Service disruption In Progress
Priority - High Affecting Other - Website hosting

The report for this issue is as follows :-

Between 12:09 and 12:42 various websites across our different platforms experienced 500 / 503 errors. The root cause of these errors was identified as a large botnet targeting a huge number of different websites on our network. Unlike the hundreds of attacks that we successfully mitigate each month, this attack had no specific origin or target. It was not a volumetric attack designed to overwhelm network capacity, but instead made proper, valid HTTP requests to thousands of different websites from thousands of different IPs to make it look genuine. The requests were so specific to ensure a single IP or domain was not targeted regularly, that our automatic defence systems saw the traffic as genuine, and no network attack sensors were triggered. Many requests also came from a network that we handle millions of requests from every day and so no request threshold was exceeded. Finally, as the requests were not from a specific netblock or towards specific websites, it was not possible to automatically redistribute the load or re-route the network traffic in order to process the requests. This ultimately resulted in a denial of service towards a large percentage of our processing capacity, resulting in the 500/503 errors experienced. Once our system administration team had identified enough of the malicious requests to identify the botnet pattern, they were able to deploy rules to the edge of our network. Once these manual block rules were in place, normal service resumed. We apologise for the inconvenience we know incidents like this cause. We've already made changes to our sensors to help identify any future re-occurrences and will continue to take steps to improve our platform's performance and availability moving forward.

1:00pm

The issue should now be fully resolved. A brief report will be posted here when available.

12.45pm

Websites should now be coming back online. Final resolutions are being implemented now and we will provide a final update in due course.

12:30pm

Progress has been made and more information will be provided shortly.

12:10pm

We have been alerted to an issue where some websites on the the following hosting plans are experiencing disruption. Nickel, Tungsten, Cobalt, Platinum, Iridium.

Email and VPS website hosting services are not currently affected.

We believe this to be a temporary problem with network connectivity at one of the datacentres and should be resolved shortly.

This notice will be continually updated as and when we obtain further information.

We apologise for any inconvenience and appreciate your patience.

Kind regards,

The Pushlogic Ltd team