We’re sorry that this morning’s DDoS attack caused an extended outage. 99% of sites are back up and running normally. We know that your websites are important, so we take these disruption very seriously.
First a note regarding sites still down on w08
Each of our web servers uses 10 different shared IP addresses. When we identify the target IP of a DDoS attack like we experienced today, we null-route (deactivate) that IP. Unfortunately, that impacts not only the target site, but the other sites using that shared IP. That null-route can last anywhere from a few hours to 24 hours, depending on how persistent the attackers are. If it persists more than 24 hours we can take further steps to start moving sites off the null-routed IP. We’re very sorry about the 100 or so of you whose sites are affected by the null-route. We hope to have you back up and running soon, but this is the only way to handle an attack like this.
Why this DDoS took down the network
Traffic flows through our network via three separate Internet backbone connections (provided by CenturyLink, Internap and Time Warner). Normally those connections – along with some other DDoS mitigation methods we employ – are more than adequate to handle the “normal” DDoS attacks we see on a rather routine basis. A large attack, however, saturates a network to the point where the routing and switching hardware at the edge of your network simply cannot accommodate the attack traffic. So the network essentially stops moving any traffic.
When an attack as large as the one we saw today happens, we have to rely on our Internet backbone providers to block (null-route) the IP under attack on their end. We can normally get a quick response from the providers, but we need at least two of them to block the DDoSed IP before our network can return to normal. That can slow down resolution of the problem.
Unfortunately, massive DDoS attacks such as we experienced today are becoming more common, and as you can see and read in the news almost every week, and they affect even the world’s largest networks. We do employ a preventative “fix” for DDoS attacks, but there is no defense against a sufficiently large attack. The only way to mitigate them is reactively, after the fact.
Again, we’re sorry the attack caused the outage that it did. We did all we could do to get your sites and email back up and running as soon as we could.