Brief Vhost-13 Outages
At approximately 10:15am November 3rd one of the Crucial web servers came under a massive coordinated DDOS (Distributed Denial of Service Attack). The attack came in three massive bursts which were effective in denying httpd service to clients of that host for very brief periods of less than 30 minutes each.
The attack came in a three separate, independent bursts which are illustrated below.
Crucial Systems and Network Administrators were immediately made aware of the DDOS Attack and took action to mitigate the attack. Our Data Center, SoftLayer, immediately enacted Cisco DDOS Guard on the IP Netblock under attack. A team of Administrators went to work on vhost-13 to defend the web server and keep other services such as email online.
During each attack non-essential services are stopped. This is why you see the long blank spots immediately following the Crucial reaction to the attack. Once the system has stabilized after the attack non essential services are restarted and you see the graphing pickup once again.
We believe we are now passed the DDOS attack itself and have moved into more of a recovery mode. There is a great possibility that if you were visiting the website(s) hosted on this server at the time of the attack you would have been considered an attacker and blocked by our administrators and automated systems.
At this time, all systems are running without issue. We have experienced zero downtime since exactly 3:20PM PST November 3rd. Anyone unable to access services on vhost-13 will most likely have been blocked and need to have their IP address manually removed from the Crucial Guard. The below graph demonstrates system availability since the attack started.
The brief blank spot at approximately 1AM is due to backups and other routine systems that run at that time.
Bookmark: Del.icio.us · Digg · Furl · Google · Reddit · Technorati · Yahoo!
Subscribe Now
Subscribe to our blog by RSS or by email.
See, this is just great. Outages happen, but honesty about what happened is top notch!
Always made me wonder what are the general costs of a DDOS like this. I never understood why DDOS’ing is done in the first place. It always takes down a lot more websites than the one DDOS’ed…