Distributed Denial of Service (DDOS) has drawn attention lately with incidents ranging from Anonymous taking down the Visa and Mastercard sites as retribution for cutting donations to Wikileaks, to Wordpress being attacked by the Chinese. A talk at the DC4420 meetup in London described DDOS as the modern political protest, comparable to a crowd protesting on Oxford street. The protest means that some people cannot go shopping, and there is media attention drawn to the cause; Paypal goes down for a few hours, the techblogs, Twitter and eventually old media play a similar role. In addition, the few million the site loses to downtime means that they may think twice about bowing to pressure so quickly from a US senator. Regardless of motives, if you operate a major website today, especially one where every minute of downtime has an impact to the bottom line, DDOS protection is something you have to think about.
The most important thing to remember is that no DDOS protection strategy will be 100% effective. If the attackers can throw 100gbps at you for long enough then you will eventually go down. For most sites paying for that kind of redundant capacity is just not economical. But it's not just brute force, mitigating resource exhaustion is also an important consideration. Throwing a massive amount of network traffic from all around the world will eventually work but it is expensive and/or hard to coordinate. If an attack can simply exhaust the CPU, memory, file descriptors or target services such as DNS, they still achieve their goal.
In designing a DDOS protection strategy for a recent project my objectives were:
- Run faster than the guy running from the bear e.g. Paypal did not go down in the Wikileaks fallout
- Survive a sustained attack - at least for 24 hours and if possible a week. If a botnet is being hired then the longer it needs operate and the larger it needs to be, the more money it is costing the attacker. Even for crowd sourced botnets like the low orbit cannon, timing is critical. A political statement is only powerful if you can announce your message and then deliver while the fickle newscycle is paying attention
- The cost for DDOS protection should be less than the downtime. The total cost of protection options can be annualized and then calculated as a cost per hour or minute. The cost of downtime is usually calculated as part of of DR or availability SLA's and even if you cannot get exact costs you should be able to make range estimates. If there is no downtime cost then there is no need to have DDOS protection. For example the Visa and Mastercard sites going down, they do not process any payments through those site so who cares? If Paypal or Amazon went down on the other hand.....Assuming there is a downtime cost your goal is something like this:
With a focus also on simplicity and defence in depth, here was my protection strategy:
- DDOS filtering service - provided by the ISP or a specialist firm such as Prolexic, Verisign, Dosarrest etc. An ISP is the most simple and most large ISP's will offer this service. Specialist services will require DNS redirection or GRE tunnel for BGP route advertisement. The service should include detection and rapid response. When a DDOS attack is detected they should have a network to filter the DDOS traffic before it gets to your network. A good post on Hacker news also backs this as the number one strategy and recommends installing the tools to be able to describe to the protection service with precise and useful data on why you think you maybe under DDOS attack if they have not detected it and not just help I'm under a DDOS attack
- Capacity to not to fail immediately - border routers, firewalls, IPS, web application firewalls need sufficient capacity to not be immediately overwhelmed. These are not so much for DDOS protection but as they are between the attacker and your servers, their capacity needs to be a consideration. My line in the sand was 5gpbs and 10 million packets per second (pps) for all in-line network kit for March 2011 considering the cost benefit equation above
- DOS protection features in network appliances - virtually all modern firewalls and IPS have some DOS protection features. A web application firewall can also perform some layer 7 defence and specialist appliances such as Cisco Guard also exist (Edit: Cisco Guard is now end of life and you could try something like the Arbour TMS). For example we have a Big-IP F5 performing load balancing and with IPS and web application firewall modules. The old school ping of death, syn floods, smurf attacks are tickbox options and not representative of a DDOS attack today but still why not enable them. The web app firewall can also perform rate limiting on IP or URL, black list IP's when it reaches a transactions per second limit or certain latency. Both of these will require monitoring and tuning and may not help you greatly against attacks launched from numerous unrelated endpoints but it is protection against the lone gunman and defence in depth. Mod evasive and other Apache tuning can provide similar features if you do not have an appliance.
- SSL accelerator - hat tip to to the DC4420 talk by THC for this one, he calculated a SSL connection takes 15 times more server resources than client resource. On a low power laptop he could generate 1000 SSL connections per second (cps), a small web server (think Amazon EC2 small allocation) could only manage 80 cps. However with an SSL accelerator it increased to 60,000 cps, so deploy an SSL accelerator
- Rehearsed plan - having DDOS monitoring and response in the security incident plan and rehearsing it. Asking penetration testers to specifically look for architectural and technical DOS vulnerabilities and advising on remediating them
Another mitigation which was not right for this project but worth exploring:
- Redundant ISP's with Nginx cluster - a great approach detailed here. Basically involves using a reverse proxy with web servers hosted at multiple ISP's:
- Roboo - HTTP robot mitigator - open source tool that uses a challenge response mechanism to identify HTTP robots. Good powerpoint deck also on DDOS presented at Blackhat Europe 2011. Works on the Ngix webserver so really good if you also using the above strategy.
Photo credit: Alatryste Flikr