We talked in depth about load balancers in use for distributing traffic to a web app the last time after I screwed up my DevOps interview. For this I would like to look a little deeper in some of the issues surrounding load balancers; Specifically how we can use them to mitigate DDoS attacks. There are many ways in which to breach and cause havoc to a web app. One of the more popular ways to make web admin’s lives harder are distributed denial of service attacks. We are going to explore some of the finer points in conducting one of these attacks, then we will explore the effects of such attacks on our differently configured load balancers.
What the fuck is a DDoS attack? You know the answer to that however, is does warrant explaining. A typical denial of service attacks (DoS) is as simple as the name suggests. It is an attack in which a host become unreachable through a flood of requests from a single client. It makes it so that regular users cannot access the site or the services as it is being inundated with too many request. A distributed denial of service attack (DDoS) is similar accept that that the attack is not coming from a single host. The attack is coming from a multiple hosts. Often these hosts are distributed (see what I did there?) across the internet. Often a DDoS attack is utilizing a network of compromised clients, often referred to as a botnet.
There are a myriad of ways in which to accomplish getting the web server to buckle under these attacks. Let’s go over some of the ways a DDoS attack can be conducted.
This attack uses errors in the TCP protocol’s “three way handshake” to tie up resources on a server. In a three way handshake the client sends a SYN request to the server, the server then replies with a SYN-ACK or SYN acknowledge message back to the client. Finally the client sends an ACK message back to the host. In a SYN flood situation many clients are sending SYN requests to a host but are not responding with ACK responses keeping the server waiting and tying up resources. Often times these clients are using spoofed IP addresses for better opacity.
This strategy utilizes the ICMP of PING protocol to occupy the resources of a web server. The clients send multiple messages to the web server as fast as they can, often without processing the replies. It hits the web servers on two front; The server is receiving and processing the messages and also sending echo reply messages back further bogging down the server.
UDP is a sessionless protocol, and in a situation where multiple clients are sending a barrage of UDP packets the server checks for the application running on whichever port is indicated. Usually there is nothing there however the act of checking clogs up resources on the web server
HTTP Flood Attacks
HTTP flood attacks are used in a far more targeted manner. Often times they utilize specifically designed GET or POST requests to be able to get the allowable maximum of data from the web server.
Zero Day DDoS Attacks
Who doesn’t love a good 0day attack? You have no idea what your attacker is exploiting and they have a sophisticated (for those who know what they are doing anyway) botnet sending a deluge of requests to your web server choking it alive. These usually take advantage of a yet unknown configuration or error in protocol and use it to have the clients send multiple requests using that error.
Using Load Balancing to Mitigate DDoS Attacks
So now that we’ve gone through some of the attack techniques that are used for DDoS, we can get an idea of what is required for hitting a single web server. Now most hostnames that have a large amount of users and prioritize up time use a distributed internal network made up of nodes (servers) which handle traffic sent by the load balancer. The load balancer being the public facing server that takes in and distributes the traffic coming from the internet. See load balancers.
TCP SYN Cookies
There are several defenses against these types of attacks using load balancers. They vary depending on the kinds of algorithms that may be running. One way in which to defend against DDoS attacks is by utilizing TCP SYN Cookies configured within our load balancer. We explained above how a SYN flood attacks occurs, however this defense works by smartly assigning a secret math function to the SYN-ACK response. Once the ACK response is gotten from the client, it is checked against the function inserted to determine if the reply is legit or not. If it checks out the connection is made.
Normally the numbers in the TCP sequence response are generally randomized and with some tools (NMap) is used to determine the OS of the client. In this case the response is specific. If the ACK response is not correct the connection is refused. This is fundamental now in defending against DDoS attacks.
Class-Based queuing also known as Class-Based Weighted Fair Queuing in Cisco IOS devices is an algorithm which divides all of our incoming traffic is distributed among classes, all of which are given the same amount of bandwidth. The classes can actually be classified in a number of ways including hardware, protocol, IP address, and priority. It takes full advantage of the IP layer which is runs, and is useful for any layer two device and any IP protocol including TCP and UDP.
One great use of CBQ against a DDoS attack is it’s core ability to be able to break up traffic into classes, as stated above. However the best part about that is that maximum and minimum bandwidth can actually be assigned as a class. Great limiting some of the crazy amount of packets coming down the pipe. How classifying traffic works means you assign the allowable bandwidth, maximum packet count, and allow for the packets to queue up and start coming through. The packets are all subject to the characteristics assigned to that class (max bandwidth, etc). After the queue has reached its packet limit depending on how it was configured packets start to drop, unless those packets which exceed our limits are dropped into Weighted Random Early Detection, which will drop our packets to avoid congestion.
The ability to limit and classify incoming traffic and assign specific configuration options is immeasurably useful once a pattern to that traffic emerges. One of the major downsides is that in this day and age where IoT devices and many others are used in large scale DDoS, it is hard to really identify a pattern in these. Making the fight against DDoS a particularly hard one.
Null routes are interesting. Upon starting my research I was not aware of this technique however I was clued into it and it is an interesting and cheap way to have your load balancer redistribute some of the traffic coming in as a result of an ongoing DDoS attack.
A null route, also known as black hole filtering, is really pretty simple. It is a network route programming into your route tables which does not exist or goes to an incorrect address such as 0.0.0.0 or a loopback address. It acts as a firewall because matching packets are either dropped or forwarded to this address. Usually it is configured with a sort of flag to be able to send those packets into oblivion.
This is great to use in conjunction with a load balancer as it acts as an extra layer of packet filtering. The technique results in little to no performance loss. It works to mitigate bottlenecks at the load balancer.
The idea of being attacked with a DDoS attack has always been a scary one. In essence someone took the time to plan, create a botnet using some vulnerability, and write a script to be able to send packets to a target network. These days it’s much more terrifying since you can always hop onto TOR and find yourself a botnet to rent and carry out these attacks (not that you ever should, seriously don’t do that). I also need to mention that often times it does not require a lot of networking, infosec, or programming skills. It’s not accessible to your average computer user however someone with a little knowledge and a grudge (or just the free time) can pretend to be an Austin Powers villain and take down a target.
These are only a few ways in which to mitigate a DDoS attack, however they are some of the only ways on your side of the network. Often times it comes down to the ISP which is hosting all your pipes to do something. It’s a scary world, however we do have some ways to defend ourselves. Happy hunting.