Security basics: DDoS attacks

No network is safe from hackers these days. In 2011, the famous blogging service WordPress.com was unavailable for an hour. And it was confirmed by Automattic (WordPress’s parent company) that the site was targeted by an extremely large DDoS attack which is affecting connectivity in some cases. The size of attack was so huge that it choked the server bandwidth, preventing its users from using it.

DDoS attacks are popular because of their simplicity. During a DDoS attack, hackers basically generate a very large number of requests to a server in order to overwhelm it. The main purpose of these kind of attacks is to prevent legitimate users from accessing the server. The attack can be initiated through any individual system or group of systems or crawlers and can use any mechanism to flood servers with huge amount of information and choke it.

DDos: A distributed denial-of-service (DDoS) attack is a malicious attempt to disrupt normal traffic of a targeted server, service or network by overwhelming the target or its surrounding infrastructure with a flood of Internet traffic.

Detection of a DDoS attack

In case such an attack is initiated from a single system, the simplest way to detect is to monitor if there is burst in request from a particular IP. If so, we can easily block that IP to mitigate the attack. But sometimes the attacker can use other users’ machines (zombies) to fire requests from multiple systems (with different IPs). And in this case, it becomes difficult to identify the (IP address of the) actual attacker. It is possible that even the zombies are unaware of this.

Attacker using zombies system to fire multiple requests (src)

Logically, it is relatively easier to identify and mitigate the bigger attacks from small number of systems rather than the other way round. For example it is easier to identify an attack originating from 5 machines sending 200 requests per second concurrently, rather than that originating from 200 machines sending 5 requests per second concurrently. In both scenarios, the server receives 1000 request per second, but in first scenario we can easily find and restrict 5 machines (since the number of concurrent requests is high), but in the second scenario it is much harder.

Mitigation of a DDoS attack

One way of mitigating a DDoS attack is by using load balancing. Load balancing is a technique to redirect incoming traffic to multiple servers. However, this can only be used if we have multiple servers at our disposal. In that case, if the attacked server goes down, traffic can be redirected to other servers. But, this is not full-proof solution. If the attacker targets our domain instead of a single server, then every time one server goes down, the requests (and hence the attack) will be transferred to other servers. And in the end, the attack will crash all the servers.

The other way is by using NGINX’s (or any server’s) rate limiting feature. Using this feature we can control the maximum incoming traffic from a source, and any additional traffic can be throttled to prevent the server from going down. This requires the source(s) of DDoS attacks to be identified so that the traffic from it/them can be filtered out.

NGINX Rate limiting feature

The ngx_http_limit_req_module module (0.7.21) is used to throttle the request processing rate per a defined key, in particular, the processing rate of requests coming from a source (mostly identified by a single IP address). This limit is set by using either “Leaky bucket” or “Token bucket” algorithm.

Leaky bucket algorithm

Leaky bucket (Img src)

Here the bucket resembles the throttle which is receiving random requests at random rate but processing at a uniform rate.

Token bucket algorithm

Token bucket (Img src)

Token bucket algorithm is used to try to control the behavior of streams on the IP-networks. By granting a stream a fixed number of tokens per time interval, one can effectively limit its bursts and the number of packets it can send. One has to spend one token per packet sent out on the network, the tokens are thus used to pay for ones use of network resources.

Leaky bucket sends output stream at a constant rate whereas in token bucket it’s variable. But when there is a burst of requests, Token Bucket can send bursts at faster rate as compared to leaky bucket. To understand it better, consider a scenario. Suppose 10 users land on the website concurrently. If the algorithm implemented is leaky bucket then requests will be processed one after another like FIFO (First In First Out) and the user (generally the last ones) might feel that the website is slow. But if the algorithm implemented is Token bucket then a burst of requests will be processed and apparently all the requests will be processed sooner.

In these algorithms, we define three things; rate, burst and nodelay in order to control which requests and in what order the requests are to be processed-

syntax to describe zone

1. rate: A zone is basically a bucket that controls the requests received by the server. When we define a zone, we can also specify on what basis requests are to be limited (based on IP, user_agent etc.). And the rate limiting feature is applicable to the complete zone. In the above pic (syntax to describe zone), the zone name is “blog” whose size is 10 MB and rate is 3 requests per second or 1 request every 300 milliseconds.
2. burst: This is an optional parameter, which defines the number of additional requests we can queue over the base rate. These additional requests get added to the queue and are processed later. Let us assume that we have burst = 2. And since the base rate is 1 request every 300 milliseconds, so now the server can handle upto 3 request for the every 300 milliseconds time span depending on the availability of burst tokens (once a request is added to a burst token, the token becomes unavailable till the queued request is processed).
3. nodelay: It is also an optional parameter, which defines how to process incoming request. If nodelay is set then the requests gets processed as soon as they arrive.

Rejection or acceptance ??

Lets consider the following NGINX conf changes for rate limiting feature

limit_req_zone $binary_remote_addr zone=blog:10m rate=1r/s;
location / {
limit_req zone=blog;                    ------ Case 1
limit_req zone=blog burst=2;            ------ Case 2
limit_req zone=blog burst=2 nodelay;    ------ Case 3
limit_req_status 503;
try_files $uri $uri/ /index.php;
}

And following graph is for the number of incoming requests per sec.

And as you can notice that we have a total of 47 requests in 10 sec.

Case 1:

Rate = 1r/s

Rate 1r/s

As we can see in the above graph the at every time span only 1 request is getting processed and the rest are rejected. So out of 47, 38 are rejected.

Case 2:

Rate = 1r/s and burst = 2

Burst is bit tricky to understand. But let’s try.

At the 1st sec, we have 10 incoming requests. We have a burst of 2, so NGINX will sent 1 request to the processor, will keep 2 in burst and the remaining 7 will be rejected.

At the 2nd sec, we have 3 incoming requests to process. Note that the rate for processing is 1r/s. So by the time the requests arrives for 2nd second, the processor has processed the previously allotted request. At this very moment the NGINX will fetch next request from the burst and send it to the processor. And at the same time NGINX will allocate empty burst token to the incoming requests. Here 1 request gets the burst token and remaining 2 will be rejected.

And this keeps on going. In this feature, the number of requests rejected drops to 35 and the requests being processed at rate of 1 for a second.

Case 3:

Rate = 1r/s and burst = 2 and no delay

The concept of burst remains the same. But what’s no delay??

Out of 10 requests 7 were rejected i.e. users were restricted, 1 is in the processing state i.e. user will receive a response in a second. What about the 2 in burst? NGINX has accepted their request but will process after certain time. Isn’t that bad for a user? User might feel that website is slow and may drop.

Now this is where “no delay” steps in. By adding “no delay” feature, NGINX processes the request as soon as it arrives but the burst token will be released at 1r/s.

So no change in the way of allocation or release of burst. Only the wait time decreases.

If the request is not processed as soon as they arrive then the User might feel that website is slow and may drop

Let’s compare

Time to do some analysis. Server received 47 requests in span of 10 seconds.

1. Case 1: No options Requests rejected = 38, which is 80%. And 9 requests gets processed in 10 seconds making average processing rate 0.9 requests/per sec.
2. Case 2: Burst =2 Requests rejected = 35, which is 74%. And 12 requests gets processed in 12 seconds making average processing rate is 1 requests/per sec.
3. Case 3: (Burst = 2 Nodelay) Requests rejected = 35, which is also 74%. And 12 requests gets processed in 10 seconds making average processing rate is 1.2 requests/per sec.

Conclusion

DDoS is one of the oldest and easiest way of attack. And with the increasing technologies and features, it has become quite easier to mitigate those attacks. Implementing DDoS using NGINX requires lots of inputs, use cases and test. I would suggest you to play with the NGINX rate limit feature and you can find some code for the same here.