A couple of days ago one of our client websites was DDoS attacked by BitTorrent clients from China. This short article explains the attack and suggests a way to better protect from this particular type of attack.
Denial of Service attacks can disrupt a service on the internet and deny users from using that service. A website has its fixed amount of valid URLs that real people with real browsers send requests to use the service. A Denial of Service attack would send a high amount of traffic to the service/website either to real valid URLs, but more commonly to a non-existent address. And the traffic originates not from your users but from the attacker at a single computer (DoS) or from attackers at multiple locations (DDoS).
As a consulting company, our clients trust us to solve problems and extinguish fires when something strange happens to their servers. One of our clients got hit by a DDoS attack, and for our consultants, it looked something like this -
There is some information there, such as the CPU and network statistics, but it is really not enough to understand what the real issue is and propose a solution. This is where we whip out our log aggregation dashboard to see all the HTTP requests to shed a lot more light on what was happening. In the access logs dashboard for this service, a regular weekend looks like this pretty colorful graph.
Wavy up and down because users are more active during work hours. Lots of green colors indicating that all is good and the business is making money. Some lighter green to show replies sent about “not modified” files (code 304), when browsers already have the file cached, correctly configured caching is excellent. A noticeable lack of red and blue colors represents no errors or missing files.
Above is another graph of a weekend that was almost identical, other than those new colors appearing and that spike in “blue” and “red” traffic representing a problem for the service. Blue being requests for missing files (code 404) and clients aborting requests in the middle (code 499), and red for unavailable service (code 503).
To better show the difference between those two hours after the attack made, we also have these pretty pie charts. On the left showing regular statistics from a week before, and on the right showing statistics for the two hours of the attack. Response codes on top and requests per country at the bottom.
Normally there are a lot of successful request 200, 304 and a few missing files (404) appearing in the responses, mostly to users in English-speaking countries such as the US, GB, and Canada. In contrast to what happened during this attack, where a high number of errors (404, 503 and 499) crept in mostly from addresses in China of all places (and a bit from Japan).
The natural knee-jerk reaction would be to start blocking IP addresses of the attackers, so off we went trying to find the attacker’s IP address to add it to the firewall rules. Which turned out to not be simple at all, we just couldn’t find unique IP addresses where requests were coming from. In fact, there was just a lot more traffic coming from a lot of different IP addresses. We didn’t know these were mostly Chinese at the time.
A Distributed Denial of Service attack executed from multiple addresses has each computer send bits of traffic each, creates a heavy load in total. These attacks are more difficult to protect against, especially when there are thousands of IP addresses sending traffic all at once — adding these to your firewall rules is possible, but first, we must find some pattern to distinguish them from regular customers.
The lead developer noticed the pattern we needed, many requests were asking for addresses like example.com/announce with a user-agent of “BitTorrent” and variants.
This was a great hint about how attackers were orchestrating this DDoS and how to block it. The attackers were BitTorrent clients! Actual users downloading a movie or a video in China were also treating this service as a tracker, consuming a considerate amount of extra traffic and creating a lot of errors.
A BitTorrent tracker keeps track of where file copies reside on peers, which peers are available at time of the client request, and helps coordinate transmission of the copied files. Clients are required to communicate with the tracker to initiate downloads, and clients that have already begun downloading communicate with the tracker periodically to negotiate transfer with new peers.
— Wikipedia / BitTorrent Tracker
The application running on a web server has two parts, first is a very fast and fairly lightweight in terms of CPU consumption — the nginx proxy server. Once nginx checked that there is no real file on disk, such as css stylesheet or javascript file that can answer the request, it redirects it to a Ruby on Rails application.
The requests coming into the application for missing URLs will return an error that there is no such address (code 404). Since Ruby on Rails is much more CPU consuming than nginx, a large spike in traffic can cause it to overload and stop answering requests fast enough (code 503). In a more extreme case, the application will just crash completely.
A quick way to mitigate the incoming traffic to an address like example.com/announce is to block these requests in nginx. This is the first thing we did.
server {
location /announc {
access_log off;
error_log off;
return 404;
}
}Some requests asked for /announce, others for /announce.php and yet others for /announce the code above covers all these scenarios and makes nginx reply with a 404 error code without involving Ruby at all. Disabling the access and error log files is also a good idea, these requests generate a lot of log entries which leads to disk i/o load.
This pretty much took care of the problem. But, sending BitTorrent clients away even more efficiently is possible, stay tuned.
For a BitTorrent client to recognize a tracker is failing, the tracker should return a text response stating a failure reason, according to BEP-31 proposal there is a way to specify when next to retry the request. Making nginx reply with this information is simple, using text/plain as the content type and encoding the response with bencode as expected by the BitTorrent clients.
server {
location /announc {
access_log off;
error_log off;
default_type text/plain;
return 410 "d14:failure reason13:not a tracker8:retry in5:nevere";
}Theoretically, this should keep well-behaving BitTorrent clients from repeatedly requesting more information from a tracker. This website is not a BitTorrent tracker, so this would be the better way to do it.
Want to protect your website from torrent clients abuse? This is the way to do it!
A quote from the BEP-31 proposal created in 2009 (six years ago) reveals that this has been a way to DDoS websites for a long time now -
HTTP server that is living at those addresses will receive a large amount of /announce and /scrape requests which it cannot fulfill as there is no BitTorrent Tracker present at that web server.
Most BitTorrent clients do not check the HTTP error codes provided by the server and this makes them ignore 404 (File Not Found) … Clients then keep on retrying forever till the user finally gives up.
With a large enough number of clients this might overwhelm the web server from serving the content that it is really supposed to perform.
We continued trying to find why most of the requests were coming from China. The requests to /announce are asking for information about specific files, we decided to try to see what these files are, and maybe understand why the attackers were Chinese.
Example of requests the server received during this DDoS attack -
17:54:53 GET /announce?info_hash=%1B%60%EC%C5%CB%F8s%AB%E1%86%0At%C4%9C%EA%D0p%80%EEJ
17:55:11 GET /announce?info_hash=kFHu%84Es%40%03%1CN%E2%D7%AA%C3I1w%5D%7D
17:55:12 GET /announce.php?info_hash=%b9qM7%10By(~%98%f1%89g%f8%8da%1f%acx%b9
17:55:12 GET /announce.php?info_hash=Ec%23%2cJy%e5%f1%97%d2%1eW%89%c2%13%ab%ee%1b%04%81
17:55:12 GET /announce?info_hash=kFHu%84Es%40%03%1CN%E2%D7%AA%C3I1w%5D%7D
17:55:14 GET /announce.php?info_hash=J%60%D0%8CP%BFp%B8%B5%B6%5B%E6%8BX%5F%F0%A5%AAv%5D
17:55:17 GET /announce?info_hash=%91l7%19T%a7%ee%a4%8ek%b3%00%16%8dU%27f%09%be%9d
17:55:20 GET /announce?info_hash=J%60%D0%8CP%BFp%B8%B5%B6%5B%E6%8BX%5F%F0%A5%AAv%5D
17:55:20 GET /announce?info_hash=ODb%7b%e8%d1%ad%04%dc%2b3%c4%a7e%10%abg%d1%ceVEven easier to get all the hashes using a bit of sed scripting.
sed -ne '/\/announc/s/^.*info_hash=\([^&]*\)&.*/\1/p' access.logThere were 12,739 unique hashes — which translates to roughly the same amount of torrent files poisoned with a fake tracker.
Each hash link to the original file, so we went to find out which files were falsely claiming a false tracker. In the world of BitTorrent, one popular way to link to files is the “magnet” resource notation. Magnet supports multiple ways to reference a file, and one of them particularly interesting is by specifying the info_hash, such as -
magnet:?xt=urn:btih:INFO_HASH_AS_HEXThe requests contain encoded hashes for URLs and before we can make magnet links out of them a small conversion is required, using Ruby -
require 'cgi'
def magnetize(url_info_hash)
"magnet:?xt=urn:btih:" + CGI.unescape(url_info_hash).unpack('H*')[0]
endPlugging these magnet links into a BitTorrent client that has DHT (distributed discovery protocol) can eventually discover the original files, which we did. Results are not surprising, most of these files are Chinese and Japanese NSFW videos that get published on popular torrent trackers and have a lot of people download. Injecting a new tracker via “Peer Exchange” is just a right-click away in almost any popular BitTorrent client like μTorrent, and an attacker can add any address, including web servers that have no torrent trackers on them.
Using popular Chinese videos also makes sense, unsurprisingly China is big and popular files create a very large pool of clients unaware that their tracker sends requests wherever the attacker wanted them to go.
Update: After some more investigation, this is probably caused by the Great Firewall of China DNS spoofing. Some old torrent files that circulate in China have tracker.thepiratebay.org specified as one of the trackers, and resolution of this domain (blacklisted in the GFC) returns random IP addresses because of DNS spoofing.
Originally published at www.prodops.io.
