The Tale of a Hacked VPS

When Vultr came out with their $2.50 a month VPS a while back, I jumped at the chance and put dokku on it to use it as a testing/staging ground for my pet projects. As of this morning, the only long-term tenant on it was a small Node.js app which forwarded texts to a Korean cell phone to an American one. (Long story short, I’m currently in an environment where I only have an access to a phone that can’t make calls but can send texts.)

The Hack

This morning, I noticed that my app wasn’t responding quickly or sometimes at all to my texts. When I logged on and hit docker ps, I saw an unfamiliar entry. (I failed to take a screenshot before I panicked and ran docker kill, which I'm still bummed about.) When I looked at its logs, it was communicating with "xmr.crypto-pool.fr." The word "crypto" immediately raised red flags in my mind - is this a malware making my server mine some cryptocurrency for this attacker? I then looked at the CPU usage logs provided by vultr:

Yup, someone is using my server for mining.

Then, I looked at the SSH access logs (sudo grep "Aug 4" /var/log/auth.log | less because I knew from the CPU logs above that this started happening on Auguest 4th):

which is, I guess, expected — bots try to brute-force your root password all the time. However, I was surprised to find these lines (there were more than one):

Somehow, my private key was compromised. This was a little scary to me, since I use this key-pair to do more than just push to dokku. (Thankfully, I only used it for pet projects — it wasn’t associated with my Github account).

Recovery

In situations like these, I was taught to “go nuclear,” so to speak. I re-imaged the Vultr instance as soon as my curiosity of how this happened was satisfied. I deleted the SSH key-pair that seems to have been compromised and stopped it from being used anywhere. I plan on re-imaging even the machine which was storing the compromised key to be extra safe.

Lessons

I am humbled by the dangers of being connected to the Internet and am reminded of the precautions we must take when setting up a public-facing server. These are the lessons I’m taking away from this scary (but honestly, quite fun) episode. I’ve found (this DigitalOcean article very educational.

1. I’m always going to set PasswordAuthentication no from now on. Bots are always going to try to bruteforce your root account - while I know mathematically they're incredibly unlikely to succeed, I'm not gonna give them even the satisfaction of trying. (This DO post has some good instructions)
2. I haven’t been setting a password for my private keys because… laziness. I really should do that from now on.
3. Unrelated to this specific break-in, but I’m going to start configuring UFW on my servers, unimportant hobby instance or no.

It’s a jungle out there… everyone be careful, and take care. :)

Originally published at blog.alexpark.me on August 5, 2017.