infected linode -> vmware & anti-virus that caught my in-the-wild (ITW) malware


tl;dr: Get a Kaspersky free trial. If it doesn’t slow down your system and you can live with broken exceptions (e.g. don’t scan your malware research directory) buy it immediately! Else, get a trial of ESET and purchase it at the end of said trial.

recently (ok november of last year; so relatively recently ;) i had a linode that was infected by malware. i was first alerted by the excessive amount of egress traffic being used on my account; however after missing the email notification(s), and therefore exhausting my monthly transfer limit, linode’s NOC (SOC?) shutdown my instance. no fault to them; that’s exactly what i would’ve done as well. after contacting support and having a very helpful, responsive, and friendly conversation i was able to gain access to the halted instance.

this is when i learned two very important lessons.

first, i used this incident as an opportunity to (finally!) learn how to migrate a linode to a vm, but more generically dd via ssh. My notes are as follows:

  1. create a new virtual machine based on your linode specs (i used vmware fusion pro to create a new vm with 1GB RAM, 24GB disk NOT split into multiple files; not sure this is needed, but AWS import made this a habit)
  2. boot into rescue disc (i used the latest kali iso)
  3. ssh user@A.B.C.D “dd if=/dev/xvda “ | dd of=/dev/sda (ok so i didn’t actually test this cli. instead i followed their docs, [ref:] copied to an image file first, (WHICH TOOK FOREVER!) and then i had to copy [again!] all 24GB to the new vm created in step1. hopefully the above works and saves me [and you!] some time in the future :)
  4. mount /dev/sda /mnt
  5. for i in /dev /dev/pts /proc /sys /run; do mount -B $i /mnt$i; done
  6. chroot /mnt
  7. update /etc/resolv.conf (i used and; which it turned out so did the malware! N.B. DNS is often overlooked, so be careful and don’t forget you have your local ISP; not to mention OpenDNS, etc.)
  8. yum install kernel grub2 (I was using CentOS; adjust for your package manager)
  9. grub2-mkconfig -o /boot/grub2/grub.cfg
  10. install grub
grub2-install --boot-directory=/boot /dev/sda --force

This allowed me to get a local infected vm for further research. Luckily (for me ;) the malcode persisted and I immediately started seeing a large stream of outbound traffic (don’t forget to use bridged networking in your test lab/cyber range [or not as the case may be ;]). Further, it seemed to be a different host from before which told me it was highly likely the malcode had checked into C2.

ok, so the hunt is on to find out what process. a quick look through netstat showed that ls was sending udp traffic. clearly; not normal. further, by inspecting /proc i quickly found the malcode in /boot. after copying what looked like a mktemp(3) filename off and checking it on i found a couple of shocking revelations…which brings me to the second lesson/teaching moment for this exercise.

anti-virus software is not only ALIVE AND WELL; there is very little AV that _actually_ catches ITW malcode! ok, so this isn’t a shocking revelation in and of itself; nor is it surprising which vendors caught this trojan: basically the same players we’ve all seen recommended time and time again

however, the surprise (for me at least) came ~2mo later when i re-analyzed

so the real story here is that only ~26% of vendors caught a known trojan 2+ months after it was publicly disclosed to exist! further, some of them are just re-bundles of the original 6 that have been “tuned” for performance.

my take away’s from all this:

  1. dd via ssh — very cool!
  2. Kaspersky and ESET NOD32 are quantified as #1 and #2 in my book for endpoint protection.
  3. The little guy is exposed, can, and will be attacked; even on modern, monitored platforms.
  4. Castle theory is dead; long live the castle mentality!
  5. Defense in Depth is more relevant than ever. (Embrace a more modern protection methodology/approach such as immunology or policing.)
  6. Use a SIEM framework/solution; if possible.

-183r space cowboy