Hunting down crypto miners on Linux using Microsoft’s Azure Security Center
This year, crypto mining replaced ransomware as the top cybersecurity threat. Malware, in general, accounts for the majority of cybersecurity threats, but crypto mining reigns supreme.
The challenge is detecting these crypto miners, as they get covertly installed on your machines. They might not be doing harm directly (eg: stealing your data) but they are taking away valuable computing power from your environment, leaving no compute cycles for your business app, and potentially increasing your cloud computing bill altogether.
Azure Security Center, which has great support for Linux and container technologies, can help you hunt those miners down.
A third of all malware is crypto mining
General malware, which includes both ransomware and crypto mining, made up 52% of cybersecurity threats in the first half of 2018, according to Webroot’s Mid-Year Threat Report Update. But crypto jacking — the nonconsensual act of crypto mining someone else’s machine — alone accounted for 35% of threats.
“In either case, it may be largely invisible to the end user, who likely won’t notice a small spike in their electric bill. But for an organization, power bills can skyrocket, especially when criminals employ scaling, i.e., keeping the drain on the CPU minimal when a keyboard or mouse is being used but scaling up to 100% at other times.”
How high CPU utilization metrics leads to detection of a crypto miner
We’re running a combination of Linux and Windows systems and servers on Azure, and together they are hosting our business application. The application has been performing slowly and logging into the Azure Portal we notice an Alert in Azure Monitor about high CPU utilization on one of the Linux servers:
DaVinci is not who we think he is
On examining the regular process and daemon list, we see nothing out of the ordinary. However, when we start investigating what else is running on the machine, we find a Docker container under the name “nervous_davinci”. Microsoft provides a solution for Container Monitoring as part of Log Analytics and provides details about the containers running on the machine. The associated CPU time of the container is 132%, and it’s executing a command that start with “cpuminer”, which raises a red flag immediately.
Both Log Analytics and Azure Security Center share the same (OMS) Workspace, thereby centralizing information collection. We head over to the Search function of the workspace and run a query to get more information about the container: when it was created, etc.
ContainerImageInventory| where Image contains “miner” and Running == 1
Kusto — Microsoft’s new powerful query language
The query language you see above is called Kusto Query Language or KQL in short. The query language was first developed by Microsoft internally, for search through large sets of data. It first surfaced externally when being used in Microsoft’s Operations Management Suite (OMS) which later became Azure Log Analytics. Today it is being used more and more in Azure services, for instance in the newly announced Azure Data Explorer. If you already know SQL, you can use this cheat sheet to quickly start using KQL.
Let’s do some more hunting
We validated that indeed there was a Docker container active on the Linux server, running the CPU miner by logging on the machine itself:
So let’s dive a little deeper. We can use another KQL query to get some more information on the ‘cpuminer’ process and the parameters and switches it is using:
ServiceMapProcess_CL| where ExecutableName_s contains “miner”| project Computer, CommandLine_s
It looks like it is connecting to digihash.co and using tcp port 3012. It also seems that it is mining based on a qubit algorithm. And there is information on the users’ crypto wallet. By going to the DigiHash website we learn that it is indeed a coin that is being mined. We resolve the domain name to an IP address:
We can run this KQL query to find all of the machines that have connections to this mining website. Luckily it is only this one machine:
VMConnection| where DestinationPort == 3012 and DestinationIp == “168.61.212.228”
Azure Security Center
Heading over to Azure Security Center, we found that it had already been raising alerts:
Clicking on such an alert will give you details about what is happening:
The great thing about Azure Security Center is its automatic ability to “group” alerts into an incident. This incident can relate to lateral movement for instance, and helps you ‘connect the dots’ on what is happening on your environment:
This way we found a suspicious event around the same time, relating to a possible SSH brute force attack:
Going from reactive to proactive
Because we were being reactive to Azure Security Center, eg: only logging in and viewing the alerts every once in a while, or when an incident occurred, we concluded that we needed to become more proactive about it. We used Azure Security Center’s playbook functionality in combination with Twilio to achieve that.
Playbooks are essentially Azure Logic Apps that you can run when an Alert is raised in Azure Security Center. An easy way to create them is to use the visual designer. You select the “when a response to an Azure Security Center alert is triggered” object as your start object, drag and drop a follow-up action (in this case: sending a text message notification to the engineering team via Twilio), and use the dynamic fields to pass information on to the next item in the ‘chain’.
Seconds after the alert has been raised in Azure Security Center, the text message arrives on your phone:
And if you have a smartwatch connected, the Azure Security Center alert shows up as well:
Linux detections try-it-yourself
Yuri Diogenes, a senior program manager at Microsoft who focuses on Azure and security, wrote a Linux Detections step-by-step workbook for Azure Security Center. If you want to try the Linux detections yourself, Yuri guides you through setting up a Kali Linux and Ubuntu Linux VM on Azure and some samples commands and scripts to trigger alerts.
Want to dive deeper? Yuri Diogenes and Erdal Ozkaya co-wrote a book on attack and defense strategies which features Azure Security Center. A must read!
Happy threat hunting!
— Maarten Goet, MVP & RD
