The other day my coworker informs me, ‘hey, there’s a weird process making network connections on your box.’ A dreaded string of words if there ever were any for the security-conscious developer.
This is what he pastes to me:
/usr/bin/curl -G —location —max-time 60 —silent —write-out %{http_code} —data since=2009-12-31%2023%3A0%3A0 —data program=prof —data count=1 —output /dev/null http://131.159.28.73/popcon.php;
Now, I fully expected this to be some normal activity observed and misinterpreted, but as I saw this I began to grow distressed. Curl POSTing to some random IP address and popcon.php?
The computer in question was not a production box, but my development VM environment. This didn’t do much to ease my worries, though. Developers are often the weakest links in the security chain. They have access to all of the source code, the ability to push to repos and often direct access to production boxes. If my dev environment was hacked, I would not be having a fun time of it.
Now, you’re probably asking yourself: how did he know this was running? Great question!
The company I work for, we kind of make it our business to know what exactly is running on your boxes, and alerting you of strange activity. New network activity from a process, well, that falls in that category. Like any good company, we dogfood our product on our dev and production boxes. My coworker saw the email alert about the strange curl process, gave me the heads up, and I proceeded right away to investigating it by diving into our user interface:
Right away, I noticed a few weird things: it’s new activity (which I observed by viewing the lonely bar in the executable history diagram), and the user that ran it was ‘root’, which was even more alarming. Looking at the process tree, I observed that its parent process was a shell script.
In Linux, all new processes are created by fork() and execve() from a parent process. We designed our UI so you can walk up the process tree to the parent process all the way up to init (pid=1), just by clicking on the parent process in the tree. So, I did exactly that. What launched this shell script? Was it some insidious piece of malware running active in memory? A new init script?
It turns out it was a perl script, launched by cron, located at /usr/share/pp-popularity-contest/cron.daily:
Hmm. On my box, I took a look at the contents of /usr/share/pp-popularity-contest/cron.daily, and I confirmed execution of the curl command in question:
I looked at the rest of the perl script, which searches for a bin /usr/bin/pp_popcon_cnt. The script appeared to post the results of this program up to 131.159.28.73 using curl.
The script didn’t look particularly malicious, although I had no idea what /usr/bin/pp_popcon_cnt was. A quick google showed me that it was part of a valid package, ‘pp-popularity-contest’:
Further investigation indicated I installed this and a bunch of other random packages earlier while performing some file modification testing! Doh!
So, despite the questionable perl script, and despite the fact I was participating in a ‘popularity contest’ I didn’t know I was participating in, it turned out this was really something benign. And I was able to determine that with < 2 minutes of investigation, which was pretty cool.
What have we learned here?
Well, for one, that my coworkers and myself may be unhealthily paranoid. ;) Certainly, 99.9% of the processes making network activity on your machines are benign, but how can you make that judgement unless you even know what is happening?
At my previous job, we observed companies with enormous security budgets who still didn’t know they were hacked for months and months. Even once they were aware of it, they still had a really tough time figuring out what boxes the attackers actually accessed, what credentials they stole, and what malware they installed, etc.
The thing is, you can save yourself a lot of headache down the line if you assume that this kind of breach can happen and prepare for it proactively. No one wants to think their access controls will fail. But at the same time, you don’t want to hear a merchant saying, ‘sorry, we did everything you can to protect your credit card numbers, but it looks like someone got them anyway and was using them for months and months, without anyone knowing.’ A little logging and a little security monitoring go a long way.
cross posted from: http://blog.threatstack.com/labs/2013/12/26/yt-seeing-something-odd-in-the-logs
