3 simple steps to take back your servers after a hack
Hiring external investigators or incidence responders can be expensive sometimes especially for small and medium-sized enterprises when there is a breach. The good news is: it’s not a rocket science and you can bounce back by following 3 basic steps in this post.
It’s tempting to just focus on keeping the attacker out the system alone as fast as possible and lose sight of the bigger picture. There is no other quick way of doing this than creating a fresh instance of your server and restore your files from a backup made before the intrusion. However, it’s not advisable to go this route because:
- You don’t get to find the holes that let the intruder in and unavoidably the holes remain. With this approach, the bad guy comes back in a couple of days.
- Extent of data theft can not be determined
- Intrusion date can not be deduced
Here are three (3) basic steps you need to follow to recover fully from a server breach.
1. Investigate:
Carry out a thorough investigation, a whole lot of servers have been compromised by exploiting vulnerabilities in the application code. You should look for files that were created or modified prior to and after the attack took place. Investigate the content of these files thoroughly. I have had to deal with a compromised server where the attacker exploited a vulnerability in the application code and uploaded a file that simply downloads a text file containing shellcode, writes the content to application root directory, and rename it to a friendly name to avoid suspicion.
Open ports and services running on your server are equally important and are worth poking a nose into. Too many open ports are like having multiple entrances into a building, the higher the chances of you leaving one or more of these entrances unchecked making bad guys enter in all smiles. Probe services running on these ports. Are they vulnerable? Do you really need them? Are they critical to the business? Are there any known vulnerabilities with the versions you are running? Attackers may have left a backdoor, this can only be uncovered through investigation.
Investigation process can be eased by using some open source tools like RootCheck, an open source command line tool that looks for indicators of compromise on Linux or BSD systems. It tries to find known backdoors, kernel-level rootkits, malware and insecure configuration settings. RooTCheck is included as part OSSEC (an excellent host-based intrusion detection system) OSSEC provides continuous endpoint monitoring, alerting and visibility into your environment. Lastly, examine your server access log, grab attacker’s IP address, you might need to hold someone responsible later.
2. Containment/Isolation
Without a thorough investigation, It’s practically impossible to know how many of your servers are affected by a breach hence making containment or isolation a losing battle.
The second step you would want to take is to isolate all affected systems, change all access credentials, stop all communications from affected systems to others and if possible disconnect them from the internet.
This step is critical as it prevents attackers from using them to further attacks or compromise other systems in your network.
3. Recovery
George Polya, a famous twentieth-century mathematician, said the first step in solving a problem is to understand it. Recovery stage assumes you have understood when and how the attack happened, vulnerabilities that let attackers in, the extent of damage done and how many users’ account were compromised? When you know all these you can better put your house in order, and blocking loopholes becomes easy than what you are thinking right now.
If vulnerabilities exist in the application code you should avoid going live without patching them, you might even want to go for a full code review and intensive application audit. The loophole may have come from your developers using vulnerable libraries or plugins in the application code.
You don’t own or control the system anymore if a rootkit backdoor was installed, a rebuild is recommended. In addition to this, it’s equally important to remove all vulnerable services you have running and close all unused ports. It’s important not to be tempted to rush back online at this point until you fix all issues identified during the investigation.
