Hacked? When in Doubt, Nuke it. It Will Make You Feel Good
Many people ask questions like “How do I know if I have been hacked?” or “What do I do if I have been hacked?” There is an easy answer to these questions — Nuke everything and start over. Then you can be fairly certain everything is okay. (For a while, at least!) This may sound a little extreme, so I will explain.
I do development. I do hacking. Often, I do them at the same time. From time to time I also conduct crazy mad-scientist experiments. All of this stuff I do typically makes a great mess out of the computer systems I work with. I wish I had unlimited resources — a huge abundance of computers, network gear, and storage. Alas, I do not. Resources cost money, and money is a finite resource. What I do have are recovery plans. That is, when something goes bad (and it always does), I have a way to get back to “normal.” Best of all, I know my recovery plans work.
How can I be so certain? Because I recently nuked one of my systems, and everything recovered just fine. At any given point in time, I make sure there are at least five copies of the important business stuff stashed in no fewer than three different physical locations. I also ensure that, in the event of a catastrophe, I have a way to restore things back to normal, and that I have tested this method and feel confident it works.
Okay, okay, everybody says backups are important. And how does having a backup help you know if you have been hacked? The answer is surprisingly simple — it does not matter if you have been hacked or not. If there is something odd or strange going on with one of your systems, nuke it and restore it to a normal state.
This may sound a little radical, but it’s a sure-fire way to get peace of mind. Both Amazon and Netflix rely on the principles of automation and restoring to a known state. Don’t juggle 10,000 variables and spend countless hours, days, or weeks worrying about whether some mysterious behavior is from a malicious intrusion or from a misconfigured computer. Get a mindset that says “This is the way things should be working, and if they are not working like I want them to, then I will nuke the resource and restore it to a known state.” It might sound a little scary, but embracing this mindset is surprisingly liberating.
By Thomas Phillips, CTO at Ridgeback Network Defense, Inc.
