Controlling Virus Outbreak — a real world example

If you have been working in iT for a few years and have not come across a virus outbreak, then you have probably spent most of the time surfing the Internet or not been good enough to notice what was going on! In the organization I have worked with over the last 5 years, we have only had two major virus outbreaks, Zotob and Sasser.

Sasser kind of caught us with out pants down and all hell was let loose as we realized we had to rapidly patch over 100,000 PCs with Windows 2000 Service Pack 4 before we could then apply the correct Microsoft security patches to halt the virus outbreak. Zotob was the same — but we were prepared and the infection was not so bad.

As a Remote Support Specialist, I was called into action to provide as much remote resolution as possible before we had to pull field engineers to sites to manually patch and clean PCs. I have listed five things that we employed to manage the workload, slow the spread of the virus and eventually keep it all under control.

1. Shutdown

When we were hit with the Zotob virus and everyone’s PCs started shutting themselves down after a few minutes, we initially thought we were hard pressed to find a remote solution. We examined the timing of the event and determined that the login script ran before this event took place. All of our users had the SMS.BAT login script as part of their login process and therefore we decided to use the SHUTDOWN.EXE file from within that login script to stop the PCs from being switched off.

SHUTDOWN -a

This command will abort any shutdown process that is running. If you action this at the right time, then the shutdown process cannot continue. In our case, the SHUTDOWN.EXE was copied to the NETLOGON share on the server and run from there.

2. Lockdown Ports

To stop Zotob from traversing networks and sites, we took the decision to block the relevant TCP and UDP ports on the routers. In the case of Zotob, this was port 445. However, this did have a knock on effect that some sites were affected by being unable to access network printers and server shares — but this is not as important compared to total network meltdown! The ports were kept locked off for over two weeks in the end before we were happy that the infection had come under our control.

3. Payment

Servers were also affected by infection. Upgrading servers to new service packs before being able to patch them with the relevant security fix from Microsoft was a real no-no idea considering there were over 5000 servers in existence on the network, all running different processes, all slightly different — but all running an out of date service pack. In the end, we went “cap in hand” to Microsoft who supplied us with a copy of the relevant patch that would protect the servers. Of course — this was not free. I did hear the sum paid for this special fix — and it was a 5 figure number! Sometimes it is worth paying for these things rather than putting your servers at the mercy of rapid Service Pack upgrades.

4. Remote Resolution

As part of the Remote Resolution team, it came to us to develop and deploy a rapid fix for the situation we now saw ourselves in. We had a mixture of clients. Desktops and laptops on the network and remote desktop and laptop users too. We needed to apply patches and fixes to them all — and fast. We developed a fix methodology for the networked clients as follows:

1. Kill the running virus processes on the remote PC using PSKILL.EXE — mainly to stop the CPU maxing at 100% so step 2 was easier.
2. Gain remote access to the PC using Dameware.
3. Use RunAs to launch an Internet Explorer window as a local admin account (shift right click Internet Explorer, choose Run As).
4. Download the latest service pack from the local server or intranet site.
5. Download the relevant patches and virus clean utilities.
6. Run the service pack upgrade, reboot and log on as local admin.
7. Repeat step 1 again.
8. Install the relevant patches and reboot.
9. Repeat step 1 again.
10. Clean the PCs using supplied cleaning tools from Symantec.

In the end we also found we were able to remotely update the Symantec Anti Virus definition files by copying the installation file to the PC and running it silently in the background using the /S switch. The Symantec clean utilities could also be run silently too if need be. This aided our ability to capture and quarantine the virus whilst we worked on the patching process.
5. CD distribution.

For customers whose PCs were not attached to the network, we developed a CD that would autorun, log on as a local admin or promote the current user and then upgrade, patch and clean the PC. This was not perfect as it had to be developed for various different PC types and builds — but it worked. We had to mail the CD out with instructions and quite a few calls were raised from issues the CDs created — but it was expected not to run perfectly.

Conclusion:

I cannot stress enough the role of the Remote Resolution teams had in controlling the outbreak. Managing the tasks using automation scripts and batch files meant that we had the fixes and were able to apply them quickly without much impact on our other workload. The correct documentation of the processes also provided to the 1st line helpdesk staff with the ability to fix them too.