Anatomy of a Botnet

When IT consultant David Soussan investigated a client’s network-connectivity problem, he used an Ethereal network sniffer to view network traffic entering the firewall and identified suspicious patterns. He then found the Microsoft Windows 2000 Server computer that was the source of the bad traffic, isolated a suspicious .exe file on that system, and determined that the program was a botnet connecting to a botmaster at various DNS addresses. David successfully disabled the botnet by putting it in an “isolation chamber” and blocking access to the TCP port that the botnet was using.

David Soussan got hooked on computers when he took a FORTRAN programming class in junior high. Now the owner of DAS Computer Consultants, not long ago David encountered a perplexing network troubleshooting challenge at a client site. The company's Internet access was down, and David’s initial investigation pointed to a bad firewall. However, the problem was actually something much worse: a botnet that was commandeering the company’s systems. In a recent conversation, David told me how he uncovered the worm’s trail and what IT pros can learn from his experience.

Explain the client's network problem and how you investigated it.
The client’s ISP said, "It looks like [the network is] up, but we're seeing some funny traffic." From inside the LAN, I got echoes pinging the firewall but not the WAN router. Then I tested the Internet connection by disconnecting the WAN router and firewall and attaching a single client to the WAN. The ISP was right: The Internet connection worked, so I knew the problem was related to the LAN. I reconnected the firewall to the WAN router, connected the PC to the closest switch, and still couldn’t ping the WAN. It appeared to be a firewall problem. I replaced the firewall with another product, and the LAN worked OK. But when I tested the firewall offline, it was good.

One of my favorite phrases is, "If it walks like a duck, and it quacks like a duck, chances are pretty good it's a duck." This problem looked like a duck and quacked like a duck, but it turned out to be a decoy. Even replacing the firewall didn’t solve the problem. That's when I started wondering, "Wow, what's going on here?"

I suspected that traffic on the LAN was killing the firewall, so my next step was to use a network sniffer to view the traffic going into the firewall. The initial sniff looked like a lot of junk flying around the LAN. By applying a succession of smaller and narrower filters, I found that one particular system’s traffic was suspicious because the system was attempting connections to many machines.

I disconnected that machine—a Windows 2000 server—from the LAN. The next step was to determine what on that computer was sending out the bad traffic. I rebooted the suspect system, then ran Mark Russinovich's TCPView and Process Explorer tools on it to see what programs might be opening network connections and what their traffic was doing. I started the sniffer, then very briefly connected it to the LAN long enough for it to get an IP address. TCPView lit up like a Christmas tree, showing a couple of hundred connections that instantly formed to batches of addresses all over the Internet. The program making all these connections was called ifconfig.exe in c:\windows\system32.

How did you determine that a botnet was running on the server?
When I viewed the ifconfig.exe process with Process Explorer and suspended it, ifconfig.exe claimed it was from Microsoft, but Process Explorer wouldn't verify ifconfig's digital signature. When I looked up the file in the c:\windows\system32 directory, it was marked read-only, system, and hidden. I opened ifconfig.exe by using the Attrib command, copied the file to a USB drive, and sneaker-netted the file to my notebook. When I copied the file from the USB key into my notebook, Trend Micro's antivirus software instantly flagged it as WORM_RBOT.CWU and didn’t let me touch the file.

I wanted to know what else the worm did. I put it into an “isolation chamber” built with a Windows 2000 Server virtual machine (VM) running on my notebook. With Ethereal sniffing the network and the worm copied into the VM, I watched as the worm was started. It immediately disappeared from the desktop copied into the c:\windows\system32 directory and started phoning home via DNS queries to a dynamic DNS service in Hungary. Later I ran a Traceroute, which localized the target IP address to a dialup account in Russia. The next morning, the bot client in the isolation chamber connected to its master, received orders, and started trying to exploit vulnerabilities in Symantec's antivirus product on port 2967 at various Internet locations. Three weeks later, the botnet's DNS name resolved to an address in Brazil. This told me that the botmaster was using other hacked machines to control the bots.

Had you ever encountered a botnet before?
I hadn't touched one up until that point. I knew the concept of a botnet. It's basically a remote controlled system, and you have bunches of remote controlled systems that communicate over Internet Relay Chat (IRC). That was the extent of my knowledge of botnets and what they did. But by watching the data flowing back and forth when I put the bot in the isolation chamber, I could see what it was sending out and receiving back from its controller. Once I could see what it was doing and how it was getting its command and control, neutering it was a simple next step.

How did you get rid of the bot?
Network traffic from the sniff indicated that the bot was “phoning home” on TCP port 4212. From another sniff, I found that many client machines were also compromised and running the same worm. Meanwhile, the isolation-chambered worm was still checking every few seconds for its master, with no answer. My client runs its own DNS servers, which forward requests for out-of-their-domain names to the ISP's DNS servers. To prevent the worm from phoning home, I added a DNS record in their DNS servers telling the servers they were the authoritative DNS for the botnet's domain name.

I stopped the worm on one server and watched. It didn’t respawn—evidence that it didn’t have another program watching its back. I also carefully watched the network traffic to detect whether any other attempts were made to connect to anywhere else—and saw none. Deleting the ifconfig.exe file and removing run keys appear to be a valid fix, though I was a bit cautious. We were lucky; this wasn’t a very smart worm. We opted to leave the less-critical server's worms suspended and manually cleaned the critical servers.

I also blocked all access to port 4212 on the firewall. I knew that copies of the worm were still running on client machines. When I viewed the traffic on port 4212, I compiled a list of infected machines by IP address; the client’s IT staff could deal with those. At this point, all the servers were either cleaned or the worms on them suspended.

   Previous  [1]  2  Next