You've Been Hacked. Now What?

Lessons learned. Before you place a publicly accessible server in the DMZ, verify with the software vendors that any programs you run on the server are secure enough for public access. Keep all servers, not just those in the DMZ, updated with the latest service packs and critical updates. Make sure that applications use SQL Server­integrated security and don't use a connection string in their code to connect to an SQL server. Embedding a connection string in the Web server code instantly gives an intruder a valid username and password. For more information about establishing a SQL Server connection from a Web server, refer to the Microsoft article "Recommendations for Connecting to Databases Through Internet Information Services" (http://support.microsoft.com/?kbid=258939). Make sure Microsoft IIS uses stored procedures to access SQL Server data, and don't let the IIS server run SQL statements. Because these steps let you grant an authenticated user Execute permissions for only specific stored procedures, you can restrict the user from running any SELECT statements on SQL Server.

VPN Client Attack
Another client's Exchange 2000 Server machine was experiencing backup problems and poor server performance when sending and receiving email. I arrived to find the problem was more serious than a failed tape drive and slow server. The server had a lot of disk activity and high CPU utilization. I opened Windows Task Manager and sorted the processes by CPU utilization. Store.exe was taking up most of the CPU cycles. The company wasn't a heavy email user and had only 15 users connected to the server. With so few users, Exchange shouldn't consume as many resources as this server was. I suspected a corrupt mail store.

Identifying the hack. I started Exchange System Manager (ESM) and selected Administrative Groups, Admin_Group_Name, Servers, Server_Name, Protocols, Smtp, Default SMTP Virtual Server, Current Sessions. I noticed that six sessions had been connected to the SMTP virtual server for longer than 5 minutes--a clue that something was very wrong on the server. Typically, an Exchange Server session lasts only a few seconds unless the connection is sending or receiving a message with a large attachment. I looked at the queues on the default SMTP virtual server and discovered more than 50 queues in various states of sending mail or waiting for a retry. Someone was using the mail server as a relay, but how? The server had the latest service packs (Win2K SP4 and Exchange 2000 SP3) and the latest critical updates, so I used the Open Relay Database's (ORDB's) test at http://www.ordb.org, which checks submitted host systems for open relays, to ensure that the relay was closed.

Whenever I tried to clear a connection to the default SMTP virtual server, the connection would reappear, usually with a different domain name but from the same IP scheme. I used the IANA to trace the IP addresses to a block allocated by an ISP in China. After verifying that the server wasn't an open relay, I concluded that someone was probably authenticating to the server and sending mail from it. Backup was failing because it was trying to back up all the mail the spammer was trying to send. I opened the Active Directory Users and Computers snap-in and removed all invalid users. I also noticed some unauthorized users in the Administrators Group and removed them. I then checked the registry and found no hacking programs loaded in the Run subkeys. I also ran a virus scan on the server, and the server was clean.

To prevent the spammer from sending more messages, I disconnected the firewall from the Internet and deleted all the active sessions from the Exchange server. I tried to use ESM to delete the messages from the mail queues, but that took too long. So I stopped all Exchange services, opened a command prompt, and used the Del command to delete the messages from the directory D:\exchsrvr\mailroot\vsi 1\queue. As soon as I stopped the Exchange services, the server's performance greatly improved. Even so, deleting the 10,000 queued messages took me more than an hour. I then looked at the bad-mail directory in D:\exchsrvr\mailroot\vsi 1\badmail. It took me approximately 8 more hours to delete all those messages. Finally, I changed the passwords for every user on the network and created a rule on the firewall to deny traffic from the IP ranges where the spam originated. After making these changes, I reconnected the firewall to the Internet and monitored the server. The spam connection didn't reappear.

This particular network had several remote sites that ran VPN tunnels. At one of the remote sites, I discovered that the remote machine contained these hacking programs: Bat.Mumu.A.Worm, Hacktool, W32.Valla.2048, W32.HLLW.Lovgate.J@mm, Bat.Boohoo.Worm, and MSBlast.

My client said that this computer was left running all the time, with the VPN tunnel active. With such an arrangement, it's only a matter of time before someone hacks the machine. I always recommend that remote clients sit behind a firewall, especially if they use a broadband connection. If you run XP over a dial-up or wireless connection, make sure you use XP's Windows Firewall (formerly Internet Connection Firewall--ICF) to protect your computer while it's connected to the Internet.

Repairing the damage. I rebuilt the workstation, placed it behind SonicWALL's TELE3 firewall, and let the firewall create the tunnel back to the corporate office. Fortunately for this client, the intruder used the server only to send spam--he or she could have caused a lot more damage.

Lessons learned. Because of this hack, the company no longer lets client machines use a mobile VPN client on a broadband connection without a firewall. If you have remote sites with VPN tunnels and broadband connections, install a firewall--or at least train users to turn off their computers when they're not in use. Also make sure each user knows how to deactivate the VPN tunnel when it's not in use.

Exchange Server SMTP AUTH Attack
A third client's Internet connection was running slowly because of heavy Internet traffic. After I asked all users to disconnect from the Internet, the traffic was still heavy. I looked at the outgoing queues on the Exchange 2000 server and discovered more than 100, with a significant number of messages in each queue. Using ESM, I inspected messages in several random queues. I discovered messages whose sender or recipient wasn't from the local domain, which means the mail server was likely being used as a mail relay. By default, Exchange 2000 and later systems allow relaying if a message sender can successfully authenticate to the mail server.

Identifying the attack. Hackers can use a couple of different methods to get a valid username and password. They can repeatedly guess a guest or user's password until they stumble upon a valid one, or they can launch a hack to obtain a valid username and password. A spammer needs only one valid username and password to relay mail, even if your mail server isn't an open relay. To determine which account the spammer was using, I started ESM and clicked Organization, Administrative Groups, Organizational Unit, Servers, then right-clicked Server Name, Properties. I selected the Diagnostics Logging tab. In the Services window, I clicked MSExchangeTransport, and in the Categories window I increased the logging level to maximum for the categories Routing Engine, Categorizer, Connection Manager, Queuing Engine, Exchange Store Driver, SMTP protocol, and NTFS store driver. I then checked the event log, looking for an authentication from an external or unknown mail server. Unsuccessful logon attempts will show up in the Security log with event ID 680. I discovered that an intruder was using a user account that wasn't a local Exchange Server account to authenticate to the mail server.

Repairing the damage. After I identified the authentication account, I took the following steps to secure the Exchange server.

1. I changed the password for the account the spammer was using. If, in a similar situation, you think a spammer might have more than one valid user ID and password, change the passwords for all users on your network. I also disabled the Guest account and set up dedicated accounts to start services on the server. Don't use the Administrator account to start services. If a machine is hacked, the account used to start the service can be compromised.

2. I disabled authentication on the outward-facing Exchange server. To do so, I started ESM and selected Organization, Administrative Groups, Organizational Unit, Servers,ServerName, Protocols, SMTP, then right-clicked the default SMTP virtual server. I selected Properties, clicked the Access tab, then clicked Authentication. I left Anonymous access enabled but cleared the Basic authentication and Integrated Windows Authentication check boxes. Clearing these check boxes essentially disables the Auth command on the SMTP server.

3. I enabled relaying for other internal Exchange servers to ensure that they can send mail to the outward-facing Exchange server. I opened ESM, right-clicked the virtual SMTP server, and selected Properties. Under the Access tab, I clicked Relay, selected Only the List Below, and listed the internal mail servers that are allowed to relay mail to the outward-facing server.

4. After making these changes, I thoroughly tested the configuration. I tested mail flow to and from the Internet and to and from all mail servers in the organization. The changes have the potential to disrupt mail flow between servers, so you might want to wait until the weekend to make them. Better yet, test these changes in a lab environment before implementing them in production.

5. In this particular incident, I discovered a machine that was severely compromised, which I completely rebuilt. In situations you might encounter, you'll need to identify all compromised machines and repair or rebuild them.

6. I checked the ORDB to determine whether the client's mail server had been blacklisted for being an open relay. Fortunately, I discovered and repaired the hack before the client's mail server was blacklisted. A mail server can be blacklisted if it's an open relay or if the mail server is identified as a server that sends large amounts of spam. Many open-relay databases exist. You can see a list of some of these databases at http://dmoz.org/computers/internet/abuse/spam/blacklists.

If your mail server is blacklisted, you can either submit a request to remove the server from the blacklist or change the outside IP address of your mail server. If you change the mail server address, you must also update the mail exchanger (MX) record for your mail server, or incoming mail will be blocked.

Lessons learned. To repair Exchange Server SMTP AUTH attacks and prevent future ones, I strongly suggest that you take the steps I did. If an intruder procures a valid user ID and password and is able to relay mail, your mail server will be placed on various email blacklists. You'll spend significantly less time preventing these attacks than troubleshooting mail delivery problems, removing your server from blacklists, and fixing the vulnerability.

Don't Panic; Be Prepared
An attack recovery plan is part of any sound IT structure. It will help you respond efficiently to a network hack instead of going into a panic. Be familiar with the tools and methods that malicious intruders use and take a proactive approach to preventing them from hacking your network. I'll discuss this subject in more detail at the next Windows Connections Conference, from October 24 to October 27 in Orlando, Florida. Hope to see you there.

End of Article

   Previous  1  2  3  [4]  Next