How We Dealt with Blaster

Version 1: September 11, 2003

Introduction:

To start with, this is not intended to be a security document, a how-to document, an official publication of my employer, or anything other than my own personal musings and experience. We predominately used shareware and other freely available tools so, though the software is modified slightly to run in our environment, we really didn't write anything original. Basically, I'm a typical network engineer who is pasting together a system using other people's code. Said another way, if you feel like thanking me, thank these people instead. In particular, we use ISC's version 9 BIND (http://www.isc.org/products/BIND/), ISC's version 3 DHCP (http://www.isc.org/products/DHCP/), and a slightly modified version of Southwestern University's NetReg code (http://sourceforge.net/projects/netreg/). The University of Connecticut also did a write-up, which can be found at http://www.security.uconn.edu/uconn_response.html, for a process similar to the one outlined here.

As some background, the University that I work for used this type of setup during the 2003 fall move in. We had about 11,000 systems arrive in our dorms over the first 10 days. During the registration process, we scanned each computer. If we caught something, then we force the system to run a list of patching/cleaning tools before we allow the system to be registered. In the first 10 days, we stopped 3,400 computers and forced them to patch/clean.

Nanog 30 Overview Presentation 2/2004

BU Security Campus - March 2004 Presentation 2/2004

The Process:

This process relies heavily on DHCP and DNS tricks, so it requires that your end hosts not have static IPs and use the DNS servers given out via DHCP. If you don't use DHCP then this isn't going to work for you.

The networks in our dorms are configured with two subnets each, a private one and a public one. For example:

	Interface Vlan101
	  ip address 192.0.2.1 255.255.255.0
	  ip address 10.0.2.1 255.255.255.0 secondary
	  ip access-group 100 in
	  ip helper-address 192.168.3.1
	!

When a system is attached to the network, our DHCP servers hand it an IP on the 10.0.2.0/24 subnet. Our network filters (ACL 100 in this example) are setup to only allow traffic from 10.0.2.0/24 to a few servers/services - DHCP to our DHCP servers, DNS to our DNS servers, open access to our scanning server, web access to our web registration server, etc. We use a shared-subnet declaration in the DHCP server configuration to accomplish this:

shared-network "ExampleNet" {
  subnet 192.0.2.0 netmask 255.255.255.0 {
  }
  subnet 10.0.2.0 netmask 255.255.255.0 {
  }
  pool {
    deny unknown-clients;
    range 192.0.2.10 192.0.2.254;
    option broadcast-address 192.0.2.255;
    option subnet-mask 255.255.255.0;
    option routers 192.0.2.1;
  }
  pool {
    allow unknown-clients;
    default-lease-time 120;
    range 10.0.2.10 10.0.2.254;
    option broadcast-address 10.0.2.255;
    option subnet-mask 255.255.255.0;
    option routers 10.0.2.1;
  }
}

You could accomplish the same thing without the second subnet by declaring different pool ranges - one range for allowed and one for non-allowed - and then set your ACL appropriately.

We have a view set up in our DNS servers that returns the IP of a special web server (192.168.2.1 in this example) for all name lookups if the client doing the query comes from our private net (10.0.2.0/24). We use something like:

Acl "dhcp-reg-required" {10.0.2.0/24;};
View "dhcp-reg" {
match-clients { "dhcp-reg-required"; };
  recursion yes;
  auth-nxdomain no;  
# [...]
  zone "." {
        type master;
        file "named.ca";
  };
}
view "default" {
  match-clients { any; };
# [...]
}

Where the "named.ca" file looks like:

$TTL 0
@ IN SOA dns01.example.com hostmaster.dns01.example.com. (
2003082501 10800 3600 604800 86400)
@ IN NS dns01.
                    0 IN A 192.168.2.3
*                  0 IN A 192.168.2.3

This causes everything to resolve to 192.168.2.3 for clients with IPs in 10.0.2.0/24 but to resolve normally for all other clients. Instead of using views, you could use different DNS servers, one for jailed hosts and one for non-jailed hosts.

When a system initiates an HTTP Get to 192.168.2.3, they are given a redirect to the first page of our registration web site, which is a virtual host on the same IP. In Apache, we do this using a configuration similar to:

NameVirtualHost 192.168.2.3 
<VirtualHost 192.168.2.3:80>
  ServerName netreg.example.com
</VirtualHost>
<VirtualHost 192.168.2.3:80>
  ServerName everything-else.example.com
  ServerAlias everything-else.example.com *
  DocumentRoot /dev/null
  RedirectMatch .* https://netreg.example.com
</VirtualHost>

The main page at netreg.example.com basically says, "this is the registration system, click here to register", though in significantly more robust and polite language. The main page also does a "are you manually jailed" check that is described below. Clicking the registration link goes to execute a CGI that is password protected, so the users is prompted for a username and password. If the username/password combination is valid, the link runs a CGI that makes an external call to a Nessus server to scan their system. In principle, one can insert anything they want at this step in the process, as long as it can communicate a Good/Bad status back to the CGI regarding the state of a system. If the Nessus server times out or gives the system an ok, they are passed through to the NetReg registration CGI. The CGI looks their IP up in our DHCP server lease's file, determines its MAC address, logs the Username/MAC pairing, then uses a dynamic OMAPI call to the DHCP server to create a host entry within the DHCP server. Now that our DHCP servers know this MAC, the system will no longer receive a 10.0.2.0/24 address but will instead receive a 192.0.2.0/24 addresses. The final page prompts the user to reboot, which effectively causes a release/renew of their IP. We've purposely set the lease time on the 10.0.2.0/24 subnet low so that, at worst, it will have expired when the host reboots. The host is now on the network and doesn't have to register again.

If the scan detects a vulnerability (or anything else you tell it to catch), it displays a "you are infected, follow this link for instructions on how to clean your system" page. In our case, the scan failure does a URL redirect, so we can redirect them to a tailored page (though we only have a single page at this point). This host is now effectively jailed until it is cleaned because it can never complete the registration process. A few nice side effects are that we don't have to track down a physical port and shut it off to jail a host and we've also provided the self-help tool for them to clean and pass the check. We also have a contact database for MAC addresses (once they are registered), so we can contact a human being down the road if we see something wrong with a system, rather than having to track down the physical port and ask around to find out "who sits here". Lastly, if a system is jailed, it is jailed no matter where they plug in on our campus so moving to another port doesn't help them.

One thing to note is that this process is a one-time scan/register process that's keyed on detecting a particular vulnerability. We've found numerous systems pass our scans but show up subsequently as being infected. In particular, we were scanning predominately for the first Window's RPC vulnerability, but the Welchia worm patches the vulnerability without always getting ride of Blaster. On the other hand, this week we are forcing people to re-register, and therefore be rescanned for the second/new Window's RPC vulnerability. Since our "here's how to get out of jail" process involves running a cleaning tool, we're hoping to catch all the Welchia's this time.

In general, there are many tools you can use to detect infected systems post-registration - your support email box, the IPAudit packet classification tool (http://ipaudit.sourceforge.net/), NetFlow statistics, honey pots, Snort (http://www.snort.org/) and other IDS systems, etc. If you'd like more information, take a look at the work that the University of Connecticut has been doing at http://www.security.uconn.edu/uconn_response.html.

When we detect a host on our network that's infected or for any reason doing things that we don't like, we use the dynamic interface to our DHCP server to de-register them. Thus, when their DHCP lease runs out, they end up back in the 10.0.2.0/24 network jail. We also add their MAC address and a reason code to a list of "jailed" hosts. Our main registration page is a CGI that looks up a host's MAC in our leases file and compares it to a list of known infected systems (see details of the registration process above). If the CGI finds their MAC in the "jailed" list, it displays a note telling the user why they've been jailed and how to clean their system, thus preventing them from re-registering. Unfortunately, we don't yet have a good way to un-jail them if they are manually jailed. We are working on a new patch/cleaning tool that will generate a unique code (hash based on MAC, Username, etc) that the user could enter into a web page to get out of jail, but that's still in development.

Final Thoughts and Known Issues:

I'm continually surprised at how many people just follow the process that we outline. Of the 11,000 or so hosts that we've registered, we've only found a few that attempted to bypass the system. We do, however, have a system that pulls ARP tables out of our routers and compares the MAC's against our registration database. If we see one that's not registered, we get an alarm. So far, we've had 5 that have used static IPs to bypass the registration process altogether, and most have been Sony Playstations.

Some of the other known issues we've seen are:

Commonnames and other spyware. As I've pointed out a few times, this whole process only works if the end users grabs an IP via DHCP, opens a web browser, does a DNS query against our special DNS servers, and is driven to our registration page. Unfortunately, spyware/adware is everywhere and it's almost impossible to get rid of. One of the unfortunate side effects of it is that some spyware/adware either overrides your DNS settings with their own or makes an HTTP call to their website before allowing the browser to download a page normally. Because it interferes with "normal" web surfing, users with spyware/adware might never be able to load your registration page. Additionally, a few people have had problems due to statically configured DNS settings that were overriding the ones pushed via DHCP.
This was used as part of a one-time registration system so it really doesn't address on-going security issues. It does, however, create the infrastructure for large-scale self-help cleaning of known infected systems. You could easily integrate the manually jailed part with any vulnerability or IDS system. We're actively working on this part.
Mac spoofing or manually changing your MAC is possible and, if you spoof a MAC that's already registered, you bypass the process completely. This is just another reason to have post-registration infection detectors.
Ok, so you've trapped someone and they need to patch. What do you do if they need something that you don't have? In our case, we hand out private IPs so the systems cannot directly access the Internet. We downloaded as many of the service packs and patches as we could, but there were still people on our network that required strange things (the Korean language version of service-pack 1 for 2000...). For the three or four cases like this that we ran into, they were all laptops that could easily be taken to our central computing lab, where we patched the system for the user.

Afterthoughts:

Again, I'd like to thank everyone out there who wrote code and made it freely available for us to use! We never would have been able to accomplish this, or even think of this process, without you!

If you have any questions, drop me note and I'll try to answer them. If I can't, I'll try to shuffle you to someone who can. Fortunately, I'm not longer among the vast, unemployed IT army, so I might not be able to get back to you immediately as my employer actually expects me to work :)

Eric :)

Version 1.0: September 11th, 2003
Writen by eric@roxanne.org