At 10:00 AM -0400 7/31/01, Dave Stewart wrote:

At 09:49 AM 7/31/2001, Jeff Ogden wrote:

...

So what, if anything, are people planning to do differently as 8 pm EDT today and the possibility of a new round of Code Red Worm activity approaches? Are there things that we as network operators can and should be doing beyond encouraging end users to patch their vulnerable systems?

You can scan your network(s) for machines that are vulnerable, and patch them. Or contact the end users and require that they patch them.... if they aren't patched by 7:45pm or so, you can block port 80 access to those machines until they are patched.

OK, but even if we get every one of the vulnerable systems on our own and our customer's networks patched, we will still be subject to probes from infected systems elsewhere. In the last go round ten or eleven days ago it was the probes of unused IP addresses more than infected systems on our network that seemed to cause problems. So while we will continue to be good network citizens and work to get systems on our network patched, we will continue to see problems as long as there are "enough" unpatched systems out there to cause problems. I suspect that that is weeks or even months in the future. Attached is a long message that was sent out to Merit's customers this morning talking about our plans. No need to read it if you don't want to. -Jeff --------------------

Date: Tue, 31 Jul 2001 01:55:24 -0400 To: michnet-inform From: Jeff Ogden <jogden@merit.edu> Subject: Merit's Tuesday evening plans related to the Code Red Worm

I am sure that most of us have seen enough announcements about the Code Red Worm by now to last a lifetime, but here is one more.

I want to outline Merit's plans for the possible reemergence of the Code Red Worm starting more or less at midnight UTC/GMT on August 1st (that is 8 pm EDT Tuesday evening here in the eastern U.S.). I say more or less because many systems don't have their clocks set exactly right or don't have their timezone set correctly, and so we could see some activity start earlier or later than the expected time by anything from a few minutes to as much as four or five hours.

First let me say that we at Merit don't know and I don't think anyone else really knows what, if anything, is going to happen starting at 8 pm Tuesday evening. There are new variants of the worm and they may behave differently. There are of course several variants of the worm that we've seen already and so we do have some idea of what to expect from them. We hope, but don't really believe, that most vulnerable systems will have been patched over the last week or ten days and that this will minimize the extent of any future problems (see below for information on why this isn't likely to be the case and about problems that may occur even after the patches have been installed on all of your local systems).

At least initially Merit does NOT plan to take any unusual steps to deal with the Code Red Worm on Tuesday evening. We are going to start out treating this as a host computer problem. Host computer problems are things that the people who are responsible for the individual computers need to deal with. We will have staff watching the network a bit more carefully than usual to spot and track signs of unusual activity or problems. We plan to work directly with some of the MichNet sites that were severely impacted by the Code Red Worm last time, both to help these sites if there are problems and to use the sites as something of an early warning indicator for what we might expect elsewhere. We will be tracking developments elsewhere including mailing lists and Web sites that have information about Code Red developments.

Sites with MichNet attachments can and should report network problems to the Network Operations Center (NOC) by e-mail or by phone. We would like to help where we can. We may be able to provide assistance, but even if we can't help, reports will give us a better view of what is actually happening across MichNet.

If it would be helpful, we can install packet filters similar to the ones we installed the last time around in routers that Merit manages. These filters block packets inbound to port 80 on host computers. This time we'd like to install these filters at the request of individual sites rather than taking this action on our own. If your site would like us to do this, contact the NOC. When you call please have a list of the IP addresses for any host computers that shouldn't be blocked. Of course many sites can and probably should take these steps themselves in the routers or firewalls that they manage.

While we hope this won't be necessary, if we start to see serious widespread problems, we may have to switch as we did last time and treat this as a network rather than as a host computer problem. If need be, we will be able to call in additional staff to work on problems either Tuesday evening or Wednesday morning. If this becomes necessary, we will post announcements to the MichNet-Inform e-mail list and on the telephone recording that the NOC maintains.

Estimates as of last Sunday are that at least 30% and perhaps as high as 80% of the 350,000 plus systems that were infected with the Code Red Worm a little more than a week ago have not yet been patched. No matter which end of the range you believe you still get big numbers. And no one knows how many vulnerable systems are out there that weren't infected the last time around, but which may be infected in the future. Estimates are that this is another large number.

Systems that only access the Internet over a dial-up line may be infected or vulnerable. New systems right out of the box may be vulnerable. Systems that belong to people on vacation or at schools that are out for the summer, may be vulnerable when they are turned back on days, weeks, or months from now. It seems certain that we are all going to be working on the Code Red and related problems for quite some time to come.

See

http://worm-security-survey.caida.org/

and

http://www.caida.org/analysis/security/code-red/

for details about the rate that patches are being installed and some very interesting analysis of the spread of the Code Red Worm ten days or so ago. If you don't have time to read all of this information, at least look at the conclusions (http://www.caida.org/analysis/security/code-red/#conclusions) which are sobering.

Even if your organization manages to patch every single vulnerable system, your site may still see network performance problems due to probes of your systems from infected computers located elsewhere. It was side effects from these probes (ARP floods caused by large numbers of probes to unused IP addresses), rather than the infected systems themselves or the traffic from the probes, that seemed to cause most of the network performance problems that individual sites on MichNet experienced ten or eleven days ago.

There are some things that individual sites can do to protect themselves beyond installing the patches in the vulnerable systems. Pay particular attention to comments about ingress and egress filtering in the section on "Good Practices" in the CERT's announcement (http://www.cert.org/advisories/CA-2001-23.html). Sites with large amounts of unused IP addresses space seem to be more vulnerable than other sites and so using filters in routers or firewalls to block access to ranges of unused IP address may be useful. Individual sites are in a much better position than Merit to install all of these types of filters.

Finally, there is a very real concern that with so much attention focused on the Code Red Worm and installing the patches from Microsoft, that we may be missing other security problems, assuming that problems are due to Code Red when in fact they are not, or not installing other patches and security fixes for other equally important problems in a timely fashion. We all need to keep in mind that the real problem here isn't the Code Red Worm, but inadequately maintained systems. We all need to put procedures in place to ensure that security patches and other fixes are installed in an on-going and timely fashion in the future.

Here is the list of some of the URLs related to the Code Red Worm that people may find useful or interesting:

http://www.digitalisland.net/codered/ (includes step by step instructions, slides, and audio from a 30 minute lecture on Code Red)

http://www.cert.org/ http://www.cert.org/archive/html/coderedannounce.html http://www.cert.org/advisories/CA-2001-23.html http://www.cert.org/advisories/CA-2001-20.html http://www.cert.org/tech_tips/home_networks.html

http://www.microsoft.com/technet/security/bulletin/MS01-033.asp

http://www.cisco.com/warp/public/707/cisco-code-red-worm-pub.shtml

http://www.caida.org/ http://worm-security-survey.caida.org/ http://www.caida.org/analysis/security/code-red/

http://www.securityfocus.com/ http://www.securityfocus.com/bugtraq/archive http://www.securityfocus.com/templates/column.html?id=13 http://www.securityfocus.com/templates/archive.pike?list=1&start=2001 -07-15&fromthread=0&threads=0&mid=197828&end=2001-07-21&

http://www.net-security.org/text/articles/coverage/code-red/ (very comprehensive collection of materials)

http://www.umich.edu/~virus-busters/bady.html

http://www.eeye.com/ (the folks that identified the vulnerability originally back in June) http://www.eeye.com/html/Research/Advisories/ http://www.eeye.com/html/Research/Tools/codered.html

http://www.nipc.gov/ http://www.nipc.gov/warnings/alerts/2001/01-016.htm

http://www.symantec.com/ http://www.symantec.com/avcenter/venc/data/codered.worm.html http://www.symantec.com/press/2001/n010720a.html

http://www.nai.com/ http://www.mcafeeasap.com/asp_subscribe/trial_cc_wormscan.asp

http://www.merit.edu/mail.archives/nanog/

Hope this is useful. Sorry there are so many of these messages and some are so long.

-Jeff Ogden Merit