Re: Santa Fe city government computers knocked out by worm
On Sun, 16 Nov 2003, Jamie Reid wrote:
There was a comment (maybe even mine) in a previous thread about accepting a base level of potentially compromised hosts on a network, as the costs of rooting out every last one becomes unwieldly. Networks are large enough that security must be viewed as an economy of controls and risks instead of as a binary state of secure or compromised.
If your policy is not to root out every last one, then you need to beef up your network so a single compromised host doesn't bring down the whole network. The Internet is evidence that a network can continue to operate even with a very large number of compromised machines on a daily basis. On the other hand, if a single user downloading a music file on your network can take your entire network off the air for several days, you may have a problem. I've often tried to explain that ISPs generally view worms as a "capacity planning" issue. Worms change the "eco-system" of the Internet and ISPs have to adapt. But ISPs generally can't "fix" the end-users or their computers. System admins were able to completely eradicate the Morris worm. But most modern worms like Nimda, Code Red I/II, Slammer stick around. Sometimes a new worm like Nachi supplants an older worm like Blaster. Even if the ISP tries to be the great network firewall, we have mobile computers with mobile code. Laptops are too common, connecting to multiple networks.
[Sorry for responding to old mail, but I'm catching up] On Sunday, November 16, 2003, at 02:12 PM, Sean Donelan wrote:
I've often tried to explain that ISPs generally view worms as a "capacity planning" issue. Worms change the "eco-system" of the Internet and ISPs have to adapt. But ISPs generally can't "fix" the end-users or their computers.
I'm curious to know if doing this is at all well understood? Those of us doing research on worm spread, I don't think have a completely clear understanding of the interaction of Internet bandwidth and worm spread. Slammer, we are pretty clear became bandwidth limited (the rate of spread slowed down dramatically about 40 seconds into the spread). But we don't really know where those chokepoints live (at the edge, or in the middle). It would seem for the Internet to reliably resist bandwidth attacks from future worms, it has to be, roughly "bigger in the middle than at the edges". If this is the case, then the worm can choke edges at the sites it infects, but the rest of the net can still function. If it's bigger at the edges than in the middle, you'd expect a big enough worm would be able to choke the core. For a given ISP, you'd want capacity to the upstream to be bigger than the capacity to downstream customers. (It would seem like this would be the reverse of what economics would tend to suggest). Do we really know much about the capacity of the Internet to carry worm traffic? (We believe Slammer used a peak bandwidth of roughly 200 Gbps). Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Stuart Staniford writes:
It would seem for the Internet to reliably resist bandwidth attacks from future worms, it has to be, roughly "bigger in the middle than at the edges". If this is the case, then the worm can choke edges at the sites it infects, but the rest of the net can still function. If it's bigger at the edges than in the middle, you'd expect a big enough worm would be able to choke the core. For a given ISP, you'd want capacity to the upstream to be bigger than the capacity to downstream customers. (It would seem like this would be the reverse of what economics would tend to suggest).
So, essentially, you are saying that the edges (customers, presumably) need to be bandwidth-limited to protect the core? This tends to happen anyway due to statistical multiplexing, but is usually not what the customers would want if they considered the question, and is not what ISPs want if they bill by the bit.
Do we really know much about the capacity of the Internet to carry worm traffic? (We believe Slammer used a peak bandwidth of roughly 200 Gbps).
I suspect that in the end the main backbone constaint will be peering links, for larger ISPs.
On Monday, November 24, 2003, at 04:59 PM, jmalcolm@uraeus.com wrote:
So, essentially, you are saying that the edges (customers, presumably) need to be bandwidth-limited to protect the core?
I wasn't advocating a solution, just observing the way things would have to be for worms to be purely a "buy a bigger box" problem (as I think Sean was suggesting if I didn't misunderstand him).
This tends to happen anyway due to statistical multiplexing, but is usually not what the customers would want if they considered the question, and is not what ISPs want if they bill by the bit.
It would generally seem that ISPs would provide more downstream capacity than upstream, since this saves money and normally not all the downstream customers will use all their bandwidth at the same time. But a big worm could well break that last assumption. So it would seem that worms are, at a minimum, not a simple or unproblematic capacity management problem. Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Stuart Staniford writes:
I wasn't advocating a solution, just observing the way things would have to be for worms to be purely a "buy a bigger box" problem (as I think Sean was suggesting if I didn't misunderstand him).
Ah.
It would generally seem that ISPs would provide more downstream capacity than upstream, since this saves money and normally not all the downstream customers will use all their bandwidth at the same time.
Right; statistical multiplexing.
But a big worm could well break that last assumption.
Yes, as could a number of events, but the response to a worm would probably be different from the latest streaming video event, or whatever.
So it would seem that worms are, at a minimum, not a simple or unproblematic capacity management problem.
Well, it would seem reasonable for an ISP to minimize a worm's effect on its non-worm customer traffic, and that might mean increasing capacity in some places, but I don't think the goal would be to move more worm traffic, but rather to reduce impact to other traffic. Presumably such activity would be combined with other anti-worm efforts.
On Mon, 24 Nov 2003, Stuart Staniford wrote:
So it would seem that worms are, at a minimum, not a simple or unproblematic capacity management problem.
Things are rarely as simple as they appear. Even buying a military grade black box may not solve the worm problem. There are some natural choke points in the Internet between ISPs and customers. The customer may have a 1000 Mbps GigE LAN and the ISP may have an OC192 backbone, but the link between them is normally much smaller. Slammer, Blaster, etc had very little impact on the major ISP backbones, but did severaly congest some of the smaller choke points. Go ahead and ask UUNET, Sprint, AT&T, etc. what impact the worms had their networks. ISPs don't have (much) control over third-party computers. But they can control their network capacity. Of course, its not a complete solution. If you are a mid-level ISP, you may have a choke point to your customer but are vulnerable from your upstream provider. A better designed worm could impact even major backbones.
On Monday, November 24, 2003, at 08:00 PM, Sean Donelan wrote:
There are some natural choke points in the Internet between ISPs and customers. The customer may have a 1000 Mbps GigE LAN and the ISP may have an OC192 backbone, but the link between them is normally much smaller. Slammer, Blaster, etc had very little impact on the major ISP backbones, but did severaly congest some of the smaller choke points. Go ahead and ask UUNET, Sprint, AT&T, etc. what impact the worms had their networks.
So you believe that the edges of the net are smaller, bandwidth-wise, than the core? So the (approximate) picture you would advocate would be that Slammer was rate limited at the customer/ISP interface? (I agree this is consistent with the fact that the tier-1s stayed up during Slammer). (I'm not trying to be difficult here - I'm just trying to figure out if we actually have any good understanding of this issue - and therefore any ability to predict what future worms might do to the Internet). (Blaster was not bandwidth limited so that's a whole different animal - it seems to have been limited by a slow scanning rate, and a poor transmission probability). Stuart. Stuart Staniford, President Tel: 707-840-9611 x 15 Silicon Defense - Worm Containment - http://www.silicondefense.com/ The Worm/Worm Containment FAQ: http://www.networm.org/faq/
Hi, Stuart. ] So you believe that the edges of the net are smaller, bandwidth-wise, ] than the core? This was certainly the case in my previous life at a large hosting provider. We had GigE LANs, used providers with OC192 backbones, but had only OC3 to OC12 links to our providers. Like most edge networks, we had CIRs on those uplinks that were considerably lower than the pipe size. A full OC12 turned out, at the time, to be darn expensive. :) Our choke points were always our peering or transit links. This was the case for our (large) enterprise customers as well. Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
On Tue, 25 Nov 2003, Rob Thomas wrote:
Our choke points were always our peering or transit links. This was the case for our (large) enterprise customers as well.
Some people refer to it as the hourglass effect, but it has more than one bump. Generally only the smallest bottleneck controls the congestion. But worms and DDOS (but not DOS) violate some of the assumptions. lower bandwidth<---->higher bandwidth Local Area Network (LAN) Campus Area Network Customer to ISP uplink ISP POP to Backbone ISP Intra-Backbone ISP to ISP transit/peer (same continent) Intercontinental circuits Of course, there are some exceptions like a customer with an OC192 uplink or an ISP running a web hosting center on a ISDN link.
Hi, Sean. ] lower bandwidth<---->higher bandwidth Great ASCII chart. :) ] Of course, there are some exceptions like a customer with an OC192 uplink ] or an ISP running a web hosting center on a ISDN link. Another bit to consider is address space. Code Red discovered a lot of folks with very small pipes (circa T1) and very large netblocks (circa /16). These folks paid a heavy price when hit with the "scan all IPs in the netblock" worms. Thanks, Rob. -- Rob Thomas http://www.cymru.com ASSERT(coffee != empty);
Sean Donelan wrote:
ISPs don't have (much) control over third-party computers. But they can control their network capacity. Of course, its not a complete solution. If you are a mid-level ISP, you may have a choke point to your customer but are vulnerable from your upstream provider. A better designed worm could impact even major backbones.
If you are an access provider, specially in the consumer space, you can do many things to help the "Greater Internet" by keeping your own back yard in good shape. In the transit business, you are expected to deliver the bits regardless of the content so there the only viable option is to drop packets where the source or destination addresses don´t make sense. Pete
On Sat, 29 Nov 2003, Petri Helenius wrote:
If you are an access provider, specially in the consumer space, you can do many things to help the "Greater Internet" by keeping your own back yard in good shape. In the transit business, you are expected to deliver the bits regardless of the content so there the only viable option is to drop packets where the source or destination addresses don�t make sense.
What is the difference between a transit provider and an access provider, specially in the consumer space? Why is a transit provider expected to deliver the bits, but the access provider isn't? Since the bulk of Internet access is actually provided by wholesale providers (e.g. AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is the access provider and who is the transit provider?
Sean Donelan wrote:
What is the difference between a transit provider and an access provider, specially in the consumer space? Why is a transit provider expected to deliver the bits, but the access provider isn't? Since the bulk of Internet access is actually provided by wholesale providers (e.g. AOL/Earthlink buy wholesale modem access from UUNET/Level3), who is the access provider and who is the transit provider?
Both of these identities can and do exist within the same ISP. Their transit product would not include active mitigation, notification and filtering but the access provider part would. Pete
participants (5)
-
jmalcolm＠uraeus.com -
Petri Helenius -
Rob Thomas -
Sean Donelan -
Stuart Staniford