- Test - lists.nanog.org

Re: sell shell accounts?
by salo＠msc.edu 24 Jul '96

24 Jul '96

> Date: Tue, 23 Jul 1996 19:03:08 -0400 > To: salo(a)msc.edu (Tim Salo) > From: Paul Ferguson <pferguso(a)cisco.com> > Subject: Re: sell shell accounts? > Cc: freedman(a)netaxs.com, nanog(a)merit.edu > > At 05:06 PM 7/23/96 -0500, Tim Salo wrote: > >The desire for a full mesh of VCs between routers becomes more compelling > >if you have more than three routers. The diagram is left as an exercise > >to the reader; I rather dislike having to draw in ASCII. > > > >Note that the two paths between A and C, (directly versus via B), seem > >to take distinctly different paths. (Perhaps, I missed your point). > > Perhaps I missed your point. > > Tim, I could've sworn that you had been following the IETF ION WG > mailing list, where this has been discussed on hundreds of occasions. > I hesitate to even mention this on NANOG (I really don't want to > re-hash this on this list), but I can't hold my tongue. > > The scalability of an ATM network where each end-point is fully meshed > to all other end-points decreases in direct proportion to the number > of end-points. This is not so much a layer 2 problem, per se, but rather > the ability to maintain functional routing (convergence, et al.) in a > fully meshed network topology. > > This does not scale. I believe that given my [secret] assumptions, (some of which are revealed below), it scales well beyond the financial abilities of most any NSP. I assumed that the wide-area ATM service supported only PVCs and PVPs. (I keep hearing promises of SVCs "next year" but I don't think anyone [domestically] has delivered them.) I believe this address your concern about maintaining functional routing, but I make no claims to being a routing expert, so I might not fully comprehend the issue. I also assumed that the going price for a DS-3 xwide-area ATM connection is, to more or less pull a number out of the air, $250,000/year. This has two implications: o Any one NSP isn't going to be able to afford all that many connections. A full mesh of 100 routers, (i.e., $25 millino/year in ATM services) would require 4,950 PVCs. That is a lot to configure, but... o Given that each link is rather expensive, it behooves the NSP to try to avoid needlessly moving a packet back and forth across a DS-3 local loop just because my ATM provider can't provide me with any more PVCs. (Which was my major point, by the way.) On the other hand, you are correct that this doesn't sale to thousands of routers. But, I didn't think that was the problem we were trying to solve here, (just yet). -tjs

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by salo＠msc.edu 24 Jul '96

24 Jul '96

> From: Dave Siegel <dave(a)rtd.net> > Subject: Re: ATM Wide-Area Networks (was: sell shell accounts?) > To: salo(a)msc.edu (Tim Salo) > Date: Tue, 23 Jul 1996 10:22:23 -0700 (MST) > Cc: dave(a)rtd.net, agislist(a)interstice.com, nanog(a)merit.edu > [...] > Even > with the improved buffering, if you fill your pipe into the ATM switch, > your ATM switch still becomes a packet shredder, compared to the more > graceful packet drops seen on a clear channel line. > [...] "Good" ATM switches are doing Early Packet Discard and Partial Packet Discard. You can ask your favorite switch vendor for details, or look at the following for a nice summary: http://www.fore.com/html/atm-edu/whitep/FTBMWP.html -tjs I am not quite sure how an ATM switch becomes a "packet shredder," or even what a "packet shredder" is...

3 2

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

It's still ATM day, 'cause something else just occurred to me: | Consider the following configuration | | ________ _______________________ ________ | | Router | loop A | | loop C | Router | | | A |=========| Wide-Area ATM Service |========| C | | |________| |_______________________| |________| | | | | local loop B | | | ________ | | Router | | | B | | |________| How is one doing bandwidth allocation in this scheme? Are we assuming CBR (to be very conservative) (or VBR with V=0 for those that don't do CBR)? If so, then local loop B is really two circuits, one of BW X and the other of BW 1-X. Moreover, continuing with the thought of CBR or the equivalent, if one has a hundred routers in one's network, how does one allocate a fixed bandwidth from B to each router? Or does one overcommit and lean on UBR or ABR or some other *BR? These are serious questions I don't have an answer to, and come a result of having remembered the paragraph below and of remembering that a couple of our competitors have very large (admittedly "Frame Relay") PVC meshes. Sean. | The desire for a full mesh of VCs between routers becomes more compelling | if you have more than three routers. The diagram is left as an exercise | to the reader; I rather dislike having to draw in ASCII.

1 0

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

It's ATM day. | Consider the following configuration | ________ _______________________ ________ | | Router | loop A | | loop C | Router | | | A |=========| Wide-Area ATM Service |========| C | | |________| |_______________________| |________| | | | | local loop B | | | ________ | | Router | | | B | | |________| Consider the following configuration: Ra----fibrepath2----------RbRc------fibrepath2--------Rd or the following: Ra---ll1---[IXCi]---ll2---RbRc---ll3---[IXCii]---ll4--Rd I wonder what the distribution is, in terms of number of backbones, of each of the three models. (You may choose to imagine easy extensions like Ra and Rd really being two routers, with physical connectivity between Ra' and Rd') In terms of designing Internet backbones, do you see any issues other than cost when choosing between your version and either one of my versions? Also, does this change when there are, say, twenty-odd routers at each of the three locations? Sean.

1 0

ATM Wide-Area Networks (was: sell shell accounts?)
by salo＠msc.edu 24 Jul '96

24 Jul '96

> From: Dave Siegel <dave(a)rtd.net> > Subject: Re: sell shell accounts? > To: freedman(a)netaxs.com (Avi Freedman) > Date: Fri, 19 Jul 1996 16:01:01 -0700 (MST) > Cc: vansax(a)atmnet.net, richards(a)netrex.com, agislist(a)interstice.com, > nanog(a)merit.edu > > > Acceptable arguments are: > > o Switches can handle more throughput > > That's difficult to quantify in theory *or* practice. > [...] I may misunderstand your assertion, but it doesn't seem all that difficult to quantify, at least to some coarse level. We have been using wide-area ATM switches at OC-3c for some time. It is pretty clear that the switches can handle OC-3c. The early switches had relatively small output buffers, so they tended to loose cells before TCP could throttle back in congested circumstances. We are now using switches with much larger output buffers, and TCP appears able to throttle back fairly gracefully. After we upgraded our switches to use much larger output buffers, most of the problems we experienced were related to the hosts, (e.g., poor TCP implementations, poor ATM interfaces, etc). By the way, I believe that most people who are using ATM wide-area networks in production or as part of the Internet are using static bandwidth allocation to avoid a number of problems. We also have a number of OC-12c interfaces for our switches. They appear to work, but we haven't really had a chance to stress them yet. (Part of the problem is that it is hard to find OC-12c data sources and sinks.) The current generation of routers appears unlikely to support OC-12c, particularly since their backplanes are only about the same speed. On a more theoretical note, switches, being circuit-switched, make the complicated decisions when the connection is established, (or configured for PVCs), and need to make relatively simple decisions to switch each cell. This probably scales very well is terms of speed, although at some point might have some difficulty in scaling to a very large number of simultaneous connections. Routers, on the other hand, have to make a bit more complicated decisions per packet. This has some limitations in terms of speed and number of simultaneous "connections." -tjs

3 2

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

| o Given that each link is rather expensive, it behooves the | NSP to try to avoid needlessly moving a packet back and | forth across a DS-3 local loop just because my ATM | provider can't provide me with any more PVCs. | (Which was my major point, by the way.) Interestingly enough, the NSPs I observe moving large amounts of traffic around are set up in such a way that their routers are generally colocated with carrier transmission gear, and so there *is* no local loop to speak of, as an added cost to an outside supplier. There are issues involving intra-POP technology and the like, though, admittedly. Moreover, the other folks I observe moving large amounts of traffic have no less than two local-loops built with maximal redundancy in mind (different trenches to different facilities and often using different local carriers), so it is much less a question of moving traffic back and forth across the same DS3 local loop, but rather an extension of the intra-POP technology problem faced by the folks with tight relationships with IXCs and overseas equivalents. Do you still see configuring up large numbers of PVCs to be an advantage in these two cases? Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| At a certain point, some of these arguments about ATM efficiency sound a bit | like saying FDDI is terrible because 4B/5B encoding is | only 80% efficient. But FDDI has a *great* admission-control scheme. However, I almost concede your point. :) [*] | I think a more interesting measure of the value of ATM | versus other wide-area technologies is some sort of | measure of throughput per dollar. No kidding. Sean. P.S.: [*] almost, because in an application where one is using ATM as a single point-to-point PVC along a long-haul trunk that is to be used essentially exclusively for IP traffic, the cell tax seems really excessive. Given the movement towards cisco's packet-over-SONET, should that work and should others implement similar and preferably interoperable point-to-point interfaces, I will have to withdraw my almost-concession, except in cases where people _really_ want to do point-to-multipoint stuff.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| For instance, if the first ATM cell containing a new | IP packet gets lost, the switch may continue to transmit the other 28 | cells in the packet even though these are now useless garbage data that | don't need to be forwarded. There are various hacks at this problem, and to be honest I haven't payed much attention to them. The ones I've looked at tend to involve a specially-marked cell that indicates one or the other end of a packet, or a train of cells each of which is marked. A switch needing to discard a cell will attempt to discard all the related cells, for instance by discarding all cells until an end-of-packet or start-of-packet or whatever is encountered. Some of the schemes are even fairly clever about trying to avoid packet shredding, however none of them are remotely as clever or as effective at congestion avoidance and management as RED is. RED on TCP flows in an ATM switch would impress me. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| It's difficult to quantify wrt to how ATM plays a role in end-to-end | performance on the Internet. There is very little research to support how | ATM affects the overall performance. Even Ameritech and PacBell restricted | the majority of their performance evaluation on | performance in the switch Well, perhaps it's better from me than from someone very fond of ATM: this is comparing apples and oranges. ATM can be used in two main modes: as a funny form of TDM or as a means of overcommitting resources. The former is in use now in the land of InternetMCI, ESNET, NSI and others, for better or for worse, and has been for some time. As Tim and others have pointed out, it works, given some tweaking here and there with things like EPD. The latter has been observed in use as an offering by MFS, and it had obvious problems. It has also been observed from time to time at the ATM-using NAPs, and also had obvious problems. These problems were arguably not directly the fault of ATM, but of arguably broken interface and adaptation devices like ADSUs, ethernet framers and FRADs. At the same time, using *BR and the like to manage the overcommitment of resources has largely been limited in scope, and in particular, I do not know of any serious use involving ABR in particular and a mix of heavily aggregated low-bandwidth traffic competing with traffic with high bandwidth*delay traffic. I also do not know of any serioufs use of IP over ATM in this mode on very long haul lines. (If anyone _does_, drop me a note). I certainly have some hypotheses (namely that it won't work very well), however they haven't been fully tested, and moreover, I don't think it's very smart to test them out using production Internet traffic, given the serious nature of the failure modes we _do_ know about. And for Tim: | if you fill your pipe into the ATM switch, | your ATM switch still becomes a packet shredder, compared to the more | graceful packet drops seen on a clear channel line. "packet shredding" here is the effect seen where attempts at traffic shaping within an ATM cloud results in cells being dropped from within IP datagrams. This is particularly toxic with TCP flows with large delay*bandwidth products, since, as we must assume large congestion windows, the arrival of a damaged segment may drop a flow's goodput through the floor. While FR^2 and SACK appear to help out, the presence of many damaged TCP segments in flight on people's wires will not really win much love for ATM by the people who pay for those wires, and that ultimately is end-to-end. A network that does not behave as a "packet shredder" is one which discards whole IP datagrams, to avoid transmitting damaged goods through fractional paths. However, drops on a clear channel line are not graceful, and only happen when there is insufficient end-to-end buffering in the presence of congestion to allow delayed ACKs to cause transmitters to slow down. On the other hand, it is more difficult to overcommit a clear-channel pipe used principally for TCP by virtue of slow-start, whereas overcommitting a pipe is often considered a feature of ATM, and most ATM switches make it very easy to do. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 23 Jul '96

23 Jul '96

| By the way, I believe that most people who are using ATM wide-area networks | in production or as part of the Internet are using static bandwidth | allocation to avoid a number of problems. Correct. Milo Medin said it best, IMHO: "ATM is really just TDM with really small time-slots". The problem is that many ATM proponents want to believe that ATM qua TDM is not enough, but rather espouse the magic of *BR, particularly ABR, to allow for an overcommittment of long-haul capacity. Nice idea, but it doesn't seem to work very well in practice, and particularly not with a mix of heavily aggregated low-bandwidth TCP flows and TCP flows with large delay*bandwith products. (This is not to say that routers in general (with a few exceptions) and ciscos in particular have fared much better at supporting TCP flows with large delay*bandwidth products, since the amount of buffering available in such routers is on the puny side. Unfortunately the vendors (and many others) have had trouble understanding the relationship between buffering and goodput on high-bandwidth, high-delay TCP streams through a fabric that is carrying other traffic). | We also have a number of OC-12c interfaces for our switches. They | appear to work, but we haven't really had a chance to stress them | yet. (Part of the problem is that it is hard to find OC-12c data | sources and sinks.) I suppose the question is one of objectives: do you want to move huge flows end-to-end, or do you want to move them around on ATM? I subscribe to the former, and to the extent that ATM is currently the only readily available fabric one can put TCP/IP on at that speed, I support playing with ATM at that speed. However, should an alternative come along, I would be very keen on seeing data move on that too (or instead). However, there are certainly a large number of people who would like to follow the second approach: run whatever data can be run at very high speeds, and run them over ATM. I believe that it isn't too much of a stretch to suggest that there is a cultural gap between the two camps. | The current generation of routers appears unlikely to support OC-12c, | particularly since their backplanes are only about the | same speed. NFK, and even if they could, good luck getting anywhere near line rate more than across town if there are multiple simultaneous TCP flows. :-( (Well, some of the current generation of routers are being built with appropriate buffering, I gather, and some might be retrofitted) | On a more theoretical note, switches, being circuit-switched, make | the complicated decisions when the connection is established, (or | configured for PVCs), and need to make relatively simple decisions | to switch each cell. This is the case in any tag-based switching scheme, even in lowly ethernet switches. The problem is what to do in the case of failures, and what the failure modes are in the first place, and, as you say below: | This probably scales very well is terms of | speed, although at some point might have some difficulty in scaling | to a very large number of simultaneous connections. But, you also say: | Routers, on the other hand, have to make a bit more complicated | decisions per packet. which is not always the case, particularly when one begins thinking of tags as a general abstraction, rather than as particular varieties like MAC addresses, VPI/VCIs and the like. Sean.

1 0