- Test - lists.nanog.org

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by salo＠msc.edu 24 Jul '96

24 Jul '96

> From: Dave Siegel <dave(a)rtd.net> > Subject: Re: ATM Wide-Area Networks (was: sell shell accounts?) > To: salo(a)msc.edu (Tim Salo) > Date: Tue, 23 Jul 1996 10:22:23 -0700 (MST) > Cc: dave(a)rtd.net, agislist(a)interstice.com, nanog(a)merit.edu > [...] > Even > with the improved buffering, if you fill your pipe into the ATM switch, > your ATM switch still becomes a packet shredder, compared to the more > graceful packet drops seen on a clear channel line. > [...] "Good" ATM switches are doing Early Packet Discard and Partial Packet Discard. You can ask your favorite switch vendor for details, or look at the following for a nice summary: http://www.fore.com/html/atm-edu/whitep/FTBMWP.html -tjs I am not quite sure how an ATM switch becomes a "packet shredder," or even what a "packet shredder" is...

3 2

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

It's still ATM day, 'cause something else just occurred to me: | Consider the following configuration | | ________ _______________________ ________ | | Router | loop A | | loop C | Router | | | A |=========| Wide-Area ATM Service |========| C | | |________| |_______________________| |________| | | | | local loop B | | | ________ | | Router | | | B | | |________| How is one doing bandwidth allocation in this scheme? Are we assuming CBR (to be very conservative) (or VBR with V=0 for those that don't do CBR)? If so, then local loop B is really two circuits, one of BW X and the other of BW 1-X. Moreover, continuing with the thought of CBR or the equivalent, if one has a hundred routers in one's network, how does one allocate a fixed bandwidth from B to each router? Or does one overcommit and lean on UBR or ABR or some other *BR? These are serious questions I don't have an answer to, and come a result of having remembered the paragraph below and of remembering that a couple of our competitors have very large (admittedly "Frame Relay") PVC meshes. Sean. | The desire for a full mesh of VCs between routers becomes more compelling | if you have more than three routers. The diagram is left as an exercise | to the reader; I rather dislike having to draw in ASCII.

1 0

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

It's ATM day. | Consider the following configuration | ________ _______________________ ________ | | Router | loop A | | loop C | Router | | | A |=========| Wide-Area ATM Service |========| C | | |________| |_______________________| |________| | | | | local loop B | | | ________ | | Router | | | B | | |________| Consider the following configuration: Ra----fibrepath2----------RbRc------fibrepath2--------Rd or the following: Ra---ll1---[IXCi]---ll2---RbRc---ll3---[IXCii]---ll4--Rd I wonder what the distribution is, in terms of number of backbones, of each of the three models. (You may choose to imagine easy extensions like Ra and Rd really being two routers, with physical connectivity between Ra' and Rd') In terms of designing Internet backbones, do you see any issues other than cost when choosing between your version and either one of my versions? Also, does this change when there are, say, twenty-odd routers at each of the three locations? Sean.

1 0

ATM Wide-Area Networks (was: sell shell accounts?)
by salo＠msc.edu 24 Jul '96

24 Jul '96

> From: Dave Siegel <dave(a)rtd.net> > Subject: Re: sell shell accounts? > To: freedman(a)netaxs.com (Avi Freedman) > Date: Fri, 19 Jul 1996 16:01:01 -0700 (MST) > Cc: vansax(a)atmnet.net, richards(a)netrex.com, agislist(a)interstice.com, > nanog(a)merit.edu > > > Acceptable arguments are: > > o Switches can handle more throughput > > That's difficult to quantify in theory *or* practice. > [...] I may misunderstand your assertion, but it doesn't seem all that difficult to quantify, at least to some coarse level. We have been using wide-area ATM switches at OC-3c for some time. It is pretty clear that the switches can handle OC-3c. The early switches had relatively small output buffers, so they tended to loose cells before TCP could throttle back in congested circumstances. We are now using switches with much larger output buffers, and TCP appears able to throttle back fairly gracefully. After we upgraded our switches to use much larger output buffers, most of the problems we experienced were related to the hosts, (e.g., poor TCP implementations, poor ATM interfaces, etc). By the way, I believe that most people who are using ATM wide-area networks in production or as part of the Internet are using static bandwidth allocation to avoid a number of problems. We also have a number of OC-12c interfaces for our switches. They appear to work, but we haven't really had a chance to stress them yet. (Part of the problem is that it is hard to find OC-12c data sources and sinks.) The current generation of routers appears unlikely to support OC-12c, particularly since their backplanes are only about the same speed. On a more theoretical note, switches, being circuit-switched, make the complicated decisions when the connection is established, (or configured for PVCs), and need to make relatively simple decisions to switch each cell. This probably scales very well is terms of speed, although at some point might have some difficulty in scaling to a very large number of simultaneous connections. Routers, on the other hand, have to make a bit more complicated decisions per packet. This has some limitations in terms of speed and number of simultaneous "connections." -tjs

3 2

Re: sell shell accounts?
by Sean Doran 24 Jul '96

24 Jul '96

| o Given that each link is rather expensive, it behooves the | NSP to try to avoid needlessly moving a packet back and | forth across a DS-3 local loop just because my ATM | provider can't provide me with any more PVCs. | (Which was my major point, by the way.) Interestingly enough, the NSPs I observe moving large amounts of traffic around are set up in such a way that their routers are generally colocated with carrier transmission gear, and so there *is* no local loop to speak of, as an added cost to an outside supplier. There are issues involving intra-POP technology and the like, though, admittedly. Moreover, the other folks I observe moving large amounts of traffic have no less than two local-loops built with maximal redundancy in mind (different trenches to different facilities and often using different local carriers), so it is much less a question of moving traffic back and forth across the same DS3 local loop, but rather an extension of the intra-POP technology problem faced by the folks with tight relationships with IXCs and overseas equivalents. Do you still see configuring up large numbers of PVCs to be an advantage in these two cases? Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| At a certain point, some of these arguments about ATM efficiency sound a bit | like saying FDDI is terrible because 4B/5B encoding is | only 80% efficient. But FDDI has a *great* admission-control scheme. However, I almost concede your point. :) [*] | I think a more interesting measure of the value of ATM | versus other wide-area technologies is some sort of | measure of throughput per dollar. No kidding. Sean. P.S.: [*] almost, because in an application where one is using ATM as a single point-to-point PVC along a long-haul trunk that is to be used essentially exclusively for IP traffic, the cell tax seems really excessive. Given the movement towards cisco's packet-over-SONET, should that work and should others implement similar and preferably interoperable point-to-point interfaces, I will have to withdraw my almost-concession, except in cases where people _really_ want to do point-to-multipoint stuff.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| For instance, if the first ATM cell containing a new | IP packet gets lost, the switch may continue to transmit the other 28 | cells in the packet even though these are now useless garbage data that | don't need to be forwarded. There are various hacks at this problem, and to be honest I haven't payed much attention to them. The ones I've looked at tend to involve a specially-marked cell that indicates one or the other end of a packet, or a train of cells each of which is marked. A switch needing to discard a cell will attempt to discard all the related cells, for instance by discarding all cells until an end-of-packet or start-of-packet or whatever is encountered. Some of the schemes are even fairly clever about trying to avoid packet shredding, however none of them are remotely as clever or as effective at congestion avoidance and management as RED is. RED on TCP flows in an ATM switch would impress me. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| It's difficult to quantify wrt to how ATM plays a role in end-to-end | performance on the Internet. There is very little research to support how | ATM affects the overall performance. Even Ameritech and PacBell restricted | the majority of their performance evaluation on | performance in the switch Well, perhaps it's better from me than from someone very fond of ATM: this is comparing apples and oranges. ATM can be used in two main modes: as a funny form of TDM or as a means of overcommitting resources. The former is in use now in the land of InternetMCI, ESNET, NSI and others, for better or for worse, and has been for some time. As Tim and others have pointed out, it works, given some tweaking here and there with things like EPD. The latter has been observed in use as an offering by MFS, and it had obvious problems. It has also been observed from time to time at the ATM-using NAPs, and also had obvious problems. These problems were arguably not directly the fault of ATM, but of arguably broken interface and adaptation devices like ADSUs, ethernet framers and FRADs. At the same time, using *BR and the like to manage the overcommitment of resources has largely been limited in scope, and in particular, I do not know of any serious use involving ABR in particular and a mix of heavily aggregated low-bandwidth traffic competing with traffic with high bandwidth*delay traffic. I also do not know of any serioufs use of IP over ATM in this mode on very long haul lines. (If anyone _does_, drop me a note). I certainly have some hypotheses (namely that it won't work very well), however they haven't been fully tested, and moreover, I don't think it's very smart to test them out using production Internet traffic, given the serious nature of the failure modes we _do_ know about. And for Tim: | if you fill your pipe into the ATM switch, | your ATM switch still becomes a packet shredder, compared to the more | graceful packet drops seen on a clear channel line. "packet shredding" here is the effect seen where attempts at traffic shaping within an ATM cloud results in cells being dropped from within IP datagrams. This is particularly toxic with TCP flows with large delay*bandwidth products, since, as we must assume large congestion windows, the arrival of a damaged segment may drop a flow's goodput through the floor. While FR^2 and SACK appear to help out, the presence of many damaged TCP segments in flight on people's wires will not really win much love for ATM by the people who pay for those wires, and that ultimately is end-to-end. A network that does not behave as a "packet shredder" is one which discards whole IP datagrams, to avoid transmitting damaged goods through fractional paths. However, drops on a clear channel line are not graceful, and only happen when there is insufficient end-to-end buffering in the presence of congestion to allow delayed ACKs to cause transmitters to slow down. On the other hand, it is more difficult to overcommit a clear-channel pipe used principally for TCP by virtue of slow-start, whereas overcommitting a pipe is often considered a feature of ATM, and most ATM switches make it very easy to do. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 23 Jul '96

23 Jul '96

| By the way, I believe that most people who are using ATM wide-area networks | in production or as part of the Internet are using static bandwidth | allocation to avoid a number of problems. Correct. Milo Medin said it best, IMHO: "ATM is really just TDM with really small time-slots". The problem is that many ATM proponents want to believe that ATM qua TDM is not enough, but rather espouse the magic of *BR, particularly ABR, to allow for an overcommittment of long-haul capacity. Nice idea, but it doesn't seem to work very well in practice, and particularly not with a mix of heavily aggregated low-bandwidth TCP flows and TCP flows with large delay*bandwith products. (This is not to say that routers in general (with a few exceptions) and ciscos in particular have fared much better at supporting TCP flows with large delay*bandwidth products, since the amount of buffering available in such routers is on the puny side. Unfortunately the vendors (and many others) have had trouble understanding the relationship between buffering and goodput on high-bandwidth, high-delay TCP streams through a fabric that is carrying other traffic). | We also have a number of OC-12c interfaces for our switches. They | appear to work, but we haven't really had a chance to stress them | yet. (Part of the problem is that it is hard to find OC-12c data | sources and sinks.) I suppose the question is one of objectives: do you want to move huge flows end-to-end, or do you want to move them around on ATM? I subscribe to the former, and to the extent that ATM is currently the only readily available fabric one can put TCP/IP on at that speed, I support playing with ATM at that speed. However, should an alternative come along, I would be very keen on seeing data move on that too (or instead). However, there are certainly a large number of people who would like to follow the second approach: run whatever data can be run at very high speeds, and run them over ATM. I believe that it isn't too much of a stretch to suggest that there is a cultural gap between the two camps. | The current generation of routers appears unlikely to support OC-12c, | particularly since their backplanes are only about the | same speed. NFK, and even if they could, good luck getting anywhere near line rate more than across town if there are multiple simultaneous TCP flows. :-( (Well, some of the current generation of routers are being built with appropriate buffering, I gather, and some might be retrofitted) | On a more theoretical note, switches, being circuit-switched, make | the complicated decisions when the connection is established, (or | configured for PVCs), and need to make relatively simple decisions | to switch each cell. This is the case in any tag-based switching scheme, even in lowly ethernet switches. The problem is what to do in the case of failures, and what the failure modes are in the first place, and, as you say below: | This probably scales very well is terms of | speed, although at some point might have some difficulty in scaling | to a very large number of simultaneous connections. But, you also say: | Routers, on the other hand, have to make a bit more complicated | decisions per packet. which is not always the case, particularly when one begins thinking of tags as a general abstraction, rather than as particular varieties like MAC addresses, VPI/VCIs and the like. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 23 Jul '96

23 Jul '96

| Back to real world considerations. Which scales better depends on | things like average packet size for the router limits and average | connection duration for the switch limits. Er, um, this is really highly dependent on the stability of the routing system. If, in fact, you have very very stable routes and a fast ATM fabric with not-so-crunchy packet-forwarders then one could argue that it may make sense to avoid creating traffic hot-spots in the routers by flattening the network into a full mesh. This still leaves two problems, and that is that this may scale poorly if your packet-forwarder ends up not being able to select among a large number of VCs at line speeds, and also routing in a very large cloud tends to scale very badly. That is, if a router is limited by the number of forwarding decisions per second per next-hop, then having many VCs to select from will reduce rather than increase the useful bandwidth available to flows passing through it, compared to shuffling traffic through only a very small set of VCs or physical circuits. That is to say that the trade off between traffic funneling and building VCs is tricky, it may be counter-intuitive, and it may not work in one's favour, depending on how one's IP router is built. Moreover, as you add instability into the system, whether external to the ATM-using network or internal to it, your ability to converge decreases in proportion to the amount of meshing in a network. This is an IP effect rather than a flaw in any particular router or routing-protocol design, although such flaws will worsen the problem considerably. So, if one identifies routing instability as a key problem in scaling the Internet, then goes off and builds a large, fully-meshed network out of boxes known to have some difficulties in managing routing instability in very simple topologies, one has to question his or her sanity. | As a result, hybrid approaches start to look attractive, | using routers on the periphery and building fat pipes | through the switches. "The switches" has a very broad range of interpretations. Essentially what one is doing here and in other schemes -- two of which I rather like, btw -- is exchanging a prefix/mask and possibly other information for a value or set of values that will be used to make switching decisions through a set of routers. One can do this with ATM. One can do this with FR. One can do this with FDDI. One can do this with ethernet. One can make up one's own tag-based switching scheme, too, optimize it for Internet traffic, and maybe learn from some of the thing that other switching schemes got right, and avoid the things they got wrong. | The state of the art of ISP needs (where the barage of tinygrams and | very short flows is felt full force) is PVC pipes between routers to | offload the routers a little bit. Hm. Well, what the state of the art ISP needs is more of a tag-based scheme, whatever the variety of tag is, but ideally one that's well-designed for the type of traffic a state of the art ISP is likely to see. Or at least, that's the fast silicon school of IP router design. There are at least two other schools of router design, one of which has been talked about by Vadim Antonov (use a hybercube computer and oh, by the way, use some of the spare CPUs to do things like web caching, news distribution, accounting, video-on-demand, playing games and so forth), and the other of which may or may not have been discussed here by its proponents. The first school is of interest because it is an obvious evolutionary step, and has been partially implemented by people throwing routers around FR switches and ATM switches and FDDI switches and the like. The second school is interesting because it has the appeal of added functionality and relatively easy scaling up to massive aggregate bandwidths, although one may have to couple that with many wires between the box in question and telco transmission MUXes or circuit-termination gear. The third school is just interesting, 'cause it's weird. :-) I am not quite sure whether I would count Craig Partridge's work as being a fourth school, actually, but what the heck. | For all the investment made in custom silicon for ATM it | may turn out that a general purpose processor and a good | router design (such as the DEC Alpha used in the BBN | router) will take us to OC12. Yes, precisely. The drawback of the first school of making IP go fast is that it takes a while and is expensive to build fast silicon. Using readily-available parts to build a router has an attraction for that reason, but the drawback there is getting the software right. This is true both of the second school and in Partridge's design, and it is also a drawback in any design where there is a temptation to do caching to squeeze more speed out because of a time lag in acquiring new parts. | Hope its been amusing. I'd say the jury is still out on | this one. :-) Well, I wouldn't go that far. I will admit that ATM hasn't been made fully irrelevant yet, although some people do have that as a goal. Sean.

1 0