- NANOG - lists.nanog.org

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| At a certain point, some of these arguments about ATM efficiency sound a bit | like saying FDDI is terrible because 4B/5B encoding is | only 80% efficient. But FDDI has a *great* admission-control scheme. However, I almost concede your point. :) [*] | I think a more interesting measure of the value of ATM | versus other wide-area technologies is some sort of | measure of throughput per dollar. No kidding. Sean. P.S.: [*] almost, because in an application where one is using ATM as a single point-to-point PVC along a long-haul trunk that is to be used essentially exclusively for IP traffic, the cell tax seems really excessive. Given the movement towards cisco's packet-over-SONET, should that work and should others implement similar and preferably interoperable point-to-point interfaces, I will have to withdraw my almost-concession, except in cases where people _really_ want to do point-to-multipoint stuff.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| For instance, if the first ATM cell containing a new | IP packet gets lost, the switch may continue to transmit the other 28 | cells in the packet even though these are now useless garbage data that | don't need to be forwarded. There are various hacks at this problem, and to be honest I haven't payed much attention to them. The ones I've looked at tend to involve a specially-marked cell that indicates one or the other end of a packet, or a train of cells each of which is marked. A switch needing to discard a cell will attempt to discard all the related cells, for instance by discarding all cells until an end-of-packet or start-of-packet or whatever is encountered. Some of the schemes are even fairly clever about trying to avoid packet shredding, however none of them are remotely as clever or as effective at congestion avoidance and management as RED is. RED on TCP flows in an ATM switch would impress me. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 24 Jul '96

24 Jul '96

| It's difficult to quantify wrt to how ATM plays a role in end-to-end | performance on the Internet. There is very little research to support how | ATM affects the overall performance. Even Ameritech and PacBell restricted | the majority of their performance evaluation on | performance in the switch Well, perhaps it's better from me than from someone very fond of ATM: this is comparing apples and oranges. ATM can be used in two main modes: as a funny form of TDM or as a means of overcommitting resources. The former is in use now in the land of InternetMCI, ESNET, NSI and others, for better or for worse, and has been for some time. As Tim and others have pointed out, it works, given some tweaking here and there with things like EPD. The latter has been observed in use as an offering by MFS, and it had obvious problems. It has also been observed from time to time at the ATM-using NAPs, and also had obvious problems. These problems were arguably not directly the fault of ATM, but of arguably broken interface and adaptation devices like ADSUs, ethernet framers and FRADs. At the same time, using *BR and the like to manage the overcommitment of resources has largely been limited in scope, and in particular, I do not know of any serious use involving ABR in particular and a mix of heavily aggregated low-bandwidth traffic competing with traffic with high bandwidth*delay traffic. I also do not know of any serioufs use of IP over ATM in this mode on very long haul lines. (If anyone _does_, drop me a note). I certainly have some hypotheses (namely that it won't work very well), however they haven't been fully tested, and moreover, I don't think it's very smart to test them out using production Internet traffic, given the serious nature of the failure modes we _do_ know about. And for Tim: | if you fill your pipe into the ATM switch, | your ATM switch still becomes a packet shredder, compared to the more | graceful packet drops seen on a clear channel line. "packet shredding" here is the effect seen where attempts at traffic shaping within an ATM cloud results in cells being dropped from within IP datagrams. This is particularly toxic with TCP flows with large delay*bandwidth products, since, as we must assume large congestion windows, the arrival of a damaged segment may drop a flow's goodput through the floor. While FR^2 and SACK appear to help out, the presence of many damaged TCP segments in flight on people's wires will not really win much love for ATM by the people who pay for those wires, and that ultimately is end-to-end. A network that does not behave as a "packet shredder" is one which discards whole IP datagrams, to avoid transmitting damaged goods through fractional paths. However, drops on a clear channel line are not graceful, and only happen when there is insufficient end-to-end buffering in the presence of congestion to allow delayed ACKs to cause transmitters to slow down. On the other hand, it is more difficult to overcommit a clear-channel pipe used principally for TCP by virtue of slow-start, whereas overcommitting a pipe is often considered a feature of ATM, and most ATM switches make it very easy to do. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 23 Jul '96

23 Jul '96

| By the way, I believe that most people who are using ATM wide-area networks | in production or as part of the Internet are using static bandwidth | allocation to avoid a number of problems. Correct. Milo Medin said it best, IMHO: "ATM is really just TDM with really small time-slots". The problem is that many ATM proponents want to believe that ATM qua TDM is not enough, but rather espouse the magic of *BR, particularly ABR, to allow for an overcommittment of long-haul capacity. Nice idea, but it doesn't seem to work very well in practice, and particularly not with a mix of heavily aggregated low-bandwidth TCP flows and TCP flows with large delay*bandwith products. (This is not to say that routers in general (with a few exceptions) and ciscos in particular have fared much better at supporting TCP flows with large delay*bandwidth products, since the amount of buffering available in such routers is on the puny side. Unfortunately the vendors (and many others) have had trouble understanding the relationship between buffering and goodput on high-bandwidth, high-delay TCP streams through a fabric that is carrying other traffic). | We also have a number of OC-12c interfaces for our switches. They | appear to work, but we haven't really had a chance to stress them | yet. (Part of the problem is that it is hard to find OC-12c data | sources and sinks.) I suppose the question is one of objectives: do you want to move huge flows end-to-end, or do you want to move them around on ATM? I subscribe to the former, and to the extent that ATM is currently the only readily available fabric one can put TCP/IP on at that speed, I support playing with ATM at that speed. However, should an alternative come along, I would be very keen on seeing data move on that too (or instead). However, there are certainly a large number of people who would like to follow the second approach: run whatever data can be run at very high speeds, and run them over ATM. I believe that it isn't too much of a stretch to suggest that there is a cultural gap between the two camps. | The current generation of routers appears unlikely to support OC-12c, | particularly since their backplanes are only about the | same speed. NFK, and even if they could, good luck getting anywhere near line rate more than across town if there are multiple simultaneous TCP flows. :-( (Well, some of the current generation of routers are being built with appropriate buffering, I gather, and some might be retrofitted) | On a more theoretical note, switches, being circuit-switched, make | the complicated decisions when the connection is established, (or | configured for PVCs), and need to make relatively simple decisions | to switch each cell. This is the case in any tag-based switching scheme, even in lowly ethernet switches. The problem is what to do in the case of failures, and what the failure modes are in the first place, and, as you say below: | This probably scales very well is terms of | speed, although at some point might have some difficulty in scaling | to a very large number of simultaneous connections. But, you also say: | Routers, on the other hand, have to make a bit more complicated | decisions per packet. which is not always the case, particularly when one begins thinking of tags as a general abstraction, rather than as particular varieties like MAC addresses, VPI/VCIs and the like. Sean.

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Sean Doran 23 Jul '96

23 Jul '96

| Back to real world considerations. Which scales better depends on | things like average packet size for the router limits and average | connection duration for the switch limits. Er, um, this is really highly dependent on the stability of the routing system. If, in fact, you have very very stable routes and a fast ATM fabric with not-so-crunchy packet-forwarders then one could argue that it may make sense to avoid creating traffic hot-spots in the routers by flattening the network into a full mesh. This still leaves two problems, and that is that this may scale poorly if your packet-forwarder ends up not being able to select among a large number of VCs at line speeds, and also routing in a very large cloud tends to scale very badly. That is, if a router is limited by the number of forwarding decisions per second per next-hop, then having many VCs to select from will reduce rather than increase the useful bandwidth available to flows passing through it, compared to shuffling traffic through only a very small set of VCs or physical circuits. That is to say that the trade off between traffic funneling and building VCs is tricky, it may be counter-intuitive, and it may not work in one's favour, depending on how one's IP router is built. Moreover, as you add instability into the system, whether external to the ATM-using network or internal to it, your ability to converge decreases in proportion to the amount of meshing in a network. This is an IP effect rather than a flaw in any particular router or routing-protocol design, although such flaws will worsen the problem considerably. So, if one identifies routing instability as a key problem in scaling the Internet, then goes off and builds a large, fully-meshed network out of boxes known to have some difficulties in managing routing instability in very simple topologies, one has to question his or her sanity. | As a result, hybrid approaches start to look attractive, | using routers on the periphery and building fat pipes | through the switches. "The switches" has a very broad range of interpretations. Essentially what one is doing here and in other schemes -- two of which I rather like, btw -- is exchanging a prefix/mask and possibly other information for a value or set of values that will be used to make switching decisions through a set of routers. One can do this with ATM. One can do this with FR. One can do this with FDDI. One can do this with ethernet. One can make up one's own tag-based switching scheme, too, optimize it for Internet traffic, and maybe learn from some of the thing that other switching schemes got right, and avoid the things they got wrong. | The state of the art of ISP needs (where the barage of tinygrams and | very short flows is felt full force) is PVC pipes between routers to | offload the routers a little bit. Hm. Well, what the state of the art ISP needs is more of a tag-based scheme, whatever the variety of tag is, but ideally one that's well-designed for the type of traffic a state of the art ISP is likely to see. Or at least, that's the fast silicon school of IP router design. There are at least two other schools of router design, one of which has been talked about by Vadim Antonov (use a hybercube computer and oh, by the way, use some of the spare CPUs to do things like web caching, news distribution, accounting, video-on-demand, playing games and so forth), and the other of which may or may not have been discussed here by its proponents. The first school is of interest because it is an obvious evolutionary step, and has been partially implemented by people throwing routers around FR switches and ATM switches and FDDI switches and the like. The second school is interesting because it has the appeal of added functionality and relatively easy scaling up to massive aggregate bandwidths, although one may have to couple that with many wires between the box in question and telco transmission MUXes or circuit-termination gear. The third school is just interesting, 'cause it's weird. :-) I am not quite sure whether I would count Craig Partridge's work as being a fourth school, actually, but what the heck. | For all the investment made in custom silicon for ATM it | may turn out that a general purpose processor and a good | router design (such as the DEC Alpha used in the BBN | router) will take us to OC12. Yes, precisely. The drawback of the first school of making IP go fast is that it takes a while and is expensive to build fast silicon. Using readily-available parts to build a router has an attraction for that reason, but the drawback there is getting the software right. This is true both of the second school and in Partridge's design, and it is also a drawback in any design where there is a temptation to do caching to squeeze more speed out because of a time lag in acquiring new parts. | Hope its been amusing. I'd say the jury is still out on | this one. :-) Well, I wouldn't go that far. I will admit that ATM hasn't been made fully irrelevant yet, although some people do have that as a goal. Sean.

1 0

Re: sell shell accounts?
by Paul Ferguson 23 Jul '96

23 Jul '96

At 05:06 PM 7/23/96 -0500, Tim Salo wrote: > >The desire for a full mesh of VCs between routers becomes more compelling >if you have more than three routers. The diagram is left as an exercise >to the reader; I rather dislike having to draw in ASCII. > >Note that the two paths between A and C, (directly versus via B), seem >to take distinctly different paths. (Perhaps, I missed your point). > Perhaps I missed your point. Tim, I could've sworn that you had been following the IETF ION WG mailing list, where this has been discussed on hundreds of occasions. I hesitate to even mention this on NANOG (I really don't want to re-hash this on this list), but I can't hold my tongue. The scalability of an ATM network where each end-point is fully meshed to all other end-points decreases in direct proportion to the number of end-points. This is not so much a layer 2 problem, per se, but rather the ability to maintain functional routing (convergence, et al.) in a fully meshed network topology. This does not scale. - paul

1 0

Re: sell shell accounts?
by salo＠msc.edu 23 Jul '96

23 Jul '96

> From: Avi Freedman <freedman(a)netaxs.com> > Subject: Re: sell shell accounts? > To: vansax(a)atmnet.net (Jim Van Baalen) > Date: Fri, 19 Jul 1996 17:17:19 -0400 (EDT) > Cc: richards(a)netrex.com, agislist(a)interstice.com, nanog(a)merit.edu > [...] > Nope, remember - there is no magic. Any mesh of PVCs that one makes > over a switched network must reflect the toplogy of that network, and > one can set up a matching set of active routing sessions and route > weights which will cause traffic to flow the same way. > [...] If I understand what you are asserting, I probably don't agree with it. I assume that we are talking about using wide-area ATM networks, (or more specifically, wide-area ATM services provided by, for example, carriers). Consider the following configuration ________ _______________________ ________ | Router | loop A | | loop C | Router | | A |=========| Wide-Area ATM Service |========| C | |________| |_______________________| |________| | | local loop B | ________ | Router | | B | |________| The "long" path in this diagram between Router A and Router C is through Router B, with a VC between (A,B) and a VC between (B,C). The "shortcut" path in this diagram is a VC between (A,C). The biggest disadvantage of routing packets between A and C via B is that every packet (needlessly) traverses local loop B twice. Furthermore, this local loop is perhaps the most expensive component in the system. The desire for a full mesh of VCs between routers becomes more compelling if you have more than three routers. The diagram is left as an exercise to the reader; I rather dislike having to draw in ASCII. Note that the two paths between A and C, (directly versus via B), seem to take distinctly different paths. (Perhaps, I missed your point). It is not clear to me that being able to ping router B along the path between A and C buys you much. It sounds a bit (well, maybe only a bit) like wanting to cut a T1 in half to install a router in the middle so that you can tell which half of the T1 went down. (Again, I am assuming that the network in question is using ATM services, so the carrier is responsible for keeping the VCs up.) -tjs

1 0

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by Paul Ferguson 23 Jul '96

23 Jul '96

Agreed. However, while one technology may be more economically efficient today, it certainly may not tomorrow. Also, economic comparisons need to also take into account functionality (or lack thereof), administrative & maintenance overhead, etc., not just monthly recurring transport costs. - paul At 02:50 PM 7/23/96 -0500, Tim Salo wrote: > >I think a more interesting measure of the value of ATM versus other >wide-area technologies is some sort of measure of throughput per dollar. > >-tjs >

1 0

T3 or not to T3
by dgaudet＠hotwired.com 23 Jul '96

23 Jul '96

I've been doing a lot of reading and thinking on what the best solution is for an application that initially requires approximately 15Mbps outgoing and 8Mbps incoming (as viewed from my router), and talks with 500000 unique hosts daily (i.e. has fairly wide coverage of the net). The application involves at least thirty machines, so colocation is likely to be cost-prohibitive. A single T3, or frac T3 isn't an option because there isn't a single provider that I can trust for the availability we want. Even ignoring availability, I seriously doubt that any provider can consistently fill a single customer's T3 pipe these days. >From stuff I've seen here and elsewhere I think the most important reason for this is congestion at NAPs making it impossible to suck (or shove) lots of bandwidth at anything but your provider's backbone. Taking all of this into account, I'm really leaning towards a solution that involves lots of small pipes to lots of providers. Essentially eliminating the need for 90% of our packets to traverse NAPs by using each backbone mostly for their own customers. Now that I've done my homework, I'd like to hear comments from some of the more experienced folk here. I haven't considered yet the maintenance/logistical cost of managing 15 T1s to 6 or 7 providers vs. the "ease" of two frac-T3s to two providers. >From a provider's point of view, if a site wanted to connect, and was willing to sign a use-policy saying they wouldn't use the connection for transit to other providers (i.e. would only ask for customer BGP and only route to the nets you provide in BGP updates), would that site have lower costs associated with it? (that you could pass on?) Thanks, Dean

11 17

Re: ATM Wide-Area Networks (was: sell shell accounts?)
by salo＠msc.edu 23 Jul '96

23 Jul '96

> Date: Tue, 23 Jul 1996 14:28:27 -0400 > From: Paul Ferguson <pferguso(a)cisco.com> > Subject: Re: ATM Wide-Area Networks (was: sell shell accounts?) > [...] > It depends on what you mean when you say 'performance'. I think > there is more and more interest being vested in ATM inefficiency, > and alternate technologies to better efficiency in the long-haul. > > Recall Jerry Scharf's numbers; they're indicative of the issue. > [...] > HDLC framing bytes = 3080633605 HDLC efficiency = 97.72 > ATM framing bytes = 3644304857 ATM efficiency = 82.61 > ATM w/snap framing bytes = 3862101043 ATM w/snap efficiency = 77.95 At a certain point, some of these arguments about ATM efficiency sound a bit like saying FDDI is terrible because 4B/5B encoding is only 80% efficient. I think a more interesting measure of the value of ATM versus other wide-area technologies is some sort of measure of throughput per dollar. -tjs

1 0