- NANOG - lists.nanog.org

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by William Allen Simpson 20 Dec '96

20 Dec '96

> From: Jim Van Baalen <vansax(a)atmnet.net> > I have a question that fits this topic. Why does everybody seem to be so > sold on Gigaswitch based Xchange points? A pretty good reason: it worked a lot better than everything else tried! As a bit of history, the only NAP that worked was with ethernet, which seemed a lot faster to most folks than the T1 links coming into the NAPs. But, the implementation was terribly flaky as a MAN technology. Then, ethernet got too congested, and folks moved to FDDI. Then, FDDI got congested and GigaSwitches were put in. They were the only thing available, and they made everything work better. My guess is that the next step will toss the LAN/MAN model for exchange GigaRouters directly connected by OC-3c/STM-1 and OC-12c/STM-4 WAN links -- but that's only a guess. It seems to be working. We are driven by things that actually work. > Based on membership and traffic it > appears that there is still a stigma associated with Xchanges (PBnap and AADS > for example) that have chosen different architectures. It was also my > impression that people were much more critical of these "other" NAPS at the > recent NANOG than SprintNAP and the MAEs. > That's because the ATM NAPs were started at the same time as MAE-East, but didn't work! They all took more than an extra year to get working. Oh, yeah, they cost a lot of money for some projects that really couldn't afford it, and it all went down the drain. Folks have a tendency to be critical of stuff with a history of failure. > In addition, > with new line cards due out early next year, the BPXs will support ABR and, > relatively speaking, huge buffers at high density OC3 and 2 port OC12. > Well, since they don't exist, why would anyone bank on deploying them? Let us know how well they work in a year or two. Meanwhile, PPP/SONET has been deployed for over 6 months at these speeds, and we are already getting experience with it. Yep, needs bigger buffers on those Ciscos, and PMC-Sierra didn't follow the SONET spec exactly, but ... experience is a much better teacher than promises. WSimpson(a)UMich.edu Key fingerprint = 17 40 5E 67 15 6F 31 26 DD 0D B9 9B 6A 15 2C 32 BSimpson(a)MorningStar.com Key fingerprint = 2E 07 23 03 C5 62 70 D3 59 B1 4F 5E 1D C2 C1 A2

3 5

RE: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by Chris A. Icide 20 Dec '96

20 Dec '96

>> And you don't find the 7500 a big jump over the 7000? >> What are you trying to do, IP tunnel your data through another backbone? > >Ok, yes they added some memory and it is faster, but they needed to put a >MUCH bigger CPU in that box. It was a jump, but not as big as the one we >all needed. The 7500 was not any new technology, just adding more of the >current. And no we are not tunneling through another backbone, we run our >backbone over Cascade 9000 with clear channel DS3s. Definitely, the 75XX is a jump over the 70XX router platform. However, it is very simple to collapse a 7513. The RSP2 does not have enough processing capability to allow economical use of the 11 available slots. We've been able to collapse several 7513's with less than half of the slots filled and heavy tuning by Cisco Engineers. Now, I am in no way, form, or manner an authoritative Cisco information source, however, I have heard some rumors which I place fairly strong reliability. 1. Cisco will be releasing a new RSP card soon (RSP4) which doubles the performance of the RSP2 2. You all know about the VIP cards along with 11.2 when all of the bugs are removed (especially those nasty memory leaks) 3. There has been a rumor that some level of load sharing between two RSP's will be available some time 2Q or 3Q '97 Again, these are but rumors I've heard. We've been spending a lot of our time lately playing with the Bay (Wellfleet) stuff, and they have some pretty neat stuff coming out (thanks to ANS, any comments, Curt?) 1. They are working on a 64M version of the ARE card (dual Power PC chips working in a full load share environment) 2. a 256M FRE3 card (dual Power PC's again) that they estimate would be completely able to support as many as 300 BGP peers with 45K route tables. 3. Channelized T3 interface, native IP over non-traditional IP media, etc. Again, mainly all rumors. Chris A. Icide Nap.Net, L.L.C.

2 1

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by salo＠msc.edu 20 Dec '96

20 Dec '96

> From: Jim Van Baalen <vansax(a)atmnet.net> > Subject: Re: NAP/ISP Saturation WAS: Re: Exchanges that matter... > Date: Fri, 20 Dec 1996 03:57:47 -0800 (PST) > > ... Why does everybody seem to be so > sold on Gigaswitch based Xchange points? ... A few random thoughts: o I believe that the DEC GigaSwitch has been demonstrated, under very heavy production loads, to be far superior to any other available technology. Technology superior to the GigaSwitch might exist, but I don't believe anyone has demonstrated that technology under heavy, 24x7 load. The alternative technology needs someone to bet their exchange on putting 200-400 mbps of continuous Internet traffic across the new technology. o When the GigaSwitch was first selected for an exchange, alternative technology was not as well developed. ATM switches still had, (with somewhere between zero and two exceptions), small buffers. A lot of ATM equipment was simply bad, at least for wide-area applications. The fact that the Cisco AIP card did not have a DS-3 interface also hurt, (fairly seriously, it turns out). o Note that if an exchange bets on ATM, its fate also depends upon the quality of the ATM interfaces in the attached routers. In particular, the Cisco AIP card becomes a critical component in the exchange. Before I placed too heavy of a bet on an ATM exchange, I would want to see a good performance analysis of the Cisco AIP card and the ATM interfaces for any other router which might attach. If the router ATM interfaces are not as good as the router FDDI interfaces, an ATM exchange, as a whole, is likely to be less successful than an FDDI exchange. By the way, I would assume that router vendors have spent more resources on improving the performance of FDDI interfaces than ATM interfaces. > Based on membership and traffic it > appears that there is still a stigma associated with Xchanges (PBnap and AADS > for example) that have chosen different architectures. ... I think several unfortunate events plagued the PacBell and AADS exchanges. o Because it was not clear that a Cisco DS-3 AIP card would be available in the required timeframe, a decision was made to use ATM DSUs and the Cisco HSSI interface, (a decision in which I played a modest role). o It turns out that the selected ATM DSU had implementation problems which took a while to work out. ATM DSUs in general, I believe, also have the problem that they simply aren't designed for the sort of load that is experienced at an Internet exchange. The Cisco HSSI card might also be weaker than the Cisco AIP card. o It is possible that the two exchanges remained committed to an ADSU solution longer than they should have, particularly when it became apparent that the Cisco DS-3 AIP card was available (something I had no role in). So, history may play a bit of a role in the current position of the PacBell and AADS exchanges. On the other hand, it appears that the Internet operational folks are unlikely to try anything as new and as unproven as ATM until they have no other choice. I suspect that the next time ATM products and standards are called upon to carry production Internet traffic, they will be ready to support a heavy duty Internet load. I think ATM products and standards are [perhaps almost, perhaps just] ready to support the traffic loads experienced at Internet exchanges. It would be nice to see a heavily loaded ATM exchange or perhaps a mixed ATM/FDDI exchange. -tjs

1 0

SprintLink Song
by Vadim Antonov 20 Dec '96

20 Dec '96

Inspired by Sprintlink's service, and written by 3 very creative guys: (apologies to whoever wrote the real song) Oh, the network outside is frightful, But on campus, it's so delightful, Our packets have nowhere to go, Net is slow, net is slow, net is slow. It doesn't show signs of stopping, All our packets, our hosts are dropping; Bandwidth is turned way down low, Net is slow, net is slow, net is slow. When we finally connect to a site, It's time to go back to the dorm; But if I could stay here all night, I could submit their Web form. The network is slowly dying, And, I fear, we're still denying, But as long as Sprint is the way to go, Net is slow, net is slow, net is slow. -(new lyrics by kube, bhchan, ekim, & norby) kube(a)csua.berkeley.edu bhchan(a)csua.berkeley.edu ekim(a)csua.berkeley.edu norby(a)csua.berkeley.edu

1 0

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by Vadim Antonov 18 Dec '96

18 Dec '96

Craig Nordin <cnordin(a)vni.net> wrote: >> Er. There's no such thing as perfect transport as long as TCP is >> concerned. If end-hosts support large windows, even a single >> TCP session will load the network to the point where it'll lose >> packets. >What you say here makes sense to me. But, out of a 500 >ping sample over the course of half a day, I was getting well >below one percent loss -- six months ago. Most TCP sessins are pretty short, in fact. They simply have no chance to open window. Also, there's a lot of old TCP implementations which have window limited to 64Kb (or less!). >Now, my same sample group is going for more than three percent, >with many ten percent loss routes showing... Well, nobody argues that Internet is overloaded. But the question of how to measure the congestion is not as simple as just pinging. The problem is that there's no reasonable way to learn loss statistics from routers. --vadim

2 1

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by Paul Ferguson 18 Dec '96

18 Dec '96

At 06:07 AM 12/17/96 -0000, James D. Wilson wrote: > >And if you are pinging across frame relay your ICMP packet loss might >be your packets above the CIR which have been discarded and also not >indicative of IP packet loss. > If you explicitly introduce L2 connectivity into this paradigm, it gets much, much more complicated. - paul

1 0

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by James D. Wilson 17 Dec '96

17 Dec '96

---------- > From: Vadim Antonov <avg(a)pluris.com> > Pings are ok, as long as you realize what exactly is being measured. > If you're pinging ciscos, you aren't measuring IP packet loss. And if you are pinging across frame relay your ICMP packet loss might be your packets above the CIR which have been discarded and also not indicative of IP packet loss. #include <standard.disclaimer> _ __ __ _____ ____ / | / /__ / /_/ ___/__ _______/ __/__ _____ / |/ / _ \/ __/\__ \/ / / / ___/ /_/ _ \/ ___/ / /| / __/ /_ ___/ / /_/ / / / __/ __/ / ================/_/=|_/\___/\__//____/\__,_/_/==/_/==\___/_/========== =====

2 1

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by Tony Li 17 Dec '96

17 Dec '96

* ICMP packets are dropped by busy routers Many routers drop ICMP packets (ping, traceroute) when busy, or alternate dropping ICMP packets. I know that this behavior occurs when the packets are directed to the specific router, I am not sure if this every occurs for packets passing through. The standby tool ping needs a more reliable replacement for testing end to end packet loss. There seems to be a great deal of (understandable) confusion on this issue. Let's set it straight: Packets which are _successfully_ forwarded through a (high end) cisco router are not (by default) prioritized by protocol type. Packets which are not forwarded require more work and are effectively rate limited (and consume large amounts of CPU time). Some effects: - Pinging a cisco is not a valid measure of packet loss. It's closer to a CPU load measure than anything else. - Pinging _thru_ a cisco is reasonable. - Traceroute to a cisco is rate limited to one reply per second, so will almost always miss the middle reply. - Traceroute _thru_ a cisco may show many drops which would NOT be seen by normal "thru" traffic. Replies generated by the cisco when the TTL expires are again thru the CPU. So you may well traceroute thru a cisco which does not reply at all. However, you can clearly see the route after that router. * Head of queue blocking in the Gigaswitch Even though the Gigaswitch has input and output queues, your output queue will block until the other providers input queue is free. My (admittedly second hand) understanding is that the Gigaswitch/FDDI actually has minimal amounts of buffering. During a congestion event, it simply withholds the token, resulting in buffering in the routers. Queues there eventually overflow, and ... If this is incorrect, I would greatly appreciate pointers to the truth. Tony

3 2

Re: Exchanges that matter...
by salo＠msc.edu 17 Dec '96

17 Dec '96

> Subject: Re: Exchanges that matter... > Date: Fri, 29 Nov 1996 11:02:24 -0800 > From: Paul A Vixie <paul(a)vix.com> > [...] > ... and they aren't subject to ATM's cell tax ... I am surprised, (well, maybe not), that you aren't concerned about the excessive overhead present in FDDI networks... -tjs From: salo(a)msc.edu (Tim Salo) Date: Tue, 23 Jul 1996 14:50:11 -0500 (CDT) > Date: Tue, 23 Jul 1996 14:28:27 -0400 > From: Paul Ferguson <pferguso(a)cisco.com> > [...] > Recall Jerry Scharf's numbers; they're indicative of the issue. > [...] > HDLC framing bytes = 3080633605 HDLC efficiency = 97.72 > ATM framing bytes = 3644304857 ATM efficiency = 82.61 > ATM w/snap framing bytes = 3862101043 ATM w/snap efficiency = 77.95 At a certain point, some of these arguments about ATM efficiency sound a bit like saying FDDI is terrible because 4B/5B encoding is only 80% efficient. I think a more interesting measure of the value of ATM versus other wide-area technologies is some sort of measure of throughput per dollar.

11 12

Re: NAP/ISP Saturation WAS: Re: Exchanges that matter...
by salo＠msc.edu 16 Dec '96

16 Dec '96

> To: salo(a)msc.edu (Tim Salo) > Date: Mon, 16 Dec 1996 17:16:40 -0500 (EST) > From: dvv(a)sprint.net (Dima Volodin) > [...] > > So, do I understand you correctly as saying that you have no interest > > in helping your customers understand the state of your service? > > No, I neither have said nor intend to say anything like that. Yes, I was being a bit unfair. Rather, I should have asked: "How do you recommend that your customers proactively monitor the status of your network and the wider Internet?" I took a quick look at the SprintLink handbook, but probably missed the recommendation. -tjs

1 0