From: Sean Donelan <SEAN@SDG.DRA.COM> To: nanog@merit.edu Subject: Re: Internet Backbone Index Date: Wednesday, July 09, 1997 5:45 PM
SAVVIS. Fortunately Our target markets are not just libraries and other information providers, it's EVERYONE that needs a T1 and above connection to the Internet. How many cities are you in Sean, where are DRA's POPs for customers to access? How much bandwidth does DRA have to get these customer to other network? Let's compare bandwidth shall we.
DRA tries to do one thing well, rather than a lot of things not so well.
Like all things, the correct answer is "It depends." If you are interested in the DRA network geography, a pretty picture of the DRA North American POPs is at <http://dranet.dra.com/dranet.html>. BTW, all those locations are up, operational, and have DRA owned and operated equipment in place. DRA's international offices are located in Montreal Canada, Paris France, and Melbourne Australia. The primary NOC is in St. Louis, MO and backup NOC operates out of Monterey, CA.
Since DRA also has a lot of private connections (in the old NSFNET AUP days these were called 'backdoor' connections) calculating total bandwidth is a bit confusing. If you follow the AGIS bandwidth counting method, DRA has about 268 Mbps of possible bandwidth to other networks through public and private inter-connects, although DRA doesn't have any single link faster than 45 Mbps. However, bandwidth is rarely the problem. I've found a well engineered 56Kbps connection outperformed a DS3 port into a poorly engineered network.
When 80 to 90 percent of the Internet traffic is to MCI, SPRINT and UUNET then our model is the right way to build this, not to try and see how many peering agreements one can get.
Except the model falls apart if 50% percent of the your Internet traffic isn't just to MCI, Sprint and UUNET, as in DRA's case. If you want to get anywhere off the commercial beaten path, like many of our customers do, things quickly get very bad if you stick with just those three providers. Most of our backdoor connections exist exactly for that reason.
Look at the KEYNOTE test sites. Although it might appear heavly weighted towards MCI, SPRINT, and UUNET (23 out of 35 sites), the network is never so simple. It works well as long as you are in a US city going to another US city. But looking at the Bell Canada graph, things don't work as well. Hint: If you are trying to reach a Canadian audience, don't put your web server in a US city. We've had a private 'backboor' connection to the University of Toronto for years for precisely this reason.
Further, about half the Keynote test sites seemed to have alternate connections. I couldn't figure out the numbers until I started doing traceroutes, and then things started making more sense. You'll discover
best route isn't always through MCI, Sprint or UUNET. Those little exchange points can make a difference.
St. Louis, MO MCI 207.230.62.16 Cybercon/Internet 1st $ traceroute 207.230.62.16 traceroute to 207.230.62.16 (207.230.62.16), 30 hops max, 38 byte packets 1 StLouis22-e3.dra.net (192.65.218.2) 200 ms 10 ms 0 ms 2 stlouix.starnet.net (198.32.132.12) 10 ms 20 ms 0 ms 3 e0-1-2.starnet2.starnet.net (199.217.255.97) 70 ms 10 ms 10 ms 4 router.cybercon.com (199.217.252.58) 10 ms 20 ms 10 ms 5 207.230.62.16 (207.230.62.16) 20 ms 10 ms 10 ms
It is amusing to look at AGIS's chart in the Boardwatch directory. The graph dramatically demostrates how badly AGIS's tough-line peering policy hurt it in this test. Refusing to peer with CRL or Goodnet wouldn't have changed CRL's or Goodnet's performance very much, but it does appear to hurt AGIS's performance. Maybe Sprint and MCI might want to re-think
Sean, we only have 805 Mbps of possible bandwidth to other networks. I not sure what you mean by a better engineered network. If you only have a routed network, not sure this is the best way to engineer a network today. Switch technology can do alot of good things. The model does fall apart if you say you are only going to buy from these three to five, but it does not fall apart if you watch your traffic and you buy from whom your traffic is going to. The idea is not to engineer poor peering in the MAEs and NAPs with a bunch of networks. Buy and manage from the networks you are going to and you will engineer a better product. I still think that if you really looked at a normal NSP traffic that the bulk of it is with 5 to 8 networks. I some what agree with some of what you say, that a peer could be a good thing if it is private and not in a shared peering in the MAEs and NAPs. You talk about engineering good networks, why then use something that is unmanaged and uses poor technology and poor engineer designs in a MAE or NAP? Where is most of the packet lost on the internet? MAEs and NAPs. Where does most of the latency come from? Routers. MAEs and NAPs are yesterdays ideas that everyone who has bought into the concept now has to live with it, they did not build the business around the fact they had to pay and manage to provide good service. The time is almost upon us that you pay for what you get on the internet and you must pay for the bandwidth that you use, not what you can over subscribe until your customer begin to leave. This is a tough model, few will make it. Some day the internet will use measurements, such as QOS and customer will be willing to pay for those who have engineered their networks to provide QOS. I only have one more question for you; how many router hops, on your network, from New York to Los Angeles? If it is more than 1, then tell me what your latency is from end to end. Let's compare, shall we. Gary Zimmerman V.P. of Network Engineering Savvis Communications Corp. email: garyz@savvis.com http://www.savvis.com Office: 314.719.2423 Address: 7777 Bonhomme Suite 1000 St. Louis, MO 63105 ---------- the their
peering policies also. On the other hand, I've never heard of CompuServe turning down an opportunity to inter-connect their network with other networks.
I'm a pragmatic person. DRA has connections to several exchange points, and has many more private 'backdoor' connections to other networks. DRA even buys (gasp) service from a few other providers, and also sells service to a few other providers. All work well, customers are happy, and DRA is profitable. Let's compare balance sheets shall we? -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation
Where does most of the latency come from? Routers.
Uhh... what about longlines??
MAEs and NAPs are yesterdays ideas that everyone who has bought into the concept now has to live with it, they did not build the business around the fact they had to pay and manage to provide good service.
Hey, if that's your opinion then that's fine. Most of us aren't so ready to throw out *ANY* of the tools available to us to build networks. Routers have a place, IXPs have a place and switching has a place. You do things your way and we'll do it our own ways and next year we will see who had the best ideas.
The time is almost upon us that you pay for what you get on the internet and you must pay for the bandwidth that you use, not what you can over subscribe until your customer begin to leave. This is a tough model, few will make it.
What makes it tough is that there is very low customer demand for this sort of network. Every experiment with pay-per-use communications services over the past twenty years has shown that customers don't like it. The Internet boom of 1995 demonstrated quite graphically that flat rate telecom has an outrageously strong customer demand. Our job is to figure out how to continue to scale a network that can profitably provide flatrate services. That's the bread and butter of this industry.
Some day the internet will use measurements, such as QOS and customer will be willing to pay for those who have engineered their networks to provide QOS.
Yes, definitely. But that's still an extra cost luxury over and above the flatrate bread and butter services. ANd there is no certainty that ATM will be needed to provide QOS. There are some interesting options in certain routers as well.
I only have one more question for you; how many router hops, on your network, from New York to Los Angeles? If it is more than 1, then tell me what your latency is from end to end. Let's compare, shall we.
You aren't going to make latency disappear by replacing router hops with ATM switch hops. Those electrons still have to cross the continent. ******************************************************** Michael Dillon voice: +1-415-482-2840 Senior Systems Architect fax: +1-415-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." ********************************************************
Michael Dillon wrote:
You aren't going to make latency disappear by replacing router hops with ATM switch hops. Those electrons still have to cross the continent.
em fields actually. Actual electron drift would be a real latency:) Seriously, your point about layer2/layer3 switching/routing tradeoffs is an excellent one. The transit time on long runs has been so thoroughly discussed as a limiting factor that I'm surprised we have to make this point again. (Marketing stuff I guess - My network is better than your network cause I've got a bigger ...) Or alternately shall we compare ... , nah. Ken Leland kwl@monmouth.com
Where does most of the latency come from? Routers.
Uhh... what about longlines??
On second thought, here are some URLs.... I went to http://www.nanog.org and searched the mailing list archives for the last time this was argued. Here are some references: http://www.cctec.com/maillists/nanog/historical/9610/msg00263.html http://www.cctec.com/maillists/nanog/historical/9604/msg00043.html http://www.cctec.com/maillists/nanog/historical/9610/msg00586.html ******************************************************** Michael Dillon voice: +1-415-482-2840 Senior Systems Architect fax: +1-415-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." ********************************************************
On Thu, 10 Jul 1997, Gary Zimmerman wrote:
You talk about engineering good networks, why then use something that is unmanaged and uses poor technology and poor engineer designs in a MAE or NAP? Where is most of the packet lost on the internet? MAEs and NAPs.
This is gratutitous and misleading to the journalists and other non-network engineers here. The packet loss at NAPs and MAEs is caused by providers not upgrading the size or number of connections they have to/from the various exchange points they are experiencing problems at. An example of this disingenous usage of exchanges is Sprint's completely saturated T1 connection to CIX. There are many other examples. The solution is to not use those routes. If a provider is leaking bad routes and flaps continously, would you peer with them? Likewise, if a provider has a notoriously saturated connection to an exchange does it them become a (misleading) casebook example of why exchange points are bad? No and No. Mike. +------------------- H U R R I C A N E - E L E C T R I C -------------------+ | Mike Leber Direct Internet Connections Voice 408 282 1540 | | Hurricane Electric Web Hosting & Co-location Fax 408 971 3340 | | mleber@he.net http://www.he.net | +---------------------------------------------------------------------------+
On Thu, 10 Jul 1997, Gary Zimmerman wrote:
NAP? Where is most of the packet lost on the internet? MAEs and NAPs. Where does most of the latency come from? Routers. MAEs and NAPs are
I only have one more question for you; how many router hops, on your network, from New York to Los Angeles? If it is more than 1, then tell me what your latency is from end to end. Let's compare, shall we.
I was going to ignore this particular pissing contest, but this caught my eye. I heard from some people that there are customers who complain about the number of IP hops through a network, and didn't get why they would ask such an irrelevant question, but now I see why. For the Nth time, routers by themselves do _NOT_ introduce any significant increased latency over switches, and whatever amount that may be is dwarfed by the propagation delay of long haul circuits. If argue otherwise is just silly marketing stunt. Now, there are failure modes in which routed networks can add large amounts of latency to flows. However, I can think of just as many failure modes in switched networks that will do the same. There are many valid reasons why you'd want to build a switched WAN backbone as opposed to a routed one in the current environment. However, to argue that a switched backbone has an inherent advantage over a routed one is at best disingenuous, and at worst dishonest. -dorian
On Sun, 13 Jul 1997, Dorian R. Kim wrote:
There are many valid reasons why you'd want to build a switched WAN backbone as opposed to a routed one in the current environment. However, to argue that a switched backbone has an inherent advantage over a routed one is at best disingenuous, and at worst dishonest.
It's been pointed out to me that I've neglected to mention the effects of buffering delay. Rather than go on, I've included an article by Scott Bradner on this topic: -dorian --- title: Religious conversions? It seems that we can not get rid of the routers vs bridges war of words, even though it is now routers vs switches. "Switch when you can and route when you must" is a mantra that one often hears from switching salesdroids. They claim that switching is faster, simpler, cheaper and smells better than routing. This is not a new ideological conflict, it goes back many years to a time when some companies were building nation-wide bridged DECnet and SNA networks and routing was some esoteric weird thing that those IP guys did in dark rooms. It seemed to die down for a while as IP became mainstream but has now flared again with the advent of Ethernet switches. (Even if many Ethernet switches are just multi-port bridges with a marketing makeover.) But I am more than a bit suspicious that much of the religious-like fervor comes less from architectural purity than from product family. Part of the FUD in this issue involves misconceptions about the performance of routing. This is perfectly exemplified by an article a few weeks ago in this paper that said that IP switching can avoid "time-consuming route table lookups." Far be it for me to bring actual facts to the discussion, they do so distort the perceptions, but in the Harvard Network Device Test Lab we measured the forwarding latency of a Cisco Systems Catalyst 5000, a level two Ethernet switch, at 66 microseconds. We measured the forwarding latency of a Bay Networks Switch Node, a level three IP router, at 72 microseconds. Thus the "time-consuming route table lookups" took about 6 microseconds additional time in the Switch Node. Somehow I do not think that a difference of 6 microseconds in router forwarding latency is going to be all that noticeable in a system latency which, at best, will be in the tens of milliseconds, and will be closer to 100 milliseconds in many cases. And just to be clear, this processing is pipelined so that these devices forward packets at wire rate, as many routers have been able to do for a number of years. Note also that buffering delays can dwarf processing delays. It takes 1.2 milliseconds to transmit one 1518 byte packet on a 10 Mbps Ethernet. It does not matter if the device is a switch or a router, if that packet is in front of your packet, your packet will be delayed by 1.2 milliseconds. Latency and its impact is just one example of the arguments that are used against routers and I, as a strong proponent of routing, expect that I could counter most of them. But that may not be required. A number of the switch companies are starting to come out with "level 3 switches", most of which are just fast, cheap routers with marketing makeovers. I am already starting to see the painful process of mental realignment going on in the sales forces of these companies. After years of damning routing they now must start to praise the concept. It is a fun conversion to watch.
There is one significant difference between routed and switched backbones. The hops on routed backbones can be seen by end users using tools such as traceroute. On switched backbones, the hops are still there, but can not be seen by end users. Hence the marketing perception is different though the results are really the same. The router side could turn the argument. "With a routed backbone, you can actually SEE what is happening to your packets. It is not a hidden unknown, thus prone to failure you can not diagnose. With routers you know it went bad at nqu1. With switches, it just went bad." randy
On Sun, 13 Jul 1997, Stephen Balbach wrote:
On Sun, 13 Jul 1997, Randy Bush wrote:
There is one significant difference between routed and switched backbones.
Doesnt an IP Switch have lower latency and higher pps?
What's an IP switch? If you can define what this is other than a marketing stunt, I'd appreciate it.
Is a Cisco running NetFlow any faster then a routed Cisco?
No. A cisco router running netflow switching doesn't make it a switch, just as a cisco router running optimum switching doesn't make it a switch either. There can be large amounts of confusion that gets created because of marketing silliness. All routers and switches forward traffic. When the forwarding decision is made in layer 3, this is usually referred to as being "routed" and when forwarding decision is made in layer 2, this is usually referred to as being "switched". People also refer to hardware/interface/link layer level forwarding decisions made in routers as "switching." Hence, "fast" switching, "optimum" switching, and "netflow" switching in cisco term, which doesn't make a router a switch. Most routers have the capability of being switches, while most switches don't have the capability of being routers. (routers by their function needs to talk to layer 2, while switches do not necessarily have to) Since people seem to think that switch has some magically theraputic quality to network performance I wonder why Bay marketing hasn't started making a big deal about the fact that their BCNs function as frame relay switches. See, it's a Switch-Router, and it's a photon accelerator too! :) -dorian
On Sun, 13 Jul 1997, Dorian R. Kim wrote:
On Sun, 13 Jul 1997, Stephen Balbach wrote:
On Sun, 13 Jul 1997, Randy Bush wrote:
There is one significant difference between routed and switched backbones.
Doesnt an IP Switch have lower latency and higher pps?
What's an IP switch? If you can define what this is other than a marketing stunt, I'd appreciate it.
Routes the first packets and switches the rest based on "flows". It is not dependent on layer 2 or PVC's to determain the correct route? This is what Ive read from the mfg's who claim higher pps via this method then straight routing. I realize this is similair to the C vs C++ argument - C++ is a method more then a language, IP Switcing is more a routing technique then a new hardware technology. But is it worth it?
Is a Cisco running NetFlow any faster then a routed Cisco?
No. A cisco router running netflow switching doesn't make it a switch, just as a cisco router running optimum switching doesn't make it a switch either.
There can be large amounts of confusion that gets created because of marketing silliness.
All routers and switches forward traffic. When the forwarding decision is made in layer 3, this is usually referred to as being "routed" and when forwarding decision is made in layer 2, this is usually referred to as being "switched".
People also refer to hardware/interface/link layer level forwarding decisions made in routers as "switching."
Hence, "fast" switching, "optimum" switching, and "netflow" switching in cisco term, which doesn't make a router a switch.
Most routers have the capability of being switches, while most switches don't have the capability of being routers. (routers by their function needs to talk to layer 2, while switches do not necessarily have to)
Since people seem to think that switch has some magically theraputic quality to network performance I wonder why Bay marketing hasn't started making a big deal about the fact that their BCNs function as frame relay switches.
I assume at some level it makes sense to do switching for topology reasons. But for performance, it is not a benefit? .stb
On Sun, 13 Jul 1997, Stephen Balbach wrote:
What's an IP switch? If you can define what this is other than a marketing stunt, I'd appreciate it.
Routes the first packets and switches the rest based on "flows". It is not dependent on layer 2 or PVC's to determain the correct route? This is what Ive read from the mfg's who claim higher pps via this method then straight routing.
Not really. Given the nature of traffic in Internet, the "cost" of flow set up and maintenance pretty much outweigh whatever gain you get from from cut-through switching. This is actually quite similiar to why forwarding caches in routers aren't very useful in the current Internet. Now, if the device had specific functions that requires it to perform actions on per-flow bases on a traffic that was deterministically long flow oriented, it would be a gain, but this sort of thing is not very useful for the Internet as we have it.
Since people seem to think that switch has some magically theraputic quality to network performance I wonder why Bay marketing hasn't started making a big deal about the fact that their BCNs function as frame relay switches.
I assume at some level it makes sense to do switching for topology reasons. But for performance, it is not a benefit?
Depends. -dorian
I assume at some level it makes sense to do switching for topology > reasons. But for performance, it is not a benefit?
I believe there are two benefits to L2 switching across the core that are rarely mentioned, but of significant help to large networks. Detached L1 reroute, and endpoint pair flow control. L2 switching across the core can benefit your L3 infrastructure by removing some of the route recalculation required by changes in L1 topology. If the L2 network can reroute without the L3 knowing about the topology change, less overhead for the L3 network to worry about. The general goal here is to 'segment the responsibility'. The same result could be found by instantiating an additional L3 routing domain across the backbone (a technique that many have done, and from which many seem to be moving away). Of course, you add an additional variable, as you must now consider the variables inherent to L2 routing, and your complexity goes up, though the pieces complexity go down. That is to say, you divide the complexity up into two pieces, with the whole of the sum being greater, but the individual pieces far less. When one is given equipment with set capacity, this is a reasonable choice. As well, the PVC endpoint flows can be engineered at a level unknown to pure L3 networks. If one has a large amount of traffic going from Boise to Dayton, and the traffic from Topeka to Evanston wants to take the same path, one can build pvcs between the two endpoint pairs, and manually route the PVCs over the preferred trunks. These two benefits - L3 detached reroutes - endpoint pair flow control are significant. I agree w/ Bradner's note that the difference in latency is trivial in a WAN environment. However the two benefits listed above are not. This is not to say that L2 switching is better for everyone. Obviously all folks must do L3 switching, at least at the edges. But for many folk [large providers] with many problems [large networks, weenie routers] L2 switching is helping to allow networks to grow as stronger routers are built. -alan
On Sun, 13 Jul 1997, Alan Hannan wrote:
These two benefits
- L3 detached reroutes
- endpoint pair flow control
are significant.
And as significant as these benefits are, one must not also forget about the significant disadvantages of L2 switching that comes with current implementations. TANSTAAFL. By having two seperate intelligent networks that do not communicate with each other, one is incurring the cost of greatly increased systemic complexity as well as chances of failure mode amplification and obfuscation between L2 and L3 infrastructure. By adding complex devices to L2, one addes more candidate for failures, and shorten mean time between failures of one's network elements. Furthermore, L3 detached reroutes has it own share of problems. Whether this trade off makes sense is up to the individual networks, but like all things each choice has it's own pains and pleasures. I personally prefer to keep L2 as simple as possible, and leave L3 to do its thing. I guess it's a different interpretation of "segment the responsibility."
But for many folk [large providers] with many problems [large networks, weenie routers] L2 switching is helping to allow networks to grow as stronger routers are built.
This is the key point. If there were real routers to buy, much of the fun games that are played by various folks would be unnecessary. Furthermore, there is no reason why L3 network can't implement things like end pair traffic management much the same way L2 networks can given correct implementations. -dorian
The router side could turn the argument. "With a routed backbone, you can actually SEE what is happening to your packets. It is not a hidden unknown, thus prone to failure you can not diagnose. With routers you know it went bad at nqu1. With switches, it just went bad."
Cynics mike argue that lack of end user visibility of network problems is a compelling argument *for* NSPs to deploy switched backbones ... ("No, our backbone is just fine, { the NAP lost your packet | it got stolen by the packet fairies } " etc ...) Alex Bligh Xara Networks
On Sun, Jul 13, 1997 at 03:08:00PM -0800, Randy Bush wrote:
There is one significant difference between routed and switched backbones. The hops on routed backbones can be seen by end users using tools such as traceroute. On switched backbones, the hops are still there, but can not be seen by end users. Hence the marketing perception is different though the results are really the same.
Actually, from the IP packet's standpoint, no, the results aren't necessarily the same. It's unlikely, but possible, that a switched mesh backbone could forward some packets that a routed one couldn't, due to TTL issues. Didn't some older kernels set rediculously low TTLs on IP packets?
The router side could turn the argument. "With a routed backbone, you can actually SEE what is happening to your packets. It is not a hidden unknown, thus prone to failure you can not diagnose. With routers you know it went bad at nqu1. With switches, it just went bad."
It's a problem I'd accept for this amount of debugging flexibility, though. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "People propose, science studies, technology Tampa Bay, Florida conforms." -- Dr. Don Norman +1 813 790 7592
Actually, from the IP packet's standpoint, no, the results aren't necessarily the same.
Indeed. As has been discussed to death many times, each has micro-problems. Bottom line seems to be that each has its role and personal tastes vary. Personally, I use both. And, at OC3 and below, I'll take routers. And soon, that border will be moving. randy
participants (10)
-
alan＠mindvision.com -
Alex.Bligh -
Dorian R. Kim -
garyz＠savvis.com -
Jay R. Ashworth -
Ken Leland -
Michael Dillon -
Mike Leber -
randy＠psg.com -
Stephen Balbach