Re: Provider credibility - does it matter? was Re: Inter-provider relations
Roger Bohn <Rbohn@UCSD.edu> wrote: At 8:36 PM -0700 10/24/96, Vadim Antonov wrote:
Note that i didn't even talk about less measurabe, but way too more important things like hosting of information suppliers. Say, Big Provider connects 1000 web sites; Small Provider hosts 1 site -- benefit from peering in terms of Web site diversity to the Big Provider's customers is 0.1%. To Small Provider's customers the benefit of peering is 99.9%.
Oops, this has an arithmetic fallacy. Assuming that Big Provider also has 1000x more customers than Small Provider, and that all 1001 web sites are equally attractive, then there are 1000 BP customers each with a .001 chance of wanting to surf the SP web site. Compare this with 1 SP customer with a .999 chance of wanting to surf a BP web site, and a .001 chance of surfing the SP web site.
No, this is not about mutual traffic -- this is about benefit to _a_ customer. Your computation is correct but is completely irrelevant from the point of view of a customer. They're interested in reacheability, not traffic at some exchange point. I.e. each small ISP's customer derives 99.9% of benefit from the peering, while large ISP's customer derives only 0.01%. Hence, the large ISP can force small ISP to pay up -- without connectivity to a large ISP the small ISP will be dead pretty soon. This is exactly like grocery chains which can (and do) force consumers to pay up for food -- although you can grow your own tomatoes in your backyard and probably sell them to those grocery chains. Good chances are, they're not interested. --vadim
I.e. each small ISP's customer derives 99.9% of benefit from the peering, while large ISP's customer derives only 0.01%. Hence, the large ISP can force small ISP to pay up -- without connectivity to a large ISP the small ISP will be dead pretty soon. This is exactly like grocery chains which can (and do) force consumers to pay up for food -- although you can grow your own tomatoes in your backyard and probably sell them to those grocery chains. Good chances are, they're not interested.
Hrm. Does it matter if the small ISP already has transit with someone else? In such a case, peering does nothing but shorten the path from SmallISP to BigISP in order to no longer make it go through SmallISPs transit provider. The 'value' delta is no longer so large; in fact is it possible that it could benefit BigISP by keeping their customers-who-are-local-to-SmallISP from transiting to the nearest BigISP-TransitISP peering point by keeping the traffic local and using the new SmallISP-BigISP peering. So the main objection to free peering is the offloading of SmallISPs traffic onto BigISPs expensive backbone. So if it were possible to have a form of 'local peering' where BigISP could 'locally peer' with SmallISP, meaning that BigISP would allow SmallISP to deliver traffic that was destined for BigISP-customers-local-to-SmallISP directly to BigISP instead of going first through TransitISP and then through the nearest TransitISP-BigISP peering point, then that would seem to make both nice-guy as well as economic sense, assuming that SmallISP is big enough to meet other requirements (like minimal problem-response-time and cluefulness tests, perhaps) To illustrate: SmallISP========TransitISP===================TransitISP/BigISP Peering \\ \\ SmallISP/BigISP_Local_Peering \\ \\ \\ BigISP_POP_local_to_SmallISP===============BigISP Backbone // \\ BigISPcustomerlocaltoSmallISP RemoteBigISPCustomers and in the above illustration realize that the 'local peering' doesn't let SmallISP talk to RemoteBigISPCustomers without going through TransitISP. Now, wait, this is the Internet of the 90s - all the Real Big ISPs use a technique of routing called 'closest exit'. Which, I think, means that they try and get the packets off of their backbone and one step closer to its destination as soon as possible. So, if we further add the condition that both peering sessions above are located at one place (the LocaltoSmallISP Nap, for instance) then the picture starts looking like: SmallISP========TransitISP===================TransitISP/BigISP Peering===TransitISP Backbone \\ / \\ // SmallISP/BigISP_Local_Peering------------+ \\ // \\ \\ // BigISP_POP_local_to_SmallISP===============BigISP Backbone=====BigISP/TransitISPPeering // \\ BigISPcustomerlocaltoSmallISP RemoteBigISPCustomers with the -- link being the (presumably fast) NAP/MAE media. Now, if TransitISP is in fact using closest-exit (aka 'hot potato') routing, then the route to even RemoteBigISPCustomers is the same distance through BigISPs Backbone whether it goes through TransitISP or the SmallISP/BigISP_Local_Peering connection. Outbound. Hrm. So the only extra cost (not insignificant, I realize) is that of the traffic outbound from RemoteBigISPCustomers to SmallISP across BigISPs backbone, which goes that way since SmallISP is peered with BigISP and is therefore 'closer' that way than via TransitISP. Hrm. Is there no metric that can be set to offset this? I admit ignorance of teh finer points of BGP. It's been interesting. --Zachary
all the Real Big ISPs use a technique of routing called 'closest exit'.
They do, and the effect is that they then wash their hands of the congestion problems that occur. Furthest exit would give a provider the most control over quality of service but would be expensive*. Best exit (perhaps closest unless there is congestion, then furthest) doesn't seem possible with current routing technology. * - but many might be willing to pay for it
On Saturday, Oct 26, 1996, Jon Zeeff writes:
all the Real Big ISPs use a technique of routing called 'closest exit'.
They do, and the effect is that they then wash their hands of the congestion problems that occur.
Congestion where? Closest Exit routing has the effect of getting the packets onto the destination's providers backbone as soon as possible; where the hops from backbone to backbone take place is all that it changes, not the number of hops there are. Which means that with the current dialup customer's typical small-outbound-big-inbound traffic patters, it is the dialup customer's provider (or their provider's provider perhaps) whos backbone is the most congested.
Furthest exit would give a provider the most control over quality of service but would be expensive*. * - but many might be willing to pay for it
I disagree; if everyone were to do 'furthest exit' (in which you transit outgoing packets across your backbone to the point closest to the destination and then pass them off to the destination's provider's backbone) , routing would still exhibit the asymmetry (and therefore would have the exact same amount of 'control over QoS) it does today. In fact, I believe that the loading would be the same as we have now, but with the labels 'Inbound Load' and 'Outbound Load' reversed. And it's only expensive because everyone is doing closest exit, and so the 'odd man out' trying to do 'closest-exit-to-destination' loses because he then bears the cost of transporting not only his incoming load across his backbone, but his outgoing load as well, when everyone else is mostly bearing the cost of just their incoming load. So in the current internet, yes, if an ISP was to do 'shortest external path' routing, their backbone would carry both their inbound and outbound traffic loads, and hence it its possible that they would have more control over the quality of service, as you pointed out. But let me point out that that situation only lasts as long as everyone else is doing closest exit routing; when they switch to 'shortest external path' then you no longer carry the brunt of the load of your inbound traffic across your backbone.
Best exit (perhaps closest unless there is congestion, then furthest) doesn't seem possible with current routing technology.
What *is* best exit? It's unclear to me which one is better; they're flip sides of the same coin. --Zachary
With all the recent talk about BGP, etc., I thought I'd see if anybody knows the reasoning behind a particular short-coming of BGP that I've noticed and found particularly bothersome... We peer, using BGP, with several "backbone" provider networks for transit purposes. Some of these links are "faster" than others (e.g. T-3 vs. multiple T-1 and single T-1) for various reasons. If our router sees a route to a particular destination via a "high-speed" link and a "low- speed" link that has the _same_ number of AS "hops", it picks the link with the "lowest" IP address! (At least that's what I'm told and what I observe...) Why doesn't BGP pick the link with the highest bandwidth, or, better yet, pick the link with the highest bandwidth AND least congestion to label as the "best" available route? The needed information is avail- able in the router (and if it was somebody doing BGP from a host that was separate from the box with the interfaces, well, then too bad I guess) and can't be _that_ hard to incorporate can it? I'll get off my soapbox now... Ed
Can't you adjust your metrics/weights to prefer the low speed links less? -Deepak. On Fri, 8 Nov 1996, Ed Morin wrote:
With all the recent talk about BGP, etc., I thought I'd see if anybody knows the reasoning behind a particular short-coming of BGP that I've noticed and found particularly bothersome...
We peer, using BGP, with several "backbone" provider networks for transit purposes. Some of these links are "faster" than others (e.g. T-3 vs. multiple T-1 and single T-1) for various reasons. If our router sees a route to a particular destination via a "high-speed" link and a "low- speed" link that has the _same_ number of AS "hops", it picks the link with the "lowest" IP address! (At least that's what I'm told and what I observe...)
Why doesn't BGP pick the link with the highest bandwidth, or, better yet, pick the link with the highest bandwidth AND least congestion to label as the "best" available route? The needed information is avail- able in the router (and if it was somebody doing BGP from a host that was separate from the box with the interfaces, well, then too bad I guess) and can't be _that_ hard to incorporate can it?
I'll get off my soapbox now...
Ed
Well, sure, but why should I _have_ to? I thought we, in part, pay the big bucks for routers that are supposed to figure some of this stuff out on their own without having to "band-aid" things with AS path manipulations, etc. On Fri, 8 Nov 1996, Deepak Jain wrote:
Can't you adjust your metrics/weights to prefer the low speed links less?
-Deepak.
On Fri, 8 Nov 1996, Ed Morin wrote:
With all the recent talk about BGP, etc., I thought I'd see if anybody knows the reasoning behind a particular short-coming of BGP that I've noticed and found particularly bothersome...
We peer, using BGP, with several "backbone" provider networks for transit purposes. Some of these links are "faster" than others (e.g. T-3 vs. multiple T-1 and single T-1) for various reasons. If our router sees a route to a particular destination via a "high-speed" link and a "low- speed" link that has the _same_ number of AS "hops", it picks the link with the "lowest" IP address! (At least that's what I'm told and what I observe...)
Why doesn't BGP pick the link with the highest bandwidth, or, better yet, pick the link with the highest bandwidth AND least congestion to label as the "best" available route? The needed information is avail- able in the router (and if it was somebody doing BGP from a host that was separate from the box with the interfaces, well, then too bad I guess) and can't be _that_ hard to incorporate can it?
I'll get off my soapbox now...
Ed
Ed Morin Northwest Nexus Inc. (206) 455-3505 (voice) Professional Internet Services edm@nwnexus.WA.COM
Well, sure, but why should I _have_ to? I thought we, in part, pay the big bucks for routers that are supposed to figure some of this stuff out on their own without having to "band-aid" things with AS path manipulations, etc.
Ed Morin
I understand that IOS 12.0 will solve the travelling salesman problem for arbitrary number of cities. Seems they had to find a way to solve NP-complete problems in sub-exponential time in order to give us better algorithms for guaranteed-optimal routing. Anyway, sorry to be snide - any suggestions you have about algorithms for exchanging routing data between (or inside of) providers that also takes into account link size and perhaps hop-count, as opposed to just AS-PATH, are welcome at the next IETF meeting. Avi
On Fri, 8 Nov 1996 19:20:40 -0800 (PST) Ed Morin <edm@halcyon.com> alleged:
Well, sure, but why should I _have_ to? I thought we, in part, pay the big bucks for routers that are supposed to figure some of this stuff out on their own without having to "band-aid" things with AS path manipulations, etc.
Try reading the manual. How is the router supposed to know what you want to do. BGP4 knows nothing of link speeds, and I hope it never does, the instabilities it could cause are frightening. I don't think the people who came up with BGP4 ever expected to see what some people do with their router configs. Regards, Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
Look, I can do a "show interface" on any interface and see what speed it's running at and if it's dropping packets. If BGP hears a route on an interface that isn't dropping packets shouldn't _that_ route be considered "best" all other things being equal (hop counts and all)? You can't tell me the router doesn't know this information because _I_ get the information from the router itself!! I understand about route instabilities, etc. All I'm talking about here is a better "tie breaker" than ordinate numbers of IP addresses. On Sat, 9 Nov 1996, Neil J. McRae wrote:
On Fri, 8 Nov 1996 19:20:40 -0800 (PST) Ed Morin <edm@halcyon.com> alleged:
Well, sure, but why should I _have_ to? I thought we, in part, pay the big bucks for routers that are supposed to figure some of this stuff out on their own without having to "band-aid" things with AS path manipulations, etc.
Try reading the manual. How is the router supposed to know what you want to do. BGP4 knows nothing of link speeds, and I hope it never does, the instabilities it could cause are frightening.
I don't think the people who came up with BGP4 ever expected to see what some people do with their router configs.
Regards, Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
Ed Morin Northwest Nexus Inc. (206) 455-3505 (voice) Professional Internet Services edm@nwnexus.WA.COM
Look, I can do a "show interface" on any interface and see what speed it's running at and if it's dropping packets. If BGP hears a route on an interface that isn't dropping packets shouldn't _that_ route be considered "best" all other things being equal (hop counts and all)? You can't tell me the router doesn't know this information because _I_ get the information from the router itself!!
I understand about route instabilities, etc. All I'm talking about here is a better "tie breaker" than ordinate numbers of IP addresses.
The router really can't see how fast an interface *can* go unless it's maxed out. That's not really useful information in a route-selection decision. It's possible that the packet loss might weight a better path but you would have to be careful about reweighting 20k paths every few minutes... Especially if you pass those decisions on to other routers :) Avi
Well, at a minimum you can do a "bandwidth 44210" (or whatever) as part of your router configuration to "tell" your router how fast it _can_ go. How much of this _really_ gets passed on to other nodes via BGP sessions. If you hear a route from 4 different links, aren't you simply passing on to your neighbor (filters aside for the moment) that you "know how to get to that net" regardless of which way you shove the packet at any particular time? That is, do you really pass on the destination link info as well as the net info? On Sat, 9 Nov 1996, Avi Freedman wrote:
Look, I can do a "show interface" on any interface and see what speed it's running at and if it's dropping packets. If BGP hears a route on an interface that isn't dropping packets shouldn't _that_ route be considered "best" all other things being equal (hop counts and all)? You can't tell me the router doesn't know this information because _I_ get the information from the router itself!!
I understand about route instabilities, etc. All I'm talking about here is a better "tie breaker" than ordinate numbers of IP addresses.
The router really can't see how fast an interface *can* go unless it's maxed out. That's not really useful information in a route-selection decision.
It's possible that the packet loss might weight a better path but you would have to be careful about reweighting 20k paths every few minutes... Especially if you pass those decisions on to other routers :)
Avi
Ed Morin Northwest Nexus Inc. (206) 455-3505 (voice) Professional Internet Services edm@nwnexus.WA.COM
Well, at a minimum you can do a "bandwidth 44210" (or whatever) as part of your router configuration to "tell" your router how fast it _can_ go.
How much of this _really_ gets passed on to other nodes via BGP sessions. If you hear a route from 4 different links, aren't you simply passing on to your neighbor (filters aside for the moment) that you "know how to get to that net" regardless of which way you shove the packet at any particular time? That is, do you really pass on the destination link info as well as the net info?
What you want is an inter-provider OSPF... Faster healing. Link size consideration. Internal hop-count consideration. Internal weighting (which BGP has) that would pass between providers. But yes, you're just saying "I know how to get to X" - or better yet - "I promise to get to X if you deliver a packet to me destined for X". The origin info can be interesting but not really used unless there's a real problem making a decision, in which case it's just there as a last step against going to pseudo-random numbers.
Ed Morin
Avi
On Sat, 9 Nov 1996 09:33:54 -0800 (PST) Ed Morin <edm@halcyon.com> alleged:
I understand about route instabilities, etc. All I'm talking about here is a better "tie breaker" than ordinate numbers of IP addresses.
OK What? Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
On Sat, 9 Nov 1996, Neil J. McRae wrote:
Try reading the manual. How is the router supposed to know what
Well, until _somebody_ writes the definitive "Nutshell" book we all know just how useful the "FM" is to "RT".
Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
Ed Morin Northwest Nexus Inc. (206) 455-3505 (voice) Professional Internet Services edm@nwnexus.WA.COM
On Sat, 9 Nov 1996 09:35:09 -0800 (PST) Ed Morin <edm@halcyon.com> alleged:
On Sat, 9 Nov 1996, Neil J. McRae wrote:
Try reading the manual. How is the router supposed to know what
Well, until _somebody_ writes the definitive "Nutshell" book we all know just how useful the "FM" is to "RT".
Up until about 1 month ago I'd never used a CISCO router in my life. Just by reading Cisco's web site I have full mesh BGP4 with reflection and aggregation... If a few people actually read what BGP4 does and what your routers implementation of BGP4 does, there would far lower problems on the net. Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
Ed Morin <edm@halcyon.com> writes * We peer, using BGP, with several "backbone" provider networks for transit * purposes. Some of these links are "faster" than others (e.g. T-3 vs. * multiple T-1 and single T-1) for various reasons. If our router sees * a route to a particular destination via a "high-speed" link and a "low- * speed" link that has the _same_ number of AS "hops", it picks the link * with the "lowest" IP address! (At least that's what I'm told and what * I observe...) Yup, this is the ultimate tie breaker if there are two routes that are otherwise identical. * Why doesn't BGP pick the link with the highest bandwidth, or, better * yet, pick the link with the highest bandwidth AND least congestion to * label as the "best" available route? The needed information is avail- The first one is easy, in fact you can do that yourself by fiddling with metrics or such on the different BGP sessions. The second one would have dramatic consequences in terms of route instability. You pick one route now because of load on the link, the load changes and you pick the other, now BGP will have to change the announcement of this network to other peers. So, now we not only have flaps because of links/routers going up and down, we also have flap because of load changes on the network. The result: you are dampened out forever, or the network falls over. -Marten
On Fri, 8 Nov 1996, Marten Terpstra wrote:
Ed Morin <edm@halcyon.com> writes
* Why doesn't BGP pick the link with the highest bandwidth, or, better * yet, pick the link with the highest bandwidth AND least congestion to * label as the "best" available route? The needed information is avail-
The first one is easy, in fact you can do that yourself by fiddling with metrics or such on the different BGP sessions. The second one would have dramatic consequences in terms of route instability. You pick one route now because of load on the link, the load changes and you pick the other, now BGP will have to change the announcement of this network to other peers. So, now we not only have flaps because of links/routers going up and down, we also have flap because of load changes on the network. The result: you are dampened out forever, or the network falls over.
Is this really true? All I'm asking for is that the route a router considers to be "best" be picked by something a little more rational than the ordinate order of its IP address relative to another link. I don't see a flap situation at all here -- only that a decision to route a packet may change more frequently based on load. Ed Morin Northwest Nexus Inc. (206) 455-3505 (voice) Professional Internet Services edm@nwnexus.WA.COM
I know this is not the cisco list, but I'm sounding a warning. We had a bad, bad, BAD experience with BGP peer groups recently, which cisco are coming in to discuss. Symptoms are your outgoing AS filter lists will not work with *some* peers! This can look really confusing if, like us, you rely on a local peer-session to do sanity checking of your filter-lists and it shows everything is OK. Version is: Cisco Internetwork Operating System Software IOS (tm) GS Software (RSP-JV-M), Version 11.1(6), RELEASE SOFTWARE (fc1) Copyright (c) 1986-1996 by cisco Systems, Inc. cisco RSP2 (R4600) processor with 131072K bytes of memory. R4700 processor, Implementation 33, Revision 1.0 I've talked to at least one other provider who has experienced similar problems. I've yet to see a bug report from cisco on this, but would appreciate any comments from other folks who see similar behaviour (directly to me, I'll summarize). Oh, and this problem can take several days to manifest itself, and it seems to have impeccable timing :-) Mark
On Fri, 8 Nov 1996, Ed Morin wrote:
Is this really true? All I'm asking for is that the route a router considers to be "best" be picked by something a little more rational than the ordinate order of its IP address relative to another link. I don't see a flap situation at all here -- only that a decision to route a packet may change more frequently based on load.
A decision to route a packet may change more frequently based on load. And those decisions are propogated and acted upon by other routers. And they send traffic down those links. And the load on those links changes. And you make a new decision to route a packet based on the new load. | _ | /| | /\ / | / \ / | -^\/\ / \ / | \/ \ / | \ / | \/ | | +---------------------------------------------- Unless you have a model which is provably resilient to these sorts of feedback loops (which, to my understanding, BGP is not), then playing these sort of tricks is a young man's game, and one guaranteed not to leave him young (or employed) for long. Statically adjust the preference on your links and count yourself lucky. 8^) __ Todd Graham Lewis Linux! Core Engineering Mindspring Enterprises tlewis@mindspring.com (800) 719 4664, x2804
Ed Morin <edm@halcyon.com> writes * > The first one is easy, in fact you can do that yourself by fiddling * > with metrics or such on the different BGP sessions. The second one * > would have dramatic consequences in terms of route instability. You * > pick one route now because of load on the link, the load changes and * > you pick the other, now BGP will have to change the announcement of * > this network to other peers. So, now we not only have flaps because of * > links/routers going up and down, we also have flap because of load * > changes on the network. The result: you are dampened out forever, or * > the network falls over. * * Is this really true? All I'm asking for is that the route a router * considers to be "best" be picked by something a little more rational * than the ordinate order of its IP address relative to another link. * I don't see a flap situation at all here -- only that a decision to * route a packet may change more frequently based on load. Ed, you have to think about this in generic terms. In your network you may not re-advertise BGP info you get from your peers. In most networks these routes do get re-advertised. On of the basic rules in BGP is that you only advertise routes you actually use. Following that rule, if you change your decision on the best route based on load, you have to change your advertisement of that route to all you announced it to. The result is a flap. -Marten
On Fri, 8 Nov 1996 18:39:09 -0800 (PST) Ed Morin <edm@halcyon.com> alleged:
We peer, using BGP, with several "backbone" provider networks for transit purposes. Some of these links are "faster" than others (e.g. T-3 vs. multiple T-1 and single T-1) for various reasons. If our router sees a route to a particular destination via a "high-speed" link and a "low- speed" link that has the _same_ number of AS "hops", it picks the link with the "lowest" IP address! (At least that's what I'm told and what I observe...)
This is correct.
Why doesn't BGP pick the link with the highest bandwidth, or, better yet, pick the link with the highest bandwidth AND least congestion to label as the "best" available route? The needed information is avail- able in the router (and if it was somebody doing BGP from a host that was separate from the box with the interfaces, well, then too bad I guess) and can't be _that_ hard to incorporate can it?
It shouldn't be a factor in BGP. BGP is used to manage route announcements and not the engineering short comings of your network, if you plan things and configure BGP correctly you'll find you can do what you are asking here. Regards, Neil. -- Neil J. McRae. Alive and Kicking. E A S Y N E T G R O U P P L C neil@EASYNET.NET NetBSD/sparc: 100% SpF (Solaris protection Factor) Free the daemon in your <A HREF="http://www.NetBSD.ORG/">computer!</A>
participants (10)
-
Avi Freedman -
Deepak Jain -
Ed Morin -
jon＠branch.net -
Mark Turner -
Marten Terpstra -
Neil J. McRae -
Todd Graham Lewis -
Vadim Antonov -
Zachary DeAquila