Stephen,
I think this is the key point. Its common sense that peering with the downstreams will improve user quality of service by both reducing latency and taking unnecessary points of failure out of the network.
Is it really common sense? If so, is the common sense correct? In fact, there is a lot of recent work that suggests that there can be a very poor (and as it turns out poorly understood) interaction between richness of interconnection and BGP dynamics; this is due, at least in part, to amplification and coupling effects that appear in some large systems. So many argue that that given the current set of protocols (i.e. BGP and its implementations), increased topological richness beyond some threshold can actually hurt robustness and reliability. And just to be clear about this, this is not a statement about peering policies themselves (I'm explicitly not commenting on that), but rather about our current understanding of some of the dynamics that exist in today's Internet. I've been trying to capture some of this in the following document (with the able help of Randy, Tim, and many others): http://www.ietf.org/internet-drafts/draft-ymbk-arch-guidelines-03.txt On the topic of interconnection richness and its (possibly unanticipated) effects, Craig and Abha's early work on this is maybe the canonical reference. For something a little more recent, see "What is the Sound of One Route Flapping", Timothy G. Griffin, IPAM Workshop on Large-Scale Communication Networks: Topology, Routing, Traffic, and Control, March, 2002. In any event, I guess the bottom line here is that sometimes what looks like common sense (or even what we have a tendency to call "conventional wisdom") may just be wrong. Dave
Preaching to the ministers here: I would like to see more data. I don't think a network with large aggregates (some who can not peer with tier 1s due to current policies) has much impact on the global routing structure. The primary problem is the noise of smaller announcements popping on and off magnified by multihoming punching holes in large aggregates. Small announcement show more churn because they are more granular. They expand the route table thus slowing convergence. Scientific investigation and public sharing of data can help networks build policies based on service criteria. Many large networks are reluctant to share internal data that could help in the broader analysis of stability issues. Where is the threshold? Can I turn it into a policy? How much does topology effect stability? Hierarchal design tends to mitigate instability when it can be localized to a small segment of your network infrastructure. Flat designs tend to ring like a bell when instability is introduced. I think we held the world record for flapping at NAP.NET in 95-96. That was a flat design executed during a time when the Cisco architecture and software could not keep up with the growth and churn rate. The inclusion of algorithms that enhanced oscillation ringing (and since has been fixed in IOS) did not help. --On Saturday, 29 June 2002 09:48 -0700 David Meyer <dmm@maoz.com> wrote:
Stephen,
I think this is the key point. Its common sense that peering with the downstreams will improve user quality of service by both reducing latency and taking unnecessary points of failure out of the network.
Is it really common sense? If so, is the common sense correct? In fact, there is a lot of recent work that suggests that there can be a very poor (and as it turns out poorly understood) interaction between richness of interconnection and BGP dynamics; this is due, at least in part, to amplification and coupling effects that appear in some large systems. So many argue that that given the current set of protocols (i.e. BGP and its implementations), increased topological richness beyond some threshold can actually hurt robustness and reliability. And just to be clear about this, this is not a statement about peering policies themselves (I'm explicitly not commenting on that), but rather about our current understanding of some of the dynamics that exist in today's Internet.
I've been trying to capture some of this in the following document (with the able help of Randy, Tim, and many others):
http://www.ietf.org/internet-drafts/draft-ymbk-arch-guidelines-03.txt
On the topic of interconnection richness and its (possibly unanticipated) effects, Craig and Abha's early work on this is maybe the canonical reference. For something a little more recent, see "What is the Sound of One Route Flapping", Timothy G. Griffin, IPAM Workshop on Large-Scale Communication Networks: Topology, Routing, Traffic, and Control, March, 2002.
In any event, I guess the bottom line here is that sometimes what looks like common sense (or even what we have a tendency to call "conventional wisdom") may just be wrong.
Dave
-- Joseph T. Klein jtk@titania.net "Why do you continue to use that old Usenet style signature?" -- anon
On Sat, Jun 29, 2002 at 07:42:03PM -0000, Joseph T. Klein wrote:
Flat designs tend to ring like a bell when instability is introduced. I think we held the world record for flapping at NAP.NET in 95-96. That was a flat design executed during a time when the Cisco architecture and software could not keep up with the growth and churn rate. The inclusion of algorithms that enhanced oscillation ringing (and since has been fixed in IOS) did not help.
Have you ever seen an InterNAP route flap? Its good for around two minutes or 120 traceroutes of pure humor, with a different loop across a different backbone in a different city with every invokation. Extensive peering relationships don't generally cause a breakdown of BGP, which is probably the reason that we have settled into using that system. Extensive transit relationships on the other hand, like those used by the "optimized routing" crowd to try and take advantage of all the "richness of paths" out there which aren't being used efficiently, break BGP very very quickly (in my experience at any rate). -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
That makes sense ... many full routing tables is fare worse than many partial routing tables. If my last resort was buying from a Tier 1 after peering out most of my traffic I would prefer "paid peering" or "partial transit". ... and one can always not listen to routes that have multiple non optimized paths via transit connections. If you have 10 ways to an ASN and 3 or four stable and clean diverse routes exist ... why chew up memory and CPU listening to the poor routes? --On Saturday, 29 June 2002 16:01 -0400 Richard A Steenbergen <ras@e-gerbil.net> wrote:
On Sat, Jun 29, 2002 at 07:42:03PM -0000, Joseph T. Klein wrote:
Flat designs tend to ring like a bell when instability is introduced. I think we held the world record for flapping at NAP.NET in 95-96. That was a flat design executed during a time when the Cisco architecture and software could not keep up with the growth and churn rate. The inclusion of algorithms that enhanced oscillation ringing (and since has been fixed in IOS) did not help.
Have you ever seen an InterNAP route flap? Its good for around two minutes or 120 traceroutes of pure humor, with a different loop across a different backbone in a different city with every invokation.
Extensive peering relationships don't generally cause a breakdown of BGP, which is probably the reason that we have settled into using that system. Extensive transit relationships on the other hand, like those used by the "optimized routing" crowd to try and take advantage of all the "richness of paths" out there which aren't being used efficiently, break BGP very very quickly (in my experience at any rate).
-- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
-- Joseph T. Klein jtk@titania.net "Why do you continue to use that old Usenet style signature?" -- anon
On Sat, 29 Jun 2002, Joseph T. Klein wrote:
That makes sense ... many full routing tables is fare worse than many partial routing tables. If my last resort was buying from a Tier 1 after peering out most of my traffic I would prefer "paid peering" or "partial transit". ... and one can always not listen to routes that have multiple non optimized paths via transit connections.
How will that work? Your last resort would then have to buy paid peering from all non-transit networks (tier 1's), which means they have to be a kind of tier 1 themselves then? But - BGP only propogates the single best route, BGP automatically removes the "multiple non optimized paths" and if they're non-optimized they will never be best and hence never cause flaps within or downstream of your network.
If you have 10 ways to an ASN and 3 or four stable and clean diverse routes exist ... why chew up memory and CPU listening to the poor routes?
I cant imagine a production router would have 10 ways tho, as above, any non optimum routes will not be propogated past the eBGP. ... and assuming you arent taking transit then you rely upon your peerings as the ONLY means of connectivity to their networks and their customer networks which means multiple interconnect points and your still going to be receiving multiple BGP routes to the same destination... Note, I dont think I'm conflicting with the comments below which are referring to route flaps at the source which therefore cross all networks and all bgp routers, thats a separate issue. Steve
--On Saturday, 29 June 2002 16:01 -0400 Richard A Steenbergen <ras@e-gerbil.net> wrote:
On Sat, Jun 29, 2002 at 07:42:03PM -0000, Joseph T. Klein wrote:
Flat designs tend to ring like a bell when instability is introduced. I think we held the world record for flapping at NAP.NET in 95-96. That was a flat design executed during a time when the Cisco architecture and software could not keep up with the growth and churn rate. The inclusion of algorithms that enhanced oscillation ringing (and since has been fixed in IOS) did not help.
Have you ever seen an InterNAP route flap? Its good for around two minutes or 120 traceroutes of pure humor, with a different loop across a different backbone in a different city with every invokation.
Extensive peering relationships don't generally cause a breakdown of BGP, which is probably the reason that we have settled into using that system. Extensive transit relationships on the other hand, like those used by the "optimized routing" crowd to try and take advantage of all the "richness of paths" out there which aren't being used efficiently, break BGP very very quickly (in my experience at any rate).
-- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)
-- Joseph T. Klein jtk@titania.net
"Why do you continue to use that old Usenet style signature?" -- anon
On Sat, Jun 29, 2002 at 07:42:03PM -0000, Joseph T. Klein wrote: [snip]
The primary problem is the noise of smaller announcements popping on and off magnified by multihoming punching holes in large aggregates.
Small announcement show more churn because they are more granular. They expand the route table thus slowing convergence.
Point: there's a body of data that indicates "multihoming" is not the culprit. There's a lot of needless de-aggregating that has little or nothign to do with multihoming, and mostly to do with lack of clue. Both WRT limiting the scope of provider-based so-called "traffic engineering" (CF ptomaine drafts) and that folks not using large tracts of space can return blocks and get blocks that actually *fit* their need. Unfortunely there's a few companies/consultants whose business plan requires them to graze on the commons and get all in a huff when any of us tell them they're filtered because they are causing incremental damage to our networks. Get over it kids; stable and deterministic behavior is required for IP to work optimally. Stability uber alles, Joe -- Joe Provo Voice 508.486.7471 Director, Internet Planning & Design Fax 508.229.2375 Network Deployment & Management, RCN <joe.provo@rcn.com>
participants (5)
-
David Meyer -
Joe Provo -
Joseph T. Klein -
Richard A Steenbergen -
Stephen J. Wilcox