Re: exponential route prefix growth, was: Re: The Cidr Report
On Fri, Sep 22, 2000 at 03:44:50PM -0400, Vijay Gill wrote:
On Fri, 22 Sep 2000, Kai Schlichting wrote:
[ http://www.employees.org/~tbates/cidr.plot.html for a plot ]
of engineering experience announcing 32 /24's out of their scattered POPs in addition to their /19 aggregate because they can't get their load balancing right without it or have never heard of the 'NO_EXPORT' attribute (I could point fingers now)?
I suspect data from inside providers with aggressive filtering policies might prove to be interesting.
Some promising local providers were past the 150k mark a while ago for sum (internal + global bgp) routes and were growing superlinearly the entire time because of the ever increasing number of customers.
I have in one location over 100k prefixes received from one nice large international provider:
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd a.b.c.d 4 701 10776831 107250 13024503 0 0 1w4d 105250
My primary concern is that the current growth is going to outscale the medium sized providers as the one vendor that the above output came from doesn't appear to be providing enough memory in their equipment, or have enough memory options available.
I am not predicting "the internet will end in 9 days, repent", but it is some cause for concern in the next 6-12 months.
What would be nice is if someone were to put together an auto-nagger similar to the "your dns server is lame" of the old days messages. Back when the network tables were about 40-45k, I would read the output of "sh ip b" and send messages to people asking them to aggregate, but those days are over... (at least for me..)
- jared
-- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine. END OF LINE | Manger of IP networks built within my own home
Take a look at: http://www.fox-den.com/routes/ This is a plot of routes v. paths on a router that peers with or at least sees the tables from a few large upstreams. Please ignore the large gaps in the chart where I <ahem> did some configuration that was contrary to the correct functioning of my statistics collection. This is a crude measurement, but it's useful for trending. I'm sure someone at CAIDA has done better research than my stupid-simple MRTG graphs, but I haven't seen theirs mentioned yet so I'll throw my $.02 in first. What is interesting in the growth difference between the two growth patterns. An interesting trend shows; there is a much larger increase in the number of paths than in the number of routes. I would assume that this indicates a growth in the interweaving of networks, or at least the interweaving of transit providing relationships. I suspect the interweaving of interconnections is growing at a similar rate, but proof of this is invisible with only a few BGP perspectives. Perhaps the more vital piece of information in this discussion is not the sudden growth of routes, but the growth of paths. The de-aggregation of routes (though I have done no research to prove this) seems to me simply a response to redundancy/load distribution issues introduced by current route selection algorithms. I think we're identifying one of the symptoms, but not the root cause of the growth in routes. * It seems that we should be saying that there is a growth in paths, not _just_ routes, and path growth with current implementation methodologies and reasoning implies route growth. * We come back to "BGP only works the way most people expect it to in a multi-home situation when you de-aggregate a route from one of your upstreams and announce it all the time to all your peers." Yes, there are many other reasons that one would de-aggregate and re-announce either back into the primary with the aggregate or others; however, I think that the (obvious) basic reasoning for the bulk of path/route announcements is redundancy and load sharing. The demand for "always on" backup ("announce all the time even in directions in which you pad the path") and cost efficiency ("we're paying for that second line, so we'd better get some use out of it") lead to increase in paths that need to be visible, and thus routes that need to be visible. Why in the last year has this been so large? You've got me there, but I'll take a swing at it. Probably something to do with the collapsing prices of bandwidth, the dispersion of BGP know-how (or the "how to set up BGP for your enterprise for dummies" books/FAQs, at least,) and the sudden expectation that all Internet traffic is mission-critical and mustmustmust always be available. The bubble of hype is finally moving down the hose and causing operational issues such as this. It's the nature of the beast; as complexity grows... complexity grows. So how does this get solved? The complex solutions of multi-provider NAT are possible, but practically impossible in a business environment where FUD rules the day. This is really a matter best discussed after at least 6 hours of deep thought, which is about 5:55 short of what has occurred for this message. :) JT
On Fri, 22 Sep 2000, John Todd wrote:
Take a look at:
Had to look at it real hard until I realized your graphs are growing to the right vs the standard MRTG output.
at least,) and the sudden expectation that all Internet traffic is mission-critical and mustmustmust always be available. The bubble of hype is finally moving down the hose and causing operational issues such as this.
It is an operation issue for Ciscos perhaps, and sure, it increases the time it takes to get a full view when a peering session is reset but, the last time I checked (a week ago), gated and zebra on the PC router running on an inexpensive pentium class machine with 256MB of memory had NO problem whatsoever with 250K prefixes. The Juniper M5 and M10 in bare bones configuration have 256MB and there is a 768MB option. IMHO, unless Cisco starts jumping through the hoops for providers, the number of "Powered by Cisco" banners we see is going to fade quickly. --- John Fraizer EnterZone, Inc
Take a look at:
Had to look at it real hard until I realized your graphs are growing to the right vs the standard MRTG output.
mrtg makes more sense to me that way also. after all, being of...uh..."english" extraction, i read just about everything else from left to right, so expecting time to rise to the right doesn't seem that far off the mark. -- |-----< "CODE WARRIOR" >-----| codewarrior@daemon.org * "ah! i see you have the internet twofsonet@graffiti.com (Andrew Brown) that goes *ping*!" andrew@crossbar.com * "information is power -- share the wealth."
John Todd wrote:
Take a look at:
http://www.fox-den.com/routes/
This is a plot of routes v. paths on a router that peers with or at least sees the tables from a few large upstreams. Please ignore the large gaps in the chart where I <ahem> did some configuration that was contrary to the correct functioning of my statistics collection. This is a crude measurement, but it's useful for trending. I'm sure someone at CAIDA has done better research than my stupid-simple MRTG graphs, but I haven't seen theirs mentioned yet so I'll throw my $.02 in first.
What is interesting in the growth difference between the two growth patterns. An interesting trend shows; there is a much larger increase in the number of paths than in the number of routes. I would assume that this indicates a growth in the interweaving of networks, or at least the interweaving of transit providing relationships. I suspect the interweaving of interconnections is growing at a similar rate, but proof of this is invisible with only a few BGP perspectives.
I don't see that, at least in RELATIVE growth. In your "Yearly" graph, number of routes at start = 64 K, number at end = 84 K, ratio = 1.31 number of paths at start = 192 K number at end = 252 K, ratio = 1.31 so both have grown by ~ 31 % in the last year. I don't see how this shows that networks are changing - wouldn't the naive model be growth in routes is proportional to growth in paths, which is what these data show ? I am not saying that the networks aren't getting more interweaved - just that I don't see how these graphs support that belief. Regards Marshall Eubanks
Perhaps the more vital piece of information in this discussion is not the sudden growth of routes, but the growth of paths. The de-aggregation of routes (though I have done no research to prove this) seems to me simply a response to redundancy/load distribution issues introduced by current route selection algorithms. I think we're identifying one of the symptoms, but not the root cause of the growth in routes. * It seems that we should be saying that there is a growth in paths, not _just_ routes, and path growth with current implementation methodologies and reasoning implies route growth. *
We come back to "BGP only works the way most people expect it to in a multi-home situation when you de-aggregate a route from one of your upstreams and announce it all the time to all your peers." Yes, there are many other reasons that one would de-aggregate and re-announce either back into the primary with the aggregate or others; however, I think that the (obvious) basic reasoning for the bulk of path/route announcements is redundancy and load sharing.
The demand for "always on" backup ("announce all the time even in directions in which you pad the path") and cost efficiency ("we're paying for that second line, so we'd better get some use out of it") lead to increase in paths that need to be visible, and thus routes that need to be visible.
Why in the last year has this been so large? You've got me there, but I'll take a swing at it. Probably something to do with the collapsing prices of bandwidth, the dispersion of BGP know-how (or the "how to set up BGP for your enterprise for dummies" books/FAQs, at least,) and the sudden expectation that all Internet traffic is mission-critical and mustmustmust always be available. The bubble of hype is finally moving down the hose and causing operational issues such as this.
It's the nature of the beast; as complexity grows... complexity grows. So how does this get solved? The complex solutions of multi-provider NAT are possible, but practically impossible in a business environment where FUD rules the day. This is really a matter best discussed after at least 6 hours of deep thought, which is about 5:55 short of what has occurred for this message. :)
JT
T.M. Eubanks Multicast Technologies, Inc. 10301 Democracy Lane, Suite 410 Fairfax, Virginia 22030 Phone : 703-293-9624 Fax : 703-293-9609 e-mail : tme@on-the-i.com http://www.on-the-i.com http://www.buzzwaves.com
participants (4)
-
Andrew Brown -
John Fraizer -
John Todd -
Thomas Marshall Eubanks