August 2001 - Test - lists.nanog.org

RE: What is the limit? (was RE: multi-homing fixes)
by Roeland Meyer 29 Aug '01

29 Aug '01

|> From: Alex Bligh [mailto:alex@alex.org.uk] |> Sent: Wednesday, August 29, 2001 11:20 AM |> |> --On Wednesday, 29 August, 2001 10:25 AM -0700 Roeland Meyer |> <rmeyer(a)mhsc.com> wrote: |> |> > To boost this point, exponential growth, approaching |> infinity, becomes |> > asymptotic. |> |> The only asymptote of an exponential curve is it's asymptote towards |> the X axis, for as X approaches negative infinity. The point remains that it is nonsensical, given a finite universe. The curve doesn't fit observable reality.

4 3

RE: What is the limit? (was RE: multi-homing fixes)
by Martin, Christian 29 Aug '01

29 Aug '01

As much as arguing with Sean is generally a lost cause (face it folks, he knows his sh*t), I would like to comment on the disasters he presents and offer a reason why this was so, from someone who wasn't there. Sometimes an outside view is a clearer one. > The Internet *FAILED* in 1994. The number of prefixes > carried globally > exceeded the ability of the global routing system to carry it, and > as a result, some parts of the world deliberately summarized away > whole large-scale ISPs merely to survive. > The Internet *FAILED* again in 1996, when the dynamicism in > the global routing system exceeded the ability of many border > routers to handle, and as a result, some networks (not just core ones) > deliberately made the Internet less quick to adjust to changes in > topology. > > Thus, the routing system FAILED twice: once because of memory, once > because of processor power. And both times because the routers themselves were not designed exclusively for the Internet and its requirements. In 1994, the Internet was running on AGS+ and 4000 boxen. Compare these boxes, 7 years later (or 3.5 Moore cycles later) to a 124xx, or an M160, or a TSR or any other vendors new gear. Today's systems are 10-20x more memory-rich, (still CPU constrained), and the routing algorithms are adapting to deal with excessive dynamism. Route damping is of huge importance, but this is something that is well known. ORF capabilities in BGP, redistribution communities, etc, will continue keeping updates frequency in check. And again, routers today are generations behind today's Moore cycle. Furthermore, we are still using general purpose CPU's. Imagine a BGP ASIC that could process a zillion updates per second! 8-> The point is that the vendors did a good job of dealing with the speed issue. In 1994, even Sean would have been impressed to see an OC-192c router interface performing at wire speed with 1000 line ACLs in and out, etc, etc. The speed problem has been addressed (the number of OC-192c/STM-64 ports in use is pretty low compared to the number of OC-48s/STM-16 ports that were in use 12 months after the those cards were released), so now the vendors can start working on the global routing system. Then again, most of them are working on MPLS which does nothing to address the GRS. *sigh* (Note to vendor gurus - isn't it interesting how an increasing number of Internet "greybeards" - sorry sean, I don't recall you being all that grey or having a beard for that matter - continue to concern themselves with GRS stability and scaling while at the same time puking on all the MPLS hype?) > If the size or the dynamicism of the global routing system grows > for a sustained period faster than the price/performance curve of > EITHER memory OR processing power, the Internet will FAIL again. <-snip-> > So, Moore's Law, or more specifically the underlying curve > which tracks the growth of useful computational power, is > exactly what we should compare with the global routing system's > growth curve. This is only the case if the algorithmic complexity is a constant. Reducing the algorithmic complexity from O(N^2) to O(N), for example, reduces exponential growth of the data processing requirements to linear growth. Going from O(N) to O(logN) makes the requirements sublinear. An example of this would be the introduction of IS-IS mesh groups. > > Note that when Moore is doing better than the Internet, > it allows for either cheaper supply of dynamic connectivity, > or it allows for the deployment of more complex handling > of the global NLRI. > > The major problem, as you have pointed out, is that processing > requirement is often bursty, such as when everyone is trying > to do a dynamic recovery from a router crash or major line fault. > We could still use 68030s in our core routers, it's just that it'd > take alot longer than it used to perform a global partition repair, > which means your TCP sessions or your patience will probably time out > alot more frequently. Are you kidding! My patience times out during single BGP session failure restoration... ;) > | I think that what we need to do is have a fourth group, > call them Internet > | Engineers for lack of a better word, come in and determine > what the sign > | should read. > > Structures built according to best known engineering practices > still fall down from time to time. That's the problem in anticipating > unforeseen failures. Consider yourself lucky that you haven't had > to experience a multi-day degredation (or complete failure!) of > service due to resource constraints. And that you haven't > run into a sign that says: "please note: if you try to have an > automatic partition repair in the event this path towards {AS SET} > fails, your local routing system will destabilize". Since I wasn't there during your problems in 1994 and 1996, I ask you, "Did you not know that you were approaching a resource constraint?" In regards to memory, we all know that we can triple or even quadruple the size of the table, today, with headroom reamining. The update frequency ceiling is less tractable, due to implementation dependence, but a correlation between table size and update frequency would certainly be of use in approximating it. I think the table size issue is solvable. Processing the table is where we may be hurting, which we are attempting to solve by constraining the update frequency. > | Finally, we have a sixth group, call them the IETF, come in > | and invent a flying car that doesn't need the bridge at all. > > As Randy (with his IETF hat) says: "send code". > I wish I could. But alas, I am too busy counseling my routers, ensuring that they are happy, trying to keep them stable. They are temperamental little b*st*rds, I tell ya. -chris > Sean. >

2 1

Re: What is the limit? (was RE: multi-homing fixes)
by smd＠clock.org 29 Aug '01

29 Aug '01

Leo - Draw two curves, the first y=x/2, the second y=x^2 Move the value of x for y=1 for the first curve left by 2, 5 or 10 and it will still be surpassed by the second curve. You will even see this for a second curve of y=x*2 or y=x. The global routing table size HAS grown exponentially in the past. Rationalize it any way you want, blame whatever you like, but there is no known way to construct a router that can handle that kind of growth in anything but a short term, and the trend for the components in the router growth curve is simply not going to increase to a long term superlinear rate. A 10x system performance boost today just moves the x point for y=1 of fundamental curve claimed by Moore's Law to the left a few notches. Or are you claiming that routing equipment will have a fundamentally different, and larger, growth curve than other computing systems? (I think there is a basis for claiming that there are some reasons which would support a _shallower_ growth curve for routing equipment, actually). In short: are you claiming that the caeteris paribus assumption in comparing Moore's Law to global routing table size is clearly false? It would be nice to see even a partial proof of such a claim. >From anyone. Sean. (today's insult-free posting)

3 2

Re: What is the limit? (was RE: multi-homing fixes)
by smd＠clock.org 29 Aug '01

29 Aug '01

| [big routers] will cope just fine with the current routing table growth. "current" being the key word here. It's not exponential today. (pace hcb) That is, after all, the entire point of this conversation: is current growth "too slow", "too fast" or "just right"? Sean.

2 1

Re: What is the limit? (was RE: multi-homing fixes)
by smd＠clock.org 29 Aug '01

29 Aug '01

| Nonetheless, it's still annoying as hell that Cisco can't just allow a GB | or more of RAM in 5-6 figure router and just be done with that aspect of | it... They used to build boxes with cheap no-ECC, no-parity DRAM and had their fingers blown off by people who started counting the difference between line BER and e2e checksum failures, particularly on faster boards. Now they use well-built ECC SDRAM in general, and retain their fingers. AFAIK they don't forbid anyone from going out and acquiring el-cheapo RAM from other suppliers. A bit-error in a patricia tree can be really yucky. Sean.

1 0

multi-homing
by steve uurtamo 29 Aug '01

29 Aug '01

i) prefix length is unfortunately not directly related to the "need" for multihoming. ii) multihoming costs the person receiving the multihoming an amount of money that is variably determined by their upstream proviers. iii) the number of prefixes that _would_ multihome if they: a) had the appropriate equipment b) had the appropriate tech-brains c) had an ASN d) had connections through multiple providers is _not_ known. if you don't believe i) just imagine a content provider that is really interested in reaching its customers through the most direct and fast path possible, and who can't afford any outages. (their content isn't really worth much during an outage). ii) and iii) are important, taken together. is _your_ router going to explode if my /24 gets added to your table? of course not. your main concern is that your router explodes if _too_ many people's prefixes, of whatever size, get added to your table. this is, i think, a good thing to worry about, but it's a little early to be worrying. if nothing else (and i'm most certainly not arguing that this is the best solution), market pressure can be applied at points ii) and iii) to reduce to an arbitrarily small number the number of prefixes of concern here. all arguments of the variety, "i pass X [MGT]bits/sec, that's why i qualify and you don't" or of the variety "i house a network of size /X and that's why i qualify and you don't" are specious as well. ISP's need to multihome for reasons quite different (well, exactly the same but in an inverted direction) from the reasons that content providers do. it's important to note that prefix length is _not_ always, or perhaps even often, directly related to amount of traffic passed. obligatory attempt at a useful comment: completely ooma(tm) speculation, but i wonder if this would work: (if someone's done this calculation, or can back-of-the-envelope do it, i'd be curious to hear the results): consider the size of an ideally "full" route table for a particular traffic-passing router. consider a) the amount of time it takes for a "maximal" percentage of particular prefixes in that table to actually pass traffic across two of your interfaces and b) acceptable latency across those two interfaces in your router. ("maximal" here means whatever your router's prefix table capacity is, and "full" means the size that we as a community need to be able to support but currently cannot.) it's possible that you can: update a "cache" of prefix entries locally against a slower, but more "expandable" prefix-server. the long and the short of it is, you offload the "live" route table to a slower (read more cheapy expandable) box that has a fast connection to your router(s). when a packet's destination route prefix isn't contained in the cache, the prefix-server is polled (remember, very low latency to this box) for the correct route, and it's added to the table. prefixes not used for awhile age out of the local cache, and prefixes currently in use are checked (through the prefix-server) periodically to determine validity. end-to-end (from far-flung BGP speaking routers) propogation time for route information isn't exactly zippy right now anyway, so it shouldn't bother anyone that "best-path as currently should be known to me" isn't always taken for every packet. so the tradeoff is more clear here -- the "slow" table server could conceivably use very cheap technology for table storage (many GB of RAM on something with a slow CPU, or even perhaps a fast-enough RAID array spring to mind), and the "fast" router only needs to poll at a rate commensurate with acceptable latency. the "slow" box and "fast" box could be independent units inside of a commercial router or not, and the "fast" link between the live table and the cache should be cheap enough. i'm not describing a route server here, really. i'm really talking about offloading the space requirements to something that's slower, and using a local cache to (presumably) keep router table size down. (this external box _would_ be the one responsible for propogating bgp updates to your peers, and inbound bgp updates would be transparently redirected by the router to this box to deal with). s.

3 2

Re: What is the limit? (was RE: multi-homing fixes)
by David Lesher 29 Aug '01

29 Aug '01

{long thread...} Routing is a Ponzi scheme. You win by getting out early...or not playing. -- A host is a host from coast to coast.................wb8foz(a)nrk.com & no one will talk to a host that's close........[v].(301) 56-LINUX Unless the host (that isn't close).........................pob 1433 is busy, hung or dead....................................20915-1433

1 0

Re: multi-homing fixes
by smd＠clock.org 29 Aug '01

29 Aug '01

Having lived for a while now in a small country, whose content for the most part is encrypted into a language only a few million people can read, I can assure you that there is nontheless alot of traffic flowing into the eyeballs of people living there (even though it's in other languages including unencrypted American), and the vast majority of that flow is paid for by the people living in that small country, whether they're paying-to-see or paying-to-be-seen. The heavy politics are all in who gets what piece of that cash flow. CNN has a local organization maintaining translated and locally-written material, for what it's worth. And anyway, with Akamizing and AOLephanting and so forth going on all over the place, they're a bad example for many other reasons. (Not least of which is that I get better news coverage elsewhere). But who would care if Bill's Bait Shop in California stopped being able to talk to a local national provider? And who would pay to fix that connectivity, if it were fixable only through the application of money? Who knows? (I'll take yer 50 cents though, Bill). Anyway, the whole idea of universal connectivity is a complete myth. Even when the connectivity is advertised at any given moment, it's not necessarily there. Multihoming in practice quickly reveals that "full routes" are not the same size and do not lead to the same goodput, as noted in part by Valdis. Sean.

1 0

RE: multi-homing fixes
by Roeland Meyer 29 Aug '01

29 Aug '01

|> From: Jonathan Disher [mailto:jdisher@ipix.com] |> Sent: Wednesday, August 29, 2001 1:11 AM |> |> > A BGP route seems to take approximately 155 bytes (based |> on "show ip |> > route summary" on a convenient GSR; YMMV). So a singe BGP |> route, at |> > $1000 per 128 meg (that is, at the absurd markup a vendor might |> > charge), you're looking 1/8 of a cent per route per router. That |> > doesn't count memory for a separate forwarding table; still, we're |> > not talking about a billable amount per router. |> |> You've forgotten CPU time (I've heard stories from my boss, |> who swear's |> he's older than dirt, of the days when people used to pay |> five, even ten |> whole dollars (wow Wally!) for the privilege of sitting in front of a |> teletype to type things in and get stuff spit out), |> $neteng's time to sit |> and type "sh ip rou x.y.z.0" or "sh ip bgp regexp xxxxx" to |> check for the |> existence of the prefix, and the labor to prepare, print, |> and mail the |> bill. The stamp alone, for a US destination, is a whole 34 cents! |> |> </sarcasm!> hehe Along with micro-allocations comes the issue of micro-payments. It costs ten times the face amount to cut a 50 cent check.

1 0

RE: multi-homing fixes
by Dave Israel 29 Aug '01

29 Aug '01

Sean, you male chauvinist pig. ;-) Actually, the following is a general rant, and not aimed at Sean. A BGP route seems to take approximately 155 bytes (based on "show ip route summary" on a convenient GSR; YMMV). So a singe BGP route, at $1000 per 128 meg (that is, at the absurd markup a vendor might charge), you're looking 1/8 of a cent per route per router. That doesn't count memory for a separate forwarding table; still, we're not talking about a billable amount per router. Now, if I am a service provider of any scale, and I do not accept your specific announcements, I run the risk of taking a sub-optimal path to you. There are lots of acceptable reasons for this to be. What if you are transitioning from provider X to provider Y, and have been given 60 days to renumber? Should you renumber on Transition Day and suffer the effects of waiting for nameservice? Or if you do mistrust your provider, or find it cheaper to connect to two smaller tier-3ish providers than one tier-1ish to get the same distance from the sites you need to reach? As a provider, I want to be able to offer the best connectivity to my customers, and even a little courtesy to the dear departed. I want to provide, and respect other people's provision of, these services. If you're not providing these services to people, then sure, filter your BGP announcements. On the other hand, if you're not providing these services, and you're not multihomed (and unless you have a /19 or so, part of the problem, in which case you have no place to bitch), what the heck do you need the table for? Assuming you're a good Internet player who has no need to be kind to your fellow providers, filter and dampen and save your time and memory. Heck, even if you are being kind, dampen; there's no reason to let a madly flapping network give your iBGP mesh a heart attack. But a stable route providing intelligent path selection to a network your customers may need to reach, or time for a site to renumber without any loss of connectivity? For an eight of a cent? I mean, come on. -Dave On 8/28/2001 at 08:41:40 -0700, Sean M. Doran said: > > "David Schwartz" <davids(a)webmaster.com> writes: > > | I'm not sure I believe that this tragedy of the commons exists where people > | route on allocation boundaries. If I make Sprint carry an extra route just > | for my little network, that helps all Sprint customers reach my little > | network. > > Cool, so you can get them to contribute a fraction of the 50 cents you owe me? > > Since I live downstream from Sprint, I tell you what: I'll give you > the first flap for free. And that's a generous offer, given that > your little network represents approximately zero percent of my traffic, > and I doubt I'll be complaining to my other personalities that I can't > get at your fascinating content without subsidizing your network's > globally-visible dynamicism. > > Sean. (who does, incidentally, subscribe to some pay-for-play > web sites, for example, which helps them pay for a part of THEIR > dynamicism. i must remember to ask for a 50 cent * n discount > at renewal time...) > > P.S.: the point here, since it's easy to miss, is that a clearing house > function is useful for containing complexity of negotiation, and > is currently being done by the RIRs on behalf of their owners > > business opportunity: a tool which reliably lets one decide if > prefix x is worth carrying in all its flapping glory, or whether > it should be bitbucketed until a bill is paid -- Dave Israel Senior Manager, IP Backbone Intermedia Business Internet

2 1