Re: "2M today, 10M with no change in technology"? An informal survey.
In another mailing list, someone has asserted that "noone believes router vendors who say [they can support 2M routes today and 10M with no change in technology]".
Haven't we gone over the CPU speed argument recently enough?
Do you believe router vendors who state they today have "capacities on the order of 2 million ipv4 routes and they have no reason to expect that they couldn't deliver 10 million route FIB products in a few years given sufficient demand."?
Pointless argument, wait and see as it doesn't matter. If they do then we're fine. If they don't then we won't have that many routes so it won't be a problem.
- where do you believe existing routing technology will fall down?
If people sit watching the train crash rather than take avoiding action
- what steps will you take/are you taking to limit your vulnerability?
As long as I can afford as big a RP as the next ISP we're fine. brandon
On Sun, Aug 26, 2007 at 09:35:07AM +0900, Randy Bush wrote:
As long as I can afford as big a RP as the next ISP we're fine.
and own enough router vendor stock that you can compensate for running your business at a loss
I think the far more interesting thing will happen long before we see the tables explode and we have the end of the internet as we know it (but i'm still fine). Vendors will come up with creative solutions and push them to the market as fast as necessary, or we'll come up with other techniques (eg: dampening to keep that pesky cpu usage down) to help us muddle through until the factory builds enough 64-way route-processors to handle the updates, fib download at 10ge speeds, etc.. that will become necessary in the platforms. The reason why I say this is because we're going to see a crunch on the bandwidth side (IMHO) before we see this crunch on the CPU side. Based on what I'm hearing from folks in the marketplace, not every network is selling 10GE's like they're candy. Even with link aggregation and other tricks, that can get you to what? 8x10G? 16x10G? That may help some folks until some faster next-gen standard comes out from IEEE or ITU. It doesn't appear that everyone has jumped on the 40G/768 bandwagon and that n*10G is much more common because it's "cheap". After we all upgrade to 100GE (or the HSII - high speed internet interface) on our routers, what's next? bundle it a up to 800G? How much of a $VENDOR chassis will that consume? How much power will it require? Will dispersion over the medium (assuming fiber) be so bad that it's unusable after going more than 100 meters? How will the larger networks interconnect? Ignoring how we deal with the rib updates, i care about how the bits will get moved about. This is far more pressing in the next few years than can $vendor slap a faster cpu on a board and call it good. I think we all know that the cpus are underpowered in most routers compared to the desktop and server space. Compared to the couple of million dollars it would take to populate a nice HFR^WCRS/t-series network, the cost of a 2.9ghz processor is only $1200 (retail/wholesale), or only about some tiny fraction of a $1m router. What will matter is how much that HSII costs and how transport, hashing, and the ability to mux all those 10G or 40G waves into something useful. So I say, Baah on the rib/fib issues for now, but since no major networks would want to disclose their backbone traffic, it'll be interesting to see how and when this exposes itself as the sleeping bear that I personally believe it will become.. (my guess, sometime 2011 or so, maybe earlier). then again, i could be some rambling fool who is out of touch. - jared (opinions don't reflect those of my employer, wife or children, or possibly even myself). -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Maybe I'm oversimplifying. Its Saturday and entirely possible I'm not thinking on all brain cells ($clue--). When it comes to an FIB, there are only say 100-200 destinations on a big router (outbound). Irrespective of the 2M or 20M routes it has. Even though the entire Internet isn't aggregated the way it should be, for FIB purposes, the router knows how it will route out to its 100-200 destinations (ports). Couldn't it just aggregate before it drops routes into its FIB and only import specifics (deagg) if a destination changes for a more specific prefix [like the swamp]? We talk about default-free zones as needing every prefix, and they do (for propagation purposes) but for FIB purposes, they really don't. So they keep a Zebra or OpenBGPd like process (thinking Juniper) that contains *everything* and handles propagation and then each FIB gets an aggregated entry dumped into each FIB on each routing card. This scales very well on the order of a well aggregated internet, and even networks with lots of customer routes (internally or because of lots of customer transit connections) each router only deals with getting the packet to its neighbor. Server CPUs & RAM can handle lots of updates on a 10MM route table all day long. FIBs only need to get involved if the update will change their view of a route -- a central CPU (or cluster of them) can decide that before the FIB sees it. If this isn't clear, I can probably explain it better, but basically what I am saying is that instead of aggregated the FIB based on CIDR rules or expecting total conformity, the router just aggregates from its point-of-view -- if an aggregate to multiple disparate netblocks all goes to the same place, it puts an aggregate in. For example, a router with only 1 connection (no matter how many routes being sent by its upstream), would only have 1 route entered into its FIB -- because no matter where the route goes, it can go upstream. If it feeds a route table to another router, the downstream will see the whole route table irrespective. I know the above degenerate case doesn't address bogons, but really we are only talking about a few dozen extra entries into an FIB for them. Deepak Jain AiNET
* Deepak Jain:
When it comes to an FIB, there are only say 100-200 destinations on a big router (outbound). Irrespective of the 2M or 20M routes it has. Even though the entire Internet isn't aggregated the way it should be, for FIB purposes, the router knows how it will route out to its 100-200 destinations (ports).
Could be more than that in some MPLS and IXP environments, but in essence, that's true.
Couldn't it just aggregate before it drops routes into its FIB and only import specifics (deagg) if a destination changes for a more specific prefix [like the swamp]? We talk about default-free zones as needing every prefix, and they do (for propagation purposes) but for FIB purposes, they really don't.
My understanding is that there are no known algorithms for fast updates (and particularly withdrawals) on aggregated FIBs, especially if those FIBs are stored in CIDR form. This is the prime reason why all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a couple of months. But I'm sure that this is not the end of the story.
For example, a router with only 1 connection (no matter how many routes being sent by its upstream), would only have 1 route entered into its FIB -- because no matter where the route goes, it can go upstream.
This will cause routing loops for unallocated address space.
My understanding is that there are no known algorithms for fast updates (and particularly withdrawals) on aggregated FIBs, especially if those FIBs are stored in CIDR form. This is the prime reason why all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a couple of months.
Do we have a real date for when this occurs? If you aren't doing uRPF, I thought they ran up to 256,000 routes. (I may not recall correctly) "Fast withdrawals".. We don't have instantaneous convergence right now, are you sure you aren't talking about "easy withdrawals" (as in, lots of compute time required per adjustment). MPLS environments would have additional entries where they are doing something with VRF or MPLS... the nodes in between wouldn't see anything as normal.
For example, a router with only 1 connection (no matter how many routes being sent by its upstream), would only have 1 route entered into its FIB -- because no matter where the route goes, it can go upstream.
This will cause routing loops for unallocated address space.
This would be addressed in the bogon case. Deepak Jain
Heya,
My understanding is that there are no known algorithms for fast updates (and particularly withdrawals) on aggregated FIBs, especially if those FIBs are stored in CIDR form. This is the prime reason why all those Cisco 65xx/76xx with MSFC2/PFC2 will be worthless junk in a couple of months.
Do we have a real date for when this occurs? If you aren't doing uRPF, I thought they ran up to 256,000 routes. (I may not recall correctly)
We ran into this hiccup a few months ago on a Sup720-3B (well, a 3BXL which mistakenly had a 3B card in the chassis, causing the SUP to clock down and act like a 3B), but I think the Sup2's are in a similar situtation. Though the box can handle up to 224k routes, they are set by default to only handle 192k IPv4 + MPLS routes plus 32k IPv6 + IP multicast routes. You can retune this so that you can get up to 224k IPv4 routes, but I've recently seen our Internet table bumping against this. My understanding is that this is a hardware limit, so upgrading is your only option. Eric :)
On Mon, 27 Aug 2007, Eric Gauthier wrote:
Do we have a real date for when this occurs? If you aren't doing uRPF, I thought they ran up to 256,000 routes. (I may not recall correctly)
We ran into this hiccup a few months ago on a Sup720-3B (well, a 3BXL which mistakenly had a 3B card in the chassis, causing the SUP to clock down and act like a 3B), but I think the Sup2's are in a similar situtation. Though the box can handle up to 224k routes, they are set by default to only handle 192k IPv4 + MPLS routes plus 32k IPv6 + IP multicast routes. You can retune this so that you can get up to 224k IPv4 routes, but I've recently seen our Internet table bumping against this. My understanding is that this is a hardware limit, so upgrading is your only option.
The sup2 can actually handle a bit more ipv4 routes than the Sup720(non-3bxl). I don't know if it can go all the way to 256k routes. I can't seem to find any cisco data sheets that specify max ipv4 routes on the sup2. The output from show mls cef hardware suggests 256k is the limit. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
According to this link, which alleges to be from cisco-nsp, an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose. The MSFC2 therefore can server 244,000 routes without uRPF turned on. Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone? http://osdir.com/ml/network.nsp.cisco/2002-08/msg00283.html Deepak
On Aug 27, 2007, at 2:49 PM, Deepak Jain wrote:
According to this link, which alleges to be from cisco-nsp, an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.
The MSFC2 therefore can server 244,000 routes without uRPF turned on.
Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?
Um? Real Soon Now? According to http://www.cidr-report.org/as2.0/ we're at 233,000 routes (as seen from AS 2.0 now) and the rate of growth as seen from http://bgp.potaroo.net/ seems pretty steep. I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?) Thanks, -drc
David Conrad wrote:
On Aug 27, 2007, at 2:49 PM, Deepak Jain wrote:
According to this link, which alleges to be from cisco-nsp, an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.
The MSFC2 therefore can server 244,000 routes without uRPF turned on.
Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?
Um?
Real Soon Now?
According to http://www.cidr-report.org/as2.0/ we're at 233,000 routes (as seen from AS 2.0 now) and the rate of growth as seen from http://bgp.potaroo.net/ seems pretty steep.
I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?)
I found that, eventually. I'm only seeing about 227K routes, but customer routes from wherever the CIDR report is getting data could be part of the difference. Where do the FIBs break on older 12000 series and M-series routers? (or pick the *next* most popular piece of network equipment that is used in full-routes scenarios). Maybe I should take a whack at my aggregation idea on an MSFC2 to see how it does. Hmmm.. Deepak
On Mon, 27 Aug 2007, Deepak Jain wrote:
Where do the FIBs break on older 12000 series and M-series routers? (or pick the *next* most popular piece of network equipment that is used in full-routes scenarios).
On the 12000, I'd give the following observations on the state of the older linecards for DFZ routing: GRP that can't handle 512 meg memory has been useless for quite some time. GRP-B with 512 megs of ram seems ok for at least 6-12 more months. PRP needs 1 gig of ram. All LCs need at least 256 megs of route memory. 4GE engine3 LC needs 512 megs of route memory. 10x1GE Engine 4 LC needs 512 megs of route memory. Engine2 LCs are starting to run out of forwarding resources, cisco states 200k routes, but obviously they still work, but for how long? Otoh the SIP-601 comes with 2 gigs of route memory, which is really nice. The 12000 with recent hardware will most likely last quite some time, but the hardware designed in the late 90ties is (not strangely) running out of steam. So if you have old hardware, you need to monitor your memory and table utilization on a monthly basis. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Mon, 27 Aug 2007, David Conrad wrote:
Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone?
Um?
Real Soon Now? ... I must be missing something obvious (or should I be dusting off my unused Y2K survival gear?)
Unlike Y2K, the end of the useful service life up the Sup2 can easily be pushed further away in time. ASnum NetsNow NetsAggr NetGain % Gain Description Table 233651 151129 82522 35.3% All ASes AS4134 1337 339 998 74.6% CHINANET-BACKBONE No.31,Jin-rong Street AS18566 1020 101 919 90.1% COVAD - Covad Communications Co. AS4323 1315 437 878 66.8% TWTC - Time Warner Telecom, Inc. AS4755 1331 507 824 61.9% VSNL-AS Videsh Sanchar Nigam Ltd. Autonomous System There's really only 151129 routes you need to have "full routes". Forcing just these top 4 slobs to aggregate reduces your global table by 3619 routes. Forcing the top 30 to aggregate frees up 15809 routes. Of course there are other reasons to upgrade (better CPU, MPLS, IPv6, etc.), but if you can't upgrade, there are alternatives to stretch the old hardware. It's not like it hasn't been done before. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
Jon, On Aug 27, 2007, at 5:50 PM, Jon Lewis wrote:
Any reasonably valid way of predicting when we'll hit 244,000 routes in the default-free zone? Real Soon Now?
According to Geoff, the BGP table is growing at around 3500 routes per month, so we're looking at blowing out MSFC2s in about 3 months if nothing changes. And here I was, wondering about 2M routes...
Unlike Y2K, the end of the useful service life up the Sup2 can easily be pushed further away in time.
"Easy" is, I suspect, in the mind of the route injector.
There's really only 151129 routes you need to have "full routes". Forcing just these top 4 slobs to aggregate reduces your global table by 3619 routes.
~1 more month.
Forcing the top 30 to aggregate frees up 15809 routes.
~3 more months.
Of course there are other reasons to upgrade (better CPU, MPLS, IPv6, etc.), but if you can't upgrade, there are alternatives to stretch the old hardware.
For a few more months. What are upgrade cycles like again? How common are the MSFC2s?
It's not like it hasn't been done before.
Yep. The nice thing about repeating history is you have a good idea of the whinage that you're in store for. "CIDR Wars 2.0: This Time It's For Real! No, really. We mean it this time." :-) Regards, -drc "I used to be disgusted, now I try to be amused ..." -- Elvis Costello
On Mon, 27 Aug 2007, David Conrad wrote:
For a few more months. What are upgrade cycles like again? How common are the MSFC2s?
I think we'll find out in a few months, when the "internet breaks" in a whole bunch of places where the admins aren't aware of this issue or operations have been downsized to the point that things are mostly on auto-pilot. I'm guessing there are a good number of Sup2's in use, and that a good % of them think they're fine...as they have 512MB RAM and on the software based routers, that's plenty for current full BGP routes. Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Wed, Aug 29, 2007 at 06:48:43PM -0400, Jon Lewis wrote:
On Mon, 27 Aug 2007, David Conrad wrote:
For a few more months. What are upgrade cycles like again? How common are the MSFC2s?
I think we'll find out in a few months, when the "internet breaks" in a whole bunch of places where the admins aren't aware of this issue or operations have been downsized to the point that things are mostly on auto-pilot. I'm guessing there are a good number of Sup2's in use, and that a good % of them think they're fine...as they have 512MB RAM and on the software based routers, that's plenty for current full BGP routes.
private replies suggest (w/ lots of handwaving) that perhaps 20-35% of the forwarding engines in use might fit this catagory.
Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread.
that would be a sucker bet
---------------------------------------------------------------------- Jon Lewis | I route
--bill
On Thu, Aug 30, 2007, bmanning@vacation.karoshi.com wrote:
Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread.
that would be a sucker bet
I've started seeing it already occasionally. I've even seen one guy here "upgrade" from Sup2 to Sup32. Adrian
So with a fairly predictable growth of 3500 routes per month, we're going to have some issues with the current equipment out there (despite this being a rather predictable situation...) So what might happen in three years if we have a sudden inflection in new routes per month due to use by major backbones of non- hierarchically allocated address space for new customer additions? I.E. If at some time unknown around 2010, ISP's stop receiving new allocations from their RIR, and instead use of many smaller "recycled" IPv4 address blocks, we could be looking at a 10x to 20x increase in routes per month for the same customer growth. Is the equipment being installed *today* and over the next two years capable of sustaining 50K new routes per month, and if so, for how long? Thanks, /John At 4:47 AM +0000 8/30/07, bmanning@vacation.karoshi.com wrote:
On Wed, Aug 29, 2007 at 06:48:43PM -0400, Jon Lewis wrote:
On Mon, 27 Aug 2007, David Conrad wrote:
For a few more months. What are upgrade cycles like again? How common are the MSFC2s?
I think we'll find out in a few months, when the "internet breaks" in a whole bunch of places where the admins aren't aware of this issue or operations have been downsized to the point that things are mostly on auto-pilot. I'm guessing there are a good number of Sup2's in use, and that a good % of them think they're fine...as they have 512MB RAM and on the software based routers, that's plenty for current full BGP routes.
private replies suggest (w/ lots of handwaving) that perhaps 20-35% of the forwarding engines in use might fit this catagory.
Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread.
that would be a sucker bet
---------------------------------------------------------------------- Jon Lewis | I route
--bill
John -- Great panic starting question. I'd guess that by 2010, we'll be worried more about IPv6 growth than IPv4 growth but the archives will bite me in the you-know-where in 3 years when I'm wrong (in either direction). And then we'll talk about how fast FIBs get eaten with both IPv4 (legacy) and IPv6 (new coolness) routes are in the same TCAMs/SRAMS/RLDRAM/whatever. How many of us realistically believe that the core routers we install today will be in use (as such) in 3-4 years??? I can't think of a time in the last 12+ years that that has been a good bet. Deepak John Curran wrote:
So with a fairly predictable growth of 3500 routes per month, we're going to have some issues with the current equipment out there (despite this being a rather predictable situation...)
So what might happen in three years if we have a sudden inflection in new routes per month due to use by major backbones of non- hierarchically allocated address space for new customer additions? I.E. If at some time unknown around 2010, ISP's stop receiving new allocations from their RIR, and instead use of many smaller "recycled" IPv4 address blocks, we could be looking at a 10x to 20x increase in routes per month for the same customer growth.
Is the equipment being installed *today* and over the next two years capable of sustaining 50K new routes per month, and if so, for how long?
Thanks, /John
At 4:47 AM +0000 8/30/07, bmanning@vacation.karoshi.com wrote:
On Mon, 27 Aug 2007, David Conrad wrote:
For a few more months. What are upgrade cycles like again? How common are the MSFC2s? I think we'll find out in a few months, when the "internet breaks" in a whole bunch of places where the admins aren't aware of this issue or operations have been downsized to the point that things are mostly on auto-pilot. I'm guessing there are a good number of Sup2's in use, and that a good % of them think they're fine...as they have 512MB RAM and on the software based routers, that's plenty for current full BGP routes.
On Wed, Aug 29, 2007 at 06:48:43PM -0400, Jon Lewis wrote: private replies suggest (w/ lots of handwaving) that perhaps 20-35% of the forwarding engines in use might fit this catagory.
Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread. that would be a sucker bet
---------------------------------------------------------------------- Jon Lewis | I route --bill
At 2:14 PM -0400 8/30/07, Deepak Jain wrote:
John --
Great panic starting question.
Sorry, not my intent. I'm just trying to get a handle on the state of the art of what's available today, and whether it has some really excellent scaling properties in case we see a much more granular block reuse. It's by no means inevitable, just a probable outcome of any routing-indifferent IP address space "market" (whether legitimate or not).
I'd guess that by 2010, we'll be worried more about IPv6 growth than IPv4 growth but the archives will bite me in the you-know-where in 3 years when I'm wrong (in either direction).
I'd love to believe that, but there's real world issues with IPv6 readiness as well as challenges communicating the business need to make the necessary preparations. When we're all still debating in technical fora the need to move to IPv6 versus viability of a heavily NAT'ed IPv4 endgame, you can't expect the folks in boardrooms to commit to any large investment decisions. /John
bmanning@vacation.karoshi.com wrote:
On Wed, Aug 29, 2007 at 06:48:43PM -0400, Jon Lewis wrote:
On Mon, 27 Aug 2007, David Conrad wrote:
For a few more months. What are upgrade cycles like again? How common are the MSFC2s? I think we'll find out in a few months, when the "internet breaks" in a whole bunch of places where the admins aren't aware of this issue or operations have been downsized to the point that things are mostly on auto-pilot. I'm guessing there are a good number of Sup2's in use, and that a good % of them think they're fine...as they have 512MB RAM and on the software based routers, that's plenty for current full BGP routes.
private replies suggest (w/ lots of handwaving) that perhaps 20-35% of the forwarding engines in use might fit this catagory.
Anyone want to bet there will be people posting to nanog and cisco-nsp in a few months asking why either the CPU load on their Sup2's has suddenly shot up or why they keep noticing parts of the internet have gone unreachable?...oblivious to this thread.
that would be a sucker bet
If Cisco could ship enough units when asked, I'd say their next couple of quarters are in the bag... but since they have such huge lead times..... well, I am guessing a lot of people will start considering taking partial routes. Transit providers would do well to have a distribute-list or similar configured to offer these guys when they call rather than trying to engineer something on an ICB basis. Deepak
On Mon, 27 Aug 2007, Jon Lewis wrote:
Of course there are other reasons to upgrade (better CPU, MPLS, IPv6, etc.),
Now if this was a dust old MSFC2 that was like 5 years old I'd say ok. The problem is twofold: 1. Cisco is still selling the 7600 with the Sup32 bundle (which is what we bought) and saying you can take a full route table on it. I could already do MPLS and IPv6 on this box. This is pretty new hardware. 2. The only thing I could buy is the top of the line Sup720 3BXL. Ok, fine, but I don't need mega-super-d00per backplane speed. I just need more TCAM like Christoper Walken needs more cowbell. Cisco needs to have a reasonable solution to this problem - especially if they want to keep selling the 7600 as a router. If I end up upgrading because of this it will probably be a forklift upgrade to another platform. And there's no guarantee that it would be a Cisco one. -- John A. Kilpatrick john@hypergeek.net Email| http://www.hypergeek.net/ john-page@hypergeek.net Text pages| ICQ: 19147504 remember: no obstacles/only challenges
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
On Mon, 27 Aug 2007, John A. Kilpatrick wrote:
a reasonable solution to this problem - especially if they want to keep selling the 7600 as a router.
and here I always looked at the 6500 as a switch...
And the 7600 is a router? :) ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Mon, 27 Aug 2007, Jon Lewis wrote:
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
On Mon, 27 Aug 2007, John A. Kilpatrick wrote:
a reasonable solution to this problem - especially if they want to keep selling the 7600 as a router.
and here I always looked at the 6500 as a switch...
And the 7600 is a router? :)
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
On Mon, 27 Aug 2007, John A. Kilpatrick wrote:
a reasonable solution to this problem - especially if they want to keep selling the 7600 as a router.
and here I always looked at the 6500 as a switch...
And the 7600 is a router? :)
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
And tagged with some white paint. Though if you've kept up with the latest IOS developments, cisco is finally differentiating the platforms we've assumed for years were only different in angle and paint. 6500's won't get to run the newest 7600 code. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
On Mon, 27 Aug 2007, Jon Lewis wrote:
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
And tagged with some white paint.
Though if you've kept up with the latest IOS developments, cisco is finally differentiating the platforms we've assumed for years were only different in angle and paint. 6500's won't get to run the newest 7600 code.
Oh poor cow :( In all seriousness though, most routing platforms have their costs and benefits. The 7600/6500 do some things nicely, apparently large FIB's aren't their strength though (in most deployed configs atleast). -Chris
On Mon, 27 Aug 2007, Jon Lewis wrote:
Though if you've kept up with the latest IOS developments, cisco is finally differentiating the platforms we've assumed for years were only different in angle and paint. 6500's won't get to run the newest 7600 code. I think Cisco is coming to their senses. SXH has *most* of SRB features, while (hopefully) more stable.
At this point, imho, the rsp720 is getting the short end of the stick, because it is only limited to SRB+, while you have a choice of SX* and SRB on the sup720. But I think, imho, this discussion belongs to cisco-nsp more than to nanog-l. -alex [not speaking as mlc blah blah]
On Tue, 28 Aug 2007, Chris L. Morrow wrote:
And the 7600 is a router? :)
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
I still needle my Cisco rep about that from time to time. IMHO, the 6500/7600 split was one of the dumbest, most poorly thought-out decisions Cisco ever made. That and they still haven't given me the warm-and-fuzzy about the plans for IOS licensing. Where I work, we're heavily invested in 6500s in the core and I don't see that changing any time soon. The borders are Junipers because they 'just plain work' :) jms
On 8/27/07, Justin M. Streiner <streiner@cluebyfour.org> wrote:
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
On Mon, 27 Aug 2007, Hex Star wrote:
On 8/27/07, Justin M. Streiner <streiner@cluebyfour.org> wrote:
I thought it was just a 6500 that sommeone got drunk and tipped over on it's side, like a cow...
http://farm.tucows.com/images/2006/07/cow_tipping.jpg :D While its occasionally amusing, can we please keep the "humour" to the minimum, while sticking to the operational content?
-alex (mlc chair)
On 8/27/07 7:36 PM, "Chris L. Morrow" <christopher.morrow@verizonbusiness.com> wrote:
and here I always looked at the 6500 as a switch...
It switches, it routes, it makes julienne fries... -- John A. Kilpatrick john@hypergeek.net Email| http://www.hypergeek.net/ john-page@hypergeek.net Text pages| ICQ: 19147504 remember: no obstacles/only challenges
On Mon, 27 Aug 2007, John A. Kilpatrick wrote:
1. Cisco is still selling the 7600 with the Sup32 bundle (which is what we bought) and saying you can take a full route table on it. I could already do MPLS and IPv6 on this box. This is pretty new hardware.
Where are they saying that? The Sup32 sounded great until it became clear that it came with PFC3B (not 3BXL), and that there was no upgrade path to 3BXL. If it was/is being sold as a BGP routing solution, it was awfully short sighted.
2. The only thing I could buy is the top of the line Sup720 3BXL. Ok, fine, but I don't need mega-super-d00per backplane speed. I just need more TCAM like Christoper Walken needs more cowbell. Cisco needs to have a
We're in the same boat. According to show catalyst6000, our Sup2's are doing just fine. If there were a Sup32-3BXL, it'd be more than sufficient for our needs.
If I end up upgrading because of this it will probably be a forklift upgrade to another platform. And there's no guarantee that it would be a Cisco one.
I guess cisco wants to play chicken with us and Juniper. Will you really do the forklift, or just bite the bullet and go Sup720-3BXL? I think they're better on the latter and counting on a bunch of hardware sales in the coming months. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
1. Cisco is still selling the 7600 with the Sup32 bundle (which is what we bought) and saying you can take a full route table on it. I could already do MPLS and IPv6 on this box. This is pretty new hardware.
Where are they saying that? The Sup32 sounded great until it became clear that it came with PFC3B (not 3BXL), and that there was no upgrade path to 3BXL. If it was/is being sold as a BGP routing solution, it was awfully short sighted. Their reps do it all the time. I worked with my rep to buy a couple of new routers. I specifically said I would be taking a full routing table on
these boxes- Cisco's rep said the Sup-32 would be fine for my needs. Now I definitely didn't do as much checking as I should have but I was busy and that's why you have rep's in the first place. (I kept thinking the Sup32 was based on the 3BXL- I have no idea why). Thankfully I don't need to take a full table on these routers and their forwarding speed among the few ports I have is more important than the FIB size. That said- if I did need the full table I would be royally ticked off at Cisco right now.
If I end up upgrading because of this it will probably be a forklift upgrade to another platform. And there's no guarantee that it would be a Cisco one.
I guess cisco wants to play chicken with us and Juniper. Will you really do the forklift, or just bite the bullet and go Sup720-3BXL? I think they're better on the latter and counting on a bunch of hardware sales in the coming months. Given how many people are tired of being screwed over by Cisco I wouldn't make that bet if I were Cisco.
-Don
On 8/27/07 9:39 PM, "Donald Stahl" <don@calis.blacksun.org> wrote:
Thankfully I don't need to take a full table on these routers and their forwarding speed among the few ports I have is more important than the FIB size. That said- if I did need the full table I would be royally ticked off at Cisco right now.
Well the way I'm putting it to my Cisco rep is "Why should I invest in 3BXLs instead of another vendor's solution?" I'm saying this repeatedly. Maybe they'll get the hint. I won't throw away the 7604s...I could totally redeploy them in my corporate infrastructure. At this point they really are Cat 6500s. I don't mind if they make a 7600-only train as long as the 7600s can still run 6500 code then at least it makes them useful. Just not as edge routers. I bet Juniper is lulzing this hardcore. -- John A. Kilpatrick john@hypergeek.net Email| http://www.hypergeek.net/ john-page@hypergeek.net Text pages| ICQ: 19147504 remember: no obstacles/only challenges
At 8:50 PM -0400 8/27/07, Jon Lewis wrote:
Unlike Y2K, the end of the useful service life up the Sup2 can easily be pushed further away in time.
ASnum NetsNow NetsAggr NetGain % Gain Description .... There's really only 151129 routes you need to have "full routes". Forcing just these top 4 slobs to aggregate reduces your global table by 3619 routes. Forcing the top 30 to aggregate frees up 15809 routes.
That's an additional ~5 months at the current rate of new routes (and current ratio of customers per new routed block.) There's a lot more than 3500 new customers per month globally and if we get to the point where they are not coming out of hierarchical PA space, the new monthly routing growth will increase dramatically. /John
On 8/27/07, Deepak Jain <deepak@ai.net> wrote:
an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.
The MSFC2 therefore can server 244,000 routes without uRPF turned on.
I'm hit square on with this because I use Sup2's with the msfc2/pfc2 for the link to both of my transit providers. I took this up with the Cisco TAC overnight to find out where I stand. Here's what I found: 1. The msfc2/pfc2 does in fact have a limit that starts at 244,000 routes. 2. Once the limit is reached, excess routes will fail over to software switching. TAC did not specify how routes are designated as excess. 3. The Sup 720 (except for the 3bxl) has a similar limit, however the "mls cef maximum-routes" command can be used to make upwards of 260,000 TCAM entries available to IPv4 unicast routing. The Sup 2 does not support this command. 4. The suggested upgrade path is the Supervisor 720-3BXL whose TCAM can support up to 1M IPv4 FIB entries or 500k IPv6 FIB entries. With a 7600 (instead of a 6500) the RSP 720-3CXL can do the same and also has a faster processor, more memory, etc. Now, my request for help: I have a leaf node on the DFZ handled by a pair of Sup2's (pfc2/msfc2), two transit providers and several peers. My focus is very heavily domestic, and I'd like to delay my upgrade. I'd like to buy some time by aggregating the incoming APNIC region prefixes (http://www.iana.org/assignments/ipv4-address-space) into the following FIB entries: 58.0.0.0/7 60.0.0.0/7 116.0.0.0/6 120.0.0.0/6 124.0.0.0/7 126.0.0.0/8 202.0.0.0/7 210.0.0.0/7 218.0.0.0/7 220.0.0.0/7 222.0.0.0/8 Can anyone suggest how to program that into the router or refer me to the URL of the correct documentation at Cisco's site? Thanks in advance, Bill Herrin -- William D. Herrin herrin@dirtside.com bill@herrin.us 3005 Crane Dr. Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
On Tue, 28 Aug 2007 15:11:52 -0400 "William Herrin" <herrin-nanog@dirtside.com> wrote:
On 8/27/07, Deepak Jain <deepak@ai.net> wrote:
an MSFC2 can hold 256,000 entries in its FIB of which 12,000 are reserved for Multicast. I do not know if the 12,000 can be set to serve the general purpose.
The MSFC2 therefore can server 244,000 routes without uRPF turned on.
<snip>
Now, my request for help:
I have a leaf node on the DFZ handled by a pair of Sup2's (pfc2/msfc2), two transit providers and several peers. My focus is very heavily domestic, and I'd like to delay my upgrade. I'd like to buy some time by aggregating the incoming APNIC region prefixes (http://www.iana.org/assignments/ipv4-address-space) into the following FIB entries:
58.0.0.0/7 60.0.0.0/7 116.0.0.0/6 120.0.0.0/6 124.0.0.0/7 126.0.0.0/8 202.0.0.0/7 210.0.0.0/7 218.0.0.0/7 220.0.0.0/7 222.0.0.0/8
Can anyone suggest how to program that into the router or refer me to the URL of the correct documentation at Cisco's site?
Probably better over at cisco-nsp, however I'd expect you'd use the "aggregate-address <prefix> <mask> summary-only" command to create aggregates, yet supressing them from being announced to any other BGP peer. I think that would still cause the more specifics to get into the FIB of the aggregating router, however there's a command I've only come across recently, under the "router bgp" section, which allows you to apply a route-map to routes as they go from the BGP RIB to the FIB. You might be able to use that to stop the more specifics getting into the FIB, with a route-map deny clause. The command is "table-map". I haven't used it myself, and the command reference says that it's only to set attributes so YMMV. I haven't had success using "deny" clauses in BGP attribute setting route-maps, so it may not be possible at all to use this command for this purpose. Another way you might avoid the more specifics getting into the FIB is to only accept a few known or selected large more specifics from those ranges from your upstreams e.g. 3 or so, dropping the rest, and use those select few to create the /6-8 aggregates you'll use internally. Probably a bit more work than the table-map method, but if that doesn't work, this is probably the way to do it. (Looks like the coffee is just kicking in this morning - I've just come up with another way just before I send this off.) Or you could set up a route server upstream of your router with the limited FIB and do the filtering and / or aggregation there. As it isn't in the forwarding path, you could probably use a lower end software Cisco platform with enough CPU and RAM just to do the BGP processing e.g. probably something as low end as an 1800 series with 1GB of RAM (I'd suggest switching CEF off to save RAM) would be quite fine to do that job. I'd even suggest an 800 series (400MHz PowerPCs are no slouches), however they've only got a max of 256MB of RAM with probably isn't enough (for a bit of fun one day, I put the full route table in a 128MB one, but it only got to 140 000 routes before it ran out of RAM.) HTH, Mark. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Bill, [...]
2. Once the limit is reached, excess routes will fail over to software switching. TAC did not specify how routes are designated as excess.
I'm not sure if the Sup2's handle this case differently from the Sup720s we were using, but, in our case, when we reached the ceilign the routes appeared in both the routing and CEF tables but were not populated into the FIB. Translation: the route was ignored.... Eric :)
[...]
2. Once the limit is reached, excess routes will fail over to software switching. TAC did not specify how routes are designated as excess.
most-specific-prefixes first. it has to be this way due to the way a TCAM search works.
I'm not sure if the Sup2's handle this case differently from the Sup720s we were using, but, in our case, when we reached the ceilign the routes appeared in both the routing and CEF tables but were not populated into the FIB.
Translation: the route was ignored....
how old is the software you were running on your cat6k? reason i ask is that since circa. 12.2(18)SXF9 (i.e. back in 2005), there has been a graceful degradation back to software forwarding for those entries that don't fit into the FIB TCAM: - when the h/w FIB is full (FIB exception) it goes into exception state where it will maintain the longest-prefix-matches by removing shortest- prefix-matches from the FIB TCAM first - it will also insert a default entry to punt lookup exceptions to software - software typically CAN maintain a full FIB, so entries which don't fit into hardware can be software forwarded in the CEF software switching path - when the h/w FIB is full, the following syslog message is generated: MLSCEF-SP-7-FIB_EXCEPTION: FIB TCAM exception, Some entries will be software switched of course, software forwarding is potentially orders-of-magnitude slower than h/w forwarding, so how much extra headroom this gives you once you exceed the capabilities is dependent on the amount of traffic to those prefixes that don't fit into the h/w tables. agree that this isn't "ideal", however Cisco has always been very specific about the h/w FIB & adjacency table sizes on the hardware in question. i know that vendor bashing is a sport in this list, but.... relevant bug-ids if you wanted to look up the details: CSCse90572 syslog message when FIB TCAM exceeds 95% utilization CSCsb18172 wrong packet forwarding at FIB exception if you need further clarification, feel free to contact me off-list, my work email address is ltd@cisco.com. cheers, lincoln. No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.484 / Virus Database: 269.12.10/977 - Release Date: 28-Aug-07 4:29 PM
agree that this isn't "ideal", however Cisco has always been very specific about the h/w FIB & adjacency table sizes on the hardware in question. i know that vendor bashing is a sport in this list, but.... Can you please point out where I can find this information ...
The only place I found information on the PFC3B was on a random page for the SUP 720-3B. I was completely unable to find the information on a Sup32 page. Now maybe my search technique isn't up to snuff- but I would hope I could find this information after searching for a couple of hours- I couldn't. I'm sure the information is on Cisco's site somewhere- but I honestly think that they could be a LOT more forward about it- rather then just very specific about it. -Don
agree that this isn't "ideal", however Cisco has always been very specific about the h/w FIB & adjacency table sizes on the hardware in question. i know that vendor bashing is a sport in this list, but....
Can you please point out where I can find this information ...
The Sup720 datasheet covers the capabilities (see http://www.cisco.com/en/US/products/hw/modules/ps2797/products_data_sheet091... 0080159856.html). I got that (cisco.com) URL from the google search 'Supervisor 720 data sheet'. I agree that the Sup32 datasheet (http://www.cisco.com/en/US/products/hw/switches/ps708/products_data_sheet090... ecd801c5cab.html) is less specific that it perhaps should be, but it does clearly talk about what "policy feature card" is onboard and from the same document hierarchy there is a link to 'Policy Feature Card-3B' which has the same data as the Sup720 datasheet on PFC3B. cheers, lincoln. NB. in my post there was a thinko, i incorrectly said "most-specific-prefixes first", i meant "least-specific-prefixes first".
On Wed, 29 Aug 2007, Lincoln Dale wrote:
reason i ask is that since circa. 12.2(18)SXF9 (i.e. back in 2005), there has
One of the problems with this is that the people that have the tendency of not knowing their hardware limitations are the same people that will be running SXD because they haven't put CFs into their SUP:s to handle the larger image sizes of SXE and later. -- Mikael Abrahamsson email: swmike@swm.pp.se
On 8/28/07 5:11 PM, "Lincoln Dale" <ltd@interlink.com.au> wrote:
agree that this isn't "ideal", however Cisco has always been very specific about the h/w FIB & adjacency table sizes on the hardware in question. i know that vendor bashing is a sport in this list, but....
The problem is that Cisco hasn't been forthcoming. To me it seems the data was hidden in a corner of a spec sheet. Meanwhile sales teams are still saying the PFC3B is acceptable for taking a full table. And the failure to produce a Sup32-3BXL or similar is also frustrating - I don't need Sup720 backplane speeds on my edge router. -- John A. Kilpatrick john@hypergeek.net Email| http://www.hypergeek.net/ john-page@hypergeek.net Text pages| ICQ: 19147504 remember: no obstacles/only challenges
participants (21)
-
Adrian Chadd -
Alex Pilosov -
bmanning＠vacation.karoshi.com -
Brandon Butterworth -
Chris L. Morrow -
David Conrad -
Deepak Jain -
Donald Stahl -
Eric Gauthier -
Florian Weimer -
Hex Star -
Jared Mauch -
John A. Kilpatrick -
John Curran -
Jon Lewis -
Justin M. Streiner -
Lincoln Dale -
Mark Smith -
Mikael Abrahamsson -
Randy Bush -
William Herrin