Gang, I have submitted an internet-draft to the IDR group on virtual aggregation (VA) (http://www.ietf.org/internet-drafts/draft-francis-idr-intra-va-00.txt). This draft suggests a few changes to routers that allow operators to control the size of their FIBs, shrinking them by 5x or 10x quite easily. This would extend the lifetime of routers that are constrained by FIB size. There has been a lively discussion of this on the IDR mailing list, including a suggestion that FIB reduction is more important for lower tier ISPs (tier 2, tier 3...) than for tier 1 ISPs. Unfortunately I don't think that people from smaller ISPs pay much attention to the IDR mailing list, so they are not being represented in this discussion. So I'm looking for input from folks who think that FIB reduction helps them, so as to better understand their requirements. Any help is much appreciated. Thanks, PF
Paul, Can this proposal be applied to IS-IS (or other IGP) as well as BGP? - Alain. On 7/20/08 8:46 AM, "Paul Francis" <francis@cs.cornell.edu> wrote:
Gang,
I have submitted an internet-draft to the IDR group on virtual aggregation (VA) (http://www.ietf.org/internet-drafts/draft-francis-idr-intra-va-00.txt). This draft suggests a few changes to routers that allow operators to control the size of their FIBs, shrinking them by 5x or 10x quite easily. This would extend the lifetime of routers that are constrained by FIB size.
There has been a lively discussion of this on the IDR mailing list, including a suggestion that FIB reduction is more important for lower tier ISPs (tier 2, tier 3...) than for tier 1 ISPs. Unfortunately I don't think that people from smaller ISPs pay much attention to the IDR mailing list, so they are not being represented in this discussion.
So I'm looking for input from folks who think that FIB reduction helps them, so as to better understand their requirements.
Any help is much appreciated.
Thanks,
PF
Certainly in principle it can, though that is not in the current proposal. The basic idea is to suppress installing routes into the FIB when there is a "virtual aggregate" that you can tunnel to instead. I remember we discussed this in San Jose NANOG, but I forget the details. Can you remind me? PF
-----Original Message----- From: Alain Durand [mailto:alain_durand@cable.comcast.com] Sent: Sunday, July 20, 2008 9:13 AM To: Paul Francis; nanog@nanog.org Subject: Re: virtual aggregation in IETF
Paul,
Can this proposal be applied to IS-IS (or other IGP) as well as BGP?
- Alain.
On 7/20/08 8:46 AM, "Paul Francis" <francis@cs.cornell.edu> wrote:
Gang,
I have submitted an internet-draft to the IDR group on virtual aggregation (VA) (http://www.ietf.org/internet-drafts/draft-francis-idr-intra-va- 00.txt). This draft suggests a few changes to routers that allow operators to control the size of their FIBs, shrinking them by 5x or 10x quite easily. This would extend the lifetime of routers that are constrained by FIB size.
There has been a lively discussion of this on the IDR mailing list, including a suggestion that FIB reduction is more important for lower tier ISPs (tier 2, tier 3...) than for tier 1 ISPs. Unfortunately I don't think that people from smaller ISPs pay much attention to the IDR mailing list, so they are not being represented in this discussion.
So I'm looking for input from folks who think that FIB reduction helps them, so as to better understand their requirements.
Any help is much appreciated.
Thanks,
PF
Paul Francis wrote:
Gang,
I have submitted an internet-draft to the IDR group on virtual aggregation (VA) (http://www.ietf.org/internet-drafts/draft-francis-idr-intra-va-00.txt). This draft suggests a few changes to routers that allow operators to control the size of their FIBs, shrinking them by 5x or 10x quite easily. This would extend the lifetime of routers that are constrained by FIB size.
There has been a lively discussion of this on the IDR mailing list, including a suggestion that FIB reduction is more important for lower tier ISPs (tier 2, tier 3...) than for tier 1 ISPs. Unfortunately I don't think that people from smaller ISPs pay much attention to the IDR mailing list, so they are not being represented in this discussion.
Software switched routers have little pressure on fib limitions. For a certain class of application the software switched edge router is in a much better position to accommodate fib growth than a device with a fixed sized cam. If you're in the business of shopping for sup-2s on ebay you'll probably see significant benefits in fib reduction.
So I'm looking for input from folks who think that FIB reduction helps them, so as to better understand their requirements.
Any help is much appreciated.
Thanks,
PF
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Software switched routers have little pressure on fib limitions. For a certain class of application the software switched edge router is in a much better position to accommodate fib growth than a device with a fixed sized cam.
I dunno about that; there's some papers floating about which look at trie type FIB representations which note significant savings in compressing FIB to unique entry set. Less memory, less comparsions, less nodes, etc. Rather interesting stuff. Try http://www.academypublisher.com/jnw/vol02/no03/jnw02031827.pdf for fun. Adrian
Adrian Chadd wrote:
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Software switched routers have little pressure on fib limitions. For a certain class of application the software switched edge router is in a much better position to accommodate fib growth than a device with a fixed sized cam.
I dunno about that; there's some papers floating about which look at trie type FIB representations which note significant savings in compressing FIB to unique entry set. Less memory, less comparsions, less nodes, etc. Rather interesting stuff.
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first?
Try http://www.academypublisher.com/jnw/vol02/no03/jnw02031827.pdf for fun.
Adrian
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first?
Well, the actual data point you should look at is "160k odd FIB from a couple years ago can fit in under 2 megabytes of memory." The random fetch time for dynamic RAM is pretty shocking compared to L2 cache access time, and you probably want to arrange your FIB to play well with your cache. Its nice that the higher end CPUs have megabytes and megabytes of L2 cache but placing a high-end Xeon on each of your interface processors is probably asking a lot. So there's still room for optimising for sensibly-specced hardware. Of course, -my- applied CPU-cache clue comes from the act of parsing HTTP requests/ replies, not from building FIBs. I'm just going off the papers I've read on the subject. :) Adrian
Adrian Chadd wrote:
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first?
Well, the actual data point you should look at is "160k odd FIB from a couple years ago can fit in under 2 megabytes of memory."
The random fetch time for dynamic RAM is pretty shocking compared to L2 cache access time, and you probably want to arrange your FIB to play well with your cache.
Its nice that the higher end CPUs have megabytes and megabytes of L2 cache but placing a high-end Xeon on each of your interface processors is probably asking a lot. So there's still room for optimising for sensibly-specced hardware.
If you're putting it on a line card it's probably more like a RAZA XLR, more memory bandwith and less cpu relative to the say the intel arch approach. That said I think you're headed to high end again. It has been routinetly posited that fib growth hurts the people on the edge more than in the center. I don't buy that for the reason outlined in my original response. If my pps requirements are moderate my software router can carry a fib of effectively arbitrary size at a lower cost than carrying the same fib in cam.
Of course, -my- applied CPU-cache clue comes from the act of parsing HTTP requests/ replies, not from building FIBs. I'm just going off the papers I've read on the subject. :)
Adrian
So, if I get you right, you are saying that edge routers have fewer CPU requirements, and so ISPs can get away with software routers and don't care about FIB. At the same time, folks in the middle are saying that in any event they need to buy high-end routers, and so can afford to buy enough hardware FIB so they also don't care (much) about FIB growth. Are there any folks for whom this statement isn't working? PF
-----Original Message----- From: Joel Jaeggli [mailto:joelja@bogus.com] Sent: Sunday, July 20, 2008 1:02 PM To: Adrian Chadd Cc: nanog@nanog.org Subject: Re: virtual aggregation in IETF
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first?
Well, the actual data point you should look at is "160k odd FIB from a couple years ago can fit in under 2 megabytes of memory."
The random fetch time for dynamic RAM is pretty shocking compared to L2 cache access time, and you probably want to arrange your FIB to play well with your cache.
Its nice that the higher end CPUs have megabytes and megabytes of L2 cache but placing a high-end Xeon on each of your interface processors is
Adrian Chadd wrote: probably
asking a lot. So there's still room for optimising for sensibly- specced hardware.
If you're putting it on a line card it's probably more like a RAZA XLR, more memory bandwith and less cpu relative to the say the intel arch approach.
That said I think you're headed to high end again. It has been routinetly posited that fib growth hurts the people on the edge more than in the center. I don't buy that for the reason outlined in my original response. If my pps requirements are moderate my software router can carry a fib of effectively arbitrary size at a lower cost than carrying the same fib in cam.
Of course, -my- applied CPU-cache clue comes from the act of parsing HTTP requests/ replies, not from building FIBs. I'm just going off the papers I've read on the subject. :)
Adrian
Paul Francis wrote:
So, if I get you right, you are saying that edge routers have fewer CPU requirements, and so ISPs can get away with software routers and don't care about FIB.
"ISP's that can get away with software routers" Also multihomed edge networks, the costs associated with multihoming aren't evenly distributed. The entities most likely to get squeezed are in the middle of the echosystem.
At the same time, folks in the middle are saying that in any event they need to buy high-end routers, and so can afford to buy enough hardware FIB so they also don't care (much) about FIB growth.
They care, but you weren't buying 2 million entry cam routers a year ago to deal with the growth of the DFZ. They bought them because they needed or would need a million or more internal routes fairly shortly, which says a lot about the complexity of a large isp. Assuming the dfz growth continues to fit the curve it's pretty easy to figure out when you need enough fib to support 500k dfz entries just as it was when we did the fib bof at nanog 39... http://www.cidr-report.org/cgi-bin/plota?file=%2Fvar%2Fdata%2Fbgp%2Fas2.0%2Fbgp-active.txt&descr=Active+BGP+entries+%28FIB%29&ylabel=Active+BGP+entries+%28FIB%29&range=Year&StartDate=&EndDate=&yrange=Auto&ymin=&ymax=&Width=1&Height=1&with=Step&color=auto&logscale=linear That's not to say that fib compression is undesirable, that approach has real benefits extending the useful life of a given platform, but there's very little about the current situation that is unexpected, or intractable.
Are there any folks for whom this statement isn't working?
PF
-----Original Message----- From: Joel Jaeggli [mailto:joelja@bogus.com] Sent: Sunday, July 20, 2008 1:02 PM To: Adrian Chadd Cc: nanog@nanog.org Subject: Re: virtual aggregation in IETF
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first? Well, the actual data point you should look at is "160k odd FIB from a couple years ago can fit in under 2 megabytes of memory."
The random fetch time for dynamic RAM is pretty shocking compared to L2 cache access time, and you probably want to arrange your FIB to play well with your cache.
Its nice that the higher end CPUs have megabytes and megabytes of L2 cache but placing a high-end Xeon on each of your interface processors is
Adrian Chadd wrote: probably
asking a lot. So there's still room for optimising for sensibly- specced hardware. If you're putting it on a line card it's probably more like a RAZA XLR, more memory bandwith and less cpu relative to the say the intel arch approach.
That said I think you're headed to high end again. It has been routinetly posited that fib growth hurts the people on the edge more than in the center. I don't buy that for the reason outlined in my original response. If my pps requirements are moderate my software router can carry a fib of effectively arbitrary size at a lower cost than carrying the same fib in cam.
Of course, -my- applied CPU-cache clue comes from the act of parsing HTTP requests/ replies, not from building FIBs. I'm just going off the papers I've read on the subject. :)
Adrian
How's this for a summary of what you are saying: Some ISPs can get away with software routers, most can't. Of those that can't, planning ahead by vendors and ISPs has allowed most ISPs to manage reasonably with well FIB growth. Nevertheless, some protocol tools to help manage FIB growth would be welcome by many. PF
-----Original Message----- From: Joel Jaeggli [mailto:joelja@bogus.com] Sent: Sunday, July 20, 2008 2:23 PM To: Paul Francis Cc: Adrian Chadd; nanog@nanog.org Subject: Re: virtual aggregation in IETF
Paul Francis wrote:
So, if I get you right, you are saying that edge routers have fewer CPU requirements, and so ISPs can get away with software routers and don't care about FIB.
"ISP's that can get away with software routers" Also multihomed edge networks, the costs associated with multihoming aren't evenly distributed. The entities most likely to get squeezed are in the middle of the echosystem.
At the same time, folks in the middle are saying that in any event they need to buy high-end routers, and so can afford to buy enough hardware FIB so they also don't care (much) about FIB growth.
They care, but you weren't buying 2 million entry cam routers a year ago to deal with the growth of the DFZ. They bought them because they needed or would need a million or more internal routes fairly shortly, which says a lot about the complexity of a large isp. Assuming the dfz growth continues to fit the curve it's pretty easy to figure out when you need enough fib to support 500k dfz entries just as it was when we did the fib bof at nanog 39...
http://www.cidr-report.org/cgi- bin/plota?file=%2Fvar%2Fdata%2Fbgp%2Fas2.0%2Fbgp- active.txt&descr=Active+BGP+entries+%28FIB%29&ylabel=Active+BGP+entries +%28FIB%29&range=Year&StartDate=&EndDate=&yrange=Auto&ymin=&ymax=&Width =1&Height=1&with=Step&color=auto&logscale=linear
That's not to say that fib compression is undesirable, that approach has real benefits extending the useful life of a given platform, but there's very little about the current situation that is unexpected, or intractable.
Are there any folks for whom this statement isn't working?
PF
-----Original Message----- From: Joel Jaeggli [mailto:joelja@bogus.com] Sent: Sunday, July 20, 2008 1:02 PM To: Adrian Chadd Cc: nanog@nanog.org Subject: Re: virtual aggregation in IETF
On Sun, Jul 20, 2008, Joel Jaeggli wrote:
Not saying that they couldn't benefit from it, however on one hand we have a device with a 36Mbit cam on the other, one with 2GB of ram, which one fills up first? Well, the actual data point you should look at is "160k odd FIB from a couple years ago can fit in under 2 megabytes of memory."
The random fetch time for dynamic RAM is pretty shocking compared to L2 cache access time, and you probably want to arrange your FIB to
your cache.
Its nice that the higher end CPUs have megabytes and megabytes of L2 cache but placing a high-end Xeon on each of your interface processors is
Adrian Chadd wrote: play well with probably
asking a lot. So there's still room for optimising for sensibly- specced hardware. If you're putting it on a line card it's probably more like a RAZA XLR, more memory bandwith and less cpu relative to the say the intel arch approach.
That said I think you're headed to high end again. It has been routinetly posited that fib growth hurts the people on the edge more than in the center. I don't buy that for the reason outlined in my original response. If my pps requirements are moderate my software router can carry a fib of effectively arbitrary size at a lower cost than carrying the same fib in cam.
Of course, -my- applied CPU-cache clue comes from the act of parsing HTTP requests/ replies, not from building FIBs. I'm just going off the papers I've read on the subject. :)
Adrian
Joel Jaeggli <joelja@bogus.com> writes:
That said I think you're headed to high end again. It has been routinetly posited that fib growth hurts the people on the edge more than in the center. I don't buy that for the reason outlined in my original response. If my pps requirements are moderate my software router can carry a fib of effectively arbitrary size at a lower cost than carrying the same fib in cam.
While that's true, planned obsolescence in terms of ram non-expandability from certain <cough> vendors leaves people on the edge (who often have to scrape by on thin budgets) hurting badly enough as to offset any theoretical software-vs-hardware router advantage they may have originally had. -r
participants (5)
-
Adrian Chadd -
Alain Durand -
Joel Jaeggli -
Paul Francis -
Robert E. Seastrom