In a message written on Mon, Oct 12, 2009 at 05:09:41PM -0700, Owen DeLong wrote:
With IPv6, it probably won't be the ideal 1:1 ratio, but, it will come much closer. Even if the average drops to 1/2, you're talking about a 70,000 route table today, and, likely growth in the 250-300,000 route range over the next 5-10 years. CAM will probably scale faster than that.
Here's a presentation from 2007. http://www.vaf.net/~vaf/apricot-plenary.pdf On page 13, you'll find a table. It starts with numbers in November of 2006, and makes projections. The 5 year projections (Nov 2011) have already been exceeded, in both IPv4 Internet Routes and Active ASN's. The problem isn't that we have 300,000 "global routes" on the Internet (http://www.cidr-report.org/as2.0/#General_Status), but that there are other things that compete for TCAM space. It's that TCAM must hold not only the global routes, but also: - Internal routes. Your IGP routes, no-exported customer deagregations, blackhole routes, etc. - MPLS Labels, including: - MPLS Traffic Engineering - MPLS VPN Identifiers - Virtual Routing Instances for Layer 3 VPN's. - ARP Entries - Multicast Routes Unfortunately details are hard to come by as most of the folks who see this in any significant way (e.g. global "tier 1" full service ISP's) tend not to release too many specific numbers for competitive reasons. That said, even using some basic assumptions (some of which are in the preso) those 300,000 global routes might have added to them: 300,000 global routes 150,000 internal routes 20,000 MPLS labels 200,000 VPN/VRF Routes 5,000 ARP Entries 20,000 Multicast Routes -------- 695,000 TCAM Entries Consumed That's today, right now, in major ISP's routers. All the sudden those "1 million route" core routers don't seem so large. Keep in mind we've passed the 2006 projection in this report in 3 years, not 5. So we're growing faster than we expected. Add in your 70,000 route IPv6 table, plus growth, and the 1 million route routers are probably failing sometime in 2011. Someone will likely pipe up, but Cisco has a 3 million route processor now! (I believe that is the spec of the just announced PRP3, but can't find a reference on Cisco's web site). Yes, that's a route processor that can do the job, but in these high end boxes the TCAM is distributed on the linecards. So upgrading from the 1 million route TCAM core routers to the 3 Million route TCAM means upgrading every linecard in each router you upgrade. Ouch. The picture I painted above is actually the rosy part of the picture. Many of these backbones have older equipment in the core which can't even do 1M routes. They use careful design and other techniques to limit the number of entries particular boxes have to see.
The problematic time scale is that time where we have to support dual stack for a majority of the network. That's what will really stress the CAM as the IPv6 table becomes meaningfully large (but not huge) and the IPv4 table cannot yet be retired.
While I think Verizon's move is somewhat premature, I can see why they might be afraid of routing table growth. I think there is an extremely high probability that given the growth of the table due to primarily to IPv6 and the growth of MPLS VPN offerings, combined with the current economic climate which has reduced the capital available for upgrades that we will see several providers "hit the wall" of various popular bits of equipment. I think some of the engineering staff at various major providers has already realized this as well. We don't seem to have a technological solution. LISP has scaling issues of its own, and would require swapping out a huge amount of equipment. TCAM scaling is at best cost prohibitive, at worst not possible due to the physical ram speed, and both are being improved at a modest rate (the preso suggests 10% per year). Worse, the problem is being made worse at an alarming rate. MPLS VPN's are quicky replacing frame relay, ATM, and leased line circuits adding MPLS lables and VPN/VRF routes to edge routers. Various RIR's are pushing "PI for all" in IPv6 based on addressing availbility. Some networks are actually finally using multicast for IPTV services, generating much larger number of entries than the global multicast table would otherwise indicate. The next 5 years may bring internet instability problems and route filtering on a scale we haven't seen since the early 90's. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/