I’m trying to run netflow on one of our Cisco core routers (SUP720-3BXL), but I think I am hitting some limitations because of this: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM Utilization [99%] The setup of netflow looks like this: ip flow-cache entries 524288 mls aging fast time 5 threshold 32 mls aging long 300 mls aging normal 60 mls netflow usage notify 80 300 mls flow ip full no mls flow ipv6 mls nde sender version 5 no mls verify ip checksum no mls acl tcam share-global ip flow-export source Loopback0 ip flow-export version 5 origin-as ip flow-export destination <ip> <port> Then I have this enabled on all border interfaces/vlans (peering / transit / other core routers) that are of interest for my stats: ip route-cache flow Some more details about the problem: #sh mls netflow table-contention detailed Earl in Module 5 Detailed Netflow CAM (TCAM and ICAM) Utilization ================================================ TCAM Utilization : 100% ICAM Utilization : 13% Netflow TCAM count : 262033 Netflow ICAM count : 17 Netflow Creation Failures : 4822220 Netflow CAM aliases : 1 #sh mls netflow table-contention aggregate Earl in Module 5 Aggregate Netflow CAM Contention Information ============================================= Netflow Creation Failures : 130003616 Netflow Hash Aliases : 4 I understand that the TCAM is full, but what can I do against it? This is a busy core router: Aggregated traffic: 7-8 GBIT/s Packets per Second: 1.0 - 1.2 Million I wouldn't mind analyzing only every 10th or 100th flow, which seems to be a common practice. Any good piece of advice is welcome. Thanks! - Andy
On 15/03/2009 01:55, Andy Bierlair wrote:
I’m trying to run netflow on one of our Cisco core routers (SUP720-3BXL), but I think I am hitting some limitations because of this:
Sounds about right for the amount of traffic you're pushing through the box. The SUP720 is a very poor netflow platform. There has been extensive discussion about this problem in cisco-nsp over the past several years, and this posting is probably more appropriate to that mailing list. But basically, there is too little netflow tcam on this card to deal with anything more than a couple of gigs of traffic. You can help things by setting the aging timer to be very aggressive, and by getting DFCs (although these are a rather expensive option). Sampling won't generally help, as the sampling is done in software, after the data has been collected. More info on:
http://www.google.com/search?q=sup720+netflow+%2Bsite:puck.nether.net/piperm...
Nick
On Sun, 15 Mar 2009, Andy Bierlair wrote:
Im trying to run netflow on one of our Cisco core routers (SUP720-3BXL), but I think I am hitting some limitations because of this:
%EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM Utilization [99%]
TCAM Utilization : 100%
Aggregated traffic: 7-8 GBIT/s
Packets per Second: 1.0 - 1.2 Million
AFAIK, at that traffic level, you will have to do sampled netflow. Try mls sampling time-based 64 [in global] mls netflow sampling [in interface] and see if that stops your TCAM utilization issues. You may have to sample even less flow data. ---------------------------------------------------------------------- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
This is, believe it or not, a feature of the device you are using. On Sun, March 15, 2009 01:55, Andy Bierlair wrote:
Im trying to run netflow on one of our Cisco core routers (SUP720 3BXL), but I think I am hitting some limitations because of this:
%EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM Utilization [99%]
-- Neil J. McRae -- Alive and Kicking. neil@DOMINO.ORG
participants (4)
-
Andy Bierlair -
Jon Lewis -
Neil J. McRae -
Nick Hilliard