Hello, I have several Catalyst 6500 (Supervisor 32) aggregation switches with WS-X6148A-GE-TX and WS-X6148-GE-TX line cards. These line cards do not support storm-control/broadcast suppression. This impacted us badly during a recent spanning tree event. As it stands, we are at risk of overwhelming control planes with excess broadcast or multicast traffic, and I need to find alternative ways to protect these switches. I have been researching STP enhancements, and control-plane policing in the following documents, and would appreciate advice from engineers who may have had to implement similar workarounds for storm-control in a service provider setting. * Configuring Denial of Service Protection http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configu... * Configuring Control Plane Policing http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/configu... * Configuring Optional STP Features http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/configu... So, if we can't mitigate against STP events using storm-control or broadcast suppression, what might be the best combination of STP enhancements and control-plane policing? For example, is it possible to rate-limit broadcast/multicast, STP and ARP on a per VLAN basis? If so, how? Many thanks, P -- Peter George Lumison t: 0845 1199 900 d: 0131 514 4022 P.S. Lumison have changed the way businesses communicate forever http://www.unified-communications.net/ ________________________________ -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. Any offers or quotation of service are subject to formal specification. Errors and omissions excepted. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Lumison. Finally, the recipient should check this email and any attachments for the presence of viruses. Lumison accept no liability for any damage caused by any virus transmitted by this email.
Peter, This question would be better directed at cisco-nsp, but... On 21/08/2009 11:39, Peter George wrote:
I have several Catalyst 6500 (Supervisor 32) aggregation switches with WS-X6148A-GE-TX and WS-X6148-GE-TX line cards.
These line cards do not support storm-control/broadcast suppression. This impacted us badly during a recent spanning tree event.
Not surprised. The 61xx cards are not service provider suitable line cards and they have proved this very clearly. Sorry to hear about these storms - they really are devastating, aren't they? But if you're running L2 customer facing services, particularly shared L2 domain access, there are two things you care about: storm control and port security (mac address counting). The 61xx cards don't do storm control.
For example, is it possible to rate-limit broadcast/multicast, STP and ARP on a per VLAN basis? If so, how?
Yes, you replace your 61xx cards with 67xx cards. You can't do this sort of thing with qos or copp. Nick
On Aug 21, 2009, at 10:23 PM, Nick Hilliard wrote:
there are two things you care about: storm control and port security (mac address counting).
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625
Roland Dobbins wrote:
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well.
Or you hit the extreme DSL concentrator end where you crank out q-in-q with roughly 1 vlan per customer (some equipment perhaps handling 1 to many with other built in security features) and let the router proxyarp between them. Unnumbered vlans and RBE saved parts of my network from pending doom. Even fixed issues with dslams that overran the arp caches causing unicast broadcast storms, but the arp cache was irrelevant when it was 1 vlan per port. I'm still waiting for other vendors to tell me how they can match that particular Cisco functionality. Jack
On 21/08/2009 16:39, Roland Dobbins wrote:
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well.
Unless your broadcast storm happens on an untagged vlan. Or unless you're running VTP and also have ipv6 hosts connected to the switch on .1q tagged ports, and consequently have to disable VTP pruning in order to get said ipv6 .1q hosts to be able to talk to each other, and then if you have a broadcast storm on any vlan, it could hose your entire l2 network, because you've disabled vtp pruning. Anyway, the point is: storm control on customer facing equipment is a basic requirement. Nick
On Aug 21, 2009, at 10:57 PM, Nick Hilliard wrote:
Or unless you're running VTP
Yes, but this is evil and dangerous in a customer-facing environment; transparent mode is the preferred option, in most circumstances. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625
On 21/08/2009 17:04, Roland Dobbins wrote:
Yes, but this is evil and dangerous in a customer-facing environment; transparent mode is the preferred option, in most circumstances.
It is very evil, yes. SXH and later support VTPv3 which allows you to disable VTP on a per port basis. But as you say, not suitable for customer facing networks. Nick
On Fri, 21 Aug 2009, Roland Dobbins wrote:
there are two things you care about: storm control and port security (mac address counting).
Chopping up the layer-2 broadcast domain for a given VLAN into smaller pieces via pVLANs can't hurt, either, as long as the hosts have no need to talk to one another - and it has other benefits, as well.
I understand why hosts need to send broadcasts. In a close/single customer environment, broadcasts could be useful. I hope most future protocol designers now think of using multicast or other discovery mechanisms besides broadcast. But in a service provider network (or any managed network), is there any reason why a customer needs to hear other customer's broadcasts? In practice, are there any useful broadcast messages in a multi-customer environment that can't/shouldn't be proxied by the network operator or handled other ways.
On Sat, 22 Aug 2009, Sean Donelan wrote:
But in a service provider network (or any managed network), is there any reason why a customer needs to hear other customer's broadcasts? In practice, are there any useful broadcast messages in a multi-customer environment that can't/shouldn't be proxied by the network operator or handled other ways.
Not that I know of, ISPs have successfully done L2 isolation of customers for 10 years and I haven't heard of any problems with it. Only bad part really is that if the customer is allowed several IPs and you use local-proxy-arp then traffic between customer computers will go via the ISP, which is one of the reasons I advocate the use of a home CPE router for IPv6, it's just a cleaner handoff. -- Mikael Abrahamsson email: swmike@swm.pp.se
On 22/08/2009 06:26, Andrew Parnell wrote:
The 67xx series cards aren't supported by the sup32, though. Would 65xx line cards do the trick?
unfortunately not:
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native...
• The following LAN switching modules do not support traffic storm control: – WS-X6148A-GE-45AF – WS-X6148A-GE-TX – WS-X6148-GE-45AF – WS-X6148-GE-TX – WS-X6148V-GE-TX – WS-X6548-GE-45AF – WS-X6548-GE-TX – WS-X6548V-GE-TX
Hmmm, expensive proposition to upgrade then - even though sup720 + ws-x67xx cards make a nice l2 platform for gig ethernet. Nick
In my opinion the Sup32 platform has some limitations when the technology is considered for high data rate, deterministic carrier customer-facing scenarios. Cisco sells the Sup32 as a wiring closet aggregation switch the main purpose of which is to connect desktop users to central core switches. In addition to the lack of storm-control/broadcast suppression mentioned below, the 61XX line cards also have a limit of, I believe, 2 ports in an Etherchannel. Additionally, and perhaps most significantly for deterministic network design, the copper cards share input hardware buffers for every 8 ports. Running one port of the 8 at wire speed will cause input drops on the other 7 ports. Also, the cards connect to the older 32 Gbps shared bus. In my view, with high data rates, it is difficult, if not impossible, to build a deterministic network with Sup32s. Cisco's solution for designing a deterministic network is the sup720 which has a 720 Gbps crossbar bus. The 67XX 48 port copper line cards have 2 20 Gbps connectors to the 720 Gbps bus, the 24 port fiber sfp line cards have a single 20 Gbps connector to the crossbar bus, and the 10 GiG 67XX line cards have 2 20 Gbps bus connectors. The crossbar bus connects line card connectors to each other in a point-to-point fashion. 67XX line cards are adequately provisioned with input and output buffers. There is still 40/48 (1 GiGE copper), 20/24 (1 GiGE sfp), and 40/160 (10 GiGE X2) oversubscription of ports to switch fabric connectors, however. Sup720 routing table lookups via Ternary Content Addressable Memory (TCAM) still use the 32 Gbps bus to access the TCAM to search for next hop for destination IP network. -----Original Message----- From: Peter George [mailto:Peter.George@lumison.net] Sent: Friday, August 21, 2009 3:40 AM To: nanog@nanog.org Subject: Alternatives to storm-control on Cat 6509. Hello, I have several Catalyst 6500 (Supervisor 32) aggregation switches with WS-X6148A-GE-TX and WS-X6148-GE-TX line cards. These line cards do not support storm-control/broadcast suppression. This impacted us badly during a recent spanning tree event. As it stands, we are at risk of overwhelming control planes with excess broadcast or multicast traffic, and I need to find alternative ways to protect these switches. I have been researching STP enhancements, and control-plane policing in the following documents, and would appreciate advice from engineers who may have had to implement similar workarounds for storm-control in a service provider setting. * Configuring Denial of Service Protection http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/con figuration/guide/dos.pdf * Configuring Control Plane Policing http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/con figuration/guide/cntl_pln.pdf * Configuring Optional STP Features http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/12.2/31sga/con figuration/guide/stp_enha.pdf So, if we can't mitigate against STP events using storm-control or broadcast suppression, what might be the best combination of STP enhancements and control-plane policing? For example, is it possible to rate-limit broadcast/multicast, STP and ARP on a per VLAN basis? If so, how? Many thanks, P -- Peter George Lumison t: 0845 1199 900 d: 0131 514 4022 P.S. Lumison have changed the way businesses communicate forever http://www.unified-communications.net/ ________________________________ -- This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender. Any offers or quotation of service are subject to formal specification. Errors and omissions excepted. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of Lumison. Finally, the recipient should check this email and any attachments for the presence of viruses. Lumison accept no liability for any damage caused by any virus transmitted by this email.
On Aug 25, 2009, at 1:03 AM, Holmes,David A wrote:
In my opinion the Sup32 platform has some limitations when the technology is considered for high data rate, deterministic carrier customer-facing scenarios
The hardware-based FPM is interesting. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sorry, sometimes I mistake your existential crises for technical insights. -- xkcd #625
On 24/08/2009 19:03, Holmes,David A wrote:
Additionally, and perhaps most significantly for deterministic network design, the copper cards share input hardware buffers for every 8 ports. Running one port of the 8 at wire speed will cause input drops on the other 7 ports. Also, the cards connect to the older 32 Gbps shared bus.
IMO, a more serious problem with the 6148tx and 6548tx cards is the internal architecture, which is effectively six internal managed gigabit ethernet hubs (i.e. shared bus) with a 1M buffer per hub, and each hub connected with a single 1G uplink to a 32 gig backplane. Ref:
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note0918...
In Cisco's own words: "These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity". In other words, these cards are fine in their place, but they are not designed or suitable for data centre usage. I don't want to sound like I'm damning this card beyond redemption - it has a useful place in this world - but at the expense of reliability, manageability and configuration control, you will get useful features (including broadcast/unicast flood control) and in many situations very significantly better performance from a recent SRW 48-port linksys gig switch than from one of these cards. Nick
On Mon, Aug 24, 2009 at 4:59 PM, Nick Hilliard <nick@foobar.org> wrote:
On 24/08/2009 19:03, Holmes,David A wrote:
Additionally, and perhaps most significantly for deterministic network design, the copper cards share input hardware buffers for every 8 ports. Running one port of the 8 at wire speed will cause input drops on the other 7 ports. Also, the cards connect to the older 32 Gbps shared bus.
IMO, a more serious problem with the 6148tx and 6548tx cards is the internal architecture, which is effectively six internal managed gigabit ethernet hubs (i.e. shared bus) with a 1M buffer per hub, and each hub connected with a single 1G uplink to a 32 gig backplane. Ref:
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note0918...
In Cisco's own words: "These line cards are oversubscription cards that are designed to extend gigabit to the desktop and might not be ideal for server farm connectivity". In other words, these cards are fine in their place, but they are not designed or suitable for data centre usage.
I don't want to sound like I'm damning this card beyond redemption - it has a useful place in this world - but at the expense of reliability, manageability and configuration control, you will get useful features (including broadcast/unicast flood control) and in many situations very significantly better performance from a recent SRW 48-port linksys gig switch than from one of these cards.
Nick
We experienced the joy of using the X6148 cards with a SAN/ESX cluster. Lots of performance issues! A fairly inexpensive solution was to switch to the X6148A card instead, which does not suffer the the 8:1 oversubscription. It also supports MTU's larger than 1500, which was another shortcoming of the older card. Mike -- Mike Bartz mob@bartzfamily.net
On 26/08/2009, at 6:21 AM, Mike Bartz wrote:
We experienced the joy of using the X6148 cards with a SAN/ESX cluster. Lots of performance issues! A fairly inexpensive solution was to switch to the X6148A card instead, which does not suffer the the 8:1 oversubscription. It also supports MTU's larger than 1500, which was another shortcoming of the older card.
Actually, the "A" variant of the x6148 is still 8:1 oversubscribed. The significant difference between the x6148 and x6148a is the buffer size. The original card had 1.4MB of buffer per port group (8 ports) while the "A" upgrade supports 5.5MB per port. Oh, that and support for 9k jumbo frames. It's still a classic bus card, it still has the same QoS queues, and is still 8:1 over subscribed. David ...
participants (10)
-
Andrew Parnell -
David Hughes -
Holmes,David A -
Jack Bates -
Mikael Abrahamsson -
Mike Bartz -
Nick Hilliard -
Peter George -
Roland Dobbins -
Sean Donelan