Paul Wall wrote:
Once upon a time, vendors released products which relied on CPU-based "flow" setup. Certain vintages of Cisco, Extreme, Foundry, Riverstone, etc come to mind. These could forward at "line rate" under normal conditions. Sufficient randomization on the sources and/or destinations (DDoS, Windows worm, portscans, ...) and they'd die a spectacular death. Nowadays, this is less of a concern, as the higher-end boxes can program a full routing table (and then some) worth of prefixes in CAM.
Either way, I think it's a good test metric. I'd be interested in hearing of why you think that's not the case. Back on topic, doing a couple of gigs of traffic at line rate is a walk in the park for any modern product billed as a "layer 3 switch". The differentiator between, say, a Dell and a Cisco, is in the software and profoundly not the forwarding performance.
Without disagreeing with your main point, there are still plenty of ways to bring even very large boxes to near-zero forwarding rates: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a "high-speed" example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. 2. Use weak hashing algorithms. Some switches (names removed to protect the guilty) hash on the low-order three bits of MAC address to decide which of eight ASICs will forward a packet. I have heard of, but have not tested, devices that use only three bits for making OSPF ECMP decisions. Not surprisingly either technique can lead to lots of hash collisions in routed environments. 3. Offer IP multicast. Maximum forwarding rates for multicast are rather different than IP unicast with some vendors' implementations, and may be affected by group count, mroute count and amount of replication. dn
On Sun, Sep 07, 2008, David Newman wrote:
1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a "high-speed" example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed.
So what you're saying is "send the right crafted packets and DoS the internet", right? (I think I know which options may make routers go all software-path on the packets but I haven't given it a run on a Cat6500. Hm, I wonder if this here 3750 in the lab will do..) Adrian
On 9/7/08 8:49 AM, Adrian Chadd wrote:
On Sun, Sep 07, 2008, David Newman wrote:
1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a "high-speed" example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed.
So what you're saying is "send the right crafted packets and DoS the internet", right?
My experience *in lab testing* is that most and perhaps all switches do slow-path processing of v4 and v6 packets with IP options set, and that slow-path forwarding rates are a tiny fraction of fast-path forwarding rates. Christian Huitema made a similar observation in one of his textbooks 10 or more years ago; tests as recently as this year suggest this is still the case. I'm not making any assertions about DoS attacks on production networks. Rate controls and other mechanisms can help mitigate the effects of flooding attacks, but that's a different topic.
(I think I know which options may make routers go all software-path on the packets but I haven't given it a run on a Cat6500. Hm, I wonder if this here 3750 in the lab will do..)
The record route option will cause rather precipitous drops in forwarding rates on both boxes (and many others). I have not tried other option types, but other testers have told me these too will be slow-pathed. Again, from the ASIC/NP/FPGA's standpoint: Fixed-length, good. Variable-length, not so much... dn
I think it would be interesting to put a table of routing devices together along with the commands it takes to knock down their forwarding rates. And to find out what platform has the higher percentage drop in forwarding rate. As mentioned elsewhere, it's not the pps, but operations per second. Frank -----Original Message----- From: David Newman [mailto:dnewman@networktest.com] Sent: Sunday, September 07, 2008 9:23 AM To: nanog@nanog.org Subject: Re: Force10 Gear Paul Wall wrote:
Once upon a time, vendors released products which relied on CPU-based "flow" setup. Certain vintages of Cisco, Extreme, Foundry, Riverstone, etc come to mind. These could forward at "line rate" under normal conditions. Sufficient randomization on the sources and/or destinations (DDoS, Windows worm, portscans, ...) and they'd die a spectacular death. Nowadays, this is less of a concern, as the higher-end boxes can program a full routing table (and then some) worth of prefixes in CAM.
Either way, I think it's a good test metric. I'd be interested in hearing of why you think that's not the case. Back on topic, doing a couple of gigs of traffic at line rate is a walk in the park for any modern product billed as a "layer 3 switch". The differentiator between, say, a Dell and a Cisco, is in the software and profoundly not the forwarding performance.
Without disagreeing with your main point, there are still plenty of ways to bring even very large boxes to near-zero forwarding rates: 1. Set IP options. A pair of Cat 6509Es using VSS can forward packets without options at up to 770 mpps, but when packets have options the maximum is more like 20 kpps. And that's a "high-speed" example; the options forwarding rate is more like 0 pps with some other devices. Silicon that forwards packets very fast is only good when header lengths are fixed. 2. Use weak hashing algorithms. Some switches (names removed to protect the guilty) hash on the low-order three bits of MAC address to decide which of eight ASICs will forward a packet. I have heard of, but have not tested, devices that use only three bits for making OSPF ECMP decisions. Not surprisingly either technique can lead to lots of hash collisions in routed environments. 3. Offer IP multicast. Maximum forwarding rates for multicast are rather different than IP unicast with some vendors' implementations, and may be affected by group count, mroute count and amount of replication. dn
On 9/7/08, Frank Bulk <frnkblk@iname.com> wrote:
I think it would be interesting to put a table of routing devices together along with the commands it takes to knock down their forwarding rates. And to find out what platform has the higher percentage drop in forwarding rate.
As mentioned elsewhere, it's not the pps, but operations per second.
Send a 3kpps stream of multicast packets with TTL=1 towards a sup720 and you can watch it keel over and cry uncle. It really, really doesn't take much these days to kill high-end hardware; they're so optimized for a specific class of traffic that they handle well in hardware, as that's what the bulk of the normal traffic is, and that's what the marketing department needs to chase to keep up with the competition; any traffic profile outside of that doesn't get the same focus from the hardware forwarding teams because that's not where the pressure to keep up from the marketplace is coming from. *Nobody* goes out and says "I have $10M to spend on routers, but to qualify they must be able to forward 10Mpps of IPv4 packets with IP options enabled, sustained rate, with no loss". That's just not a driving market force right now. I think you would find that your table simply reflects what the *bulk* of the traffic profiles from major customers represent; those areas that cause the routers pain in terms of forwarding are exactly those traffic patterns that are *not* highly represented among the majority of the customer base. Matt
participants (4)
-
Adrian Chadd -
David Newman -
Frank Bulk -
Matthew Petach