Chris:

I've been thinking about leasing some dark fiber and running one of the new 10gigE blades for the Cat 6500 chassis.

Be careful here. Last I tested (at one of our channels that also resells Cisco) is that the 10GbE on the Catalyst 6500 hasn't broken 4G throughput yet. Sort of like buying a GbE interface for a 7200 (It only get's 10% throughput... Why waste the money, just buy FE!). The GSR is up to about 8G throughput nowadays from what I've seen. Foundry Networks (my company) can get a perfect clean 8G throughput on all of our chassis with management modules M2 or above (we don't support 10GbE on the legacy M1). Our NG chassis will be available later in the year for those folks that want 4 X 10 GbE on each module (8 slot chassis). I expect this will be a perfect 40G throughput since I've never seen us do anything less than perfect (been working here since August). Additionally, you would be the first customer I've heard about doing standards based 10GbE on a Catalyst. (feel free to chime in if you're doing this... Can I bring my SmartBits 600 to your site to test throughput?). Good luck! Foundry has a few references: Deployed: http://www.foundrynet.com/about/newsevents/releases/pr4_3_02.html http://www.foundrynet.com/about/newsevents/releases/pr4_2_02.html http://www.foundrynet.com/about/newsevents/releases/pr2_11_02.html Many others that we don't press release. We've got these blades running in production networks here in Japan that I'm not allowed to talk about. Also many other places. Deploying: http://www.foundrynet.com/about/newsevents/releases/pr5_8_02.html Performance: http://www.spirentcom.com/news/press.cfm?id=87

Throw in the Cisco "Flamethrower" GBIC and I should be good for 50 miles. Has anyone tried this?

Foundry Network's Long Haul (LHB: 150 km, LHA: 70 km) Ethernet optics exceed Cisco's on GbE (ZX: 100 km). I'm sure we exceed them on the ER LAN PHY for 10GbE. We've only tested to 85 kilometers (ER). 802.3ae standard is 40 km: http://biz.yahoo.com/prnews/020508/nyw068_1.html Cisco's website says they can do the 802.3ae standard 40 km on the 1550 nm blade. I'm not sure if the optics are changeable either: http://www.cisco.com/warp/public/cc/pd/ifaa/6500ggml/ I doubt if there is a GBIC for 10GbE available. We use the same blade with changeable optics; however, I would not call the SR (300 meters), LR (10 km), and ER LAN PHY optics GBIC's... Moral of this story is that BEFORE you buy these blades from Cisco (or anybody), test them! If you don't have 10GbE SmartBits or IXIA, you can use 1GbE interfaces and wrap them around until you get 8G (no need to produced anything higher 'cause the Cat 6500 has an 8G throughput limitation). Don't test latency with this method :-). I don't believe the marketing from any company, not even my own. I test, then tell. I've personally never seen a packet drop at a steady 8G rate for up to 72 hours; however, one of our customers evaluating the 10GbE blades reported 2 64 byte packet's were dropped in a 12 hour line rate test. I suspect they had bad fiber. Gary Blankenship Systems Engineer Foundry Networks

On Mon, May 20, 2002 at 10:33:32PM -0600, Chance Whaley wrote:

Oh phuleeese.... Stop drinking your own Kool-Aid(tm). To honestly suggest that Foundry, or any other vendor for that matter, never does 'anything less than perfect' is nothing less than idiotic. If Foundry does things so 'perfect' why do they have a TAC? Why do they have bugs? Why do they even need to release new software ever again? Obviously what is out now will solve every possible issue - its 'perfect' right? The only possible answer according to your logic, is to support customers who are 'doing it wrong' and need to be educated.

Personally I would say that Foundry does EVERYTHING less than perfect. Nearly everyone I'm aware of (including myself) who has had to misfortune to try and use their devices in a service provider environment and a layer 3 role has come away with a universal loathing of biblical proportions. I really can't stress this enough, it DOES NOT MATTER how many gigabits your box forwards. A router is ONLY as useful as the quality of its software and support, if you can't login to it or have working routing protocols, it's just a big paperweight. The only "wannabe cisco" company I have seen learn this lesson is Juniper, and I am firmly convinced this is the reason for their success in the core. Whenever I read a press release about Foundry in the core, I stop and take a moment to laugh uncontrollably. It has nothing to do with ISIS or MPLS, it has to do with making your existing functionality work correctly and behave in a sensible fashion. Nothing personal against Foundry, but the people in charge couldn't possibly "not get it" any more than they do now. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

I recall that, early in my career I had the opportunity to build a new LAN backbone for a 6 story office building. It was going to be Category 5! Woohoo. With a 12/24 fiber backbone. ATM in a LAN environment was new at the time but I was going to make sure I had an OC3 backhauling each of the floors to a central switch. I thought this design was beautiful and marvelous. There was a neat new company that made LAN-style ATM gear with performance specs that would just blow your mind. So when I took the design to the board they loved the fastethernet fiber blah blah and gave approval. But when it came down to selecting vendors for the hardware I ran right into a brick wall with questions like: How long has this company been in business? Are they using open standards? Do they have knowldgeable tech support? ..and so on. So, regardless of whether the hardware is the fastest thing on the block, pushing 10 nanobits at a megaflop, you can look like a fool if you don't consider the business repercussions of the vendor you choose. In the end, I didn't get my design approved until I chose Cisco. Was I pissed, sure! Did I ship off white papers and other propaganda to support my case? Yes! But the company went bankrupt about 2 weeks after I submitted the bid. Just my .02, Regards, Christopher J. Wolff, VP CIO Broadband Laboratories http://www.bblabs.com -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Gary Sent: Tuesday, May 21, 2002 12:37 AM To: Richard A Steenbergen Cc: nanog@merit.edu Subject: RE: The market must be coming back Richard:

Personally I would say that Foundry does EVERYTHING less than perfect.

Nearly everyone I'm aware of (including myself) who has had to misfortune to try and use their devices in a service provider environment and a layer 3 role has come away with a universal loathing

of biblical proportions.

Not worth a response. Can't please everybody and you CAN'T design everyone's network for them. Sort of like EIGRP. Even the worst network engineer can look great with it. Perhaps you should read JANOG. Maybe they can help you. Search for フアウンドリ。 (note, if you cannot read this, it is Japanese for Foundry in unicode).

I really can't stress this enough, it DOES NOT MATTER how many gigabits your box forwards. A router is ONLY as useful as the quality of its software and support, if you can't login to it or have working routing protocols, it's just a big paperweight. The only "wannabe cisco" company I have seen learn this lesson is Juniper, and I am firmly convinced this is the reason for their success in the core.

Juniper is an OUSTANDING company. Much better than many networking companies in many respects. I've also heard nothing but good things about Unisphere here in Japan, so perhaps this will be a good marriage with benefits to service providers. I'll enjoy competing. We will compete.

Whenever I read a press release about Foundry in the core, I stop and take a moment to laugh uncontrollably. It has nothing to do with ISIS or MPLS, it has to do with making your existing functionality work correctly and behave in a sensible fashion. Nothing personal against Foundry, but the people in charge couldn't possibly "not get it" any more than they do now.

Remember what you said in this paragraph. I will refer to it later. Yoroshiku, Gary

Good point! The other one is "Choose your battles wisely." -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Patrick Sent: Tuesday, May 21, 2002 9:52 PM To: Christopher J. Wolff Cc: nanog@merit.edu Subject: Re: The business side of the coin. WAS RE: The market must be coming back On Tue, 21 May 2002, Christopher J. Wolff wrote:

So, regardless of whether the hardware is the fastest thing on the block, pushing 10 nanobits at a megaflop, you can look like a fool if you don't consider the business repercussions of the vendor you choose. In the end, I didn't get my design approved until I chose Cisco. Was I pissed, sure! Did I ship off white papers and other propaganda to support my case? Yes! But the company went bankrupt about 2 weeks after I submitted the bid.

"No one gets fired for buying IBM." /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ /\/\/\ Patrick Greenwell Asking the wrong questions is the leading cause of wrong answers \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ \/\/\/

Chance:

...

that want 4 X 10 GbE on each module (8 slot chassis). I expect this will be a perfect 40G throughput since I've never seen us do anything less than perfect (been working here since August).

Oh phuleeese.... Stop drinking your own Kool-Aid(tm). To honestly suggest that Foundry, or any other vendor for that matter, never does 'anything less than perfect' is nothing less than idiotic. If Foundry does things so 'perfect' why do they have a TAC? Why do they have bugs? Why do they even need to release new software ever again? Obviously what is out now will solve every possible issue - its 'perfect' right? The only possible answer according to your logic, is to support customers who are 'doing it wrong' and need to be educated.

Topic is performance. Not sugary beverages. Sorry for not making that clear. Let me reword. My bad: "perfect performance on 10GbE". I believe I also mentioned our 8G per slot throughput limitation not to mislead people to think we do 10GbE non-blocking. Same limitation as the Cat6500 once it gets up to speed.

Go find the nice black shirts that were passed out at Foundry's last Kool-Aid fest. You are in obvious need of one. This is NOT the place to post vendor FUD. All you are doing is making Foundry look bad, and making yourself look even worse.

Didn't you pass out those shirts? Everything I posted concerning performance of 10GbE I saw for myself. All other information was publicly available and concerns operators interested in 10GbE. Many of them are unaware of their options and I wanted to bring Foundry to light. Reading NANOG you would think that the only way to spot Nimda would be NBAR and the only MPLS is Juniper. The post I replied to is a person considering 10GbE in a 6500. I've seen the performance on this at a customer site with SmartBits. The channel became a Foundry reseller because of this specific issue. Now the same configuration comes up on NANOG and I wanted the person thinking about the 6500/10GbE solution to be aware of what I saw. Perhaps the performance is faster than 4G today (My info is a month old). If I were to leave Foundry today (to make them look better) and work for another company (McDonalds?), I would have sent the same post (would you like fries with that?). You can't forget what you see. I have tested our 10GbE personally. Gary

Adam:

[...] Sort of like buying a GbE interface for a 7200 (It only get's

...

10% throughput... Why waste the money, just buy FE!).

How did the Foundry test lab arrive at those figures, and what substances were consumed at the time?

I used a Cisco 7200 VXR with NPE-400. I used two different 7200's with the exact same results. Bidirectional throughput on 1GbE is a fraction above 10%. Unidirectional is a bit better (23%). Singl line ACL drops it to 8% (permit ip any any). FE performance doesn't start to drop below line rate until you put more than two in the box. I have a powerpoint if you'd like it, but it is not meant to slander Cisco, just to convince my customers NOT to put GbE in a 7200! It is not a GbE platform!

I'd say 300+ mbit/sec on a PA-GE is a more accurate real-world limit, assuming you've got plenty of spare CPU cycles to burn, and no ACL's.

Besides, that's really an apples to oranges comparison. I don't think anyone, including Cisco, has ever made the claim that it can do line rate GbE; that's not to say it isn't useful for certain topologies requiring slightly-faster-than-fast-e router<->switch uplinks, etc.

My powerpoint compares the 7200 with the FastIron 4802 Premium. It is line rate with less than 7 us latency on the two GbE ports. I tested this myself. I can forward this to you if you like. It is a bunch of SmartApps screen captures of the testing. I really like the 7200 VXR. It is a good 10M and minimum FE platform. It can switch DS0 on the midplane and it supports a wide array of interfaces! I just don't like to see it oversubsribed. Many of our customers use the 7200 and have nothing bad to say about it when deployed properly. Gary

On Tue, May 21, 2002 at 07:27:35AM +0200, Mikael Abrahamsson wrote:

I have personally seen a 7200 with PXF-chip and two PA-GE do NAT at 300megabit with a few (10-15) ftp streams going thru it. With more random load it wouldn't go much above 100 meg, though.

I have done 400Mbit with an NPE400, though that's pushing the box close to its limits. But really, a good engineer knows his tools and knows how to choose them for the task. If you want to push 900Mbps, you don't pick a router with a central software based route lookup system and PCI based backplane. On the other hand, if you need to do "complex" things, a 7200 may be your best bet simply because of its simplicity. All the nasty bugs that make using a GSR so miserable almost never manifest themselves on a 7200. If you're adventurous you can even install the "latest" code and probably not pay for your transgression against the IOS gods within 48 hours. :)

And please, lab tests doesnt show it all. Does the Foundry have a route cache? How many entries? I have seen equipment that performs perfectly in the lab start to bog down when you put real traffic on them, because of route cache limitations (for instance, 256.000 entries starts to be problematic when you have thousands of customers running real internet traffic thru the device).

A classic Foundry flaw, which you can get around to some extent with ip net-agg or dr-agg. I've found it best to treat a Foundry doing layer 3 like you would a 7500. You know, tiptoe when you walk by it, try not to give it any funny looks, only login to it when you REALLY need to, only make changes at 2am, etc, it is usable in a customer aggregation role. Anything more is tempting fate. And if^H^Hwhen you run into a really fun issue, don't even think about calling Foundry TAC after hours, all you'll get is someone's house with their screaming kids in the background. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

On Tue, May 21, 2002 at 04:55:51PM +0900, Gary wrote:

Richard:

...

And if^H^Hwhen you run into a really fun issue, don't even think about calling Foundry TAC after hours, all you'll get is someone's house with their screaming kids in the background.

Our TAC is 24/7 and has been 24/7 for years. I work in the Support Center for Japan. We have not gone 24/7 yet, but it is under investigation. Sitting 2 feet from me is a gentleman who has been working with Foundry products since '97. He has called almost every day since then and not once has had the problem you described. I did not mention to him why I was asking these questions and he is honest. Did you call the wrong number? This looks a bit personal...

I didn't say it wasn't 24/7, I just said it rang through to someones house with their screaming kids in the background on a regular basis. I do know how to operate a telephone, thanks. :) And it's nothing personal, I have actually been one of Foundry's biggest supporters compared to almost every other engineer I know. Everyone else gave up using them in layer 3 a long time ago. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras PGP Key ID: 0x138EA177 (67 29 D7 BC E8 18 3E DA B2 46 B3 D8 14 36 FE B6)

On 2002-05-21-01:12:25, Gary <garyb@foundrynet.com> wrote:

I used a Cisco 7200 VXR with NPE-400. I used two different 7200's with the exact same results. Bidirectional throughput on 1GbE is a fraction above 10%. Unidirectional is a bit better (23%). Singl line ACL drops it to 8% (permit ip any any). FE performance doesn't start to drop below line rate until you put more than two in the box. I have a powerpoint if you'd like it, but it is not meant to slander Cisco, just to convince my customers NOT to put GbE in a 7200! It is not a GbE platform!

Send it over, I'd be interested in how you're conducting these tests. I'm not trying to accuse you of lying or slandering your competitors or anything, but well, those numbers sound a bit funny.

...

Besides, that's really an apples to oranges comparison. [...] My powerpoint compares the 7200 with the FastIron 4802 Premium. It is line rate with less than 7 us latency on the two GbE ports. I tested this myself. I can forward this to you if you like. It is a bunch of SmartApps screen captures of the testing.

Not a meaningful comparison; vastly different architectures and purposes. I'd be more interested in seeing empirical data comparing the FastIron 4802 to... say... a Catalyst 2948G-L3 or Extreme Summit 48i. Maybe a Cat6k/MSFC2 as well, seeing as the pricing is roughly comparable in the used hardware market, even if the density is not.

I really like the 7200 VXR. It is a good 10M and minimum FE platform. It can switch DS0 on the midplane and it supports a wide array of interfaces!

Sounds reminiscent of the dot.gone wastefulness that killed many companies. :-) -a