At 01:10 AM 4/20/2003, Richard A Steenbergen wrote:

As much as I hate to say this, stock FreeBSD makes a terrible high performance router. The route-cache is horribly out of date with modern techniques, and there just aren't that many wackjobs out there trying to shove a hungred megs through a unix box to fully debug it (with the exception of a certain notoriously cheap people who will probably respond to this email talking about their success with FORE ATM OC3 cards :P).

Ok, I'll bite. We're routinely shoving 500mbps through our FreeBSD system running zebra, and we've never hit the 50% cpu mark. 3 GigE BGP peers passing me a full table and one GigE LAN interface, a few VLANs, lengthy IPFW rules, and tons of "count" rules so I can MRTG each IP passing through it. In some off network synthetic testing, I easily maxed out our GigE LAN interface before the router dropped a packet. All this on a $1800 Dell server with a $150 Intel PCI-X card slapped in there. This system's been up for 6 months now. Zero crashes, zero hung interfaces, zero problems. I'm not saying a FreeBSD+Zebra system is going to do everything that your high end router will, but I haven't run into anything that I couldn't find some way of doing with the tools that I had. IPFW, Dummynet, tcpdump and other tools that come "stock" have saved me quite a bit of effort over some other much more costly solutions that couldn't do some things that we depend on now. I know quite a few would consider this a bit of ghetto networking, but I've even worked out a pretty reliable hot-standby system for all our web servers by running zebra/bgpd on each of them and having them announce /32's for the IP's that the web server is listening on to a route reflector. Have another box setup as a standby system announcing the same /32 to our router with higher metric, and the failover is instant. No extra hardware, no fancy load balancers, and the web servers don't even have to know anything happened. Zebra/FreeBSD aren't the best things out there, but when you have essentially no budget, there are a lot of us out there who've figured out how to make our networks operate pretty well. I really don't mean this as a flame, Richard... Just that I think a lot of people out there have discounted this without trying it or researching what others have gotten to work. Not all of us have the luxury of working somewhere where not being "cheap" is an option. :) -- Kevin

From my personal experience, the "Olive" is just plain old JunOS. When you install it on a PC, it does not include the hardware related elements, I found this out when the M10 crashed and I installed from scratch outside the M10 hardware (on a new disk on a PC). For the record, all you have to do is reinstall the package and the hardware will show up.

I ran several Ethernet ports and it worked fine. I am not sure about PCI GE ports and if they are supported. Bottom line, if you have JunOS you can make a stand alone pc work as a router (did this with 5.2R1.4-domestic). This was a great solution for a lab router doing protocols, not sure about performance. All the best, JK -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Omachonu Ogali Sent: Sunday, April 20, 2003 8:05 PM To: Kevin Day Cc: nanog@merit.edu Subject: Re: Zebra Router??? I had 1 GigE, 2 FastE interfaces, no full feeds, about 16k routes, pushing 100-120Mbps, and I always hit 100% cpu, causing Zebra's BGP process to die repeatedly. And, whenever I removed a route from the Zebra daemon, it didn't get removed from the kernel routing table, and a simple 'route delete x.x.x.x' did not work, so the route was effectively STUCK until the next reboot. (No, killing and restarting the Zebra processes did not work either, nor did a 'route flush'). I've tried different CPUs, motherboards, hard drives, network cards, kernels, sysctl variables, memory modules, no, some, and all (about 80) IPFW rules, and still, same results. 100% CPU while the box is sucking ass at 40-50kpps. Bottom line, from my personal experiences, FreeBSD sucks as a router, and Zebra sucks as a routing process. Yeah yeah yeah, Juniper uses FreeBSD for their code, but a) it's heavily modified, modifications you won't find on the net for yourself, b) they use ASICs to speed up certain operations, and there is no PCI card with a PacketGoFaster(tm) chipset you can buy to get better performance out of your desktop gaming Samba-serving router. I hear Olive has some good performance, but guess what, it's not public domain and your chances of finding it are very slim, so you're SOL for now. But hey, your mileage may vary.

Any experiences doing this with Linux, as opposed to FreeBSD? How's the Linux route cache implementation? -- Bruce Robertson, President/CEO +1-775-348-7299 Great Basin Internet Services, Inc. fax: +1-775-348-9412 http://www.greatbasin.net

At 03:56 PM 21/04/2003 -0400, Deepak Jain wrote:

If you go higher than 100Mb/s speeds, look for zero-copy boards, especially on the Gigabit side that FreeBSD supports. You have drastically low interrupt overhead and your speeds to move packets around increase. In my experience, its not the CPU that gets bogged down (first) but the interrupt load on a properly configured (software) box.

... The em driver now supports polling. The author of much of the polling code (luigi@iet.unipi.it) claimed to push 650Kpps on a 2.4Ghz PIV machine. YMMV based on the hardware you use, and the cards you combine. ---Mike

On Mon, 21 Apr 2003, Ryan Mooney wrote:

I don't see any reason that you shouldn't be able to basically saturate a 100Mb line with a box of this speed, although I have no publishable information to back me up.

You can saturate 100mb ether with a low end classic pentium (eg p90). 100mb ethernet is not magic. GbE is a different story though :-) -Dan -- [-] Omae no subete no kichi wa ore no mono da. [-]