Hi all! There's been some discussion on the list regarding software routers lately and this piqued my interest. Does anybody have any recent performance and capability statistics (eg. forwarding rates with full BGP tables and N ethernet interfaces) or any pointer to what the current state of art in software routers is? - Zed
On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
There's been some discussion on the list regarding software routers
The performance of "software routers" has always had a hardware component. Basically, for the vast majority of them, take your PCI bus bandwidth, count how many times a packet has to cross it, and do the math. You can't forward more than that much traffic no matter *what* software you run on that box. If that number falls short, stop right there and look for some box of different design that has the required backplane bandwidth. You will, of course, take additional performance hits due to locking issues and similar in your software stack (that, and most "software" routers will suffer from not having special hardware assist for routing table lookups). Let us know if you find a suitable chassis/motherboard that has enough bandwidth to make it worth thinking about for anything other than the smaller edge routers that most providers have zillions of... :)
Valdis.Kletnieks@vt.edu wrote:
On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
There's been some discussion on the list regarding software routers
The performance of "software routers" has always had a hardware component.
Basically, for the vast majority of them, take your PCI bus bandwidth, count how many times a packet has to cross it, and do the math. You can't forward more than that much traffic no matter *what* software you run on that box. If that number falls short, stop right there and look for some box of different design that has the required backplane bandwidth.
This might be of interest: http://nrg.cs.ucl.ac.uk/mjh/tmp/vrouter-perf.pdf -- Charles Wyble (818) 280 - 7059 http://charlesnw.blogspot.com CTO Known Element Enterprises / SoCal WiFI project
On Wed, Jul 23, 2008, Charles Wyble wrote:
This might be of interest:
Various FreeBSD related guys are working on parallelising the forwarding layer enough to use the multiple tx/rx queues in some chipsets such as the Intel gig/10ge stuff. 1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?) Linux apparently is/has headed down this path. If someone were to spend some time dissecting the rest of the code to also optimise the single-core throughput then you may see some interesting software routers using commodity hardware (for values of "commodity" roughly equal to "PC servers", rather than "magic lotsacore core MIPS with some extra glue for jacking packets around." Sure its not a CRS-1, but reliably doing a mil pps with a smattering of low-touch features would be rather useful, no? (Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..) Adrian
Adrian Chadd wrote:
1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?)
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html has all the details. It's rather long thread but 1mpps was achieved on a single cpu IIRC (the server had multiple cpus but only one being used for forwarding). Firewall rules slowed it down quite a bit but theres also some work out there being done to minimize this. Regards, Chris
On Wed, Jul 23, 2008, Chris Marlatt wrote:
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00364.html has all the details. It's rather long thread but 1mpps was achieved on a single cpu IIRC (the server had multiple cpus but only one being used for forwarding). Firewall rules slowed it down quite a bit but theres also some work out there being done to minimize this.
Yah, all of that is happening. Some people keep asking why FreeBSD-4 forwarding was always much faster than same-hardware forwarding under current FreeBSD but at least thats finally being worked on. Of course, with my FreeBSD advocacy hat on, if you -want- to see something like FreeBSD handle 1mil+ pps forwarding then you should really drop the FreeBSD Foundation a line and introduce yourself. There are developers working on this (note: not me! :) who would benefit from equipment and funding. Anyway. Some PC class hardware is pretty damned fast. Some vendors even build highish-throughput firewalls and proxies out of PC class hardware. :) The "wah wah PC class hardware has anemic bus IO/memory IO/ CPU speed/ethernet modules and is thus too crap for serious routing" argument is pretty much over for at least 1 mil pps, perhaps more. 2c, Adrian
That is a very interesting paper. Seriously, 7mpps with an off-the-shelf Dell 2950? Even if it were -half- that throughput, for a pure ethernet forwarding solution that is incredible. Shoot, buy a handful of them as hot spares and still save a bundle. Highly recommended reading, even if (like me) you're anti-commodity routing. Cheers, Randal On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian@creative.net.au> wrote:
On Wed, Jul 23, 2008, Charles Wyble wrote:
This might be of interest:
Various FreeBSD related guys are working on parallelising the forwarding layer enough to use the multiple tx/rx queues in some chipsets such as the Intel gig/10ge stuff.
1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?)
Linux apparently is/has headed down this path.
Is anyone using Vyatta for routing? I sure would like to know about any experience with it in production. http://www.vyatta.com/ -- Tim Sanderson, network administrator tims@donet.com -----Original Message----- From: randal k [mailto:nanog@data102.com] Sent: Wednesday, July 23, 2008 1:46 PM To: Adrian Chadd Cc: nanog@merit.edu Subject: Re: Software router state of the art That is a very interesting paper. Seriously, 7mpps with an off-the-shelf Dell 2950? Even if it were -half- that throughput, for a pure ethernet forwarding solution that is incredible. Shoot, buy a handful of them as hot spares and still save a bundle. Highly recommended reading, even if (like me) you're anti-commodity routing. Cheers, Randal On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian@creative.net.au> wrote:
On Wed, Jul 23, 2008, Charles Wyble wrote:
This might be of interest:
Various FreeBSD related guys are working on parallelising the forwarding layer enough to use the multiple tx/rx queues in some chipsets such as the Intel gig/10ge stuff.
1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?)
Linux apparently is/has headed down this path.
Yes. We put in some Vyatta routers to extend our corporate network into another building as a temporary solution (the building had a very short lease, so our boss didn't want to spend any money on Juniper which is our usual net gear vendor). Consequently, we are still there.. go figure. When we started w/ them, they were still using the XORP routing engine (and we haven't upgraded to the new platform yet). My experience wasn't terribly good. The first issue was a bad memory leak in the router manager process when VRRP hello times were set to 1 second. The first indication of something wrong is that our master router crashed, followed by his backup. Had to physically reboot the boxes to get them back online, which involved driving there as no one onsite had access to the cage at the office. All voice and data ran through these routers, basically rendering every employee useless until we got it back online. It wasn't a happy day. After that we had to monitor memory and do controlled reboots every month or so. We eventually convinced Vyatta of this memory leak and they were able to fix it, but that was a very frustrating process, and time consuming for us, which is why the next problems I describe, we have just found our own workarounds. The next problem was a combination of a problem with the Vyatta and a problem w/ our IP phones. The Vyatta was sending garp's for the data vrrp address out the voice vlan (same 2 routers are default gate on both data and voice vlans). All of the workstations run through the phones (which sit tagged on voice vlan, and pass traffic from workstation untagged to data vlan). The phone, seeing the arp for the data vrrp address on its voice vlan, would send traffic to that address out the voice vlan, effectively taking that workstation off the net for anything other than local traffic. That was a bugger to figure out, and basically we solved it with an arptables rule on the vyatta's. That was the one advantage of using a Linux (debian) based router platform, was that we could load other 'unsupported' packages to solve problems like this. The last thing is that OSPF never really converges correctly. You can view the OSPF database, and see which default the routers should converge to, but they do not. They will sit converged to one path for a while, and if you reboot the other router that generates default, they will reconverge to it for a while. This hasn't been a big enough problem for me to worry about it. Last thing to say is, I haven't tried upgrading since Vyatta abandoned the XORP platform and moved to the Quagga platform, but I'm guessing (based on experience w/ Quagga) that they have a lot fewer of these quirks that I've described. IMHO, YMMV, etc --Justin Tim Sanderson wrote:
Is anyone using Vyatta for routing? I sure would like to know about any experience with it in production.
-- Tim Sanderson, network administrator tims@donet.com
-----Original Message----- From: randal k [mailto:nanog@data102.com] Sent: Wednesday, July 23, 2008 1:46 PM To: Adrian Chadd Cc: nanog@merit.edu Subject: Re: Software router state of the art
That is a very interesting paper. Seriously, 7mpps with an off-the-shelf Dell 2950? Even if it were -half- that throughput, for a pure ethernet forwarding solution that is incredible. Shoot, buy a handful of them as hot spares and still save a bundle.
Highly recommended reading, even if (like me) you're anti-commodity routing.
Cheers, Randal
On Wed, Jul 23, 2008 at 10:17 AM, Adrian Chadd <adrian@creative.net.au> wrote:
On Wed, Jul 23, 2008, Charles Wyble wrote:
This might be of interest:
Various FreeBSD related guys are working on parallelising the forwarding layer enough to use the multiple tx/rx queues in some chipsets such as the Intel gig/10ge stuff.
1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?)
Linux apparently is/has headed down this path.
Last thing to say is, I haven't tried upgrading since Vyatta abandoned the XORP platform and moved to the Quagga platform, but I'm guessing (based on experience w/ Quagga) that they have a lot fewer of these quirks that I've described.
Quagga is pretty decent, but it is not uncommon for serious bugs to go unaddressed for a long time. For example, this bug renders Quagga nearly unusable for OSPF on FreeBSD 7, http://bugzilla.quagga.net/show_bug.cgi?id=420 which resulted in some finger-pointing, but the last I heard, it was due to a kernel interface change where FreeBSD multicast code had been rewritten and was DTRT, while Linux was doing something else. This is probably still better than the XORP platform, but it is unfortunate. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
It would be very useful if there was an effort from the telecom community to develop a dynamic routing frontend like Quagga. The amount of human work that it requires in order to build up a product is enormous. If only someone with millions of dollars could donate engineers. It would allow the deployment of small branch office systems at a much lower cost. Would you rather deploy a $3000 cisco edge box which is a unexpandable, 100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE platform? Joe Greco wrote:
Last thing to say is, I haven't tried upgrading since Vyatta abandoned the XORP platform and moved to the Quagga platform, but I'm guessing (based on experience w/ Quagga) that they have a lot fewer of these quirks that I've described.
Quagga is pretty decent, but it is not uncommon for serious bugs to go unaddressed for a long time. For example, this bug renders Quagga nearly unusable for OSPF on FreeBSD 7,
http://bugzilla.quagga.net/show_bug.cgi?id=420
which resulted in some finger-pointing, but the last I heard, it was due to a kernel interface change where FreeBSD multicast code had been rewritten and was DTRT, while Linux was doing something else.
This is probably still better than the XORP platform, but it is unfortunate.
... JG
-- +1.925.202.9485 Sargun Dhillon deCarta sdhillon@decarta.com www.decarta.com
Would you rather deploy a $3000 cisco edge box which is a unexpandable, 100 mbit piece of crap, or throw two $2000 Dell boxes and have a 1 GigE platform?
You don't need two $2000 Dell boxes to get a 1G platform, but this isn't the list for that. You also don't need a ton of money to do open source development (we do one major project here in-house, one that's responsible for generating a noticeable amount of traffic on most networks). But this also isn't the list for that discussion. I suspect software routing is going to continue to be a growing factor in the packet herding world, as more people want to do more interesting things, and the CPU's are able to deal with more, more, more. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Adrian Chadd wrote:
On Wed, Jul 23, 2008, Charles Wyble wrote:
Sure its not a CRS-1, but reliably doing a mil pps with a smattering of low-touch features would be rather useful, no?
(Then, add say, l2tp/ppp into that mix, just as a crazy on-topic example..)
Sounds like a Juniper J-series. Have a look at the forwarding figures for the J6350. It does something around 2mpps and it's just an intel CPU with some PCI/PCI-X interfaces. The device just below it, the J4350 uses a 2.53Ghz celeron. I'm not sure what the J6350 uses. adam.
Once upon a time, Adam Armstrong <lists@memetic.org> said:
Sounds like a Juniper J-series. Have a look at the forwarding figures for the J6350. It does something around 2mpps and it's just an intel CPU with some PCI/PCI-X interfaces. The device just below it, the J4350 uses a 2.53Ghz celeron. I'm not sure what the J6350 uses.
IIRC the new slots (the EPIMs) are PCI-E. The J6350 CPU is a P4 3.4GHz. It is my understanding that the J-series use a real-time layer under the FreeBSD kernel and have a real-time thread for forwarding (as opposed to the M-series with a hardware forwarding engine). -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
On Wed, Jul 23, 2008 at 11:17 AM, Adrian Chadd <adrian@creative.net.au> wrote:
Various FreeBSD related guys are working on parallelising the forwarding layer enough to use the multiple tx/rx queues in some chipsets such as the Intel gig/10ge stuff.
Linux apparently is/has headed down this path.
I've seen some notes from Dave Miller about adding multiple TX queues to the 2.6.27 kernel. See Dave's blog for the gory details: http://vger.kernel.org/~davem/cgi-bin/blog.cgi http://git.kernel.org/?p=linux/kernel/git/davem/net-tx-2.6.git;a=summary AFAIK he hasn't made any claims about performance improvements. I don't know the state of RX queues in Linux. Jeff
* Adrian Chadd:
1 mil pps has been broken that way, but it uses lots of cores to get there. (8, I think?)
Was this with one packet flow, or with millions of them? Traditionally, software routing performance on hosts systems has been optimized for few and rather long flows. Anyway, with multi-core, you don't need funky algorithms for incremental FIB updates anymore (if you don't need sub-second convergence and stuff like that). As a result, you can use really dumb multi-way trees for which a lookup takes something like 100 CPU cycles (significantly less for non-DoS traffic with higher locality).
On Sat, Jul 26, 2008, Florian Weimer wrote:
Was this with one packet flow, or with millions of them?
I believe it was >1 flow. The guy is using an Ixia; I don't know how he has it configured.
Traditionally, software routing performance on hosts systems has been optimized for few and rather long flows.
Yup. And I always ask that question when people claim really high(!) throughput on software forwarding. It turns out their throughput was single source/single dest, and/or large packets (so high throughput, but low pps.) Adrian
On 2008/07/26 01:05 PM Adrian Chadd wrote:
On Sat, Jul 26, 2008, Florian Weimer wrote:
Traditionally, software routing performance on hosts systems has been optimized for few and rather long flows.
Yup.
And I always ask that question when people claim really high(!) throughput on software forwarding. It turns out their throughput was single source/single dest, and/or large packets (so high throughput, but low pps.)
I assume though that all of this is on x86 platform hardware. How does this compare to Linux or FreeBSD running on something else like the Cavium Octeon and other 64bit MIPS based processors?
On Sat, Jul 26, 2008, Colin Alston wrote:
And I always ask that question when people claim really high(!) throughput on software forwarding. It turns out their throughput was single source/single dest, and/or large packets (so high throughput, but low pps.)
I assume though that all of this is on x86 platform hardware. How does this compare to Linux or FreeBSD running on something else like the Cavium Octeon and other 64bit MIPS based processors?
You'll have to ask the people playing with it on that. Me, I've been looking for some multicore MIPS + fruit for some Squid related hackery but I've been busy with other things (like, you know, making Squid-2 be able to be run on multi-core hardware in the first place..) so it'll have to wait.. :) Adrian
Ok, it's probably a stupid question, but given the relative ease of putting 4gb+ ram on a 64bit platform, could packet per second performance be improved by brute forcing the route lookup as an array of 1 byte destination interface indexes for a contiguous swath of /32's from bottom to top? Route updates would be a little ugly, 2^24 bytes to rewrite for a /8, but forwarding lookups out to be a single indexed read ? On Sat, Jul 26, 2008 at 7:31 AM, Adrian Chadd <adrian@creative.net.au>wrote:
And I always ask that question when people claim really high(!)
on software forwarding. It turns out their throughput was single
On Sat, Jul 26, 2008, Colin Alston wrote: throughput source/single
dest, and/or large packets (so high throughput, but low pps.)
I assume though that all of this is on x86 platform hardware. How does this compare to Linux or FreeBSD running on something else like the Cavium Octeon and other 64bit MIPS based processors?
You'll have to ask the people playing with it on that.
Me, I've been looking for some multicore MIPS + fruit for some Squid related hackery but I've been busy with other things (like, you know, making Squid-2 be able to be run on multi-core hardware in the first place..) so it'll have to wait.. :)
Adrian
On Sat, Jul 26, 2008 at 7:41 AM, Dorn Hetzel <dhetzel@gmail.com> wrote:
Ok, it's probably a stupid question, but given the relative ease of putting 4gb+ ram on a 64bit platform, could packet per second performance be improved by brute forcing the route lookup as an array of 1 byte destination interface indexes for a contiguous swath of /32's from bottom to top?
Route updates would be a little ugly, 2^24 bytes to rewrite for a /8, but forwarding lookups out to be a single indexed read ?
Dorn, In theory with about 6 gigs of ram for the IPv4 FIB, sure. But: You're significantly multiplying the likelihood of a cache miss when performing a lookup. You can do a fair amount of tree traversal in cache for the price of one miss. You're a tad shy on ram to try this with IPv6. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
* Dorn Hetzel:
Ok, it's probably a stupid question, but given the relative ease of putting 4gb+ ram on a 64bit platform, could packet per second performance be improved by brute forcing the route lookup as an array of 1 byte destination interface indexes for a contiguous swath of /32's from bottom to top?
8 bits per destination are not enough because you really want to put all the necessary L2 information into the FIB. Perhaps even AS numbers, for flow export.
On Sat, 26 Jul 2008, Dorn Hetzel wrote:
Ok, it's probably a stupid question, but given the relative ease of putting 4gb+ ram on a 64bit platform, could packet per second performance be improved by brute forcing the route lookup as an array of 1 byte destination interface indexes for a contiguous swath of /32's from bottom to top?
Much easier if you filter out any longer prefixes than /24 :-) Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ IRISH SEA: VARIABLE 3 OR 4 BECOMING NORTHEAST 4 OR 5. SLIGHT. FOG PATCHES, THUNDERY SHOWERS LATER. MODERATE OR GOOD, OCCASIONALLY VERY POOR.
On Sat, Jul 26, 2008, Florian Weimer wrote:
Was this with one packet flow, or with millions of them?
I believe it was >1 flow. The guy is using an Ixia; I don't know how he has it configured.
Traditionally, software routing performance on hosts systems has been optimized for few and rather long flows.
Yup.
And I always ask that question when people claim really high(!) throughput on software forwarding. It turns out their throughput was single source/single dest, and/or large packets (so high throughput, but low pps.)
I'm not sure where the claims about "{one, few} flow{s}" are coming from. Certainly the number of flows on a typical UNIX box acting as a router is not that relevant unless you specifically configure something like stateful firewalling, because the typical UNIX box simply doesn't have a *concept* of "flows." It deals with packets. This has its own problems, of course, but handling high levels of traffic in many flows is not one of them. There are other software routing platforms that DO flows, but the above mentions "host[s] systems", so I'm reading that as "UNIX router." On the flip side, packet size is definitely a consideration. This topic has been beaten to death on the Zebra mailing lists by myself and others in the past. With yesterday's technology (P4 3.0G, 512MB RAM, PCI-X, FreeBSD 4) we were successfully dealing with >300Kpps about 3 years ago, without substantial work. That was single source/single dest, but with a full routing table. There's no real optimization for that within the FreeBSD framework, so it is about the same performance you'd have gotten with multi source/multi dest. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
This is not exactly true. The modern Linux kernel (2.6) uses some amount of flow tracking in order to do route caching. You can check this out on your system by: "ip route show cache" It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most systems have the iptables modules loaded in kernel and the conntrack module in kernel. This immediately activates connection tracking, therefore considerably slowing down software routing. The most optimal way of speeding this up would be sticking the route cache into somewhat faster memory. Though it would be fairly nice to get rid of the route cache as that can cause problem with eccentric setups. Also, as cache entries take a moment to be deleted, or degrade leading to convergence times being higher. Joe Greco wrote:
On Sat, Jul 26, 2008, Florian Weimer wrote:
Was this with one packet flow, or with millions of them?
I believe it was >1 flow. The guy is using an Ixia; I don't know how he has it configured.
Traditionally, software routing performance on hosts systems has been optimized for few and rather long flows.
Yup.
And I always ask that question when people claim really high(!) throughput on software forwarding. It turns out their throughput was single source/single dest, and/or large packets (so high throughput, but low pps.)
I'm not sure where the claims about "{one, few} flow{s}" are coming from. Certainly the number of flows on a typical UNIX box acting as a router is not that relevant unless you specifically configure something like stateful firewalling, because the typical UNIX box simply doesn't have a *concept* of "flows." It deals with packets. This has its own problems, of course, but handling high levels of traffic in many flows is not one of them.
There are other software routing platforms that DO flows, but the above mentions "host[s] systems", so I'm reading that as "UNIX router."
On the flip side, packet size is definitely a consideration. This topic has been beaten to death on the Zebra mailing lists by myself and others in the past.
With yesterday's technology (P4 3.0G, 512MB RAM, PCI-X, FreeBSD 4) we were successfully dealing with >300Kpps about 3 years ago, without substantial work. That was single source/single dest, but with a full routing table. There's no real optimization for that within the FreeBSD framework, so it is about the same performance you'd have gotten with multi source/multi dest.
... JG
-- +1.925.202.9485 Sargun Dhillon deCarta sdhillon@decarta.com www.decarta.com
This is not exactly true. The modern Linux kernel (2.6) uses some amount of flow tracking in order to do route caching. You can check this out on your system by: "ip route show cache"
Okay... # ip route show cache ip: Command not found. # So I guess that's all well and good for me.
It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most systems have the iptables modules loaded in kernel and the conntrack module in kernel. This immediately activates connection tracking, therefore considerably slowing down software routing. The most optimal way of speeding this up would be sticking the route cache into somewhat faster memory. Though it would be fairly nice to get rid of the route cache as that can cause problem with eccentric setups. Also, as cache entries take a moment to be deleted, or degrade leading to convergence times being higher.
Note .. to .. self .. Linux .. makes .. crappy .. router. Got it. Guess we'll continue to use FreeBSD, and the lesson to come away with is that it probably pays to avoid technologies that are suboptimal for the task at hand. Not everything is created equal. It also pays to tune things. If "conntrack" hurts, then remove it. With the emergence of computers with many cores, it will be very interesting to see how this develops. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
It keeps track of Src/Dst/QoS/Ethernet adapters/etc.. Additionally most systems have the iptables modules loaded in kernel and the conntrack module in kernel. This immediately activates connection tracking, therefore considerably slowing down software routing. The most optimal way of speeding this up would be sticking the route cache into somewhat faster memory. Though it would be fairly nice to get rid of the route cache as that can cause problem with eccentric setups. Also, as cache entries take a moment to be deleted, or degrade leading to convergence times being higher.
Note .. to .. self .. Linux .. makes .. crappy .. router. Got it.
Guess we'll continue to use FreeBSD, and the lesson to come away with is that it probably pays to avoid technologies that are suboptimal for the task at hand. Not everything is created equal. It also pays to tune things. If "conntrack" hurts, then remove it.
You can use Linux without conntrack. You can either do "rmmod ip_conntrack" (unload the module), rm /var/lib/modules/ip_conntrack (or something like that to erase the file) or use the RAW queue to forward some packets without connection tracking (-j NOTRACK) and some others with conntrack (proxy redirection, captive portal and thinks like that requires stateful forwarding in any platform). I would be more worried about the prefix match and route cache done by the operating system you are considering for use as a router. That cannot be circunverted by turning off conntrack, pf or anything that might do more with the packet that plain simple routing. Rubens
Rubens Kuhl Jr. wrote:
You can use Linux without conntrack. You can either do "rmmod ip_conntrack" (unload the module), rm /var/lib/modules/ip_conntrack (or something like that to erase the file) or use the RAW queue to forward some packets without connection tracking (-j NOTRACK) and some others with conntrack (proxy redirection, captive portal and thinks like that requires stateful forwarding in any platform).
I would be more worried about the prefix match and route cache done by the operating system you are considering for use as a router. That cannot be circunverted by turning off conntrack, pf or anything that might do more with the packet that plain simple routing.
Hi, As of 2.6.x kernel version (at least on 2.6.17) there is a FIB implementation called LC_Trie which supposedly does an O(1) route lookup which is very fast. Where I live there are a lot of linux boxes deployed as routers pushing line rate GE for hundreds to thousand nodes computer networks while also deliverying QoS for each and every node. From what I see in this thread you're more worried about T3/E3 linecards than the actual Linux performance as a router. As a personal example, I use a celeron 2.53Ghz with 512Mb of ram to push line rate 3 x 100Mbps cards wihout any discernable load reported either by top or uptime and that on top of Quagga with about ~ 5k prefixes. Also, as an experiment I loaded a full routing table from one of my peers and besides of the increased RAM usage by Quagga to about 50MB the machine forwarded at the same rate, _maybe_ 1% incresed load.
Sargun Dhillon wrote:
This is not exactly true. The modern Linux kernel (2.6) uses some amount of flow tracking in order to do route caching. You can check this out on your system by: "ip route show cache"
Did you mean "route -C" ? I like the idea and price point of the ImageStream products, but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing. ~Seth
but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.
Enough with the bipartisan politics. There are more choices than just Linux and FreeBSD for software routing. Click for instance <http://read.cs.ucla.edu/click/> --Michael Dillon
michael.dillon@bt.com wrote:
but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.
Enough with the bipartisan politics. There are more choices than just Linux and FreeBSD for software routing.
Click for instance <http://read.cs.ucla.edu/click/>
--Michael Dillon
Anyone have experience with RouterOS (http://www.mikrotik.com/)? Created mostly to run on these guys I think (http://www.routerboard.com/comparison.html) which generally don't get above 200k pps on the higher models.. But will RouterOS run on bigger boxen?
Justin Sharp wrote:
michael.dillon@bt.com wrote:
but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.
Enough with the bipartisan politics. There are more choices than just Linux and FreeBSD for software routing.
Click for instance <http://read.cs.ucla.edu/click/>
--Michael Dillon
Anyone have experience with RouterOS (http://www.mikrotik.com/)? Created mostly to run on these guys I think (http://www.routerboard.com/comparison.html) which generally don't get above 200k pps on the higher models.. But will RouterOS run on bigger boxen?
Yes I do, and I'm still in therapy. I was pushing 30mbit, and I can't remember how many PPS through one, and it crashed about once a month requiring onsite intervention (usually at midnight). This was running on a Compaq Deskpro I think. It doesn't have much support for good network cards. I suspect the Realtek's were behind the crashes. Andrew
Andrew D Kirch wrote:
Justin Sharp wrote:
michael.dillon@bt.com wrote:
Yes I do, and I'm still in therapy. I was pushing 30mbit, and I can't remember how many PPS through one, and it crashed about once a month requiring onsite intervention (usually at midnight). This was running on a Compaq Deskpro I think. It doesn't have much support for good network cards. I suspect the Realtek's were behind the crashes.
Yeah.... or any number of cheap consumer parts in the Deskpro. I would love to see RouterOS running on an HP or Dell mid range server and how that performs. I'm guessing it would be quite nice. -- Charles Wyble (818) 280 - 7059 http://charlesnw.blogspot.com CTO Known Element Enterprises / SoCal WiFI project
Andrew D Kirch wrote:
Anyone have experience with RouterOS (http://www.mikrotik.com/)? Created mostly to run on these guys I think (http://www.routerboard.com/comparison.html) which generally don't get above 200k pps on the higher models.. But will RouterOS run on bigger boxen? Yes I do, and I'm still in therapy. I was pushing 30mbit, and I can't remember how many PPS through one, and it crashed about once a month requiring onsite intervention (usually at midnight). This was running on a Compaq Deskpro I think. It doesn't have much support for good network cards. I suspect the Realtek's were behind the crashes. The Realteks were almost certainly to blame. Just like any other "server," good hardware is well worth it. RouterOS 2.9 and 3.x support Intel's gigabit server NICs, which work quite well.
www.mikrotikrouter.com sells a few nice purpose-built rackmount appliances for this sort of thing. (Just a happy customer, don't work there or anything.) If you have the budget, and need the single-purpose horsepower, you'll usually be happier with Cisco or Juniper or someone that makes dedicated routing kit. If money's tight, or you need one box to do several network-related jobs for whatever reason, Mikrotik's software is another useful tool to have. David Smith MVN.net
michael.dillon@bt.com wrote:
but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.
Enough with the bipartisan politics. There are more choices than just Linux and FreeBSD for software routing.
Click for instance <http://read.cs.ucla.edu/click/>
--Michael Dillon
Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution. ~Seth
Seth Mattinen wrote:
michael.dillon@bt.com wrote:
but knowing how bad Linux is at being a router and that their products are Linux-based, I'm afraid to give one a try. J products are based on a competing non-Linux platform that has a better reputation for routing.
Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.
Hmmmm. Well then you probably don't want to use Linux/BSD as a router, as a substantial amount of DIY is required for anything beyond relatively simple routing. MPLS support (on Linux) for example is in early phases and requires integrating separate pieces and is best supported on Fedora9. Needless to say, Fedora isn't designed for reliable/stable operation and long term deployment. I have yet to look into *BSD based solutions, but hear very good things about firewall performance. I don't know about BGP/OSPF/MPLS etc support on FreeBSD but am going to wager a guess its on par with Linux if not better. To address another point made in this thread, see http://ols.fedoraproject.org/OLS/Reprints-2007/zhu-Reprint.pdf which addresses hardware multiqueue device support under Linux. Its from 2007. I think there was a question about Linux/multiqueue support in this thread, but I am not 100% sure. :) I think there was mention of Vyatta earlier in the thread and some talk about it switching from Xorp to Quagga, and a supposition that should improve it. -- Charles Wyble (818) 280 - 7059 http://charlesnw.blogspot.com CTO Known Element Enterprises / SoCal WiFI project
Hmmmm. Well then you probably don't want to use Linux/BSD as a router, as a substantial amount of DIY is required for anything beyond relatively simple routing. MPLS support (on Linux) for example is in early phases and requires integrating separate pieces and is best supported on Fedora9. Needless to say, Fedora isn't designed for reliable/stable operation and long term deployment.
I have yet to look into *BSD based solutions, but hear very good things about firewall performance. I don't know about BGP/OSPF/MPLS etc support on FreeBSD but am going to wager a guess its on par with Linux if not better.
The underlying OS is responsible for packet forwarding, but none of them do any significant routing protocols natively. Adding on a package such as Quagga or OpenBGPD is required for that, and the results of each should be relatively similar across platforms. The only major caveat is that Quagga OSPF is currently a disaster on FreeBSD 7. Don't try it. We added a server that was advertising some stuff, with multiple interfaces, using a config identical to what we do under FreeBSD 6. Not only did it randomly not work, but it also randomly killed OTHER OSPF speakers elsewhere in the network, including on non-directly attached networks in another OSPF area (we'd log in and see no neighbors). OpenOSPFD appears to be the fix for that. Simpler, smaller, but dumb enough that it advertised 127.0.0.1 into our OSPF environment when we were trying to get some aliases on lo0 advertised, which caused freaking out of pretty much every OSPF-speaking UNIX server we have (sigh). BGP is straightforward, except for things like MD5, which can be a bit dicey. Quagga is very good, and much less expensive than, something like Cisco for a route server, from what I've heard over the years. You'll notice some of the Route-views boxes are Quagga or Zebra (its predecessor). ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On 2008-07-28, Joe Greco <jgreco@ns.sol.net> wrote:
I have yet to look into *BSD based solutions, but hear very good things about firewall performance. I don't know about BGP/OSPF/MPLS etc support on FreeBSD but am going to wager a guess its on par with Linux if not better.
The underlying OS is responsible for packet forwarding, but none of them do any significant routing protocols natively.
OpenBSD has OpenOSPFD/OpenBGPD in the base OS rather than as a port/ package, so it's fully coupled with any kernel changes (and supports some things missing from the FreeBSD port).
* Stuart Henderson <stu@spacehopper.org> [2008-08-01 19:06]:
On 2008-07-28, Joe Greco <jgreco@ns.sol.net> wrote:
I have yet to look into *BSD based solutions, but hear very good things about firewall performance. I don't know about BGP/OSPF/MPLS etc support on FreeBSD but am going to wager a guess its on par with Linux if not better.
The underlying OS is responsible for packet forwarding, but none of them do any significant routing protocols natively.
OpenBSD has OpenOSPFD/OpenBGPD in the base OS rather than as a port/ package, so it's fully coupled with any kernel changes (and supports some things missing from the FreeBSD port).
can't be stressed enough; the concept of OpenBGPD/OSPFD/RIPD/DVRMPD/OSPF6D (did I forget one again?) is not too be just another daemon implementing the protocol at hand, they come with massive changes to the OpenBSD kernel to offer an alternative to other solutions, including "hardware" routers. Now it is quite clear that you don't want to run 5 loaded 10GE ports on any Hardware OpenBSD currently supports (it's not just PCs), but there are enough installations with smaller bandwidth requirements where it is a very viable alternative. -- Henning Brauer, hb@bsws.de, henning@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Click for instance <http://read.cs.ucla.edu/click/>
Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.
Sorry, but you're in the wrong place. The IP networking consultants are over thataway, and if you pay them a nice daily fee they will sort out your problem for you. But if you want free suggestions, then you'll have to put up with half answers, vendor fanboys, and the usual ruckus of NANOG. --Michael Dillon P.S. that was a serious suggestion up above. If you have an interest in software routers, then you should look at it. If you just want to buy products then all routers are software routers, most especially the ones that depend on something called IOS or Junos. Focus on the capabilities that you need and the prices. Don't try to be pretend to be a router designer.
michael.dillon@bt.com wrote:
Click for instance <http://read.cs.ucla.edu/click/>
Thanks for being oh-so-helpful with a serious question. Got any useful answers for me? Give me a vendor that offers your suggestion. I don't have time for a make-it-myself solution.
Sorry, but you're in the wrong place. The IP networking consultants are over thataway, and if you pay them a nice daily fee they will sort out your problem for you.
But if you want free suggestions, then you'll have to put up with half answers, vendor fanboys, and the usual ruckus of NANOG.
--Michael Dillon
P.S. that was a serious suggestion up above. If you have an interest in software routers, then you should look at it. If you just want to buy products then all routers are software routers, most especially the ones that depend on something called IOS or Junos. Focus on the capabilities that you need and the prices. Don't try to be pretend to be a router designer.
I'm aware of Cisco IOS, then BSD-based and Linux-based platforms that are actually sold as routing products. I also know there are a billion "yay, router!" things out there. Rather than reinvent the wheel alone, nanog has to contain the highest concentration of people that have tried various things and already know what will work and what won't work. I'm not looking for OS politics, just operational experience from people who have access to more money and more hardware than I do to have tried more stuff. ~Seth
On Mon, Jul 28, 2008 at 10:08:32PM +0100, michael.dillon@bt.com wrote:
But if you want free suggestions, then you'll have to put up with half answers, vendor fanboys, and the usual ruckus of NANOG.
As much as I hate to contribute to the problem, I'd like to point out that the barrage of useless, off-topic, empty traffic on this list in the last week is, in my estimation, quite a bit above the "usual" ruckus of NANOG. While I'm not one to thunk down the rulebook, can you all collectively knock it off? Cheers, -jp -- -------------------------------------------------------- Rev. Jeffrey Paul -datavibe- sneak@datavibe.net aim:x736e65616b pgp:0xD9B3C17D phone:1-800-403-1126 9440 0C7F C598 01CA 2F17 D098 0A3A 4B8F D9B3 C17D "Virtue is its own punishment." --------------------------------------------------------
On Mon, Jul 28, 2008 at 10:08:32PM +0100, michael.dillon@bt.com wrote:
But if you want free suggestions, then you'll have to put up with half answers, vendor fanboys, and the usual ruckus of NANOG.
As much as I hate to contribute to the problem, I'd like to point out that the barrage of useless, off-topic, empty traffic on this list in the last week is, in my estimation, quite a bit above the "usual" ruckus of NANOG.
While I'm not one to thunk down the rulebook, can you all collectively knock it off?
Cheers, -jp I haven't followed the other threads in the last week, but I don't think
Rev. Jeffrey Paul wrote: that a discussion of routers is off topic. While Michael's opinion was expressed in a fairly tongue-in-cheek method as would be required by his response, I don't see anything offtopic, perhaps just hotly worded. Anderw
Andrew D Kirch wrote:
On Mon, Jul 28, 2008 at 10:08:32PM +0100, michael.dillon@bt.com wrote:
But if you want free suggestions, then you'll have to put up with half answers, vendor fanboys, and the usual ruckus of NANOG.
As much as I hate to contribute to the problem, I'd like to point out that the barrage of useless, off-topic, empty traffic on this list in the last week is, in my estimation, quite a bit above the "usual" ruckus of NANOG.
While I'm not one to thunk down the rulebook, can you all collectively knock it off?
Cheers, -jp I haven't followed the other threads in the last week, but I don't think
Rev. Jeffrey Paul wrote: that a discussion of routers is off topic. While Michael's opinion was expressed in a fairly tongue-in-cheek method as would be required by his response, I don't see anything offtopic, perhaps just hotly worded.
I wasn't too thrilled about being accused of OS politics when I was genuinely concerned about deploying a software router based on things I've heard in passing or read about here previously. It *is* nice to know that someone found out that FreeBSD 7 hates OSPF - since I actually use OSPF - and that would have tormented me for a while had I gone that route. Back to the topic at hand, unfortunately I wouldn't have the luxury of converting T1/T3 to Ethernet. I've used cards from Digium and Sangoma in the past for T1 and been relatively pleased, although older Digium cards hated sharing an IRQ with anything. ~Seth
I wasn't too thrilled about being accused of OS politics when I was genuinely concerned about deploying a software router based on things I've heard in passing or read about here previously. It *is* nice to know that someone found out that FreeBSD 7 hates OSPF - since I actually use OSPF - and that would have tormented me for a while had I gone that route.
Nonononono ... QUAGGA hates FreeBSD 7, and therefore Quagga OSPF does not work on FreeBSD 7. OpenOSPFD has worked like a CHAMP. Any issues I have with OpenOSPFD are related to it not being Quagga, or as flexible as Quagga, but I have had >0< issues with OpenOSPFD's reliability. With only a relatively short period to judge it, I'll note, but still, easy to install, easy to deploy... Easily enough that I'm having semi-serious thoughts ... The problem appears to be related to FreeBSD having made changes to the multicast API to become RFC-compliant. Quagga has a bunch of workarounds for various forms of brokenness present in Linux, etc., and my reading suggests that Quagga is doing the wrong thing. Quite frankly, this, and the loopback implosion OpenOSPFD caused when we misconfigured it, are the worst things I've heard about software routing this year. Unlike most Cisco or Juniper issues, it's not a "reboot the router" or "that'll be fixed 'soon'" type of thing. You're free to open up the hood and experiment yourself. If your Cisco OSPF wasn't working, and Cisco didn't show any signs of fixing it, it's a little more difficult to just pop the top and drop a different routing protocol engine in. Sorry for any misinterpretation. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Mon, 28 Jul 2008, Rev. Jeffrey Paul wrote:
As much as I hate to contribute to the problem, I'd like to point out that the barrage of useless, off-topic, empty traffic on this list in the last week is, in my estimation, quite a bit above the "usual" ruckus of NANOG.
While I'm not one to thunk down the rulebook, can you all collectively knock it off?
I gotta disagree with you, especially with regard to this thread. Much of the conversations on this topic have ancillary benefits, specifically for folks who need to populate networks with things like 10g forensic sensors or similiar. I don't see commodity hardware router discussions being any different from a foundry vs juniper vs crisco discussion, be it typical fanboy nonsense or otherwise. As far as active threads on nanog go, the signal to noise ratio on this one has already far exceeded more 'operational' ones. Even anecdotal experiences noted thus far have been pretty insightful, and useful. I even totally resisted the urge of bombing the thread by extolling the virtues of the Killer NIC as a solution to all the throughput problems people have, because I felt it would really derail what has thus far been a fairly educational thread. That said though, the more I thought about it (the killer nic joke), the more I looked at it. What's the state of NPU offloading amongst software routers? Is the notion even viable? I've seen a couple remarks about various brands of network cards having various buffer and interrupt driven issues as serious limiters to pps throughput, which is what prompted me to think of it in the first place. - billn
* Joe Greco:
I'm not sure where the claims about "{one, few} flow{s}" are coming from. Certainly the number of flows on a typical UNIX box acting as a router is not that relevant unless you specifically configure something like stateful firewalling, because the typical UNIX box simply doesn't have a *concept* of "flows." It deals with packets.
You are mistaken. Linux routing is flow-based. Ever wondered what those "dst cache overflow" messages mean you see during a DoS attack? It's the flow cache complaining that it can't expire records in an organic manner. I don't know much about FreeBSD. I think it got a route cache after FreeBSD 4, too. That's the reason why the FreeBSD 4 IP stack is still so popular.
On Wed, Jul 23, 2008 at 11:24 AM, <Valdis.Kletnieks@vt.edu> wrote:
On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
There's been some discussion on the list regarding software routers
The performance of "software routers" has always had a hardware component.
Basically, for the vast majority of them, take your PCI bus bandwidth, count how many times a packet has to cross it, and do the math. You can't forward more than that much traffic no matter *what* software you run on that box. If that number falls short, stop right there and look for some box of different design that has the required backplane bandwidth.
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from the NIC to main DRAM. They claim a full 10gbps on a PCIE bus. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
I wonder, has anyone heard of this used for IDS? I've been looking at building a commodity SNORT solution, and wondering if a powerful network card will help, or would the bottleneck be in processing the packets and overhead from the OS? - naveen
On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <naveen@calpop.com> wrote:
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
I wonder, has anyone heard of this used for IDS? I've been looking at building a commodity SNORT solution, and wondering if a powerful network card will help, or would the bottleneck be in processing the packets and overhead from the OS?
http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS snort at 1g & 10g -chris
We use them here and there (the 1Gig versions). The biggest thing to think about is the types of rule-sets you'll be using compounded by the number of flows being created / expired. Once tuned, they work quite well, but the balance is how fast you can pull/analyze out of RAM. Compiling the rules down to the card's level speeds things up a bit, but at the loss of using more dynamic rulesets. If you can get the raw data to some sort of larger medium (say, rotating pcaps on a disk), you length the buffer-window. FWIW however, probably the best way to scale this is get an Xport fiber regen tap, populate with a few of these, tune them to monitor different segments based on address space or port ranges. You'll have yourself a relatively cheap solution, but extremely effective solution. I've yet to test out the NinjaProbes... It's on my todo list... On Jul 23, 2008, at 2:21 PM, Christopher Morrow wrote:
On Wed, Jul 23, 2008 at 11:05 AM, Naveen Nathan <naveen@calpop.com> wrote:
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
I wonder, has anyone heard of this used for IDS? I've been looking at building a commodity SNORT solution, and wondering if a powerful network card will help, or would the bottleneck be in processing the packets and overhead from the OS?
http://www.endace.com/our-products/ninja-appliances/NinjaProbe-NIDS
snort at 1g & 10g
-chris
-- Wes Young Network Security Analyst CIT - University at Buffalo http://claimid.com/saxjazman9
* William Herrin:
The Endace DAG cards claim they can move 7 gbps over a PCI-X bus from the NIC to main DRAM. They claim a full 10gbps on a PCIE bus.
But they are receive-only, right? The main problem for "software routing" seems to be that it's basically Ethernet-only because other interfaces are very difficult to find.
Valdis.Kletnieks@vt.edu wrote:
On Wed, 23 Jul 2008 02:52:56 PDT, Zed Usser said:
There's been some discussion on the list regarding software routers
The performance of "software routers" has always had a hardware component.
Basically, for the vast majority of them, take your PCI bus bandwidth, count how many times a packet has to cross it, and do the math. You can't forward more than that much traffic no matter *what* software you run on that box. If that number falls short, stop right there and look for some box of different design that has the required backplane bandwidth.
You will, of course, take additional performance hits due to locking issues and similar in your software stack (that, and most "software" routers will suffer from not having special hardware assist for routing table lookups).
The current state of the art is around 2 million pps for fast intel arch system.
Let us know if you find a suitable chassis/motherboard that has enough bandwidth to make it worth thinking about for anything other than the smaller edge routers that most providers have zillions of... :)
Hi all!
There's been some discussion on the list regarding software routers lately and this piqued my interest. Does anybody have any recent performance and capability statistics (eg. forwarding rates with full BGP tables and N ethernet interfaces) or any pointer to what the current state of art in software routers is?
- Zed I'd like to be wrong, but there's no way that any PC/Commodity routing system is going to work (in any environment other than Ethernet). For
Zed Usser wrote: the small ISP starting out (you know, the ones selling T1's/xDSL), there are no Channelized T3/OC3 cards available for a PeeCee, which means you still need to buy something from Cisco or Juniper. And the major carriers are already using Cisco/Juniper, because even at the price they charge they aren't unreasonable because they support their product. I don't care how many packets per second, or simultaneous route flows you can do if I can't plug anything besides Ethernet into it. If you can show me the hardware, great I'll take a look at it, otherwise these simply don't matter all that much. Andrew
Once upon a time, Andrew D Kirch <trelane@trelane.net> said:
I'd like to be wrong, but there's no way that any PC/Commodity routing system is going to work (in any environment other than Ethernet). For the small ISP starting out (you know, the ones selling T1's/xDSL), there are no Channelized T3/OC3 cards available for a PeeCee, which means you still need to buy something from Cisco or Juniper.
First hit on a Google search: http://www.imagestream.com/PCI_921-CDS.html -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Chris Adams wrote:
Once upon a time, Andrew D Kirch <trelane@trelane.net> said:
I'd like to be wrong, but there's no way that any PC/Commodity routing system is going to work (in any environment other than Ethernet). For the small ISP starting out (you know, the ones selling T1's/xDSL), there are no Channelized T3/OC3 cards available for a PeeCee, which means you still need to buy something from Cisco or Juniper.
First hit on a Google search:
ImageStream isn't really "look I just bought a Dell and now I need to route my DS3 with BGP" is it? Does anyone reading this use any ImageStream gear in production? I'm curious what your experiences are. ~Seth
I have one of these (Imagestream T3 WAN adapter on an Imagestream router) for 5+ years to back up my Cisco 7204 with a channelized T3 card. I like the system, I like the card. The other engineers in my office call it the "bling router". Lots of gold chrome. Pimped out. --Patrick Darden -----Original Message----- From: Chris Adams [mailto:cmadams@hiwaay.net] Once upon a time, Andrew D Kirch <trelane@trelane.net> said:
there are no Channelized T3/OC3 cards available for a PeeCee, which means you still need to buy something from Cisco or Juniper.
First hit on a Google search: http://www.imagestream.com/PCI_921-CDS.html -- Chris Adams <cmadams@hiwaay.net>
participants (33)
-
Adam Armstrong -
Adrian Chadd -
Andrew D Kirch -
Bill Nash -
Charles Wyble -
Chris Adams -
Chris Marlatt -
Christopher Morrow -
Colin Alston -
Darden, Patrick S. -
David E. Smith -
Dorn Hetzel -
Eugeniu Patrascu -
Florian Weimer -
Henning Brauer -
Jeffrey Ollie -
Joe Greco -
Joel Jaeggli -
Justin Sharp -
michael.dillon＠bt.com -
Naveen Nathan -
randal k -
Rev. Jeffrey Paul -
Rubens Kuhl Jr. -
Sargun Dhillon -
Seth Mattinen -
Stuart Henderson -
Tim Sanderson -
Tony Finch -
Valdis.Kletnieks＠vt.edu -
Wes Young -
William Herrin -
Zed Usser