I also wonder about re-inventing the wheel. The router part is easy, you could even do that with a windows box (that's a joke). Obviously capital cost is part of it, but the man hours involved in doing what you're talking about, especially since you are talking about a telco.... whatever you come up with has to be pretty darn reliable... Certainly would be interested in a little more information about the use case. Eric On Dec 26, 2013, at 8:46 AM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:

I am a believer of not having to re-invent the wheel...

Having said that.. have you looked at 'purpose built appliances' e.g.

http://www.lannerinc.com/ http://us.axiomtek.com/

If you are looking for a full router.... Consider such as these... http://www.linktechs.net/ http://www.maxxwave.com/

and there are a few others but the concept is the same

Personally, I am not a believer in making a single device be the do all / end all of everything.. While one can do everything on a big server .. however breaking things out e.g. voip trans-coding and routing make maintenance, availability, and ability to create redundancy much more practical.

Regards

Faisal Imtiaz Snappy Internet & Telecom

----- Original Message -----

...

From: "Nick Cameo" <symack@gmail.com> To: nanog@nanog.org Sent: Thursday, December 26, 2013 11:33:13 AM Subject: The Making of a Router

Hello Everyone,

We are looking to put together a 2u server with a few PCIe 3 x8 (recommendations appreciated). The router will take a voip transcoding line card, and will act as an edge router for a telecom company.

For things like BGP (Quagga, Zebra, all that lovely stuff!!!), static routes, and firewall capabilities we are thinking gentoo linux stripped for sure however, what about the BSDs? FreeBSD or OpenBSD. Any comments, feedback, does, and don'ts are much appreciated.

Kind Regards,

Nick.

On Thu, Dec 26, 2013 at 05:21:11PM +0000, Warren Bailey wrote:

Not to mention the fact that this "router" will require support. The build before buy people are silly. Let the smart router guys do their thing and use their box accordingly. When it breaks call to inform them it broke and they will fix it.

Unless they deem that it's "outside of scope". Or they can't get anyone to you inside of SLA[1]. Or they send someone incompetent. Or it's a problem that's never happened before.

Diy projects are a nightmare to support.

*Everything* is a nightmare to support. A DIY project just means that you're betting you're smarter than whoever the vendor sends to fix their thing. Maybe it's a good bet, maybe it isn't. I'm sure you've got plenty of horror stories of DIY project support; I've got plenty of horror stories of vendor support. Perhaps we can get together some day and have a story-off. <grin> - Matt [1] So you might get some SLA credits at some point in the future. So what? It won't even cover your SLA payouts to your customers, let alone the lost business and reputation.

----- Original Message -----

From: "Matt Palmer" <mpalmer@hezmatt.org>

I've got plenty of horror stories of vendor support. Perhaps we can get together some day and have a story-off. <grin>

Just subject-tag it so we can archive it, ok guys? Cheers, -- jr 'whacky weekend' a -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA #natog +1 727 647 1274

On 12/26/13, 8:46, Faisal Imtiaz wrote:

I am a believer of not having to re-invent the wheel...

Having said that.. have you looked at 'purpose built appliances' e.g.

http://www.lannerinc.com/ http://us.axiomtek.com/

If you are looking for a full router.... Consider such as these... http://www.linktechs.net/ http://www.maxxwave.com/

I bought two Maxxwaves with the Core2 Duo processor and it's an Axiomtek NA-822 inside. It's a nifty platform, I'm probably going to put BIRD on it and make some BGP RRs one of these fine days. I'm too OCD to suffer a "server" in the routing/switching bays as the only thing with ports on the back. I wouldn't imagine trying to route 10 gig with it though, or sharing any functions (like VOIP) on it. ~Seth

On 12/26/13, 11:28, Faisal Imtiaz wrote:

Ahh.. so you are the one who bought those two from Ebay ! I was watching, but got to them rather late.

If you are the one who got them.. you got a great deal.

These have Mikrotik ROS license with them, you can do BGP/ OSPF etc. with them If you want to reload them with other OS, all you got to do is pull out the FlashCard, and install anything else you want.

The Core2 Duo Model with Mikrotik ROS ver 5 will handle about 1G to 1.5G of traffic with much trouble. With ROS ver 6 .. they will do about 25-30% more.

The i5 & i7 versions will handle 3-5Gig of traffic easily... and these do support 10G SFP+ Intel nic's with ROS 6.x

Nope, that wasn't me. I got mine probably a year ago. I pulled the flash and tried Vyatta on it, but Vyatta turned out to be a buggy unpleasant experience. RouterOS had an IPv6 OSPF bug where it ignores some LSAs that made it a showstopper. ~Seth

Totally agree that a routing box should be standalone for tons of reasons. Even separating network routing and call routing. It used to be that BSD's network stack was much better than Linux's under load. I'm not sure if this is still the case - I've never been put in the situation where the Linux kernel was at its limits. FWIW Jared Mauch <jared@puck.nether.net> wrote:

Have to agree on the below. I've seen too many devices be so integrated they do no task well, and can't be rebooted to troubleshoot due to everyone using them.

Jared Mauch

...

On Dec 26, 2013, at 10:55 AM, Andrew D Kirch <trelane@trelane.net> wrote:

Don't put all this in one box.

You can build using commodity hardware and get pretty good results. I've had really good luck with Supermicro whitebox hardware, and Intel-based network cards. The "Hot Lava Systems" cards have a nice selection for a decent price if you're looking for SFP and SFP+ cards that use Intel chipsets. There might be some benefits in going with something like FreeBSD, but I find that Linux has a lot more eyeballs on it making it much easier to develop for, troubleshoot, and support. There are a few options if you want to go the Linux route. Option 1: Roll your own OS. This takes quite a bit of effort, but if you have the tallant to do it you can generally get exactly what you want. Option 2: Use an established distribution. Vyatta doesn't seem to be doing much with its FOSS release "Vyatta Core" anymore, but the community has forked the GPL parts into "VyOS". I've been watching them pretty closely and helping out where I can; I think the project is going to win over a lot of people over the next few years. http://www.vyatta.org/ http://www.vyos.net/ The biggest point of failure I've experienced with Linux-based routers on whitebox hardware has been HDD failure. Other than that, the 100+ units I've had deployed over the past 3+ years have been pretty much flawless. Thankfully, they currently run an in-memory OS, so a disk failure only affects logging. If you want to build your own OS, I'll shamelessly plug a side project of mine: RAMBOOT http://ramboot.org/ RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process (added into initramfs tools, so kernel updates generate the right kernel automatically). Essentially, I use a kernel ramdisk instead of an HDD for the root filesystem and "/" is mounted on "/dev/ram1". The bootflash can be removed while the system is running as it's only mounted to save system configuration or update the OS. I haven't polished it up much, but there is enough there to get going pretty quickly. You'll also want to pay attention to the settings you use for the kernel. Linux is tuned as a desktop or server, not a router, so there are some basics you should take care of (like disabling ICMP redirects, increasing the ARP table size, etc). I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING or http://soucy.org/tmp/netfilter.txt (more recent, but includes firewall examples). Also a note of caution. I would stick with a longterm release of Linux. I've had good experience with 2.6.32, and 3.10. I'm eager to use some of the post-3.10 features, though, so I'm anxious for the next longterm branch to be locked in. If running a proxy server of any kind, you'll want to adjust TCP_TIMEWAIT_LEN in the header file and re-compile the kernel, else you'll run into ephemeral port exhaustion before you touch the limits of the CPU. I recommend 15 seconds (the default in Linux is 60). Routing-engine -wise. I currently have a large XORP 1.6 deployment because I have a need for multicast routing (PIM-SM), but XORP is very touchy and takes quite a bit of operational experience to avoid problems. Quagga has much more active development and eyeballs. BIRD is also very interesting. I like the model of BIRD a lot (more of a traditional daemon than trying to be a Cisco or Juniper clone). It doesn't seem to be as far along as Quagga though. One of the biggest advantages is the low cost of hardware allows you to maintain spare systems, reducing the time to service restoration in the event of failure. Dependability-wise, I feel that whitebox Linux systems are pretty much at Cisco levels these days, especially if running in-memory. On Thu, Dec 26, 2013 at 1:07 PM, jim deleskie <deleskie@gmail.com> wrote:

I've recently pushed a "large" BSD box to a load of over 300, for more then an hour, while under test, some things slowed a little, but she kept on working!

-jim

On Thu, Dec 26, 2013 at 1:59 PM, Shawn Wilson <ag4ve.us@gmail.com> wrote:

...

Totally agree that a routing box should be standalone for tons of reasons. Even separating network routing and call routing.

It used to be that BSD's network stack was much better than Linux's under load. I'm not sure if this is still the case - I've never been put in the situation where the Linux kernel was at its limits. FWIW

Jared Mauch <jared@puck.nether.net> wrote:

...

Have to agree on the below. I've seen too many devices be so integrated they do no task well, and can't be rebooted to troubleshoot due to everyone using them.

Jared Mauch

...

On Dec 26, 2013, at 10:55 AM, Andrew D Kirch <trelane@trelane.net> wrote:

Don't put all this in one box.

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net

Inline response exist, On 12/26/13, Ray Soucy <rps@maine.edu> wrote:

You can build using commodity hardware and get pretty good results.

I've had really good luck with Supermicro whitebox hardware, and Intel-based network cards. The "Hot Lava Systems" cards have a nice selection for a decent price if you're looking for SFP and SFP+ cards that use Intel chipsets.

I like the supermicro as well however we have a couple of IBM x3250 with 2 pcie v3 x8 that are begging for a intel network card.

There might be some benefits in going with something like FreeBSD, but I find that Linux has a lot more eyeballs on it making it much easier to develop for, troubleshoot, and support. There are a few options if you want to go the Linux route.

This is very important to consider. I would be speculating, or even worse, expecting the same type of community support from the BSD verse that I have been getting from the linux community.

Option 1: Roll your own OS. This takes quite a bit of effort, but if you have the tallant to do it you can generally get exactly what you want.

If Free/OpenBSD is ruled out, I could crack open the LFS project. You only have to do it once right? Or maybe just reach out to the gentoo community for a stripped version, and build outwards.

The biggest point of failure I've experienced with Linux-based routers on whitebox hardware has been HDD failure. Other than that, the 100+ units I've had deployed over the past 3+ years have been pretty much flawless.

SSD

Thankfully, they currently run an in-memory OS, so a disk failure only affects logging. If you want to build your own OS, I'll shamelessly plug a side project of mine: RAMBOOT

http://ramboot.org/

RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process (added into initramfs tools, so kernel updates generate the right kernel automatically). Essentially, I use a kernel ramdisk instead of an HDD for the root filesystem and "/" is mounted on "/dev/ram1".

The bootflash can be removed while the system is running as it's only mounted to save system configuration or update the OS.

I haven't polished it up much, but there is enough there to get going pretty quickly.

Ummm, if it's ok with the community, can you kindly elaborate :). I am not too fond of Debian since my horrible experience with Squeeze Desktop. I would maybe like to try this using the combination of SSD, in memory, and Gentoo?

You'll also want to pay attention to the settings you use for the kernel. Linux is tuned as a desktop or server, not a router, so there are some basics you should take care of (like disabling ICMP redirects, increasing the ARP table size, etc).

Totally strip it as much as possible. If anyone has a Gentoo stripped kernel config that they would like to share, please do :).

I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING or http://soucy.org/tmp/netfilter.txt (more recent, but includes firewall examples).

Will definitely look into all your sites.

Also a note of caution. I would stick with a longterm release of Linux. I've had good experience with 2.6.32, and 3.10. I'm eager to use some of the post-3.10 features, though, so I'm anxious for the next longterm branch to be locked in.

We are comfy with 3.4 right now...

One of the biggest advantages is the low cost of hardware allows you to maintain spare systems, reducing the time to service restoration in the event of failure. Dependability-wise, I feel that whitebox Linux systems are pretty much at Cisco levels these days, especially if running in-memory.

Really interested with the "in-memory", however, I would love to implement it using gentoo as mentioned above. Kind Regards, N.

Oh my bad. I did not mean it like that at all! I am more that capable of putting it together using gentoo instead of debian (a little pedagogy goes a long way). And if he would like, he can post the ISO on his webstie alongside the different distro. This is what I was leaning too... Please don't be offended. N.

In talking about RAMBOOT I also realized the instructions are out of date on the website. The "ramboot" boot target script was updated since I created the initial page to generate the correct fstab, and enable the root account, set a hostname, etc. so you can actually use the OS until you create a new image. I extracted the script from the initird to make it easier to grab: http://ramboot.org/download/RAMBOOT/RAMBOOT-pre0.2/SYSLINUX/initrd/scripts/r... Essentially, by adding a new "ramboot" script to "/usr/share/initramfs-tools/scripts" along side "nfs" and "local" it creates a new "boot=" target (since the init script looks for "scripts/${BOOT}". As mentioned on the website, the ramboot process needs a more complete version of busybox (for tar archive support) and the mke2fs tool added to "/usr/lib/initramfs-tools/bin/" so they will be available to the initrd. Once you configure networking (see the "INSTALL/setup_network" script) you can do an apt-get update and apt-get the packages you need from Ubuntu 12.04 LTS. Example starting point: apt-get install sudo apt-get install nano apt-get install ssh apt-get install vlan apt-get install bridge-utils On Thu, Dec 26, 2013 at 8:27 PM, Ray Soucy <rps@maine.edu> wrote:

The basic idea of RAMBOOT is typical in Embedded Linux development.

Linux makes use of multi-stage boot process. One of the stages involves using an initial ramdisk (initrd) to provide a base root filesystem which can be used to locate and mount the system root, then continue the boot process from there.

For an in-memory OS, instead of the boot process mounting a pre-loaded storage device with the expected root filesystem (e.g. your installed HDD), you modify it to:

1) Create and format a ramdisk. 2) Create the expected directory structure and system files for the root filesystem on that ramdisk.

The root filesystem includes the /dev directory and appropriate device nodes, the basic Linux filesystem and your init program.

The easy way to do that is just to have a TAR archive that you extract to the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that the archive can be read from storage (e.g. USB flash) more quickly.

Today, the initramfs in Linux handles a lot more than simply mounting the storage device. It performs hardware discovery and loads appropriate modules. As such the Debian project has a dynamic build system for initramfs that is run to build the initrd when a new kernel package is installed, it's called "initramfs-tools".

You can manually build your own initramfs using the examples on the RAMBOOT website, but the point of RAMBOOT is to make building an in-memory OS quick and simple.

RAMBOOT instead adds configuration to initramfs-tools so that each time a new initrd is generated, it includes the code needed for RAMBOOT.

The RAMBOOT setup adds handling of a new boot target called "ramboot" to the kernel arguments. This allows the same kernel to be used for a normal installation and remain unaffected, but when you add the argument "boot=ramboot" as a kernel option to the bootloader, it triggers the RAMBOOT process described above.

Having a common kernel between your development environment and embedded environment makes it much easier to test and verify functionality.

The other part of RAMBOOT is that it makes use of "Ubuntu Core". Ubuntu Core is a stripped down minimal (and they really do mean minimal) root filesystem for Embedded Linux development. It includes apt-get, though, so you can install all the packages you need from Ubuntu on the running system.

RAMBOOT then has a development script to make a new root filesystem archive with the packages you've installed as a baseline. This allows for you to boot a RAMBOOT system, install your desired packages and change system configuration files as desired, then build a persistent image of that install that will be used for future boots.

I also have the start of a script to remove unused kernel modules, and other files (internationalization for example) which add to the OS footprint.

You could build the root filesystem on your own (and compile all the necessary packages) but using Ubuntu Core provides a solid base and allows for the rapid addition of packages from the giant Ubuntu repository.

Lastly, I make use of SYSLINUX as a bootloader because my goal was to use a USB stick as the bootflash on an Atom box. Unfortunately, the Atom BIOS will only boot a USB device if it has a DOS boot partition, so GRUB was a no-go. The upside is that since the USB uses SYSLINUX and is DOS formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy new images or configuration to it easily.

For the boot device I make use of the on-board vertical USB socket on the system board (typical for most system boards these days) and a low-profile USB stick. I find the Verbatim "Store 'n' Go" 8GB USB stick ideally suited for this as it's less than a quarter-inch high after the USB adapter.

RAMBOOT as a project is in the very early stages, so you should be comfortable with Linux before you build a system on it. And I really feel it's more of an example than anything at this point.

There are several advantages though:

The most common point of failure on a Linux system is the storage device (either HDD or SSD). The biggest bottleneck in system performance is storage IO. Using a ramdisk eliminates both these concerns (in fact, even an Atom system has surprisingly great performance when run using a ramdisk).

The result is that you get a very reliable, high-performance system.

The other benefit to RAMBOOT is that the root filesystem is NOT persistent. This means that like a Cisco device, every boot of the system brings you to a known working state OS-wise. There are hundreds of system files in a Linux system; any one of which being modified could cause problems. For both security and availability concerns a lot of effort is invested in detecting changes to system files and avoiding them. With RAMBOOT the problem is easily avoided.

A minimal system can fit within a 512MB ramdisk. But with RAM being so cheap these days, I think even reserving up to 2GB of RAM for a ramdisk would be fine (e.g. for a 4-8GB system).

Here is the hardware configuration I originally started RAMBOOT for in 2011 (wanted to avoid the cost of an HDD):

$326.36 (shipping included): 1U rack-mount case Supermicro CSE-502L-200B Intel Atom D510 system board with dual Gigabit (Intel 82574L) Supermicro MBD-X7SPA-H-O 2GB RAM 8GB low-profile USB flash drive (which will connect to the internal USB port and be low enough to fit in the case) Verbatim "Store n Stay".

No HD; the system will boot off the 8GB flash into RAM and run the OS on a ramdisk.

Using the RAMBOOT release that's currently up, I can build a custom Linux in-memory OS in half a day. I can easily update packages for security updates from the Ubuntu project and re-generate a new, updated, image in less time than that. So the initial goal of being able to build something useful quickly was satisfied, at least. My attention has now moved on to building a configuration management system, similar to Vyatta or VyOS and building a real distribution. I was going to call it "Carrier-grade Linux" (cglinux.org), but given the momentum VyOS has I might try to help the VyOS community instead of doing something new on my own.

For what it's worth, I'm actually working with the VyOS project to try and incorporate some of the RAMBOOT ideas into VyOS as an install option for in-memory only.

If you make use of RAMBOOT I would love to hear about it. :-)

On Thu, Dec 26, 2013 at 4:22 PM, Nick Cameo <symack@gmail.com> wrote:

...

Inline response exist,

...

You can build using commodity hardware and get pretty good results.

I've had really good luck with Supermicro whitebox hardware, and Intel-based network cards. The "Hot Lava Systems" cards have a nice selection for a decent price if you're looking for SFP and SFP+ cards

On 12/26/13, Ray Soucy <rps@maine.edu> wrote: that

...

use Intel chipsets.

I like the supermicro as well however we have a couple of IBM x3250 with 2 pcie v3 x8 that are begging for a intel network card.

...

There might be some benefits in going with something like FreeBSD, but I find that Linux has a lot more eyeballs on it making it much easier to develop for, troubleshoot, and support. There are a few options if you want to go the Linux route.

This is very important to consider. I would be speculating, or even worse, expecting the same type of community support from the BSD verse that I have been getting from the linux community.

...

Option 1: Roll your own OS. This takes quite a bit of effort, but if

you

...

have the tallant to do it you can generally get exactly what you want.

If Free/OpenBSD is ruled out, I could crack open the LFS project. You only have to do it once right? Or maybe just reach out to the gentoo community for a stripped version, and build outwards.

...

The biggest point of failure I've experienced with Linux-based routers on whitebox hardware has been HDD failure. Other than that, the 100+ units I've had deployed over the past 3+ years have been pretty much flawless.

SSD

...

Thankfully, they currently run an in-memory OS, so a disk failure only affects logging. If you want to build your own OS, I'll shamelessly plug a side project of mine: RAMBOOT

http://ramboot.org/

RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process (added into initramfs tools, so kernel updates generate the right kernel automatically). Essentially, I use a kernel ramdisk instead of an HDD for the root filesystem and "/" is mounted on "/dev/ram1".

The bootflash can be removed while the system is running as it's only mounted to save system configuration or update the OS.

I haven't polished it up much, but there is enough there to get going pretty quickly.

Ummm, if it's ok with the community, can you kindly elaborate :). I am not too fond of Debian since my horrible experience with Squeeze Desktop. I would maybe like to try this using the combination of SSD, in memory, and Gentoo?

...

You'll also want to pay attention to the settings you use for the

kernel.

...

Linux is tuned as a desktop or server, not a router, so there are some basics you should take care of (like disabling ICMP redirects, increasing the ARP table size, etc).

Totally strip it as much as possible. If anyone has a Gentoo stripped kernel config that they would like to share, please do :).

...

I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING or http://soucy.org/tmp/netfilter.txt (more recent, but includes

firewall

...

examples).

Will definitely look into all your sites.

...

Also a note of caution. I would stick with a longterm release of Linux. I've had good experience with 2.6.32, and 3.10. I'm eager to use some

of

...

the post-3.10 features, though, so I'm anxious for the next longterm branch to be locked in.

We are comfy with 3.4 right now...

...

One of the biggest advantages is the low cost of hardware allows you to maintain spare systems, reducing the time to service restoration in the event of failure. Dependability-wise, I feel that whitebox Linux systems are pretty much at Cisco levels these days, especially if running in-memory.

Really interested with the "in-memory", however, I would love to implement it using gentoo as mentioned above.

Kind Regards,

N.

-- Ray Patrick Soucy Network Engineer University of Maine System

T: 207-561-3526 F: 207-561-3531

MaineREN, Maine's Research and Education Network www.maineren.net

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net

On 12/26/13, 9:24, Andrew D Kirch wrote:

If he can afford a 10G link... he should be buying real gear... I mean, look, I've got plenty of infrastructure horror stories, but lets not cobble together our own 10gbit solutions, please? At least get one of the new microtik CCR's with a 10gig sfp+? They're only a kilobuck... If you can't afford that I suggest you can't afford to be an ISP.

Unless all the money is going into the 10 gig link. ~Seth

On Fri, Dec 27, 2013 at 1:33 AM, <Valdis.Kletnieks@vt.edu> wrote:

On Thu, 26 Dec 2013 11:16:53 -0800, Seth Mattinen said:

...

On 12/26/13, 9:24, Andrew D Kirch wrote:

...

If he can afford a 10G link... he should be buying real gear... I mean, look, I've got plenty of infrastructure horror stories, but lets not cobble together our own 10gbit solutions, please? At least get one of the new microtik CCR's with a 10gig sfp+? They're only a kilobuck... If you can't afford that I suggest you can't afford to be an ISP.

Unless all the money is going into the 10 gig link.

If you've sunk so much into the 10G link (or anything else, for that matter) that you don't have a kilobuck to spare, you're probably undercapitalized to be an ISP.

I have issue with this line of thought. Granted, a router is built with custom ASICs and most network people understand IOS. However, this is where the benefit of a multi-thousand buck router ends. Most have limited RAM, so this limits the size of your policies and how many routes can be stored and the likes. With a computer with multi 10s or 100s of gigs of RAM, this really isn't an issue. Routers also have slow-ish processors (which is fine for pure routing since they are custom chips but) if you want to do packet inspection, this can slow things down quite a bit. You could argue that this is the same with iptables or pf. However, if you just offload the packets and analyze generally boring packets with snort or bro or whatever, packets flow as fast as they would without analysis. If you have multiple VPNs, this can start to slow down a router whereas a computer can generally keep up. ... And then there's the money issue. Sure, if you're buying a gig+ link, you should be able to afford a fully spec'd out router. However, (in my experience) people don't order equipment with all features enabled and when you find you need a feature, you have to put in a request to buy it and then it takes a month (if you're lucky) for it to be approved. This isn't the case if you use ipt/pf - if te feature is there, it's there - use it. And if a security flaw is found in a router, it might be fixed in the next month... or not. With Linux/BSD, it'll be fixed within a few days (at the most). And, if your support has expired on a router or the router is EOL, you're screwed. I think in the near future, processing packets with GPUs will become a real thing which will make doing massive real time deep packet inspection at 10G+ a real thing. Granted, your network people knowing IOS when they're hired is a big win for just ordering Cisco. But, I don't see that as a show stopper. Stating the scope of what a box is supposed to be used for and not putting endless crap on it might be another win for an actual router. However, this is a people/business thing and not a technical issue. Also, I'm approaching this as more of a question of the best tool for the job vs pure economics - a server is generally going to be cheaper, but I generally find a server nicer/easier to configure than a router.

On Fri, 27 Dec 2013 07:23:36 -0500 (EST), "Justin M. Streiner" <streiner@cluebyfour.org> said: > You end up combining some of the downsides of a hardware-based > router with some of the downsides of a server (new attack > vectors, another device that needs to be backed up, patched, and > monitored... Might be a good idea to back up, patch and monitor your routers too... Just sayin'

On 12/26/13, Alessandro Ratti <lord2y@gmail.com> wrote:

if you want build by yourself I will suggest gentoo and/or freebsd with bird (http://bird.network.cz/) for routing stuff (maybe with 10G nics).

Hello Alessandro, Any benchmarks of freebsd vs openbsd vs present day linux kern?

Chipsets and drivers matter a lot in the 1G+ range. I've had pretty good luck with the Intel stuff because they offload a lot in hardware and make open drivers available to the community. On Thu, Dec 26, 2013 at 7:48 PM, Olivier Cochard-Labbé <olivier@cochard.me>wrote:

Le 26 déc. 2013 22:02, "Nick Cameo" <symack@gmail.com> a écrit :

...

Any benchmarks of freebsd vs openbsd vs present day linux kern?

Hi,

Here are my own benchs using smallest packet size (sorry no Linux): http://dev.bsdrp.net/benchs/BSD.network.performance.TenGig.png

My conclusion: building a line-rate gigabit router (or a few rules ipfw firewall) is possible on commodity server without problem with FreeBSD. Building a 10gigabit router (this mean routing about 14Mpps) will be more complex in present day. Note: The packet generator used was the high-perf netmap pkg-gen, allowing me to generate about 13Mpps on this same hardware (under FreeBSD), but I'm not aware of forwarding tools that use netmap: There are only packet generator and capture tools available.

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net

I need a solution for everything except the last-mile customers. The customers are connected to a Zhone PON switch. From there they will arrive at our core switch as Q-in-Q vlans, one vlan per customer. I need a router that will do two full routing tables for our uplinks, a number of partial routing tables for our IX peers, IPv6 support, IPv4 proxy arp support and the ability to handle a large number of Q-in-Q vlans. And of course I will need two for redundancy. The uplinks, the links to edge switches and many of the IX peers are all 10 Gbit/s links. IPv4 proxy arp is especially important given the state of IPv4 exhaustion. Being a new ISP in the RIPE region, we only got 1024 IPs. When we run out of that initial assignment, we have to buy IP-addresses at a steep price. Therefore we can not afford to give each home a full IPv4 subnet. They will have to share the subnet with multiple other customers. This is achieved through proxy arp on the switch. We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital. Like Nick Cameo I have seriously considered going with a Linux solution. I know I can build it. I just don't know if I can make it stable enough or make it perform good enough. I am looking into an OpenFlow solution as a middle ground. It allows me to buy cheaper switches/routers. The servers will do the "thinking" but the actual work of moving packets is still done in hardware on the switches. OpenFlow supports controller fail over, so I will not go down with just one server crash. Poor performance on the servers will not affect customer traffic directly. Regards, Baldur On Fri, Dec 27, 2013 at 2:11 PM, Eugeniu Patrascu <eugen@imacandi.net>wrote:

On Fri, Dec 27, 2013 at 3:05 PM, Baldur Norddahl < baldur.norddahl@gmail.com> wrote:

...

On the topic of building a software router for an ISP, has anyone tried it using OpenFlow? The idea is to have a Linux server run BGP and a hardware switch to move the packets. The switch would be programmed by the Linux server using the OpenFlow protocol.

I am looking at the HP 5400 zl switches as the hardware platform and RouteFlow https://sites.google.com/site/routeflow/ to program the BGP rules.

One issue is that the HP switch will only allow a limited amount of rules to be processed in hardware (about 4096 rules I believe). Will this be enough to cover most of the traffic of a FTTH ISP on the fast path?

You want to use the switch for what ? To connect last-mile customers ? For L3 aggregation ? You want to run the switch as an edge router with limited BGP ? What's the exact use case you are thinking about ?

Eugeniu

On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands <fohdeesha@gmail.com> wrote:

On Dec 27, 2013 10:08 AM, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:

...

We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital.

Then you didn't ask for nearly enough capital.

Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links. What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket. Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost. Regards, Baldur

On Fri, Dec 27, 2013 at 6:48 PM, Justin M. Streiner <streiner@cluebyfour.org

wrote:

If you want to use servers as routers, that's your choice. I think what most people in the thread have been saying is not to use one server (or even a pair of servers) for everything. It's one thing if server XYZ goes down and some web services are offline. It's another thing entirely if that same server goes down and your entire business is offline.

You need at least two servers. Much of the functional separation can be achieved by using virtualisation: Two virtual servers per function, but only two physical servers. It is a lot cheaper to just get two beefy servers than to invest in expensive 10G routers. Later on it is easy to move the VMs to more servers if needed. The exception might be any function that moves packets. It is hard to get the necessary performance from dedicated hardware. Trying to do it from a VM will not help. In that case I would dedicate two physical machines to the purpose. However I am actually trying to avoid having the server move packets - I want to use an OpenFlow switch to do the brunt work. Running a BGP daemon on a VM to remote control an OpenFlow switch should not be a problem. Regards, Baldur

It seems to be a pretty "hot button" issue, but I feel that modern hardware is more than capable of pushing packets. The old wisdom of "only hardware can do it efficiently" is starting to prove untrue. 10G might still be a challenge (I haven't tested), but 1G is not even close to being an issue. Depending on the target for your deployment, it might make sense to whitebox a router or firewall instead of spending 20K on it. Especially if you're working with any kind of scale. TL;DR I think the backlash against anything but big iron routing is becoming an old way of thinking. On Fri, Dec 27, 2013 at 6:56 PM, Jon Sands <fohdeesha@gmail.com> wrote:

On 12/27/2013 4:23 PM, Matt Palmer wrote:

...

There *is* a world outside of Silly Valley, you know... a world where money doesn't flow like a mighty cascade from the benevolent wallets of vulture capitalists, into the waiting arms of every crackpot with an elevator pitch. - Matt

Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k.

-- Jon Sands

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net

In article <xs4all.CALFTrnNyr4V_Op0Rg4MGfN+8zX6474p80UpX3TM35y8kyYZLqA@mail.gmail.com> you write:

It seems to be a pretty "hot button" issue, but I feel that modern hardware is more than capable of pushing packets. The old wisdom of "only hardware can do it efficiently" is starting to prove untrue. 10G might still be a challenge (I haven't tested), but 1G is not even close to being an issue. Depending on the target for your deployment, it might make sense to whitebox a router or firewall instead of spending 20K on it. Especially if you're working with any kind of scale.

Yes well, but also remember that bandwidth is not everything. Packets per second is. And if you're going to provide internet connectivity to endusers, some of them /will/ get hit with DDOS attacks. With a hardware router you can survive that as long as the DDOS is not consuming all your bandwidth. A software router being bombarded with a few gigabits of 64 byte packets .. not so much. This is also the reason btw that you should look into shaping the outgoing bandwidth to each enduser, to prevent one of them being DDOSsed filling up the entire link he/she is on. Mike.

On 12/27/2013 8:18 PM, Baldur Norddahl wrote:

Brocade NetIron CER 2024F-4X goes for about $21k

As one last aside, if you're paying 21k, you're paying a little more than twice too much. Call Brocade and get yourself a real quote. I think peoples main point here is that any handful of thousand dollars you think you will be saving by going PC based, is going to dissapear the very first downtime you experience with zero support. Also, all the features you mention needing are standard features that any modern router should have no problem with. It's been a while but if I remember right you can terminate and route up to 4096 qinq based ve's on a single CER. If you're network topology has you terminating ten thousand qinq networks on a single box, I would take a step back and examine my network topology (if it were me, anyway) -- Jon Sands

On a side note, Q-in-Q support has been added to the recent 3.10 Linux kernel, configured using the "ip" command. It will be popping up in distributions "soon [tm]". Another interesting addition is IPv6 NAT (transparent redirect, prefix translation, etc). On Fri, Dec 27, 2013 at 8:18 PM, Baldur Norddahl <baldur.norddahl@gmail.com>wrote:

On Sat, Dec 28, 2013 at 12:56 AM, Jon Sands <fohdeesha@gmail.com> wrote:

...

Yes, and in that world, one should probably not start up a FTTH ISP when one has not even budgeted for a router, among a thousand other things. And if you must, you should probably figure out your cost breakdown beforehand, not after. Baldur, you mention $200k total to move 10gb with Juniper (which seems insanely off to me). Look into Brocades CER line, you can move 4x 10gbe per chassis for under 12k.

I was saying $100k for two Juniper routers total.

Perhaps we could get back on track, instead of trying to second guess what we did or did not budget for. You have absolute no information about our business plans.

The Brocade BR-CER-2024F-4X-RT-AC - Brocade NetIron CER 2024F-4X goes for about $21k and we need two of them. That is enough to buy a full year of unlimited 10G internet. And even then, we would be short on 10G ports.

It is not that we could not bring that money if that was the only way to do it. It is just that I have so many other things that I could spend that money on, that would further our business plans so much more.

I can not even say if the Juniper or the Brocade will actually solve my problem. I need it to route to ten of thousands of VLANS (Q-in-Q), both with IPv4 and IPv6. It needs to act as IPv6 router on every VLAN, and very few devices seems to like having that many IP-addresses assigned. It also needs to do VRRP and proxy arp for every VLAN.

The advantage of a software solution is that I can test it all before buying. Also to some limited degree, I am able to fix shortcomings myself.

Regards,

Baldur

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net

On Sat, Dec 28, 2013 at 3:50 AM, Brian Loveland <brian@aereo.com> wrote:

Interested on where you are buying transit at $1750/mo for full 10G ports ($0.175/meg)?

I did not that claim that. I said two times $21k divided by 12 = $3500 per month. Try he.net. Regards, Baldur

On Sat, Dec 28, 2013 at 4:10 AM, Randy Bush <randy@psg.com> wrote:

clearly you have a deep understanding of what you are doing, the market, what costs and capabilities are, and where to get what you need. now please remind me of what it was you were asking.

randy

I asked if anyone here has experience using OpenFlow in an ISP setting. Clearly the answer is no. Regards, Baldur

Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?)

not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.

I propose cage fighting at the next NANOG summit. Sent from my Mobile Device. -------- Original message -------- From: Randy Bush <randy@psg.com> Date: 12/27/2013 7:07 PM (GMT-09:00) To: Valdis.Kletnieks@vt.edu Cc: North American Network Operators' Group <nanog@nanog.org> Subject: Re: The Making of a Router

Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?)

not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.

At the very least, could we fight about something worthwhile? I¹m all for a good fight, and I¹m the first one to trigger the nuclear explosion.. But the subject matter of this peepee competition is tiring. I query the nanog gods frequently, sometimes you get useful feedback and sometimes you get a bunch of haters trying to piss on your parade. There is probably some validity on both sides, but a Friday night email fight should be reserved for those email blacklist douchebags.. Arguing over DIY routers being shittier than radical COTS equipment can be done off list. It¹s supposed to be Christmas.. Have a coke and a smile and STFU if you aren¹t going to add some value. I can¹t wait for this thread to fan back up on Monday morning when the normal(er) people read this. The first rule of fight club is.. You don¹t talk about fight club. On 12/27/13, 8:58 PM, "Shawn Wilson" <ag4ve.us@gmail.com> wrote:

This has gotten a bit ridiculous.

I was hoping someone could give technical insight into why this is good or not and not just "buy a box branded as a router because I said so or your business will fail". I'm all for hearing about the business theory of running an ISP (not my background or day job) but didn't think that's what the OP was asking about (and it didn't seem they were taking business suggestions very well anyway).

This thread started cool and about 10 posts in, started sucking.

Warren Bailey <wbailey@satelliteintelligencegroup.com> wrote:

...

I propose cage fighting at the next NANOG summit.

Sent from my Mobile Device.

-------- Original message -------- From: Randy Bush <randy@psg.com> Date: 12/27/2013 7:07 PM (GMT-09:00) To: Valdis.Kletnieks@vt.edu Cc: North American Network Operators' Group <nanog@nanog.org> Subject: Re: The Making of a Router

...

Right. And the point that others are trying to make clear is that if that $100K is half your capitalization, you have $200K - and that's nowhere near enought to cover all the stuff you're going to hit starting an ISP. (Hint - what's your projected burn rate for the first two months of business?)

not to worry. growth is not going to be an issue doing openflow due to today's tcam limits.

Chris Adams <cma@cmadams.net> wrote:

...

I was hoping someone could give technical insight into why this is good or not and not just "buy a box branded as a router because I said so or your business will fail". I'm all for hearing about the business

Once upon a time, Shawn Wilson <ag4ve.us@gmail.com> said: theory of running an ISP (not my background or day job) but didn't think that's what the OP was asking about (and it didn't seem they were taking business suggestions very well anyway).

There's been some technical insight here I would say. I'm a big Linux, Open Source, and Free Software advocate, and I'll use Linux-based systems for routing/firewalling small stuff, but for high speed/PPS, get a router with a hardware forwarding system (I like Juniper myself).

You can build a decently-fast Linux (or *BSD) system, but you'll need to spend a good bit of time carefully choosing motherboards, cards, etc. to maximize packet handling, possibly buying multiple of each to find the best working combination. Make sure you buy a full set of spares once you find a working combination (because in the PC industry, six months is a lifetime). Then you have to build your OS install, tweaking the setup, network stack, etc.

After that, you have to stay on top of updates and such (so plan for more reboots); while on a hardware-forwarding router you can mostly partition off the control plane, on a Linux/*BSD system, the base OS is the forwarding plane. Also, if something breaks, falls over under an attack, etc., you're generally going to be on your own to figure it out. Maybe you can Google the answer (and hope it isn't "that'll be fixed in kernel 3.<today's version+2>. Not saying that doesn't happen with router vendors (quoting RFCs at router engineers is "fun"), but it is IMHO less often.

The question becomes: what is your time worth? You could spend hundreds of hours going from the start to your satisfactory in-service router, and have a potentially higher upkeep cost. Can you hire somebody with all the same Linux/*BSD knowlege as yourself, so you are not on-call for your home-built router around the clock?

I've used Linux on all my computers for almost 20 years, I develop on Linux, and contribute to a Linux distribution. However, when I want to record TV to watch later, I plug in a TiVo, not build a MythTV box. There is a significant value in "just plug it in and it works", and if you don't figure your time investment (both up-front and on-going) into the cost, you are greatly fooling yourself.

I agree with all of this to some degree. IDK whether cost of ownership on a hardware router or a desktop is more or less - I jus haven't done the research. We use them at work and at home I have Cisco and Linksys gear (plus Linux doing some things the router could like DHCP) - go figure. I agree that some network cards and boards work better than others (and am partial to the Intel Pro cards - though I'm unsure if they're still the best). I would also hesitate to route that much traffic with a PC. Though, I have no technical reason for this bias. If you have hardware in production, you really should have a spare - whether we're talking servers, HDDs, batteries, or routers. Ie, that comment is not unique to servers. I also don't think warranty has any bearing on this - I've seen servers stay down for over a day because (both HP and Dell for their respective hardware) screwed up and the company didn't budget for a spare board and I've seen a third of a network be taken out because multiple switch ports just died. How much would a spare switch have cost compared to 50 people not online? At any rate, I'm interested in this because I've worked in both environments and haven't seen a large difference between the two approaches (never worked at an ISP or high bandwidth web environment though). I do like the PC router approach because it allows more versatility wrt dumping packets (no need to dig out that 10mbit dumb hub and throttle the whole network), I can run snort or do simple packet inspection with iptables (some routers can do this but most can't or require a license). So I'm sorta leaning to the PC router as being better - maybe not cheaper but better.

Pretty much what everyone else said. I'm a huge linux person, almost everything I use is linux, run full Myth set up etc, but I wouldn't use it for a high PPS situation like this. It's just asking for suffering later, at the worst possible times.

to paraphrase MO from many years ago (as i do not have a direct quote), do not use home appliances as production routers. there are niches where routeflow/vandervecken are usable. there are niches where quagga/bird/openbgp are useful. but not core/backbone of any non-trivial network. they might be usable for _slightly_ more than trivial if your business model values your time at zero. randy

Warren Bailey via<http://support.google.com/mail/bin/answer.py?hl=en&answer=1311182&ctx=mail> nanog.org : I propose cage fighting at the next NANOG summit.

Reminds me of some of the BOFs in 2000ish. Anyway, Ray's "TL;DR I think the backlash against anything but big iron routing is becoming an old way of thinking." should send a message to C&J that for other than "Tier 1" providers, a lot of people are looking for something else that pencils out better.. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474

I Am not trying to be inflammatory.. Why does everything has to be measured in Extremes ? e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ? I think we often confuse the performance of "Software" and Hardware as being monolithic, which is far from the truth. In today fast moving world, we don't even bother to look under the hood to see what is it that is being taxed.. the Software ? or Hardware ? or process of how we are doing what we want to get accomplished ? While it is easy to talk about the extremes, we often forget to point out to others that the 'Middle' is a huge place..... Regards Faisal Imtiaz Snappy Internet & Telecom ----- Original Message -----

From: "Eugeniu Patrascu" <eugen@imacandi.net> To: "Baldur Norddahl" <baldur.norddahl@gmail.com> Cc: nanog@nanog.org Sent: Friday, December 27, 2013 3:23:11 PM Subject: Re: The Making of a Router

On Fri, Dec 27, 2013 at 10:00 PM, Baldur Norddahl <baldur.norddahl@gmail.com

...

wrote:

...

On Fri, Dec 27, 2013 at 4:18 PM, Jon Sands <fohdeesha@gmail.com> wrote:

...

On Dec 27, 2013 10:08 AM, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:

...

We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital.

Then you didn't ask for nearly enough capital.

Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links.

What you should understand is not the fact that a 10G interface is expensive, but what you can do with that interface tends to get very expensive. If you want to move traffic from one interface to another, you can achieve this today with two physical interfaces on a Linux box. How many PPS ? Well, that's another story. You then want shaping, Q-in-Q and other stuff which consume a lot of resources even on dedicated hardware.

...

What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that. Even if I did not have a clear limit on my capital, I would be looking at avoiding paying that kind of money, because in the end the money comes out of my own pocket.

You can build your ISP without getting big routers but you need to cut back a little bit on your expectations about what you can in terms of features: - Do pool NAT for your users if they accept this. You can easily squeeze a lot of users on a single IP address. Downside is that if one of them does something bad, that IP might get blackholed on some providers and the rest will suffer. Also, you might want to take into consideration regulatory requirements like to know what users used what port to what destination for a certain number of months (in Europe regulations vary, but the smallest period is 6 months). - If you give them VoIP/IPTV then assign a VLAN for VOIP and another for IPTV and run it to all your users to their STBs and make use of IGMP snooping for Multicast traffic on all your switches - You can run full table BGP with Quagga on Linux (it worked for me when the DFZ was at around 270k prefixes, I assume it will work with 480k prefixes today) - also, do you really need full tables ?. Your IGP, if you don't run anything fancy should be a few tens of routes, that can be achieved with modest L3 switches that do 64/128 routes in hardware.

...

Everybody have critical services running on servers. DHCP, DNS, Radius and so on are all on servers and you will be down if these services are down. What is with the knee jerk reaction for suggesting that the BGP daemon could also be run on a server? There seems to be many advantages of doing it this way, and not all of them are related to cost.

For the sake of a good night sleep, you would want to separate all the services on different physical machines for redundancy/availability and load sharing.

Once you grow, you can move to more powerful and dedicated hardware for your networking needs.

Eugeniu

Fair point.. but in real life, isn't that true for everything... I say the same .... be familiar(honest awareness) with the limits (limitations) and capabilities of your specific solution, be it a 'dyi' or a commercial solution, before pushing it to the limit. Unless of course, you have factored in the ability to deal with the consequences. Most 'DYI' solutions, make the non-techy bean counters very nervous, and seeing a major 'name brand' label for some odd reason makes them real comfortable, ir-respective of the capabilities or function of either solution. If you have to answer to the bean counters, then this is a very valid point to be considered. :) Faisal Imtiaz Snappy Internet & Telecom 7266 SW 48 Street Miami, FL 33155 Tel: 305 663 5518 x 232 Help-desk: (305)663-5518 Option 2 or Email: Support@Snappytelecom.net ----- Original Message -----

From: "Jared Mauch" <jared@puck.nether.net> To: "Faisal Imtiaz" <faisal@snappytelecom.net> Cc: "Eugeniu Patrascu" <eugen@imacandi.net>, "North American Network Operators' Group" <nanog@nanog.org> Sent: Friday, December 27, 2013 4:04:12 PM Subject: Re: The Making of a Router

On Dec 27, 2013, at 3:37 PM, Faisal Imtiaz <faisal@snappytelecom.net> wrote:

...

e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ?

Those of us with experience know that when “something bad(tm)” happens, those features and “expensive silicon” start to show some ROI. Is it a full trade-off? Depends on the risks of your business and exposure.

You can get some inexpensive hardware to do fairly fancy features these days. That can be very good, but caries that risk. Make sure you evaluate it carefully.

- Jared

On Fri, 27 Dec 2013 20:37:52 +0000, Faisal Imtiaz said:

e.g. If someone says I need a 10g interface, why is it automatically assumed that the router is going to be running @ Full Line Rate ?

It may not be full line rate - but it's a pretty sure bet that you plan to run it at a fairly high percentage duty cycle, either right now or in the fairly near future. If you weren't, you'd have bought a 1G interface instead.

On Fri, 27 Dec 2013, Baldur Norddahl wrote:

Another told Nick Cameo that if he can afford a 10G link, he can afford Juniper. You could not be more wrong. The 10G uplink goes for $0 in initial fee and less than $4k / month with unlimited traffic. The Juniper gear is $100k up front for two routers able to handle the 10G links.

What I get from you guys is that in your opinion it is not possible to set up a small ISP without spending a ton on Juniper or Cisco. I am not buying that.

Small ISPs don't need 10gb transit connections. Even if you did want cisco gear capable of handling 10gb transit, it can be done on the secondary market for a fraction of that $100k figure. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

On Fri, Dec 27, 2013 at 10:18:47AM -0500, Jon Sands wrote:

On Dec 27, 2013 10:08 AM, "Baldur Norddahl" <baldur.norddahl@gmail.com> wrote:

...

We are an upstart and just buying the fancy Juniper switch times two would burn half of my seed capital.

Then you didn't ask for nearly enough capital.

There *is* a world outside of Silly Valley, you know... a world where money doesn't flow like a mighty cascade from the benevolent wallets of vulture capitalists, into the waiting arms of every crackpot with an elevator pitch. - Matt