Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT translation capability for a few public IP addresses to private addresses are the only requirements. Public or private responses are welcome! Thanks! Blake Pfankuch Network Engineer
would pfsense work for you? On Wed, Feb 10, 2010 at 4:12 PM, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT translation capability for a few public IP addresses to private addresses are the only requirements. Public or private responses are welcome!
Thanks! Blake Pfankuch Network Engineer
On Wed, 10 Feb 2010, Bryan Irvine wrote:
would pfsense work for you?
pfSense has ipv6, since it's essentially just a freebsd kernel with a layer on top. However, ipv6 support in the GUI is fairly minimal to non-existant, and I wouldn't recommend it if you really want to use ipv6. Mind you, I'm a fan of pfSense, it's just too bad it's not ipv6-friendly :) -- Peter van Arkel T: +31 623988844 | p.vanarkel@gmail.com RIPE: PvA63-RIPE | PGP: 0xA0991D6B
Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 >support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for >something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT >translation capability for a few public IP addresses to private addresses are the only requirements. Public or private >responses are welcome!
Not sure if they support IPV6 or not, but Imagestream makes Linux based routers, and everyone I've ever talked to that owns one has nothing bad to say about them.
On Wed, Feb 10, 2010 at 7:12 PM, Blake Pfankuch <bpfankuch@cpgreeley.com> wrote:
Anyone have some insight on a good dual stack Linux (or BSD) router distro?
Mikrotik RouterOS. It is based on Linux and a bit more feature-rich than some of the linux router distros I've tried such as IPCop. Licenses costs a few bucks but its worth it IMHO. Regards, Mark
On 2010-02-10 at 17:12:28 -0700, Blake Pfankuch wrote:
Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT translation capability for a few public IP addresses to private addresses are the only requirements. Public or private responses are welcome!
I'm not sure if the GUI is a requirement, but I'm a huge fan of Shorewall. It has support for both v4 and v6 along along with the usual router requirements. Since it's just a linux box with a few iptables rules, you can easily load openvpn, ipsec, quagga, etc... It's all text files and a 'shorewall start|stop|check' script. If you want something with a GUI, pfSense is your best bet, or you could use something like fwbuilder to build your iptables rules. -A
I actually spaced about vyatta when I wrote this email. I have since been forcefully reminded. About 30 times :) In the process of testing it, however my main concern is some of the complexity of the config options. The GUI is a welcome addition since 4, however I still find it a bit lacking. I may go the vyatta route anyway based only on my sheer curiosity and future possible needs. Thank you all for your input! -----Original Message----- From: Carlos A. Carnero Delgado [mailto:carloscarnero@gmail.com] Sent: Wednesday, February 10, 2010 9:19 PM To: Blake Pfankuch Cc: nanog@nanog.org Subject: Re: Linux Router distro's with dual stack capability Have you checked Vyatta? HTH, Carlos.
On Wed, 2010-02-10 at 17:12 -0700, Blake Pfankuch wrote:
Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT translation capability for a few public IP addresses to private addresses are the only requirements. Public or private responses are welcome!
We are having moderate success with IPv6 on Vyatta, but we have seen neighbour discovery glitches in the current production images. The prerelease subscription code crashes on our vyatta appliances, so we haven't tested that yet. William
Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See the freebsd-isp list. -Jack Carrozzo On Thu, Feb 11, 2010 at 3:23 AM, William Pitcock <nenolod@systeminplace.net> wrote:
On Wed, 2010-02-10 at 17:12 -0700, Blake Pfankuch wrote:
Anyone have some insight on a good dual stack Linux (or BSD) router distro? Currently using IPCop but it lacks ipv6 support. I've used SmoothWall Express but not in some time and not sure how well it works with IPv6. Not looking for something huge, just something for the equivalent of a small branch office. Site to Site VPN support and NAT translation capability for a few public IP addresses to private addresses are the only requirements. Public or private responses are welcome!
We are having moderate success with IPv6 on Vyatta, but we have seen neighbour discovery glitches in the current production images.
The prerelease subscription code crashes on our vyatta appliances, so we haven't tested that yet.
William
Hi, On Thu, 2010-02-11 at 13:05 -0500, Jack Carrozzo wrote:
Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See the freebsd-isp list.
FreeBSD's network stack chokes up in DDoS attacks due to interrupt flooding. We used to use FreeBSD for firewalling and basic routing, but when noticing that we had horizontal scalability (e.g. a Celeron 667mhz performed nearly as well as a dual dual-core Xeon system when DDoS attacks happened), we switched to Vyatta, and generally have not looked back. William
On Thu, Feb 11, 2010 at 04:12:03PM -0600, William Pitcock wrote:
On Thu, 2010-02-11 at 13:05 -0500, Jack Carrozzo wrote:
Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See the freebsd-isp list.
FreeBSD's network stack chokes up in DDoS attacks due to interrupt flooding. We used to use FreeBSD for firewalling and basic routing, but when noticing that we had horizontal scalability (e.g. a Celeron 667mhz performed nearly as well as a dual dual-core Xeon system when DDoS attacks happened), we switched to Vyatta, and generally have not looked back.
Have you tried using FreeBSD's polling mode instead of interrupt mode? No experience with it myself, but it sounds cool: http://info.iet.unipi.it/~luigi/polling/
Date: Thu, 11 Feb 2010 18:20:13 -0500 From: Chuck Anderson <cra@WPI.EDU>
On Thu, Feb 11, 2010 at 04:12:03PM -0600, William Pitcock wrote:
On Thu, 2010-02-11 at 13:05 -0500, Jack Carrozzo wrote:
Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See the freebsd-isp list.
FreeBSD's network stack chokes up in DDoS attacks due to interrupt flooding. We used to use FreeBSD for firewalling and basic routing, but when noticing that we had horizontal scalability (e.g. a Celeron 667mhz performed nearly as well as a dual dual-core Xeon system when DDoS attacks happened), we switched to Vyatta, and generally have not looked back.
Have you tried using FreeBSD's polling mode instead of interrupt mode?
No experience with it myself, but it sounds cool:
Polling is excellent for low speed lines, but for Gig and faster, most newer interfaces support interrupt coalescing. This easily resolves the issue in hardware as interrupts are only issued when needed but limited to a reasonable rate, Polling does not use interrupts, but consumes system resources regardless of traffic. FreeBSD has supported polling for a long time (V6?) and interrupt coalescing since some release of V7. (Latest release is V8.) -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
On Thu, Feb 11, 2010 at 03:46:13PM -0800, Kevin Oberman wrote:
Polling is excellent for low speed lines, but for Gig and faster, most newer interfaces support interrupt coalescing. This easily resolves the issue in hardware as interrupts are only issued when needed but limited to a reasonable rate, Polling does not use interrupts, but consumes system resources regardless of traffic.
FreeBSD has supported polling for a long time (V6?) and interrupt coalescing since some release of V7. (Latest release is V8.)
I'm pretty sure it's been around for a lot longer than that. I seem to recall playing with both back in 4.x. Of course interrupt coalescing is mostly a function of the NIC (though some driver involvement is required to take advantage of it), so the quality of the implementations have varied significantly over the years. The first generation GE NICs which offered it didn't do a particularly good job with it though, so for example it was still possible to cripple a box with high interrupt rates while the same box would be perfectly fine with polling. That said, I think your use case for polling is backwards. As you say, "normally" the NIC fires off an interrupt every time a packet is received, and the kernel stops what it is doing to process the new packet. On a low speed (or at least low traffic) interface this isn't a problem, but as the packet/sec rate increases the amount of time wasted as interrupt processing "overhead" becomes significant. For example, even a GE interface is capable of doing 1.488 million packets/sec. By switching to a polling based model, you switch off the interrupt generation completely and simply check the NIC for new packets a set rate (for example, 1000 times/sec). This gives you a predictable and consistent CPU use, so even if you had 1.488M/s interrupts coming in you would still only be checking 1000 times/sec. If you did less than 1000pps it would be a net increase in CPU, but if you do more (or ever risk doing more, such as during a DoS attack) it could be a net benefit. This is makes the most sense for people doing a lot of traffic regardless. Of course the downside is higher latency, since you're delaying the processing of the packet by some amount of time after it comes in. In the 1000 times/sec example above, you could be delaying processing of your packet by up to 1ms. For most applications this isn't enough to cause any harm, but it's something to keep in mind. Interrupt coalescing works around the problem of large interrupt rates by simply having the NIC limit the number of interrupts it generates under load, giving you the benefits of low-latency processing and low-interrupt rate under high load. I haven't played with this stuff in many many years, so I'm sure modern interrupt coalescing is much better than it used to be, and the extra work of configuring polling and dealing with the potential latency/jitter implications isn't worth the benefits for most people. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Also IIRC you can tune the hash cache / tree algorithm - ie if your traffic is mostly a few addresses then the default prefix search is fine (with the caching) but for more sparse traffic as you'd see at an edge, disabling the cache and using the other algo proved a lot faster. There's a paper on this I saw a few years ago, will forward if I find it. -Jack Carrozzo On Thu, Feb 11, 2010 at 7:41 PM, Richard A Steenbergen <ras@e-gerbil.net> wrote:
On Thu, Feb 11, 2010 at 03:46:13PM -0800, Kevin Oberman wrote:
Polling is excellent for low speed lines, but for Gig and faster, most newer interfaces support interrupt coalescing. This easily resolves the issue in hardware as interrupts are only issued when needed but limited to a reasonable rate, Polling does not use interrupts, but consumes system resources regardless of traffic.
FreeBSD has supported polling for a long time (V6?) and interrupt coalescing since some release of V7. (Latest release is V8.)
I'm pretty sure it's been around for a lot longer than that. I seem to recall playing with both back in 4.x. Of course interrupt coalescing is mostly a function of the NIC (though some driver involvement is required to take advantage of it), so the quality of the implementations have varied significantly over the years. The first generation GE NICs which offered it didn't do a particularly good job with it though, so for example it was still possible to cripple a box with high interrupt rates while the same box would be perfectly fine with polling.
That said, I think your use case for polling is backwards. As you say, "normally" the NIC fires off an interrupt every time a packet is received, and the kernel stops what it is doing to process the new packet. On a low speed (or at least low traffic) interface this isn't a problem, but as the packet/sec rate increases the amount of time wasted as interrupt processing "overhead" becomes significant. For example, even a GE interface is capable of doing 1.488 million packets/sec.
By switching to a polling based model, you switch off the interrupt generation completely and simply check the NIC for new packets a set rate (for example, 1000 times/sec). This gives you a predictable and consistent CPU use, so even if you had 1.488M/s interrupts coming in you would still only be checking 1000 times/sec. If you did less than 1000pps it would be a net increase in CPU, but if you do more (or ever risk doing more, such as during a DoS attack) it could be a net benefit. This is makes the most sense for people doing a lot of traffic regardless.
Of course the downside is higher latency, since you're delaying the processing of the packet by some amount of time after it comes in. In the 1000 times/sec example above, you could be delaying processing of your packet by up to 1ms. For most applications this isn't enough to cause any harm, but it's something to keep in mind. Interrupt coalescing works around the problem of large interrupt rates by simply having the NIC limit the number of interrupts it generates under load, giving you the benefits of low-latency processing and low-interrupt rate under high load. I haven't played with this stuff in many many years, so I'm sure modern interrupt coalescing is much better than it used to be, and the extra work of configuring polling and dealing with the potential latency/jitter implications isn't worth the benefits for most people. :)
-- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
William Pitcock wrote:
FreeBSD's network stack chokes up in DDoS attacks due to interrupt flooding. We used to use FreeBSD for firewalling and basic routing, but when noticing that we had horizontal scalability (e.g. a Celeron 667mhz performed nearly as well as a dual dual-core Xeon system when DDoS attacks happened), we switched to Vyatta, and generally have not looked back.
William
Which version of FreeBSD and how much traffic/pps? I believe that there has been significant improvements to the networking stack in recent versions of FreeBSD, plus there are also a lot of sysctl tunables which can significantly improve networking performance. I have a hard time believing that the networking performance of recent versions of FreeBSD would not be competitive in comparison to other unixes. -M
Jack Carrozzo wrote:
Lots of people roll FreeBSD with Quagga/pf/ipfw for dual stack. See the freebsd-isp list.
Raises hand. I do, on these boxes: http://www.mikrotikrouter.net/ Steve
I was wondering what kind of experience the nanog userbase has had with these two packages. Thanks -- Jason Fried This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message. Any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited. [v.E.1]
Fried, Jason (US - Hattiesburg) wrote:
I was wondering what kind of experience the nanog userbase has had with these two packages.
Quagga++. I've never tried the other. I use Quagga for OSPF, OSPFv3 and BGP (IPv4 and IPv6). With a bit of trickery, it fits in nicely with my RANCID setup, and what I like best is that it (mostly) follows Cisco's command convention. There are also very active developer and user mailing lists. For the most part, I wouldn't know if I was writing a config for a Cisco or for a Quagga box. fwiw, I've also heard good things about bgpd(8) and ospfd(8), but I haven't tried those either...zebra/Quagga just stuck. Steve
http://www.uknof.org.uk/uknof15/ Has quite a few talk about Quagga/Bird as they are used as route servers in Europe. For a route server use, BGP under very high number of peers, it seems bird now behave better than anything else. so for "normal" use, it would seems that whatever you pick will work but quagga is surely the most deployed. Thomas On 12 Feb 2010, at 22:51, Steve Bertrand wrote:
Fried, Jason (US - Hattiesburg) wrote:
I was wondering what kind of experience the nanog userbase has had with these two packages.
Quagga++.
I've never tried the other.
I use Quagga for OSPF, OSPFv3 and BGP (IPv4 and IPv6). With a bit of trickery, it fits in nicely with my RANCID setup, and what I like best is that it (mostly) follows Cisco's command convention.
There are also very active developer and user mailing lists.
For the most part, I wouldn't know if I was writing a config for a Cisco or for a Quagga box.
fwiw, I've also heard good things about bgpd(8) and ospfd(8), but I haven't tried those either...zebra/Quagga just stuck.
Steve
On 13/02/2010, at 11:51 AM, Steve Bertrand wrote:
fwiw, I've also heard good things about bgpd(8) and ospfd(8), but I haven't tried those either...zebra/Quagga just stuck.
OpenBGPd would be great for a public route server at an IX. It's not so great for use in a network unless you run it on OpenBSD - FreeBSD has no metric attribute in it's routing tables, so next-hop IGP metric cannot be compared as the two daemons do not communicate directly at all. If you're on anything other than OpenBSD, I recommend Quagga. I can't comment on BIRD as I have no experience with it yet. XORP is also interesting, it's a more JunOS like interface. It's also some quite heavy C++, so running it on the tiny Soekris boxes that I had meant it wouldn't work for me. If you can spare the CPU and RAM then give XORP a go. -- Nathan Ward
On 13.02.2010 02:01 Nathan Ward wrote
On 13/02/2010, at 11:51 AM, Steve Bertrand wrote:
fwiw, I've also heard good things about bgpd(8) and ospfd(8), but I haven't tried those either...zebra/Quagga just stuck.
OpenBGPd would be great for a public route server at an IX.
Be cautious when doing filtering. bgpctl will hang for minutes, even hours. Otherwise OpenBGPD seems to be very performant. Quagga does not really behave well with lots of peers (lots >> 200), but there will be an optimized route server version soon. BIRD seems to do fine. Best regards, Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arnold@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333
Quagga does not really behave well with lots of peers (lots >> 200), but there will be an optimized route server version soon.
This was discussed today at Linx 68. Linx is very pleased with Bird - they could not get Quagga working due to load issues. With large numbers of peers, the update processing can cause the program to hit his peer HoldTime Timer (with a domino's effect as well). EuroIX is sponsoring some work on Quagga to get the KeepAlive management moved into a separate Thread. During the discussion, a developers of Bird said that their filtering code _may_ still have bugs (when performing community based filtering). Thomas
On 16/02/2010 19:47, Thomas Mangin wrote:
During the discussion, a developers of Bird said that their filtering code _may_ still have bugs (when performing community based filtering).
medium-long term, community based route-server filtering has no future. There will be two reasons for its demise: it cannot easily accommodate asn32 and it does not allow predetermined filtering and hence sane loc-rib instance management. I touched on this briefly at my uknof talk recently, but long term, the writing is on the wall for this sort of filtering. Nick
medium-long term, community based route-server filtering has no future. There will be two reasons for its demise: it cannot easily accommodate asn32 and it does not allow predetermined filtering and hence sane loc-rib instance management.
i would add decades of bad anecdotes where the data plane is not congruent with the control plane. in general, when plane N is not congruent with plane N+1, management and debugging are problematic. randy
As in SS7, which has successfully managed the phone system for decades, where the control and data plane are explicitly separated? There's significant theoretical work, backed up with lots of practical experience connecting a lot more nodes in real time in a lot more places than the Internet currently does, that posits that the control and forwarding plane should actually ALWAYS be separate, and control higher priority, so that state management converges faster than the dataflows. I'd like to see the countervailing, peer reviewed, references.
-----Original Message----- From: Randy Bush [mailto:randy@psg.com] Sent: Tuesday, February 16, 2010 5:19 PM To: Nick Hilliard Cc: NANOG list Subject: Re: BIRD vs Quagga
medium-long term, community based route-server filtering has no future. There will be two reasons for its demise: it cannot easily accommodate asn32 and it does not allow predetermined filtering and hence sane loc-rib instance management.
i would add decades of bad anecdotes where the data plane is not congruent with the control plane. in general, when plane N is not congruent with plane N+1, management and debugging are problematic.
randy
Good point regarding non-congruence not necessarily meaning non-separation, and touché on margins (by which I presume you mean SS7 has massive overprovisionining for average traffic). However, the fact remains, it has proven itself to work for a lot longer, and a for much larger subscriber base, with far fewer systemic failures (especially on a per subscriber/expected availability basis), than the current Internet. I notice you didn't answer my request for the peer reviewed literature to support your assertion. To support mine I give (there are hundreds in the literature): Gopel (Nokia): HSN and Multimedia Apps, 5th Conf, IEEE, 2002: Print ISBN: 0-7803-7600-5 PP 161-166 Ramjee et al (Lucent Bell Labs): Comsware 2006: Print ISBN: 0-7803-9575-1 PP 1-10 Khalios et al (City College of NY): IEEE Globecom 2003: Print ISBN: 0-7803-7974-8 PP 3984-3989 Never mind all of Shannon's work and everything bell labs did in developing digital switching. You can always have control traffic follow the same path in a different channel, so you get the same effect of physical interruption, and therefore the topography alerting of an interrupted link, without the issues of pathological traffic in the bearer channel interrupting your control traffic (as with ISDN subscriber trunks).
-----Original Message----- From: Randy Bush [mailto:randy@psg.com] Sent: Tuesday, February 16, 2010 7:56 PM To: Tomas L. Byrnes Cc: Nick Hilliard; NANOG list Subject: Re: BIRD vs Quagga
As in SS7, which has successfully managed the phone system for decades, where the control and data plane are explicitly separated?
and has such wonderful margins
and, btw, separation is not necessarily non-congruence
On Tue, Feb 16, 2010 at 10:55 PM, Randy Bush <randy@psg.com> wrote:
As in SS7, which has successfully managed the phone system for decades, where the control and data plane are explicitly separated?
and has such wonderful margins
and, btw, separation is not necessarily non-congruence
<cough> decisions per second rate</cough> <cough> path information per 'call' or 'decision'</cough> ...these aren't like examples... (apples to oranges discussion, move along, nothing to see here)
On 2010-02-16, at 19:53, Tomas L. Byrnes wrote:
There's significant theoretical work, backed up with lots of practical experience connecting a lot more nodes in real time in a lot more places than the Internet currently does, that posits that the control and forwarding plane should actually ALWAYS be separate, and control higher priority, so that state management converges faster than the dataflows.
I'd like to see the countervailing, peer reviewed, references.
I have no shortage of anecdotes where a non-trivial layer-2 topology at an exchange point has left my router and provider X's router both able to talk to a route server, but unable to talk to each other directly. Since the NEXT_HOP on routes we each learnt from the route server pointed at an address we couldn't talk to, the result was a black hole. So while your theoretical work might well have substantial merit, its application to the example at hand seems potentially lacking. I am somewhat intrigued at this network you mention with which people have practical experience that has more nodes than the Internet does, though. That'd be quite a network. Joe
On Wed, Feb 17, 2010 at 12:50 AM, Joe Abley <jabley@hopcount.ca> wrote:
I am somewhat intrigued at this network you mention with which people have practical experience that has more nodes than the Internet does, though. That'd be quite a network.
what's the current estimate on PSTN endpoints? 2-3B globally? is that more/less than the IP/Internet world? I bet it's, at this time, fairly close to the same number. Potential references: <http://www.funtrivia.com/askft/Question25278.html> thinks the ITU estimated ~33% of 1.4b devices were mobile phones (in 2002) <http://en.wikipedia.org/wiki/List_of_countries_by_number_of_mobile_phones_in_use> wikipedia thinks a total mobile phone market is ~4.1b phones with ~60% global penetration. <http://en.wikipedia.org/wiki/List_of_countries_by_number_of_telephone_lines_in_use> number of PSTN lines globally ~= 1.3b So a total across the wikipedia links says ~5.5b phone devices. That seems significantly more than IP devices. I did not, obviously, factor in phones which are also IP devices (your iPhone type thingy) -Chris
On 2010-02-16, at 22:00, Christopher Morrow wrote:
On Wed, Feb 17, 2010 at 12:50 AM, Joe Abley <jabley@hopcount.ca> wrote:
I am somewhat intrigued at this network you mention with which people have practical experience that has more nodes than the Internet does, though. That'd be quite a network.
what's the current estimate on PSTN endpoints? 2-3B globally?
True, I was thinking about packet-switched networks, since it always seems to me that a circuit-switched network is only as big at any time as the number of circuits that exist, not the number of possible termination points for circuits. Quite possibly I'm just smoking crack, however. Joe
On Wed, Feb 17, 2010 at 1:03 AM, Joe Abley <jabley@hopcount.ca> wrote:
On 2010-02-16, at 22:00, Christopher Morrow wrote:
On Wed, Feb 17, 2010 at 12:50 AM, Joe Abley <jabley@hopcount.ca> wrote:
I am somewhat intrigued at this network you mention with which people have practical experience that has more nodes than the Internet does, though. That'd be quite a network.
what's the current estimate on PSTN endpoints? 2-3B globally?
True, I was thinking about packet-switched networks, since it always seems to me that > a circuit-switched network is only as big at any time as the number of circuits that exist,
I almost made a comment that the PSTN is really (as far as routing is concerned) lots of disparate networks with no 'global view' of the problem. It can be argued (and randy likely will, or bmanning even) that the Internet doesn't have a single view either... There's loop protection in the routing data, which the PSTN doesn't really have. (which is a side problem)
not the number of possible termination points for circuits. Quite possibly I'm just smoking crack, however.
doubtful... probably me missing a terminology collision :( It also depends on how Tomas was defining his problem. I still say the PSTN and Internet are apples/oranges in so many ways you can't use them in an comparision. -chris
On Tue, 2010-02-16 at 21:50 -0800, Joe Abley wrote:
On 2010-02-16, at 19:53, Tomas L. Byrnes wrote:
There's significant theoretical work, backed up with lots of practical experience connecting a lot more nodes in real time in a lot more places than the Internet currently does, that posits that the control and forwarding plane should actually ALWAYS be separate, and control higher priority, so that state management converges faster than the dataflows.
I'd like to see the countervailing, peer reviewed, references.
I have no shortage of anecdotes where a non-trivial layer-2 topology at an exchange point has left my router and provider X's router both able to talk to a route server, but unable to talk to each other directly. Since the NEXT_HOP on routes we each learnt from the route server pointed at an address we couldn't talk to, the result was a black hole.
I have similar anecdotes... and I was on the side of running the route-servers. This gets to be a tough nut to crack especially if you happen to have multiple RSes on opposite ends of a layer2 failure (a case where intended redundancy resulted in unintended new failure modes). The best solution we came up with at the time was to add some control knobs to rsd in order to allow us to quickly take down the BGP session to the peer on the falsely advertising RS. Figuring out which third-party negotiated "pairwise peering" was being effected during a switch fabric breakage was done manually at the time and not all that accurate nor of course was it expedient. We attempted to automate that part without too much success. -- /*=================[ Jake Khuon <khuon@NEEBU.Net> ]=================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | -------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==================================================================*/
On Tue, 2010-02-16 at 23:03 -0800, Jake Khuon wrote:
The best solution we came up with at the time was to add some control knobs to rsd in order to allow us to quickly take down the BGP session to the peer on the falsely advertising RS.
Sorry... this was poorly worded. We did not actually tear down the BGP sessions. I should have placed quotes around "BGP session". What we did was virtually nuked the "view" in the RS of the pairwise peering thus forcing a BGP withdrawal to the effected peers of the RS and hopefully leaving only valid third-party views intact. Again, the greatest problem was detection and modeling. -- /*=================[ Jake Khuon <khuon@NEEBU.Net> ]=================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | -------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==================================================================*/
On 17/02/2010 01:19, Randy Bush wrote:
i would add decades of bad anecdotes where the data plane is not congruent with the control plane. in general, when plane N is not congruent with plane N+1, management and debugging are problematic.
I've always maintained publicly and privately that route servers are not for everyone. A good rule of thumb is: "using a route collector for peering is probably suitable for your network unless you know why it isn't". Interesting choice of wording: "decades of bad anecdotes". Does that mean anecdata? Nick
During the discussion, a developers of Bird said that their filtering code _may_ still have bugs (when performing community based filtering).
Someone rightly pointed to me that the commenter was not a BIRD developer .. my mistake sorry. I will "recall" my statement until I can watch to the webcast archive of the meeting. Thomas
On 13 Feb 2010, at 01:01, Nathan Ward wrote:
On 13/02/2010, at 11:51 AM, Steve Bertrand wrote:
fwiw, I've also heard good things about bgpd(8) and ospfd(8), but I haven't tried those either...zebra/Quagga just stuck. OpenBGPd would be great for a public route server at an IX.
Nathan has made a good point. Deploying them in an IX environment, with features like per-peer RIBs, very complex filtering, and the numbers of peers you might expect on a route-server environment, is a very different beast to (and more complicated than) deploying them in a network edge/forwarding role. In a forwarding role, the underlying OS's features and the robustness of the daemon under load matters in different ways. So what's best ? I have used all three in a forwarding role and found BIRD on Debian a pretty solid combination. I found OpenBGPd on OpenBSD a pain to use - it converged really slowly and bgpctl seemed to lock up for a while after startup in an environment with *many* peers, and the behaviour with ospf3 used to change quite a lot. Quagga on Linux or FreeBSD seemed to work ok, and the interface will be quite familiar to Cisco users. Using all three as an injector for Anycast or similar leads to quite similar outcomes. However you might find ExaBGP more lightweight in this role - see http://bgp.exa.org.uk/ - do check it out. This has an interface which will feel extremely comfortable to Juniper users. You should still go to the IX Route-Servers panel to learn more about the software in question :-) And its really very good research being presented - but I am biased here. Best wishes Andy -- // www.netsumo.com // Professional network engineering consultancy // // uk ddi: +44(0)20 7993 1702 // us ddi: (415) 520 3589 //
There will be a presentation comparing BIRD with Quagga at NANOG week after next in Austin. II believe it will be a part of the Route Servers Track on Monday afternoon. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
participants (26)
-
Aaron C. de Bruyn -
Andy Davidson -
Arnold Nipper -
Blake Pfankuch -
Bryan Irvine -
Carlos A. Carnero Delgado -
Christopher Morrow -
Chuck Anderson -
Fried, Jason (US - Hattiesburg) -
Gregory J. Boehnlein -
Jack Carrozzo -
Jake Khuon -
Joe Abley -
Kevin Oberman -
Mark Price -
Marty Anstey -
Matthew Palmer -
Nathan Ward -
Nick Hilliard -
Peter van Arkel -
Randy Bush -
Richard A Steenbergen -
Steve Bertrand -
Thomas Mangin -
Tomas L. Byrnes -
William Pitcock