advice on BGP + CARP setup on FreeBSD
Hi fellow operators, We are slowly and carefully joining the fray of autonomous systems and started announcing our own netblock, a first test that started last week. So far, things are going well, but before going further along this setup, I would be curious to hear experience from other operators about the plan we are thinking of deploying. Our requirements: * free software, as much as possible * inexpensive * using existing operating system expertise (FreeBSD or Debian) So far, we have: * our own ASN * a /21 assigned by ARIN * two uplinks deployed (Netelligent and Cogent) * Netelligent announces 3 /24 netblocks for us * we announce the last /24 through a BGP link with cogent We have some horrible diagrams describing the setup here: https://wiki.koumbit.net/RoutingService/RoadMap As you can see, the uplinks are connected directly into a switch, in two separate VLANs. The reason for this is we want to be able to hotswap the routers in case of a hardware failure, but we have understood from Cogent's documentation that this is not a good practice because the links appears up even if the router goes down. What is your opinion on this? Also, we currently testing OpenBGPd for the announcements, and we are very pleased with it. The syntax is clear and it just works, with minimal memory usage: https://wiki.koumbit.net/OpenBgpdMaintenance#Checking_memory_usage However, this seems to be a fairly exotic platform, most people running BGP with Cisco, Juniper or, in some cases Quagga or Bird for Linux machines. Are there recmomendations on using OpenBGP in production? Good / bad experiences? How many people are running Linux routers vs dedicated Cisco/Juniper/etc routers? Finally, we are likely to complete this setup with a CARP (the free equivalent of VRRP) on the inside of the network. FreeBSD can apparently group interfaces and communicate with OpenBGPd - did anyone deploy such a thing here? What are your experiences or advice? Thanks for any advice, A. -- Sous un gouvernement qui emprisonne injustement, la place de l’homme juste est aussi en prison. - La désobéissance civile, Henry David Thoreau
On 05/11/2013 19:31, Antoine Beaupré wrote:
Our requirements:
* free software, as much as possible * inexpensive * using existing operating system expertise (FreeBSD or Debian)
You need to make a decision on how to spend your money: on commodity router hardware where you can easily get support if there's a problem, or on more FOSS operating systems with a routing layer (e.g. openbgpd / bird / quagga / etc).
As you can see, the uplinks are connected directly into a switch, in two separate VLANs. The reason for this is we want to be able to hotswap the routers in case of a hardware failure, but we have understood from Cogent's documentation that this is not a good practice because the links appears up even if the router goes down. What is your opinion on this?
Cogent is correct and their reasoning is correct.
However, this seems to be a fairly exotic platform, most people running BGP with Cisco, Juniper or, in some cases Quagga or Bird for Linux machines. Are there recmomendations on using OpenBGP in production? Good / bad experiences? How many people are running Linux routers vs dedicated Cisco/Juniper/etc routers?
I run lots of different routing systems for a lot of different situations (am currently using quagga, bird, openbgpd, cisco ios, cisco xr, junos and brocade ironware for bgp stuff). For small setups, it really doesn't make a whole lot of difference so long as you run with a configuration which supports both ibgp and an interior routing protocol like ospf or isis. It's not going to make a whole lot of difference to you whether you use quagga, openbgpd or bird because you're not going to stress the RIB engine with only two providers. Usually, it's better to run COTS routers (e.g. juniper / cisco / etc). If you don't want to do this, you will probably end up spending roughly the same in terms of manpower, so don't be tempted to think that you're going to save a whole lot with a free unix based system. If you want a FOSS system and you have no preconceptions about routing, I'd suggest using linux/freebsd + bird because bird is a truly wonderful RIB engine. If you are already familiar with cisco syntax, linux/freebsd + quagga will do the job just fine. If you have decided that you like openbgpd and want all the features of openbgpd (including md5 passwords), then you need openbsd + opengpd + openospfd, all of which I have found to be frankly a pain to operate and maintain, although I think openbsd has improved since the last time I used it in anger which was 3-4 years ago.
Finally, we are likely to complete this setup with a CARP (the free equivalent of VRRP) on the inside of the network. FreeBSD can apparently group interfaces and communicate with OpenBGPd - did anyone deploy such a thing here? What are your experiences or advice?
linux carp is hopeless and I would strongly advise not to use linux if you want to implement vrrp / carp. Incidentally if anyone feels this is unfair, they need to take a long hard look at the linux vmac implementation and if they don't run screaming, I'll take my hat off. The FreeBSD CARP implementation (which is borrow directly from openbsd) usually works fine, but i've seen more than my fair share of kernel panics on relatively recent freebsd relating to carp. Srsly, get a cisco / juniper router. Unless you're doing some incredibly specialised large scale router implementation and you really know what you're doing and why you're doing it, using a FOSS system will end up being more expensive in terms of your time. Nick
First, my warm thanks to everyone to responded on and off list, an amazing response that truly speaks for the opennness and incredible skill of this community. We are likely to change the setup to make sure the switch fabric sits behind the edge routers, and thanks to my new understanding of iBGP, will simply associate different upstream with the different edge routers and run BGP between them. The downside of this setup is that if a router falls over, we loose an uplink, but that's a minor problem considering how it makes the whole setup much simpler, and completely removes the single point of failure of the switch. And anyways since the uplinks are directly in the router, the downtime should be negligible in such a (rare) occurence. We will keep on experimenting with OpenBGPd, but at the first sign of trouble we will switch to what seems to be the more widely accepted alternative in the *BSD world, Bird, which also allows for a clean transition to GNU/Linux if we ever make the jump. CARP will come later, but will still be in the picture. Both routers will be in production at all time, and we'll use CARP to elect the gateway for the internal network. We prefer CARP to VRRP because it seems well supported in *BSD world and because VRRP is patent-encumbered. I am worried, however, of rumours of kernel panics associated with CARP, but I am confident that the very responsive FreeBSD community will be able to help with that. Thanks again for all your feedback, you guys rock. Cheers, A. -- A ballot is like a bullet. You don't throw your ballots until you see a target, and if that target is not within your reach, keep your ballot in your pocket. - Malcom X
On 07/11/2013 16:26, Antoine Beaupré wrote:
and because VRRP is patent-encumbered
CARP is probably patent encumbered too. I don't understand why people claim that using CARP as a default gateway isn't patent encumbered because it uses pretty much exactly the same process that the VRRP patent covers. To be honest, I wouldn't get too upset about it as the patent expires early next year, so this is not important. Also bear in mind that if you decide to migrate to VRRP later on and for some reason use the same VRRP group as CARP vhid, you will trash your network connectivity. The OpenBSD people specifically designed CARP so that it would break VRRP on the same network if they use the same parameters, but sadly they refuse to document this. It's documented elsewhere on the web though, so bear it in mind if you ever implement CARP and then move to VRRP.
I am worried, however, of rumours of kernel panics associated with CARP, but I am confident that the very responsive FreeBSD community will be able to help with that.
I hope so. There have been some long-standing problems, but hopefully the work done on fixing carp in freebsd 10 will finally see the end of these issues. Nick
participants (2)
-
Antoine Beaupré -
Nick Hilliard