BGP Enabled transit in Chicago (River North) and equipment recommendation
I have a need for a BGP enabled connection in the River North section of Chicago. We have a small number of IP blocks that we want to use. Currently, we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking at downsizing and bringing stuff in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine). I know RCN offers business cable-modem service but probably not BGP. Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size router, but none of them seem to do BGP (not surprising). Any recommendations on hardware would be welcome as well
Might be worth to consider running a software router on that scale with perhaps some cheap quad-port GbE PCIe NICs. BIRD would be the BGP daemon to go, or FRRouting if you want an integrated shell. Hardware routers for 100 Mbit egress seem a bit overpowered, however, as scaleable you want to go, some Ubiquiti routers might be a cheap option. As for transit, have you considered a redundant tunnel-based solution instead? You can run that transparently on top of your RCN connection, with negligible costs for your commit and no additional connection fees. On Sep. 3 2019, at 6:17 pm, ADNS NetBSD List Subscriber <nanog2@adns.net> wrote:
I have a need for a BGP enabled connection in the River North section of Chicago. We have a small number of IP blocks that we want to use. Currently, we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking at downsizing and bringing stuff in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine).
I know RCN offers business cable-modem service but probably not BGP.
Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size router, but none of them seem to do BGP (not surprising). Any recommendations on hardware would be welcome as well
I’ve had BGP from comcast business in River North before, not sure what their minimum bandwidth is for that. Tunnels may be simplest at that bandwidth level.
On Sep 3, 2019, at 12:52 PM, Florian Brandstetter via NANOG <nanog@nanog.org> wrote:
Might be worth to consider running a software router on that scale with perhaps some cheap quad-port GbE PCIe NICs. BIRD would be the BGP daemon to go, or FRRouting if you want an integrated shell. Hardware routers for 100 Mbit egress seem a bit overpowered, however, as scaleable you want to go, some Ubiquiti routers might be a cheap option.
As for transit, have you considered a redundant tunnel-based solution instead? You can run that transparently on top of your RCN connection, with negligible costs for your commit and no additional connection fees.
On Sep. 3 2019, at 6:17 pm, ADNS NetBSD List Subscriber <nanog2@adns.net> wrote: I have a need for a BGP enabled connection in the River North section of Chicago. We have a small number of IP blocks that we want to use. Currently, we have some equipment at 350 E. Cermak (Steadfast Networks) and are looking at downsizing and bringing stuff in-house. Our bandwidth requirements are miniscule (10MB/Sec is fine).
I know RCN offers business cable-modem service but probably not BGP.
Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size router, but none of them seem to do BGP (not surprising). Any recommendations on hardware would be welcome as well
On Tue, Sep 3, 2019 at 12:44 PM ADNS NetBSD List Subscriber <nanog2@adns.net> wrote:
Also, we’d like to ditch our 3640 router in favor of a smaller “desktop” size router, but none of them seem to do BGP (not surprising). Any recommendations on hardware would be welcome as well
I can think of lots of smaller, even desktop sized routers that have no problem doing BGP assuming you're only taking a single transit circuit and only accepting a default route from your upstream provider. Ubiquiti's tiniest EdgeRouters which cost around $100 will do BGP just fine under those circumstances if you're not pushing tons of traffic or pps. If you want something a little nicer/fancier, you can get a Juniper SRX (probably a 300, 320, or 345 depending on how much bandwidth you're going to use). If you want to run multiple transits and take full tables, then at that point I'd recommend going a bit bigger: maybe a Juniper MX or a Cisco ASR. But even the higher-end Ubiquiti EdgeRouter series products can handle full tables if you understand and accept their limitations in doing so if budget is a huge concern but you still need to take full tables. Take care, Matt
On 9/3/2019 12:19 PM, Matt Harris wrote:
But even the higher-end Ubiquiti EdgeRouter series products can handle full tables if you understand and accept their limitations in doing so if budget is a huge concern but you still need to take full tables.
As long as you stick with the 1.10.10 series of firmware, the EdgeRouter series is pretty stable. I run an EdgeRouter Infinity (8 x 10G SFP+) at home with both IPv4 and IPv6 BGP feeds on a CenturyLink Fiber+ connection. You can get the original EdgeRouter Lite for sub $100 and it will have no problem taking a full feed for both v4 and v6... However, better choice is the EdgeRouter 4, which is a bit newer and much faster. There's also the ER6P if you need something with more ports, or even the ER12/12P if you want something with a built in network switch. Just avoid the budget oriented ERX and ER10X which lack the RAM. All of the ERs are low power consumption and Linux based with a Vyatta/VyOS/Juniper-like CLI, so pretty easy to work with. (I do Ubiquiti deployments, so can answer a lot of questions anyone might have). -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buff ers on sockets. I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached. Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%. You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it.
On 9/3/2019 1:54 PM, Florian Brandstetter wrote:
I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.
I'm going based on actual tests done when the ERL came out in 2012. While its cores are anemic by today's standards, it handled multiple full tables fine back then. Since then, the ERs have stopped using Quagga/Zebra and now use ZebOS (which is also used in several other commercial products). It's a bit faster and smaller. In today's world, its tight, but it will still work.
You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it.
As I pointed out, its the dirt cheap option. Better option is the ER4 and later which is better suited to the increased table size. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra features. On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote:
Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets.
I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.
Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%.
You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it. Sent from Mailspring
I've got a Cisco 881 with the "Advanced IP features" This will do for what I'm trying to accomplish. I think I'm going to go with a BGP tunnel. No one at RCN has any clue about this - they may not even provide the server. The sales droids only know how to sell their pre-packaged plans. Does anyone know who provides BGP tunnel session? Doesn't really need to be RCN as I can create a tunnel with any peer. Thanks
They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra features.
On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote:
Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets.
I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.
Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%.
You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it. Sent from Mailspring
--------------69793807A24007030ACBABEA Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra features.<br> </p> <div class="moz-cite-prefix">On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote:<br> </div> <blockquote type="cite" cite="mid:69414933-770B-464C-B9DA-A8F7A61566A1@getmailspring.com"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div>Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets.</div> <br> <div>I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.</div> <br> <div>Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%.</div> <br> <div>You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it.</div> <img class="mailspring-open" alt="Sent from Mailspring" style="border:0; width:0; height:0;" src="https://link.getmailspring.com/open/69414933-770B-464C-B9DA-A8F7A61566A1@get..." moz-do-not-send="true" width="0" height="0"> </blockquote> </body> </html>
--------------69793807A24007030ACBABEA--
If you're okay with a tunnel, you may want to check out http://bgp.services. On Wed, Oct 16, 2019 at 8:36 AM John Palmer <nanog2@adns.net> wrote:
I've got a Cisco 881 with the "Advanced IP features" This will do for what I'm trying to accomplish.
I think I'm going to go with a BGP tunnel.
No one at RCN has any clue about this - they may not even provide the server. The sales droids only know how to sell their pre-packaged plans.
Does anyone know who provides BGP tunnel session? Doesn't really need to be RCN as I can create a tunnel with any peer.
Thanks
They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra features.
On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote:
Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets.
I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.
Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%.
You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it. Sent from Mailspring
--------------69793807A24007030ACBABEA Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>They are obviously not running full tables on their 3640. I'd imagine a raspberry pi would have more BGP capability and throughput than a 3640, though I don't recommend doing that even as a joke. But an ERR would be fine if they're expecting nothing more than a slightly faster 3640 with maybe some extra
features.<br>
</p> <div class="moz-cite-prefix">On 9/3/19 3:54 PM, Florian Brandstetter via NANOG wrote:<br> </div> <blockquote type="cite" cite="mid:69414933-770B-464C-B9DA-A8F7A61566A1@getmailspring.com"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div>Ubiquiti's EdgeRouter Lite is equipped with 512 MiB of DDR2 memory, of which after startup, roughly 491 MiB can be utilized. 119 MiB of the remaining memory are allocated by the base of the router already, which leaves you with a remainder of 372 MiB memory. Memory usage depends on the architecture for objects, for example there's a large difference between x86 and x86_64, since on x86_64, the compiler will generally use 64bit boundaries to be faster; the ERL runs on a MIPS64 architecture, which will have a similar trade-off. To get to the point, let's have a quick look at the components using memory: bgpd, zebra, kernel. Roughly 180 MiB of memory are required to keep a single full table in bgpd alone, leaving you with 192 MiB of free memory. Accounting further, zebra will eat at least another 100 MiB for exporting the BGP RIB to the Kernel (FIB), leaving you with 100 MiB. At this point, you have a mere 92 MiB left for fitting the routes into the kernel, and to leave room for RX buffers on sockets.</div> <br> <div>I don't see full tables happening from a memory perspective on the EdgeRouter Lite, you would want to look at something with at least 2 GiB of memory to keep the whole system running smoothly, and when using Quagga and Zebra, that's still aimed rather low. FRRouting at this point uses 2 GiB for 4 full tables on an x86 system, without any magic attached.</div> <br> <div>Having kept it unmentioned, the EdgeRouter Lite has a dual-core with 500 MHz, and surely your BGP updates processing isn't offloaded, hence you will pretty quickly kill the whole router when you flood it with a full table, unless you set very low queue sizes, which isn't really reliable though since you generally want BGP to converge fast - not after a period of 15 minutes with the CPU sitting on 100%.</div> <br> <div>You might want to install something like OpenWRT (which I don't know the possibility of on an ERL), and run BIRD if you're tied to a low memory footprint, however, in a base vendor-generic setup of the ERL, it's beyond my understanding why one would even suggest running a full table on it.</div> <img class="mailspring-open" alt="Sent from Mailspring" style="border:0; width:0; height:0;" src="
https://link.getmailspring.com/open/69414933-770B-464C-B9DA-A8F7A61566A1@get... "
moz-do-not-send="true" width="0" height="0"> </blockquote> </body> </html>
--------------69793807A24007030ACBABEA--
I will note that Comcast will do BGP on their enterprise fiber circuits. Comcast DIA (which they call EDI) comes in increments of 1M up to 10M, then 10M up to 100M, etc. So you could get 10M or 80M (not sure if "10MB/Sec" means 10Mbps or 80MBps) and do BGP over that, if it's available. RCN is likely similar but I'm not as familiar with their offerings. On Tue, Sep 3, 2019 at 3:35 PM Brielle <bruns@2mbit.com> wrote:
On 9/3/2019 12:19 PM, Matt Harris wrote:
But even the higher-end Ubiquiti EdgeRouter series products can handle full tables if you understand and accept their limitations in doing so if budget is a huge concern but you still need to take full tables.
As long as you stick with the 1.10.10 series of firmware, the EdgeRouter series is pretty stable. I run an EdgeRouter Infinity (8 x 10G SFP+) at home with both IPv4 and IPv6 BGP feeds on a CenturyLink Fiber+ connection.
You can get the original EdgeRouter Lite for sub $100 and it will have no problem taking a full feed for both v4 and v6... However, better choice is the EdgeRouter 4, which is a bit newer and much faster.
There's also the ER6P if you need something with more ports, or even the ER12/12P if you want something with a built in network switch. Just avoid the budget oriented ERX and ER10X which lack the RAM.
All of the ERs are low power consumption and Linux based with a Vyatta/VyOS/Juniper-like CLI, so pretty easy to work with.
(I do Ubiquiti deployments, so can answer a lot of questions anyone might have).
-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org
participants (8)
-
ADNS NetBSD List Subscriber -
Brielle -
Darrell Budic -
Florian Brandstetter -
John Palmer
-
Matt Harris -
Paul Timmins -
Ross Tajvar