I am inheriting a WISP network with Mikrotik equipment throughout. One of my first duties is to make the network multihomed. We have our first internet connection at one location and our second internet connection will be delivered at a second location in a week or so. I understand all of the steps I need to go through with ARIN in terms of getting an ASN and so forth. My question is about BGP on the Mikrotik platform. The guy who I am supplanting swears that we are supposed to be bringing the second internet link to the same place as the first internet link for BGP to work properly. Obviously that is not true with major brand routers which would do the BGP job just fine. (And he's the same guy that has bridged this whole network, so it is easy to disbelieve his opinion.) But maybe he knows that Mikrotik can't perform BGP in the same way that other routers can. So here's the question. Is there something about running BGP on a Mikrotik platform that precludes having the internet connections come in at different locations? Sincerely, Lorell Hathcock OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c) 713-992-2343 (f) | lorell@officeconnect.net Texas State Security Contractor License | ONSSI Certified Channel Partner Axis Communications Channel Partner | BICSI Corporate Member Leviton Authorized Installer
On 21/05/2010 13:16, Lorell Hathcock wrote:
job just fine. (And he's the same guy that has bridged this whole network, so it is easy to disbelieve his opinion.)
ew. nasty.
So here's the question. Is there something about running BGP on a Mikrotik platform that precludes having the internet connections come in at different locations?
I will refrain from making any smart-ass comments about Mikrotik and BGP, but no: there is no reason whatever that you can't take your internet feeds from different locations, so long as you have a good quality interior network link between those two locations, and your two routers talk iBGP to each other. Just make sure your boxes have enough RAM to cope with a full dfz feed. I.e. it's just the same as using any other router in this regard. Nick
On 05/21/2010 08:23 AM, Nick Hilliard wrote:
I will refrain from making any smart-ass comments about Mikrotik and BGP, but no: there is no reason whatever that you can't take your internet feeds from different locations, so long as you have a good quality interior network link between those two locations, and your two routers talk iBGP to each other. Just make sure your boxes have enough RAM to cope with a full dfz feed.
I.e. it's just the same as using any other router in this regard.
Nick
I've used Mikrotiks for everything except BGP, but we don't use Mikrotiks for BGP only because we already had BGP on a different platform...personally, when it comes to BGP, I think people are better off running it on devices they are familiar with rather then trying to learn the idiosyncrasies of a new platform. Bret
On 21/05/10 13:39, Bret Clark wrote:
On 05/21/2010 08:23 AM, Nick Hilliard wrote:
I will refrain from making any smart-ass comments about Mikrotik and BGP, but no: there is no reason whatever that you can't take your internet feeds from different locations, so long as you have a good quality interior network link between those two locations, and your two routers talk iBGP to each other. Just make sure your boxes have enough RAM to cope with a full dfz feed.
I.e. it's just the same as using any other router in this regard.
Nick
I've used Mikrotiks for everything except BGP, but we don't use Mikrotiks for BGP only because we already had BGP on a different platform...personally, when it comes to BGP, I think people are better off running it on devices they are familiar with rather then trying to learn the idiosyncrasies of a new platform.
While Mikrotik's BGP implementation isn't very sofisticated, there is no reason, why you can't have your feeds in different places. As Nick outlined, you need to set iBGP up between the boxes. I'm running myself a ISP on mainly Mikrotik basis (basestations and clients, approx 2500 users) and I've been extensively testing Mikrotik's BGP stack in the last 4 years (from 2.9 and up). Mikrotik wrote the whole routing stack from scratch in 3.x, which resultet in tons of problems and bugs. In my opinion, it still isn't where it should be. Don't get me wrong, but there are several pitfalls. - Mikrotik still has some memory leaks in the BGP stack somewhere, causing funny issues at times. - Filters aren't adequate for my use, and lacking a lot on IPv4, but even more on IPv4. First of all, you will need at least a RB1000, RB1100 or a PC based Mikrotik router to get enough ram, to accomodate one full-table or more. Anything less and you can forget it. I'm running a mix of Quagga boxes, Cisco and recently Juniper instead for BGP. For our internal routing OSPF on Mikrotik definatly does the job. Just my 2c. Kind regards, Martin List-Petersen -- Airwire - Ag Nascadh Pobail an Iarthair http://www.airwire.ie Phone: 091-865 968
On 2010/05/21 11:56 PM, Martin List-Petersen wrote:
- Mikrotik still has some memory leaks in the BGP stack somewhere, causing funny issues at times.
- Filters aren't adequate for my use, and lacking a lot on IPv4, but even more on IPv4.
I haven't seen either of those issues running the v4.x stream of RouterOS. The memory leak was solved a while ago and Mikrotik has fairly short release cycles. We have extensive inbound and outbound filters on our eBGP doing most of the normal things that you would do on a cisco. The IPv6 filters must be built via the terminal to avoid limitations with the current GUI but they also work very well -- Graham Beneke
On Sun, 23 May 2010 08:21:47 +0200, Graham Beneke wrote:
On 2010/05/21 11:56 PM, Martin List-Petersen wrote:
- Mikrotik still has some memory leaks in the BGP stack somewhere, causing funny issues at times.
- Filters aren't adequate for my use, and lacking a lot on IPv4, but even more on IPv4.
I haven't seen either of those issues running the v4.x stream of RouterOS. The memory leak was solved a while ago and Mikrotik has fairly short release cycles.
We have extensive inbound and outbound filters on our eBGP doing most of the normal things that you would do on a cisco. The IPv6 filters must be built via the terminal to avoid limitations with the current GUI but they also work very well
In some ways, I find the MikroTik RouterOS routing filter syntax a little more powerful than Cisco's route-maps. As routing filters work the same way as firewall filters, you can group rules in "chains" and reuse parts of your filters in other filters by jumping to another chain. This could be used, for instance, on a peering setup, where you have a number of rules per peer but also some common filtering for all peers, or to handle specific and generic filtering for your customers. I haven't yet found anything that I missed being able to with filters, at least with BGP. With other routing protocols, it's another story. Regards, Allan Eising
in V3 RouterOS's BGP support is very decent. We typically don't have any issues with it! :) Whats nice is a router with 2 gig of RAM (cheap RAM too) can take multiple full table BGP feeds without issues. Something else that's nice on our Dual Core systems is that while you are receiving the routes, you are only doing so on one core, instead of hitting high CPU while you receive all those, you only go up to 50% (on dual core system, and lower for quad and dual-quad systems). So you don't have the huge CPU issue when you pull those routes. We had some upstream limit the BGP to something stupid like 128k! Takes 50 min to get all the routes! ----------------------------------------------------------- Dennis Burgess, CCNA, Mikrotik Certified Trainer, MTCNA, MTCRE, MTCWE, MTCTCE, MTCUME Link Technologies, Inc -- Mikrotik & WISP Support Services Office: 314-735-0270 Website: http://www.linktechs.net LIVE On-Line Mikrotik Training - Author of "Learn RouterOS" -----Original Message----- From: Allan Eising [mailto:allan.eising+gmane@gmail.com] Sent: Monday, May 24, 2010 11:29 AM To: nanog@nanog.org Subject: Re: Mikrotik BGP Question On Sun, 23 May 2010 08:21:47 +0200, Graham Beneke wrote:
On 2010/05/21 11:56 PM, Martin List-Petersen wrote:
- Mikrotik still has some memory leaks in the BGP stack somewhere, causing funny issues at times.
- Filters aren't adequate for my use, and lacking a lot on IPv4, but even more on IPv4.
I haven't seen either of those issues running the v4.x stream of RouterOS. The memory leak was solved a while ago and Mikrotik has fairly short release cycles.
We have extensive inbound and outbound filters on our eBGP doing most of the normal things that you would do on a cisco. The IPv6 filters must be built via the terminal to avoid limitations with the current GUI but they also work very well
In some ways, I find the MikroTik RouterOS routing filter syntax a little more powerful than Cisco's route-maps. As routing filters work the same way as firewall filters, you can group rules in "chains" and reuse parts of your filters in other filters by jumping to another chain. This could be used, for instance, on a peering setup, where you have a number of rules per peer but also some common filtering for all peers, or to handle specific and generic filtering for your customers. I haven't yet found anything that I missed being able to with filters, at least with BGP. With other routing protocols, it's another story. Regards, Allan Eising
On 24/05/10 17:28, Allan Eising wrote:
In some ways, I find the MikroTik RouterOS routing filter syntax a little more powerful than Cisco's route-maps. As routing filters work the same way as firewall filters, you can group rules in "chains" and reuse parts of your filters in other filters by jumping to another chain. This could be used, for instance, on a peering setup, where you have a number of rules per peer but also some common filtering for all peers, or to handle specific and generic filtering for your customers.
I haven't yet found anything that I missed being able to with filters, at least with BGP. With other routing protocols, it's another story.
It's different thinking for every router platform/os, really. On Cisco/Quagga you can also reuse filtering rules by using peering-groups. At the end of the day, everybody has to find their best medium. Kind regards, Martin List-Petersen -- Airwire - Ag Nascadh Pobail an Iarthair http://www.airwire.ie Phone: 091-865 968
On Fri, May 21, 2010 at 8:23 AM, Nick Hilliard <nick@foobar.org> wrote:
On 21/05/2010 13:16, Lorell Hathcock wrote: each other. Just make sure your boxes have enough RAM to cope with a full dfz feed.
note that you do NOT have to have a full feed on either location, if your goal is simply primary/backup links... getting default from both providers and sending your prefixes out to both (potentially preferring one with an intentionally longer aspath, or other normal tricks/config) will accomplish primary/backup just fine. Don't use a sledghammer when a push pin works. -chris
On Friday 21 May 2010 05:16, Lorell Hathcock wrote:
I am inheriting a WISP network with Mikrotik equipment throughout. One of my first duties is to make the network multihomed. We have our first internet connection at one location and our second internet connection will be delivered at a second location in a week or so. [snip] My question is about BGP on the Mikrotik platform. The guy who I am supplanting swears that we are supposed to be bringing the second internet link to the same place as the first internet link for BGP to work properly. Obviously that is not true with major brand routers
And it is not true with Mikrotik either... I work for a WISP that uses Mikrotik almost exclusively, everything from our core to customer CPEs. We have multiple Mikrotik edge routers at diverse locations, with 200+Mbs internet connections thru different providers, all running full BGP feeds, and all sharing those feeds between each other. A simple 1U box with a good MB, 1-2GB RAM, flash drive for booting, and good multi-port Gb ethernet cards for each is all that is needed. We are a small ISP by most standards, but we have had no problem running 180Mbs and 40,000pps in/out on just one of our edges, while carrying on with multiple BGP feeds and exchange between our internal routers. Adrian
Tutorial: Introduction to BGP http://nanog.org/meetings/nanog47/abstracts.php?pt=MTQ0MSZuYW5vZzQ3&nm=nanog47 Tutorial: BGP 102 http://nanog.org/meetings/nanog48/abstracts.php?pt=MTUyMiZuYW5vZzQ4&nm=nanog48 http://wiki.mikrotik.com/wiki/Manual:BGP_Case_Studies On 2010-05-21 14:46, Choprboy wrote:
On Friday 21 May 2010 05:16, Lorell Hathcock wrote:
I am inheriting a WISP network with Mikrotik equipment throughout. One of my first duties is to make the network multihomed. We have our first internet connection at one location and our second internet connection will be delivered at a second location in a week or so. [snip] My question is about BGP on the Mikrotik platform. The guy who I am supplanting swears that we are supposed to be bringing the second internet link to the same place as the first internet link for BGP to work properly. Obviously that is not true with major brand routers
And it is not true with Mikrotik either... I work for a WISP that uses Mikrotik almost exclusively, everything from our core to customer CPEs. We have multiple Mikrotik edge routers at diverse locations, with 200+Mbs internet connections thru different providers, all running full BGP feeds, and all sharing those feeds between each other. A simple 1U box with a good MB, 1-2GB RAM, flash drive for booting, and good multi-port Gb ethernet cards for each is all that is needed.
We are a small ISP by most standards, but we have had no problem running 180Mbs and 40,000pps in/out on just one of our edges, while carrying on with multiple BGP feeds and exchange between our internal routers.
Adrian
Dear Lorell,
My question is about BGP on the Mikrotik platform. The guy who I am supplanting swears that we are supposed to be bringing the second internet link to the same place as the first internet link for BGP to work properly. Obviously that is not true with major brand routers which would do the BGP job just fine. (And he's the same guy that has bridged this whole network, so it is easy to disbelieve his opinion.) But maybe he knows that Mikrotik can't perform BGP in the same way that other routers can.
So here's the question. Is there something about running BGP on a Mikrotik platform that precludes having the internet connections come in at different locations?
That depends on the netwoek in between this two locations. There could be a lot of good reasons why this is no good idea; please bring some light into this. Kind regards, Ingo Flaschberger
My question is about BGP on the Mikrotik platform. The guy who I am supplanting swears that we are supposed to be bringing the second internet link to the same place as the first internet link for BGP to work
We are putting a private PTP metro ethernet (fiber based) link between the two locations. And both locations will have one internet connection. I am reading that Mikrotik has a memory leak in its BGP implementation. Any more info about this? Sincerely, Lorell Hathcock OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c) 713-992-2343 (f) | lorell@officeconnect.net Texas State Security Contractor License | ONSSI Certified Channel Partner Axis Communications Channel Partner | BICSI Corporate Member Leviton Authorized Installer -----Original Message----- From: Ingo Flaschberger [mailto:if@xip.at] Sent: Friday, May 21, 2010 6:43 PM To: Lorell Hathcock Cc: nanog@nanog.org Subject: Re: Mikrotik BGP Question Dear Lorell, properly.
Obviously that is not true with major brand routers which would do the BGP job just fine. (And he's the same guy that has bridged this whole network, so it is easy to disbelieve his opinion.) But maybe he knows that Mikrotik can't perform BGP in the same way that other routers can.
So here's the question. Is there something about running BGP on a Mikrotik platform that precludes having the internet connections come in at different locations?
That depends on the netwoek in between this two locations. There could be a lot of good reasons why this is no good idea; please bring some light into this. Kind regards, Ingo Flaschberger
Dear Lorell,
We are putting a private PTP metro ethernet (fiber based) link between the two locations. And both locations will have one internet connection.
this network between should be no problem, what routing protocols do you use in your network? ospf? Kind regards, Ingo Flaschberger
We will implement OSPF. Sincerely, Lorell Hathcock OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c) 713-992-2343 (f) | lorell@officeconnect.net Texas State Security Contractor License | ONSSI Certified Channel Partner Axis Communications Channel Partner | BICSI Corporate Member Leviton Authorized Installer -----Original Message----- From: Ingo Flaschberger [mailto:if@xip.at] Sent: Saturday, May 22, 2010 6:07 PM To: Lorell Hathcock Cc: nanog@nanog.org Subject: RE: Mikrotik BGP Question Dear Lorell,
We are putting a private PTP metro ethernet (fiber based) link between the two locations. And both locations will have one internet connection.
this network between should be no problem, what routing protocols do you use in your network? ospf? Kind regards, Ingo Flaschberger
On 2010-05-23 18:55, Ingo Flaschberger wrote:
Dear Lorell,
We will implement OSPF.
so what arguments speak against 2 bgp upstreams?
It's not an either or proposition... ospf carries your internal routes, ibgp carries you external routes between internal routers. you can carry default around in either in fact you probably should since routers that don't need a nuanced view of the outside world don't need to carry such a big table.
Kind regards, Ingo Flaschberger
-----Original Message----- From: joel jaeggli [mailto:joelja@bogus.com] Sent: Sunday, May 23, 2010 10:27 PM To: Ingo Flaschberger Cc: nanog@nanog.org Subject: Re: Mikrotik BGP Question
On 2010-05-23 18:55, Ingo Flaschberger wrote:
Dear Lorell,
We will implement OSPF.
so what arguments speak against 2 bgp upstreams?
It's not an either or proposition...
Well, I believe the original poster said that one of his colleagues swore that BGP multihoming wouldn't work unless both feeds terminated on the same router. I suppose said colleague has never heard of iBGP between two routers of the local AS. Those two routers should probably take a full table and exchange them between the two but going inside the network, yeah, they should probably simply originate a default into the the ospf routing. But I am making some assumptions here. I am assuming the two routers have connectivity between them sufficient to handle the required traffic in case one of the upstreams fails (backhaul bandwidth is at least equal to upstream bandwidth). Maybe the colleague knew that the links between the sites were insufficient and that is why both links were desired on the same physical unit or something. It is impossible to sort out other people's networking from short blurbs on a mailing list. George
* George Bonser:
Well, I believe the original poster said that one of his colleagues swore that BGP multihoming wouldn't work unless both feeds terminated on the same router. I suppose said colleague has never heard of iBGP between two routers of the local AS. Those two routers should probably take a full table and exchange them between the two but going inside the network, yeah, they should probably simply originate a default into the the ospf routing.
Does this really work that well? Won't you still get loops or blackholes unless the eBGP routes on all border routers are identical? I think you also need iBGP speakers along all feasible paths between eBGP speakers.
-----Original Message----- From: Florian Weimer Sent: Monday, May 24, 2010 2:35 AM To: George Bonser Cc: joel jaeggli; Ingo Flaschberger; nanog@nanog.org Subject: Re: Mikrotik BGP Question
* George Bonser:
Does this really work that well? Won't you still get loops or blackholes unless the eBGP routes on all border routers are identical?
As opposed to what, injecting the entire BGP table into your igp? That generally doesn't work well.
I think you also need iBGP speakers along all feasible paths between eBGP speakers.
I was assuming the eBGP speakers were directly connected over some sort of interconnecting backhaul. Again, you can't really figure out what someone's topology is from a short blurb on a mailing list. Yes, if there are intervening hops, they will need to speak iBGP as well and possibly configured as route reflectors if it isn't practical to fully mesh everything. Maybe there is a reason the legacy operator said both uplinks must be connected to the same router. If the two locations are not interconnected, that would be one reason. I don't believe the original poster described their internal connectivity. George
* George Bonser:
Does this really work that well? Won't you still get loops or blackholes unless the eBGP routes on all border routers are identical?
As opposed to what, injecting the entire BGP table into your igp?
As opposed to just injecting defaults.
Maybe there is a reason the legacy operator said both uplinks must be connected to the same router. If the two locations are not interconnected, that would be one reason. I don't believe the original poster described their internal connectivity.
There was a follow-up that mentioned that there's a direct connection, so they just have to make the other paths infeasible.
None in my mind. The legacy network operator was unfamiliar with actual best practice enterprise/carrier networking policies that he thought that for BGP to work on a two internet feed network, both internet connections have to be delivered to the same location. I thought since he has more insight into Mikrotik, that he knew about a bug with Mikrotik that made the argument true. Feedback from NANOG list members that also run Mikrotik has proven that there is no problem with running current rev levels of the Mikrotik RouterOS and BGP with internet feeds at two different locations. Sincerely, Lorell Hathcock OfficeConnect.net | 832-665-3400 x101 (o) | 832-782-4656 (c) 713-992-2343 (f) | lorell@officeconnect.net Texas State Security Contractor License | ONSSI Certified Channel Partner Axis Communications Channel Partner | BICSI Corporate Member Leviton Authorized Installer -----Original Message----- From: Ingo Flaschberger [mailto:if@xip.at] Sent: Sunday, May 23, 2010 8:56 PM To: Lorell Hathcock Cc: nanog@nanog.org Subject: RE: Mikrotik BGP Question Dear Lorell,
We will implement OSPF.
so what arguments speak against 2 bgp upstreams? Kind regards, Ingo Flaschberger
participants (13)
-
Allan Eising -
Bret Clark -
Choprboy -
Christopher Morrow -
Dennis Burgess -
Florian Weimer -
George Bonser -
Graham Beneke -
Ingo Flaschberger -
joel jaeggli -
Lorell Hathcock -
Martin List-Petersen -
Nick Hilliard