The best IX list I've found is Open-IX as it's the only one I've found dedicated to IXes while still being public. Tried to join the Euro-IX ones, but as you indicated... members only. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com ----- Original Message ----- From: "Andy Davidson" <andy@nosignal.org> To: "Goran Slavic" <gslavic@sox.rs> Cc: nanog@nanog.org Sent: Friday, April 24, 2015 4:30:12 AM Subject: Re: Euro-IX quagga stable download and implementation Hi, Goran, everyone -- On 23 Apr 2015, at 09:06, Goran Slavic <gslavic@sox.rs> wrote:

at the mailing list and have an interest in downloading and implementing the Euro-IX version of Quagga in our Internet exchange. My questions are simple: - Considering the time when the post is written (2012) - what is the current status of the Euro-IX Quagga ? - Where can it be downloaded as a stable release / version ?

This email is a comment on using this software as a route-server, and not a comment on using this software as a RIB manager on a forwarding device - if you’re a reader from the future trying to understand about running this software on a router, then please bear this in mind. There are three well known open source BGP implementations which are commonly used as a route-server - BIRD, Quagga, and OpenBGPd. It is typical to configure them today in a way that has the route-server calculate a different RIB for every connected ASN on your exchange. This is because it is also common to allow route-server users to filter (prevent their prefixes reaching) other participants. Information about why this is important has been published in various presentations and papers at IX and operator events. Calculating best-path for every participant becomes complex when you have a lot of participants, further when the number of prefixes on the exchange grows. OpenBGPd will stay up but take a very long time to process and forward announce/withdraw BGP messages. On a ~100 ASN/participant/table system with ~5000 prefixes, it can take anywhere up to an hour for a withdraw to be processed and forwarded which means your participants will get a route that is withdrawn for a long time and blackhole traffic at the exchange. It is therefore problematic to use this software on all but the smallest exchanges. It’s OK on small instances but does not scale. Quagga’s vanilla build will fail to stay up with large numbers of tables and participants. Chris Hall did an amazing job at making a build that was more prone to staying up and his build is doing a sterling job at LINX (alongside BIRD) but I understand that this source tree is no longer maintained and that the task of merging his stability fixes into the mainline or OSR (https://www.opensourcerouting.org) version is not a simple job and has not been done. This gives me a serious concern about the future of this branch. BIRD just doesn’t die, no matter what scale we seem to throw at it. This thing just keeps flying. We now have two (busy) BIRD instances at the LONAP exchange in London where most of our 150 exchange point members use the service. Goran - SOX is a member of the Euro-IX association for exchange points and there is a private mailing list for members who operate route-servers. There may be a greater concentration of route-server operators on that list so it’s probably worth continuing the discussion there? You sign in to the website and visit https://www.euro-ix.net/mailing-list-archives to subscribe. With best wishes, Andy Davidson (Relevant Hats: LONAP, IXLeeds, Euro-IX, IIX, NapAfrica)

Hello Andy, Martin and everyone else First I would like to thank you all on extremely well written and well-argumented posts. SOX operated Quagga route servers for many years but as the number of peers (and more importantly prefixes) has grown we began to see very disturbing warning signs that the stability of those servers is at peril. "Clear" commands that take forever, over processing of requests that makes Quagga forget to send keep-alive pockets to peers, constant memory leakage etc. Considering that those problems have began to multiply and escalate with every new peering - I was tasked to find and implement the alternative for current route servers in order to improve stability or find the alternative to program packages we currently use. Considering what I have learned in your posts (and on other places that I have informed myself) I will definitely suggest to SOX management to go the way similar to what LINX did (1 Bird + 1 Quagga as route servers) for the simple reason that 2 different solution provides more security in context of "new program update->new bugs" problems and incidents and prevents other potential problems. I am extremely grateful for your help specially in the context of how much time it has saved me and good arguments it has given me for the solution I plan to implement. I hope we will continue future discussions and exchange of ideas on Euro-IX forums/mailing lists. Regards, Goran Slavić SOX -----Original Message----- From: Martin Winter [mailto:mwinter@noaccess.com] Sent: Saturday, 25 April 2015 03:41 To: Andy Davidson Cc: Goran Slavic; nanog@nanog.org Subject: Re: Euro-IX quagga stable download and implementation Andy, Goran (and everyone else) Disclaimer first: I work full-time for OpenSourceRouting on Quagga. On Fri, 24 Apr 2015, Andy Davidson wrote:

Hi, Goran, everyone --

On 23 Apr 2015, at 09:06, Goran Slavic <gslavic@sox.rs> wrote:

...

at the mailing list and have an interest in downloading and implementing the Euro-IX version of Quagga in our Internet exchange. My questions are simple: - Considering the time when the post is written (2012) - what is the current status of the Euro-IX Quagga ? - Where can it be downloaded as a stable release / version ?

[...]

Quagga's vanilla build will fail to stay up with large numbers of tables and participants. Chris Hall did an amazing job at making a build that was more prone to staying up and his build is doing a sterling job at LINX (alongside BIRD) but I understand that this source tree is no longer maintained and that the task of merging his stability fixes into the mainline or OSR (https://www.opensourcerouting.org) version is not a simple job and has not been done. This gives me a serious concern about the future of this branch.

On the Chris Hall branch: Chris did some great work fixing many issues, but unfortunatly, mostly in a solo mission on it's own. The idea (from the beginning when Euro-IX sponsored his work) was to get this integrated back into the mainline Quagga. However, by the time we got access to the code, it was a basically one large diff of 1000's of lines with no git history. This would be a lot of work to pick it apart again, review the code and commit it (in pieces) into the mainline. We talked about supporting it as an alternative BGP daemon, but he changed quite a bit in zebra as well, so this was still too much work. When I say "too much", the issue was that noone was willing to sponsor the work (person or money) to get this integrated. We did (actually multiple times) look into the issues and made different plans on how to get the BGP performance fixed. But so far (in the past), everyone who sponsors us doesn't care much about the BGP scale and cares more on IPv6 with OSPFv3, ISIS etc. So that's where most of our work went. (Plus a lot of testing. I think Quagga is the only Open Source routing platform which is tested against protocol fuzzers and for RFC compliance) There is now (again) some interest (mainly form european IX'es) to look into the problems and we started (again) to evaluate, measure and see how we can fix it on a limited budget. The idea is to really get Quagga usable as a RouteServer to have a 2nd choice (beside Bird). Happy to get donations (We are a US 501c3 non-profit) to actually make it happen. Overall, if everyone here who complains would just donate a little bit money (or some work), then the whole issue would be long solved.

BIRD just doesn't die, no matter what scale we seem to throw at it. This thing just keeps flying.

Short term, if you are ok with a single solution and need something now for a route-server, I think Bird is the solution. Long term, I hope to get Quagga as an alternative (and for everyone who wants 2 different solutions). Bird initially was (and still is) focused to the Route server & Route reflector application and has some unique features there. Quagga is today more focused as a full routing daemon and mostly used in virtual routers, SDN applications and ToR routers. Regards, Martin Winter (OpenSourceRouting, NetDEF)

Andy, Believe me when I say: I would never have the idea to think about attempting to try to test my ability to generate configurations for this "2 route servers/ 2 different programs that run them" solution without the IXP Manager :-) I am familiar with the work INEX has been doing with IXP Manager and have for some time attempted to find time from regular SOX operation to implement it in our IX. This migration gives me the excellent opportunity and arguments to finally allocate time, resources and manpower for installation and implementation of IXP Manager as the route server configuration generator at SOX. Regards G.Slavic -----Original Message----- From: Andy Davidson [mailto:andy@nosignal.org] Sent: Saturday, 25 April 2015 21:34 To: Goran Slavić Cc: nanog@nanog.org Subject: Re: Euro-IX quagga stable download and implementation On 25 Apr 2015, at 15:16, Goran Slavić <gslavic@sox.rs> wrote:

Considering what I have learned in your posts (and on other places that I have informed myself) I will definitely suggest to SOX management to go the way similar to what LINX did (1 Bird + 1 Quagga as route servers) for the simple reason that 2 different solution provides more security in context of "new program update->new bugs" problems and incidents and prevents other potential problems.

Goran - glad to have helped. One last piece of advice which might be useful - to help to guarantee consistency of performance between the two route-servers, you should consider a configuration generator so that your route-server configs are in sync. The best way to implement this at your exchange is to use IXP Manager, maintained by the awesome folks at the Irish exchange point, INEX. https://github.com/inex/IXP-Manager IXP Manager will get you lots of other features as well as good route-server hygiene. There's also a historic perl-script that does this on my personal github. Both of these solutions allow you to filter route-server participants based on IRR data, which has proved to be a life-saver at all of the exchanges I help to operate. Having my horrible historic thing is maybe better than no thing at all, but I deliberately won't link to it as you should really use IXP Manager. :-) Andy=

sorry, for the double post. dmarc fuckup... Hey there, considering the state of this discussion, BIRD seems to be the only scalable solution to be used as a route server at IXPs. I have built a large code base around BGP for the hoofprints project [1] and BRITE [2] and would enjoy building another state-of-the-art open-source route-server implementation for IXPs. Would you be so kind to send me your feedback on this idea? Do you think, it makes sense to pursue such a project or is it not relevant enough for you? Best regards, Sebastian 1: https://github.com/sspies8684/hoofprints/ 2: https://brite.antd.nist.gov/statics/about Am 25.04.2015 um 22:06 schrieb Goran Slaviæ:

Andy,

Believe me when I say: I would never have the idea to think about attempting to try to test my ability to generate configurations for this "2 route servers/ 2 different programs that run them" solution without the IXP Manager :-)

I am familiar with the work INEX has been doing with IXP Manager and have for some time attempted to find time from regular SOX operation to implement it in our IX. This migration gives me the excellent opportunity and arguments to finally allocate time, resources and manpower for installation and implementation of IXP Manager as the route server configuration generator at SOX.

Regards G.Slavic

-----Original Message----- From: Andy Davidson [mailto:andy@nosignal.org] Sent: Saturday, 25 April 2015 21:34 To: Goran Slaviæ Cc: nanog@nanog.org Subject: Re: Euro-IX quagga stable download and implementation

On 25 Apr 2015, at 15:16, Goran Slaviæ <gslavic@sox.rs> wrote:

...

Considering what I have learned in your posts (and on other places that I have informed myself) I will definitely suggest to SOX management to go the way similar to what LINX did (1 Bird + 1 Quagga as route servers) for the simple reason that 2 different solution provides more security in context of "new program update->new bugs" problems and incidents and prevents other potential problems. Goran - glad to have helped.

One last piece of advice which might be useful - to help to guarantee consistency of performance between the two route-servers, you should consider a configuration generator so that your route-server configs are in sync. The best way to implement this at your exchange is to use IXP Manager, maintained by the awesome folks at the Irish exchange point, INEX. https://github.com/inex/IXP-Manager

IXP Manager will get you lots of other features as well as good route-server hygiene.

There's also a historic perl-script that does this on my personal github. Both of these solutions allow you to filter route-server participants based on IRR data, which has proved to be a life-saver at all of the exchanges I help to operate. Having my horrible historic thing is maybe better than no thing at all, but I deliberately won't link to it as you should really use IXP Manager. :-)

Andy=

My experience tells me Martins direction is a good one. You would be surprised to learn how much time already went into whats out there that people trust now. Besides - it has very limited marketing appeal. The IXs number is small. The big ones already have something working well. I wouldn't implement something new. When I chose, I went for something a big network ran for years. As a result it was reliable and easy to maintain. Had few and simple problems. Simply ran 2 and had people get a session with both. No one ever lost routes when I took one down to upgrade - or when we had a hardware failure. Thank You Bob Evans CTO

On Mon, 4 May 2015, Sebastian Spies wrote:

...

sorry, for the double post. dmarc fuckup...

Hey there,

considering the state of this discussion, BIRD seems to be the only scalable solution to be used as a route server at IXPs. I have built a large code base around BGP for the hoofprints project [1] and BRITE [2] and would enjoy building another state-of-the-art open-source route-server implementation for IXPs. Would you be so kind to send me your feedback on this idea? Do you think, it makes sense to pursue such a project or is it not relevant enough for you?

How about (instead of another implementation) helping one of the existing projects? Writing another implementation is easy. Keeping it up to date, testing it and supporting it over multiple years is what I would worry about.

I would *strongly* suggest to solve that issue first before starting on another implementation.

- Martin

Hi Sebastian, I highly support your idea! Almost all large IXPs switched to bird. As we know from different route server studies almost 1/3 of the traffic handled by IXPs is managed by route servers. This shows that route servers play an important role in the IXP ecosystem. So, depending on only one implementation comes with a lot of operational risks at least. The beauty of your suggestion is that a route server that is just that (and not a routing daemon with many unneeded features as bird or quagga). This could limit the complexity of the source code which means the effort needed to maintain the code should be a lot smaller compared to bird/quagga. I see no reason why there is not enough room for at least one or two more route server implementations besides bird. Best regards, Thomas (with no hat on -> my personal opinion)