Peering Exchange Configurations
Hello All, First, apologies for the change in email address.. my work account was getting a little busy so I've moved my lists to my Gmail account. But onward.. I'm interested in peering exchange design. We are not lucky enough to have access to a peering exchange so I have no direct experience. My questions are as follows: 1) Is a private AS typically used for the exchange side of the session? 2) Are RFC1918 IPs typically used for the p2p links into the exchange? 3) Do peering exchanges typically remove their AS from the path advertised to exchange participants? 3a) If no: Do participants typically preference exchange-learned routes over other sources? 4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server? 5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges? I very much appreciate any responses. -- Brad Fleming
On 08/04/10 18:02, Brad Fleming wrote:
1) Is a private AS typically used for the exchange side of the session?
No. Everybody uses his own AS number to establish sessions at peering points.
2) Are RFC1918 IPs typically used for the p2p links into the exchange?
No. You usually get an IP address from the IX which pertains to the IX's AS, and sits in a class that's specifically not announced to the outside world.
3) Do peering exchanges typically remove their AS from the path advertised to exchange participants?
This is the case that happens if you use a route server. Being at a peering exchange point means you have the chance to sit on a switch where other participants are directly connected. At this point you can either establish direct peering relationships (configuring a session for each peering agreement you get) or create a session with the local route server, getting routes from all the other participants.
3a) If no: Do participants typically preference exchange-learned routes over other sources?
It depends. It's mostly a matter of economics more than a personal choice.
In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server?
The route server usually sits on the same LAN segment as the IXP participants.
5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges?
I would say that OpenBGPd and BIRD are your best choices for this. Quagga is getting better now, but suffered lots of problems with a high number of peers in the past at many IXes. That's why many migrated either to OpenBGPd, BIRD or both. Ciao! -- Massimiliano Stucchi BrianTel Srl stucchi@briantel.com
On 2010-04-08, at 12:02, Brad Fleming wrote:
1) Is a private AS typically used for the exchange side of the session?
No. Also many exchange points do not run route servers at all, and expect participants to build bilateral BGP sessions directly between each other.
2) Are RFC1918 IPs typically used for the p2p links into the exchange?
No. Participants in an exchange typically number their exchange-facing interfaces out of a larger (non-p2p) subnet, e.g. an IPv4 /24 or /23, or an IPv6 /48 (or both).
3) Do peering exchanges typically remove their AS from the path advertised to exchange participants?
Some do, I hear. See above regarding route servers.
3a) If no: Do participants typically preference exchange-learned routes over other sources?
Many people apply a higher LOCAL_PREF to routes learnt over an exchange in order to prefer cheap peering routes over more expensive transit routes. This is not universal, however. I know of networks who deliberately flatten LOCAL_PREF across peering and transit sessions in order to use different discriminators for exit selection (e.g. AS_PATH length).
4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast
I'm quite ignorant of multicast. IPv6 unicast peering is common.
In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server?
In all exchange points I have seen where a route server was available, the route server appears on the shared fabric numbered just as any other participant would be.
5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges?
BIRD seems to be the choice du jour based on idle hallway chatter, but I have not compared them. Joe
Re JOe, jabley@hopcount.ca (Joe Abley) wrote:
1) Is a private AS typically used for the exchange side of the session? No. Also many exchange points do not run route servers at all, and expect participants to build bilateral BGP sessions directly between each other.
...which is a shame. Routeservers in place gives you a nice benefit upon hooking up to the exchange and before you have even found out who is on the grid (anyone have a list for NOTA?). Basically, the bigger exchange points (as in "european" mostly) all have routeservers ready, but not everybody chooses to use them.
2) Are RFC1918 IPs typically used for the p2p links into the exchange? No. Participants in an exchange typically number their exchange-facing interfaces out of a larger (non-p2p) subnet, e.g. an IPv4 /24 or /23, or an IPv6 /48 (or both).
Using RFC1918 for oft-traversed addresses is also not a good idea ;)
3) Do peering exchanges typically remove their AS from the path advertised to exchange participants? Some do, I hear. See above regarding route servers.
None of the routeservers I am peering with does insert their ASN. On direct peering sessions, there is of course nobody in between.
4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast
I'm quite ignorant of multicast. IPv6 unicast peering is common.
Multicast is still seen as something special, sometimes even on dedicated hardware, or on different VLANs. It's certainly possible, but usually there are not so many participants...
5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges?
BIRD seems to be the choice du jour based on idle hallway chatter, but I have not compared them.
I was thinking "plat du jour"...and well, it's "du jour", so it can change in an instant. Cheers, Elmar.
On 2010-04-08, at 12:42, Elmar K. Bins wrote:
jabley@hopcount.ca (Joe Abley) wrote:
1) Is a private AS typically used for the exchange side of the session? No. Also many exchange points do not run route servers at all, and expect participants to build bilateral BGP sessions directly between each other.
...which is a shame. Routeservers in place gives you a nice benefit upon hooking up to the exchange and before you have even found out who is on the grid (anyone have a list for NOTA?).
I've never had a problem getting a participant list for NOTA from Terremark. One down-side of route servers on a shared exchange fabric is that the layer-2 path through the exchange for the BGP sessions does not always match the layer-2 path through the exchange for traffic. This means that AS1 might continue to learn AS2's routes through the route server even though there's a layer-2 problem that prevents AS1 and AS2's peering routers from talking directly to each other. Hilarity may result. I've never seen such a problem on small exchanges where the layer-2 fabric is simple, but I have seen it more than once on larger, more complicated exchanges. My personal preference is to focus peering energy on bilats, and not to rely on a route server. But I understand the savings in opex that route servers can provide on busy exchanges. Joe
Some Research&Education type peering exchanges, like Pacific Wave http://www.pacificwave.net/ , support ipv4 multicast forwarding. As an exchange operator you'd want to support PIM-Snooping and the ability to disable DR-Flooding to control those flows just to the networks that joined them. Chris -- Chris Costa CENIC ccosta@cenic.org On Apr 8, 2010, at 9:29 AM, Joe Abley wrote:
4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast
On 8-4-2010 18:02, Brad Fleming wrote:
1) Is a private AS typically used for the exchange side of the session?
No.
2) Are RFC1918 IPs typically used for the p2p links into the exchange?
No. In EU usually it is separate public /24, /23 or /22. The IPv6 range in RIPE region for exchanges is assigned from within special pool 2001:7f8::/32 (each IX gets /48).
3) Do peering exchanges typically remove their AS from the path advertised to exchange participants?
The direct peering is between participants AS'es, there is nothing in between, including IX AS. Route-servers based on Cisco box put their AS number in between, but Quagga/Bird usually remove the IX as, however it may be configured per peer not to do so.
3a) If no: Do participants typically preference exchange-learned routes over other sources?
Most do.
4) Do exchanges typically support the following address families? IPv4 Multicast
no
IPv6 Unicast
Almost all.
IPv6 Multicast
No.
In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server?
Route-server is just like one of the members of the exchange. You get /23 or similar prefix of the exchange. You may have a BGP session with the route-server, but you are also free to have direct BGP sessions with other parties. Route-servers are mostly used by peers with open peering policies, but you still may steer your announcements basing on BGP communities.
5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges?
Quagga was used most often, but recently most biggest EU exchanges replaced it with Bird and it is much more stable. -- Grzegorz Janoszka
On Thu, 2010-04-08 at 11:02 -0500, Brad Fleming wrote:
1) Is a private AS typically used for the exchange side of the session?
Not in a typical public internet exchange. that said, there is no reason why one could not build an exchange point that uses private ASNs. One might do this for a specialised application of PPVPN peering partners but that is beyond what you are asking.
2) Are RFC1918 IPs typically used for the p2p links into the exchange?
No. Public IXPs will provide an address space for its participants to use. That said, once again, there might be some very specific non-public applications of exchange points which may use RFC1918 address space.
3) Do peering exchanges typically remove their AS from the path advertised to exchange participants?
I'm assuming you are talking about routes exchanged via a route-server. Most moderm deployments of RSes do act transparently but in the past there have been cases where some exchange point participants did want to see the RS' ASN in the ASPATH. Merit's old RSes had the capbility to turn on or off transparency on a peer-by-peer basis. I am not sure about modern RS implementations.
3a) If no: Do participants typically preference exchange-learned routes over other sources?
That is a matter of personal reference and religion. But in my experience at larger exchanges with bigger players, the RSes' routes are considered secondary and less preferred.
4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast
IPv4 multicast and IPv6 unicast seems standard in modern IXPs but I'm not sure about IPv6 multicast.
In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server?
Not in modern exchange points. In some older ones this was the case but I think that's largely historic.
5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges?
I wouldn't say clear but BIRD seems to be gaining ground on the other two. There were some talk given at the last NANOG meeting in the route- servers track... http://www.nanog.org/meetings/nanog48/abstracts.php?pt=MTUxMyZuYW5vZzQ4&nm=nanog48 -- /*=================[ Jake Khuon <khuon@NEEBU.Net> ]=================+ | Packet Plumber, Network Engineers /| / [~ [~ |) | | -------- | | for Effective Bandwidth Utilisation / |/ [_ [_ |) |_| NETWORKS | +==================================================================*/
I operate the exchange point in the Kansas City area so I'll answer your questions based on how we do it. 1) Is a private AS typically used for the exchange side of the session? No. Each participant uses their own ASN. 2) Are RFC1918 IPs typically used for the p2p links into the exchange? No. Exchanges typically have their own IPs assigned by their RIR and pass them out to the members for connections to the exchange. 3) Do peering exchanges typically remove their AS from the path advertised to exchange participants? There is no peering directly with the exchange in a private link. In the case of peering with a route server on the exchange then it is considered best practices to do so. 3a) If no: Do participants typically preference exchange-learned routes over other sources? Yes. As far as I know all our members set routes learned through the exchange fabric higher than anything else. That's kind of the point as exchange traffic is free so you always want to use it first. 4) Do exchanges typically support the following address families? IPv4 Multicast IPv6 Unicast IPv6 Multicast No Yes No In exchanges where a route server is employed: 4) Do participants have a p2p link into a simple routing environment then multi-hop to a route server? No. The route server is typically accessed like any other peer on the fabric. 5) I see that Bird, OpenBDGd, and Quagga are all options for route server software. Does one of those packages stand out as the clear current choice for production peering exchanges? We use Quagga. It's what we we're most familiar with and we haven't had any issues.
I very much appreciate any responses.
No Problem. Feel free to stop by and check out our fabric for yourself. Aaron
3a) If no: Do participants typically preference exchange-learned routes over other sources?
Yes. As far as I know all our members set routes learned through the exchange fabric higher than anything else. That's kind of the point as exchange traffic is free so you always want to use it first.
Actually, the order of preference is usually: 1. Private Interconnects (direct private peering) 2. Non-metered paid peering/transit 3. Exchange Points 4. Metered paid peering/transit Owen
On Apr 8, 2010, at 2:08 PM, Owen DeLong wrote:
3a) If no: Do participants typically preference exchange-learned routes over other sources?
Yes. As far as I know all our members set routes learned through the exchange fabric higher than anything else. That's kind of the point as exchange traffic is free so you always want to use it first.
Actually, the order of preference is usually:
Where 'usually' here is rather nebulous. I am not trying to say Owen is wrong, just don't think the way any network uses interconnectivity is somehow standard. Every network is different, and even similar links in the same network are different. IXPs are standard ('usually' :), networks are not. -- TTFN, patrick
1. Private Interconnects (direct private peering) 2. Non-metered paid peering/transit 3. Exchange Points 4. Metered paid peering/transit
Owen
-----Original Message----- On Apr 8, 2010, at 2:08 PM, Owen DeLong wrote:
3a) If no: Do participants typically preference exchange-learned routes over other sources?
Yes. As far as I know all our members set routes learned through the exchange fabric higher than anything else. That's kind of the point as exchange traffic is free so you always want to use it first.
Actually, the order of preference is usually:
Where 'usually' here is rather nebulous. I am not trying to say Owen is wrong, just don't think the way any network uses interconnectivity is somehow standard. Every network is different, and even similar links in the same network are different. IXPs are standard ('usually' :), networks are not. -- TTFN, patrick
1. Private Interconnects (direct private peering) 2. Non-metered paid peering/transit 3. Exchange Points 4. Metered paid peering/transit
Owen
----------------------------------------------------------- My answers were based on what I know about members of our exchange. In our market there is little to no private peering. Everyone connects through the exchange so that's their only source of peering. Although we don't require our members to use the route server, all of them do.
participants (10)
-
Aaron Wendel -
Brad Fleming -
Chris Costa -
Elmar K. Bins -
Grzegorz Janoszka -
Jake Khuon -
Joe Abley -
Massimiliano Stucchi -
Owen DeLong -
Patrick W. Gilmore