Bill Woodcock writes:

To expand upon that a little, while an exchange point itself is usually a layer-2 entity (that's a good thing), the ISPs which are exchanging traffic across it need to assign IP addresses to their router ports which are connected to it, and it's good to use a single common block of addresses for that purpose, and manage them in such a way that the in-addr DNS for the block provides useful information about which ISP they're associated with, and contact info, and so forth. IP addresses are available for this purpose from Bill Manning, bmanning@ep.net, who also provides a management infrastructure for registering the in-addr names and contact information.

RIRs may also set aside addresses to be used for exchange points, and you may want to check with your local RIR for this address space. It is generally a good thing to establish a relationship with the local RIR in any case.

> the ix provider may also choose to provide services on the side... > Like other routed networks, if they are multi-homed, they tend to > have their own asn.

One of the services that some exchange points choose to provide is a route-server or "looking glass" which is a router which all or most of the others peer with, and which either forwards routes between the other peers (aggregating the routes into single peering sessions) or allows participants to log in and see what routes other participants are or should be offering. As a BGP peer, this obviously needs an AS of its own. Packet Clearing House, a research organization that I coordinate, offers free route-servers to exchange points, and we also provide the AS number for the route-servers.

There are actually several different services here, and I think it would be useful if we had an agreed set of terms for those services. Below I've put the terms I've heard most often and my sense of their meaning; if there are other accepted terms or meanings, I would be happy to hear of them and try to come up with a generic vocabulary. One is a "route collector", with which exchanges participants peer in order to give the exchange operator a view into what is going on at layer 3 within the exchange; the primary use of this is to allow the exchange operator to correlate layer 2 behavior with layer 3 behavior when trouble shooting. Another is a "looking glass" which allows some group (ranging from participants to the general public) to see layer 3 adjacencies and some statistics for the exchange; this can be derived from data seen by a participant at an exchange or from a route collector. The third is a route server, whose function was originally developed as part of the routing arbiter project. The route servers allow exchange participant to outsource the routing task (but not the forwarding of packets) to a specialized host within the exchange. regards, Ted Hardie

One difference is that Route Servers, like the ones run by Merit RSNG team, are based on the Internet Routing Registry, whereas route reflectors are not. Route Server routes are re-announced based upon configured IRR policy. I also think of Route Reflectors as being both internal AS (IGP) and external AS (BGP) re-announcers whereas Route Servers are strictly inter-AS (BGP). Bill

I've also heard some symantic confusion between route-servers and route reflectors. In conversation, I usually assume that distinction to be between functionally equivalent boxes operating in the plenum between a number of administrative domains (a route-server) or as glue between regions or ASes within one administrative domain (a route reflector). I don't know how common that understanding would be, though. Anyone have any better thoughts on the difference between a route-server and a route reflector?

One difference is that Route Servers, like the ones run by Merit RSNG team, are based on the Internet Routing Registry, whereas route reflectors are not. Route Server routes are re-announced based upon configured IRR policy.

Actually, a key difference is that Merit Route Servers allow for *multiple* views, whereas route reflectors and (most router-based) route servers provide only *one* view of routes. -abha ;)

Actually, that isn't completely true.

+ Router Server - cost is workstation, software, and training. You need to buy the RS software. $120,000 (Merit's price) is a lot of money to collect for an IXP project in places like Kenya, Sri Lanka, Uganda, Philippines, etc.).

If you want to get the fully supported Merit does everything version, then sure, it is going to cost some money. Mind you that price tag also includes 24x7 NOC coverage and helping peers out with their policies, etc. The service you get is worth the price. If you want a cheap Route Server, download the code (free) and install it on a cheap PC running FreeBSD or something (close to free). This can support many many peers just fine.

+ Router Reflector - cost is a router (or a unix box with gated) and training. (3620 works fine for a IXP with +60 ISPs).

About the same price as a cheap Route Server. Folks often prefer using routers over a unix box "Route Server" because the interface is familiar. So, it is understandable why this is a common choice, however, once you spend the time getting the hang of RSd, there is a lot you can do... Just my thoughts.... -abha ;)

On Mon, Jan 08, 2001 at 05:35:41PM -0800, Randy Bush wrote:

...

Another difference is cost:

+ Router Server - cost is workstation, software, and training. You need to buy the RS software. $120,000 (Merit's price) is a lot of money to collect for an IXP project in places like Kenya, Sri Lanka, Uganda, Philippines, etc.).

+ Router Reflector - cost is a router (or a unix box with gated) and training. (3620 works fine for a IXP with +60 ISPs).

switch - cost $200-500

Not getting e-mail for every new prefix being announced: priceless -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org

...

...

switch - cost $200-500 Not getting e-mail for every new prefix being announced: priceless

rofl!

but folk not at linx will not understand this 'joke'. they're lucky.

and, from an american commercial point of view, this week's wonderful linx-ism, linx admin publicly announcing that a peer's bill is overdue, and giving their name etc., was absolutely over the top.

randy

A peer's bill is overdue? But I thought the Lords were becoming even more irrelevant?

but folk not at linx will not understand this 'joke'. they're lucky.

LINX and every other exchange point in Europe and several large US ISPs, and as I understand it there are other IXP's in the world that have this practise, whilst I don't think its useful, other people do.

and, from an american commercial point of view, this week's wonderful linx-ism, linx admin publicly announcing that a peer's bill is overdue, and giving their name etc., was absolutely over the top.

No it wasn't. It was done for operational reasons, I think most people would rather that they knew that one of their peers had been taken down and why that had happened in case it affected their customers. Clearly you don't feel so strongly about good customer service, I and I believe the majority of the LINX membership do believe in good customer service. You don't have any kind of involvement with other European Exchange points do you Randy? Regards, Neil.

Bill Woodcock writes

> One is a "route collector", with which exchanges participants peer in > order to give the exchange operator a view into what is going on > > Another is a "looking glass" which allows some group (ranging from > participants to the general public) to see layer 3 adjacencies

Hmmm... These would seem to me to be the same thing, just a difference of who's allowed to log in. I'd call both of these a looking glass.

I guess I see them as two functions which may or may not be connected. The route collector does the work of peering with the participants. An exchange operator can interact with a route collector as if it were a standard router; there is no need for any interface other than the cli (or other standard interface) for the trouble shooting function. The looking glass can be tied to the same information source, but its primary function is to present that data. In a lot of cases the interface used for that isn't different from the interface used by the exchange operator, so the two are strongly tied. They can be weakly tied, though, as the looking glass might use a different information source (say, a participant's router) or may have filters which limit what data can be presented to the looking glass's user base. Does that match your sense of the functional difference? regards, Ted Hardie

BGP Route Reflector IXPs need a AS number. I'll send you a URL with a whitepaper. The BGP Route Reflector IXPs have proved to offer a low entry cost for ISPs (for those places that do not have the deep pockets to get big routers). It also does not remove the option of scaling to point-to-point peering (pointed out below does not need an AS number) or route server (which also does not need a AS number).

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Randy Bush Sent: Saturday, January 06, 2001 6:36 AM To: deen@slt.lk Cc: nanog@merit.edu Subject: Re:

...

Is a separate AS Number required for an Internet Xchange point network.

almost all ixen are at layer two. hence they have no ip addresses, let alone layer three routing.

of course, the ix provider may also choose to provide services on the side, such as ntp, measurement, ... those are servers and run at layer three+. hence they are routed. like other routed networks, if they are multi- homed, they tend to have their own asn.

randy

Randy;

...

BGP Route Reflector IXPs need a AS number. I'll send you a URL with a whitepaper. The BGP Route Reflector IXPs have proved to offer a low entry cost for ISPs (for those places that do not have the deep pockets to get big routers).

except that big routers are not needed for small-isp exchanges. remember, an isp participating in such an exchange has only to add the prefixes of their local peers to their routing, typically a dozen or so. there are very successful layer-two exchanges where the peers use what we think of as cpe routers, e.g. cisco 2501s. and what's nice is that this is on the right path to exchange growth.

l3 exchange ponints are a labor suck and are fragile.

Maybe. However, l2 is for telco. l2 exchange ponints are a labor suck and are fragile. The right path is l1, though, then, there is less reason to have exchange points. It will be more obvious as the peering speed between two ISPs exceeds that of a single physical interface. Masataka Ohta

You spelled 'wisdom' wrong Randy. Now be nice, eh?

...

Maybe. However, l2 is for telco.

l2 exchange ponints are a labor suck and are fragile.

The right path is l1, though, then, there is less reason to have exchange points.

It will be more obvious as the peering speed between two ISPs exceeds that of a single physical interface.

glad to have words of practical wisedom from your experience as a large provider.

randy

-- () ascii ribbon campaign - against html mail /\ - against ms attachments

Daniel;

This is an especially strange comment, as almost everyone who peers, interconnects in multiple places - thus, exceeding the capacity of a single interface.

I mean peering speed between a single pair of ISPs at a single exchange (or peering) point exceeds that of a single interface. And, if you need many, say 10, interfaces, l1 have all the flexibilities Vadim want.

Layer 1 peering (or pooling, as it's more usually known) is great for interconnecting fiber networks, fast provisioning, and all that.

You may say that we are not ready for full fiber networking, yet. But, we, at least Randy, are talking about "the right path". OK?

However, I fail to see the connection between Layer 1 interconnection and an IP exchange point of any kind. This seems apples and oranges.

Except that there are private peering at exchange points.

Layer 2 exchange points are the only efficient way to go for IP traffic. History and the "invisible hand" of the market have endorsed this path.

It is merely that an l3 exchange point over an l2 shared medium is a bad idea. Masataka Ohta

Bill;

...

I mean peering speed between a single pair of ISPs at a single exchange (or peering) point exceeds that of a single interface.

And, if you need many, say 10, interfaces, l1 have all the flexibilities Vadim want.

...

Layer 1 peering (or pooling, as it's more usually known) is great for interconnecting fiber networks, fast provisioning, and all that.

You may say that we are not ready for full fiber networking, yet.

Any given interface is inherently rate limited. When demand exceeds the capacity, something must be done. Often this is done w/ "striping" or "muxing" where multiple "low-speed" channels are "bonded" into a single virtual path. L1 is not that different than L2 & L3 in these cases. The specific dynamics are unique per layer but the problem remains the same.

When l1 is p2p, you can do nothing at l2, unless you additionally introduce l2 switches which are as expensive as and often slower (as is the case with ATM) than l3 routers. But, the point here is that it is not or will not be necessary to share a single "high-speed" channel for "low-speed" pathes to multiple ISPs through L2 switches. Masataka Ohta

Bill;

...

...

Any given interface is inherently rate limited. When demand exceeds the capacity, something must be done. Often this is done w/ "striping" or "muxing" where multiple "low-speed" channels are "bonded" into a single virtual path. L1 is not that different than L2 & L3 in these cases. The specific dynamics are unique per layer but the problem remains the same.

When l1 is p2p, you can do nothing at l2, unless you additionally introduce l2 switches which are as expensive as and often slower (as is the case with ATM) than l3 routers.

when L1 is p2p (generally true for most deployed L1 technologies) you still need something to originate/terminate the information transmission. and when the offered load exceeds the L1 capabilities, you are stuck. Striping or changing L1 media are the primary options.

And we rely on l3.

Your other points wrt introduction of more stuff are true.

I never said we should get rid of l3. :-)

...

But, the point here is that it is not or will not be necessary to share a single "high-speed" channel for "low-speed" pathes to multiple ISPs through L2 switches.

Thats true today and has been for several decades.

Partly because telco has been offering l2 service to carry l3. Things are changing partly because dark fibers, raw l1 media, are available.

But it might be desirable, for any number of reasons.

The primary reason should be to make telco profitable forever. Masataka Ohta

Randy;

...

Maybe. However, l2 is for telco.

l2 exchange ponints are a labor suck and are fragile.

The right path is l1, though, then, there is less reason to have exchange points.

It will be more obvious as the peering speed between two ISPs exceeds that of a single physical interface.

glad to have words of practical wisedom from your experience as a large provider.

My humble wisdom is not to have experience with telco style (thus large) ISPs with impractical (that is, extremely costly) internetworking. :-) Masataka Ohta

There are a number of boxes that can do this, or are in beta. It would be a horrific mistake to base an exchange point of any size around one of them. Talk about difficulty troubleshooting, not to mention managing the exchange point. Get a Foundry BigIron 4000 or a Riverstone SSR. Exchange point in a box, so to say. The Riverstone can support the inverse-mux application nicely, on it's own, as can a Foundry, when combined with a Tiara box. Daniel Golding NetRail,Inc. "Better to light a candle than to curse the darkness" On Mon, 8 Jan 2001, Vadim Antonov wrote:

There's another option for IXP architecture, virtual routers over a scalable fabric. This is the only approach which combines capacity of inverse-multiplexed parallel L1 point-to-point links and flexibility of L2/L3 shared-media IXPs. The box which can do that is in field trials (though i'm not sure the current release of software supports that functionality).

--vadim

On Tue, 9 Jan 2001, Masataka Ohta wrote:

...

Randy;

...

...

BGP Route Reflector IXPs need a AS number. I'll send you a URL with a whitepaper. The BGP Route Reflector IXPs have proved to offer a low entry cost for ISPs (for those places that do not have the deep pockets to get big routers).

except that big routers are not needed for small-isp exchanges. remember, an isp participating in such an exchange has only to add the prefixes of their local peers to their routing, typically a dozen or so. there are very successful layer-two exchanges where the peers use what we think of as cpe routers, e.g. cisco 2501s. and what's nice is that this is on the right path to exchange growth.

l3 exchange ponints are a labor suck and are fragile.

Maybe. However, l2 is for telco.

l2 exchange ponints are a labor suck and are fragile.

The right path is l1, though, then, there is less reason to have exchange points.

It will be more obvious as the peering speed between two ISPs exceeds that of a single physical interface.

Masataka Ohta

At 03:32 -0800 2001/01/09, Vadim Antonov wrote:

You mean you really have any other option when you want to interconnect few 300 Gbps backbones? :) Both mentioned boxes are in 120Gbps range fabric capacity-wise. If you think that's enough, i'd like to point out at the DSL deployment rate. Basing exchange points at something which is already inadequate is a horrific mistake, IMHO.

Exchange points are major choke points, given that 80% or so of traffic crosses an IXP or bilaterial private interconnection. Despite the obvious advantages of the shared IXPs, the private interconnects between large backbones were a forced solution, purely for capacity reasons.

--vadim

exchange points being choke points are more complex than that: - backbones direct interconnect because it makes what was public traffic stats now private. also is more financially sound model than a 3rd party being involved. it minimize expenses. - backbones limiting bandwidth into an Exchange Point also makes it a choke point. - pulling out of an Exchange or demoting it's importance to a particular backbone means a justification for not having equitable peering. - knowing so much traffic goes between backbones makes it a political tug of war that brought on direct interconnects. - private interconnects were not a forced solution. they were for revenue and political, not purely for capacity reasons. there has been this notion of Tier 1,2,3 ... because of this. - equitable financial return at an Exchange. means turning smaller peers into customers. i am sure i have not nearly covered everything here. -craig

On Mon, 8 Jan 2001, Daniel L. Golding wrote:

...

There are a number of boxes that can do this, or are in beta. It would be a horrific mistake to base an exchange point of any size around one of them. Talk about difficulty troubleshooting, not to mention managing the exchange point. Get a Foundry BigIron 4000 or a Riverstone SSR. Exchange point in a box, so to say. The Riverstone can support the inverse-mux application nicely, on it's own, as can a Foundry, when combined with a Tiara box.

Daniel Golding NetRail,Inc. "Better to light a candle than to curse the darkness"

On Mon, 8 Jan 2001, Vadim Antonov wrote:

...

There's another option for IXP architecture, virtual routers over a scalable fabric. This is the only approach which combines capacity of inverse-multiplexed parallel L1 point-to-point links and flexibility of L2/L3 shared-media IXPs. The box which can do that is in field trials (though i'm not sure the current release of software supports that functionality).

--vadim

The use of a BGP Route Reflector at an L2 IXP does not make it a L3 IXP. IMHO - L3 IXPs do not work any more. The "IXs" that use them are really 'international transit services' using the "IX" term to sell their service. The IXPs with lots of 2501s have scaled because of BGP Route Reflectors. The RR is a cheap low maintenance, low learning curve alternative to a Router Server. We know that cause it has proven it self in action for the past four years. Over time, the IXP can transition from a RR to a RS - like some large IXPs are doing now.

...

BGP Route Reflector IXPs need a AS number. I'll send you a URL with a whitepaper. The BGP Route Reflector IXPs have proved to offer a low entry cost for ISPs (for those places that do not have the deep pockets to get big routers).

except that big routers are not needed for small-isp exchanges. remember, an isp participating in such an exchange has only to add the prefixes of their local peers to their routing, typically a dozen or so. there are very successful layer-two exchanges where the peers use what we think of as cpe routers, e.g. cisco 2501s. and what's nice is that this is on the right path to exchange growth.

l3 exchange ponints are a labor suck and are fragile.

On Tue, Jan 09, 2001 at 09:37:32AM -0800, Randy Bush wrote:

here i quibble. i don't really like either reflectors or route servers. let's ignore route servers.

:-)

so, rather than do the rr thing, which is labor intensive and a dead end in the long run, just jump to routers that can handle it, like small 36xx (or 26xx if they have the ram, i really don't know that end of your products well).

Many small providers aren't very good at filtering their routes in the first place. Sloppiness in a mini-nap is expected. One person with a little clue and a route server can make _everyones_ lives easier in such a situation. Scaling routers is relatively easy. Scaling clue is harder. :-) (Note that I wont disagree that in such a situation, full-mesh peering is a very viable option.)

randy

-- Jeffrey Haas - Merit RSng project - jeffhaas@merit.edu