More Questions of Exchange Points
Hi, I am a graduate student of computer science, now I am working on Internet exchange points. Following the discussion of exchange points last month, I have some more specific questions. 1, Some of the exchange points are layer 2 facilities, then why do they need register IP addresses? Furthermore, those IP addresses do appears in the traceroute traces (from the skitter data of caida). Does this mean that these IP addresses are actually in use? Then how the layer 2 exchange points work? 2, How do you categorize the exchange points into large/local IXs, transit/peering IXs (besides look into the peering policies)? From the number of participants? Or from who are the participants? Thanks. Ruomei Gao Email: gte489q@prism.gatech.edu
> 1, Some of the exchange points are layer 2 facilities, then why do they need > register IP addresses? Furthermore, those IP addresses do appears in the > traceroute traces (from the skitter data of caida). Does this mean that these > IP addresses are actually in use? Nearly all exchange points are layer 2. This means that they consist of a layer 2 switch, normally Ethernet, on rare occasions ATM or frame relay, or even something more exotic. The participating ISPs bring routers, which they all connect to the switch. Each of those routers must have an IP address in order to communicate with the others, and the IP addresses must all be within the same subnet. That being the case, the correct procedure is for a block of addresses to be allocated to the exchange, rather than through any one of the ISPs, so that the rest of the participants aren't dependent upon any one ISP which might be providing the address space. Also, that way no ISP is forced to provide transit for the exchange-point addresses, which theoretically don't need it. So yes, those addresses are very much in use, but in somewhat the same way that the /30 on a point-to-point link would be. As a means for the two adjacent routers to communicate, and pass on traffic which is coming from and going to points much more distant. > 2, How do you categorize the exchange points into large/local IXs, > transit/peering IXs (besides look into the peering policies)? From the number > of participants? Or from who are the participants? The difference between a local and a regional exchange is typically one of size of participants. It isn't a technical difference, so it might be a little hard to arrive at complete consensus on, with respect to any particular exchanges. It's probably safe to say that in the U.S., PAIX in Palo Alto, MAE-East ATM, and Equinix Ashburn are regional exchanges, that in Europe the LINX and AMS-IX are regional exchanges, and that in Asia JPIX, NXP-ISP2, and HKIX are regional exchanges. They're where large regional ISPs would go to peer with ISPs from outside the region. A regional exchange would often be thought of as one that an ISP from outside the region would go to first. The difference between a peering exchange and a transit exchange is a much more easily technically-defined difference: a peering exchange is one across which, by and large, the participants just exchange peering routes. A transit exchange is one across which many of the participants are exchange full transit. The consequences of this distinction are pretty far-reaching, and generally mean that only one large peering exchange can exist in a region, and it'll be inexpensive, whereas several smaller, more expensive transit exchanges can coexist in the same region. Phil Smith, Keith Mitchell, and I will be presenting a paper on the topic at the next NANOG in Toronto. -Bill
> 1, Some of the exchange points are layer 2 facilities, then why do they need > register IP addresses? Furthermore, those IP addresses do appears in the > traceroute traces (from the skitter data of caida). Does this mean that these > IP addresses are actually in use?
Nearly all exchange points are layer 2. This means that they consist of a layer 2 switch, normally Ethernet, on rare occasions ATM or frame relay, or even something more exotic. The participating ISPs bring routers, which they all connect to the switch. Each of those routers must have an IP address in order to communicate with the others, and the IP addresses must all be within the same subnet. That being the case, the correct procedure is for a block of addresses to be allocated to the exchange, rather than through any one of the ISPs, so that the rest of the participants aren't dependent upon any one ISP which might be providing the address space. Also, that way no ISP is forced to provide transit for the exchange-point addresses, which theoretically don't need it.
Er, -ALL- exchanges have a layer2 component. Some institute policy at layer 3. Since, in general, we are talking about INTERNET exchanges it makes sense that IP comes into play. If there is a shared medium that is used as a single broadcast domain, then a common subnet makes life easier for everyone. It is possible to use divergent networks (see Sleepy Bills (woodcock) list) on the same shared media. His list argues that these distinct subnets are unique exchanges. My take is that they are not but that is a nit argument. There are lots of ways to slice the exchange point.
So yes, those addresses are very much in use, but in somewhat the same way that the /30 on a point-to-point link would be. As a means for the two adjacent routers to communicate, and pass on traffic which is coming from and going to points much more distant.
What he said. It is the Internet after all.
> 2, How do you categorize the exchange points into large/local IXs, > transit/peering IXs (besides look into the peering policies)? From the number > of participants? Or from who are the participants?
The difference between a local and a regional exchange is typically one of size of participants. It isn't a technical difference, so it might be a little hard to arrive at complete consensus on, with respect to any particular exchanges. It's probably safe to say that in the U.S., PAIX in Palo Alto, MAE-East ATM, and Equinix Ashburn are regional exchanges, that in Europe the LINX and AMS-IX are regional exchanges, and that in Asia JPIX, NXP-ISP2, and HKIX are regional exchanges. They're where large regional ISPs would go to peer with ISPs from outside the region. A regional exchange would often be thought of as one that an ISP from outside the region would go to first.
only the very brave or very foolish will attempt such catagorization. Posh Bill (norton) clearly points out that the value of an exchange, like beauty, lies in the eyes/network of the beholder. local/regional - peering/transit.... the key thing is "whats in it for me?"
The difference between a peering exchange and a transit exchange is a much more easily technically-defined difference: a peering exchange is one across which, by and large, the participants just exchange peering routes. A transit exchange is one across which many of the participants are exchange full transit. The consequences of this distinction are pretty far-reaching, and generally mean that only one large peering exchange can exist in a region, and it'll be inexpensive, whereas several smaller, more expensive transit exchanges can coexist in the same region. Phil Smith, Keith Mitchell, and I will be presenting a paper on the topic at the next NANOG in Toronto.
Humph. Difference w/o (significant) distinction. If -ANY- isp provides transit off the exchange fabric, does that make it a transit exchange? If not, why not? And what about exchanges that have -NO- routing protocol at all? (can you say ARP.... sure you can.) Not peering or transit. Or are they? For me, the key point is that an exchange acts as an aggregation point for the participants. Generating value off aggregation can take many forms. Peering and Transit are but two vectors that are effected by aggregation. --grumpy bill (manning)
Ruomei Gao Email: gte489q@prism.gatech.edu On Sat, 6 Apr 2002 bmanning@karoshi.com wrote:
> 1, Some of the exchange points are layer 2 facilities, then why do they need > register IP addresses? Furthermore, those IP addresses do appears in the > traceroute traces (from the skitter data of caida). Does this mean that these > IP addresses are actually in use?
Nearly all exchange points are layer 2. This means that they consist of a layer 2 switch, normally Ethernet, on rare occasions ATM or frame relay, or even something more exotic. The participating ISPs bring routers, which they all connect to the switch. Each of those routers must have an IP address in order to communicate with the others, and the IP addresses must all be within the same subnet. That being the case, the correct procedure is for a block of addresses to be allocated to the exchange, rather than through any one of the ISPs, so that the rest of the participants aren't dependent upon any one ISP which might be providing the address space. Also, that way no ISP is forced to provide transit for the exchange-point addresses, which theoretically don't need it.
Er, -ALL- exchanges have a layer2 component. Some institute policy at layer 3. Since, in general, we are talking about INTERNET exchanges it makes sense that IP comes into play. If there is a shared medium that is used as a single broadcast domain, then a common subnet makes life easier for everyone. It is possible to use divergent networks (see Sleepy Bills (woodcock) list) on the same shared media. His list argues that these distinct subnets are unique exchanges. My take is that they are not but that is a nit argument. There are lots of ways to slice the exchange point.
So yes, those addresses are very much in use, but in somewhat the same way that the /30 on a point-to-point link would be. As a means for the two adjacent routers to communicate, and pass on traffic which is coming from and going to points much more distant.
What he said. It is the Internet after all.
> 2, How do you categorize the exchange points into large/local IXs, > transit/peering IXs (besides look into the peering policies)? From the number > of participants? Or from who are the participants?
The difference between a local and a regional exchange is typically one of size of participants. It isn't a technical difference, so it might be a little hard to arrive at complete consensus on, with respect to any particular exchanges. It's probably safe to say that in the U.S., PAIX in Palo Alto, MAE-East ATM, and Equinix Ashburn are regional exchanges, that in Europe the LINX and AMS-IX are regional exchanges, and that in Asia JPIX, NXP-ISP2, and HKIX are regional exchanges. They're where large regional ISPs would go to peer with ISPs from outside the region. A regional exchange would often be thought of as one that an ISP from outside the region would go to first.
only the very brave or very foolish will attempt such catagorization. Posh Bill (norton) clearly points out that the value of an exchange, like beauty, lies in the eyes/network of the beholder. local/regional - peering/transit.... the key thing is "whats in it for me?"
The difference between a peering exchange and a transit exchange is a much more easily technically-defined difference: a peering exchange is one across which, by and large, the participants just exchange peering routes. A transit exchange is one across which many of the participants are exchange full transit. The consequences of this distinction are pretty far-reaching, and generally mean that only one large peering exchange can exist in a region, and it'll be inexpensive, whereas several smaller, more expensive transit exchanges can coexist in the same region. Phil Smith, Keith Mitchell, and I will be presenting a paper on the topic at the next NANOG in Toronto.
Humph. Difference w/o (significant) distinction. If -ANY- isp provides transit off the exchange fabric, does that make it a transit exchange? If not, why not?
And what about exchanges that have -NO- routing protocol at all? (can you say ARP.... sure you can.) Not peering or transit. Or are they?
For me, the key point is that an exchange acts as an aggregation point for the participants. Generating value off aggregation can take many forms. Peering and Transit are but two vectors that are effected by aggregation.
--grumpy bill (manning)
Sorry about the empty mail sent by mistake just now.
list) on the same shared media. His list argues that these distinct subnets are unique exchanges. My take is that they are not but that is a nit argument. There are lots of ways to slice the exchange point.
I did observe 2 exchange points have direct connection between them, does it mean they belong to the same switch fabiric?
The difference between a peering exchange and a transit exchange is a much more easily technically-defined difference: a peering exchange is one across which, by and large, the participants just exchange peering routes.
Do you mean the participants just exchange BGP routing information? So the traceroute data will only discover the peering point they exchange traffic? ISPs exchange their traffic at IXs or private peering points, so which is more important to the ISPs (in term of traffic volume or other measures)? Maybe I should also mention co-locators, then what's the difference between co-locator and the "carrier hotel"? Are they like "physical layer exchange points" (if there is such a concept)? Are there any other methods to exchange traffic between peers?
Humph. Difference w/o (significant) distinction. If -ANY- isp provides transit off the exchange fabric, does that make it a transit exchange? If not, why not?
Are those private peering points? Regards, Ruomei
>> The difference between a peering exchange and a transit exchange is a much >> more easily technically-defined difference: a peering exchange is one >> across which, by and large, the participants just exchange peering routes. > > Do you mean the participants just exchange BGP routing information? So the > traceroute data will only discover the peering point they exchange traffic? The assumtion is that all ISPs exchange routes via BGP. What's at issue is the degree of redundancy in the routes which they're exchanging. If they're purchasing transit at or through a facility, it's to provide reachability to things that they couldn't otherwise reach, either normally or under conditions of failed peering. That makes the service much more critical than peering, which is, by definition, an economic optimization over transit. Thus, people are willing to spend much more money on a facility through which they're putting transit, and they're willing to tolerate a divided marketplace, as long as each facility is able to maintain at least three sellers. > ISPs exchange their traffic at IXs or private peering points, so which > is more important to the ISPs (in term of traffic volume or other > measures)? Maybe I should also mention co-locators, then what's the > difference between co-locator and the "carrier hotel"? Are they like > "physical layer exchange points" (if there is such a concept)? These aren't necessarily useful distinctions you're making. They're distinctions of marketing positioning. What matters economically and technically is how people use the facilities, not what they're called. -Bill
On Sat, 6 Apr 2002, I wrote: > The assumption is that all ISPs exchange routes via BGP. Before someone else finds a need to argue with this, I'll clarify it myself: The assumption is that all ISPs WHICH exchange routes DO SO via BGP. -Bill
>> The difference between a peering exchange and a transit exchange is a much >> more easily technically-defined difference: a peering exchange is one >> across which, by and large, the participants just exchange peering routes. > > Do you mean the participants just exchange BGP routing information? So the > traceroute data will only discover the peering point they exchange traffic?
The assumtion is that all ISPs exchange routes via BGP. What's at issue
well, there are a fair number of exchanges built on a model developed by B.Greene, when BGP capability was not always there (often not a technology problem... :) They are still a number of them in existence. The exchange routes using an IGP and fate share over the exchange. The BGP'ness occurs at each of their upstreams.
is the degree of redundancy in the routes which they're exchanging. If they're purchasing transit at or through a facility, it's to provide reachability to things that they couldn't otherwise reach, either normally or under conditions of failed peering. That makes the service much more critical than peering, which is, by definition, an economic optimization over transit. Thus, people are willing to spend much more money on a facility through which they're putting transit, and they're willing to tolerate a divided marketplace, as long as each facility is able to maintain at least three sellers.
I really need to read your paper. There appear to be a number of presumptions that are "cultural", for want of a better term, which bias your conclusions.
> ISPs exchange their traffic at IXs or private peering points, so which > is more important to the ISPs (in term of traffic volume or other > measures)? Maybe I should also mention co-locators, then what's the > difference between co-locator and the "carrier hotel"? Are they like > "physical layer exchange points" (if there is such a concept)?
These aren't necessarily useful distinctions you're making. They're distinctions of marketing positioning. What matters economically and technically is how people use the facilities, not what they're called.
Amen.
-Bill
are not but that is a nit argument. There are lots of ways to slice the exchange point.
I did observe 2 exchange points have direct connection between them, does it mean they belong to the same switch fabiric?
What does this mean?
If -ANY- isp provides transit off the exchange fabric, does that make it a transit exchange? If not, why not?
Are those private peering points?
confusion of terms. When bits cross an administrative boundary that can be called a "peering point". Often times that administrative boundary has a policy associated with it. Policies may be implementated via BGP, ACLS, etc. The pathological case is the T1 between Sprint and my home network. The two endpoints of that circuit comprise a peering point. Sprint controls one end, I control the other and we have agreed to fate share a common communications path to swap bits. Multiple parties can agree to share a layer 2 media for exchanging bits. For Internet, I make the distinction that the layer 1 media (glass, copper, freq.) must implement a shared broadcast domain, e.g. I can ARP between the MAC addresses of the connecting devices. Again, for Internet, the presumption is IP. It is conceivable that an operator might get a big'ol switch (layer one) and configure it so that ports 1-10 are one broadcast domain, 11 & 12 are a second broadcast domain, and 13-20 are a third, leaving 21-24 for the fourt broadcast domain. Or... four VLANS. One switch, four networks. Assign an IP subnet for each. That would be four exchanges. Now Zocalo & JAM, running on the first VLAN/exchange are assigned 192.168.10.4 and 192.168.10.5 & can ping/peer with everyone else on VLAN 1-10. HOWEVER, Zocalo & JAM want to do some nifty/cool things that they really don't want anyone else to sniff out. So they create a VPN (extra credit for defining at least four ways to do this... over the SAME VLAN) and use 10.168.10.4 and 10.168.10.5 for their private VPN. So. Is this one exchange point (one switch), four exchange points ( 4 VLANS), or five exchange points ( 5 subnets)? Which ones are public? Which ones are private? and why?
Regards,
Ruomei
On Sun, 7 Apr 2002 bmanning@karoshi.com wrote: > So. Is this one exchange point (one switch), four exchange points > ( 4 VLANS), or five exchange points ( 5 subnets)? Which ones are > public? Which ones are private? and why? Troublemaker! :-) -Bill
I did observe 2 exchange points have direct connection between them, does it mean they belong to the same switch fabiric? What does this mean?
I mean in a trace (from traceroute probing), 2 exchange points (in Mr Woodcock's list) are next to each other. .... ip of AS1, ip of EP1, ip of EP2, ip of AS2, .... I thought they are connected to each other directly (probably the connecion is not as simple as the p2p link between 2 routers).
Are those private peering points?
confusion of terms. When bits cross an administrative boundary [...]
Sorry, I did confuse peering with transit. But I thought those private peering points are somewhat similiar too the IXs, ISPs exchange traffic there and they may also provide transit to the customers there.
So. Is this one exchange point (one switch), four exchange points ( 4 VLANS), or five exchange points ( 5 subnets)? Which ones are public? Which ones are private? and why?
Is this case very common?
I did observe 2 exchange points have direct connection between them, does it mean they belong to the same switch fabiric? What does this mean?
I mean in a trace (from traceroute probing), 2 exchange points (in Mr Woodcock's list) are next to each other.
.... ip of AS1, ip of EP1, ip of EP2, ip of AS2, ....
I thought they are connected to each other directly (probably the connecion is not as simple as the p2p link between 2 routers).
I 'spect that you are seeing something like secondary IP s on the same phys interface.
Are those private peering points?
confusion of terms. When bits cross an administrative boundary [...]
Sorry, I did confuse peering with transit. But I thought those private peering points are somewhat similiar too the IXs, ISPs exchange traffic there and they may also provide transit to the customers there.
Nope. You remain confused. exchanges are not "peering" or "transit" or a floor wax. ISP policies determine if they propgate routes or not and if they are willing to propogate third party routes e.g. routes not originated by their direct neighbors. exchanges are just a way to interconnect between ISPs.
So. Is this one exchange point (one switch), four exchange points ( 4 VLANS), or five exchange points ( 5 subnets)? Which ones are public? Which ones are private? and why?
Is this case very common?
More than you would want to beleive. Now answer the question. --bill
On Sat, 6 Apr 2002, Ruomei Gao wrote:
I did observe 2 exchange points have direct connection between them, does it mean they belong to the same switch fabiric? I mean in a trace (from traceroute probing), 2 exchange points (in Mr Woodcock's list) are next to each other.
.... ip of AS1, ip of EP1, ip of EP2, ip of AS2, ....
I thought they are connected to each other directly (probably the connecion is not as simple as the p2p link between 2 routers).
If they connected directly then you would see only two IPs when tracing through, one on the ISP side of the router connecting the the IX and one on the IX side of the next hop router. Theres prolly some other magic going on to give that trace.
Are those private peering points?
confusion of terms. When bits cross an administrative boundary
Sorry, I did confuse peering with transit. But I thought those private peering points are somewhat similiar too the IXs, ISPs exchange traffic there and they may also provide transit to the customers there.
The rule is there are no rules, stop drawing boxes
So. Is this one exchange point (one switch), four exchange points ( 4 VLANS), or five exchange points ( 5 subnets)? Which ones are public? Which ones are private? and why?
Is this case very common?
Why care, you are now at the level of asking individual ISPs how they configure their network, and theres a lot of ISPs to ask before you can begin to answer your question! As per the previous emails, the only thing that matters here is what you do on your network which therefore defines what you connect with, how you connect up and how you configure. I think your mistaken to believing the Internet is structured and organised in some way! :) Prof Einstein can offer you some wise words to help right now- that it is all relative. Steve
Stephen J. Wilcox wrote (to Ruomei Gao):
I think your mistaken to believing the Internet is structured and organised in some way! :)
The internet was an experiment to design a network resiliant to attack, seeing as www.gov.ps has gone offline when Palestine was attacked, the experiment is a dismal failure, so we should all pack up and get new jobs? :-) The internet has a structure from a given viewpoint, a path layed out before you to any given destination, or at least the rechable ones. And a different structure from any other viewpoint. You are in a maze of twisty passages, all alike. Although slightly more asymmetric than that one. David. The Internet is closed for cleaning. Return to your lives, citizens.
>list argues that these distinct subnets are unique exchanges. Don't anthromophize lists, they don't like it. No argument is being made. The property of principal interest to me in the list is the subnets, so that's what the list is of. Subnets of machines that can talk to each other at layer 2. Many of them overlap in different ways, but by definition, they don't overlap in subnet terms, which is what governs layer 3 reachability over layer 2 media. > only the very brave or very foolish will attempt such > catagorization. Posh Bill (norton) clearly points out that > the value of an exchange, like beauty, lies in the eyes/network > of the beholder. local/regional - peering/transit.... the > key thing is "whats in it for me?" Right, but everyone categorizes exchanges _for their own set of criteria_. It's feckless to attempt to argue that anyone's set of criteria are generally applicable, but it's the intersections of different folks utility-sets which govern the formation and endurance of exchanges. > Difference w/o (significant) distinction. Not at all. Read the paper before making pronouncements about it. > If -ANY- isp provides transit off the exchange fabric, > does that make it a transit exchange? Of course, for that ISP. -Bill
participants (5)
-
Bill Woodcock -
bmanning＠karoshi.com -
David Luyer -
Ruomei Gao -
Stephen J. Wilcox