Do ATM-based Exchange Points make sense anymore?
Hi all - I've been working with a number of ISPs on a research paper that builds on the previous peering research papers (Internet Service Providers and Peering, A Business Case for Peering, The Art of Peering, Interconnection Strategies for ISPs, etc.) that applies the Peering Modelling tools in a comparison of ATM and Ethernet-based Internet Exchanges. Both of these IXes are compared against each other and against the cost of buying transit. The paper applies recent price quotes for transport and transit, costs for ATM and Ethernet-based IX participation, to answer the question: Do ATM-based Exchange Points make sense anymore? I'd like to speak with additional ISP Peering Coordinators and Network Architects (preferable ones that have experience with peering across both ATM and Ethenet-based IXes) to walk through this paper and help me check that I have the technical and business details right. I would need about 20 minutes or so on the phone to walk you through the paper, the financial models, the cost points, and get feedback on the conclusions...preferably sometime in the next couple weeks. If you are a Peering Coordinator I think you will find at least a couple of findings in this research *very* interesting. In any case, if you can help, please send me an e-mail at wbn@equinix.com and let me know when we could chat. Thanks - Bill PS - As with any these Peering White Papers, this white paper will be freely available once enough folks have walked through it and verify that we have things right. ------------------------------------------ Abstract --------------------------------------------------- During the NSFNET transition from the Authorized Use Policy Internet to the Commercial Internet, several Network Access Points (NAPs) were created to facilitate the traffic exchange between the Internet Service Providers, two of which were ATM-based. Internet Service Providers were initially required to connect to three of the four NAPs in order to receive NSF funds (indirectly through their NSF-sponsored customers) during this transition period. During the years that followed, this requirement was dropped and the costs models of Internet Operation have changed dramatically. Technologies such as Wave Division Multiplexing and Long Haul Fiber Improvements have led to radical a decrease in the cost of transport and a corresponding drop in the price of transit. At the same, the cost of peering at ATM-based exchange points has not substantially dropped in cost, leading to the question in the Peering Coordinator Community: "Do ATM-based Internet Exchange Points make sense anymore?" In this paper we apply the peering financial models to this question, using current market prices to compare the price of transit against the costs of peering at ATM-based NAPs and Ethernet-based Internet Exchange Points. We build upon the previous research on Peering by introducing the notion of an Effective Peering Range (EPR) to describe the "useful life" of an Internet Exchange. We also highlight a potentially costly EPR Gap, an interim range between Peering Capacity points where peering is more expensive than transit. The financial models presented that produced the graphs are included in the Appendix so that ISPs can apply these cost models to their specific situation.
Hi again - A couple points (based on some interactions with folks privately). This is not an ATM is bad, or general ATM-bashing paper. It simply applies the same Peering Analysis that ISPs are applying to determine if and when IXes make sense. With the transit prices and transport prices dropping, this is a reasonable question, worthy of greater analysis than "well, ATM is expensive so ATM is bad." To give you a flavor, given a set of assumptions, OC-3 (155Mbps) transport into an ATM-based IX has an "Effective Peering Range" (where peering across them is cheaper than transit) of 75-90Mbps, while given the same assumptions, Fast Ethernet-based IXes also at OC-3 have an Effective Peering Range of 40-70Mbps. The "Minimum Cost of Traffic Exchange" for this ATM solution is $122/Mbps while FastE is $80/Mbps. At higher capacity the interconnect analysis is more dramatic: Given the relatively high price point of transport and port cost, the Effective Peering Range for ATM/OC-12 Peering is a narrow 236Mbps to 375Mbps with a Minimum Cost of Traffic Exchange of $69/Mbps. The GigE/OC-12 equivalent range is 109Mbps-466Mbps with a Minimum Cost of Traffic Exchange of $25/Mbps. What was unexpected in this analysis was the Effective Peering Range Gap. When an ISP upgrades the ATM OC-3 to OC-12, the gap between the Effective Peering Bandwidth of the OC-3 (90Mbps) and the Peering Breakeven Point (the point at which the Peering Costs are totally offset by the cost savings of peering vs. transit) at 208Mbps is huge. This 118Mbps gap is where an ISP should rationally prefer to purchase transit until 208Mbps can be sent in peering relationships over the ATM fabric, and only then upgrade the peering connection to OC-12! There is also an Ethernet EPR Gap but it is only about 40 Mbps, and once at the GigE/OC-12 capacity, it gets you an Effective Peering Range up to 475Mbps. In any case, this is the analysis that the paper walks through, and since the spreadsheets are in the paper, one can muck around with the assumptions and cost points, key of which are: 1) ATM OC-3 Port Cost $8000/mo, ATM OC-3 Circuit Cost $3000/mo, ATM OC-12 Port Cost $17000/mo, ATM OC-12 Circuit Cost $8000/mo 2) FastE Port & Rack Space $2500/mo, OC-3 Circuit $3500/mo, GigE Port & Rack Price$5000/mo, OC-12 Circuit $7000/mo 3) Transit Price: if you peer at OC-3, you probably pay $125/Mbps, peer at OC-12,$110/Mbps 4) ATM Overhead (aka cell tax): 20% 5) Assumption that ISP upgrade capacity when avg utilization >75% Effective Peering BW Let me know if you violently object to any of these data points. These are culled from a lot of conversations in the field. The rest of the paper is simply plugging these data points into the equations and analyzing the results. Bill At 04:36 PM 8/7/2002 -0700, William B. Norton wrote:
Hi all -
I've been working with a number of ISPs on a research paper that builds on the previous peering research papers (Internet Service Providers and Peering, A Business Case for Peering, The Art of Peering, Interconnection Strategies for ISPs, etc.) that applies the Peering Modelling tools in a comparison of ATM and Ethernet-based Internet Exchanges. Both of these IXes are compared against each other and against the cost of buying transit. The paper applies recent price quotes for transport and transit, costs for ATM and Ethernet-based IX participation, to answer the question:
Do ATM-based Exchange Points make sense anymore?
I'd like to speak with additional ISP Peering Coordinators and Network Architects (preferable ones that have experience with peering across both ATM and Ethenet-based IXes) to walk through this paper and help me check that I have the technical and business details right. I would need about 20 minutes or so on the phone to walk you through the paper, the financial models, the cost points, and get feedback on the conclusions...preferably sometime in the next couple weeks.
If you are a Peering Coordinator I think you will find at least a couple of findings in this research *very* interesting. In any case, if you can help, please send me an e-mail at wbn@equinix.com and let me know when we could chat.
Thanks -
Bill
PS - As with any these Peering White Papers, this white paper will be freely available once enough folks have walked through it and verify that we have things right.
------------------------------------------ Abstract --------------------------------------------------- During the NSFNET transition from the Authorized Use Policy Internet to the Commercial Internet, several Network Access Points (NAPs) were created to facilitate the traffic exchange between the Internet Service Providers, two of which were ATM-based. Internet Service Providers were initially required to connect to three of the four NAPs in order to receive NSF funds (indirectly through their NSF-sponsored customers) during this transition period.
During the years that followed, this requirement was dropped and the costs models of Internet Operation have changed dramatically. Technologies such as Wave Division Multiplexing and Long Haul Fiber Improvements have led to radical a decrease in the cost of transport and a corresponding drop in the price of transit. At the same, the cost of peering at ATM-based exchange points has not substantially dropped in cost, leading to the question in the Peering Coordinator Community:
"Do ATM-based Internet Exchange Points make sense anymore?"
In this paper we apply the peering financial models to this question, using current market prices to compare the price of transit against the costs of peering at ATM-based NAPs and Ethernet-based Internet Exchange Points. We build upon the previous research on Peering by introducing the notion of an Effective Peering Range (EPR) to describe the "useful life" of an Internet Exchange. We also highlight a potentially costly EPR Gap, an interim range between Peering Capacity points where peering is more expensive than transit.
The financial models presented that produced the graphs are included in the Appendix so that ISPs can apply these cost models to their specific situation.
--------------------------------------------------------------------------------------------------------------- William B. Norton <wbn@equinix.com> 650.315.8635 Co-Founder and Chief Technical Liaison Equinix, Inc. Yahoo Instant Messenger ID: WilliamBNorton
Everything I say below is IMHO.
This is not an ATM is bad, or general ATM-bashing paper. It simply applies the same Peering Analysis that ISPs are applying to determine if and when IXes make sense. With the transit prices and transport prices dropping, this is a reasonable question, worthy of greater analysis than "well, ATM is expensive so ATM is bad."
Yes. ATM is priced expensive from the ATM NAP providers, and no one seems to know why. Personally, I don't believe that ATM is 'bad' for shared-fabric exchange point. I mean, it works, and solves several problems quite easy: a) it's easily distributed via SONET services to folks who are not next to the ATM switch, b) it makes interconnection between networks safer (ie, not dealing with broadcast issues on a ethernet nap), c) virtual PI connections are easily accomplished, d) there are varying degrees of interconnection speed (agreeably, less important), e) it allows for things other than IP, or packet-based traffic to be exchanged (a la Verizons' video-portal service) (agreeably, again, less important).
To give you a flavor, given a set of assumptions, OC-3 (155Mbps) transport into an ATM-based IX has an "Effective Peering Range" (where peering across them is cheaper than transit) of 75-90Mbps, while given the same assumptions, Fast Ethernet-based IXes also at OC-3 have an Effective Peering Range of 40-70Mbps. The "Minimum Cost of Traffic Exchange" for this ATM solution is $122/Mbps while FastE is $80/Mbps.
The pricing model is wholy irrelevant of the platform used. I think that it just happens to be that AADS and WCOM rape the folks who peer.
At higher capacity the interconnect analysis is more dramatic: Given the relatively high price point of transport and port cost, the Effective Peering Range for ATM/OC-12 Peering is a narrow 236Mbps to 375Mbps with a Minimum Cost of Traffic Exchange of $69/Mbps. The GigE/OC-12 equivalent range is 109Mbps-466Mbps with a Minimum Cost of Traffic Exchange of $25/Mbps.
Agreed, however, even at $25/meg, when you factor in the cost of equipment, rack space, and the build of network to get you to the place to peer, you can easily be over the price of purchasing bandwidth again.
What was unexpected in this analysis was the Effective Peering Range Gap. When an ISP upgrades the ATM OC-3 to OC-12, the gap between the Effective Peering Bandwidth of the OC-3 (90Mbps) and the Peering Breakeven Point (the point at which the Peering Costs are totally offset by the cost savings of peering vs. transit) at 208Mbps is huge. This 118Mbps gap is where an ISP should rationally prefer to purchase transit until 208Mbps can be sent in peering relationships over the ATM fabric, and only then upgrade the peering connection to OC-12!
Certainly a problem, but this should more be addressed as a problem with price points that they charged not being relative to actual costs.
In any case, this is the analysis that the paper walks through, and since the spreadsheets are in the paper, one can muck around with the assumptions and cost points, key of which are: 1) ATM OC-3 Port Cost $8000/mo, ATM OC-3 Circuit Cost $3000/mo, ATM OC-12 Port Cost $17000/mo, ATM OC-12 Circuit Cost $8000/mo 2) FastE Port & Rack Space $2500/mo, OC-3 Circuit $3500/mo, GigE Port & Rack Price$5000/mo, OC-12 Circuit $7000/mo 3) Transit Price: if you peer at OC-3, you probably pay $125/Mbps, peer at OC-12,$110/Mbps 4) ATM Overhead (aka cell tax): 20% 5) Assumption that ISP upgrade capacity when avg utilization >75% Effective Peering BW
I am not looking to start a "my transit provider is cheaper than yours" arguement, but at 100 mbit/second committments, they are a wide variety of providers in the $50 to $100/meg range (with a nice cluster around $75). -- Alex Rubenstein, AR97, K2AHR, alex@nac.net, latency, Al Reuben -- -- Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
> Personally, I don't believe that ATM is 'bad' for > shared-fabric exchange point. I mean, it works, and solves several > problems quite easy: a) it's easily distributed via SONET services to > folks who are not next to the ATM switch, b) it makes interconnection > between networks safer (ie, not dealing with broadcast issues on a > ethernet nap), c) virtual PI connections are easily accomplished, d) there > are varying degrees of interconnection speed (agreeably, less important), All of the above are true of frame relay as well, which has the additional benefit of not being funamentally incompatible with data networking. :-) > e) it allows for things other than IP, or packet-based traffic to be > exchanged (a la Verizons' video-portal service) (agreeably, again, less > important). Ah, yes, I agree with this entirely. If you're building a voice exchange, rather than a data exchange, ATM is excellent technology. I believe Vinnie can give us some personal experience on this front. -Bill
On Fri, Aug 09, 2002 at 01:13:04PM -0700, Bill Woodcock wrote:
> Personally, I don't believe that ATM is 'bad' for > shared-fabric exchange point. I mean, it works, and solves several > problems quite easy: a) it's easily distributed via SONET services to > folks who are not next to the ATM switch, b) it makes interconnection > between networks safer (ie, not dealing with broadcast issues on a > ethernet nap), c) virtual PI connections are easily accomplished, d) there > are varying degrees of interconnection speed (agreeably, less important),
All of the above are true of frame relay as well, which has the additional benefit of not being funamentally incompatible with data networking. :-)
I doubt that any of the ATM-based echanges were built because of a deep affection for ATM. More likely, it was the only virtual circuit techonlogy around at the the time that a certain router vendor supported at speeds greater than DS3. ATM worked reasonably well for that application, once there were switches with adequate buffering. Anyone building a similar exchange today would have new choices not available three or more years ago. Steve
At 01:13 PM 8/9/2002 -0700, Bill Woodcock wrote:
> Personally, I don't believe that ATM is 'bad' for > shared-fabric exchange point. I mean, it works, and solves several > problems quite easy: a) it's easily distributed via SONET services to > folks who are not next to the ATM switch, b) it makes interconnection > between networks safer (ie, not dealing with broadcast issues on a > ethernet nap), c) virtual PI connections are easily accomplished, d) there > are varying degrees of interconnection speed (agreeably, less important),
All of the above are true of frame relay as well, which has the additional benefit of not being funamentally incompatible with data networking. :-)
You guys might find this interesting.... I'd like to share the more common "Religious debate points" regarding ATM-based vs. Ethernet-based IXes that I heard during the walk throughs (about 50 so far) of this paper (v1.6): ------------------------------------------------------------------------------------------------------------------ ATM Advocate: First off, make sure you mention that ATM solves the key problem with "Broadcast Domain Internet Exchanges". Broadcast Domain Issues refers to problems when all ISP attachments are on the same Ethernet segment. Broadcast storms and other anomalies caused by one attached customer can adversely affects all others. Ethernet-based IX operators try and solve this problem administratively with MOUs Memorandum of Understanding (see the Appendix of the http://www.linx.net/joining/mou.thtml for an example of this) but it is solved in ATM by the private nature of PVCs, yielding a more stable peering infrastructure. Ethernet Advocate: Ethernet-based IXes can address these "Broadcast Domain" issues technically via private VLANs and direct cross connects between members. The "nature" of ATM as you describe it has a high overhead associated with it, specifically, by statically allocating bandwidth to a peering session with a historically spiky cyclical traffic characteristic. This static allocation of bandwidth prevents the multiplexing benefits of aggregation of lots of peering traffic sources. In addition to Ethernet-IXes being generally less expensive than ATM-based IXes, Ethernet interfaces are generally less expensive. This reduces the total peering costs, breakeven points, and increases the attractiveness of the Ethernet-based IX and therefore its likelihood of succeeding. Finally, Ethernet is what ISPs know, it is what they love. This ubiquity of and familiarity with Ethernet reduces the costs of operation, since operations folks only need to know how the Ethernet stuff works. ATM Advocate: Most Ethernet-based IXes primarily use the default LAN and are therefore subject to the "Broadcast Domain" issues. Ethernet may be less expensive but it is not shaping the traffic as ATM does. You pay for that functionality and stability. As for the operations argument, naturally one technology is easier to support than multiple technologies. This isn't a specific fault of ATM or Ethernet but rather and indication that choosing one and sticking with one is advantageous. Ethernet Advocate: In the Ethernet-based IX model, private cross connects allow one to scale beyond OC-12, the maximum reasonable capacity of ATM. The OC-48 ATM cards cost $250K, making it unreasonably expensive for the exchange of Internet Peering traffic. At the same time, the Ethernet-based model allows collocated folks to interconnect at Gigabit Ethernet, 10-GigE, whatever the peers choose. These direct connects are typically not allowed (for policy reasons) to occur at collocated ATM IXes. And PVCs are not the same things as a PNI as there are active electronics in the middle, some thing that can break, obscure troubleshooting, and limit flexibility with respect to interconnect types. ------------------------------------------------------------------------ Interesting points, and although orthogonal to the analysis in "Do ATM-based Internet Exchange Points Make Sense Anymore?", I am including these in the appendix to show these alternate views of the world. Am I missing any of the major (fact-based) views? Bill
Interesting points, and although orthogonal to the analysis in "Do ATM-based Internet Exchange Points Make Sense Anymore?", I am including these in the appendix to show these alternate views of the world. Am I missing any of the major (fact-based) views?
There is this "small" thing that higher speed ATM interfaces are not actively being developed. It´s very hard to come by STM16 ATM interface on a router with STM64 being non-existent on both switches and routers. This is not due technological barriers but commercial reality. ATM would be going away faster if DSLAM´s wouldn´t be there increasing the traffic, but because of the extensive investments to ADSL gear, it´ll be there in 5, probably 10 years time. Because of the above reason, there is no point in setting up an ATM based exchange today. The first one here which was built 1996 is being retired by the end of year, and the other probably sometime next year. Pete
participants (5)
-
Alex Rubenstein -
Bill Woodcock -
Petri Helenius -
Steve Feldman -
William B. Norton