From this point on the focus will be on the more scalable of the two approaches (the exchange-based interconnection method) and describe the selection criteria an ISP typically uses when selecting an exchange. Note
Hi all - I want to thank all of you who participated in the Peering BOF Monday night at the last NANOG meeting in Montreal. It was lively and very productive. To document the BOF, and for the benefit of those who could not attend NANOG, I am sending out my BOF meeting notes. Comments/Corrections welcome. As an aside, I'd like to thank my old colleagues at Merit for inviting me to return to the stage and MC the Montreal NANOG. It was a lot of fun - Thanks! Hope these notes help. Cheers - Bill ============================================================================ ==== Peering BOF - NANOG 17 - Montreal Monday October 4, 1999 7:30 PM EST Moderator: William B. Norton, Equinix, <wbn@umich.edu> Attendees: About 150? from NANOG meeting The agenda consisted of three items: 1) Ground Rules, 2) Presentation and Discussion of the Peering Decision Tree white paper, and 3) Populating the initial Peering Contact Database to facilitate peering 1) The Ground Rules were ------------------------ A) Not focus on about Peering politics, who gains, etc. B) Not about settlement, nor valuation of peering, etc. C) Instead, Focus on positive things we can do to facilitate peering Capture (document) the essence of peering process 2) Peering Contact Database --------------------------- The Peering Decision Tree interviews (about a dozen) turned up a key challenge ISP Peering Coordinators face: identifying the right staff to speak with at the other ISP. To this end, we helped with this process by having participating Peering Coordinators toss their business cards into the hat. Information they *did not* want in the peering contact database was crossed out. I assembled the cards and e-mailed an excel spreadsheet of all the Peering Coordinators to all the Peering Coordinators that tossed in their cards. <Note: About 50 ISPs tossed in their card and received a copy of the Peering Contact Database. I'm updating the database : if you are a involved in peering and would like to be listed in the Peering Contact Database (and receive a copy of the current Peering Contact Database), please send the relevant contact information (Contact Name & Title, Company Name, address, AS #, peering@<ispdomain>.net address for peering, phone numbers, etc.) to wbn@umich.edu. I will not send the database to those who do not contribute info to the database, nor folks who are not peering coordinators. I'll send out periodic updates as appropriate.> Naturally, the value of the Peering Contact Database to the community is proportional to the number and type of peering coordinators listed. To help maximize this, I'll try to grow the population of this database by repeating the peering BOF at a variety of forums domestic and international. Suggestions welcome. 3) The Peering Decision Tree Discussion --------------------------------------- The majority of the meeting was focused on the Peering Decision Tree. I interviewed about a dozen ISPs and documented the peering process in the Peering Decision Tree white paper. We walked through the paper in the BOF and validated the decision model; this roughly matches the peering coordinator logic. I started to write up a summary of this part of the meeting and found myself rewriting the paper! To save time, I'm simply including a raw text draft of the peering white paper. (Prettier version (pictures, etc.) available as a word document for those who participate in the Peering Contact Database or are willing to be interviewed/add to the document.) Please consider this a work in progress and send me comments! I'll try and incorporate those comments back into the document for the community. ------------------------------- snip ----------------------------------- Peering Decision Tree William B. Norton <wbn@umich.edu> DRAFT v 0.7 Abstract -------- Internet Service Provider (ISP) peering is an interconnection business relationship that decreases the cost and reliance on purchased Internet transit. As the single greatest operating expense, ISPs seek to minimize these telecommunications costs. Interviews with Internet Service Providers have highlighted three distinct decision phases of the peering process: Identification (Traffic Engineering Data Collection and Analysis), Contact & Qualification (Initial Peering Negotiation), and Implementation Discussion (Peering Methodology). The first phases identifies the who and the why, while the last phase focuses on the how. The appendix includes a diagram highlighting key questions asked when identifying peering candidates and determining methods of peering. I. Phase 1: Identification of Potential Peer: Traffic Engineering Data Collection and Analysis ---------------------------------------------------------------------------- ---- Choices made by Internet Service Providers (ISP) are often dominated by telecommunications cost issues. Highest among these costs are Internet transit costs that provide the ISP with connectivity to the global Internet. Transit Prices for DS-3 transit for example vary by circuit mile but can be as high as $50,000/month, and OC-3 transit can cost up to $150,000/month . To reduce these costs, ISPs seek peering (-zero or reduced cost) relationships with other ISPs that provide more direct traffic exchange and reduce the load on these expensive transit services (as shown below). To identify potential peers, ISPs use a variety of criteria. There may be existing business relationships that can be leveraged to include peering. Quantities of traffic distributed between networks often set the pace of the negotiation; to quantify this, ISPs may systematically sample inbound and outbound traffic flows. Flows then are mapped to originating AS, and calculations are made to determine where peering (direct interconnections) would most reduce the load on the expensive transit paths. There is substantial work involved here, as this traffic sampling results in a large number of data. The end result of this first phase is list of the top ISP candidates for peering. The greatly simplified peer qualification decision tree looks something like this: Once the measurements have been made and analyzed, and it appears to be of benefit to peer,, the ISP enters into Phase 2, Contact & Qualification, Initial Peering Negotiation. II. Phase 2: Contact & Qualification, Initial Peering Negotiation ---------------------------------------------------------------------------- ---- Internet Service Providers typically have a person or group specifically tasked with peering and traffic engineering issues. For example, UUNet has a "Peering Steering Committee" to evaluate peering requests. Some variations of the following steps lead to the parties either leaving the negotiation or proceeding to peering methodology discussions. The first step is for one of the parties to initiate contact with the potential peer. Today, discussion starts in one of the following ways: a) as part of a larger business transaction, b) via electronic mail, using the pseudo standard peering@<ispdomain>.net or a personal contact, c) from contacts listed on an exchange point participant list, d) with tech-c or admin-c from DNS or ASN registries, e) informal meeting in an engineering forum like NANOG, IETF, RIPE, etc., f) at trade shows from introductions among speakers, or with booth staff, g) from the target ISP sales force, h) from the target ISP NOC. The interviews to date have highlight a key challenge for ISPs: finding the right person to speak with is a difficult and time intensive process. Mergers and acquisitions further cloud lines of communication. This is where "people networking" helps a great deal, and hiring experience for their contacts speeds this initial contact process up quite a bit. Second, mutual non-disclosures may be negotiated and signed, and a discussion of peering policy and prerequisites follow. Traffic engineering discussions and data disclosure may be used to justify the peering relationship. Each ISP typically has a set of requirements for peering that include peering at some number of geographically distributed locations, sometimes at public exchange points. Traffic volume is usually a key determining factor. The decision rule hinges upon whether or not there is sufficient savings from peering to justify spending capital on a port on a router and/or a portion of the interconnection costs or augmenting existing capacity into an exchange point. A Bilateral Peering Agreement (BLPA) is the legal form that details each parties understanding of acceptable behavior, and defines the arms length interactions that each would agreed to. Another motivation for peering to factor in includes lower latency and/or more regional distribution of traffic than existing connections allow. This process is diagrammed below. After this initial discussion, either party may decide to walk away from peering discussions until certain criteria are met . If both parties agree that their requirements are met sufficiently to discuss methodology, and they both benefit from the peering relationship, they move onto Phase 3: Implementation Discussions. III. Phase 3: Implementation Discussions: Peering Methodology ---------------------------------------------------------------------------- ---- Since peering is of mutual benefit, both parties next explore the interconnection method(s) that most effectively exchange traffic to and from each other's customers. The primary goal is to establish mutual point(s) of interconnection, and secondarily detail optimal traffic exchange behavior (MEDs). For interconnections, ISPs face three options: Direct Circuit Interconnection, Exchange-Based Interconnection or some global combination thereof. The "Interconnection Strategies for ISPs" white paper quantifies the economics and technical tradeoffs between the first two options. To summarize this report, the preferred methodology depends on the number of peers participating in the region and bandwidth required for its regional interconnections. ISPs that expect to interconnect at high or rapidly increasing bandwidth within the region, or expect interconnections with more than five parties in the region often prefer the exchange-based solution. Those that do not anticipate a large number of regional interconnects prefer direct-circuits and typically decide to split the costs of interconnection with the peer by region. On occasion the costs are covered in whole by one peer. For direct-circuit interconnects, key issues center upon interconnection location(s) and who pays for and manages the interconnection. This becomes a material cost issue as traffic grows and circuits increase in size and cost. In either case, ISPs generally have the following goals for establishing peering: 1. fulfill obligations of larger business agreement, 2. get peering set up as soon as possible, 3. minimize the cost of the interconnection and their transit costs, 4. maximize the benefits of a systematic approach to peering, and 5. execute the regional operations plan as strategy dictates (may be architecture/network development group goal). Exchange Environment Selection Criteria ======================================= that these issues are listed in no particular order. Telecommunications Access Issues -------------------------------- How fast can circuits be bought into the interconnection environment? How many carriers compete for my business for circuits back to my local Point of Presence (POP)? For facilities-based ISPs, what is the cost of trenching into the exchange (how far away and what obstacles present themselves)? Are there nearby fiber providers that lease strands? These answers will answer the most important question to ISPs: How fast can my peer and I get connectivity into the exchange? Multiple carriers lead to speed and cost efficiencies. Some ISPs have volume deals with certain carriers or otherwise preferred carriers so prefer exchanges where these carriers can quickly provision circuits. These answers strongly impact the desirability of the exchange environment. Deployment Issues ----------------- How do I get my equipment into the exchange (assuming it supports collocation)? Do I ship equipment in or do I have to bring it with me as I fly in? Will someone act as remote hands and eyes to get the equipment into the racks or do I do the installation myself? What are the costs associated with deployment (travel, staff time, etc.)? Does the exchange have sufficient space, power. The answers to these questions impact the deployment schedule for the ISP engineers and the costs of the interconnection method. ISP Current Presences --------------------- The most inexpensive (and expedient) peering arrangements are the ones made between ISPs that are already located in the same exchange with sufficient capacity to interconnect. Cross-connects or switching fabrics can easily establish peering within hours or at most days. ISPs will prefer to interact where one of both ISP already has a presence. Somewhat related, is the exchange in the right geographic location? Operations Issues ----------------- Once the ISP equipment and/or circuit into the exchange is installed, these issues focus on the ongoing operations activities allowed within the exchange. Does the exchange allow private network interconnections? Are there requirements to connect to a central switch? How secure is the facility? Is there sufficient power, HVAC, capacity at the switch, space for additional racks, real time staff support? Is it easy to upgrade my presence over time? Upgrading in this context means the ability to increase the speed of circuits into the exchange, the ability to purchase dark fiber, the ability to increase the number of racks and cross connects in the exchange, the ease of increasing the speed of interconnection. ISPs will prefer bandwidth-rich, ISP-friendly exchanges over those with restrictions over future operations. Business Issues --------------- Perhaps the most far-reaching issue is strategic: do we want to support this exchange operator, and do their interests enhance or conflict with ours? According to Lauren Nowlin, Peering Coordinator at PGExpress, "Bandwidth, strategic partner alliances, and corporate ties often override the technical justification ." Will using this exchange support a competitor (contribute to their net income, their credibility, their positioning)? Does the exchange have requirements (require use of their carrier or ISP services) that limit the market for services within the exchange? Since it is difficult and disruptive to move equipment out of an exchange, ISPs will prefer a neutrally operated exchange environment that doesn't suffer from market distortions and limitations due to business conflicts of interest. Cost Issues ----------- What is the cost of using this exchange? What are the rack fees, cross connect fees, port fees, installation fees? What are the future operating fees going to be? What are the motivations and parameters surrounding these fees? Cost issues shadow most of the other issues listed in this paper. All else being equal, ISPs will seek to minimize the costs, particularly upfront costs, associated with the interconnection for peering. Credibility Issue ----------------- Does the exchange exist today and will it exist tomorrow? Who will be there in the future? The benefit to the ISP grows (as pictured below) as the number of interconnection possibilities grow, as the number of transit customers grows, and as the bandwidth between the exchange participants grow. During the early stages of the exchange, ISPs are asked to make a leap of faith when committing, and therefore prefer an exchange with strong backing and the credibility to ensure the ISP obtains value. Does the exchange operator have the backing and credibility to attract the more valuable peering candidates? Who is managing the exchange and what technology is in use signals the credibility and survivability of the exchange. ISPs will prefer an exchange with credibility - one that is financially and technically well backed and likely to attract the most desirably peering candidates. Exchange Population ------------------- Are there other side benefits to using this exchange? Are there other ISPs there that are peering candidates? Are there transit sales possible at the exchange? In the context of the credibility issue discussed above, who will likely be at the exchange in the future, and when will the cost of participation equal the value of the interconnection (also known as the Critical Mass Point)? ISPs will prefer established and well-populated exchanges, particularly those with potential customers that can generate revenue for the ISP. Existing Exchange vs. New Exchange? ----------------------------------- There are many regional "meet me" exchange points in each region of the U.S. There are also emerging exchanges that may be considered. However, given the pace of ISP expansion, it is unlikely that emerging exchange offerings are differentiated or compelling enough to be preferred over existing exchanges. Chronic traffic congestion can influence the decision to plan to peer in an existing exchange or wait until a better exchange opens. Customers with heavy flows of regional traffic can also influence the decision. Long term benefits (scalability) may lead to preferring a next generation exchange. However, all else considered equal, ISPs generally prefer an existing exchange to an emerging one. This third discussion phase can be summed up by the following graph. IV. Summary =========== The paper provides a rough description of the decision process ISPs follow to obtain peering relationships and reduce their transit costs. It focuses on the elements of the decision that lead to the selection of a specific exchange environment for peering. The results of the interviews with ISP Peering Coordinators can be summarized with the following observations: 1) To reduce transit costs, peering goals for ISPs include a) to get peering set up as soon as possible, b) to minimize the cost of peering, c) to maximize the benefits of whatever architecture is selected, and d) to leverage the architecture to accomplish their strategic regional goals. 2) The selection of an exchange is made relatively late in the peering process, and the selection of a soon-to-be-available exchange is a very difficult sell. This is particularly true when there are existing exchanges in the region that meet the ISPs requirements. 3) Most critical to the ISP are issues surrounding business opportunities presented at an exchange, telecommunications access, deployment, population, operations, cost, and credibility. ISPs will prefer and exchange environment that best suit these needs. 4) One major challenge facing Peering Coordinators is the identification of potential peers and initiating discussions. Acknowledgements ---------------- This paper was the result of interviews with key Internet peering coordinators, interactions with ISP support staff, and side conversations at various meetings and forums. Special thanks to a few folks who contributed their time and ideas to help create this early draft report: Ren Nowlin (PGExpress/Onyx), Joe Payne (IXC), Dave Diaz (Netrail), Jake Khuon and Alan Hannan (Frontier GlobalCenter), Dan Gisi and Jeff Rizzo (Equinix), Patricia Taylor-Dolan (Level 3 Communications), Sean Donelan (AT&T Labs). As interviews continue, this list will expand to identify other key contributors to this paper. References ---------- [1] "Maturation in a Free Market: The Changing Dynamics of Peering in the ISP Market" by Jennifer DePalma [2] "NAPs, Exchange Points, and Interconnection of Internet Service Providers: Recent Trends, Part I: 1997 Survey of Worldwide NAPs and Exchange Points", Mark Knopper Appendix A: Peering Decision Tree Appendix B: Detail Peering Decision Tree Some interesting side notes from the BOF ------------------------------------------- +At least two countries including Israel have regulators that forced ISPs to peer. +Formal legal documents are not required for peering, although commonplace +Keith Mitchel (LINX) has a prototype Bi_Lateral Peering Agreement (BLPA) on-line: http://www.linx.net/joininfo/peering-template/agreement-v4.html +The group wanted the Peering Contact Database NOT to be public, but rather limited to the folks participating. +Non-U.S. ISPs seem to have greater difficulty establishing peering here in North America. Language, culture issues are mixed with technical issues. ---------------------------------------------------------------- William B. Norton <wbn@equinix.com> +1 650.298.0400 x2225 Equinix Co-Founder, Director of Business Development