PSINet and C&W peering
C&W did indeed shutdown their peering connections to PSINet this weekend. While there are many potential explanations for their actions, I have no visibility into their decision process. I am disappointed with their decision to disconnect. PSINet continues to seek a resolution with C&W to restore normal connectivity in order to avoid further negative impact to both companies and the Internet. Their decision is hard to understand based on the following: - C&W and PSINet upgraded circuits used for peering between the two networks earlier this year. C&W's recent action seems inconsistent with the strategy that led to these upgrades. - PSINet's recent addition of direct private peering with several of C&W's transit customers relieved the peering connections between the networks of a couple hundred Mbps of traffic (improving connectivity overall and, undoubtedly, lowering costs for those transit customers). This is significant only because C&W claims PSINet no longer has sufficient traffic to justify the connections according to their published standards. In fact, PSINet's overall traffic continues to grow. - Most of the PSINet traffic previously destined for sites behind C&W has alternative paths through other providers. While this sounds like a generally good thing, especially given the actions C&W has taken, it does make it difficult for those that require certain traffic levels to be maintained consistently for peering. Specifically, C&W's customers (or C&W itself) could alter "natural" traffic flow to favor (or not) various connections to meet their published standards (or not). PSINet demonstrated to C&W that if naturally less favorable announcements were preferred, PSINet could make an almost arbitrarily large (or small) amount of traffic flow between the peers. Even so, in C&W's opinion, PSINet will not be able to comply with their peering policy's traffic standards. It is gratifying to note that even without C&W peering, substantially all of the traffic previously flowing between PSINet and C&W continues to be delivered. - At this time PSINet has not disabled the C&W peering interfaces nor decommissioned any facilities. If C&W chooses to, they can re-enable interfaces on their side and bring back the connectivity lost between their non-transit customers and PSINet. PSINet remains open to discuss with them a new bilateral peering agreement if they so choose. PSINet remains committed to servicing its customers and the Internet with the best possible infrastructure and policies. PSINet still maintains hundreds of peering connections with other ISPs throughout the world. While posting about matters between PSINet and its peering partners is not typical, the circumstances and questions arising from C&W's decision required some clarification. Hopefully this additional clarification helps everyone understand the current situation. -Mitch Levinn PSINet
On Tue, 5 Jun 2001, Mitchell Levinn wrote:
PSINet remains committed to servicing its customers and the Internet with the best possible infrastructure and policies. PSINet still maintains hundreds of peering connections with other ISPs throughout the world. While posting about matters between PSINet and its peering partners is not typical, the circumstances and questions arising from C&W's decision required some clarification. Hopefully this additional clarification helps everyone understand the current situation.
and you did what to Exodus, Abovenet, and maybe others? PSI shut off other peering sessions. I know why and maybe you can share this information with the rest of nanog. If PSI really cared about their customers and the Internet as you so write, you would have direct connections to companys like Exodus and Abovenet. Also, is there a isp-peering list we can move this to? Thanks Christian
Christian Nielsen wrote:
PSI shut off other peering sessions. I know why and maybe you can share this information with the rest of nanog.
If PSI really cared about their customers and the Internet as you so write, you would have direct connections to companys like Exodus and Abovenet.
Also, is there a isp-peering list we can move this to?
What, you think this isn't on-topic for NANOG? Seems operational to me. Besides, I want to hear why PSI shut off the other peering sessions. -- Tired of Earthlink? Get JustTheNet! Nationwide Dialup, ISDN, DSL, ATM, Frame Relay, T-1, T-3, and more. EARTHLINK AMNESTY PROGRAM: Buy a year, get two months free More info coming soon to http://JustThe.net, or e-mail me! B!ff: K3wl, w3'v3 r00t3D da N@vy... 0h CrAp, INC0M!Ng $%^NO CARRIER
oddly enough this looks exactly like the statments made by a few providers when PSI decided to de-peer. what goes around comes around. On Tue, 5 Jun 2001, Mitchell Levinn wrote:
C&W did indeed shutdown their peering connections to PSINet this weekend. While there are many potential explanations for their actions, I have no visibility into their decision process. I am disappointed with their decision to disconnect. PSINet continues to seek a resolution with C&W to restore normal connectivity in order to avoid further negative impact to both companies and the Internet. Their decision is hard to understand based on the following:
- C&W and PSINet upgraded circuits used for peering between the two networks earlier this year. C&W's recent action seems inconsistent with the strategy that led to these upgrades. - PSINet's recent addition of direct private peering with several of C&W's transit customers relieved the peering connections between the networks of a couple hundred Mbps of traffic (improving connectivity overall and, undoubtedly, lowering costs for those transit customers). This is significant only because C&W claims PSINet no longer has sufficient traffic to justify the connections according to their published standards. In fact, PSINet's overall traffic continues to grow. - Most of the PSINet traffic previously destined for sites behind C&W has alternative paths through other providers. While this sounds like a generally good thing, especially given the actions C&W has taken, it does make it difficult for those that require certain traffic levels to be maintained consistently for peering. Specifically, C&W's customers (or C&W itself) could alter "natural" traffic flow to favor (or not) various connections to meet their published standards (or not). PSINet demonstrated to C&W that if naturally less favorable announcements were preferred, PSINet could make an almost arbitrarily large (or small) amount of traffic flow between the peers. Even so, in C&W's opinion, PSINet will not be able to comply with their peering policy's traffic standards. It is gratifying to note that even without C&W peering, substantially all of the traffic previously flowing between PSINet and C&W continues to be delivered. - At this time PSINet has not disabled the C&W peering interfaces nor decommissioned any facilities. If C&W chooses to, they can re-enable interfaces on their side and bring back the connectivity lost between their non-transit customers and PSINet. PSINet remains open to discuss with them a new bilateral peering agreement if they so choose.
PSINet remains committed to servicing its customers and the Internet with the best possible infrastructure and policies. PSINet still maintains hundreds of peering connections with other ISPs throughout the world. While posting about matters between PSINet and its peering partners is not typical, the circumstances and questions arising from C&W's decision required some clarification. Hopefully this additional clarification helps everyone understand the current situation.
-Mitch Levinn PSINet
/rf
[Disclaimer: I work for MFN/Above.net, that that for what you will.] On Tue, Jun 05, 2001 at 09:35:20AM -0400, Mitchell Levinn wrote:
C&W did indeed shutdown their peering connections to PSINet this weekend. While there are many potential explanations for their actions, I have no visibility into their decision process. I am disappointed with their decision to disconnect. PSINet continues to seek a resolution with C&W to restore normal connectivity in order to avoid further negative impact to both companies and the Internet. Their decision is hard to understand based on the following:
I find this quite ironic, as PSINet decided to terminate peering with AboveNet in November of 2000. AboveNet was disappointed with PSI's decision, and had worked hard to avoid the disconnection. To this day, AboveNet is still unsure why PSI took that action, with the damage that resulted. The really interesting part, is the claims made below, which I will address one by one.
- C&W and PSINet upgraded circuits used for peering between the two networks earlier this year. C&W's recent action seems inconsistent with the strategy that led to these upgrades.
The same was true for PSINet and AboveNet. AboveNet ordered multiple OC-3 peering circuits to PSI. The first was turned up under our agreement. It was not until we tried to turn up the second that PSI decided to reevaluate AboveNet as a peer. We were very much caught by surprise when PSINet abruptly reverse course.
- PSINet's recent addition of direct private peering with several of C&W's transit customers relieved the peering connections between the networks of a couple hundred Mbps of traffic (improving connectivity overall and, undoubtedly, lowering costs for those transit customers). This is significant only because C&W claims PSINet no longer has sufficient traffic to justify the connections according to their published standards. In fact, PSINet's overall traffic continues to grow.
This one one of the reasons mentioned by PSINet in the disconnection proceedings for AboveNet. Clearly AboveNet could not be an important peer due to the lack of traffic. AboveNet has maintained since day one of its existance a very open peering policy, and the low traffic to PSI was very much in part due to the robust peering that AboveNet had with many of PSI's downstreams. In all fairness, the issue of traffic ratio also came up with PSINet (so they don't come back and claim they never cared about traffic levels, just ratio). The ratio is equally funny though, due to the next point.
- Most of the PSINet traffic previously destined for sites behind C&W has alternative paths through other providers. While this sounds like a generally good thing, especially given the actions C&W has taken, it does make it difficult for those that require certain traffic levels to be maintained consistently for peering. Specifically, C&W's customers (or C&W itself) could alter "natural" traffic flow to favor (or not) various connections to meet their published standards (or not). PSINet demonstrated to C&W that if naturally less favorable announcements were preferred, PSINet could make an almost arbitrarily large (or small) amount of traffic flow between the peers. Even so, in C&W's opinion, PSINet will not be able to comply with their peering policy's traffic standards. It is gratifying to note that even without C&W peering, substantially all of the traffic previously flowing between PSINet and C&W continues to be delivered.
This is true for any provider with a reasonable amount of traffic. With the number of multi-homed customers behind {Sprint, C&W, UUnet, AT&T, Genuity, Level 3, Qwest} (and possibly others) it is easy to move hundreds of megabits of traffic between those providers while still honoring peering agreements. It is something PSINet ignored, as well as C&W. The fact is, most large ISP's can set the ration on any given peering interface to be whatever they want, of course at the expense of altering a ratio somewhere else. It's like a baloon, the total volume (ratio) is fixed, but by squeezing or pulling it you can make it into limitless shapes.
- At this time PSINet has not disabled the C&W peering interfaces nor decommissioned any facilities. If C&W chooses to, they can re-enable interfaces on their side and bring back the connectivity lost between their non-transit customers and PSINet. PSINet remains open to discuss with them a new bilateral peering agreement if they so choose.
How lovely. :-) Let me try from our side "At this time AboveNet has not disabled PSINet's peering interfaces nor decommissioned any facilities, if PSI chooses to, they can re-enable interfaces on their side and bring back the connectivity lost..." AboveNet is of course always open to discussing peering with PSINet, we hold no grudges.
PSINet remains committed to servicing its customers and the Internet with the best possible infrastructure and policies. PSINet still maintains hundreds of peering connections with other ISPs throughout the world. While posting about matters between PSINet and its peering partners is not typical, the circumstances and questions arising from C&W's decision required some clarification. Hopefully this additional clarification helps everyone understand the current situation.
Now this is clearly not true. As much as we had to swallow or pride to do it, AboveNet took steps to make sure that even though PSINet terminated peering, AboveNet customers experienced uninterrupted (if a bit rocky for a short time) connectivity to PSINet. Our commitment to our customers included making sure they could connect to PSINet. PSINet clearly doesn't care if their customers can reach C&W. I feel no sympathy for PSINet, as I am personally a firm believer in "connectivity is its own reward", and I think the fairly open AboveNet peering policy reflects that as well. No one wins when connectivity is broken like this, I have to believe both C&W and PSI customers are feeling some real pain right now. I hope this lesson makes people think twice about terminating such peering relationships in the future. That said, the irony here is amazing. PSINet has been put in the exact same position by C&W that they decided to put AboveNet (and possibly others, I'll let them comment on their situation) into several months ago, and isn't even big enough to work around the problem. "What goes around comes around" is very often true, the provider that was big yesterday can be small tomorrow, and could possibly even come back to be big again. If PSINet would like to start down the road to recovery, I suggest they start with mending some of the relationships they chose to sever in the past. If a PSINet representative would like to talk about the AboveNet situation in particular, please mail me privately and I'll make sure you get in touch with the right people. Also, I would like to suggest that if you are a Cable and Wireless customer, and disagree with their current action of terminating peering relationships (we have nasa.gov and psi.net confirmed on Nanog, and rumors of many others in the works) I suggest you let them know how you feel, first with your words, and if that doesn't work with your dollars, by taking them elsewhere. I know a lot of people think ISP's don't listen to their customers, and sometimes it may seem that they don't, but by and large they do try to keep their customer base happy. -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
On Tue, 5 Jun 2001, Mitchell Levinn wrote:
- PSINet's recent addition of direct private peering with several of C&W's transit customers relieved the peering connections between the networks of a couple hundred Mbps of traffic (improving connectivity overall and, undoubtedly, lowering costs for those transit customers). This is significant only because C&W claims PSINet no longer has sufficient traffic to justify the connections according to their published standards. In fact, PSINet's overall traffic continues to grow.
Now, while I agree that this would provide a better path between 174 and whoever these multi-hundred Mb/s peers are, I can also understand why C&W might have gotten a bit miffed. That's a couple of hundred megabytes of flow that they can no longer bill to those customers. I'd ALWAYS rather have my customers use our network for transit than have them peer directly with my peers and bypass the toll booth. That said, if this was their reasoning for terminating the peering agreement, I don't agree with them at all. I'd be upset about it if I were them but, I wouldn't be stupid about the situation. --- John Fraizer EnterZone, Inc
That's a couple of hundred megabytes of flow that they can no longer bill to those customers. I'd ALWAYS rather have my customers use our network for transit than have them peer directly with my peers and bypass the toll booth.
What an ideal carrot to ensure that we operators plan our networks well, if we do - we keep our customers. If we don't our customers find alternative means to provide what we are supposed too, and one way or another thats going to cost you. Personally speaking, I'm a little worred that this isn't the last C+W peering issue that we will hear about in the near future, unless of the new flows of customers abandoning them changes a few minds at corporate HQ, I've certainly heard from a large European C+W customer who is now seriously thinking about finding alternatives because of this action. Also, I couldn't recommend buying connectivity from any organisation who randomly disconnects people without consultation/communication to their customer base, and operates draconian and stupid peering policies. Regards, Neil.
If exchange of IP traffic is ruled to be commerce then peering policies would be regulated under the Sherman antitrust act. It shall be unlawful for any person engaged in commerce, in the course of such commerce, to be a party to, or assist in, any transaction of sale, or contract to sell, which discriminates to his knowledge against competitors of the purchaser, in that, any discount, rebate, allowance, or advertising service charge is granted to the purchaser over and above any discount, rebate, allowance, or advertising service charge available at the time of such transaction to said competitors in respect of a sale of goods of like grade, quality, and quantity; to sell, or contract to sell, goods in any part of the United States at prices lower than those exacted by said person elsewhere in the United States for the purpose of destroying competition, or eliminating a competitor in such part of the United States; or, to sell, or contract to sell, goods at unreasonably low prices for the purpose of destroying competition or eliminating a competitor. ... if IP traffic exchange is interstate commerce ... At 16:47 +0100 05-06-2001, Neil J. McRae wrote:
That's a couple of hundred megabytes of flow that they can no longer bill to those customers. I'd ALWAYS rather have my customers use our network for transit than have them peer directly with my peers and bypass the toll booth.
What an ideal carrot to ensure that we operators plan our networks well, if we do - we keep our customers. If we don't our customers find alternative means to provide what we are supposed too, and one way or another thats going to cost you.
Personally speaking, I'm a little worred that this isn't the last C+W peering issue that we will hear about in the near future, unless of the new flows of customers abandoning them changes a few minds at corporate HQ, I've certainly heard from a large European C+W customer who is now seriously thinking about finding alternatives because of this action.
Also, I couldn't recommend buying connectivity from any organisation who randomly disconnects people without consultation/communication to their customer base, and operates draconian and stupid peering policies.
Regards, Neil.
-- Joseph T. Klein +1 414 915 7489 Senior Network Engineer jtk@titania.net Adelphia Business Solutions joseph.klein@adelphiacom.com "... the true value of the Internet is its connectedness ..." -- John W. Stewart III
At 16:24 +0000 05-06-2001, Joseph T. Klein wrote:
If exchange of IP traffic is ruled to be commerce then peering policies would be regulated under the Sherman antitrust act.
blah .. blah .. I was trying to make a point. Consistent and publicly announced peering policies are in the best interest of any entity who desires to keep the regulators out. -- Joseph T. Klein +1 414 915 7489 Senior Network Engineer jtk@titania.net Adelphia Business Solutions joseph.klein@adelphiacom.com "... the true value of the Internet is its connectedness ..." -- John W. Stewart III
[deleted] Personally speaking, I'm a little worred that this isn't the last C+W peering issue that we will hear about in the near future, unless of the new flows of customers abandoning them changes a few minds at corporate HQ, I've certainly heard from a large European C+W customer who is now seriously thinking about finding alternatives because of this action. Also, I couldn't recommend buying connectivity from any organisation who randomly disconnects people without consultation/communication to their customer base, and operates draconian and stupid peering policies. Regards, Neil. ----- I think this is a great point. Wouldn't BOTH networks be to blame from a Customer point of view? I know if I were a customer of either network, I would move to one not involved in either place. Frankly I don't think customers care about where the mud is slung. *IF* a site they want is effected, they will move when it is economically feasible to do so. I would say that being unable to reach a site you expect to reach is a significant drop in service, possibly below acceptable contractual levels. Then I'd leave it in the hands of the attorneys. There is no way [the company who terminated peering] is going to make money off of this venture, because invariably whoever the transit is bought from will be a peer of [the company who terminated peering]. And rather than encourage transit relationships, it encourages more and more direct relationships with large customers. Years ago peering was only between large networks -- the "technical complexity" was beyond mere "customers". Now large/regional customers try to design relationships with other large/regional customers. Eventually it may go to even smaller shops. Deepak Jain AiNET
Deepak Jain wrote:
Also, I couldn't recommend buying connectivity from any organisation who randomly disconnects people without consultation/communication to their customer base, and operates draconian and stupid peering policies.
I've seen telcos do that to ISPs before, though not on that level... and C&W is most definitely a telco. -- Tired of Earthlink? Get JustTheNet! Nationwide Dialup, ISDN, DSL, ATM, Frame Relay, T-1, T-3, and more. EARTHLINK AMNESTY PROGRAM: Buy a year, get two months free More info coming soon to http://JustThe.net, or e-mail me! B!ff: K3wl, w3'v3 r00t3D da N@vy... 0h CrAp, INC0M!Ng $%^NO CARRIER
On Tue, 5 Jun 2001, John Fraizer wrote:
and whoever these multi-hundred Mb/s peers are, I can also understand why C&W might have gotten a bit miffed. That's a couple of hundred megabytes of flow that they can no longer bill to those customers. I'd ALWAYS rather have my customers use our network for transit than have them peer directly with my peers and bypass the toll booth. That said, if this was their reasoning for terminating the peering agreement, I don't agree with them at all. I'd be upset about it if I were them but, I wouldn't be stupid about the situation.
Here are some points to think about: 1) Some peers dont upgrade peering links quick enough. If this is the case, companies go after the largest src/dest for IP. 2) Some IP providers charge for IP If this is the case, companies go after the largest src/dest to reduce the amount they might have to pay. 3) Some IP providers wont peer. If this is the case, companies go after the largest src/dest first 4) and lastly, some companies have very open peering policys And if this is the case... those companies get asked to peer all the time. Personally, it would make much easier to peer with fewer larger players than many little players. but heh, this is the internet right??? BTW, to solve these types of problems, Smaller companies need to ban together and start peering with each other in the local metro area. Christian
Date: Tue, 5 Jun 2001 10:26:29 -0700 (PDT) From: Christian Nielsen <cnielsen@nielsen.net>
[ snip ]
BTW, to solve these types of problems, Smaller companies need to band together and start peering with each other in the local metro area.
Until the smaller companies get bought out by OneM^H^H^H^H a national provider who has no interest in peering. Quite frankly, it seems that most smaller companies have no clue what the heck peering is or why it's good. I've approached a few, and none seem to get it. One (cable company, mind you) wanted to *charge more for peering than for transit*. They thought that we would be getting some sort of "priority access" to their network, and wanted to charge a hefty price... and this was talking to a net admin, not a droid. Jeez. Eddy --------------------------------------------------------------------------- Brotsman & Dreger, Inc. EverQuick Internet Division Phone: (316) 794-8922 --------------------------------------------------------------------------- Date: Mon, 21 May 2001 11:23:58 +0000 (GMT) From: A Trap <blacklist@brics.com> To: blacklist@brics.com Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to <blacklist@brics.com>, or you are likely to be blocked.
On Tue, Jun 05, 2001 at 05:39:41PM +0000, E.B. Dreger wrote:
BTW, to solve these types of problems, Smaller companies need to band together and start peering with each other in the local metro area.
Until the smaller companies get bought out by OneM^H^H^H^H a national provider who has no interest in peering.
Quite frankly, it seems that most smaller companies have no clue what the heck peering is or why it's good. I've approached a few, and none seem to get it.
i have been involved with a couple small exchanges, and the way it worked for us was "build it and they will come". initially TorIX was only a few ISP's who happened to be in the same building. then a few other ISP's pulled in circuits and peered. then i bumped into an @home tech, and they jacked in. it took 4+ years to get the momentum going, but now we have a pretty good peering thing going on. so, if you can at least get one other ISP to peer, pick a good place, and drive a stake in the ground. others may migrate towards it. (some of the selling points are sharing usenet news feeds and/or http caches) -- [ Jim Mercer jim@reptiles.org +1 416 410-5633 ] [ Now with more and longer words for your reading enjoyment. ]
On Tue, 5 Jun 2001, E.B. Dreger wrote:
From: Christian Nielsen <cnielsen@nielsen.net> BTW, to solve these types of problems, Smaller companies need to band together and start peering with each other in the local metro area.
Hear hear. The local exchanges that are not corrupted by their sponsors' "upsell" agenda have somehow sustained themselves, because they solve a problem for their participants. Local exchanges (regional exchange points, community internet exchanges) are becoming more critical parts of the infrastructure. With the convergence of the metro network (look at the growth rate of investment in the metro network compared to the national network--that's not just for national-network connectivity), voice moving onto data networks (80% of voice traffic stays within the local exchange) and the migration of new applications to the Internet making the Internet Matter, traffic within the metro area will become a significant, if not dominant, segment of the overall traffic.
Until the smaller companies get bought out by a national provider who has no interest in peering.
Quite frankly, it seems that most smaller companies have no clue what the heck peering is or why it's good. I've approached a few, and none seem to get it.
I've talked to a lot of companies about a lot of different kinds of technology. I found that if you find a problem they have that they really care about, or that you can educate them to care about, typically they will appreciate the value of the solution, and be motivated to get it. Talking to a small company about peering (which they would be unlikely to know about, since I was probably the first to approach them) resulted in blank stares and a quick dismissal. Talking to a company about saving money on their transit, improving performance and reliability, diversifying routing paths, adding value to their customers, differentiating themselves from competition, and doing so in a manner catered to their skill-set, resulted in most companies at least acknowledging that the solution (peering) was right for them, and many of them moving forward. Fitting it into their business and financial models was a different hurdle, but much more easily overcome when they were helping. As you might suspect, there are different methods to sell to the networking guy vs. the business manager. Since the networking guy rarely has a company check-book, I found the most effective approach was to convince both of them, or at a minimum the business guy.
One (cable company, mind you) wanted to *charge more for peering than for transit*. They thought that we would be getting some sort of "priority access" to their network, and wanted to charge a hefty price... and this was talking to a net admin, not a droid. Jeez.
This doesn't seem too far off from the state of peering in general today. Isn't that what ratios and traffic levels and geography policies are all about, enforcing that peers are somehow "equal" and deserve to share traffic freely, and the rest of us can pay for "priority access"? Pete. pete@commix.org [ Support your neighborhood and community network projects. ] [ Create a community Internet exchange. Ask me how. ]
participants (12)
-
Christian Nielsen -
Deepak Jain -
E.B. Dreger -
Jim Mercer -
John Fraizer -
Joseph T. Klein -
Leo Bicknell -
Mitchell Levinn -
neil＠DOMINO.ORG -
Pete Kruckenberg -
rich＠nullroute.net -
Steve Sobol