Sprint v. Cogent, some clarity & facts
Having skimmed the Sprint / Cogent threads, I saw multiple errors and lots of really bad guesses. Instead of replying individually, I thought I would sum up a few facts so everyone was on the same page. This way when we run off into another 100 post thread, we can at least -start- from reality (although I would bet serious cash on long odds we will diverge from it soon enough). 1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?) Whether one or both _should_ have transit is not a fact, and therefore outside the scope of this e- mail, but that neither have transit today is a fact. (And please don't tell me how Network X has 100 Mbps of transit in Sri Lanka because they are too lazy to lease undersea cable. If you don't understand what I am saying here, stop reading now.) 2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1. 3. Standard transit contracts do not guarantee full connectivity If you are a Cogent customer, it is very unlikely your contract will allow you SLA or other credits for not being able to reach Sprint unless you negotiated something special. I doubt Sprint's standard contract is much different. Transit contract SLAs end at AS boundaries. This is because Network A has no control over Network B and therefore will not give credit if Network B fails. Of course, you can still sue, threaten to terminate, etc., but the letter of the contract almost certainly says nothing about packets going beyond your transit provider's ASN. 4. There is a reason behind ratios which has nothing to do with telco "sender-pays" Hot potato routing + very poor ratios puts much more of the cost on the receiving network. This is a valid, logical, and costly concern for receiving networks. The concern can be alleviated by cold-potato routing through accepting MEDs, anycast, CDN, and other technologies; to which the receiving network may say they cannot send proper MEDs, etc. Whether the problem can or should be worked through is not a fact, though. That this issue has nothing to do with telco "sender- pays" mentality is. (Of course, the telcos might still have that mentality, but that doesn't change the facts.) 5. Cogent has been disconnected several times Cogent has been de-peered (e.g. Teleglobe, L3, Sprint) and/or performed de-peering themselves (e.g. Telia) multiple times. Cogent has been disconnected from another network more times & for longer (in each instance?) than every other transit free network combined for the last decade. (In fact, if memory serves, for the history of the Internet - but I'm not quite sure enough to guarantee it as fact.) Cogent has also de-peered many non-transit-free networks, at least sometimes without even notifying the peer prior to disconnection. Whether that makes Cogent the bully-er or the bully-ee is not fact, so I will not comment on that here. There are probably other errors I missed while skimming the longer posts. But this should get us started on a good, clean, factual footing for future flights of fancy. -- TTFN, patrick
On Mon, Nov 3, 2008 at 1:26 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?)
How do you explain Cogent's arrangement with NTT (AS 2914)? If it's not transit, what is it? Does Akamai have peering arrangements with Cogent directly? Paul
On Mon, Nov 3, 2008 at 3:35 AM, Paul Wall <pauldotwall@gmail.com> wrote:
On Mon, Nov 3, 2008 at 1:26 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?)
How do you explain Cogent's arrangement with NTT (AS 2914)? If it's not transit, what is it?
So you are asking to have what was Cogent's arrangement with NTT? That's like asking what was AOL's relationship with Level(3) when they had their peering spat with Sprint. Just because a relationship existed in the past does not mean the same relationship exists today. Accept the fact that both providers are transit free and move forward.
Does Akamai have peering arrangements with Cogent directly?
Akamai are self declared peering sluts. So, yes, they have direct peering arrangements with Cogent. charles
Does Akamai have peering arrangements with Cogent directly?
Akamai are self declared peering sluts. So, yes, they have direct peering arrangements with Cogent.
Hrm, so after I posted this, I looked a bit deeper into it and found: 3 vl3493.mpd03.jfk02.atlas.cogentco.com (154.54.5.226) 0.665 ms 0.670 ms 0.667 ms 4 te9-4.mpd01.jfk05.atlas.cogentco.com (154.54.26.62) 0.697 ms 0.686 ms 0.653 ms 5 gblx.jfk05.atlas.cogentco.com (154.54.11.138) 0.721 ms 0.717 ms 0.802 ms 6 te3-2-10g.ar4.nyc1.gblx.net (67.16.131.105) 0.986 ms 0.934 ms 1.181 ms 7 bandwidth-consulting.tengigabitethernet8-3.ar4.nyc1.gblx.net (64.210.29.14) 0.696 ms 2.665 ms 0.700 ms 8 te-8-3.bbr1.ash1.bandcon.com (216.151.179.225) 7.187 ms 7.177 ms 7.073 ms 9 209.234.254.198 (209.234.254.198) 7.322 ms 7.141 ms 7.217 ms 10 84.53.144.71 (84.53.144.71) 7.053 ms 7.071 ms 7.016 ms So, for at least for traceroute www.akamai.com, Coget is using their peer (Global Crossing) to reach Akamai transit provider. Guess I got caught up in the past myself, thinking "Cogent would never depeer a slut like Akamai". charles
On Nov 3, 2008, at 2:35 AM, Paul Wall wrote:
On Mon, Nov 3, 2008 at 1:26 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?)
How do you explain Cogent's arrangement with NTT (AS 2914)? If it's not transit, what is it?
I do not know, and neither do you. But I do know it is not "transit", at least not to Sprint. It is trivial to prove to yourself if Cogent has transit. Find me any AS path in the global table showing "_TF1_TF2_174_", there "TF1" and "TF2" are the ASNs of two of the other 13 transit free networks. (Modulo a few leaked prefixes, which always seem to crop up. For instance, if a network has 40K prefixes in its cone, showing O(10) paths is not proof.) This is a positive test - if you see it, you know they have transit, if you do not see it, you do not know they do not have transit. But combined with bifurcation when Sprint drops peering to Cogent, one can _know_ Cogent does not have full transit, or partial transit to Sprint. It is possible (although I personally believe unlikely) Cogent has partial transit to some other transit free network that you cannot see right now because their peering to that network is up and overriding the AS paths in the global table. But that doesn't matter to this discussion.
Does Akamai have peering arrangements with Cogent directly?
That is none of your business, not to mention completely irrelevant to the topic at hand as Akamai is neither a network nor transit free. -- TTFN, patrick
Sorry for my possible ignorance, but could you explain me what are you calling "transit-free"? I mean, the ISP I work for, has contract for several STM-4 links with Sprint (at least for 8 years now), and for sure they do have transit, at least for us (as we publish our customers ASs to them and they publish them to other carriers). ¿is that what you call transit? Thanks in advance, Nicolas. Patrick W. Gilmore wrote:
On Nov 3, 2008, at 2:35 AM, Paul Wall wrote:
On Mon, Nov 3, 2008 at 1:26 AM, Patrick W. Gilmore <patrick@ianai.net> wrote:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?)
How do you explain Cogent's arrangement with NTT (AS 2914)? If it's not transit, what is it?
I do not know, and neither do you. But I do know it is not "transit", at least not to Sprint.
It is trivial to prove to yourself if Cogent has transit. Find me any AS path in the global table showing "_TF1_TF2_174_", there "TF1" and "TF2" are the ASNs of two of the other 13 transit free networks. (Modulo a few leaked prefixes, which always seem to crop up. For instance, if a network has 40K prefixes in its cone, showing O(10) paths is not proof.)
This is a positive test - if you see it, you know they have transit, if you do not see it, you do not know they do not have transit. But combined with bifurcation when Sprint drops peering to Cogent, one can _know_ Cogent does not have full transit, or partial transit to Sprint. It is possible (although I personally believe unlikely) Cogent has partial transit to some other transit free network that you cannot see right now because their peering to that network is up and overriding the AS paths in the global table. But that doesn't matter to this discussion.
Does Akamai have peering arrangements with Cogent directly?
That is none of your business, not to mention completely irrelevant to the topic at hand as Akamai is neither a network nor transit free.
On Mon, Nov 03, 2008 at 04:34:16PM -0200, Nicolas Antoniello wrote:
Sorry for my possible ignorance, but could you explain me what are you calling "transit-free"?
Transit-free means that you don't pay anyone else to reach some 3rd-party network. In other words, if I'm Sprint, I don't pay UUNET to get to X. Either X connects directly with me or X pays someone else to get to me. If I can make that claim for all values of X, then I am transit-free. Note that while I don't pay another network for access to its *peers* (that's transit) I might pay for access to its customers. This is typically called "paid peering" or "settlement-based peering", but sometimes it can just be plain transit that's modified with communities to look like peering. To add to the confusion, the latter case might be described differently by both parties; the seller probably says "X is a transit customer of mine", and the buyer says "I have peering with Y", and in this case, neither one is lying (mostly). If you didn't see the reference a month or so ago when Paul sent it, the following link might be interesting to you: http://arstechnica.com/guides/other/peering-and-transit.ars --Jeff
* Patrick W. Gilmore:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?) Whether one or both _should_ have transit is not a fact, and therefore outside the scope of this e- mail, but that neither have transit today is a fact. (And please don't tell me how Network X has 100 Mbps of transit in Sri Lanka because they are too lazy to lease undersea cable. If you don't understand what I am saying here, stop reading now.)
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
Out of curiosity, what would happen if one of the parties got transit from a business POV? Not just in this particular case, but in general. Doesn't this work because they are so large that any such arrangement would immediately threaten traffic ratios at the (transit-free) transit provider?
3. Standard transit contracts do not guarantee full connectivity
If this were true, why would end users (or, more generally, not significantly multi-homed networks) buy transit from such networks?
On Mon, 03 Nov 2008 10:26:59 +0100, Florian Weimer said:
* Patrick W. Gilmore:
3. Standard transit contracts do not guarantee full connectivity
If this were true, why would end users (or, more generally, not significantly multi-homed networks) buy transit from such networks?
Quite frankly, if any potential transit provider tried to make noises about being able to *guarantee* full connectivity, I'd show him the door. Consider the average length of an AS path. Now consider that your AS is at one end, your transit provider is the next hop - and there's often 5 or 6 or more AS hops past that. And that potential transit provider has absolutely *no* control over what some backhoe just did to connectivity 4 AS down the path... For example, look at the traceroute from my desktop to where your mail originated: traceroute to 212.9.189.177 (212.9.189.177), 30 hops max, 60 byte packets 1 isb-6509-1.vl103.cns.vt.edu (128.173.12.1) 0.394 ms 0.712 ms 0.791 ms 2 isb-6509-2.po51.cns.vt.edu (128.173.0.5) 0.597 ms 0.681 ms 0.756 ms 3 isb-7606-2.ge1-1.cns.vt.edu (192.70.187.218) 0.740 ms 0.709 ms 0.687 ms 4 192.70.187.10 (192.70.187.10) 7.590 ms 7.610 ms 7.647 ms 5 te2-1--580.tr01-asbnva01.transitrail.net (137.164.131.177) 89.583 ms 89.618 ms 89.797 ms 6 llnw-peer.asbnva01.transitrail.net (137.164.130.30) 11.956 ms 9.450 ms 9.473 ms 7 ve5.fr3.iad.llnw.net (69.28.171.213) 17.243 ms 9.689 ms 17.443 ms 8 * * * 9 FRA-3-eth0-403.de.lambdanet.net (81.209.156.9) 99.266 ms 99.180 ms 99.163 ms 10 FRA-1-eth000.de.lambdanet.net (217.71.96.69) 98.342 ms 98.436 ms 98.283 ms 11 STU-3-pos330.de.lambdanet.net (217.71.96.82) 111.748 ms 111.764 ms 107.438 ms 12 bond0.border2.LF.net (212.9.160.73) 104.380 ms 104.404 ms 104.262 ms 13 em1.core.LF.net (212.9.160.65) 104.622 ms 104.761 ms 104.504 ms 14 dsl-gw.ispeg.de (212.9.161.26) 106.013 ms 105.999 ms 105.973 ms 15 dsl.enyo.de (213.178.172.64) 135.094 ms 136.729 ms 136.007 ms Are you saying that you'd accept a contract where ispeg.de or LF.net are making claims they can guarantee connectivity to AS1312 no matter what transitrail is doing? (I admit being surprised - I was *expecting* the traceroute to go through Level3 or Sprint, actually. When did lambdanet land in DE? ;) (And the real kicker - if transitrail burps, is ispeg or LF able to find us via our Level3 or Sprint connections? Maybe, maybe not...)
Quite frankly, if any potential transit provider tried to make noises about being able to *guarantee* full connectivity, I'd show him the door.
Let's not make the perfect the enemy of the good. All that's required is that they promise to make a good faith effort to interconnect with anyone else who makes a similar good faith effort. Now we can all fight about whether Cogent and/or Sprint are making such a good faith effort. ;) DS
* Valdis Kletnieks:
On Mon, 03 Nov 2008 10:26:59 +0100, Florian Weimer said:
* Patrick W. Gilmore:
3. Standard transit contracts do not guarantee full connectivity
If this were true, why would end users (or, more generally, not significantly multi-homed networks) buy transit from such networks?
Quite frankly, if any potential transit provider tried to make noises about being able to *guarantee* full connectivity, I'd show him the door.
Obviously, nothing won't stop them from disconnecting customers which are not sufficiently multi-homed, which might adversely affect me, independently of the size of the disconnected network or the nature of the dispute. That being said, there's a difference between disconnecting a customer and making sure, through action or inaction, that their network is no longer reachable from yours. I'd need to litigate to be sure, but I think the latter actually violates contracts we have at work.
Consider the average length of an AS path.
Well, in this context, the affected AS paths are really, really short. 8-)
Now consider that your AS is at one end, your transit provider is the next hop - and there's often 5 or 6 or more AS hops past that. And that potential transit provider has absolutely *no* control over what some backhoe just did to connectivity 4 AS down the path...
There's a difference between random events such as backhoes and self-inflicting damage as the result of DSWs.
For example, look at the traceroute from my desktop to where your mail originated:
I expect LF.net to isolate me from the results of those wars, both by making the right decisions in advance, and to act to correct problems when they arise. This hasn't always been possible. For instance, during one of the European routing wars in the 90s, I couldn't reach ftp.funet.fi for a couple of days.
On Nov 3, 2008, at 4:26 AM, Florian Weimer wrote:
* Patrick W. Gilmore:
1. Neither Sprint nor Cogent have transit Both Sprint & Cogent are transit-free networks. (Notice how I carefully avoided saying "tier one"?) Whether one or both _should_ have transit is not a fact, and therefore outside the scope of this e- mail, but that neither have transit today is a fact. (And please don't tell me how Network X has 100 Mbps of transit in Sri Lanka because they are too lazy to lease undersea cable. If you don't understand what I am saying here, stop reading now.)
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
Out of curiosity, what would happen if one of the parties got transit from a business POV? Not just in this particular case, but in general.
From a business perspective, one of the two parties would then be paying a third party to reach the other. In fact, a year ago this is exactly what was happening - Cogent bought partial transit from Verio to reach Sprint and AOL. Neither believes this is in their best interest. I cannot tell you if that is true.
Doesn't this work because they are so large that any such arrangement would immediately threaten traffic ratios at the (transit-free) transit provider?
Obviously not since it was happening in the past. But you make a good point. Traffic from either of these networks is probably large enough to push at least one of the other transit-free networks over their peering ratios with someone else. But probably not all. It is probable some transit free networks gets more traffic from Sprint than they send, so selling transit to Cogent would not hurt them. Not so sure any transit-free network pushes more to Cogent than they receive, but I cannot prove it. And even if selling to Cogent would put them over their ratio requirements, perhaps they could negotiate a better settlement deal, so they get more from Cogent than they pay to Sprint. Despite the fact I believe Cogent is heavy outbound to all other transit free networks, there are solutions that would allow a network to sell Sprint transit to Cogent. Etc., etc. It is a business problem, it has multiple business solutions.
3. Standard transit contracts do not guarantee full connectivity
If this were true, why would end users (or, more generally, not significantly multi-homed networks) buy transit from such networks?
"If this were true"? "Why would end users [...] buy transit from such networks"? Please show me a transit contract - just one - that guarantees connectivity beyond the transit AS boundary. Put another way, since _every_ network does this, if you do not want to buy from 'such networks', you cannot buy transit. -- TTFN, patrick
Put another way, since _every_ network does this, if you do not want to buy from 'such networks', you cannot buy transit.
Let's put it another 'nother way. Would an end user get better connectivity by buying from a reseller of transit? In other words, buying transit from a network which also buys transit. Presumably up near the top of the chain (Tier 1 vicinity), that transit reseller has a lot of peering in place with other folks in the same neighborhood (Tier 1 vicinity). But as long as a network is a transit reseller (i.e. they buy transit), then they are less likely to suffer from partition events caused by fractious peering negotiations. --Michael Dillon frac*tious (frakshus) adj. 1. Inclined to make trouble; unruly. 2. Having a peevish nature; cranky. Also likely to cause your network having connectivity to only a fraction of the Internet.
-----Original Message----- From: michael.dillon@bt.com [mailto:michael.dillon@bt.com] Sent: Monday, November 03, 2008 8:55 AM
Let's put it another 'nother way. Would an end user get better connectivity by buying from a reseller of transit? In other words, buying transit from a network which also buys transit. Presumably up near the top of the chain (Tier 1 vicinity), that transit reseller has a lot of peering in place with other folks in the same neighborhood (Tier 1 vicinity). But as long as a network is a transit reseller (i.e. they buy transit), then they are less likely to suffer from partition events caused by fractious peering negotiations.
--Michael Dillon
Can anyone explain to me why end users find it so important to label carriers as "Tier 1" or "Tier 2"? The prevailing theory in the heads of prospective customers is that a "Tier 1" is somehow inherently better than a "Tier 2" (or lower), even though they don't quite understand the concepts behind why the "Tier" designation even exist(s/ed). These labels, at least to me, are no longer very relevant in today's internet world. In fact, would anyone agree that being a "Tier 1", as Cogent believes themselves to be, leaves that network in a very painful position when things like their frequent peering disputes happen? For an NSP, it's obviously a "good thing" to be SFI-only, as in theory, it _should_ lower your costs. YMMV, as mentioned in a previous thread. However, what does it really matter to an end-user, especially if they are biased towards using "Tier 1" networks only? Why does a network who purchases transit give the impression to end users that that network's internet genitalia is somehow smaller than, say, Verizon or AT&T? I can see merit in touting the size and coverage of the actual network, but it's always been my understanding that this is not the true definition of the tiered system. -evt
On Nov 3, 2008, at 10:02 AM, Eric Van Tol wrote:
-----Original Message----- From: michael.dillon@bt.com [mailto:michael.dillon@bt.com] Sent: Monday, November 03, 2008 8:55 AM
Let's put it another 'nother way. Would an end user get better connectivity by buying from a reseller of transit? In other words, buying transit from a network which also buys transit. Presumably up near the top of the chain (Tier 1 vicinity), that transit reseller has a lot of peering in place with other folks in the same neighborhood (Tier 1 vicinity). But as long as a network is a transit reseller (i.e. they buy transit), then they are less likely to suffer from partition events caused by fractious peering negotiations.
--Michael Dillon
Can anyone explain to me why end users find it so important to label carriers as "Tier 1" or "Tier 2"?
In my experience, end users generally don't know and almost never care. It's the sales people who talk about tiers. Regards Marshall
The prevailing theory in the heads of prospective customers is that a "Tier 1" is somehow inherently better than a "Tier 2" (or lower), even though they don't quite understand the concepts behind why the "Tier" designation even exist(s/ed). These labels, at least to me, are no longer very relevant in today's internet world. In fact, would anyone agree that being a "Tier 1", as Cogent believes themselves to be, leaves that network in a very painful position when things like their frequent peering disputes happen?
For an NSP, it's obviously a "good thing" to be SFI-only, as in theory, it _should_ lower your costs. YMMV, as mentioned in a previous thread. However, what does it really matter to an end- user, especially if they are biased towards using "Tier 1" networks only? Why does a network who purchases transit give the impression to end users that that network's internet genitalia is somehow smaller than, say, Verizon or AT&T? I can see merit in touting the size and coverage of the actual network, but it's always been my understanding that this is not the true definition of the tiered system.
-evt
Patrick W. Gilmore wrote:
4. There is a reason behind ratios which has nothing to do with telco "sender-pays"
There is an alleged reason.
Hot potato routing + very poor ratios puts much more of the cost on the receiving network. This is a valid, logical, and costly concern for receiving networks.
So what? So the argument is: 1) Your customers want to receive from my customers. 2) Receiving is more expensive. 3) Therefore you should pay me? I want to send, and sending is cheap. Your customers want to do the expensive receiving, not mine. My customers want to do the cheap sending. The ratio argument is nonsense. If your customers want to receive mostly, and receiving is expensive, they should pay you more to cover your higher costs in receiving traffic. If my customers mostly want to send, and sending is cheap, then I should pay less, since I want to do the cheap thing and you want to do the expensive thing. If we hold a convention in Idaho, the people who live near Idaho don't pay money to the people who came from out-of-town. And if you live in Alaska, get used to the fact that you're going to be paying more than your share of transportation expenses to go to conventions. If you don't like it, don't live in Alaska.
That this issue has nothing to do with telco "sender- pays" mentality is. (Of course, the telcos might still have that mentality, but that doesn't change the facts.)
You are probably right that it's not an old-fahsioned "sender-pays" mentality. But it is complete nonsense. Your customers pay you to receive their traffic, even if that's more expensive than other things they might want to do. Your customers pay you to carry their traffic across your network between them and the next network in the line. There is no reason anyone else should compensate you for doing this. DS
David Schwartz wrote:
Your customers pay you to carry their traffic across your network between them and the next network in the line. There is no reason anyone else should compensate you for doing this.
What it all comes down to is that the majority of eyeballs are on "residential" connections that are relatively expensive to provide but for which are sold at a relatively low price (often 1/10th as much per megabit of capacity). Those eyeball ISPs cannot or will not charge their customers the full cost of "receiving" traffic so they want money from the more profitable content ISPs "sending" the traffic to offset their losses. This is also one of the reasons eyeball ISPs want to stamp out P2P: both ends of the connections are on unprofitable lines and there is _nobody_ paying for the traffic. Just follow the money. S
* Stephen Sprunk
What it all comes down to is that the majority of eyeballs are on "residential" connections that are relatively expensive to provide but for which are sold at a relatively low price (often 1/10th as much per megabit of capacity). Those eyeball ISPs cannot or will not charge their customers the full cost of "receiving" traffic so they want money from the more profitable content ISPs "sending" the traffic to offset their losses.
Another point worth mentioning is that the traffic is going to flow between those two ISPs _anyway_. Therefore, in many cases the only ones to profit from them not reaching a peering agreement (settlement-free or not) is their upstream(s), who is probably delighted to be able to charge them both for the transit traffic. Regards, -- Tore Anderson
On Nov 3, 2008, at 10:41 AM, Tore Anderson wrote:
Another point worth mentioning is that the traffic is going to flow between those two ISPs _anyway_.
I believe the events of 2-3 days ago disproves your assertion.
Therefore, in many cases the only ones to profit from them not reaching a peering agreement (settlement-free or not) is their upstream(s), who is probably delighted to be able to charge them both for the transit traffic.
Again, supposed facts not in evidence. I mentioned in the thread earlier that it is entirely possible Eyeball Network saves money by turning down peering and paying a transit provider to deliver the packets where Eyeball Network wants. Fiber, routers, IX ports, engineers, etc. are all expensive. Transit these days is not. Doesn't mean Eyeball Network actually does save money. Just means you don't know either way. -- TTFN, patrick
* Patrick W. Gilmore
On Nov 3, 2008, at 10:41 AM, Tore Anderson wrote:
Another point worth mentioning is that the traffic is going to flow between those two ISPs _anyway_.
I believe the events of 2-3 days ago disproves your assertion.
Having partitioned transit-free networks is going to continue to be the exception and not the rule, or at least I hope so...
Therefore, in many cases the only ones to profit from them not reaching a peering agreement (settlement-free or not) is their upstream(s), who is probably delighted to be able to charge them both for the transit traffic.
Again, supposed facts not in evidence.
It does happen. I've experienced it myself.
I mentioned in the thread earlier that it is entirely possible Eyeball Network saves money by turning down peering and paying a transit provider to deliver the packets where Eyeball Network wants.
But I never said that this could never be the case either, in fact I think you're right; it is indeed entirely possible that this in many cases is the reason for one network to turn down the other's peering request. If only one of the networks is geographically large or the proposed peering agreement does not have provisions about multiple peering locations (and respecting MED), it's probably even more likely to be the case. However, it is also entirely possible that the networks simply are too stubborn to either accept paid peering, to loosen up on any requirements of balanced ingress/egress ratio, or to commit to cold potato routing, or whatever. I suspect that the likelyhood of this beeing the case is dependant on the size of the networks involved, though. Regards, -- Tore Anderson
David Schwartz wrote:
The ratio argument is nonsense. If your customers want to receive mostly, and receiving is expensive, they should pay you more to cover your higher costs in receiving traffic. If my customers mostly want to send, and sending is cheap, then I should pay less, since I want to do the cheap thing and you want to do the expensive thing.
If it costs one party to an SFI agreement more than the other (total cost, including intangibles) this makes the agreement less attractive, perhaps to the point of inequitability. Where one party profits more from the agreement than another, there is less incentive for the interconnection to be settlement-free. There is no father figure standing there saying 'Party A and Party B must SFI regardless of cost' - that decision is up to the relevant commercial minds within Party A and Party B to carry out the required analysis and negotiate as required. Will
On Nov 3, 2008, at 10:03 AM, David Schwartz wrote:
Patrick W. Gilmore wrote:
4. There is a reason behind ratios which has nothing to do with telco "sender-pays"
There is an alleged reason.
Peering rations were first 'big news' when BBN wanted to de-peer Above.Net, Global Center, and Exodus in 1998. I spent a long time chatting with BBN's CTO about why BBN wanted to do this. I am convinced the facts are correct. Perhaps more importantly, anyone who understands how BGP, fiber, routers, etc. work can figure this out for themselves without even talking to another network. Put another way, this is not a fantasy, supposition, bluster, etc. What do you have to convince people otherwise?
Hot potato routing + very poor ratios puts much more of the cost on the receiving network. This is a valid, logical, and costly concern for receiving networks.
So what? So the argument is:
1) Your customers want to receive from my customers.
2) Receiving is more expensive.
3) Therefore you should pay me?
I don't remember saying that at all. Perhaps you should re-read my post.
I want to send, and sending is cheap. Your customers want to do the expensive receiving, not mine. My customers want to do the cheap sending.
The ratio argument is nonsense. If your customers want to receive mostly, and receiving is expensive, they should pay you more to cover your higher costs in receiving traffic. If my customers mostly want to send, and sending is cheap, then I should pay less, since I want to do the cheap thing and you want to do the expensive thing.
The ratio argument is not nonsense. And fortunately, what you spout on NANOG has no effect on reality.
Your customers pay you to carry their traffic across your network between them and the next network in the line. There is no reason anyone else should compensate you for doing this.
<eyeball-network advocate> Your customers pay you to deliver their traffic to my eyeballs. There is no reason I should compensate you for doing so. </advocate> The FACT is that a point-source sending traffic to distributed receivers combined with hot-potato routing puts more of the cost on the receiver. That fact is not in dispute, apparently even you agree. From that fact, you can argue whether that is grounds for de-peering, settlements, etc. But the fact stands. Also, please note no one is forcing you to pay anyone. Cogent decided not to pay. There is no law forcing them, Sprint is not holding a gun to Dave's head. But just like no one is forcing the sender to pay, no one is forcing the receiver to pay either. Personally I think business problems have a business solution. For the ratio problems in 1998, Above.Net (and others, probably), agreed to carry the traffic and deliver it closer to BBN's eyeballs, thereby shifting the majority of the cost to AN. Dave (Rand this time, not Schaeffer) actually preferred it that way, saying he trusted his network more than BBN's and AN's customers pay him for quality. Hrmm, sounds like just the opposite of how you treat your customer's traffic.... -- TTFN, patrick
In a message written on Mon, Nov 03, 2008 at 10:40:46AM -0500, Patrick W. Gilmore wrote:
The FACT is that a point-source sending traffic to distributed receivers combined with hot-potato routing puts more of the cost on the receiver. That fact is not in dispute, apparently even you agree.
s/more of the cost/more of the network transport cost/. Having been at AboveNet when several providers tried to tell us it was unfair that we had data centers near exchanges and it cost them a lot of money to put in access to the end user I simply pointed them to our stock reports where we were spending hundreds of millions of dollars building data centers, and our customers were further buying hundreds of millions of dollars of servers to serve up the content. You are correct that hot potato routing makes more of the bits flow on the "eyeball" network. This of course can be mitigated by using alternate routing strategies; for instance AboveNet actively encouraged other providers to send us Meds and in some cases deaggregate to us such that we could carry the bits on our network. However I am skeptical when looking at the total system cost that the costs are as disproportionate as you suggest. It's great to build an eyeball network, but unless someone invests in data centers and servers it's not going to get to any content. The view that network transport cost is the only interesting figure is a historical artifact of the days of $5000/megabit transit. There was a time it was more expensive to carry bits across the country than to build a data center; if anything I think it's now the exact opposite. The data center providers are actually taking more of the costs than many end user ISP's are due to the falling price of transport and the relatively static price of real estate, generators, air conditioners, and the like. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Incase this has not hit the list yet: http://www.pcworld.com/businesscenter/article/153194/sprint_reconnects_cogen... Sprint Reconnects Cogent, but Differences Are Unresolved Mikael Ricknäs, IDG News Service Monday, November 03, 2008 7:50 AM PST On Sunday Sprint Nextel reconnected its network with Cogent Communications after severing it earlier last week. The reconnection is only temporary, as the core issues in this dispute have not changed, Sprint said in a statement to its customers. As a result, it is again possible for Sprint customers and Cogent customers to directly communicate across the Internet. Data supplied by Keynote Systems confirms that the two networks are again communicating with each other. Sprint's view of what led up to its disconnecting from Cogent Communications on Oct. 30 differs substantially from what Cogent has stated. In shutting down the peering between the two, Sprint violated a contractual obligation to exchange Internet traffic with Cogent on a settlement-free peering basis, according to Cogent. But that's just fiction, according to Sprint, because at no time did the two enter into an actual contract. In 2006, Sprint and Cogent formed a trial agreement that ended in September last year. A three-month commercial trial indicated that Cogent didn't meet the minimum traffic exchange criteria agreed to by both parties, according to Sprint. As a result, settlement-free peering was not established, Sprint said. Instead, Sprint wants Cogent to pay for its ongoing connection to the Sprint network. But despite repeated collection attempts by Sprint, Cogent has not done that. Nonpayment on Cogent's part is the reason Sprint decided to disconnect from Cogent last week, a process that had started on Oct. 7, and shouldn't have come as a surprise for Cogent, Sprint said in its customer statement. What happens next remains to be seen. The two operators are involved in litigation over the matter. Sprint filed a lawsuit against Cogent on Sept. 2 in Fairfax County Circuit Court in Virginia for breach of contract. On its part, Cogent said it wants settlement-free peering with Sprint.
Barrett Lyon wrote:
Incase this has not hit the list yet:
http://www.pcworld.com/businesscenter/article/153194/sprint_reconnects_cogen...
Sprint Reconnects Cogent, but Differences Are Unresolved Mikael Ricknäs, IDG News Service
Monday, November 03, 2008 7:50 AM PST On Sunday Sprint Nextel reconnected its network with Cogent Communications after severing it earlier last week. The reconnection is only temporary, as the core issues in this dispute have not changed, Sprint said in a statement to its customers.
As a result, it is again possible for Sprint customers and Cogent customers to directly communicate across the Internet. Data supplied by Keynote Systems confirms that the two networks are again communicating with each other.
Sprint's view of what led up to its disconnecting from Cogent Communications on Oct. 30 differs substantially from what Cogent has stated.
In shutting down the peering between the two, Sprint violated a contractual obligation to exchange Internet traffic with Cogent on a settlement-free peering basis, according to Cogent. But that's just fiction, according to Sprint, because at no time did the two enter into an actual contract.
In 2006, Sprint and Cogent formed a trial agreement that ended in September last year. A three-month commercial trial indicated that Cogent didn't meet the minimum traffic exchange criteria agreed to by both parties, according to Sprint. As a result, settlement-free peering was not established, Sprint said.
Instead, Sprint wants Cogent to pay for its ongoing connection to the Sprint network. But despite repeated collection attempts by Sprint, Cogent has not done that. Nonpayment on Cogent's part is the reason Sprint decided to disconnect from Cogent last week, a process that had started on Oct. 7, and shouldn't have come as a surprise for Cogent, Sprint said in its customer statement.
What happens next remains to be seen. The two operators are involved in litigation over the matter. Sprint filed a lawsuit against Cogent on Sept. 2 in Fairfax County Circuit Court in Virginia for breach of contract.
On its part, Cogent said it wants settlement-free peering with Sprint.
So basically, it won't be resolved until Cogent gets to keep free access to Sprint. Where's my free access to Sprint? Can I cry and scream that I have to pay for my access too because I don't qualify for free peering? ~Seth
On Mon, Nov 3, 2008 at 1:34 PM, Nicolas Antoniello <nantoniello@antel.net.uy> wrote:
Sorry for my possible ignorance, but could you explain me what are you calling "transit-free"?
Transit: You pay an ISP to send and receive traffic to and from "the Internet." "The Internet" consists of: his paid customers, his peers and anybody he pays for an Internet Transit connection. When you say, "I have an Internet connection," that means a transit connection. Peering: You arrange with an ISP to send and receive traffic to and from his paid customers only. The peer network -does not- pass your traffic to his peers or transit connections, only to his paid customers Because his paid customers have already paid him to carry this traffic from you (just as yours have paid you to carry the traffic to him) such a connection is often "settlement free" meaning that neither peer network pays the other for the privilege of a connection or the packets sent across it. Transit Free: A network which does not purchase any Transit connections at all. This is presumably because your customers plus the paid customers of all of your Peers together include 100% of the users on the Internet. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004
And a 'Tier One' nework is a transit-free network that can reach all end points (end user IP addresses)? Roderick S. Beck Director of European Sales Hibernia Atlantic 13-15, rue Sedaine, 75011 Paris http://www.hiberniaatlantic.com Wireless: 1-212-444-8829. French Landline: 33+1+4355+8224 French Wireless: 33-6-14-33-48-97. AOL Messenger: GlobalBandwidth rod.beck@hiberniaatlantic.com rodbeck@erols.com ``Unthinking respect for authority is the greatest enemy of truth.'' Albert Einstein.
And a 'Tier One' nework is a transit-free network that can reach all end points (end user IP addresses) A "Tier One" is best defined as "the ISP the salesman represents." It originally referred to transit-free, settlement-free ISPs, but over time, bigger ISPs began to play with the definition to try to differentiate themselves from the smaller ISPs that did not have the reach they had, and smaller players began glossing over paid peering and similar arrangements and claiming Tier One status. Since there's no
Rod Beck wrote: formal definition, anybody can claim they are Tier One or that somebody else is not. Don't trust the term.
On Nov 3, 2008, at 3:49 PM, Rod Beck wrote:
And a 'Tier One' nework is a transit-free network that can reach all end points (end user IP addresses)?
A transit free network that has no settlements. Which means no network is strictly "tier one". Read <http://en.wikipedia.org/w/index.php?title=Tier_1_network
. [*]
Interestingly, I wrote in the article that Cogent has settlement with Sprint and is therefore not "tier one". Apparently Cogent disagreed with me.... :-) -- TTFN, patrick [*] I got into a bit of a disagreement with others on Wikipedia because there is no citation for the "facts" in the article. While I understand the desire to have only verifiable, objective facts in Wikipedia, the alternative is to have no information. Perhaps I was being silly, but I prefer to have what I believe is correct info, properly caveated, over nothing.
The FACT is that a point-source sending traffic to distributed receivers combined with hot-potato routing puts more of the cost on the receiver. That fact is not in dispute, apparently even you agree.
Mmm, that's really not a fact. I like the way you painted it though. When you're looking at fundamental cost, let's not get confused by labelling it as "distributed" receivers. There are several elements to the cost of getting data from point A to B. Consider some examples. Let's pretend: You are a "point-source" sender on the east coast. I am a "point-destination" receiver in the midwest. The receiving network can build out its own national network, so that a node is near you, and you (or someone that transits your packets to it) peer with it. In such a case, yes, hot-potato routing dumps most of the cost on the receiving network, which is the sort of case that commonly exists these days. However, there is no reason that receivers need to build national networks. A small receiver (think: regional ISP) could certainly bring a line down to Chicago, get relatively inexpensive transit bandwidth, and leave the sending site with the task of purchasing transit from someone who had a national network in order to get packets to the receiving network. Now, there are a variety of problems here. One problem is that there is inherently a much higher cost-per-bit to actually deliver bits on a residential broadband network than there is where bits are delivered in a data center. Despite this, residential broadband network feel a downward pressure on price. No successful residential broadband network actually has a network capable of delivering large numbers of bits to all its customers simultaneously, and reliance on incredible overcommit is very common. But there's another factor. Many ISP's "feel" that it is cheaper to internalize network costs; a regional network might still get a connection to the east and west coasts, and peer on each coast, on the theory that doing this was "cheaper" than paying transit. In doing so, however, they internalize that cost for what-used-to-be- transit, and remember that transit used to be paid for at both ends, by the sender and the receiver. Now it's all being paid for by the receiver, at least in this example. Therefore, I will take issue with the statement "puts more of the cost on the receiver." I will certainly agree that the receiver has underwritten more of the cost, but as a deliberate choice. This is, of course, only a very small component of a much more complex picture, but it can be used to help people understand why things such as paid peering exist. In the case of Cogent, I have seen numerous examples of situations where data entered the Cogent network and left it within the same building; one has to assume that this is very inexpensive for Cogent, of course based on the fact that their peering with other networks is settlement-free. At least in major POP's such as Ashburn, this is actually more the norm than not, last I was able to see live data from a customer to Cogent. $10/meg from a paying customer, spit out onto other networks settlement-free, Cogent's costs are rather minimal there.
From that fact, you can argue whether that is grounds for de-peering, settlements, etc. But the fact stands.
No, it doesn't. It's not even a fact. It's just the way things often work right now, because most have chosen to work it that way.
Personally I think business problems have a business solution. For the ratio problems in 1998, Above.Net (and others, probably), agreed to carry the traffic and deliver it closer to BBN's eyeballs, thereby shifting the majority of the cost to AN. Dave (Rand this time, not Schaeffer) actually preferred it that way, saying he trusted his network more than BBN's and AN's customers pay him for quality.
That would seem a reasonable solution. Cold-potato routing is relatively difficult to do well on a large scale, but perhaps a focus on this could reduce the "political" problem. Cogent has been depressing the price of bandwidth for years, largely because of their hot-potato routing strategy. Perhaps this isn't Cogent's fault so much as it is a failure to engineer equitable cost-management strategies that would discourage companies like Cogent from selling for $4/meg. Please note that I'm mainly trying to paint a more accurate and comprehensible picture for the audience here, Patrick. I expect that none of this is news to /you/. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
In a message written on Mon, Nov 03, 2008 at 01:26:14AM -0500, Patrick W. Gilmore wrote:
Having skimmed the Sprint / Cogent threads, I saw multiple errors and lots of really bad guesses. Instead of replying individually, I thought I would sum up a few facts so everyone was on the same page. This way when we run off into another 100 post thread, we can at least -start- from reality (although I would bet serious cash on long odds we will diverge from it soon enough).
I notice a significant bit of information which has not been posted in any of the posts I have read. If I were to rank networks by the difficulty of getting settlement free peering I believe Sprint would be at the top of the list. The traffic volume they demand is pretty much higher than anyone else, the ratio they demand is tighter than anyone else, and they demand you meet them at Sprint POP's. Not friendly carrier neutral colos, they are stuck in the Telco world of we run half the circuits to you and run half the circuits to us. Never mind that only a few carriers are even in a Sprint POP to deliver a circuit. I don't necessarily fault Sprint for setting their requirements as such; if I were them I may well do the same. Basically they have decided they have enough peers and attempt to keep their requirements high enough such that they need not ever add more. However, looking at the issue from an external view, since I work for neither Cogent nor Sprint, I do believe there are two fundamental "fairness" questions: 1) Do all of Sprint's current peers meet Sprint's peering policy? 2) Due to the size and customer base of Sprint's network is it possible for anyone to meet their peering policy? While these are rarely tested by anyone who could enforce them (save the odd merger of mega players) they really are the interesting questions. Several of the "good old boys" in the settlement free club have fallen from where they once were, yet they don't get depeered. If you have peers that no longer meet your current peering policy and don't get depeered, how is that fair? Of course, the way the industry is structured those arrangements are completely hidden, so we'll never know if everyone meets the criteria or no one does. The second is also very interesting. Most networks came up with their criteria (say, 2:1 ratio) 10, 15, or even 20 years ago. The network was a very different place at that time, when a T1 was a large circuit and interconnects were DS-3's. In particular, most access was symmetrical; T1's, DS-3's, maybe ethernet (the 10M kind) in a colo; and what's more, everyone was selling the same products. There was no DSL, no cable modems, no EV-DO cards. There have been two interesting developments in the industry since that time, and the traditional criteria don't serve them well. Asymmetrical access and specialization. Take a pure end user play network, like most of the cable modem networks, and peer them with any content player and the ratio is never going to be 1.5:1 or 2:1, but they both get benefit from the arrangement. Many of the new networks have realized this and adjusted their criteria, many of the old players, like Sprint, may have morphed into one of these new buckets, but not realized it yet. This is an area where I believe our industry needs to grow up. We like to operate on the assumption that if we have to jump over a particular bar to peer with someone that policy has been applied fairly across all players and is reasonable. However, there is no way for anyone to verify that; short of the DoJ stepping in during a merger. I am fairly sure that the requirements are not enforced evenly by some players (no comment on Sprint or Cogent in this case); that they use the fact that actual traffic volumes and ratios are hidden to play poker with other networks. While I'm generally pro business and think this is a good thing; I believe it has reached a point in our industry where it is damaging the Internet. The increase in peering spats to me is an indication that the players are not looking out for stability, performance, and insuring their customers are getting the access that they pay for, but rather that they are interested in playing poker with each other. I don't even think it's about the money, but rather about the power. The interesting thing is that there is a disruptive force on the horizon. The IPv6 transition will change the peering landscape. Traffic volumes and ratios will be moving to new providers as people transition technologies. This is going to lead to quite a shake up on the peering front I'm afraid, and I fear the result is going to be great instability. The lack of transparency means that those in power will try and bluff their way forward; the question is how many people who have some visibility will call that bluff. In all, this situation just reminds me of what I already knew. When there is a depeering both networks are at fault. The fact that these two players couldn't come to some commercially acceptable terms to both of them speaks volumes about both. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Patrick writes:
3. Standard transit contracts do not guarantee full connectivity If you are a Cogent customer, it is very unlikely your contract will allow you SLA or other credits for not being able to reach Sprint unless you negotiated something special. I doubt Sprint's standard contract is much different. Transit contract SLAs end at AS boundaries. This is because Network A has no control over Network B and therefore will not give credit if Network B fails. Of course, you can still sue, threaten to terminate, etc., but the letter of the contract almost certainly says nothing about packets going beyond your transit provider's ASN.
I am not aware of any major content provider who still has any agreements in place that don't say anything about routing past the provider's network. Some weren't paying any attention when they signed up initially and didn't get the specific provisions. But once bitten by such, renewals Do Not Happen without additional clauses being inserted. I don't rule out there still being such agreements, but I think that anyone with a clue and enough traffic to think about multihoming has been exposed to this and should have insisted on some legal protection about best-effort to route to rest of world. It never fails to impress me how many people have little clue, though... -george william herbert gherbert@retro.com
On Nov 3, 2008, at 8:16 PM, George William Herbert wrote:
Patrick writes:
3. Standard transit contracts do not guarantee full connectivity If you are a Cogent customer, it is very unlikely your contract will allow you SLA or other credits for not being able to reach Sprint unless you negotiated something special. I doubt Sprint's standard contract is much different. Transit contract SLAs end at AS boundaries. This is because Network A has no control over Network B and therefore will not give credit if Network B fails. Of course, you can still sue, threaten to terminate, etc., but the letter of the contract almost certainly says nothing about packets going beyond your transit provider's ASN.
I am not aware of any major content provider who still has any agreements in place that don't say anything about routing past the provider's network.
Content hosting != transit
Some weren't paying any attention when they signed up initially and didn't get the specific provisions. But once bitten by such, renewals Do Not Happen without additional clauses being inserted.
I don't rule out there still being such agreements, but I think that anyone with a clue and enough traffic to think about multihoming has been exposed to this and should have insisted on some legal protection about best-effort to route to rest of world. It never fails to impress me how many people have little clue, though...
"Never underestimate the power of human stupidity." -- TTFN, patrick
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones. Dave.
On Nov 4, 2008, at 9:49 AM, David Freedman wrote:
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones.
If two transit free networks have a technical failure which disables all peering between them, the Internet cannot route around it. Intention is not the gating factor here. Intention just -guarantees- a problem exists, turning off BGP sessions and/or light is still the base problem. :-) -- TTFN, patrick
Patrick W. Gilmore wrote:
On Nov 4, 2008, at 9:49 AM, David Freedman wrote:
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones.
If two transit free networks have a technical failure which disables all peering between them, the Internet cannot route around it.
From a technical standpoint, the Internet is always suffering from multiple
Sure it can. The traffic just flows through any of the providers that still have reliable high-bandwidth connectivity to both of those providers. Unless, of course, a pre-existing political failure prohibits this traffic. The Internet can't route around that political failure. political failures. This leaves it vulnerable to small technical failures it could otherwise route around. DS
On Nov 4, 2008, at 11:02 AM, David Schwartz wrote:
Patrick W. Gilmore wrote:
On Nov 4, 2008, at 9:49 AM, David Freedman wrote:
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones.
If two transit free networks have a technical failure which disables all peering between them, the Internet cannot route around it.
Sure it can. The traffic just flows through any of the providers that still have reliable high-bandwidth connectivity to both of those providers.
Unless, of course, a pre-existing political failure prohibits this traffic. The Internet can't route around that political failure.
Perhaps you missed the "transit free" part. If Sprint & UUNET have a technical failure causing all peering to go down, Level 3 will not magically transport packets between the two, despite the fact L3 has "reliable high-bandwidth connectivity to both of those providers". How would you propose L3 bill UU & Sprint for it? On second thought, don't answer that, I don't think it would be a useful discussion. Or are you claiming the fact every network does not give every other network transit a "political failure". If you are, we should agree to disagree and move on.
From a technical standpoint, the Internet is always suffering from multiple political failures. This leaves it vulnerable to small technical failures it could otherwise route around.
See above. I do not think it is a "political failure" that I do not give you free transit. -- TTFN, patrick
The concept of "Transit Free" is a political failure, not a technical one. The protocols are designed, and the original concept behind the Internet is, to propagate all reachability via all paths. IE to use Transit if peering fails. Not doing so is a policy decision that breaks the redundancy in the original design.
-----Original Message----- From: Patrick W. Gilmore [mailto:patrick@ianai.net] Sent: Tuesday, November 04, 2008 8:10 AM To: NANOG list Subject: Re: Sprint v. Cogent, some clarity & facts
On Nov 4, 2008, at 11:02 AM, David Schwartz wrote:
Patrick W. Gilmore wrote:
On Nov 4, 2008, at 9:49 AM, David Freedman wrote:
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones.
If two transit free networks have a technical failure which disables all peering between them, the Internet cannot route around it.
Sure it can. The traffic just flows through any of the providers that still have reliable high-bandwidth connectivity to both of those providers.
Unless, of course, a pre-existing political failure prohibits this traffic. The Internet can't route around that political failure.
Perhaps you missed the "transit free" part.
If Sprint & UUNET have a technical failure causing all peering to go down, Level 3 will not magically transport packets between the two, despite the fact L3 has "reliable high-bandwidth connectivity to both of those providers". How would you propose L3 bill UU & Sprint for it? On second thought, don't answer that, I don't think it would be a useful discussion.
Or are you claiming the fact every network does not give every other network transit a "political failure". If you are, we should agree to disagree and move on.
From a technical standpoint, the Internet is always suffering from multiple political failures. This leaves it vulnerable to small technical failures it could otherwise route around.
See above. I do not think it is a "political failure" that I do not give you free transit.
-- TTFN, patrick
On Nov 4, 2008, at 11:51 AM, Tomas L. Byrnes wrote:
The concept of "Transit Free" is a political failure, not a technical one.
We disagree.
The protocols are designed, and the original concept behind the Internet is, to propagate all reachability via all paths. IE to use Transit if peering fails.
Not doing so is a policy decision that breaks the redundancy in the original design.
Using the 'original design' (and honestly, your assertion is debatable) would not have allowed the Internet to scale to the size it is today. Or even the size it was 10 years ago. So I guess you could say the current situation is a political success. -- TTFN, patrick
-----Original Message----- From: Patrick W. Gilmore [mailto:patrick@ianai.net] Sent: Tuesday, November 04, 2008 8:10 AM To: NANOG list Subject: Re: Sprint v. Cogent, some clarity & facts
On Nov 4, 2008, at 11:02 AM, David Schwartz wrote:
Patrick W. Gilmore wrote:
On Nov 4, 2008, at 9:49 AM, David Freedman wrote:
2. The Internet cannot "route around" de-peering I know everyone believes "the Internet routes around failures". While occasionally true, it does not hold in this case. To "route around" the "failure" would require transit. See item #1.
The internet "routes around" technical failures, not political ones.
If two transit free networks have a technical failure which disables all peering between them, the Internet cannot route around it.
Sure it can. The traffic just flows through any of the providers that still have reliable high-bandwidth connectivity to both of those providers.
Unless, of course, a pre-existing political failure prohibits this traffic. The Internet can't route around that political failure.
Perhaps you missed the "transit free" part.
If Sprint & UUNET have a technical failure causing all peering to go down, Level 3 will not magically transport packets between the two, despite the fact L3 has "reliable high-bandwidth connectivity to both of those providers". How would you propose L3 bill UU & Sprint for it? On second thought, don't answer that, I don't think it would be a useful discussion.
Or are you claiming the fact every network does not give every other network transit a "political failure". If you are, we should agree to disagree and move on.
From a technical standpoint, the Internet is always suffering from multiple political failures. This leaves it vulnerable to small technical failures it could otherwise route around.
See above. I do not think it is a "political failure" that I do not give you free transit.
-- TTFN, patrick
The concept of "Transit Free" is a political failure, not a technical one.
We disagree.
Perhaps some examples are needed? If you drive in a screw with a big hammer, the end result is not pleasing. For one, a screw will not have the holding power of a nail. For another, the screw and the hammer are both likely to damage the objects being attached. Nevertheless, you would be hardpressed to say that this is a technical failure. A wise person could have imposed the policy of always using screwdrivers to drive in a screw, and to only drive in nails when using a hammer. Same technology, different results. In the case of peering arrangements, the term "transit free" hides a multitude of sins. It is pure spin, dreamed up by marketing people back in the 90's when the Big Five ISPs were trying to control the market and make it hard for competitors to gain mythical Tier 1 status. In the end, everyone drank the koolaid and the whole arena of network operations has been poisoned by it. Has anyone heard of a backup route? With a longer path so it is never used unless there is a real emergency? Why was there no backup route available to carry the Sprint <-> Cogent traffic? Because there was a political failure in both Sprint and Cogent. Back in 2000 it was acceptable for the big New York banks to have all their eggs in one basket in central Manhattan. In 2002, it was no longer acceptable. Do we really need a 911 magnitude of disaster on the Internet for people to wake up and smell the coffee? The Internet is no longer a kewl tool built and operated by the cognoscenti to meet their own interests. It is now part of every nation's and everbody's critical infrastructure. It needs to be engineered and operated better so that it does not end up partitioning for dumb reasons. --Michael Dillon
(note: i don't think sprint or cogent is being evil in this situation.) <michael.dillon@bt.com> writes:
Has anyone heard of a backup route? With a longer path so it is never used unless there is a real emergency? Why was there no backup route available to carry the Sprint <-> Cogent traffic? Because there was a political failure in both Sprint and Cogent.
what you're calling a political failure could be what others call a rate war. i'd imagine that cogent's cost structure lower than most networks' (since their published prices are so low and since they're not pumping new money in from the outside and since they have no non-IP business they could be using to support "dumping"). if cogent's rates aare also lowest then the other large networks might be losing customers toward cogent and those other large networks might feel they are hurting their own cause by peering, settlement free, with this new-economy competitor. if that's the case then the "political failure" you describe might be a matter of cogent saying "we don't want our prices to our customers to reflect the capital inefficiencies of other networks" and those other networks saying "we do." note, i'm not on the inside and i'm working only from public knowledge here and so this is all supposition. but calling it "political failure" when these possibilities exist seems like a stretch. there'd be no other leverage whereby cogent could protect the price point its customers seem to like so well. i'm not saying a cogent customer will be glad to trade some instability to get those prices, but i am saying that if this long chain of guesses is accurate it likely also represents the ONLY way to drive efficiency in a competitive capital-intensive market.
Back in 2000 it was acceptable for the big New York banks to have all their eggs in one basket in central Manhattan. In 2002, it was no longer acceptable. Do we really need a 911 magnitude of disaster on the Internet for people to wake up and smell the coffee? The Internet is no longer a kewl tool built and operated by the cognoscenti to meet their own interests. It is now part of every nation's and everbody's critical infrastructure. It needs to be engineered and operated better so that it does not end up partitioning for dumb reasons.
that sounds like justification for government regulation, if true. -- Paul Vixie
what you're calling a political failure could be what others call a rate war.
I only used the term "political failure" because it was the best match of the two options given. But you are right that it is necessary to let go of those terms and maybe, define your own, if you want to get to a deep understanding of what is going on.
but i am saying that if this long chain of guesses is accurate it likely also represents the ONLY way to drive efficiency in a competitive capital-intensive market.
In other words, the market demands more efficiency and gives their resources (money) to those willing to provide it. This is generally how any given industry makes step changes in efficiency, due to competitive pressures.
It is now part of every nation's and everbody's
critical infrastructure. It needs to be engineered and operated better so that it does not end up partitioning for dumb reasons.
that sounds like justification for government regulation, if true.
Not at all. It is justification for ISP management to stop peddling the same old, same old, and to restructure their businesses to be more like a utility. Granted, there is more margin to be made in vale-added services, but the core network operation needs to be run as an efficient utility, not have its inefficiency propped up by some lucrative voice products. Cogent seems to be operating according to the pure utility model. If the other ISPs don't want to get dragged into that space, they need to partition their businesses so that the high-priced services provide actual added value to customers over and above the generic Internet transit service. Regulation is just a way to prop up inefficient businesses as we should have all learned with the global telco disaster from the 1960's onward. As the sophistication of technology rose exponentially along with drastic drops in prices, telcos just ate up the extra margin by becoming more inefficient, until finally the regulators said enough is enough, and opened the doors to competition. --Michael Dillon
-----Original Message----- From: michael.dillon@bt.com [mailto:michael.dillon@bt.com] Sent: Wednesday, November 05, 2008 10:52 AM To: nanog@merit.edu Subject: RE: Sprint v. Cogent, some clarity & facts
what you're calling a political failure could be what others call a rate war.
I didn't really care about this, but now I'm curious. Since their peering was a 'trial', I'm assuming it hasn't always been there. Prior to Sprint and Cogent peering directly with each other, how did they communicate? Why was that functionality broken after they started peering?
On Wed, 5 Nov 2008, Church, Charles wrote:
I didn't really care about this, but now I'm curious. Since their peering was a 'trial', I'm assuming it hasn't always been there. Prior to Sprint and Cogent peering directly with each other, how did they communicate? Why was that functionality broken after they started peering?
They purchased transit (through NTT I believe) for connectivity to sprint. They removed that, because their goal has been to be transit-free. --- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html
Cogent transition or prep work this morning? AS 2914 NTT? Anyone else seeing similar changes abroad? ************ CORE INFRASTRUCTURE AFFECTING ALERT ************* -------------------------------------------------------------- BGP Status Change Sequence No: 1225957222 -------------------------------------------------------------- Change Status Type : origin / Origin Set Change -------------------------------------------------------------- BGP Monitored Prefix : 63.166.22.0/24 -------------------------------------------------------------- Update/Detection Time: 1225942611 / 1225957221 BGP ASN Update Time : 1225942611 -------------------------------------------------------------- BGP Set : 1239 -------------------------------------------------------------- BGP Gain(s) : 1239 -------------------------------------------------------------- BGP Loss(s) : -------------------------------------------------------------- Originating Entity : Synips / L2Networks -------------------------------------------------------------- ************ CORE INFRASTRUCTURE AFFECTING ALERT ************* -------------------------------------------------------------- BGP Status Change Sequence No: 1225957222 -------------------------------------------------------------- Change Status Type : last-hop / Last Hop Change -------------------------------------------------------------- BGP Monitored Prefix : 63.166.22.0/24 -------------------------------------------------------------- Update/Detection Time: 1225942611 / 1225957221 BGP ASN Update Time : 1225942611 -------------------------------------------------------------- BGP Set : 2914 -------------------------------------------------------------- BGP Gain(s) : 2914 -------------------------------------------------------------- BGP Loss(s) : -------------------------------------------------------------- Originating Entity : Synips / L2Networks -------------------------------------------------------------- OrgName: NTT America, Inc. OrgID: NTTAM-1 <http://ws.arin.net/whois/?queryinput=O%20!%20NTTAM-1> Address: 8005 South Chester Street Address: Suite 200 City: Centennial StateProv: CO PostalCode: 80112 Country: US ReferralServer: rwhois://rwhois.gin.ntt.net:4321/ ASNumber: 2914 ASName: NTT-COMMUNICATIONS-2914 <http://ws.arin.net/whois/?queryinput=A%20.%20NTT-COMMUNICATIONS-2914> ASHandle: AS2914 <http://ws.arin.net/whois/?queryinput=A%20!%20AS2914> Comment: RegDate: 1998-12-07 Updated: 2006-09-06 RTechHandle: PEERI-ARIN<http://ws.arin.net/whois/?queryinput=P%20!%20PEERI-ARIN> RTechName: Peering RTechPhone: +1-303-645-1900 RTechEmail: peering@ntt.net OrgAbuseHandle: NAAC-ARIN<http://ws.arin.net/whois/?queryinput=P%20!%20NAAC-ARIN> OrgAbuseName: NTT America Abuse Contact OrgAbusePhone: +1-800-551-1630 OrgAbuseEmail: abuse@ntt.net OrgNOCHandle: NASC-ARIN<http://ws.arin.net/whois/?queryinput=P%20!%20NASC-ARIN> OrgNOCName: NTT America Support Contact OrgNOCPhone: +1-800-551-1630 OrgNOCEmail: support@us.ntt.net OrgTechHandle: VIPAR-ARIN<http://ws.arin.net/whois/?queryinput=P%20!%20VIPAR-ARIN> OrgTechName: VIPAR OrgTechPhone: +1-303-645-1900 OrgTechEmail: vipar@us.ntt.net
On Tuesday 04 November 2008 11:55:01 Patrick W. Gilmore wrote:
On Nov 4, 2008, at 11:51 AM, Tomas L. Byrnes wrote:
The concept of "Transit Free" is a political failure, not a technical one.
We disagree. [snip] So I guess you could say the current situation is a political success.
I would say a 'qualified' political success. Like most other political solutions to technical problems, the concept of transit-free (and even the differentiation between transit and settlement-free peering) is a best-effort compromise, and works pretty well under most circumstances. But, which is worse: a completely 100% reachable Internet* with massive transit congestion that is massively expensive or the current partitionable Internet* that actually works and is affordable? Note: * There is no such thing as a 100% reachable 'Internet,' just a tangled lattice of interconnections of autonomous systems who provide best-effort interconnections. To what extent there IS an 'Internet' changes every second.
* tomb@byrneit.net (Tomas L. Byrnes) [Tue 04 Nov 2008, 17:51 CET]:
The concept of "Transit Free" is a political failure, not a technical one.
Yeah, networks should be free! And Cogent, if they don't get access to Sprint directly, should just set a default route over some public IX where Sprint is also present at to reach their network!! And then hack their routers to do likewise.
The protocols are designed, and the original concept behind the Internet is, to propagate all reachability via all paths. IE to use Transit if peering fails.
Yeah, the original concept of the internet. Like classful IP routing.
Not doing so is a policy decision that breaks the redundancy in the original design.
Because the original design totally had in mind established players locking out cheaper newcomers and explicitly specified a maximum band where prices for transit had to exist inside of. Please stop it. We've had enough. -- Niels. -- "We humans get marks for consistency. We always opt for civilization after exhausting the alternatives." -- Carl Guderian
On Tue, 04 Nov 2008 11:09:31 EST, "Patrick W. Gilmore" said:
If Sprint & UUNET have a technical failure causing all peering to go down, Level 3 will not magically transport packets between the two, despite the fact L3 has "reliable high-bandwidth connectivity to both of those providers". How would you propose L3 bill UU & Sprint for it? On second thought, don't answer that, I don't think it would be a useful discussion.
You have to admit that it's probably a very tempting concept for some L3 beancounter, unless the resulting UU<-L3->Sprint firehose is too big for L3's core to drink from...
Patrick W. Gilmore wrote:
From a technical standpoint, the Internet is always suffering from multiple political failures. This leaves it vulnerable to small technical failures it could otherwise route around.
See above. I do not think it is a "political failure" that I do not give you free transit.
We3 have, I think, a reality failure. The terminology comes from ARP and UUCP days. The reality comes from today, where traffic flows, as a maximum, between nodes that think there is something it for them to allow it to flow. If a packet has evidence of "fare paid" flows. End of story. The difference between "peer" and "transit is the coin used to pay the fare.
participants (31)
-
Barrett Lyon
-
Charles Gucker
-
Church, Charles
-
Dave Israel
-
David Freedman
-
david raistrick
-
David Schwartz
-
Eric Van Tol
-
Florian Weimer
-
George William Herbert
-
Jeff Aitken
-
Joe Greco
-
Kraig Beahn
-
Lamar Owen
-
Larry Sheldon
-
Leo Bicknell
-
Marshall Eubanks
-
michael.dillon@bt.com
-
Nicolas Antoniello
-
Niels Bakker
-
Patrick W. Gilmore
-
Paul Vixie
-
Paul Wall
-
Rod Beck
-
Seth Mattinen
-
Stephen Sprunk
-
Tomas L. Byrnes
-
Tore Anderson
-
Valdis.Kletnieks@vt.edu
-
Will Hargrave
-
William Herrin