Level 3 Peering Guidelines
I'm sorry to interrupt the discussion of "how long is your rack?" and "what do you do when your home ISP is down?" with something impacting some folks' cost and manner of selling services, however Level 3 just published its new peering guidelines, buried in comments on the L3/GX merger: http://fjallfoss.fcc.gov/ecfs/document/view?id=7021703819 http://fjallfoss.fcc.gov/ecfs/document/view?id=7021703818 FYI, -a
In a message written on Fri, Aug 19, 2011 at 04:29:05PM -0400, Adam Rothschild wrote:
I like to see Level 3 arguing this with the regulators. AboveNet persued this line of thinking with a number of ISP's in the late 1990's with some success, and I believe others did as well. AboveNet implmented it by honoring MEDs from peers, and thus doing a cold potato routing and carrying a higher bit-mile cost. Ratio is the most broken part about modern peering agreements. Ratio really has no bearing on the costs to either ISP, it is an artifact of their position in the world. That is to say the type of ISP (end user, content) location (urban, rural) or technologies (dsl, cable, leased line) along with user behavior determine the ratio. Early ISP's that had similar customer mixes, locations, and technologies could use ratio as an easy proxy, but those days are long gone. The primary challenge to change is the technical community coming up with some metric that is easy to measure and senior management can understand. You can go to a VP and say "the ratio to them is 1.5:1" and they get it (or so they think). Trying to make the same argument that on some vague level you are deriving "equal benefit" is much larger. I like Level 3's effort in using the bit-mile cost but I don't know any way to measure that metric easily on a large network. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Aug 19, 2011, at 4:51 PM, Leo Bicknell wrote:
In a message written on Fri, Aug 19, 2011 at 04:29:05PM -0400, Adam Rothschild wrote:
I like to see Level 3 arguing this with the regulators. AboveNet persued this line of thinking with a number of ISP's in the late 1990's with some success, and I believe others did as well. AboveNet implmented it by honoring MEDs from peers, and thus doing a cold potato routing and carrying a higher bit-mile cost.
Yes, Above.Net broke the original peering-ratio fight that way. Thank you for that. Too bad it didn't last.
Ratio is the most broken part about modern peering agreements. Ratio really has no bearing on the costs to either ISP, it is an artifact of their position in the world. That is to say the type of ISP (end user, content) location (urban, rural) or technologies (dsl, cable, leased line) along with user behavior determine the ratio. Early ISP's that had similar customer mixes, locations, and technologies could use ratio as an easy proxy, but those days are long gone.
There are many broken parts of 'modern' peering agreements. "You must spend as much on your backbone as I spend on mine." "You must be on at least 3 continents (no matter where the traffic goes to or from)." "'Peer' means 'equal'." "You have to be wearing a blue hat when you request peering." Whatever. I've said it before, more times than I can count. Peering is a business tool, a means to an end. The goal, the 'end', of for-profit companies is to make a profit. Sounds obvious, but surprising how many people forget this. If peering with another network will increase your profit and you turn down the peering request, you should be fired. Your ego should not be substituted for business decisions. Some people rationalize these decisions (read: "con themselves into believing") by saying if they do not peer with a network, they will gain revenue from that network. It even works sometimes. Most of the time it does not work, and in more & more cases ensures no revenue will -ever- flow between the companies. Just so no one jumps all over me, I am not even hinting that peering requests must always be accepted. There are rational, business reasons to turn down peering. But a one-size-fits-all policy is idiotic, as are most of the reasons peering requests are denied these days. More importantly, the real issue is power (leverage, whatever you want to call it). In the 90s, all users had to do to switch ISPs is change a few digits in their modem dialer. Content owners had to do quite a bit more to move a web site between hosting providers. These days, it is very difficult for end users to change broadband ISPs, frequently requiring equipment changes (e.g. DSL -> Cable), moving phone & TV, etc. Worse, for many people in the US, they have only one provider who can give them more than 1.5 Mbps.[*] Content owners can move hosting companies easily. Not hard to figure out who had more power in the 90s, and who has it now.
The primary challenge to change is the technical community coming up with some metric that is easy to measure and senior management can understand. You can go to a VP and say "the ratio to them is 1.5:1" and they get it (or so they think). Trying to make the same argument that on some vague level you are deriving "equal benefit" is much larger. I like Level 3's effort in using the bit-mile cost but I don't know any way to measure that metric easily on a large network.
Having a metric sr. mgmt can understand will not change which company has power. Companies that have power -will- exercise it to make money. L3 is on the wrong side of the power equation today, so they are trying to move the discussion to something that has real business logic. Their only hope of the leverage some other companies have is to either get the gov't to step in (EWW!), or pray the threat of gov't regulation will be enough. Dangerous game, if you ask me. -- TTFN, patrick [*] Anyone know what %-age of North American users have multiple choices for real broadband (e.g. > 1.5 Mbps, or even > 4 Mbps as the FCC now defines it)? I searched, but can't find it. I can find how many people have > 4 Mbps available, but not more than one choice.
On Aug 19, 2011, at 3:40 PM, Patrick W. Gilmore wrote:
[*] Anyone know what %-age of North American users have multiple choices for real broadband (e.g. > 1.5 Mbps, or even > 4 Mbps as the FCC now defines it)? I searched, but can't find it. I can find how many people have > 4 Mbps available, but not more than one choice.
I tried getting some data on that a few weeks back when reading the FCC Broadband Report. The best I could find was below which still leaves the users vs. markets unclear (which most likely skews the end result): Today, 290 million Americans—95% of the U.S. population—live in housing units30 with access to terrestrial, fixed broadband infrastructure capable of supporting actual download speeds of at least 4 Mbps.31 Of those, more than 80% live in markets with more than one provider capable of offering actual download speeds of at least 4 Mbps.32 Meanwhile, 14 million people in the United States living in 7 million housing units do not have access to terrestrial broadband infrastructure capable of this speed.33 Although housing units without access to terrestrial broadband capable of 4 Mbps download speeds exist throughout the country, they are more common in rural areas (see Exhibit 3-D).34 Above is from http://www.broadband.gov/plan/3-current-state-of-the-ecosystem/ which I got referenced from the FCC Broadband Report. That further references the OBI report on broadband availability gap which I never finished reading so more details might be in there. /nco
There might be something in the recent CITI report http://isoc-ny.org/p2/?p=2352 j :
On Aug 19, 2011, at 3:40 PM, Patrick W. Gilmore wrote:
[*] Anyone know what %-age of North American users have multiple choices for real broadband (e.g. > 1.5 Mbps, or even > 4 Mbps as the FCC now defines it)? I searched, but can't find it. I can find how many people have > 4 Mbps available, but not more than one choice.
--
Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com VP (Admin) - ISOC-NY - http://isoc-ny.org -------------------------------------------------------------- -
In a message written on Fri, Aug 19, 2011 at 05:40:43PM -0400, Patrick W. Gilmore wrote:
Yes, Above.Net broke the original peering-ratio fight that way. Thank you for that. Too bad it didn't last.
I forgot to write back when you first posted this, but the recent follow ons reminded me... While it is not a wide spread practice, I am 99.99% sure several networks are still using this method in private. I also know of at least two networks that have in the past deaggregated their large supernets to peers to affect a similar result. The interesting thing is that this is not done to elminate the peering ratio argument in the aggregate, but to make up for the geography of the two networks. Maybe one network insisted on peering in a city that the other only has a few customers in, so the link is mostly idle while others run too hot. Rather than provision more bandwidth they agree to meds or deaggregates to get the traffic to the lightly loaded link. The ratio in the aggregate is basically unchanged.
I've said it before, more times than I can count. Peering is a business tool, a means to an end. The goal, the 'end', of for-profit companies is to make a profit. Sounds obvious, but surprising how many people forget this. If peering with another network will increase your profit and you turn down the peering request, you should be fired. Your ego should not be substituted for business decisions.
I agree with your premise, and think everyone involved _thinks_ they are making the best business decision for their company. However I believe many folks are in fact making poor decisions for their company because they lack data. It's interesting to look at the first order costs of peering; the port, cross connect, and time to set it up. In my experience though the second and third order effects are far more important; how it affects your ratio with other peers, your customer's perception of your network performance, etc. These are much harder to measure, and near as I can tell virtually no network does. I've been asked many times how the government could step in and help, aka regulate peering to make it better. In thinking about all the things that could be mandated, the interactions that can be regulated, I see them generally as a univeral bad. With one, major, glaring exception.... Right now peering is all done in secret. You don't really know if your competitor is getting free peering, paying to peer, or getting paid to peer. You don't know if they are really insisting on the same locations with other parties they insist with for you. It's a perfect game of poker. This is in stark contrast to say, transit, where you can get prices from 20 people and compare. AboveNet also used to put graphs of almost all of its peering links online, in real time. A few peers flat out refused, but most were there. You could see where they peered, at what speed, and how much traffic was flowing. You could also calculate the ratio. I think the Government could bring a lot of sunshine to the party if it simply required all ISP's to do just that. Post a graph of every peering link or port to a public exchange, updated in real time. They all have the data internally, they are all collecting it for their own needs. You still wouldn't know if any money is changing hands, but imagine how it would change many of the interactions. Oh, you require me to hit 3Tb aggregate, when you have 20 peers that are less than half that? Really, you're going to require me to peer with you in Nome Alaska when you don't have any other peers at that location? It would also let ISP's make an informed decision about depeering events. The next time cogent splits from someone you can go look at the graphs on both sides, and make your own decision who was being more reasonable. It would also let you, the transit customer, evaluate the extent and free capacity to peers your ISP has before buying. I have yet to find a down side to this sort of sunshine. I'd wecome anyone who thinks its a bad idea to educate me. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
I have yet to find a down side to this sort of sunshine. I'd wecome anyone who thinks its a bad idea to educate me.
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary. Next time Cogent de-peers someone, customers do not care who was being more reasonable. They care that their links are broken. Customers do not care if links are out of ratio, they care if packets are dropped. Further there are companies who can push 75 Gbps on an 8x10 LAG for 12 hours without dropping a packet, while other companies cannot push 60 Gbps on that same link without some intermittent packet loss. Explaining to customers which are 'good' and which are 'bad' is an impossible task. Finally, if this were to happen, a lot of links would be quickly disconnected. Perhaps that could be considered a feature, but I am not at all certain it would be. In summary: I do not like gov't involvement, even if it is just to require some random set of data to be made public. -- TTFN, patrick On Aug 26, 2011, at 2:46 PM, Leo Bicknell wrote:
In a message written on Fri, Aug 19, 2011 at 05:40:43PM -0400, Patrick W. Gilmore wrote:
Yes, Above.Net broke the original peering-ratio fight that way. Thank you for that. Too bad it didn't last.
I forgot to write back when you first posted this, but the recent follow ons reminded me...
While it is not a wide spread practice, I am 99.99% sure several networks are still using this method in private. I also know of at least two networks that have in the past deaggregated their large supernets to peers to affect a similar result.
The interesting thing is that this is not done to elminate the peering ratio argument in the aggregate, but to make up for the geography of the two networks. Maybe one network insisted on peering in a city that the other only has a few customers in, so the link is mostly idle while others run too hot. Rather than provision more bandwidth they agree to meds or deaggregates to get the traffic to the lightly loaded link. The ratio in the aggregate is basically unchanged.
I've said it before, more times than I can count. Peering is a business tool, a means to an end. The goal, the 'end', of for-profit companies is to make a profit. Sounds obvious, but surprising how many people forget this. If peering with another network will increase your profit and you turn down the peering request, you should be fired. Your ego should not be substituted for business decisions.
I agree with your premise, and think everyone involved _thinks_ they are making the best business decision for their company. However I believe many folks are in fact making poor decisions for their company because they lack data. It's interesting to look at the first order costs of peering; the port, cross connect, and time to set it up. In my experience though the second and third order effects are far more important; how it affects your ratio with other peers, your customer's perception of your network performance, etc. These are much harder to measure, and near as I can tell virtually no network does.
I've been asked many times how the government could step in and help, aka regulate peering to make it better. In thinking about all the things that could be mandated, the interactions that can be regulated, I see them generally as a univeral bad. With one, major, glaring exception....
Right now peering is all done in secret. You don't really know if your competitor is getting free peering, paying to peer, or getting paid to peer. You don't know if they are really insisting on the same locations with other parties they insist with for you. It's a perfect game of poker. This is in stark contrast to say, transit, where you can get prices from 20 people and compare.
AboveNet also used to put graphs of almost all of its peering links online, in real time. A few peers flat out refused, but most were there. You could see where they peered, at what speed, and how much traffic was flowing. You could also calculate the ratio.
I think the Government could bring a lot of sunshine to the party if it simply required all ISP's to do just that. Post a graph of every peering link or port to a public exchange, updated in real time. They all have the data internally, they are all collecting it for their own needs. You still wouldn't know if any money is changing hands, but imagine how it would change many of the interactions. Oh, you require me to hit 3Tb aggregate, when you have 20 peers that are less than half that? Really, you're going to require me to peer with you in Nome Alaska when you don't have any other peers at that location?
It would also let ISP's make an informed decision about depeering events. The next time cogent splits from someone you can go look at the graphs on both sides, and make your own decision who was being more reasonable. It would also let you, the transit customer, evaluate the extent and free capacity to peers your ISP has before buying.
I have yet to find a down side to this sort of sunshine. I'd wecome anyone who thinks its a bad idea to educate me.
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Fri, 26 Aug 2011 21:32:00 EDT, "Patrick W. Gilmore" said:
Next time Cogent de-peers someone, customers do not care who was being more reasonable. They care that their links are broken.
Wouldn't that mostly affect people who are silly enough to single-home to a Tier-1 that gets involved in a peering dispute?
On Aug 26, 2011, at 11:10 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Fri, 26 Aug 2011 21:32:00 EDT, "Patrick W. Gilmore" said:
Next time Cogent de-peers someone, customers do not care who was being more reasonable. They care that their links are broken.
Wouldn't that mostly affect people who are silly enough to single-home to a Tier-1 that gets involved in a peering dispute?
Which $PROBLEM does not affect only those who are "silly enough" to not put mitigation tools for $PROBLEM? And the customers still don't care. They just care _that_ it affected them - at least during the problem. Although one can hope they care enough to change their behavior afterward. -- TTFN, patrick
On Sat, 27 Aug 2011 13:56:35 EDT, "Patrick W. Gilmore" said:
And the customers still don't care. They just care _that_ it affected them - at least during the problem. Although one can hope they care enough to change their behavior afterward.
And yet, people still single-home to Tier-1s. Go figure.
On 8/27/11 5:30 PM, Valdis.Kletnieks@vt.edu wrote:
On Sat, 27 Aug 2011 13:56:35 EDT, "Patrick W. Gilmore" said:
And the customers still don't care. They just care _that_ it affected them - at least during the problem. Although one can hope they care enough to change their behavior afterward.
And yet, people still single-home to Tier-1s. Go figure.
Because they care and don't care at the same time, although most don't want to be bothered to spend the money on a second provider. ~Seth
On Aug 27, 2011, at 8:30 PM, Valdis.Kletnieks@vt.edu wrote:
On Sat, 27 Aug 2011 13:56:35 EDT, "Patrick W. Gilmore" said:
And the customers still don't care. They just care _that_ it affected them - at least during the problem. Although one can hope they care enough to change their behavior afterward.
And yet, people still single-home to Tier-1s. Go figure.
People still single home to "tier-2s" that have a single transit provider - which is objectively worse. People still smoke too. Go figure. -- TTFN, patrick
In a message written on Fri, Aug 26, 2011 at 09:32:00PM -0400, Patrick W. Gilmore wrote:
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary.
It is, but it's also in the effort of allowing consumers to be informed. Why do we require food items to list the ingredients? Why do public companies have to disclose particular events to the SEC? Why do car companies have to list the MPG on their cars? The FCC is very interested right now in finding out if when ISP X says "your service is 16M down" you can get 16M. Peering has an impact on that. If I have a 16M service, and every single pering link they have is flatlined 10 hours a day I won't be able to get 16M down for those 10 hours. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Aug 28, 2011, at 3:51 PM, Leo Bicknell wrote:
In a message written on Fri, Aug 26, 2011 at 09:32:00PM -0400, Patrick W. Gilmore wrote:
Why is that any different than forcing businesses to explain which links are paid? Or any other internal data? Private businesses are private. Their relationships with other private businesses are private. Saying some data should be public while others are not is arbitrary.
It is, but it's also in the effort of allowing consumers to be informed. Why do we require food items to list the ingredients? Why do public companies have to disclose particular events to the SEC? Why do car companies have to list the MPG on their cars?
While I see some similarities, I do believe there are differences between food ingredients & financial disclosures and peering ports. I guess one could argue MPG is a comparable metric, but it's not really a useful number. (Perhaps why it is comparable? :)
The FCC is very interested right now in finding out if when ISP X says "your service is 16M down" you can get 16M. Peering has an impact on that. If I have a 16M service, and every single pering link they have is flatlined 10 hours a day I won't be able to get 16M down for those 10 hours.
Without disagreeing with your last point, we all know it will not be that simple. Suppose I have 100 peering partners and 3 are congested. Further assume those 3 are the three smallest peers I have, and refuse to upgrade because they do not have the cash (or whatever)? How many customers are going to be mad at me, even though it likely has zero effect on them? And what should I do about it? (If your answer is "explain to them why this is not a problem", then I submit you have never spoken to a pissed-off, irrational customer.) Some more questions: Are you going to limit this to just networks that sell consumer access? What will you do if the peering links are outside the US? Will some countries become havens for peering (like Lichtenstein is for banking)? How many links do you think will be dropped if this rule is put into place? Are transit ports required to be shown? Etc., etc. That said, I'm not going to Washington to lobby against it if this law is written. -- TTFN, patrick
What are thoughts on public disclosure limited to capacity constraints? There is ample business reason for making the terms of specific interconnects private. On the other hand, knowing definitively that {mon,du}opoly broadband provider A is running its connections to transit provider B hot could be in the public interest, and allow operators to make informed routing decisions. Bringing these metrics into the public light might also encourage operators to upgrade more responsibly, though this could be wishful thinking on my part. :-) (This is entirely food for thought, I've not yet formed any opinions.) -a
participants (7)
-
Adam Rothschild -
Joly MacFie -
Leo Bicknell -
Niclas Comstedt -
Patrick W. Gilmore -
Seth Mattinen -
Valdis.Kletnieks＠vt.edu