I'd have to agree with Brian. There is no simple answer to this one... If the ultimate cause is the abuse of bandwidth, I can understand this... BUT if the underlying motive is to squash competition then shame on you! -----Original Message----- From: Rettke, Brian [mailto:Brian.Rettke@cableone.biz] Sent: Monday, November 29, 2010 4:41 PM To: Patrick W. Gilmore; NANOG list Subject: RE: Level 3 Communications Issues Statement Concerning Comcast's Actions Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures. I can understand both the side taken by Comcast, and the side of the content provider, but I don't think it's as simple as the slogans spewed out regarding "Net Neutrality", which has become so misused and abused as a term that I don't think it has any credulous value remaining. I'm hoping that there is an eventual meeting of the minds wherein some sort of collaboration takes place. If this gets additional government regulations I fear no one will like the result. Sincerely, Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services -----Original Message----- From: Patrick W. Gilmore [mailto:patrick@ianai.net] Sent: Monday, November 29, 2010 3:28 PM To: NANOG list Subject: Level 3 Communications Issues Statement Concerning Comcast's Actions <http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp> I understand that politics is off-topic, but this policy affects operational aspects of the 'Net. Just to be clear, L3 is saying content providers should not have to pay to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on this list are happy to hear L3's change of heart and will be applying for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are competing and L3 is putting up a "toll booth" on the Internet, would they?) -- TTFN, patrick

It may have something to do with that Level3 is now hosting all the streaming content for Netflixs. Cheers Ryan -----Original Message----- From: Thomas Donnelly [mailto:tad1214@gmail.com] Sent: Monday, November 29, 2010 5:52 PM To: Rettke, Brian; Patrick W. Gilmore; NANOG list; Guerra, Ruben Subject: Re: Level 3 Communications Issues Statement Concerning Comcast'sActions "On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast's customers who request such content." If the issue is bandwidth, then why not charge for bandwidth? Picking a specific service says we are trying to squash the competition. On Mon, 29 Nov 2010 16:48:06 -0600, Guerra, Ruben <Ruben.Guerra@arrisi.com> wrote:

I'd have to agree with Brian. There is no simple answer to this one...

If the ultimate cause is the abuse of bandwidth, I can understand this... BUT if the underlying motive is to squash competition then shame on you!

-----Original Message----- From: Rettke, Brian [mailto:Brian.Rettke@cableone.biz] Sent: Monday, November 29, 2010 4:41 PM To: Patrick W. Gilmore; NANOG list Subject: RE: Level 3 Communications Issues Statement Concerning Comcast's Actions

Essentially, the question is who has to pay for the infrastructure to

support the bandwidth requirements of all of these new and booming streaming ventures. I can understand both the side taken by Comcast, and the side of the content provider, but I don't think it's as simple as

the slogans spewed out regarding "Net Neutrality", which has become so

misused and abused as a term that I don't think it has any credulous value remaining.

I'm hoping that there is an eventual meeting of the minds wherein some

sort of collaboration takes place. If this gets additional government

regulations I fear no one will like the result.

Sincerely,

Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services

-----Original Message----- From: Patrick W. Gilmore [mailto:patrick@ianai.net] Sent: Monday, November 29, 2010 3:28 PM To: NANOG list Subject: Level 3 Communications Issues Statement Concerning Comcast's

Actions

<http://www.marketwatch.com/story/level-3-communications-issues-statemen t-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.

Just to be clear, L3 is saying content providers should not have to

pay

to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on

this list are happy to hear L3's change of heart and will be applying

for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are

competing and L3 is putting up a "toll booth" on the Internet, would they?)

-- TTFN, patrick

-- Using Opera's revolutionary email client: http://www.opera.com/mail/

[Changed long CC list to BCC] On 12/2/10 12:49 AM, Frank Bulk wrote:

George Ou touches on a similar point at the end of his article: http://www.digitalsociety.org/2010/11/level-3-outbid-akamai-on-netflix-by-re selling-stolen-bandwidth/

The Ou article makes no sense at all! It's based on the premise that Level 3 and Comcast are peering, and that traffic should be symmetric. Everywhere else, the articles and pundits indicate that Comcast is a transit customer of Level 3. All actual network operators know that traffic isn't symmetric! Ou's hit piece reads more like a pseudo-libertarian rant. In fact, other Ou posts listed there have titles that read like an ultra-conservative cum social-conservative rant: * Wrong On The Internet » Another Net Neutrality ‘violation’ debunked * Why Viacom and others justified in blocking Google TV * Wrong On The Internet » Genachowski pushing ahead with Net Neutrality during lame duck * Google hypocrisy on content blocking * Hijacking the Internet is trivial today You have to consider the source. If Ou doesn't understand contracts, peering, and/or transit, just take his posts with a grain of salt.

Then ignore Ou's post and focus on the point I tried to make: that Level3 has a vested interest in making sure the Comcast users have a good Netflix experience. =) Frank -----Original Message----- From: William Allen Simpson [mailto:william.allen.simpson@gmail.com] Sent: Thursday, December 02, 2010 8:28 AM To: NANOG list Subject: Re: Level 3 Communications Issues Statement Concerning Comcast's Actions [Changed long CC list to BCC] On 12/2/10 12:49 AM, Frank Bulk wrote:

George Ou touches on a similar point at the end of his article:

http://www.digitalsociety.org/2010/11/level-3-outbid-akamai-on-netflix-by-re

selling-stolen-bandwidth/

The Ou article makes no sense at all! It's based on the premise that Level 3 and Comcast are peering, and that traffic should be symmetric. Everywhere else, the articles and pundits indicate that Comcast is a transit customer of Level 3. All actual network operators know that traffic isn't symmetric! Ou's hit piece reads more like a pseudo-libertarian rant. In fact, other Ou posts listed there have titles that read like an ultra-conservative cum social-conservative rant: * Wrong On The Internet > Another Net Neutrality 'violation' debunked * Why Viacom and others justified in blocking Google TV * Wrong On The Internet > Genachowski pushing ahead with Net Neutrality during lame duck * Google hypocrisy on content blocking * Hijacking the Internet is trivial today You have to consider the source. If Ou doesn't understand contracts, peering, and/or transit, just take his posts with a grain of salt.

On Thu, Dec 2, 2010 at 5:10 PM, Matthew Petach <mpetach@netflight.com> wrote:

fair game for reverse billing.  If it does, it's going to completely eliminate "transit" as a commercial offering; instead, we'll all be stuck doing settlements in every direction for traffic...and that's just *way* too much paperwork.  ^_^;

oh! that's the LD network.. that worked out so darned well, can we do it again? and can we have the ITU manage it for us? please? please? please? :) -chris

On Thu, 2 Dec 2010, Matthew Petach wrote:

So, one wonders why Level3 didn't just say "look, I'm the vendor, you're the customer; the customer pays the vendor for service, period.

There's no wonder here at all. It's not at all hard to imagine the conversation: Level3: I'm the vendor, you're the customer; the customer pays the vendor for service, period. Comcast: Okay vendor, we aren't going to pay you any more. Go ahead and shut down our circuits. We'll go ahead and pay you the early termination penalties or whatever, but keep in mind that the Level3 network has no way to reach Comcast through any other path thanks to our clever routing tricks, so your customers, including Netflix, won't be able to reach our customers. Level3: But, but, but, you are the customer! Comcast: Go ahead, shut us down, we dare you. Perhaps you'll want to find someone to buy transit from that CAN reach us? I have to say, it's not that hard to imagine because it's exactly what I would have done in their position. If I were them, I would then proceed to do the exact same thing to every other "vendor" that they have until they are a transit free network. Then I might even start demanding payments from my peers. Why not? Comcast has all the power. It's exactly what the government has incentivized them to do by allowing them to have all of those cable monopolies around the country. That's right, government is the real problem here, Comcast is simply acting in their own best interest. Now where did I put that CMCS stock... -- Brandon Ross AIM: BrandonNRoss ICQ: 2269442 Skype: brandonross Yahoo: BrandonNRoss

On 11/29/2010 14:49, Aaron Wendel wrote:

A customer pays them for access to the Internet. If that access demands more infrastructure then Comcast needs to build out the infrastructure and pass on the costs to the customers demanding it.

But then Comcast might have to raise prices on their customers. This way they don't. ~Seth

On 11/29/2010 4:49 PM, Aaron Wendel wrote:

A customer pays them for access to the Internet. If that access demands more infrastructure then Comcast needs to build out the infrastructure and pass on the costs to the customers demanding it.

I agree. This type of maneuver is no different than ESPN3 charging the ISP for the ISP customers to access the content. Both are unscalable models that threaten the foundation of an open Internet. As an ISP, I could care less what is in the packets my customers send and receive. The exception to this, of course, is malicious packets but they keep refusing to set the evil bit. Jack

Netflix pays someone to get access to "the internet" and that someone has some sort of relationship with Comcast, or gets to Comcast through a third party who has that relationship. No one is getting anything for free. I don't think it's unreasonable to expect customers to bear the cost of their provider doing business. If that business calls for the buildout of additional infrastructure to remain competitive then so be it. Comcast customers pay their provider, Netflix pays its provider. I think what this really boils down to is an effect of shoddy marketing. Access providers want to offer "unlimited" everything and don't want to have to go back to their customer base and say, "oh, sorry, we didn't really mean unlimited. We didn't think you'd really use that much." So they are looking for ways of making up for the increased costs without having to look like idiots to their customers. My problem is, what happens if this becomes the new model? What if Comcast comes to me and says, "Oh, we've noticed X Mbits originating from your network coming through ours. Here's the bill of $X per bit." What happens when I counter with, "Ok, and I see X bits originating from your network. Here's my bill, too." Do they agree to an exchange of money for an exchange of bits or do I get an "F you. Pay your bill to us and we're not giving you crap." Aaron From: Rettke, Brian [mailto:Brian.Rettke@cableone.biz] Sent: Monday, November 29, 2010 5:21 PM To: Jack Bates; Aaron Wendel Cc: 'Patrick W. Gilmore'; 'NANOG list' Subject: RE: Level 3 Communications Issues Statement Concerning Comcast's Actions On 11/29/2010 4:49 PM, Aaron Wendel wrote:

A customer pays them for access to the Internet. If that access demands more infrastructure then Comcast needs to build out the infrastructure and pass on the costs to the customers demanding it.

I'd change this to "A customer pays for SHARED access to the Internet." Unless your customer is paying for a direct fiber or internet circuit (~$500 - $10,000 per month) they aren't paying for independent and sole access to the internet. It's another term that I think has lost its actual meaning, "Unlimited access." I don't have a problem, as a customer or as a Service Provider, passing along the bill to the top 5% that are using a disproportionate amount of bandwidth. I can see the Internet reaching a fair-use model, as opposed to a free-use model that is unsustainable, as was previously said. Here's one specific example I can think of to discuss: Netflix uses about a third of Internet bandwidth, in some cases going over the HTTP traffic use for most customers. Netflix charges customers a fee to use their service, but they don't pay the providers required to supply the bandwidth for the customer leg. I don't think ISPs charging Netflix is a sustainable model either. A mutual endeavor involving shared interconnect costs and intelligent placement of proxies would be something I could think of to make the process beneficial for all parties. The end goal would be that the "Shared Media Customer" has no idea what we are doing, but does not see performance degradation in their HTTP or Netflix traffic, and that it does not pass along additional cost to them. After all, to both Netflix and the ISP, it is in their best interests to keep that customer a happy and paying customer. Sincerely, Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services -----Original Message----- From: Jack Bates [mailto:jbates@brightok.net] Sent: Monday, November 29, 2010 4:11 PM To: Aaron Wendel Cc: Rettke, Brian; 'Patrick W. Gilmore'; 'NANOG list' Subject: Re: Level 3 Communications Issues Statement Concerning Comcast's Actions On 11/29/2010 4:49 PM, Aaron Wendel wrote:

A customer pays them for access to the Internet. If that access demands more infrastructure then Comcast needs to build out the infrastructure and pass on the costs to the customers demanding it.

I agree. This type of maneuver is no different than ESPN3 charging the ISP for the ISP customers to access the content. Both are unscalable models that threaten the foundation of an open Internet. As an ISP, I could care less what is in the packets my customers send and receive. The exception to this, of course, is malicious packets but they keep refusing to set the evil bit. Jack _____ No virus found in this message. Checked by AVG - www.avg.com Version: 10.0.1170 / Virus Database: 426/3287 - Release Date: 11/29/10

You and I both know that. I'll bet the vast majority of comcast customers don't. Sent via DROID on Verizon Wireless -----Original message----- From: William Warren <hescominsoon@emmanuelcomputerconsulting.com> To: 'NANOG list' <nanog@nanog.org> Sent: Tue, Nov 30, 2010 01:24:40 GMT+00:00 Subject: Re: Level 3 Communications Issues Statement Concerning Comcast's Actions On 11/29/2010 6:45 PM, Aaron Wendel wrote:

I think what this really boils down to is an effect of shoddy marketing. Access providers want to offer "unlimited" everything and don't want to

have

to go back to their customer base and say, "oh, sorry, we didn't really mean unlimited. We didn't think you'd really use that much." So they are looking for ways of making up for the increased costs without having to look like idiots to their customers. Unlimited access is already NOT unlimited access. A transfer cap isn't unlimited..while Comcast has a generous cap..it's still a transfer cap.

My problem is, what happens if this becomes the new model? What if Comcast comes to me and says, "Oh, we've noticed X Mbits originating from your network coming through ours. Here's the bill of $X per bit." What happens when I counter with, "Ok, and I see X bits originating from your network. Here's my bill, too." Do they agree to an exchange of money for an exchange of bits or do I get an "F you. Pay your bill to us and we're not giving you crap."

I've read through the entire thread thus far, and there are several very interesting points. I'd like to know more about the Australian experiment? But there were a couple of disparate comments that seem highly related, so I'll reply to them jointly here: On 11/30/10 2:59 AM, JC Dill wrote:

What is happening now between L3 and Comcast also reminds me of the dial-tone settlement deals in the 1990s. The big telcos thought they could push small telcos out by making it more expensive to place calls (paying a fee to the telco that "terminates" the call) and less expensive to receive calls (receiving the termination fee). They mistakenly thought the startup telcos would go after consumers (who typically place more calls than they receive) and they didn't think about startup telcos going after ISP dial-up services (which receive more calls than they place) and then being forced to pay those startups settlement fees for all the calls their consumer customers made into the startup telco's ISP customer's modem banks.

But I remember what happened next. BellSouth refused to pay their settlements. The CLECs sued and went bankrupt. BellSouth had deeper pockets and more lawyers.

We don't have an interstate telephone settlement system or PUC to "decide" what the rules will be for settlements between content providers and eyeball providers. I believe that in the end it will come down to market forces and which group can better marshal customer angst to their side when packets don't flow freely between these two types of networks.

Maybe. But I'm hoping the consumer angst gives us a better FCC. The "market" hasn't worked before, and isn't working in this case. So, maybe there isn't a "market" after all.... On 11/30/10 2:47 AM, Kevin Blackham wrote:

I'm not convinced. Either I'm calculating something wrong, or greed is at work.

Greed. Reminder: Comcast drastically raised their rates a few years back, saying to local cable commissions that they needed to "invest" in digital infrastructure. Instead, they took the massive profits and invested in NBC/Universal. When a cable "node" is an entire neighborhood of 500+ homes, because Comcast never bothered to split the nodes down to a reasonable networking size (as opposed to CATV-sized), then it's a Comcast greed problem.... A half year ago or so, talking with a Google manager about a certain fiber project, we ended up arguing about the size of cable nodes. He seemed to think everywhere was like Mountain View. I was trying not to embarrass him; just let it stand at -- as you drive, you don't look overhead at the cable infrastructure much, do you? (He admitted he doesn't.) On 11/29/10 11:27 PM, Jared Mauch wrote:

The issue here is cost of infrastructure. The last mile generally is more valuable than the long-distance part. Everyone can build a nationwide network for a nominal amount of money. All the carriers can provide circuits at the same IXPs where you can public/private peer. The question does become, who is in those smaller and mid-markets. Not everyone is going to build fiber in Akron, Eugene, nor Madison. It gets even more interesting if you look at what happened with Fairpoint in the northeast IMHO. Verizon realized they would not make money there and sold it off. The promises and costs consumed them and forced bankruptcy.

I'm not saying that will happen to Comcast, but it may cause them to divest the unprofitable parts as well, leaving some parts of the country worse-off than we would be today.

Or in this case, invest in something else more profitable, NBC/Universal; and then try to leverage their customer base to gouge their CDN competitors. I'd like to see Level 3 pull a Disney/ABC or a Murdock/Fox, and publicly announce that they expect Comcast to share *their* revenue. And be willing to pull the plug! (Admittedly, I thought Disney/ABC and Murdock/Fox are evil, too. That model was only reasonable as the CATV channels had no advertising. All we have left now is Turner Classic Movies. A pox on *all* their houses!) It's really time for some anti-trust legislation/regulation. The last mile market has failed.

Well, I don't work for the NBN, but I do live here and follow the politics with interest. So far the 'experiment' is on track. The political parties who support the NBN are the majority by a slim margin (2 or 3 seats) and the project seems to be going forward. Most recently legislation passed that creates the NBN as a corporation among other things. If you're truly interested: http://australianpolitics.com/downloads/10-11-24_nbn-co-business-case-summar... jy On 01/12/2010, at 12:56 AM, William Allen Simpson wrote:

I've read through the entire thread thus far, and there are several very interesting points. I'd like to know more about the Australian experiment?

But there were a couple of disparate comments that seem highly related, so I'll reply to them jointly here:

On 11/30/10 2:59 AM, JC Dill wrote:

...

What is happening now between L3 and Comcast also reminds me of the dial-tone settlement deals in the 1990s. The big telcos thought they could push small telcos out by making it more expensive to place calls (paying a fee to the telco that "terminates" the call) and less expensive to receive calls (receiving the termination fee). They mistakenly thought the startup telcos would go after consumers (who typically place more calls than they receive) and they didn't think about startup telcos going after ISP dial-up services (which receive more calls than they place) and then being forced to pay those startups settlement fees for all the calls their consumer customers made into the startup telco's ISP customer's modem banks.

But I remember what happened next. BellSouth refused to pay their settlements. The CLECs sued and went bankrupt. BellSouth had deeper pockets and more lawyers.

...

We don't have an interstate telephone settlement system or PUC to "decide" what the rules will be for settlements between content providers and eyeball providers. I believe that in the end it will come down to market forces and which group can better marshal customer angst to their side when packets don't flow freely between these two types of networks.

Maybe. But I'm hoping the consumer angst gives us a better FCC. The "market" hasn't worked before, and isn't working in this case. So, maybe there isn't a "market" after all....

On 11/30/10 2:47 AM, Kevin Blackham wrote:

...

I'm not convinced. Either I'm calculating something wrong, or greed is at work.

Greed.

Reminder: Comcast drastically raised their rates a few years back, saying to local cable commissions that they needed to "invest" in digital infrastructure. Instead, they took the massive profits and invested in NBC/Universal.

When a cable "node" is an entire neighborhood of 500+ homes, because Comcast never bothered to split the nodes down to a reasonable networking size (as opposed to CATV-sized), then it's a Comcast greed problem....

A half year ago or so, talking with a Google manager about a certain fiber project, we ended up arguing about the size of cable nodes. He seemed to think everywhere was like Mountain View. I was trying not to embarrass him; just let it stand at -- as you drive, you don't look overhead at the cable infrastructure much, do you? (He admitted he doesn't.)

On 11/29/10 11:27 PM, Jared Mauch wrote:

...

The issue here is cost of infrastructure. The last mile generally is more valuable than the long-distance part. Everyone can build a nationwide network for a nominal amount of money. All the carriers can provide circuits at the same IXPs where you can public/private peer. The question does become, who is in those smaller and mid-markets. Not everyone is going to build fiber in Akron, Eugene, nor Madison. It gets even more interesting if you look at what happened with Fairpoint in the northeast IMHO. Verizon realized they would not make money there and sold it off. The promises and costs consumed them and forced bankruptcy.

I'm not saying that will happen to Comcast, but it may cause them to divest the unprofitable parts as well, leaving some parts of the country worse-off than we would be today.

Or in this case, invest in something else more profitable, NBC/Universal; and then try to leverage their customer base to gouge their CDN competitors.

I'd like to see Level 3 pull a Disney/ABC or a Murdock/Fox, and publicly announce that they expect Comcast to share *their* revenue. And be willing to pull the plug!

(Admittedly, I thought Disney/ABC and Murdock/Fox are evil, too. That model was only reasonable as the CATV channels had no advertising. All we have left now is Turner Classic Movies. A pox on *all* their houses!)

It's really time for some anti-trust legislation/regulation. The last mile market has failed.

* Valdis Kletnieks:

On Mon, 29 Nov 2010 17:11:18 CST, Jack Bates said:

...

I agree. This type of maneuver is no different than ESPN3 charging the ISP for the ISP customers to access the content. Both are unscalable models that threaten the foundation of an open Internet.

Oddly enough, cable channels like ESPN asking for a per-subscriber fee from cable delivery networks like Comcast has been a mostly-scalable model for the cable-TV arena for three or four decades now....

And your TV bandwidth usage doesn't count towards your monthly limit, either. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99

On 11/29/2010 14:40, Rettke, Brian wrote:

Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures. I can understand both the side taken by Comcast, and the side of the content provider, but I don't think it's as simple as the slogans spewed out regarding "Net Neutrality", which has become so misused and abused as a term that I don't think it has any credulous value remaining.

Is Level3 the content provider though? Or did Comcast just decide they don't want to do the settlement free peering thing anymore for traffic transiting via Level 3? ~Seth

From: Rettke, Brian Sent: Monday, November 29, 2010 2:41 PM To: Patrick W. Gilmore; NANOG list Subject: RE: Level 3 Communications Issues Statement Concerning Comcast'sActions

Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures.

From a cultural standpoint, we in the US are used to a model where the sender pays the postage for unsolicited content and the requestor pays the shipping for requested content. So asking an ad network to pay Comcast for shipping their ads is valid but in a request where the end user specifically asked for a movie, the user should be expected to pay for that. What Comcast appears to be doing is subsidizing "flat rate" customer rates by charging the providers "metered" access (assuming the fee charged the provider isn't also a "flat rate"). If so, that really isn't fair because:

1. The provider has no control over the size of or number of requests that are made. The provider is essentially agreeing to an unknown quantity in advance. 2. There is no way to ensure that a request is legitimate and not a request generated simply to generate revenue (sort of like click fraud ... stream fraud). 3. It opens the provider up to a "denial of sustainability" attack where a bot net requests many copies of various streams, sends them to the equivalent of /dev/null and the provider is presented with a huge bill. 4. The only way a provider could mitigate increases in fees is to meter access causing a sub-optimal user experience. Shouldn't Level3 turn around and charge Comcast for distributing NBC/Universal content? The whole thing is like a movie theater charging the studios to show movies while selling tickets to the public to watch them. Actualy, it is like Universal opening their own movie theaters and charging competing studios to show their movies while still charging the public to see them. Comcast is simply imposing a tariff on competing content. If I were level3, I would have denied the request. Customers on Comcast would then discover they have sub-optimal Internet service and gone to a competitor (AT&T Uverse or Verizon FIOS, for example). As owner of NBC Universal, Comcast is a producer as well as a distributor of content. That puts them in direct competition with other producers regardless of the distributor. Level3 should deny the request and Comcast users will have "Internet" access instead of Internet access. Comcast doesn't have the captive audience they once had in many places and when customers discover their choices are limited when they choose Comcast, they will go elsewhere. I hope Level3's acquiescence is temporary or the FTC puts a stop to it. It is sort of like FedEx owning the freeway and charging UPS to use it.

On Nov 29, 2010, at 3:42 PM, George Bonser wrote:

...

From: Rettke, Brian Sent: Monday, November 29, 2010 2:41 PM To: Patrick W. Gilmore; NANOG list Subject: RE: Level 3 Communications Issues Statement Concerning Comcast'sActions

Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures.

...

From a cultural standpoint, we in the US are used to a model where the sender pays the postage for unsolicited content and the requestor pays the shipping for requested content. So asking an ad network to pay Comcast for shipping their ads is valid but in a request where the end user specifically asked for a movie, the user should be expected to pay for that. What Comcast appears to be doing is subsidizing "flat rate" customer rates by charging the providers "metered" access (assuming the fee charged the provider isn't also a "flat rate"). If so, that really isn't fair because:

On the internet, how would you tell these apart? First, as I have said in a previous message, I think Comcast's actions here are deplorable. However... Generally speaking, most (legitimate) ads are delivered as content on web pages at various sites. As a result, they are "requested" specifically by the user's browser right alongside all the other content the end user actually asked for. Perhaps part of the problem here is that Comcast comes from a culture where advertisers pay to deliver their advertising which is delivered directly alongside the content that the consumer actually wants. Perhaps Comcast is having a hard time realizing that the internet is not broadcast television, or, perhaps they think they can convert the rest of us to thinking of the internet along those lines. Certainly the ad-delivery method on the web is actually more in line with that of broadcast television than it is in line with postal delivery of advertising.

1. The provider has no control over the size of or number of requests that are made. The provider is essentially agreeing to an unknown quantity in advance. 2. There is no way to ensure that a request is legitimate and not a request generated simply to generate revenue (sort of like click fraud ... stream fraud). 3. It opens the provider up to a "denial of sustainability" attack where a bot net requests many copies of various streams, sends them to the equivalent of /dev/null and the provider is presented with a huge bill. 4. The only way a provider could mitigate increases in fees is to meter access causing a sub-optimal user experience.

All valid points. In fact, this points out that while the approach Comcast is taking appears to resemble the "sponsor pays" model of broadcast television, the metered aspect is a major difference which changes the game significantly. I suspect that Comcast is operating from the assumption that every request for content to the provider is money flowing to the provider so that the money collected by Comcast for the corresponding traffic is merely "their cut" of the revenue. Of course we all know this is a flawed assumption on Comcast's part, but, if you look at it from that mind set the belief can be rationalized, even if it is misguided.

Shouldn't Level3 turn around and charge Comcast for distributing NBC/Universal content?

Seems like a reasonable response, but, hopefully Level 3 will not surrender the high ground since they have it for the moment.

The whole thing is like a movie theater charging the studios to show movies while selling tickets to the public to watch them. Actualy, it is like Universal opening their own movie theaters and charging competing studios to show their movies while still charging the public to see them. Comcast is simply imposing a tariff on competing content. If I were level3, I would have denied the request. Customers on Comcast would then discover they have sub-optimal Internet service and gone to a competitor (AT&T Uverse or Verizon FIOS, for example).

That's great in areas where that is an option. There are many areas served by Comcast where Uverse, FIOS, etc. are not available. I live in San Jose, California, the so called Capitol of Silicon Valley. In my neighborhood, Comcast is the only cost effective alternative for more than 1.5mbps/384kbps. (A residential DS-3 circuit is not cost effective).

As owner of NBC Universal, Comcast is a producer as well as a distributor of content. That puts them in direct competition with other producers regardless of the distributor. Level3 should deny the request and Comcast users will have "Internet" access instead of Internet access. Comcast doesn't have the captive audience they once had in many places and when customers discover their choices are limited when they choose Comcast, they will go elsewhere.

I hope Level3's acquiescence is temporary or the FTC puts a stop to it. It is sort of like FedEx owning the freeway and charging UPS to use it.

I hope so, too. Unfortunately, Comcast is very experienced at these kinds of bullying tactics and they have used them very successfully with broadcasters and others for many years. If there is hope, it likely lies with the regulators. Owen

http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html On Mon, Nov 29, 2010 at 7:51 PM, Dave CROCKER <dhc2@dcrocker.net> wrote:

On 11/29/2010 2:40 PM, Rettke, Brian wrote:

...

Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures. I can understand both the side taken by Comcast, and the side of the content provider, but I don't think it's as simple as the slogans spewed out regarding "Net Neutrality", which has become so misused and abused as a term that I don't think it has any credulous value remaining.

I find it helpful to distinguish "participant neutrality" from "service neutrality". The first says that you and I pay the same rate. The second says the my email costs the same as my voip.

As described, it appears that Level3 is being singled out, which makes for participant non-neutrality. On the other hand, if Comcast were charging itself for xfinity traffic, this might qualify as service non-neutrality (assuming there is a plausible meaning to "charging itself"...

d/

--

Dave Crocker Brandenburg InternetWorking bbiw.net

On Mon, 2010-11-29 at 20:02 -0500, Bret Clark wrote:

On 11/29/2010 07:55 PM, Ren Provo wrote:

...

http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html

On Mon, Nov 29, 2010 at 7:51 PM, Dave CROCKER<dhc2@dcrocker.net> wrote:

Okay's let's say L3 gives in to Comcast and pays them.

L3 gave into Comcast and paid them already according to a press release they issued. William.

On Mon, 29 Nov 2010, Bret Clark wrote:

Okay's let's say L3 gives in to Comcast and pays them. L3 then turns around and charges us (providers) more to cover the additional money they have to pay Comcast now.

Why don't you, and other providers, demand L3 give you the same settlement-free peering they want from Comcast? Then you won't need to pay L3 anything because of L3's deal with Comcast? Oh, what? You say that L3 won't peer with you on a settlement-free basis, L3 wants you to pay them? Or why don't you build a network to places that Comcast peers at; and bypass L3 completely and negotiate a peering relationship directly with Comcast? Peering battles are so much fun because every side can think up all sorts of reasons why they should or should not pay or be paid. There is no right or wrong answer.

On Tue, 30 Nov 2010, Bret Clark wrote:

...

Or why don't you build a network to places that Comcast peers at; and bypass L3 completely and negotiate a peering relationship directly with Comcast?

We tried Comcast wouldn't peer with us because they considered us a compeititor.

Seriously this has nothing to do with L3 but more with Netflix...it's clear that the Netflix business model is eating into Comcast VoD business and so they are strong arming other providers to affect Netflix's business model. But as others have stated what would happen if Comcast starts coming after every service provider's hosting services that Comcast doesn't like?

Comcast claims it offered Level3 the same CDN deal it has with other Netflix CDN competitors. Level3 didn't want the same deal. According to Comcast, Level 3 wants a 'special' deal. Of course, Level 3 spins it the other way and claims that it offered Comcast a settlement-free deal, but Comcast didn't want it now. Level 3 has been trying to strong arm other providers for a decade. MCI, Sprint, ANS, UUNET, and others lost in history, have been doing it even longer. As BBN showed with the WORLDCOM/MCI/UUNET merger, now is an opportune time for Level 3 to obtain concessions from Comcast. Its always fun watching one long time toll-booth operator (Level 3) complain when someone new sets up another toll-booth (Comcast).

Does "build it, and they will come" now become a business liability? Yes, a business should stake out appropriate agreements in order to ensure relevant product delivery, but they also shouldn't be punished (for lack of a better word) for not foreseeing the success of said product- perhaps a "share the wealth" mentality is in effect here?

On Nov 30, 2010, at 11:47 AM, George Bonser wrote:

...

Seriously this has nothing to do with L3 but more with Netflix...it's clear that the Netflix business model is eating into Comcast VoD business and so they are strong arming other providers to affect Netflix's business model. But as others have stated what would happen if Comcast starts coming after every service provider's hosting services that Comcast doesn't like?

Bret

I think it has more to do with this:

http://www.insidebayarea.com/oaklandtribune/localnews/ci_16526623?source =rss

The cable companies are losing subs at an increasing rate. People are using them for internet and not buying the television programming. If Comcast can't collect from their "cord-cutting" customers, then they will collect from the content providers whose products their customers are using.

I have been told that "cutting the cord" are the 3 most frightening words in the cable industry today. IMO, they need to see that they are service providers, not gate-keepers. I am afraid that the Level-3 response here may help them to cling on to the legacy business model and avoid facing the new situation before them. Regards Marshall

In summary: Level3 is crying foul while their CDN competitors have quietly bought into Comcast's racket. I applaud Level3 for calling attention to this matter. Owen (Speaking strictly for myself) On Nov 29, 2010, at 4:55 PM, Ren Provo wrote:

http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html

On Mon, Nov 29, 2010 at 7:51 PM, Dave CROCKER <dhc2@dcrocker.net> wrote:

...

On 11/29/2010 2:40 PM, Rettke, Brian wrote:

...

Essentially, the question is who has to pay for the infrastructure to support the bandwidth requirements of all of these new and booming streaming ventures. I can understand both the side taken by Comcast, and the side of the content provider, but I don't think it's as simple as the slogans spewed out regarding "Net Neutrality", which has become so misused and abused as a term that I don't think it has any credulous value remaining.

I find it helpful to distinguish "participant neutrality" from "service neutrality". The first says that you and I pay the same rate. The second says the my email costs the same as my voip.

As described, it appears that Level3 is being singled out, which makes for participant non-neutrality. On the other hand, if Comcast were charging itself for xfinity traffic, this might qualify as service non-neutrality (assuming there is a plausible meaning to "charging itself"...

d/

--

Dave Crocker Brandenburg InternetWorking bbiw.net

On 11/29/2010 5:46 PM, Mark Wall wrote:

Between the lines: Comcast wants to end mutual peering agreements (due to: ratios, politics , greed) but we are going to spin it due to net neutrality making it main stream media and hoping we can get comcast clients to complain...

Not the worse angle we've seen

...

I think Karl Denninger has this one called right: http://market-ticker.org/post=173522

On Nov 29, 2010, at 15:57, William Warren <hescominsoon@emmanuelcomputerconsulting.com> wrote:

I think Karl Denninger has this one called right: http://market-ticker.org/post=173522

I don't think so. Let's do a little math exercise: Comcast charges me $75/mo for my pipe, but let's discount that for bundling, promos and lower tier services. $30-40 avg ok? For that money I get 250GB a month. Let's assume I actually use it - which I never do, even with Netflix, other VOD, and many habits common to eyeballs - but for the sake of a number to work with, I do. That's less than 1Mbps average per month. I'm not factoring in deviation from avg to peak, so I am going to assume 1Mbps per sub is peak per sub and 250GB is not the average for the user base. That is at least $40/Mbps paid by the eyeballs... or if I am very wrong, $20/Mbps. This is unsustainable and requires income at both ends for a healthy business model? I'm not convinced. Either I'm calculating something wrong, or greed is at work.

On 30/11/2010, at 6:17 PM, Kevin Blackham wrote:

On Nov 29, 2010, at 15:57, William Warren <hescominsoon@emmanuelcomputerconsulting.com> wrote:

...

I think Karl Denninger has this one called right: http://market-ticker.org/post=173522

I don't think so. Let's do a little math exercise:

Comcast charges me $75/mo for my pipe, but let's discount that for bundling, promos and lower tier services. $30-40 avg ok?

For that money I get 250GB a month. Let's assume I actually use it - which I never do, even with Netflix, other VOD, and many habits common to eyeballs - but for the sake of a number to work with, I do. That's less than 1Mbps average per month. I'm not factoring in deviation from avg to peak, so I am going to assume 1Mbps per sub is peak per sub and 250GB is not the average for the user base.

Average is easy - but the "not factoring in the deviation from avg to peak" is basically ignoring the actual meat of the problem. The human being using a network wants a quite large instantaneous peak during, say, 5pm to 11pm week nights. If you're doing network dimensioning and look at the 5min/avg and assume that's enough then you're wrong and will see packet loss. The more customers the smoother the curve, but at the far edge of the network near the last mile where aggregation starts the difference in cost to cope with this starts to add up when you start doing it cookie cutter style over hundreds/thousands or more sites. Especially if these sites are remote and have power/size restrictions. MMC

On 11/29/2010 15:24, Phil Bedard wrote:

Is L3 hosting content for Netflix? Netflix has become a large source of traffic going to end users. L3 likely could have held out on this one if the content they were hosting is valuable enough to Comcast's customers, but maybe what Comcast was asking for wasn't much in the grand scheme of things.

Obviously someone has to pay for the access infrastructure and Comcast would much rather get the content provider to pay for it versus passing it along to their customers. I think they probably just took a stab and L3 complied.

My take on this is that settlement free peering only remains free as long as it is beneficial to both sides, i.e. equal amounts of traffic exchanged. If it becomes wildly lopsided in one direction, then it becomes more like paying for transit. Perhaps this is the "cost" of acquisitions and mergers, like acquiring a CDN product that dramatically screws with your peering ratios. ~Seth

In a message written on Mon, Nov 29, 2010 at 03:34:52PM -0800, Seth Mattinen wrote:

My take on this is that settlement free peering only remains free as long as it is beneficial to both sides, i.e. equal amounts of traffic exchanged. If it becomes wildly lopsided in one direction, then it becomes more like paying for transit.

When you have users and no content how can the traffic be equal? When you have content and no users how can the traffic be equal? Ratio is horribly outdated. Cable and DSL providers enforce out of ratio at the edge with technology and policy. My cable modem is 8 down 2 up, yet my traffic profile is supposed to be equal? I can't host any "servers" by my TOS, but aggregated up the ratio is supposed to be 1:1? No one will ever be in ratio compliance with an eyeball dominant network. Ever. Period. It's not possible via technology and TOS. Enforcing it as an eyeball network just forces content providers to aquire eyeballs, e.g. compete with you. That's bad business. But this isn't a technology problem, or a ratio problem. Peering spats like this are ego problems. It's one VP/SVP/CTO/CFO deciding that "my sandbox is more important than your sandbox", or "I'm going to get revenue even if the world hates me for it and I'm going to burn all my bridges in the process." If they actually wanted to equalize the costs, they could do that. Decide on better peering locations, use cold potato routing, locate caching/cdn things inside the other network, etc. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

On 11/29/10 3:59 PM, Leo Bicknell wrote:

But this isn't a technology problem, or a ratio problem.

Comcast's blog specifically mentions unbalanced ratios as an issue. ~Seth

On Mon, 29 Nov 2010, Seth Mattinen wrote:

On 11/29/10 3:59 PM, Leo Bicknell wrote:

...

But this isn't a technology problem, or a ratio problem.

Comcast's blog specifically mentions unbalanced ratios as an issue.

They're an "eyes network". What do they expect? Look at typical traffic profiles for home users. They send out tiny requests, and receive big packets of data (web pages, images, streaming media). I find it ironic that when we were an "eyes network" of dial-up users, we bought transit to bring traffic in for our customers. Now that we're a hosting network and our transit bandwidth is lopsided the other direction, the big "eyes networks" are saying we should pay them to deliver the traffic their customers request. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

AFAIC, Comcast really doesn't have a leg to stand on by doing this. Unfortunately L3 set a very bad precedent by caving into Comcast's pressure.

Actually, by paying but crying foul, I think L3 is doing the best they can with a bad situation and as much as it pains me, I applaud L3 for this. The other options were: + Cave without crying foul -- Bad on both fronts. + Refuse -- Moral high ground, but, degraded service harmful to L3 and Comcast customers alike. I think it is better to try and preserve good user experiences and resolve the situation through "diplomatic" methods as L3 appears to be attempting to do. Hopefully this will lead to sufficient "peer pressure" (no pun intended) to get Comcast to eventually renege. Owen

In a message written on Mon, Nov 29, 2010 at 10:22:34PM -0500, Christopher Morrow wrote:

see craig's report from nanog47: <http://www.nanog.org/meetings/nanog47/presentations/Monday/Labovitz_ObserveReport_N47_Mon.pdf>

not for a time has Comcast been solely an 'eye-ball' network... or so they think.

I think you are misreading the data. From googling around it appears there are somewhere between 90 and 100 million "broadband subscribers" in the united states. Comcast claims to have somewhere between 15 and 17 million broadband subscribers, and they are the largest cable company in the US. With around 18-20% of all broadband end-users in the US Comcast, if you believe Arbor's numbers, generate 3.12% of all Internet traffic. Comcast also sells business service (not cable modem, but like GigE to the prem) which is propping that up a bit. If the FCC wanted to do something useful they would look at the combined ratio of all /customers/ of an ISP, and then require their peering policy to allow for around 2x of that. For instance, if you summed all Comcast customers and did the ratio of out:in and got 3:1, they should at a minimum be required to peer with someone at 6:1 IMHO. I have no idea in Comcast's case specifically, or in any recent case as my skin isn't in the game right now. However I am quite sure in the past I have delt with networks who wanted 2:1 on peering, but where I was nearly positive their customer base was 3:1 or 4:1. Basically the ratio became an excuse to depeer anyone they didn't like, it was all a sham. While I think ratio requirements are just plain stupid, I do think it needs to be considered when looking at peering. If you do hot potato routing the person on the "wrong end" of the ratio ends up carrying the traffic longer distances. If you look at long haul bandwidth on a bit-mile basis this can be unfair in some circumstances. The thing is though it's easy to fix. Networks could use MEDs (yes, they work on Internet scale routing), selective leaking (w/no-export), peering with regional ASN's (many of the large eyeball networks are subdivided internally) or any number of other very simple configurations to balance this issue. But I come back to my fundamental beef with cable and DSL providers, when you're selling 50/5 (10:1 ratio), 25/5 (5:1 ratio), 12/2 (6:1 ratio) services, you can't expect to maintain a 2:1 or 3:1 ratio with your peers. If you look at the TV side of the business the eyeball network is the whipping boy 99 times out of 100. Look at the recent Fox v Cable Vision dispute, Cable Vision caved. Users want content, users pay Cable Vision, Cable VIsion gets millions of angry calls, Fox runs a few ads how Cable Vision is the big bad guy and they have deals with everyone else. Go back to previous cases, almost always the eyeballs cave. Provides are trying to change this in IP space, because they don't like it. They want Netflix/Amazon/Apple/RIAA/MPAA to pay, and not be in charge. For the moment this works, if Netflix can't deliver via the Internet their users just request DVD's in the mail; a peering spat hurts Netflix more than Comcast. But, as users cut the cord, and get more of their content over the Internet I think we'll see the same shift. Outside of Nanog Ma and Pa Citizen don't even know what the word peering means. All they know is when they can't get their Netfix streaming to work they call their provider and complain, possibly going as far as to switch services. Now, while it may seem I'm taking Level 3's side of this dispute I am not. Sadly when these things spill out in public like this it is generally because both sides have been acting like idiots with each other in private for months or years. Maybe Level 3's been a model citizen in this case and has been wronged, but I doubt it. The problem is it all happened in private, and nice press releases from both parties aside we really have no idea what happened behind closed doors, who asked for what, who's egos got out of control, etc. I'm not going to call a winner or a loser, just point out how broken some of the arguments put forth are..... -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

In a message written on Mon, Nov 29, 2010 at 11:08:53PM -0500, Joe Provo wrote:

On Mon, Nov 29, 2010 at 08:03:27PM -0800, Leo Bicknell wrote: [snip]

...

If the FCC wanted to do something useful they would look at the combined ratio of all /customers/ of an ISP, and then require their peering policy to allow for around 2x of that. [snip]

...or maybe not get involved in peering policies at all? DO we really want regulatory oversight here, happily driving traffic away from the US?

I will be the first to advocate the government use minimal to no regulation where there is active competition and consumer choice, and thus folks can "vote with their dollars". Broadband in the US is not in that boat. Too many consumers have a "choice" of a single provider. The vast majority of the rest have the "choice" of two providers. We make these monopoly or duoploies come begging to local government to get approval for rate increases and channel lineup changes, they appear to be demonstrating that we need to do the same on the Internet side. I also fail to see how this drives traffic away from the US, it's not like Ma Citizen in Des Moines Iowa is going to move to Tokyo to get Gigabit service for $18. The only reason it looks like traffic is going away from the US is the rest of the world is leaving us behind, more bandwidth for less money. The current experiment going on in Australia is very interesting to watch.... -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

On Mon, Nov 29, 2010 at 11:03 PM, Leo Bicknell <bicknell@ufp.org> wrote:

In a message written on Mon, Nov 29, 2010 at 10:22:34PM -0500, Christopher Morrow wrote:

...

see craig's report from nanog47: <http://www.nanog.org/meetings/nanog47/presentations/Monday/Labovitz_ObserveReport_N47_Mon.pdf>

not for a time has Comcast been solely an 'eye-ball' network... or so they think.

I think you are misreading the data.

s/you are/Craig is/ I was just passing along a study presented at a nanog meeting about this kind of topic... I really do like to know next to nothing about peering.

I have no idea in Comcast's case specifically, or in any recent case as my skin isn't in the game right now.  However I am quite sure in the past I have delt with networks who wanted 2:1 on peering, but where I was nearly positive their customer base was 3:1 or 4:1. Basically the ratio became an excuse to depeer anyone they didn't like, it was all a sham.

sure, there are more variables (I gather) than just bits in/out... like 'but my customers complain more if you are further away/slower/more-lossy' etc. None of those factors are in peering agreements I would bet, though clearly ratios are, so that stick is used to whack the other-guy over the head.

But I come back to my fundamental beef with cable and DSL providers, when you're selling 50/5 (10:1 ratio), 25/5 (5:1 ratio), 12/2 (6:1 ratio) services, you can't expect to maintain a 2:1 or 3:1 ratio with your peers.

web traffic (as a measure) seems to be ~10:1 when I look at my interface at home (vz-consumer-type), without packet-loss and over a decent sample of time. As with all of the 'peering disputes' over the last few years, it'll be a fun ride to watch from the outside :) -Chris

On Nov 29, 2010, at 6:34 PM, Seth Mattinen wrote:

My take on this is that settlement free peering only remains free as long as it is beneficial to both sides, i.e. equal amounts of traffic exchanged. If it becomes wildly lopsided in one direction, then it becomes more like paying for transit.

Ratios were an excuse used by GTEi to try and force Exodus, Above.Net, and Global Center to pay for peering back in 1998. It had a valid, technical reason behind it - the cost of bit-miles.[*] Unfortunately, most people have forgotten this and simply claim one side is more 'valuable' than the other. In reality, the "value" of a relationship is NOT related to the number of bits flowing in either direction. More importantly, today large content providers & CDNs either carry the traffic longer, or deliver it close to the user, so the bit-mile argument is invalid. (For smaller providers, especially single-location hosting providers, the argument still holds.) But people do not actually care why ratios were originally brought up, ratios are simply used as a reason to bludgeon other providers into paying. Not that it ever mattered. Business relationships are not predicated on equal value. You do not refuse to buy copier paper unless the paper provider proves that he is not making more money off selling you paper than you are making off using the paper. You may go to the next provider who sells paper, but if they all make more than you do, you don't decide to go without. You buy paper and run your business. Peering is a business relationship. If your company can make more or spend less by peering with another company, you should do it. If you do not consummate that relationship, you are hurting your business. This should be the only reason to peer or not peer. Of course, some people will tell you there is an opportunity cost. Perhaps other networks would have bought from you for far more than you will save by peering this one network, and if you do peer those networks will expect it for free. This may a valid point, for a very limited set of providers negotiating with a very limited set of customers. The vast majority of the time, it is complete BS. Mostly that is just someone's ego talking, not a true business decision. Unfortunately, too many business decisions (not just on the Internet) are made for reasons more to do with ego than dollars. Which is a shame. The Internet is a business, and we would all do better to treat it as such. -- TTFN, patrick [*] 10 second explanation for those who do not understand: I hand you a small HTTP GET request, you carry it across the country. You had me a 1500 byte web page, I carry it across the country. My costs are much higher than yours, you need to compensate me for the additional costs. BTW: The attempt failed. Dave @ Above got Exodus & Global Center to agree to pull a Cogent if GTEi pulled a Level 3. GTEi blinked, and the rest is history.

On Nov 30, 2010, at 4:46 AM, John Curran wrote:

On Nov 29, 2010, at 11:47 PM, Patrick W. Gilmore wrote:

...

BTW: The attempt failed. Dave @ Above got Exodus & Global Center to agree to pull a Cogent if GTEi pulled a Level 3. GTEi blinked, and the rest is history.

Patrick -

Your summary is incorrect. To be perfectly clear on the history: In summer of 1997, GTEi did indeed have a dispute with Exodus regarding traffic levels on peering interconnects, and indicated that it would cease peering. On 16 Sep 1998, the dispute was resolved when Exodus signed an agreement with GTEi which was covered by non-disclosure at Exodus's request[1][2].

Well... Yes, Exodus signed an agreement under NDA and GTE got their ounce of flesh from Exodus, but, Patrick is correct in that GTE did not continue to extort GC and Above and got a much smaller reward than they were initially seeking (pound of flesh). The result of the NDA, however, was that from outside perceptions, it all looked like GTEi blinked, which, was very good for the industry. It would have been better if certain players on the Exodus side hadn't been quite so spineless, but, I guess to satisfy Godwin's law one can only say they were following the example of the French in early WWII.

...

Peering is a business relationship. If your company can make more or spend less by peering with another company, you should do it. If you do not consummate that relationship, you are hurting your business. This should be the only reason to peer or not peer.

Correct, and indeed that was basic principle in operation during the GTEi/Exodus peering dispute.

Sort of. It was clearly in GTEi and Exodus interest to continue exchanging traffic and obviously would have hurt both companies had actual depeering occurred. The question is which company would have been harmed the most and thus succumbed to settling the dispute. Owen

On Mon, Nov 29, 2010 at 8:47 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:

On Nov 29, 2010, at 6:34 PM, Seth Mattinen wrote:

...

My take on this is that settlement free peering only remains free as long as it is beneficial to both sides, i.e. equal amounts of traffic exchanged. If it becomes wildly lopsided in one direction, then it becomes more like paying for transit. ... [*] 10 second explanation for those who do not understand: I hand you a small HTTP GET request, you carry it across the country.  You had me a 1500 byte web page, I carry it across the country.  My costs are much higher than yours, you need to compensate me for the additional costs.

Clearly, to balance out the traffic ratios, content providers should set their server MTUs to 64 bytes. That way, small HTTP request packets will be nicely balanced out by small HTTP reply packets. If the content providers also turn off SACK, and force ACKs for each packet, they can achieve nearly the perfect traffic ratios the eyeball networks seem to desire. Small packet one way, equivalent small packet the other way, and everyone is happy. Obviously those recent infidels pushing for the so-called "Jumbo Frames" here on NANOG were nothing more than shills for the eyeball networks, seeking to get more and more networks out of ratio, in an effort to get them to cough up money. Fie on them, I say--instead of JumboFrames, we need MicroFrames! Exchange points should start enforcing a maximum frame size of 64 bytes, to truly bring the internet into perfectly-balanced ratio-ness. Matt (*in search of forceps to extract a tongue planted far too forcefully into the cheek*)

It seems that Comcast(AS7922) peers directly with Netflix(AS2906)....? -----Original Message----- From: Phil Bedard [mailto:bedard.phil@gmail.com] Sent: Monday, November 29, 2010 5:24 PM To: NANOG list Subject: Re: Level 3 Communications Issues Statement Concerning Comcast's Actions Is L3 hosting content for Netflix? Netflix has become a large source of traffic going to end users. L3 likely could have held out on this one if the content they were hosting is valuable enough to Comcast's customers, but maybe what Comcast was asking for wasn't much in the grand scheme of things. Obviously someone has to pay for the access infrastructure and Comcast would much rather get the content provider to pay for it versus passing it along to their customers. I think they probably just took a stab and L3 complied. Phil On 11/29/10 5:28 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

<http://www.marketwatch.com/story/level-3-communications-issues-statement- concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.

Just to be clear, L3 is saying content providers should not have to pay to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on this list are happy to hear L3's change of heart and will be applying for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are competing and L3 is putting up a "toll booth" on the Internet, would they?)

-- TTFN, patrick

Now we know what "Xfinity" means :-) Jeff

Trying to follow this - so, if I have followed it correctly, L3 hosts high-bandwitdh services (namely NetFlix) to which an abundance of Comcast users subscribe? And Comcast is crying foul, and claiming a portion of L3's revenue is rightfully theirs, for being "last mile" to a significant portion of the CDN/NetFlix customer base? Does L3 even service a home user market, in the same vein as Comcast or Verizon? On Mon, Nov 29, 2010 at 8:55 PM, Marshall Eubanks <tme@americafree.tv>wrote:

On Nov 29, 2010, at 6:24 PM, Phil Bedard wrote:

...

Is L3 hosting content for Netflix?

You bet.

http://blogs.barrons.com/techtraderdaily/2010/11/11/level-3-signs-deal-to-be...

• NOVEMBER 11, 2010, 9:13 AM ET

Level 3 Signs Deal To Be A Primary Netflix CDN; Shares Rally

Regards Marshall

...

Netflix has become a large source of traffic going to end users. L3 likely could have held out on this one if the content they were hosting is valuable enough to Comcast's customers, but maybe what Comcast was asking for wasn't much in the grand scheme of things.

Obviously someone has to pay for the access infrastructure and Comcast would much rather get the content provider to pay for it versus passing it along to their customers. I think they probably just took a stab and L3 complied.

Phil

On 11/29/10 5:28 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

...

< http://www.marketwatch.com/story/level-3-communications-issues-statement- concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.

Just to be clear, L3 is saying content providers should not have to pay to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on this list are happy to hear L3's change of heart and will be applying for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are competing and L3 is putting up a "toll booth" on the Internet, would they?)

-- TTFN, patrick

-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy

OK...as I was driving up and back on I-95 through Connecticut up toward Boston, I noticed that at all the rest/gas-up stops, there was a single restaurant - McDonalds. No Burger Kings, Wendy, etc...just McDonalds. Now I'm relatively certain that McD's had to pony up significant coin to be the restaurant of choice for the I-95 corridor through Connecticut. That commercial agreement was made with the owners of the infrastructure (in the case, the state of Connecticut) prior to McDonald establishing a presence on the highway. It would be bad form, IMO, for the state to come back to Mc'D's and say "hey...you guys are doing a thriving business here...we want a bigger cut, and if we don't get it, we'll barracade the exits and you'll do NO business in these shops you've stood up. Furthermore, we don't care if our customers (drivers on the highway) have bought the McD's meal plan for their frequent trips up and down the road...they can't do business here." Comcast has essentially quarantined off part of the 'net. The point was made earlier - does Comcast take the same stand against Google? Are they going to tax Amazon (Holiday traffic, you know) for their traffic, or any other online merchant that may not use Comcast as a primary provider? Comcast's infrastructure is getting overrun by legitimate traffic requested by their own customers - I'm not a NetFlix subscriber, but the way I understand it is that the customer requests content in the form of movies over the net - content is not arbitrarily provided by the content provider. Instead of rasing the tarriffs for the true consumers in this case - being their customers, they're going after the content provider. Anti-competitive at best. The truth is, given today's media-rich content, Comcast can't deliver a 15mb constant stream of traffic (or whatever it is they claim to offer), at $19.95/month - they've made a business decision to over-subscribe their infrastructure, and had a formula for the level to which they were going to over-subscribe their network. That formula either wasn't accurate then, or didn't take into account the media-rich content that would be delivered over the net. Now, the second explanation really doesn't hold water in light of the fact that Comcast also offers premium content similar to NetFlix. A significant portion of the Comcast subscriber base has decided the content available from NetFlix either in content or delivery, superior to that offered by Comcast. So Comcast is now saying ..."wait...to get your content to our customers, your traffic has to traverse our backbone, therefore, we should be able to extract a tarriff for said traffic." But the NetFlix subscription is a fully "opt-in" model. L3, via NetFlix, is simply delivering the content Comcast customers have requested, and using the Internet to delivery it. Comcast has truly handled this all wrong...they should either a) charge their customers a more realistic monthly fee - one more in line with what it takes to deliver what their level-of-service claims they offer, or b) improve the quality and/or delivery of their premium content, or make it more cosst-effective to their customers, so their customers don't have to go "off-net" to get the premium content they desire. What they did is take the easy route, which is/was to try to get into the pockets of the successful content provider, because their network is ill-equipped to handle their own level-of-service claims. On Mon, Nov 29, 2010 at 9:44 PM, Marshall Eubanks <tme@americafree.tv>wrote:

On Nov 29, 2010, at 9:03 PM, Steven Fischer wrote:

...

Trying to follow this - so, if I have followed it correctly, L3 hosts high-bandwitdh services (namely NetFlix) to which an abundance of Comcast users subscribe?

That is my understanding.

...

And Comcast is crying foul, and claiming a portion of L3's revenue is rightfully theirs, for being "last mile" to a significant portion of the CDN/NetFlix customer base?

That is my reading of these diplomatic notes.

...

Does L3 even service a home user market, in the same vein as Comcast or Verizon?

Not as far as I know, although they made enough acquisitions I wouldn't be surprised if they had the odd neighborhood.

Regards Marshall

...

On Mon, Nov 29, 2010 at 8:55 PM, Marshall Eubanks <tme@americafree.tv> wrote:

On Nov 29, 2010, at 6:24 PM, Phil Bedard wrote:

...

Is L3 hosting content for Netflix?

You bet.

http://blogs.barrons.com/techtraderdaily/2010/11/11/level-3-signs-deal-to-be...

...

• NOVEMBER 11, 2010, 9:13 AM ET

Level 3 Signs Deal To Be A Primary Netflix CDN; Shares Rally

Regards Marshall

...

Netflix has become a large source of traffic going to end users. L3 likely could have held out on this one

...

...

the content they were hosting is valuable enough to Comcast's customers, but maybe what Comcast was asking for wasn't much in the grand scheme of things.

Obviously someone has to pay for the access infrastructure and Comcast would much rather get the content provider to pay for it versus passing it along to their customers. I think they probably just took a stab and L3 complied.

Phil

On 11/29/10 5:28 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

...

< http://www.marketwatch.com/story/level-3-communications-issues-statement- concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.

Just to be clear, L3 is saying content providers should not have to

if pay

...

...

...

to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on this list are happy to hear L3's change of heart and will be applying for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are competing and L3 is putting up a "toll booth" on the Internet, would they?)

-- TTFN, patrick

-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy

-- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy

On Nov 29, 2010, at 10:51 PM, Ben Butler wrote:

In the Uk, we used to have 2MB DSL, and business providers like myself would happily provide it on the basis of CBR 2Mbit and we did'nt care what you did with it. 2Mbit is more than enough for streaming and I challenge anyone otherwise.

I say otherwise. So do many customers who want 720 or 1080 lines on their TV. So do many content providers who want to satisfy their customers. But it is your network, your rules. If your customers do not want "HD quality", and are happy with 1.5 Mbps streams per DSL line, that's between you & your customers. Of course, if your customers want more, that's between your customers and your competitors.... -- TTFN, patrick

On Mon, Nov 29, 2010 at 11:23 PM, Ben Butler <ben.butler@c2internet.net> wrote:

Ok, you have a point with SD vs HD which is encoded at 8 rather than 2 on our digital terrestrial and satellite broadcasters in the UK.

So why 24mb or 50mb access speeds, what is it actually being used for, I do not believe that streamed video is the culprit here with most codecs doing about ~700 kbits.

why NOT 24mb or 50mb or 500mb? what applications/innovations/ideas are being left on the table because folk can't do them in a reasonable period of time with <2mbps and longer queue times on their home network? If you could provision, cheaply, more bandwidth to the end-users in a community (or really several communities) do you think there would be new applications or new services provided to these folks? Video HD or SD is just one application/service... they get lots of hype today, but tomorrow what things may be possible? -chris

On Tue, 30 Nov 2010 04:23:41 +0000 Ben Butler <ben.butler@c2internet.net> wrote:

Part of the problem is the content providers do not encode properly, we have seen this all along with images on webs sties as access speeds have increased. There is no penalty on the content provider for lazy programming, cpu cycles or codec licensing to stop them making the access network carry larger streams than nessacery.

Hi Ben, Content providers do have some motivation to use resources, including bandwidth, efficiently: 1. it costs money to serve that content - bandwidth isn't free (not on my network anyway) 2. the consumers of that content may have restrictive access speeds or poorly performing access services. Now, that doesn't necessarily mean that content providers will make their content as bandwidth-efficient as may be technically possible... Greater bandwidth efficiency also comes at a price - newer technologies, greater computing resource, expertise, etc - and those costs may outweigh the benefit of enabling access for a portion of the end-user community. You will see different approaches by different folks. This isn't much different from, for example, attitudes to AAAA records in DNS. Google are extremely cautious because they see great value in not disrupting the end user experience, whilst others are content to publish the records accepting that some end users may be affected. If you believe that providers of oversized content should pay towards the recipient's and/or his/her agents' costs, it is perhaps suprising that you accompany so few lines of contribution to this discussion with such a large and unnecessary e-mail "signature" which includes several lines of HTML markup and misguided legal boilerplate, not to mention the failure to appropriately trim Patrick's comments to which you are responding. d.

Owen DeLong wrote:

You can stream 1080p/5.1 128khz over 2mbps at high quality using codecs that were available 2 years ago. (VP6, VP7 can do this, for example).

Over the 'Internet'? Why do you think http://www.vudu.com/ tells me I need 4.5Mbpps? "Required Internet Speed: Customers should have at least a 1 Mbps broadband internet connection in order to enjoy the VUDU streaming service. With faster broadband connections, customers can enjoy VUDU's 720p HD and industry leading 1080p HDX format. Minimum requirements for the VUDU streaming service are as follows: SD (480p) requires 1 Mbps HD (720p) requires 2.25 Mbps HDX (1080p) requires 4.5 Mbps" --Michael

Same hymn sheet, if they pay enough the cost averaging model works again and we don't have to worry about latency critical or transfer volume. The problem is that they wont pay for it. -----Original Message----- From: wherrin@gmail.com [mailto:wherrin@gmail.com] On Behalf Of William Herrin Sent: 30 November 2010 04:17 To: Ben Butler Cc: NANOG list Subject: Re: Level 3 Communications Issues Statement Concerning Comcast'sActions On Mon, Nov 29, 2010 at 10:51 PM, Ben Butler <ben.butler@c2internet.net> wrote:

Then consumer broadband came along, the subs went down, the headline speeds went up, service delivery becomes impossible in the face of the marketing BS ---- and here we are.

Hi Ben, So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage. There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem. Another problem is that the price of electricity has been very stable for a very long time, as has the general character of devices which consume it. Consumers have a gut understanding of the cost of leaving the light on. But what is a byte? How much to load that web page? Watch that movie? And doesn't Moore's Law mean that 18 months from now it should cost half as much? If I can't tell whether or not I'm being ripped off, I'm probably being ripped off. A third problem is the whole regulated monopoly thing. The electric company had to be slapped down hard by the government to make its billing process fair. Anything we can do to avoid that fate is money in the bank, even if it means allowing the occasional customer to get more than he paid for. So if we can't bill you by usage, and at a consumer level we can't, then we have to find another way. Statistics and prayer isn't working out as well as we'd hoped so we're looking at double-billing schemes. Bad plan! Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004 -------------------------------------------------------------------------- BODY { MARGIN: 0px}.footerdark { LINE-HEIGHT: 13px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #001a35; FONT-SIZE: 9px; FONT-WEIGHT: normal; TEXT-DECORATION: none}.blackcopy { LINE-HEIGHT: 12px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #000000; FONT-SIZE: 10px; FONT-WEIGHT: bold; TEXT-DECORATION: none}.bluecopy { LINE-HEIGHT: 12px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #29aae2; FONT-SIZE: 10px; FONT-WEIGHT: bold; TEXT-DECORATION: none}.address { LINE-HEIGHT: 12px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #000000; FONT-SIZE: 10px; TEXT-DECORATION: none}.footerlight { LINE-HEIGHT: 13px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #667891; FONT-SIZE: 9px; FONT-WEIGHT: normal; TEXT-DECORATION: none}.pinkcopy { LINE-HEIGHT: 12px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #ed174d; FONT-SIZE: 10px; FONT-WEIGHT: bold; TEXT-DECORATION: none} Ben Butler Director Tel: 0333 666 3332 Fax: 0333 666 3331 C2 Business Networking Ltd The Paddock, London Road, Nantwich, Cheshire, CW5 7JL http://www.c2internet.net/ Part of the Atlas Business Group of Companies plc Registered in England: 07102986 Registered Address: Datum House, Electra Way, Crewe CW1 6ZF Vat Registration No: 712 9503 48 This message is confidential and intended for the use only of the person to whom it is addressed. If you are not the intended recipient you are strictly prohibited from reading, disseminating, copying, printing, re-transmitting or using this message or its contents in any way. Opinions, conclusions and other information expressed in this message are not given or authorised by the Company unless otherwise indicated by an authorised representative independent of this message. The Company does not accept liability for any data corruption, interception or amendment to any e-mail or the consequences thereof.Emails addressed to individuals may not necessarily be read by that person unless they are in the office.Calls to and from any of the Atlas Business Group of Companies may be recorded for the purposes of training, monitoring of quality and customer services.

It's a popular concept that competition will resolve NN concerns. A couple of weeks back I taped Barbara Van Schewick expounding on her theme that blocking, discrimination, and/or access charges, ARE acceptable if at the users - rather than provider's - discretion. http://www.isoc-ny.org/p2/?p=1459 Afterwards, I asked her about the effect of competition. She remarked that, according to her research, countries with competition, such as the Euro unbundling regimes like the UK, actually had a much higher likelihood of such network management practices that the duopolist USA as the providers were under greater pressure to optimize the economic value of every bit put through. <http://www.isoc-ny.org/p2/?p=1459>Plusnet's transparency would seem to be indicative of a trend toward Van Schewick style solutions, where user's have a bandwidth dashboard where they can opt to throttle application-by-application, plus possibly receive targeted ads, to get a cheaper connection. j On Tue, Nov 30, 2010 at 3:13 AM, Michael Painter <tvhawaii@shaka.com> wrote:

Ben Butler wrote:

...

Same hymn sheet, if they pay enough the cost averaging model works again and we don't have to worry about latency critical or transfer volume. The problem is that they wont pay for it.

I became interested in these guys: http://www.plus.net/?home=hometop in 2008 because they were one of the first to use DPI (and admit it) to enforce their TOS. Every time I check their site (~every 8-10months), they seem to have won another award. Is 'Net Neutrality', the FCC, or something else preventing a model like this from having success in the U.S.? Or does it exixt and I just haven't heard about it?

--Michael

...

-----Original Message----- From: wherrin@gmail.com [mailto:wherrin@gmail.com] On Behalf Of William Herrin Sent: 30 November 2010 04:17 To: Ben Butler Cc: NANOG list Subject: Re: Level 3 Communications Issues Statement Concerning Comcast'sActions

On Mon, Nov 29, 2010 at 10:51 PM, Ben Butler <ben.butler@c2internet.net> wrote:

...

Then consumer broadband came along, the subs went down, the headline speeds went up, service delivery becomes impossible in the face of the marketing BS ---- and here we are.

Hi Ben,

So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage.

There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem.

Another problem is that the price of electricity has been very stable for a very long time, as has the general character of devices which consume it. Consumers have a gut understanding of the cost of leaving the light on. But what is a byte? How much to load that web page? Watch that movie? And doesn't Moore's Law mean that 18 months from now it should cost half as much? If I can't tell whether or not I'm being ripped off, I'm probably being ripped off.

A third problem is the whole regulated monopoly thing. The electric company had to be slapped down hard by the government to make its billing process fair. Anything we can do to avoid that fate is money in the bank, even if it means allowing the occasional customer to get more than he paid for.

So if we can't bill you by usage, and at a consumer level we can't, then we have to find another way. Statistics and prayer isn't working out as well as we'd hoped so we're looking at double-billing schemes. Bad plan!

Regards, Bill Herrin

-- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com Secretary - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------

aka the Australian NBN model ? Or taking the Allied Fiber 'real-estate' model to the edge. It's not beyond possibility that some US muni's may go for it. j

Recipe for success is to have a neutral entity whose business idea is to rent out fiber to anyone who wants to rent it, and who goes all the way to residential customers. Aggregate at nodes with several thousand households and let ISPs colocate at these nodes to reach end users.

Think COs but instead of copper, use fiber, and the entitity who owns this doesn't do anything but L1.

-- Mikael Abrahamsson email: swmike@swm.pp.se

-- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com Secretary - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------

On Mon, Nov 29, 2010 at 22:17, William Herrin <bill@herrin.us> wrote:

So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage.

There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem.

Not quite. Look at mobile data plans. A very few are unlimited, most are per byte.

Another problem is that the price of electricity has been very stable for a very long time, as has the general character of devices which consume it. Consumers have a gut understanding of the cost of leaving the light on. But what is a byte? How much to load that web page? Watch that movie? And doesn't Moore's Law mean that 18 months from now it should cost half as much? If I can't tell whether or not I'm being ripped off, I'm probably being ripped off.

Yep, sure seems that way when I get my mobile bill with roaming data charges. Consumers learn what it costs per byte, apps are created for them to manage their download amounts. Carriers send messages alerting consumers of their usage.

A third problem is the whole regulated monopoly thing. The electric company had to be slapped down hard by the government to make its billing process fair. Anything we can do to avoid that fate is money in the bank, even if it means allowing the occasional customer to get more than he paid for.

So if we can't bill you by usage, and at a consumer level we can't, then we have to find another way. Statistics and prayer isn't working out as well as we'd hoped so we're looking at double-billing schemes. Bad plan!

If double billing is such a bad plan, what are your proposed alternatives? Andy Koch

On Nov 29, 2010, at 9:09 PM, Andrew Koch wrote:

On Mon, Nov 29, 2010 at 22:17, William Herrin <bill@herrin.us> wrote:

...

So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage.

There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem.

Not quite. Look at mobile data plans. A very few are unlimited, most are per byte.

And I am on Sprint because they are one of the few.

...

Another problem is that the price of electricity has been very stable for a very long time, as has the general character of devices which consume it. Consumers have a gut understanding of the cost of leaving the light on. But what is a byte? How much to load that web page? Watch that movie? And doesn't Moore's Law mean that 18 months from now it should cost half as much? If I can't tell whether or not I'm being ripped off, I'm probably being ripped off.

Yep, sure seems that way when I get my mobile bill with roaming data charges. Consumers learn what it costs per byte, apps are created for them to manage their download amounts. Carriers send messages alerting consumers of their usage.

I simply avoid using roaming services. Frankly, my carrier could double their revenue from me and significantly increase their profits if they would offer me a global unlimited data/voice plan for twice what I currently pay for domestic. (If any of you cellular companies are listening, that's right, I'd be willing to pay ~$250/month for global unlimited voice/data and my usage would not increase very much above what you're already providing). I also happen to know that I'm not the only consumer that would very much like to be able to purchase this kind of service. Owen

On Tue, Nov 30, 2010 at 1:52 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

Considering there are mobile roaming partners that charge USD10-15 per megabyte, unfortunately that proposition is really hard to do in todays global market.

but really, the 'cost' here is the same as a local wireless user for air-time traffic, then 1$/mbps to ship the bits off their network across the internet, right? So... same cost regardless of 'roaming' or 'local user', at the bit level, right? Perhaps some charge to do the LNP (or equivalent to find/locate the handset in the world) lookup but that can't be 10-15$/mb equivalent is it? -Chris

I spent have the GDP of Kyrgyzstan on their roaming charges when I thought my Google Voice was using the WiFi vs. the mobile network. Jeff On Tue, Nov 30, 2010 at 2:18 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

On Tue, 30 Nov 2010, Christopher Morrow wrote:

...

On Tue, Nov 30, 2010 at 1:52 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:

...

Considering there are mobile roaming partners that charge USD10-15 per megabyte, unfortunately that proposition is really hard to do in todays global market.

but really, the 'cost' here is  the same as a local wireless user for air-time traffic, then 1$/mbps to ship the bits off their network across the internet, right?

Unfortunately the "cost" and the "price" has a very low correlation in this market. The same carriers who charge USD10 per megabyte for roaming might charge 0.1USD per megabyte to some of their own end users.

-- Mikael Abrahamsson    email: swmike@swm.pp.se

-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions

On Nov 29, 2010, at 10:58 PM, George Bonser wrote:

...

I'd be willing to pay ~$250/month for global unlimited voice/data and my usage would not increase very much above what you're already providing). I also happen to know that I'm not the only consumer that would very much like to be able to purchase this kind of service.

Owen

So would I. I make a fair amount of international calls/month but not very many (say less than a dozen in a normal month and that might increase to 20 if I had such a plan). Thing is that when I do, they are often conference calls that last a while.

The problem with "unlimited" is the 85/15 problem where 15% of your users are responsible for 85% of your traffic.

That's only a problem if it's always the same 15% of users and the other 85% don't see a value proposition to the rate they are paying. Also, I don't care if it's unlimited international -> international calling, mainly I want unlimited international data roaming and the ability to receive and make calls wherever<->US.

Give someone unlimited international and you are going to get a bunch of people such as foreign students who will be chatting with their friends back home all day. The issue with terminating international calls is that often the terminating telecom monopoly charges the initiating telco per minute for the call.

Today this can be overcome with rational VOIP delivery options that can make the call termination local in the receiving country with the internet covering the international legs. Sure, it messes with Caller ID, but, in my experience, international caller ID on cellular is, well, unreliable at best anyway, so, I'm not seeing a problem.

Unlimited mobile to mobile might be an option in some countries, though. Calling landlines in foreign countries doesn't scale when "unlimited" is in the equation.

Like I said, I'm not so worried about calling foreign landlines unlimited, but, I want to be able to call the US from wherever I am, get calls on my phone wherever I am, and, use data services wherever I am. For that, I'm willing to pay $250/month. Owen

On Tue, Nov 30, 2010 at 12:09 AM, Andrew Koch <andrew.koch@gawul.net> wrote:

On Mon, Nov 29, 2010 at 22:17, William Herrin <bill@herrin.us> wrote:

...

So if we can't bill you by usage, and at a consumer level we can't, then we have to find another way. Statistics and prayer isn't working out as well as we'd hoped so we're looking at double-billing schemes. Bad plan!

If double billing is such a bad plan, what are your proposed alternatives?

Hi Andrew, Back in the dialup days the solution was a single word: attended. The $100 account was 24/7 and the $20 account was "unlimited attended dialup." First month you got a warning so we could be sure you understood that 24/7 was a different account. Second month was upgrade or goodbye. -Bill -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Tue, Nov 30, 2010 at 9:47 AM, William Warren <hescominsoon@emmanuelcomputerconsulting.com> wrote:

On 11/30/2010 12:09 AM, Andrew Koch wrote:

...

On Mon, Nov 29, 2010 at 22:17, William Herrin<bill@herrin.us>  wrote:

...

So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage.

There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem.

Not quite.  Look at mobile data plans.  A very few are unlimited, most are per byte.

I don't know of a single data plan that's unlimited.  they all have either 5 gig or lower transfer caps.  That's not unlimited no matter what the lawyers or marketers day.

William, My Verizon Blackberry plan says unlimited data. Including the tether. IIRC, Clear's 4G service has no monthly cap. Regardless, we were talking about residential Internet, not mobile Internet. There's a market expectation that mobile systems cost more than their wireline counterparts and have usage-based billing even if their wireline counterparts don't. Moving the market expectations for wireline Internet in the face of your competitor's ability to move a different way is tough. This, by the way, is where Verizon is going to take some of you to the cleaners. With fiber all the way to the premises and control of the key transit-free who everyone else either peers with or pays, they can jack up their data capacity much more cost effectively than you can. And let's face it, when they tell you that you're upgrading your peering port from 10G to 100G in order to keep it, you will comply rather than lose reciprocal peering. The more you complain, the rosier they'll smell replying that, "Oh, we don't see the problem. We just increased our data rates. We only see a need for caution on the highly competitive wireless side which because of competition doesn't need regulation anyway." Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

MetroPCS also offers unlimited EVDO. Owen On Nov 30, 2010, at 8:22 AM, Brielle Bruns wrote:

On 11/30/10 9:07 AM, William Herrin wrote:

...

My Verizon Blackberry plan says unlimited data. Including the tether.

Its 5GB, trust me on that one. Former roommate worked for Verizon Wireless as a high level blackberry tech in the local call center - they quietly added the cap to all plans over the past year after adding all these little disclaimers to sales docs, websites, etc.

She came home and warned us one day that our EVDO modem on the business account was now capped, even though it was originally 'unlimited'. IIRC, they'll start billing you per megabyte or gigabyte after 5GB. I've not had an oppertunity to test this, so I'm only going by what I was told.

...

IIRC, Clear's 4G service has no monthly cap.

It does, 5GB as well, but I believe they throttle you down majorly once you hit the cap. I'll keep my eyes on the fine print next time I see a Clear commercial here.

-- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

I used to have an unlimited EVDO service from Sprint, when they changed to 5GB I called to complain and was advised that my plan was grandfathered, my new limit 5GB but with $0/GB overage. Jeff On Nov 30, 2010 11:24 AM, "Brielle Bruns" <bruns@2mbit.com> wrote: On 11/30/10 9:07 AM, William Herrin wrote:

My Verizon Blackberry plan says unlimited data. Inclu...

Its 5GB, trust me on that one. Former roommate worked for Verizon Wireless as a high level blackberry tech in the local call center - they quietly added the cap to all plans over the past year after adding all these little disclaimers to sales docs, websites, etc. She came home and warned us one day that our EVDO modem on the business account was now capped, even though it was originally 'unlimited'. IIRC, they'll start billing you per megabyte or gigabyte after 5GB. I've not had an oppertunity to test this, so I'm only going by what I was told.

IIRC, Clear's 4G service has no monthly cap.

It does, 5GB as well, but I believe they throttle you down majorly once you hit the cap. I'll keep my eyes on the fine print next time I see a Clear commercial here. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

---------- Original Message ----------- From: William Herrin <bill@herrin.us> Sent: Tue, 30 Nov 2010 13:17:45 -0500

I checked it out when I updated my credit card number online recently. The billing page has a place to describe a cap and overage charges. It's listed as unlimited. Not saying you're wrong. Just saying that the billing documentation disagrees.

It's 'unlimited' up to 5Gb -- big lawyers make that work I guess. And yes I've also been grandfathered in from almost 8 years ago when I first got it -- for these types of accounts they shut you off instead of billing overusage. -Randy

I just wanted to stop and say I'm glad we can have this kind of debate :) I think we need to start with education at every level. Watching 1-2 movies a day, some additional streaming content, using the VoIP phone whenever, and surfing the web is normal behavior. Running occasional P2P is normal behavior. You'd never leave the water running all day, even though if you rent it probably wouldn't cost you any more (landlord usually pays for water). It's not simply a question of "what can I get," it's a question of being a good internet citizen. There will never be a network so robust that everyone in the world could go full throttle all the time at the same time, so we have to share. I myself am against a lot of regulation of the free market. I want to be able to use P2P without it being relegated to scavenger, though I don't use it all the time. I want to watch Hulu or Discovery or Netflix when something is on that I want to see. I've heard of and seen implemented some rather generous leaking token bucket scenarios that keep the average user unaware of any bandwidth restrictions, while causing slower service for those people that use everything at full speed all the time. Since I pay the same (or more) than most of the other shared media users for my service, I think that is a good implementation of fair use. They can still use critical services, VoIP, HTTP, and some video, but they don't get the same kind of full-throttle download anymore. Sincerely, Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services

On the subject of marketing for years the wireless operators sold unlimited data plans. Now they are coming back and saying well unlimited is really 5 GB. -----Original Message----- From: William Herrin [mailto:bill@herrin.us] Sent: Monday, November 29, 2010 11:17 PM To: Ben Butler Cc: NANOG list Subject: Re: Level 3 Communications Issues Statement ConcerningComcast'sActions On Mon, Nov 29, 2010 at 10:51 PM, Ben Butler <ben.butler@c2internet.net> wrote:

Then consumer broadband came along, the subs went down, the headline speeds went up, service delivery becomes impossible in the face of the marketing BS ---- and here we are.

Hi Ben, So you're saying: treat it like electrical service. I have a 200 amp electrical service at my house. But I don't pay for a 200 amp service, I pay for kilowatt-hours of usage. There are several problems transplanting that billing model to Internet service. The first you've already noticed - marketing activity has rendered it unsalable. But that's not the only problem. Another problem is that the price of electricity has been very stable for a very long time, as has the general character of devices which consume it. Consumers have a gut understanding of the cost of leaving the light on. But what is a byte? How much to load that web page? Watch that movie? And doesn't Moore's Law mean that 18 months from now it should cost half as much? If I can't tell whether or not I'm being ripped off, I'm probably being ripped off. A third problem is the whole regulated monopoly thing. The electric company had to be slapped down hard by the government to make its billing process fair. Anything we can do to avoid that fate is money in the bank, even if it means allowing the occasional customer to get more than he paid for. So if we can't bill you by usage, and at a consumer level we can't, then we have to find another way. Statistics and prayer isn't working out as well as we'd hoped so we're looking at double-billing schemes. Bad plan! Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 11/29/10 7:51 PM, Ben Butler wrote:

In the Uk, we used to have 2MB DSL, and business providers like myself would happily provide it on the basis of CBR 2Mbit and we did'nt care what you did with it. 2Mbit is more than enough for streaming and I challenge anyone otherwise.

While this whole discussion was going on, I took a break to watch Tears of the Sun on Netflix streaming via my Roku. The utilization looked like this: http://ninjamonkey.us/wordpress/wp-content/uploads/2010/11/netflixonroku.png ~Seth

I know the content providers hate this, I know the consumers will not pay more for their access

Yes they will. They are going to have to. To expect end user traffic to increase by an order of magnitude without any corresponding upgrade of the infrastructure to support that is living in a fantasy world. Comcast believes they are big enough to force the content providers to pay. If the content providers say "no", then Comcast will be forced to increase subscription fees in order to finance that upgrade. That will, in turn, allow their competitors to also increase subscription rates and upgrade their infrastructures to support today's traffic demands. If Comcast can force the providers to pay, they are betting that their competitors won't and Comcast will be able to undercut the subscription rates of their competition or force them into sub-standard service from the traffic loads. It is basically an economic game of "chicken". If the providers simply say "no" and disconnect, Comcast loses. There is a compromise solution but the politicians have neutered that idea. The idea would be to simply continue to carry the traffic but prioritize it down if you don't pay. Basically two levels of service ... standard and sub-standard (no "premium"). If a content provider pays, their traffic stays as it is. If a content provider doesn't pay, their stuff gets QoS second tier. This is a little different than the model of selling "premium" level access to a network. Heck, if I were Level3, I might even drop the level of traffic in my own network bound for Comcast and have a note on the site that Comcast users can expect poor performance. Something like "Comcast is too poor to upgrade their network and has attempted to extort payment from us under threat of disconnection from their internet users. Though we have refused to pay the "eyeball ransom", we have decided to help Comcast out anyway by bandwidth limiting traffic to their poor wittle network. As a result, Comcast users might experience reduced performance."

On Mon, Nov 29, 2010 at 9:29 PM, Ben Butler <ben.butler@c2internet.net> wrote:

 It is not double billing, it is shared billing.

Hi Ben, Nice try, but no. There are a couple forms of shared billing. The one you're probably talking about is The Dance. Everybody pays to get in to the dance. The organizer provides a ballroom and a DJ. And then you dance with whoever else pays. One key criteria for The Dance is that all participants are on an equal footing. If the DJ never gets around to playing my song because someone else gave him a list and a fifty then I've been ripped off. Netflix and John Residential Doe are not on equal footing. Netflix will always be the controlling voice in that partnership. Hence Dance-style shared billing is inappropriate -- from John's perspective he should have to pay Netflix or you but not both. Another form of shared billing is when two parties pool resources to deal with a third. For example, you and your insurance company at the pharmacy. Or my neighbor and I buying a jointly-owned snow-blower. I don't think that's the kind of shared billing you meant. Roommates appears to be shared billing but it isn't. In one version, a particular roommate is responsible for the entire rent whether the others cough up their share or not. That's called "subletting." In another, all roommates are responsible for the entire rent regardless of whether the others do what they're supposed to. Never sign the latter rent contract. Just don't. When it goes bad you get royally screwed. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Dec 1, 2010, at 3:38 PM, Derek J. Balling wrote:

On Nov 29, 2010, at 11:20 PM, Leo Bicknell wrote:

...

Broadband in the US is not in that boat. Too many consumers have a "choice" of a single provider. The vast majority of the rest have the "choice" of two providers.

I dunno. I've lived in areas where I had two dozen local providers vying for my last-mile residential connectivity business. Perhaps this is something for you to bring up with your local municipality, tell them to stop strangling the businesses that want to offer service to their residents.

I live in an area without two dozen local providers that offer services to my address. Neither T nor CMCSA offer service at my address nor will they even return calls about price quotes to build. The local municipalities were uninterested as well, including putting pressure on the local utilities (T/CMCSA) that have major offices/callcenters located in the township. Ultimately I managed to work something out and get service, but for those on the "edge" areas, its much harder than you would think to gain access. I suspect there will be ongoing property devaluation as a consequence of lack of these utilities.. - Jared

On Wed, Dec 1, 2010 at 3:38 PM, Derek J. Balling <dredd@megacity.org> wrote:

On Nov 29, 2010, at 10:25 PM, William Herrin wrote:

...

There are a couple forms of shared billing.

There's a third kind you failed to mention that doesn't require equal footing of the parties. The broker.

I might pay an apartment broker $X to help find me an apartment. In turn the apartment broker might match me up with an apartment, and charge the landlord $Y for a successful tenancy.

Hi Derek, For the most part the apartment broker process doesn't work quite the way you think. Generally he either gets a fee from you to find you the best apartment or a fee from the landlord to find him a tenant (a "no fee" listing). But not both. Read http://www.nakedapartments.com/blog/broker-fees-explained/. Sometimes the landlord will agree to cover part of the broker's fee but the legal fiction is that the landlord is paying the renter who is paying the broker. Also bear in mind that apartment brokers tend to be a New York City phenomenon where regulated rent stabilization laws and related heavy regulation apply. They exist elsewhere but all top 20 Google hits for "apartment broker fees" were NYC. Let's consider a related example that's more ubiquitous than New York City apartment brokers: the real estate agent. The seller's agent collects a commission. So does the buyer's agent. If they're the same person, they get both commissions. Right? http://homebuying.about.com/od/glossaryd/g/DualAgency.htm "Dual agency is not legal in all 50 states." http://homebuying.about.com/od/realestateagents/qt/92807_DualAgncy.htm "Dual agency must be agreed to in writing between [all three] parties." The problem with dual agency is it's a classic conflict of interest. That's why both buyer and seller have to agree to it and go in eyes-wide-open, even where it's legal. What's more, in the highly competitive real estate market, savvy buyers know it's time to apply the screws -- the agent will earn more money even if he takes a big hit on the buyer's commission. Kinda the opposite of the monopoly/duopoly ISP who doesn't seek your permission in dealing with anyone else. Finally, realize that in both cases (real estate agent and apartment broker) you're dealing with a competitive negotiated process. The law allows -many- things in negotiated contracts that are flat illegal in the contracts of adhesion typically offered to the residential Internet buyer. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On Dec 1, 2010, at 5:47 PM, William Herrin wrote:

On Wed, Dec 1, 2010 at 3:38 PM, Derek J. Balling <dredd@megacity.org> wrote:

...

On Nov 29, 2010, at 10:25 PM, William Herrin wrote:

...

There are a couple forms of shared billing.

There's a third kind you failed to mention that doesn't require equal footing of the parties. The broker.

I might pay an apartment broker $X to help find me an apartment. In turn the apartment broker might match me up with an apartment, and charge the landlord $Y for a successful tenancy.

Hi Derek,

For the most part the apartment broker process doesn't work quite the way you think. Generally he either gets a fee from you to find you the

Regardless of whether the apartment broker comparison holds up, there are many examples of what economists call two-sided markets: http://en.wikipedia.org/wiki/Two-sided_market They don't all have the same fee-splitting systems, and you can find an example to site as precedent for just about any system you could reasonably advocate. An example raised in a talk I heard a few years ago was of scholarly journals that collect money from both their subscribers and their authors. The authors need to be published in order to get tenure, and the readers pay because they want to know what the authors are saying. Another example is the Golden Gate Bridge, which was funded in the 1930s by the rural counties north of the bridge (including one ~300 miles north), who wanted connectivity to San Francisco. It's probably reasonable to generalize a bit and say that in the systems not imposed by regulators, the distribution of costs has something to do with how much each party cares, within the limits of each party's resources. Whether the response produced by the market is at all fair is another -- far more subjective -- question, and that's where regulators come in. -Steve

On Thu, Dec 2, 2010 at 7:25 PM, William Herrin <bill@herrin.us> wrote:

On Thu, Dec 2, 2010 at 4:28 PM, Steve Gibbard <scg@gibbard.org> wrote:

...

http://en.wikipedia.org/wiki/Two-sided_market

They don't all have the same fee-splitting systems, and you can find an example to site as precedent for just about any system you could reasonably advocate.  An example raised in a talk I heard a few years ago was of scholarly journals that collect money from both their subscribers and their authors. The authors need to be published in order to get tenure, and the readers pay because they want to know what the authors are saying.

I received an interesting comment off list to the effect that there's been a rise in journals in which the authors pays while the reader has open (free) access via the web citing this example: http://openwetware.org/wiki/Publication_fees Modern newspapers have gravitated towards a comparable model in which the advertisers bear the cost of the content and the readers have free access via the web. I, for example, read washingtonpost.com most days. In the abstract, the model looks like this: 1. Pick which side of the two sided market you expect to always pay you. For example, the advertisers in the newspapers' case. 2. Offer _fully functional_ free access to the other side of the market, where "fully functional" is largely defined by that side's nature. For example washingtonpost.com 3. Ask a convenience fee for access which does not impact functionality but is in some way more desirable to one or another segment of the otherwise unpaid side of the market. For example, paying a subscription to have the hardcopy version of the newspaper delivered. The jury is still out on whether this model is economically sustainable. Still, I think it could offer useful insight to folks like Comcast. I think it likely that the recipient side of the market is going to be the always-payer for ISP service on an eyeball network. That means giving the content side basic fully functional access for free and convenience enhancements for a fee. That's why I suggested: "Maybe you'll openly peer with all comers but only at 100 mbps in any single location. You'll open as many locations deep in the network as they want, but it's the peer's problem to connect there. Naturally you'll sell a convenience service to backhaul all those connection points to a convenient location for the peers... or they can make their own arrangements but either way they don't get to massively consume your backbone for free. There's probably enough separation there between what you sell customer B and what you sell customer C to eke over to the "good" side of the ethics line. And by the way an open peering policy with those parameters would make you the Chamber of Commerce's new best friend, enabling small business to vend innovative products directly to your customers (and then pay you for the convenience of aggregation once they build up a customer base)." The key pitfall with this model is function versus convenience. Your paid enhancements to the second side of the market can offer greater convenience but you cross the line if they offer greater functionality. Connecting where I want (instead of where you offer) is convenient. Connecting with enough bandwidth for my service to be usable is functional. Regards, Bill Herrin -- William D. Herrin ................ herrin@dirtside.com  bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

On 11/30/2010 6:33 AM, Jeff Young wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On 30/11/2010, at 9:28 AM, Patrick W. Gilmore wrote:

...

<http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.

Just to be clear, L3 is saying content providers should not have to pay to deliver content to broadband providers who have their own product which has content as well. I am certain all the content providers on this list are happy to hear L3's change of heart and will be applying for settlement free peering tomorrow. (L3 wouldn't want other providers to claim the Vyvx or CDN or other content services provided by L3 are competing and L3 is putting up a "toll booth" on the Internet, would they?)

-- TTFN, patrick

So in this particular game of chicken, Comcast wins. Shame that L3 agreed to this, sets a bad precedent. I have to imagine that Comcast would have been the worse for wear, their phone lines would have lit up like a Christmas tree -- why can't I access...?

jy -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iF4EAREIAAYFAkz04QkACgkQxvthcni5E2+LwgD+NAie3r+r1dniJNRPMVKAJEj7 BQIympMzCXji7NveWicA/ReSLZgW92LT4cY/yMnsw3EkrD8mL1rkhAzicifOoCwe =GPm+ -----END PGP SIGNATURE-----

This whole mess concerns me about the future of the internet. If the traffic can't get to the clients by routing around a depeering..is the internet really working as designed? I don't think so. Peering has become the gateway to the ultimate in network control...while it's the provider's prerogative who access their network..peering has become a club for access and has become the instrument of removing the basic design wins of the internet.

On 11/30/2010 9:06 AM, William Warren wrote:

This whole mess concerns me about the future of the internet. If the traffic can't get to the clients by routing around a depeering..is the internet really working as designed? I don't think so. Peering has become the gateway to the ultimate in network control...while it's the provider's prerogative who access their network..peering has become a club for access and has become the instrument of removing the basic design wins of the internet.

For a home user, it means knowledge and ability to setup a tunnel to somewhere else to receive the traffic. For netflix, you could setup a v6 tunnel and stream it via ipv6.netflix.com. Jack

Maybe I am oversimplifying this a bit, but the way I see this situation is this: 1. L3 is carrying traffic for a popular service 2. Comcast customers want that service. 3. Comcast and L3 peer with each other (i.e. very little cost for either) (So, Comcast is paying very little to get that data into their network) 5. Comcast wants L3 to pay them for the traffic (WTF?) Nobody pays me for bandwidth that *I* request and use. That is just ridiculous. I wonder how Comcast would feel if L3 said screw it, and sent all the traffic a different route. If it routed through a different provider (lets say "ProviderX"), would Comcast try to get ProviderX to pay for traffic it was sending? -Randy -- | Randy Carpenter | Vice President, IT Services | Red Hat Certified Engineer | First Network Group, Inc. | (419)739-9240, x1 ----

Great detective work and it feels very probable that you are largely correct. The pieces together quite nicely. Love the L3 LG part. I dont think they were out to get Comcast specifically but the whole internet, L3 is a large global player and sell lots of transit bits. More bits to sell and peering agreement ratios that is affected by a move like this. If large parts of the internet pays to get to your network why not get more of the internet to give to them? makes perfect sense. /Christian Karlsson Teknikmejeriet Sweden On 2010-11-30 19:02, Leo Bicknell wrote:

Having been involved with a few peering spats in the past I know what is said publically rarely matches the reality behind the scenes. In this particular case my spidy sense tells me there is absolutely something interesting behind the scenes, but the question is what.

I'd never really paid attention to how Netflix delivers its content. It's obviously a lot of bandwidth, and likely part of the issue here so I thought I would investigate.

Apparently Akamai has been the primary Netflix streaming source since March. LimeLight Networks has been a secondary provider, and it would appear those two make up the vast majority of Netflix's actual streaming traffic. I can't tell if Netflix does any streaming out of their own ASN, but if they do it appears to be minor.

Here's a reference from the business side of things: http://www.fool.com/investing/general/2010/11/11/netflix-takes-streaming-to-...

This is also part of the reason I went back to the very first message in this thread to reply:

In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. Gilmore wrote:

...

<http://www.marketwatch.com/story/level-3-communications-issues-statement-concerning-comcasts-actions-2010-11-29?reflink=MW_news_stmp>

I understand that politics is off-topic, but this policy affects operational aspects of the 'Net. Patrick works for Akamai, it seems likely he might know more about what is going on. Likely he can't discuss the details, but wanted to seed a discussion. I'd say that worked well.

I happen to be a Comcast cable modem customer. Gooling for people who had issues getting to Netflix streaming turned up plenty of forum posts with traceroutes to Netflix servers on Akamai and Limelight. I did traceroutes to about 20 of them from my cable modem, and it's clear Comcast and Akamai and Comcast and Limelight are interconnected quite well. Akamai does not sell IP Transit, and I'm thinking it is extremely unlikely that Comcast is buying transit from Limelight. I will thus conclude that these are either peering relationships, or that they have cut some sort of special "CDN Interconnect" deal with Comcast.

But what about Level 3? One of my friends I was chatting with on AIM said they thought Comcast was a Level 3 customer, at least at one time. Google to the rescue again.

Level 3 provides fiber to Comcast (20 year deal in 2004): http://blog.tmcnet.com/blog/tom-keating/voip/level-3-and-comcast.asp

Level 3 provides voice services/support to Comcast: http://cable.tmcnet.com/news/2005/jul/1168088.htm

Perhaps the most interesting though is looking up an IP on Comcast's local network here in my city in L3's looking glass: http://lg.level3.net/bgp/bgp.cgi?site=sjo1&target=68.86.240.141

Slightly reformatting for your viewing pleasure, along with my comments:

Community: North_America Lclprf_100 Level3_Customer # Level 3 thinks they are a customer United_States San_Jose EU_Suppress_to_Peers Suppress_to_AS174 # Cogent Suppress_to_AS1239 # Sprint Suppress_to_AS1280 # ISC Suppress_to_AS1299 # Telia Suppress_to_AS1668 # AOL Suppress_to_AS2828 # XO Suppress_to_AS2914 # NTT Suppress_to_AS3257 # TiNet Suppress_to_AS3320 # DTAG Suppress_to_AS3549 # GBLX Suppress_to_AS3561 # Savvis Suppress_to_AS3786 # LG DACOM Suppress_to_AS4637 # Reach Suppress_to_AS5511 # OpenTransit Suppress_to_AS6453 # Tata Suppress_to_AS6461 # AboveNet Suppress_to_AS6762 # Seabone Suppress_to_AS7018 # AT&T Suppress_to_AS7132 # AT&T (ex SBC)

So it would appear Comcast is a transit customer of Level 3 (along with buying a lot of other services from them). I'm going to speculate that the list of supressed ASN's are peers of both Level 3 and Comcast, and Comcast is going that so those peers can't send some traffic through Level 3 in attempt to game the ratios on their direct connections to Comcast.

Now a more interesting picture emerges. Let me emphasize that this is AN EDUCATED GUESS on my part, and I can't prove any of it.

Level 3 starts talking to Netflix, and offers them a sweetheart deal to move traffic from Akamai to Level 3. Part of the reason they are willing to go so low on the price to Netflix is they will get to double dip by charging Netflix for the bits and charging Comcast for the bits, since Comcast is a customer! But wait, they also get to triple dip, they provide the long haul fiber to Comcast, so when Comcast needs more capacity to get to the peering points to move the traffic that money also goes back to Level 3! Patrick, from Akamai, is unhappy at losing all the business, and/or bemused at the ruckus this will cause and chooses to kick the hornets nest on NANOG.

One last thing, before we do some careful word parsing. CDN's like Akamai and LimeLight want to be close to the end user, and the networks with end users want them to be close to the end user. It doesn't make sense to haul the bits across the country for any party involved. Akamai does this by locating clusters inside providers networks, LimeLight does it by provisioning bandwidth from their data centers directly to distribution points on eyeball networks.

So let's go back and look at the public statements now:

Level 3 said: http://www.marketwatch.com/story/level-3-communications-issues-statement-con... -2010-11-29?reflink=MW_news_stmp

"On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast's customers who request such content."

Comcast said: http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html

"Comcast has long established and mutually acceptable commercial arrangements with Level 3's Content Delivery Network (CDN) competitors in delivering the same types of traffic to our customers. Comcast offered Level 3 the same terms it offers to Level 3's CDN competitors for the same traffic."

You can make both of these statements make sense if the real situation is that Comcast told Level 3 they needed to act like a CDN if they were going to host Netflix. Rather than having Comcast pay as a customer, they needed to show up in various Comcast distribution centers around the country where they could drop traffic off "locally". To Comcast this is the same deal other CDN's get, it matches their statement. To Level 3, this means paying a fee for bandwidth to these locations, and being that they are Comcast locations it may even mean colocation fees or other charges directly to Comcast. Comcast said if you're not going to show up and do things like the CDN players then we're going to hold you to a reasonable peering policy, like we would anyone else making us run the bits the old way.

The most interesting thing to me about all of this is these companies clearly had a close relationship, fiber, voice, and IP transit all on long term deals. If my speculation is right I'm a bit surprised Level 3 would choose to piss off such a long term large customer by bringing Netflix to the party like this, which is one of the reasons I doubt my speculation a bit.

But, to bring things full circle, neither of the public statements tell the whole story here. They each tell one small nugget, the nugget that side wants the press to run with so they can score political points.

Business is messy, and as I've said throughout this thread this isn't about peering policies or ratios, there are deeper business interests on both sides. This article: http://www.marketwatch.com/story/level-3-outlines-network-expansion-on-netfl...

Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix. I'm sure a lot of that is going to Comcast users, since they are the largest residential broadband ISP.

Messy. Very messy.