I would have said OK, and then we'll go ahead and renew your contract with us at current price + $X/Mbps. Jeff On Tue, Nov 30, 2010 at 9:12 PM, Richard A Steenbergen <ras@e-gerbil.net> wrote:

On Tue, Nov 30, 2010 at 11:45:53AM -0800, Kevin Oberman wrote:

...

We have seen the same thing with other carriers. As far as I can see, Comcast is congested, at least at Equinix in San Jose. Since this is all over private connections (at least in our case), the fabric is not an issue.

Maybe they will be using the money from Level(3) to increase capacity on the peerings with the transit providers. (Or maybe not.)

I don't know about their connection to TWT, but Comcast has definitely been running their transits congested. The most obvious one from recent months is Tata, which appears to be massively congested for upwards of 12 hours a day in some locations. Comcast has been forcing traffic from large networks who refuse to peer with them (e.g. Abovenet, NTT, Telia, XO, etc) to route via their congested Tata transit for a few months now, their Level3 transit is actually one of the last uncongested providers that they have.

The part that I find most interesting about this current debacle is how Comcast has managed to convince people that this is a peering dispute, when in reality Comcast and Level3 have never been peers of any kind. Comcast is a FULL TRANSIT CUSTOMER of Level3, not even a paid peer. This is no different than a Comcast customer refusing to pay their cable modem bill because Comcast "sent them too much traffic" (i.e. the traffic that they requested), and then demanding that Comcast pay them instead. Comcast is essentially abusing it's (in many cases captive) customers to extort other networks into paying them if they want uncongested access. This is the kind of action that virtually BEGS for government involvement, which will probably end badly for all networks.

If there is any doubt about any of this, you can pop on over to lg.level3.net and look at the BGP communities Comcast is tagging on their Level3 transit service, preventing the routes from being exported to certain peers. For example, to my home cable modem:

Community: North_America Lclprf_100 Level3_Customer United_States Chicago2 EU_Suppress_to_Peers Suppress_to_AS174 Suppress_to_AS1239 Suppress_to_AS1280 Suppress_to_AS1299 Suppress_to_AS1668 Suppress_to_AS2828 Suppress_to_AS2914 Suppress_to_AS3257 Suppress_to_AS3320 Suppress_to_AS3549 Suppress_to_AS3561 Suppress_to_AS3786 Suppress_to_AS4637 Suppress_to_AS5511 Suppress_to_AS6453 Suppress_to_AS6461 Suppress_to_AS6762 Suppress_to_AS7018 Suppress_to_AS7132

-- Richard A Steenbergen <ras@e-gerbil.net>       http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions

On Tue, Nov 30, 2010 at 06:45:57PM -0800, Leo Bicknell wrote:

Actually it appears to be Level 3 who fired the first PR salvo running to the FCC, if the date stamps on the statements are right. So it's really Level 3 framing as a net neutrality peering issue the fact that Comcast balked at paying them more.

I never said otherwise. The PR is pretty clear: Level 3 says that Comcast, their TRANSIT CUSTOMER, demanded that Level 3 pay them because of a ratio imbalance. Level 3, not wanting to cause massive disruptions to their other customers who would then no longer be able to reach Comcast (or depending on your point of view, because of an extreme lack of testicular fortitude), complied, and then put out a PR whining about it. In some ways it IS a net neutrality issue. Comcast is effectively "too big" to turn off, and has used the threat of disruption to it's massive customer base to bully a transit provider into paying its customer for the right to deliver service. Comcast has made it quite clear that their goal is to charge content companies for access to their customers, which if I'm not mistaken is what the whole net neutrality thing (at least originally) was all about. :)

Netflix is today apparently delivered via Akamai, who has nodes deep inside Comcast. Maybe Akamai pays Comcast, I actually don't think that is the case from an IP transit point of view, but I think they do pay for space and power in Comcast data centers near end users. But anyway, this Netflix data is close to the user, and going over a settlement free, or customer connection.

Netflix is today delivered by LimeLight and Akamai, who are both very clearly and publicly acknowledged customers of Comcast (though the LLNW deal is VERY fresh), as well as by Level 3 CDN. Level 3 CDN recently (and very publicly) won a lot of Netflix's business, but they're by no means new customers.

Level 3 appears to have sucked Netflix away, and wants to double dip charging Netflix for the transit, and Comcast for the transit. Worse,

Absolutely they wanted to double dip. If you've seen the prices that Level 3 is selling it's CDN services for, you'd know they'll need to quadruple dip just to break even. :) Comcast wants to double dip too. They're not satisfied with receiving the traffic via a peer for free, they want to be paid on both sides. So yes you effectively have a battle of two companies who want to double dip. The major difference is that Level 3 accomplished its double dip by providing quality service at a reasonable price in an environment with a significant amount of competition, while Comcast accomplished its double dip by hosting its (mostly captive) customer base hostage, and intentionally creating congestion via every alternate path. If Comcast was winning customers by offering better, cheaper, faster service, they would have a leg to stand on, but the reality is the only thing they're offering is access to their captive eyeball customers. The funny part is that Level 3 was clearly ill prepared for the PR war, whereas Comcast, being the first mover (if not the first PR issuer), was well prepared.

they get to triple dip, since they are Comcast's main fiber provider. Comcast will have to buy more fiber to haul the bits from the Equinix handoffs to the local markets where Akamai used to dump it off. Worse still, Level 3 told them mid-novemeber that the traffic would be there in december. Perhaps 45 days to provision backbone and peering to handle this, during the holiday silly season. Perhaps Level 3 wanted to quadruple dip with the expedite fees.

I think you're making a lot of assumptions which have no basis in fact above, unless you know something I don't, which based on what I've read so far I don't think you do. Again, there is no peering, Comcast is a Level 3 transit customer. Until a month ago a lot of this content was being delivered by LLNW via Global Crossing, until Comcast threatened LLNW with intentional congestion of it GX paid peers, and forced them to buy directly to keep Netflix happy. This is far from the first time this issue has come up, and Comcast has established a very clear pattern of trying everything in its power to force content companies to pay for uncongested access.

Yet with all of this Level 3 runs to the FCC screaming net neutrality. Wow. That takes balls. Comcast did itself no favors respnding with "it's a ratio issue" rather than laying out the situation.

If you refused to pay your transit provider, they'd probably just shut you off. The problem is that Comcast is too big to just shut off, and would no doubt tell it's customers that "Level 3 did it" (just like they have every other time someone has complained about their congested transits), that's why they're whining.

What I wonder is why Netflix and Comcast are letting middle men like Level 3 and Akamai jerk both of them around. These two folks need to get together and deal with each other, cutting out the middle man....

Netflix is a Comcast customer too (again well established publicly and easily provable via the global routing table), but they don't run their own server infrastructure, and Comcast doesn't offer a CDN service... The reality is that Level 3 offered Netflix a cut-throat price on CDN service to steal the business from Akamai, probably only made possible by the double dipping mentioned above. They were already in for a world of hurt based on their CDN infrastructure investment and the revenue they were able to extract from it, this certainly isn't going to help things. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Tue, Nov 30, 2010 at 07:53:25PM -0800, Leo Bicknell wrote:

I'm not privy to the deal, but I will point out as reported it makes no sense, so there is something else going on here. This is where both sids are hiding the real truth. I suspect it's one of two scenarios:

- Comcast demanded a lower price from Level 3, which Level 3 has spun as paying Comcast a monthly fee.

- Comcast said they would do settlment free peering with Level 3, in addition to, or in place of transit. Level 3 is spinning the cost of turning this up as paying Comcast a fee.

I suspect we'll not know what terms were offered for many years.

While obviously nobody is going to come out and officially acknowledge the exact terms on the NANOG mailing list, I'd say this is far too massive a leap of logic to make any kind of sense. Both Level 3 and Comcast seem to acknowledge that Comcast is asking for Level 3 to pay, is it really so hard to believe that this is the case? :)

Yes and no. First off, network neutrality is a vaguely defined term, so I'm not going to use it. Rather I'm going to say I think many people agree there is a concept that when it comes to traffic between providers there should be roughly similar terms for all players. Comcast shouldn't give Netflix a sweetheart deal while making Youtube pay through the nose.

Why shouldn't they? Charging different people different rates based on their willingness to pay is perfectly legal last I looked, and goes on in every industry. Personally I thought net neutrality was about not charging Netflix a special fee or else risk having their services "degraded" (in the same way that the mob makes sure "nothing bad happens" to your store :P), so they don't compete with an internal VOD service which doesn't get such fees applied. But obviously net neutrality is like "tier 1", you can apply any definition you'd like. :)

...

The funny part is that Level 3 was clearly ill prepared for the PR war, whereas Comcast, being the first mover (if not the first PR issuer), was well prepared.

Really? I just checked google news again, and the first statement I can find by either side was a Level 3 submission to business wire:

I believe that's what I said. To be perfectly clear, what I'm saying is: * Comcast acted first by demanding fees * Level 3 went public first by whining about it after they agreed to pay * Comcast was well prepared to win the PR war, and had a large pile of content that sounds good to the uninformed layperson ready to go.

...

The reality is that Level 3 offered Netflix a cut-throat price on CDN service to steal the business from Akamai, probably only made possible by the double dipping mentioned above. They were already in for a world of hurt based on their CDN infrastructure investment and the revenue they were able to extract from it, this certainly isn't going to help things. :)

I feel you undercut your network neutrality argument right here, because you make an argument that this is just two competitive businesses trying to get a leg up on each other. You can't have the fairness part of network neutrality and try and stab each other in the back at every step.

The net neutrality part comes from the fact that Level 3 can't just turn Comcast off for non-payment without risking massive impact to their customers. I'm pretty sure Level 3 is still allowed to charge people for transit services. If Comcast didn't want to buy from Level 3 they could have easily gone elsewhere, the part where the gov't steps in is when someone is abusing a monopoly/duopoly position.

Neither Level 3 nor Comcast here are interested in the fairness of network neutraility, or even interested in helping their customers. They are interested in hurting their "competitors" and boosting their own bottom line.

Probably true, but I'm sure someone somewhere (i.e. the consumers who have little to no choice in their home broadband) cares about the fairness just a little.

I bet the cash spent on lawyers and lobbiests taking this to the FCC on both sides could pay for enough backbone bandwidth and router ports to make this problem go away on both sides many times over. If they really cared about the customers experience and good network performance they would put away the press release swords, the various VP and CxO's egos, and come up with a solution.

Do you really think Comcast cares about the $50k router ports (by their own accounts, though personally I'd suggest they get off the CRS-1 tippe if they actually wanted to save some money :P), or might they actually be more interested in establishing themselves as a new Tier 1? :) At the end of the day both companies have made their share of mistakes, but I have a lot more respect for the ones who compete fairly and honestly, rather than by forcing people to use their services "or else". -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Wed, Dec 01, 2010 at 06:31:39AM -0800, Leo Bicknell wrote:

In a message written on Tue, Nov 30, 2010 at 10:59:25PM -0600, Richard A Steenbergen wrote:

...

I believe that's what I said. To be perfectly clear, what I'm saying is:

* Comcast acted first by demanding fees * Level 3 went public first by whining about it after they agreed to pay * Comcast was well prepared to win the PR war, and had a large pile of content that sounds good to the uninformed layperson ready to go.

I think I can make this very simple. What I am saying is that you're missing a step before your 3 bullet points. Before any of the three things you describe, Level 3 demanded fees from Comcast. Level 3 is doing a great job of getting folks to ignore that fact.

Do you have any basis for this claim, or are you just making it up as a possible scenario that would explain Comcast's actions? I have it on good authority that Level 3 did not attempt to raise their prices or ask for additonal fees beyond their existing contract, nor was their contract coming to term where they could "renegotiate" for more favorable terms. Comcast simply said, we've decided we don't want to pay you, you should pay us instead, and you're going to bend over and like it if you want to be able to reach our customers. Obviously the version I've heard and the version you're pitching can't co-exist, so either you have some REALLY interesting inside info that I don't (which I honestly find hard to believe given your knowledge of the facts so far), or you're stating a theory with no possible basis that I can find as a fact. If it's just a theory, please say so, then we don't keep having to argue these positions that can clearly never converge.

Comcast is a customer of L3, and pays them for service. Brining on Netflix will cause Comcast to pay L3 more. More interestingly, in this case it's likely Level 3 went to Comcast and said we don't think your existing customer ports will handle the additional traffic....so...um...you should buy more customer ports.

Comcast is th customer, they have complete and total control of the traffic being exchabged over their transit ports. If they wanted less traffic, they could announce fewer routes, or add more no-export communities. They also have complete control of traffic being sent outbound, and since Level3 is more than capable of handling 300Gbps (the capacity comcast claims they have), if Comcast actually had 300Gbps of outbound traffic to send they could easily have had a 1:1 ratio. Framing this as a peering ratio debate is absurd, because there two networks were NEVER peers. Any customer could have sent addtional bits to Level3 at any time, and Comcast should be prepared to deal with the TE as a result. That's life on the Internet.

Does network neutrality work both ways? If it is bad for Comcast to hold the users hostage to extort more money from Level 3, is it also bad for Level 3 to hold the content hostage to extort more money from Comcast?

You know, most people manage to buy sufficient transit capacity to support the volume of traffic that their customers pay them to deliver. Only Comcast seems to feel that it is proper to use their captive customer base hostage to extort content networks into paying for uncongested access. Level 3 is free to sell full transit or CDN to whomever they like, just as Comcast is free to not buy transit from Level 3 when their contract is up. The net neutrality part starts when Level 3 is NOT free to turn off their customer for non-payment just like what would happen to anyone else who suddenly decided they didn't think they should keep paying their bills, because Comcast maintains so little transit capacity that to shut them off would cause mssive disruptions to large portions of the Internet. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

I've collected my fav links (inc. nanog posts) on this topic on http://www.isoc-ny.org/p2/?p=1504. If there are issues with my brief explanation please let me know. j On Wed, Dec 1, 2010 at 12:34 PM, Leo Bicknell <bicknell@ufp.org> wrote:

Comcast has released additional details publically. Of course, this is their side of the story, so I wouldn't believe it hook line and sinker but it helps fill in the gaps.

http://blog.comcast.com/2010/11/comcasts-letter-to-fcc-on-level-3.html

-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/

-- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com Secretary - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------