GigaOm has begin tracking this story: http://gigaom.com/2010/11/30/a-play-by-play-on-the-comcast-and-level-3-spat On Tue, Nov 30, 2010 at 1:02 PM, Leo Bicknell <bicknell@ufp.org> wrote:
Having been involved with a few peering spats in the past I know what is said publically rarely matches the reality behind the scenes. In this particular case my spidy sense tells me there is absolutely something interesting behind the scenes, but the question is what.
I'd never really paid attention to how Netflix delivers its content. It's obviously a lot of bandwidth, and likely part of the issue here so I thought I would investigate.
Apparently Akamai has been the primary Netflix streaming source since March. LimeLight Networks has been a secondary provider, and it would appear those two make up the vast majority of Netflix's actual streaming traffic. I can't tell if Netflix does any streaming out of their own ASN, but if they do it appears to be minor.
Here's a reference from the business side of things: http://www.fool.com/investing/general/2010/11/11/netflix-takes-streaming-to-...
This is also part of the reason I went back to the very first message in this thread to reply:
In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. Gilmore wrote:
I understand that politics is off-topic, but this policy affects operational aspects of the 'Net.
Patrick works for Akamai, it seems likely he might know more about what is going on. Likely he can't discuss the details, but wanted to seed a discussion. I'd say that worked well.
I happen to be a Comcast cable modem customer. Gooling for people who had issues getting to Netflix streaming turned up plenty of forum posts with traceroutes to Netflix servers on Akamai and Limelight. I did traceroutes to about 20 of them from my cable modem, and it's clear Comcast and Akamai and Comcast and Limelight are interconnected quite well. Akamai does not sell IP Transit, and I'm thinking it is extremely unlikely that Comcast is buying transit from Limelight. I will thus conclude that these are either peering relationships, or that they have cut some sort of special "CDN Interconnect" deal with Comcast.
But what about Level 3? One of my friends I was chatting with on AIM said they thought Comcast was a Level 3 customer, at least at one time. Google to the rescue again.
Level 3 provides fiber to Comcast (20 year deal in 2004): http://blog.tmcnet.com/blog/tom-keating/voip/level-3-and-comcast.asp
Level 3 provides voice services/support to Comcast: http://cable.tmcnet.com/news/2005/jul/1168088.htm
Perhaps the most interesting though is looking up an IP on Comcast's local network here in my city in L3's looking glass: http://lg.level3.net/bgp/bgp.cgi?site=sjo1&target=68.86.240.141
Slightly reformatting for your viewing pleasure, along with my comments:
Community: North_America Lclprf_100 Level3_Customer # Level 3 thinks they are a customer United_States San_Jose EU_Suppress_to_Peers Suppress_to_AS174 # Cogent Suppress_to_AS1239 # Sprint Suppress_to_AS1280 # ISC Suppress_to_AS1299 # Telia Suppress_to_AS1668 # AOL Suppress_to_AS2828 # XO Suppress_to_AS2914 # NTT Suppress_to_AS3257 # TiNet Suppress_to_AS3320 # DTAG Suppress_to_AS3549 # GBLX Suppress_to_AS3561 # Savvis Suppress_to_AS3786 # LG DACOM Suppress_to_AS4637 # Reach Suppress_to_AS5511 # OpenTransit Suppress_to_AS6453 # Tata Suppress_to_AS6461 # AboveNet Suppress_to_AS6762 # Seabone Suppress_to_AS7018 # AT&T Suppress_to_AS7132 # AT&T (ex SBC)
So it would appear Comcast is a transit customer of Level 3 (along with buying a lot of other services from them). I'm going to speculate that the list of supressed ASN's are peers of both Level 3 and Comcast, and Comcast is going that so those peers can't send some traffic through Level 3 in attempt to game the ratios on their direct connections to Comcast.
Now a more interesting picture emerges. Let me emphasize that this is AN EDUCATED GUESS on my part, and I can't prove any of it.
Level 3 starts talking to Netflix, and offers them a sweetheart deal to move traffic from Akamai to Level 3. Part of the reason they are willing to go so low on the price to Netflix is they will get to double dip by charging Netflix for the bits and charging Comcast for the bits, since Comcast is a customer! But wait, they also get to triple dip, they provide the long haul fiber to Comcast, so when Comcast needs more capacity to get to the peering points to move the traffic that money also goes back to Level 3! Patrick, from Akamai, is unhappy at losing all the business, and/or bemused at the ruckus this will cause and chooses to kick the hornets nest on NANOG.
One last thing, before we do some careful word parsing. CDN's like Akamai and LimeLight want to be close to the end user, and the networks with end users want them to be close to the end user. It doesn't make sense to haul the bits across the country for any party involved. Akamai does this by locating clusters inside providers networks, LimeLight does it by provisioning bandwidth from their data centers directly to distribution points on eyeball networks.
So let's go back and look at the public statements now:
Level 3 said: http://www.marketwatch.com/story/level-3-communications-issues-statement-con... -2010-11-29?reflink=MW_news_stmp
"On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast's customers who request such content."
Comcast said: http://blog.comcast.com/2010/11/comcast-comments-on-level-3.html
"Comcast has long established and mutually acceptable commercial arrangements with Level 3's Content Delivery Network (CDN) competitors in delivering the same types of traffic to our customers. Comcast offered Level 3 the same terms it offers to Level 3's CDN competitors for the same traffic."
You can make both of these statements make sense if the real situation is that Comcast told Level 3 they needed to act like a CDN if they were going to host Netflix. Rather than having Comcast pay as a customer, they needed to show up in various Comcast distribution centers around the country where they could drop traffic off "locally". To Comcast this is the same deal other CDN's get, it matches their statement. To Level 3, this means paying a fee for bandwidth to these locations, and being that they are Comcast locations it may even mean colocation fees or other charges directly to Comcast. Comcast said if you're not going to show up and do things like the CDN players then we're going to hold you to a reasonable peering policy, like we would anyone else making us run the bits the old way.
The most interesting thing to me about all of this is these companies clearly had a close relationship, fiber, voice, and IP transit all on long term deals. If my speculation is right I'm a bit surprised Level 3 would choose to piss off such a long term large customer by bringing Netflix to the party like this, which is one of the reasons I doubt my speculation a bit.
But, to bring things full circle, neither of the public statements tell the whole story here. They each tell one small nugget, the nugget that side wants the press to run with so they can score political points.
Business is messy, and as I've said throughout this thread this isn't about peering policies or ratios, there are deeper business interests on both sides. This article: http://www.marketwatch.com/story/level-3-outlines-network-expansion-on-netfl...
Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix. I'm sure a lot of that is going to Comcast users, since they are the largest residential broadband ISP.
Messy. Very messy.
-- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/