"potential new and different architectural approach" to solve the Comcast - L3 dispute
Since it is Friday, maybe some of peering experts have some time to speculate what this new approach proposed by Comcast might be, as they assert it would represent "a significant shift of Internet infrastructure." http://www.lightreading.com/document.asp?doc_id=202121 http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3--... Well, their previous proposal was already representing quite a significant shift, and not in a good way, but I wonder what the new offer could be so that they figured it would be more acceptable to Level 3. I hope due to the speculative nature of the question it will not be considered off-topic. -Lorand Jakab
On Dec 17, 2010, at 9:57 AM, Loránd Jakab wrote:
Since it is Friday, maybe some of peering experts have some time to speculate what this new approach proposed by Comcast might be, as they assert it would represent "a significant shift of Internet infrastructure."
http://www.lightreading.com/document.asp?doc_id=202121 http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3--...
I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either. If anybody knows for sure, I'd be interested to hear. Cheers, -Benson
On Dec 17, 2010, at 9:57 AM, Lor=E1nd Jakab wrote:
Since it is Friday, maybe some of peering experts have some time to speculate what this new approach proposed by Comcast might be, as they assert it would represent "a significant shift of Internet = infrastructure." =20 http://www.lightreading.com/document.asp?doc_id=3D202121 = http://blog.comcast.com/2010/12/comcast-continues-discussions-with-level-3= ----offers-to-trial-new-solutions.html
I have no direct knowledge of the situation, but my guess: I suspect = the proposal was along the lines of longest-path / best-exit routing by = Level(3). In other words, if L(3) carries the traffic (most of the way) = to the customer, then Comcast has no complaint--the costs can be more = fairly distributed. The "modest investment" is probably in tools to = evaluate traffic and routing metrics, to make this work. This isn't = really *new* to the peering community, but it isn't normal either.
If anybody knows for sure, I'd be interested to hear.
How effective have variations on hot potato routing been, historically? I seem to recall Cogent made lots of noises early on about how they could do hot potato routing to encourage peering, but over the years that didn't seem to pan out that way. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Dec 17, 2010, at 11:23 AM, Joe Greco wrote:
How effective have variations on hot potato routing been, historically? I seem to recall Cogent made lots of noises early on about how they could do hot potato routing to encourage peering, but over the years that didn't seem to pan out that way.
I can't comment on Cogent... But, in general: hot-potato reduces network costs but doesn't eliminate them--more capacity is still required to carry more traffic. The goal is to balance out the cost, assuming the traffic is of adequate value (or equal value, ideally) to both networks. Cheers, -Benson
On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser <bensons@queuefull.net> wrote:
I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.
That is a reasonable guess, but Level3's FCC filing yesterday spells out with certainty that Level3 did offer to "cold potato" traffic onto Comcast (it does not mention the technical means e.g. MED honoring, CDN smarts, or otherwise) and that Comcast refused. I agree that the proposed Comcast solution may not be truly "new" but instead unusual, but unless "Backdoor Santa" tells us what they really have in mind, I suppose we won't know. If I were Comcast, I would want to move the significant cost of detailed netflow collection and analysis infrastructure onto backbone providers by wrapping that accounting mechanism up into my settlement agreements with peers, as well as the expense of a cost-ineffective network, and demand that Level3 and Comcast really calculate how much each network spends on each bit, and share in that cost. In theory, this is what happens when an ILEC opens a rate case with its state regulator; and it is how settlements for POTS calls work (at a very basic level.) Actually, if I were Comcast, I would focus on running my business more efficiently, as Level3 has thrown down the gauntlet with the FCC and requested that the FCC dictate to Comcast specifically, and explicitly all other broadband access providers, how they will interconnect with peers and transit suppliers. Level3 must think that their business would be better off with regulatory oversight of peering, or they would not have taken this action. Comcast should realize that, of the three potential motives for their recent actions I have previously outlined, #1 and #3 are not just highly unlikely, but would be practically impossible in a regulated environment. As such, they should further realize that their peering committee is driven by motive #2, ego, and find the best way to change their position without losing too much credibility. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
Level3 must think that their business would be better off with regulatory oversight of peering, or they would not have taken this action. Comcast should realize that, of the three potential motives for their recent actions I have previously outlined, #1 and #3 are not just highly unlikely, but would be practically impossible in a regulated environment. As such, they should further realize that their peering committee is driven by motive #2, ego, and find the best way to change their position without losing too much credibility.
-- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
Or maybe Level(3) thinks the entire game could potentially change and are attempting to head that off at the pass. What if instead of the end users paying for Internet service, the content providers did. Sort of like broadcast TV where the broadcasters pay the freight and the user simply turns on their device and they get content. In that model, the providers of the traffic pay the delivery costs of the content. So you would have "consumer" access that is mainly paid for by the content providers and "business" access which would be paid by the end users but would have less "consumer" traffic such as Netflix, Hulu, Facebook, Twitter, etc. If you look at the revenues being reported by some of these content providers, someone might be looking at those numbers saying "why *shouldn't* they pay? They are making money from the end users via ad sales just like broadcasters do, why shouldn't the model be the same?". I am not making any statement of my opinion, simply looking at a possibility. If there were such a sea change, Level3 now being a major content provider might find its long range plans have had a wrench thrown in them.
On Friday, December 17, 2010 12:51:02 pm George Bonser wrote:
What if instead of the end users paying for Internet service, the content providers did.
I've been following these threads with some interest, and even replying in a couple of places, but now it hits me that a sea change has already occurred, and it's the whole content provider / end user *thing* versus the original 'a host is a host is a host' IP *thing*. But content providers already pay more for their 'service' than the typical asymmetric-towards-the-customer bandwidth user does.
On 12/17/2010 12:45 PM, Lamar Owen wrote:
But content providers already pay more for their 'service' than the typical asymmetric-towards-the-customer bandwidth user does.
Agreed, though I think they pay less than most eyeball networks pay (the ISP, not the user), depending on where they host it (we have a lot of hauling we have to do). I'd also note, that the Internet is continuing to push more towards blurring the lines of content provider/eyeball, as p2p continues to be deployed with more technologies and for more uses. As households are constantly on, there is benefit in the household hosting content which can be reached directly by those you are sharing it to. As the market shifts to containing a larger market share of households with symmetric bandwidth, we can expect to see this improve (asymmetric last miles has hindered many innovations). Jack
On Fri, 17 Dec 2010, George Bonser wrote:
What if instead of the end users paying for Internet service, the content providers did. Sort of like broadcast TV where the broadcasters
Um. I'm a content provider. I pay a -lot- for internet service already. That's how my bits and bytes arrive in the tubes for those end users to recieve... -- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html
On Dec 17, 2010, at 1:59 PM, david raistrick wrote:
On Fri, 17 Dec 2010, George Bonser wrote:
What if instead of the end users paying for Internet service, the content providers did. Sort of like broadcast TV where the broadcasters
Um.
I'm a content provider.
I pay a -lot- for internet service already. That's how my bits and bytes arrive in the tubes for those end users to recieve...
+1 from here. Regards Marshall Eubanks AmericaFree.TV
-- david raistrick http://www.netmeister.org/news/learn2quote.html drais@icantclick.org http://www.expita.com/nomime.html
On Dec 17, 2010, at 11:35 AM, Jeff Wheeler wrote:
... Level3 must think that their business would be better off with regulatory oversight of peering, or they would not have taken this action.
And they might be correct in thinking that, if we assume the peering ecosystem is changing i.e. such that traditional "backbones" are being bypassed. Regulatory oversight might have the effect of locking-in today's interconnect regime, which would be ideal for Level(3). Cheers, -Benson
On Dec 17, 2010, at 12:35 PM, Jeff Wheeler wrote:
On Fri, Dec 17, 2010 at 12:15 PM, Benson Schliesser <bensons@queuefull.net> wrote:
I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.
That is a reasonable guess, but Level3's FCC filing yesterday spells out with certainty that Level3 did offer to "cold potato" traffic onto Comcast (it does not mention the technical means e.g. MED honoring, CDN smarts, or otherwise) and that Comcast refused. [...]
Comcast's latest: http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
http://fcc.gov/ NOTICE: The FCC website and related electronic filing systems and documents (except for NORS) will be unavailable beginning 6:00 p.m. (EST) Friday, December 17 through 6:00 a.m. (EST) Monday, December 20 for scheduled maintenance. :( On Fri, Dec 17, 2010 at 3:42 PM, Steve Schultze <sjs@princeton.edu> wrote:
That is a reasonable guess, but Level3's FCC filing yesterday spells out with certainty that Level3 did offer to "cold potato" traffic onto Comcast (it does not mention the technical means e.g. MED honoring, CDN smarts, or otherwise) and that Comcast refused. [...]
Comcast's latest: http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
-- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com VP (Admin) - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------
http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.htm... On Dec 17, 2010, at 10:25 PM, Joly MacFie wrote:
NOTICE: The FCC website and related electronic filing systems and documents (except for NORS) will be unavailable beginning 6:00 p.m. (EST) Friday, December 17 through 6:00 a.m. (EST) Monday, December 20 for scheduled maintenance.
:(
On Fri, Dec 17, 2010 at 3:42 PM, Steve Schultze <sjs@princeton.edu> wrote:
That is a reasonable guess, but Level3's FCC filing yesterday spells out with certainty that Level3 did offer to "cold potato" traffic onto Comcast (it does not mention the technical means e.g. MED honoring, CDN smarts, or otherwise) and that Comcast refused. [...]
Comcast's latest: http://fjallfoss.fcc.gov/ecfs/comment/view?id=6016064677
-- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com VP (Admin) - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------
On 12/18/2010 12:38 AM, Steve Schultze wrote:
http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.htm...
I very much doubt whether my comment on the blog will survive their moderation process, so here it is: === I am a Comcast residential HSI customer, and have many clients who are business HSI Comcast customers. At the same time, I do maintain servers in my own racks at a datacenter. What is not mentioned in this letter, is that Comcast is already being paid - by me, and by every other customer, for access to the content. Note that Comcast has never said that the Level3/Netflix issue is about users exceeding their allotted bandwidth (currently at about 250GB/month for residential); presumably, were a Comcast user to use 249GB of bandwidth downloading cute pictures of cats, Comcast would have no objection. It appears to be the specific issue that Netflix is a possible competitor to Comcast's TV business, that somehow causes Comcast to decide that there is a problem. Understand this: every Netflix video to be streamed, is specifically requested by a Comcast user, operating under the Comcast-advertised "High Speed Internet" service and presumably within the bandwidth caps that Comcast's own contract allows. That Comcast presumes to have the right to limit, modify, or decide for me which pieces of the Internet I can have access to, removes Comcast's common carrier protections, calls into question the truth of your advertisements for the HSI service, and raises the issue of whether Comcast is dealing in bad faith with each and every Comcast HSI subscriber. ==== --Patrick
On Sat, Dec 18, 2010 at 01:07:15AM -0500, Patrick Giagnocavo wrote:
Note that Comcast has never said that the Level3/Netflix issue is about users exceeding their allotted bandwidth (currently at about 250GB/month for residential); presumably, were a Comcast user to use 249GB of bandwidth downloading cute pictures of cats, Comcast would have no objection.
I believe they want the cat people to pay too, it's just easier to go after Netflix first. Lets say for a moment that Comcast's overall ratio with its customers is approximately the same as their ratio in the leaked Tata graphs (yes I know that this proves nothing, but lets just assume it for a moment), i.e. 5:1. They then ask that every network who sends them traffic, even their transit providers (in the case of Level 3) be under 2:1. What is the point of insisting on a ratio that is not supported by the traffic their customers actually request? Because it gives them a convenient excuse to demand payment from nearly everyone on the Internet for being out of ratio, and to restrict capacity to those who do not pay. With so many transit ports running hot, and even peering ports running hot as in the recent example where they intentionally turned down Global Crossing capacity (which they claim is settlement free) and CAUSED congestion, the ISP who hosts the cute cat pictures may have little choice but to pay Comcast for access, or risk losing their cute cat hosting business to someone else who is willing to do so. I've also seen Comcast ignore several offers to honor MEDs or accept more-specifics from networks who DO meet their published peering requirements in every way except ratios, so I don't think they're interested in technical solutions a potential transport cost imbalance either. If it was about anything other than trying to extract a toll from content providers, one of these technical solutions would clearly have been better for them then continuing to force the traffic into their congested transit ports, which they not only pay for, but then also do the backhaul for across their own network. BTW, they rejected my very nice comment on their blog asking if they would be willing to share the graphs of their transit provider interfaces (which are NOT peering relationships, and not under NDA) to back up their claims that the published graphs are false, so I'm positive yours isn't going to get through. :) -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
If I was Comcast and I got this deal I'd set up scripts to continuously spoof requests to Netflix, I mean hey I get paid for the traffic.. j -- --------------------------------------------------------------- Joly MacFie 218 565 9365 Skype:punkcast WWWhatsup NYC - http://wwwhatsup.com http://pinstand.com - http://punkcast.com VP (Admin) - ISOC-NY - http://isoc-ny.org ---------------------------------------------------------------
Richard A Steenbergen wrote:
BTW, they rejected my very nice comment on their blog asking if they would be willing to share the graphs of their transit provider interfaces (which are NOT peering relationships, and not under NDA) to back up their claims that the published graphs are false, so I'm positive yours isn't going to get through. :)
Seems as though, in both this case and Steve's case, Comcast is going out of their way to spin as much FUD as they can against those who dare speak out and are making a concerted effort to censor their "blog" (ie, press release machine). It's not going unnoticed and I hope they realize that instead of commenting on their blog, "we" will turn to the official FCC comment process. -Dave
+1 In fact, I feel that at home, I need fast, reliable internet access. I wish I could get that from one provider. Unfortunately, instead, I get fast internet service from Comcast (most of the time) and I get reliable internet service from Raw Bandwidth (DSL, 1.5mbps/768k). Owen (Comcast Business HSI customer) On Dec 17, 2010, at 10:07 PM, Patrick Giagnocavo wrote:
On 12/18/2010 12:38 AM, Steve Schultze wrote:
http://blog.comcast.com/2010/12/comcasts-responds-to-level-3s-fcc-filing.htm...
I very much doubt whether my comment on the blog will survive their moderation process, so here it is:
=== I am a Comcast residential HSI customer, and have many clients who are business HSI Comcast customers. At the same time, I do maintain servers in my own racks at a datacenter.
What is not mentioned in this letter, is that Comcast is already being paid - by me, and by every other customer, for access to the content.
Note that Comcast has never said that the Level3/Netflix issue is about users exceeding their allotted bandwidth (currently at about 250GB/month for residential); presumably, were a Comcast user to use 249GB of bandwidth downloading cute pictures of cats, Comcast would have no objection.
It appears to be the specific issue that Netflix is a possible competitor to Comcast's TV business, that somehow causes Comcast to decide that there is a problem.
Understand this: every Netflix video to be streamed, is specifically requested by a Comcast user, operating under the Comcast-advertised "High Speed Internet" service and presumably within the bandwidth caps that Comcast's own contract allows.
That Comcast presumes to have the right to limit, modify, or decide for me which pieces of the Internet I can have access to, removes Comcast's common carrier protections, calls into question the truth of your advertisements for the HSI service, and raises the issue of whether Comcast is dealing in bad faith with each and every Comcast HSI subscriber.
====
--Patrick
On Fri, Dec 17, 2010 at 11:15:14AM -0600, Benson Schliesser wrote:
I have no direct knowledge of the situation, but my guess: I suspect the proposal was along the lines of longest-path / best-exit routing by Level(3). In other words, if L(3) carries the traffic (most of the way) to the customer, then Comcast has no complaint--the costs can be more fairly distributed. The "modest investment" is probably in tools to evaluate traffic and routing metrics, to make this work. This isn't really *new* to the peering community, but it isn't normal either.
Nah, you're still thinking about this like it was a classic peering dispute over ratios, when nothing could be further from the truth. First off, by the very nature of a CDN, all of the Netflix/etc traffic is going to be delivered to the best exit on the long-haul network already. Second, Comcast is a FULL TRANSIT CUSTOMER of Level 3. Typically the customer gets to dictate the handoff point to the provider, by either advertising MEDs, or by sending inconsistent routes. The fact that the existing Level3/Comcast routing DOESN'T make Level 3 haul all of the bits to the best exit mean it's highly likely that Comcast agreeing to haul the bits was part of their commercial transit agreement, probably in exchange for lower transit prices. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Fri, Dec 17, 2010 at 12:48 PM, Richard A Steenbergen <ras@e-gerbil.net> wrote:
advertising MEDs, or by sending inconsistent routes. The fact that the existing Level3/Comcast routing DOESN'T make Level 3 haul all of the bits to the best exit mean it's highly likely that Comcast agreeing to haul the bits was part of their commercial transit agreement, probably in exchange for lower transit prices.
It's worth asking why Comcast did not accept Level3's suggestion that they use MED as a face-saving maneuver, which would have allowed both sides to declare victory. A) Comcast may already have the contractual right to use MED but chooses not to. I agree with you that this is unlikely, not for pure reasons of economics, but because Comcast has some of the same set of motives not to send MED to their transit provider as every other network: prefix aggregation, quality control, and ego. I'll discount geography, marketing, and inability to calculate useful MED values. For argument's sake, let's say they currently can start sending MEDs to Level3 whenever they want. This being the case, Level3's "offer" would have amounted to Level3 telling Comcast upper management that Comcast's engineering people are leaving a huge amount of money on the table, that Level3 is far more cost-effective at running its long-haul network than Comcast, and that they should leave the big networking to the big boys. Comcast management could either react badly to this, or go back to their network folks and ask why they can't be as cost-effective as Level3. B) Comcast may not be able to use MED today. In this case, management may be asking themselves why. An essentially similar scenario can play out; they can either react badly to Level3, or ask their own staff why they are wasting money. C) Comcast doesn't care about MED or the actual cost of doing business. They are boldly moving towards a future that is opposite the one "net neutrality" folks advocate, one that looks like my "Comcast Motive #3." D) Comcast does not think that beginning to use MED (whether currently enabled or not) is enough to satisfy the federal regulators and legislators who are now taking interest in this game of interconnection brinkmanship, involving 17 million households, between a major IP carrier delivering traffic from everyone including a household name like Netflix, and a major cable company that is waiting for government approval to purchase NBC. They feel they must demand something very concrete to demonstrate that they are looking out for consumers' best interest, which means they must make Level3 and/or Netflix look like the bad guy. E) Comcast thinks that a system of accounting for the cost of bearing traffic and dividing it among the involved parties will actually be good for their business, because they can over-build their infrastructure as much as they like, perhaps even improving quality for end-users, and only have to pay for about half of it. The cost of being inefficient, stupid, or committing purchasing or forecasting errors drops by half. This looks very much like my "Comcast Motive #1." E1) Comcast may also know a thing or two about Hollywood Accounting. If you do not understand this reference, simply look it up on Wikipedia. It suffices to say that cost/revenue sharing agreements of this nature can be manipulated in gross ways to the advantage of the party doing the bulk of the book-keeping. F) Management has the same case of ego-driven decision-making that their technical staff have demonstrated. I find this unlikely but still possible. We all know this has been the case at the CEO level in some major interconnection disputes of the past. I believe this outlines the reasonable scenarios for Comcast avoiding a face-saving maneuver with Level3. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator / Innovative Network Concepts
participants (16)
-
Benson Schliesser
-
Dave Temkin
-
david raistrick
-
George Bonser
-
Jack Bates
-
Jeff Wheeler
-
Joe Greco
-
Joly MacFie
-
Lamar Owen
-
Loránd Jakab
-
Marshall Eubanks
-
Owen DeLong
-
Patrick Giagnocavo
-
Richard A Steenbergen
-
Steve Schultze
-
Steve Schultze