Re: Some truth about Comcast - WikiLeaks style
Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses: my management is pretty disgusted with the badmouthing and accusation slinging on nanog.org btw the demands to disclose confidential data on the blog aren't helping either the budget for hosting will be impacted I guarantee because it came out of folks who are being hassled's budget there is a meeting today to discuss the value of supporting the NANOG community Apparently Comcast's support and sponsorship of NANOG has actually been a ploy to buy our silence, and if we keep talking poorly of them they're going to cut off the funding. Shhhhh don't tell anyone.
On Thu, Dec 16, 2010 at 9:53 AM, Backdoor Parrot <backdoorparrot@hotmail.com> wrote:
Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses:
my management is pretty disgusted with the badmouthing and accusation slinging on nanog.org btw the demands to disclose confidential data on the blog aren't helping either the budget for hosting will be impacted I guarantee because it came out of folks who are being hassled's budget there is a meeting today to discuss the value of supporting the NANOG community
Apparently Comcast's support and sponsorship of NANOG has actually been a ploy to buy our silence, and if we keep talking poorly of them they're going to cut off the funding. Shhhhh don't tell anyone.
Any more details to those logs? Timestamps, channel names, nicknames? -- Brent Jones brent@servuhome.net
Pardon my ignorance here but what does Comcast do for the NANOG community? I know they attend many conferences and share their experiences with a lot of us which is very much appreciated... Just asking ;) -----Original Message----- From: Backdoor Parrot [mailto:backdoorparrot@hotmail.com] Sent: December-16-10 12:53 PM To: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses: my management is pretty disgusted with the badmouthing and accusation slinging on nanog.org btw the demands to disclose confidential data on the blog aren't helping either the budget for hosting will be impacted I guarantee because it came out of folks who are being hassled's budget there is a meeting today to discuss the value of supporting the NANOG community Apparently Comcast's support and sponsorship of NANOG has actually been a ploy to buy our silence, and if we keep talking poorly of them they're going to cut off the funding. Shhhhh don't tell anyone.
On Dec 16, 2010, at 1:37 PM, Paul Stewart wrote:
Pardon my ignorance here but what does Comcast do for the NANOG community? I know they attend many conferences and share their experiences with a lot of us which is very much appreciated...
I'm sure the concern is that Comcast signed up to return NANOG (newNOG?) to philly. I think they may be overly sensitive to some of the comments, just as if people were posting similar comments about my employer, I would likely be sensitive. (Also there are a lot of people who post stuff but don't actually attend NANOG meetings. There is this overlap but disjoint as well between the two in my experience. Hope everyone is wearing their teflon pants). Aside from the 'public comments', the leaked graphics (which I personally would believe are accurate, but the motives of the leakers not obvious), I don't directly have a role here. I understand comcast has a lot of infrastructure and costs. Likely more fiber than the incumbent telcos, and they are constrained by a variety of local business conditions from doing what may be a more optimal solution for themselves. All that said, the whole issue of 'local content' is going to continue to rage on for years to come. Getting the content closer to the end user is going to be a key to reducing costs for the long-tail providers to homes and businesses. Should it be incumbent on the CDNs to pay for colo at the headend? That's a business decision that will entirely be driven by these ongoing disputes. It surely feels like we are slowly going down the road of telco-style settlement based on call direction. I've observed some trends that point at this happening when someone has a fortress they wish to defend, monetize or subsidize further. Will it win out? I'm not entirely sure. - Jared
On Dec 16, 2010, at 1:58 PM, Jared Mauch wrote:
On Dec 16, 2010, at 1:37 PM, Paul Stewart wrote:
Pardon my ignorance here but what does Comcast do for the NANOG community? I know they attend many conferences and share their experiences with a lot of us which is very much appreciated...
I'm sure the concern is that Comcast signed up to return NANOG (newNOG?) to philly.
They also were the sponsor for IETF-71 in Philly in 2008. Regards Marshall
I think they may be overly sensitive to some of the comments, just as if people were posting similar comments about my employer, I would likely be sensitive. (Also there are a lot of people who post stuff but don't actually attend NANOG meetings. There is this overlap but disjoint as well between the two in my experience. Hope everyone is wearing their teflon pants).
Aside from the 'public comments', the leaked graphics (which I personally would believe are accurate, but the motives of the leakers not obvious), I don't directly have a role here. I understand comcast has a lot of infrastructure and costs. Likely more fiber than the incumbent telcos, and they are constrained by a variety of local business conditions from doing what may be a more optimal solution for themselves.
All that said, the whole issue of 'local content' is going to continue to rage on for years to come. Getting the content closer to the end user is going to be a key to reducing costs for the long-tail providers to homes and businesses. Should it be incumbent on the CDNs to pay for colo at the headend? That's a business decision that will entirely be driven by these ongoing disputes.
It surely feels like we are slowly going down the road of telco-style settlement based on call direction. I've observed some trends that point at this happening when someone has a fortress they wish to defend, monetize or subsidize further. Will it win out? I'm not entirely sure.
- Jared
All that said, the whole issue of 'local content' is going to continue to rage on for years to come. Getting the content closer to the end user is going to be a key to reducing costs for the long-tail providers to homes and businesses. Should it be incumbent on the CDNs to pay for colo at the headend? That's a business decision that will entirely be driven by these ongoing disputes.
What I still don't understand is this (and please pardon my ignorance): If the issue is the costs that long-tail providers must bear to transit content across their own network, and the solution is to move the content closer to the providers' customers, (why) is the content provider obligated to subsidize that? If collocating equipment to the headend is truly the correct response (if it truly reduces the ISP's costs to provide access to that content, and truly results in a better customer experience), then surely the savings would cover the ISP's cost of collocating equipment at that ISP's own headends? It seems reasonable to expect that a content provider come up with the equipment to be collocated, as well as bear the cost-burden of supporting that equipment, so there can't be a significant capex for the ISP... The idea of buying colocation from a last-mile ISP to reduce that last-mile ISP's costs seems (at first glance) to be a hysterically unfair proposition - though it seems that incumbent ISPs may have great enough leverage to extract this revenue if they really want to. Or am I off my rocker? What is in the best interests of the customer? Nathan
That seems to be "Off Topic". The operational implications for most of us is, most likely, much more technical bookkeeping and data storage. On Dec 16, 2010, at 2:24 PM, Nathan Eisenberg wrote:
What is in the best interests of the customer?
Nathan
James R. Cutler james.cutler@consultant.com
On Dec 16, 2010, at 2:24 PM, Nathan Eisenberg wrote:
The idea of buying colocation from a last-mile ISP to reduce that last-mile ISP's costs seems (at first glance) to be a hysterically unfair proposition - though it seems that incumbent ISPs may have great enough leverage to extract this revenue if they really want to. Or am I off my rocker?
What is in the best interests of the customer?
I think the balance here is: If you can buy wholesale IP for $X/meg from a generic provider that delivers the bits to all destinations vs Buying local IP for $Y (where Y>X) in the local network access, you will pay the $X rate. If there were some price advantage for the CDNs, I doubt the discussion would be happening at all. Some people call this "dumping", others call it market forces. I'm not sure debating the business merits here make sense, as I'm neither comcast nor a CDN, and all my data is based on similar 'backdoor' or 'whisper' comments over many years. I seriously doubt the CDNs care about much other than the price:quality ratio. Clearly what happened here was a business decision that has been dragged out too long in public. If you can't figure that out from this thread yet, you may not "get it" even if you saw an xls telling you the same thing. Most of the companies involved are publicly traded, read their 10-K's and extrapolate the costs and pressures. Either the costs involved here represent enough to be material and something to be noted in a filing at edgar, or they are people fighting over loose change. - Jared
On 16-Dec-2010, Paul Stewart <paul@paulstewart.org> sent:
Pardon my ignorance here but what does Comcast do for the NANOG community? I know they attend many conferences and share their experiences with a lot of us which is very much appreciated...
Just asking ;)
http://nanog.org/meetings/nanog46/ -- Chip Marshall <chip@2bithacker.net> http://weblog.2bithacker.net/ KB1QYW PGP key ID 43C4819E v4sw5PUhw4/5ln5pr5FOPck4ma4u6FLOw5Xm5l5Ui2e4t4/5ARWb7HKOen6a2Xs5IMr2g6CM
the demands to disclose confidential data on the blog aren't helping either
It's always interesting how things like bandwidth displays are considered "confidential data" particularly when they show something bad. The best service providers will actually provide the statistics without being asked, even to the public, for example: https://noc.iphouse.com/?skin=print Comcast may need a reminder that an Internet Service Provider's job is to provide internet service to its customers. If you cannot do the job, open up your infrastructure to sharing and let someone else have a go at it. This leveraging-captive-customers-to-get-money-from- others game is fundamentally dirty, at least if the rumors about your transit connections are true. Which probably brings us around to the reasons that it'd be interesting to have Comcast volunteer the information. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Dec 16, 2010, at 11:53 AM, Backdoor Parrot wrote:
Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses: (snip)
With all due respect, logs or GTFO. I can find no mention of this outside of your email. I would expect there to be quite a few mentions of such a statement made in "a public IRC channel with many witnesses".
On 12/16/2010 09:38 AM, Daniel Seagraves wrote:
On Dec 16, 2010, at 11:53 AM, Backdoor Parrot wrote:
Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses: (snip)
With all due respect, logs or GTFO. I can find no mention of this outside of your email. I would expect there to be quite a few mentions of such a statement made in "a public IRC channel with many witnesses".
So far this whole thing disturbs me. We've gone from "Backdoor Santa" dropping graphs that we can't specifically attribute to Comcast, through to "Backdoor Parrot" now adding IRC communication that yet again we can't attribute to Comcast. In the former case we've gone from disbelief through to academic "what if", swiftly moving on to damning accusation without there being /any /supporting evidence, as far as I can see, that the graphs are anything to do with Comcast. I fear we're likely to see the same results from these IRC logs. All we're ending up with is what is mostly hearsay being treated as facts. Paul
On Thu, 2010-12-16 at 09:47 -1000, Paul Graydon wrote:
(...) All we're ending up with is what is mostly hearsay being treated as facts.
One consumer organization in France during the ongoing debate with regulators on network neutrality called for network operator to publish some verifiable information on their bandwidth issues: http://www.arcep.fr/index.php?id=10387 http://www.alain-bazot.fr/index.php/neutralite-du-net-n-oublions-pas-l-inter... http://www.pcinpact.com/actu/news/55827-alain-bazot-neutralite-ufc-arcep.htm Alain Bazot, president of "UFC - Que Choisir" a well-known french consumer organization wrote on his blog: << (...) Avant toute intervention, l’opérateur devrait prouver qu’il y a un réel problème sur son réseau, comme une congestion. Alors que les témoignages quant à la réalité de la saturation des réseaux divergent, cette condition me semble essentielle. (...)
My poor translation: << (...) Before any change the network operator must prove he has a real congestion issue. Since informations on the reality of network saturation are divergent, this condition seems essential to me. (...)
Regulators and the public need data for proper regulation and future changes in regulation, and the issue is the same everywhere :). Sincerely, Laurent PS: sorry for my miscalculation AMSIX 1.2Tbit/s cost is $2.25 per month per Comcast subscriber assuming 16 millions customers and $30/Mbit/s/month transit but as pointed out by participants of this list for a 10G port at Comcast cost is likely to be closer to $3 Mbit/s so it all cancels out to my original erroneous $0.225 :).
Earlier this morning a Comcast peering manager had the following things to say about the recent NANOG thread, in a public IRC channel with many witnesses: (snip)
With all due respect, logs or GTFO. I can find no mention of this outside of your email. I would expect there to be quite a few mentions of such a statement made in "a public IRC channel with many witnesses".
I was in the IRC channel at the time and saw it. It's real. I don't support the posting of IRC logs, but can't control that either. Randy
On Thu, Dec 16, 2010 at 02:48:56PM -0500, Randy Epstein wrote:
I was in the IRC channel at the time and saw it. It's real.
I don't support the posting of IRC logs, but can't control that either.
I saw it too. I don't support posting of IRC logs trying to get people "in trouble" (though lord knows it wouldn't be the first time that has happened :P), but I also completely disagree with Comcast's position on this (big shocker, I know). As one of the people who has spoken out against Comcast's actions the most vocally, I suppose the original sentiment might very well be targeted at me. Personally I really don't think that people on the NANOG list posting about their network issues or actions has ANYTHING to do with their sponsorship of the NANOG conferences or community, and I suppose I should be shocked and appalled that it might come down to these type of threats to silence people who have something negative to say. I'm a Comcast customer too (50M/10M or 6M/768K DSL at home, gee, decisions decisions :P), what are they going to do next, shut off my cable modem for TOS violations? :) Seriously guys, this is an operator forum and you're running a congested network, to expect that people are not going to comment on those facts just because you've put money into NANOG sponsorship is absurd. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On Thu, Dec 16, 2010 at 02:13:47PM -0600, Richard A Steenbergen wrote:
Seriously guys, this is an operator forum and you're running a congested network, to expect that people are not going to comment on those facts just because you've put money into NANOG sponsorship is absurd.
Forgot to attach a giant disclaimer on the previous post: I'm speaking solely for myself, and not in any way, shape, or form, for the NANOG, NewNOG, or any other organization. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
On 12/16/2010 2:13 PM, Richard A Steenbergen wrote:
On Thu, Dec 16, 2010 at 02:48:56PM -0500, Randy Epstein wrote:
I was in the IRC channel at the time and saw it. It's real.
I don't support the posting of IRC logs, but can't control that either.
I saw it too. I don't support posting of IRC logs trying to get people "in trouble" (though lord knows it wouldn't be the first time that has happened :P), but I also completely disagree with Comcast's position on this (big shocker, I know).
I think the "post the logs" comment was due to no one speaking up as to having seen the quote. Anonymous posting without collaborating evidence is useless. I may not agree with Comcast, but I also can't agree with people quoting them without any evidence; and multiple people having seen it is acceptable evidence. Jack
participants (15)
-
Backdoor Parrot -
Brent Jones -
Chip Marshall -
Cutler James R -
Daniel Seagraves -
Jack Bates -
Jared Mauch -
Joe Greco -
Laurent GUERBY -
Marshall Eubanks -
Nathan Eisenberg -
Paul Graydon -
Paul Stewart -
Randy Epstein -
Richard A Steenbergen