gin-nto-icore1 is a Tata router at Equinix in NY. Whether or not that port belongs to Comcast is anyone's guess. Jeff On Tue, Dec 14, 2010 at 1:39 AM, Justin M. Streiner <streiner@cluebyfour.org> wrote:

On Tue, 14 Dec 2010, Jack Bates wrote:

...

On 12/13/2010 11:07 PM, Backdoor Santa wrote:

...

 Ever wonder what Comcast's connections to the Internet look like? In the  tradition of WikiLeaks, someone stumbled upon these graphs of their TATA  links.

Forgive me for being the skeptic, but I presume there is at least a traceroute  with rDNS mentioning one of the 3 10G interfaces on gin-nto-icore1 from comcast?

It's not like the image lists the customer name on it; disregarding photoshop concerns. At least wikileaks documents look like they came from the government and have lots of details. :)

Agreed.  There's no independently verifiable detail to lend any credence to the source(s) of the data.  It just shows some 10G links flat-topping due to saturation.  There's not enough here to get particularly excited.

jms

-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions

* Richard A Steenbergen

FWIW these graphs pretty much reflect the massive congestion that I've been observing between Tata and Comcast. I've also seen some third party Smokeping graphs which visually show the rate of loss, and the pattern looks very very similar, but I'll let someone who actually maintains them be the one to post them.

Voxel have also reported seeing congestion to Comcast via Tata: http://www.voxel.net/blog/2010/12/peering-disputes-comcast-level-3-and-you Best regards, -- Tore Anderson Redpill Linpro AS - http://www.redpill-linpro.com Tel: +47 21 54 41 27

On Tue, Dec 14, 2010 at 1:53 AM, Rettke, Brian <Brian.Rettke@cableone.biz>wrote:

I don't see anything listed that indicates operation that is at all different from any other service provider network.

Yeah, the 30 day looks like a classic uptick in traffic toward the holidays. Some bellhead beancounter maybe took out capacity in the summer lull and ignored the engineers. Or they just have stupidly-slow install intervals. Same crap I've seen on loads of provider networks.

The "capacity" issue listed is not an issue at all. It's simply inciting anger and the same rhetoric that pollutes the legitimate discussion of backbone network constraints.

When you shout "conspiracy" without offering verifiable facts, and not accounting for the cost (and time) it takes to upgrade networks (much less the fact that it requires capacity upgrades on both sides, in this case between TATA and Comcast), it makes the whole argument invalid in my opinion.

If they wanted to be tru to the claim of "wikileaks style" in the subject line, they'd have an actual memo from some executive stating the policy of purposefully starving traffic. Never attribute to malice* *that which is adequately explained by stupidity.

That and the "backdoor santa" thing makes me believe the whole thread is designed to flame rather than promote the discourse that is the hallmark of NANOG. I really hope that there are moderators about to verify this: With these kinds of people about I'm less likely to post anything of substance.

Sincerely,

Brian

-----Original Message----- From: Mikael Abrahamsson [mailto:swmike@swm.pp.se] Sent: Monday, December 13, 2010 11:45 PM To: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style

On Mon, 13 Dec 2010, Backdoor Santa wrote:

...

Another thing to notice is the ratio of inbound versus outbound. Since Comcast is primarily a broadband access network provider, they're going to have millions of eyeballs (users) downloading content.

Actually, there are plenty of access providers with 2:1 ratio (more ul than dl). It's not a matter if you're access provider or not, it's a matter if you offer decent upstream speed or not.

In my experience, someone with 10/10 megabit/s ETTH compared to someone with 24/1 ADSL will download the same amount of data on average, but the 10/10 will have four (4) times more upload usage, bringing the ratio from 2:1 (Dl:Ul) on ADSL to 1:2 (Dl:Ul) on ETTH.

So because Comcast is offering low upload speeds, they'll have low outgoing amount of traffic compared to incoming. With more and more ISPs offering more symmetric dl/ul speeds, we'll approach 1:1 ratio more and more...

-- Mikael Abrahamsson email: swmike@swm.pp.se

On Tue, 2010-12-14 at 16:20 -0500, Ricky Beam wrote:

On Tue, 14 Dec 2010 11:24:45 -0500, Craig L Uebringer <cluebringer@gmail.com> wrote:

...

Same crap I've seen on loads of provider networks.

No ISP I've ever worked for or with has ever willingly ran their transit (or peering) links at capacity.

(Granted, I've been responsible for saturating links, but I moved user traffic off of them first.)

--Ricky

PS: TATA confirmed Comcast's behavior before anyone found any traffic graphs. We already knew they were gaming their own customer base.

According to: http://en.wikipedia.org/wiki/Comcast "Comcast has 15.930 million high-speed internet customers" If a 10G port for transit is paid by comcast $30/Mbit/s monthly that's 0.19 cent/internet customer/month for a new 10G port to properly desaturate this particular link. Did I compute something wrong? Laurent

On Wed, 2010-12-15 at 05:31 -0500, Randy Epstein wrote:

Laurent,

...

If a 10G port for transit is paid by comcast $30/Mbit/s monthly that's 0.19 cent/internet customer/month for a new 10G port to properly desaturate this particular link.

...

Did I compute something wrong?

...

Laurent

Yes, now you need to multiply that by the numerous other ports that have the same conditions and need upgrades.

If I look at: http://www.ams-ix.net/statistics/ That's 1.2 Tbit/s peak for European biggest IX so 120 10G ports so about 22 cent/customer/month assuming Comcast alone generates this kind of bandwidth and pays what mom & pop AS pay for transit. It still doesn't compute to me... Laurent

On Dec 15, 2010, at 10:09 AM, ML wrote:

...

According to: http://en.wikipedia.org/wiki/Comcast "Comcast has 15.930 million high-speed internet customers"

If a 10G port for transit is paid by comcast $30/Mbit/s monthly that's 0.19 cent/internet customer/month for a new 10G port to properly desaturate this particular link.

Did I compute something wrong?

Laurent

Assuming that I did my math right.

It's actually 1.9 cents/month/per customer.

Assuming they pay $30/meg...

Assuming you understand some networks end-to-end costs without access to their data may result in suboptimal outcomes. - Jared

On 2010-12-15-12:15:47, Kevin Neal <kevin@safelink.net> wrote:

Also assuming the backbone and distribution upgrades required between their data centers and their customers costs nothing. It's not free to get bandwidth from Point A (port with TATA) to Point B (Customer).

I don't see how this point, however valid, should factor into the discussion. Missing from this thread is that Comcast's topology and economics for hauling bits between a neutral collocation facility and broadband subscriber are the _same_ whether they ingest traffic by way of a settlement-free peer, customer, or paid transit connection. Speaking to Richard's earlier observations, we too have run into issues attempting to deliver content by way of Comcast's Tata transit, dating back to July of this year. (It's possible the issues might have begun sooner, however this is as far back as our analytics go. I've actually been spending some time documenting how we've been measuring this loss, and how folk might measure it on their production infrastructure utilizing policy routing, routing-instances, and the like -- any interested content folk are welcome to contact me off-list. Suffice it to say, configs are the easy part, the hard part is building a statistically valid sample set without degrading connectivity for paying customers...) Whatever the cause, five months should be ample time to turn up some additional transit capacity or otherwise work around the issues; we're talking commodity transit ports in neutral facilities, such as Equinix sites, after all. What we have here is Comcast holding its users captive, plain and simple. They have established an ecosystem where, to reach them, one must pay to play, otherwise there's a good chance that packets are discarded. Alternate paths simply aren't there, given the no-export communities deployed. As it stands, I could multi-home to NTT, Telia, Tata, and XO, and still get stuck with no good paths to Comcast. While this has happened before (see: DTAG, FT, ...), this is probably the first we've seen it occur in the United States, at scale. Folk in content/hosting should find this all more than a little bit scary. -a

This should also be a wake-up call that for whatever reason (who cares what for this discussion), if our bandwidth demands exceed our bandwidth supply, we must become more efficient at using our bandwidth. I'm hoping that we not only discuss peering and bandwidth, management and implementation, but look at the Content providers with the same level of scrutiny that we hold the Backbone transit providers to. We should look at video compression and codecs with the same level of urgency that we do bandwidth, because there will never be enough if both sides are not looked at. Sincerely, Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services -----Original Message----- From: Jack Bates [mailto:jbates@brightok.net] Sent: Wednesday, December 15, 2010 6:05 PM To: Adam Rothschild Cc: Kevin Neal; nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style On 12/15/2010 4:47 PM, Adam Rothschild wrote:

Folk in content/hosting should find this all more than a little bit scary.

So you don't think the money content providers will pay Comcast won't reflect on other eyeball networks who aren't important/large enough to request financing? ie, Comcast could run lower rates and offer better service by charging the content provider, while competitive eyeball networks won't get the option to receive compensation from content providers and have to charge appropriate rates to their customers. Jack

On Wed, 15 Dec 2010 19:05:26 CST, Jack Bates said:

request financing? ie, Comcast could run lower rates and offer better service by charging the content provider, while competitive eyeball networks won't get the option to receive compensation from content providers and have to charge appropriate rates to their customers.

Yes, Comcast *could* do that. But let's stick to plausible scenarios, OK?

On Wed, Dec 15, 2010 at 5:47 PM, Adam Rothschild <asr+nanog@latency.net> wrote:

I don't see how this point, however valid, should factor into the discussion.  Missing from this thread is that Comcast's topology and economics for hauling bits between a neutral collocation facility and broadband subscriber are the _same_ whether they ingest traffic by way of a settlement-free peer, customer, or paid transit connection.

Given that transit must be an incredibly small portion of Comcast's cost to provide IP service to its customers, I think there are only three possible reasons why Comcast would focus so much energy on congesting transit to force content networks to purchase connectivity for them, rather than upgrade transit or engage in more peering: 1) Comcast believes they can exact a great deal of revenue from content networks. For this to be comparable to their captive customers, per-megabit rates must be reminiscent of pre-Level3 days, when $30/Mb was a bargain. This would spell bad news for Netflix. Of course, since cable companies typically must pay network affiliates and media companies great sums for television programming packages, it is in direct opposition to the TV content/delivery model. It would be hard to argue both sides if both businesses were faced with like-minded regulators. 2) Comcast is making its engineering decisions in an ego-driven manner, with little or no practical basis for their peering or transit purchasing strategy. 3) Comcast is hoping the phrase "net neutrality" becomes a thing of the past, and that, at some point in the future, they will be free to block or QoS down anyone they please, including content networks, search engines, or MP3 stores that compete with their own offerings. I bet Comcast would love to have a few cents off every iTunes purchase through their network, a handling charge for every amazon.com transaction, or to coerce a million Netflix subscribers into a Comcast-owned service. This is as good a way as any for Comcast to argue their side to potential regulators. In any case, the "net neutrality" side gains credibility anytime a media company can be made to look like they are constraining users' choices by exacting a price from content providers. There has been talk of regulating Internet peering on this list since DIGEX disconnected from ANS, if not before. Reasons in favor of doing it continue to become easier for a lay-person to understand. In my state, there is a law against walking down the street with an ice cream cone in your pocket. I don't know the origin of that law, but I strongly suspect some person did it enough times, for a dumb enough reason, to attract legislative interest. Comcast should keep that in mind before engaging in further peering brinkmanship. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator  /  Innovative Network Concepts

On Thu, Dec 16, 2010 at 12:15 PM, Dave Temkin <davet1@gmail.com> wrote:

I disagree.  Even at $1/Mbit and 6Tbit of traffic (they do more), that's still $72M/year in revenue that they weren't recognizing before.  Given that that traffic was actually *costing* them money to absorb before, turning the balance and making that kind of money would be very favorably looked upon in

Yeah, because it makes a lot of sense to fuck with a billion dollar a month revenue stream so you can extract a few million dollars more per month from IP carriers. This definitely makes more sense than, say, running the billion dollar a month side a little more efficiently. You need to understand the scale of comcast's expenses and revenue on the access and transport side of their business, in order to have a remotely intelligent opinion about whether or not they are doing anything smart with the peering/transit side, in these conditions. If you really think it's a good idea to attract the attention of government regulators, newspapers, customers, and every major ISP by making a lot of noise over something that might allow you to make 0.5% more money off a product where you could probably save an order of magnitude more money through any number of ignored efficiencies within the organization, I would love for you to post that. I suspect that most folks who are of the opinion that Comcast is motivated by anything but the three things I mentioned have not clearly considered the proportionately small benefit they could gain from selling access to their network at anything approaching a nominal fee. It must be either 1) very high per-Mb price; 2) ego and stupidity; 3) greed of such magnitude that it would make Gordon Gecko proud. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator  /  Innovative Network Concepts

On Dec 16, 2010, at 10:53 AM, Dave Temkin wrote:

Jeff Wheeler wrote:

...

On Thu, Dec 16, 2010 at 12:15 PM, Dave Temkin <davet1@gmail.com> wrote:

...

I disagree. Even at $1/Mbit and 6Tbit of traffic (they do more), that's still $72M/year in revenue that they weren't recognizing before. Given that that traffic was actually *costing* them money to absorb before, turning the balance and making that kind of money would be very favorably looked upon in

Yeah, because it makes a lot of sense to fuck with a billion dollar a month revenue stream so you can extract a few million dollars more per month from IP carriers. This definitely makes more sense than, say, running the billion dollar a month side a little more efficiently.

You need to understand the scale of comcast's expenses and revenue on the access and transport side of their business, in order to have a remotely intelligent opinion about whether or not they are doing anything smart with the peering/transit side, in these conditions.

I do. And yes, they are happy to "fuck with a billion dollar a month revenue stream" (that happens to be low margin) in order to set a precedent so that when traffic is 60Tbit instead of 6Tbit, across the *same* customer base they have today that's insisting on getting that $19.99/month promo deal for life they make up the infrastructure investment on the backs of the content providers and not their customers. $1B/month from your customers + $1B/month from your content providers is what they'd ideally like to see and this is just laying the groundwork for it.

They have a captive audience. What percentage of their customers who they're offering 10Mbit+ connections to do you think have a 10Mbit+ alternative? It's not very many.

-Dave

Well said, Dave... I've been mostly ignoring this thread in recent days because I had pretty much said my piece. However, if people still aren't understanding that Comcast is attempting to leverage a monopoly here for anti-competitive ends, it boggles the mind. Level3 is no angel, either. IMHO, both organizations are posterchildren for burdensome regulation (no, I'm not a fan of burdensome regulation). The world needs more open peering policies and denser connection between networks. Recouping access costs on the backs of content providers is absurd. So is trying to recoup the costs of content delivery on the backs of access networks (Level3's traditional model). I suspect that is what will happen in the long run. THe question now is whether it will happen through cooperative competition as is the tradition of the internet, or, whether these bozos will force the government into turning it into a system like the telco settlements that made telephony so expensive for so long. Owen

-----Original Message----- From: Jeff Wheeler [mailto:jsw@inconcepts.biz] Sent: Thursday, December 16, 2010 1:22 PM To: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style

...

I do.  And yes, they are happy to "fuck with a billion dollar a month revenue stream" (that happens to be low margin) in order to set a

On Thu, Dec 16, 2010 at 1:53 PM, Dave Temkin <davet1@gmail.com> wrote: precedent

...

so that when traffic is 60Tbit instead of 6Tbit, across the *same* customer

Turn the question around. What would any provider think if a city said "sure, you can have access to our residents' eyeballs. It will cost you $5 per subscriber per month". Would Comcast or anyone go for that? That is a real question, by the way. For all I know some municipality might already do that. But say one with something between 100,000 and 1,000,000 potential subscribers did that. Would any of the providers think that is "fair"? Particularly *after* the provider is already providing services to those subscribers and then has the rules changed on them after they already have contracts in place with the subscribers? It just seems to me to be an evil Pandora's box that once opened, there is no potential end to. What if several cities ganged up and together decided to charge a last mile provider access to eyeballs? Better in my opinion to let the end user pay for what they use. It doesn't have to be strictly metered per meg but can be put into tiers (as most providers already do anyway). Sort of like "smart meters" they are doing with electricity. People will modify their usage according to what they can afford. Pricing bandwidth according to basic principles of supply and demand would probably work better. Those that use more would pay more, those that use less would pay less.

----- Original Message -----

From: "George Bonser" <gbonser@seven.com>

Turn the question around. What would any provider think if a city said "sure, you can have access to our residents' eyeballs. It will cost you $5 per subscriber per month". Would Comcast or anyone go for that? That is a real question, by the way. For all I know some municipality might already do that. But say one with something between 100,000 and 1,000,000 potential subscribers did that. Would any of the providers think that is "fair"? Particularly *after* the provider is already providing services to those subscribers and then has the rules changed on them after they already have contracts in place with the subscribers?

I believe you're looking for Rose.net/CNS in Thomasville GA: http://www.cns-internet.com/aboutcns.shtml Why not go *ask* competing providers what they think?

It just seems to me to be an evil Pandora's box that once opened, there is no potential end to. What if several cities ganged up and together decided to charge a last mile provider access to eyeballs?

What about it?

Better in my opinion to let the end user pay for what they use. It

That's orthogonal to who should be providing it, so the rest of your graf:

doesn't have to be strictly metered per meg but can be put into tiers (as most providers already do anyway). Sort of like "smart meters" they are doing with electricity. People will modify their usage according to what they can afford. Pricing bandwidth according to basic principles of supply and demand would probably work better. Those that use more would pay more, those that use less would pay less.

is a strawman. And note that I don't *care* whether commercial entities think a given approach is "fair" or not: they sure don't care whether *we* think their practices are "fair". No one is entitled to continue to make a living in any particular way, by law or any other facility. I thought that was attributable to Judge Learned Hand, but as it turns out, I stole it from Robert Heinlein, who used it in a speech from a judge in his very first published story, Lifeline. Perhaps Bill Patterson, his biographer, knows where he stole it from. It's still an excellent thing to remember. Lots of companies have sprung up to fulfill a niche -- full motion NTSC video processing in PCs, frex -- and then had to find something else to do when the pendulum swung from hardware back to software. Cheers, -- jra

----- Original Message -----

From: "JC Dill" <jcdill.lists@gmail.com> If I drive from SF to LA for business or for personal purposes, my costs for the drive are the same. But the economy of doing it for business depends on what the client is willing to pay me. If they want me to drive to LA but only pay $10, it's not economical (from a business perspective) for me to do it. Right now, Comcast is carrying content to their customers "for free" and they want to be paid by the content providers (thru paid transit connections) to cover the cost of carrying that content traffic across their network to the end customer.

Comcast is acting, collectively, as the agent of their customers, who I'm sure would tell you if you asked them that they believe the contract is "I pay you, and you carry my packets back and forth as I direct, as long as I follow your TOS" -- which pulling movies from Netflix does not presently violate, AFAICT.

Sure, Comcast's customers are also paying Comcast. But Comcast wants to get paid from the content provider. I think they are betting that in the long run it's easier to make money from content providers (and have the content providers charge customers or advertisers as necessary to make a profit) than to make money from the end consumer. And I think they are right about this "easier" part. I think that they will succeed at pressuring big content providers to play by Comcast's rules and shift the cost of running Comcast's network from consumers to content providers.

I'm sure that Comcast does think it's easier. But that doesn't mean it's a valid legal interpretation of their contracts with their direct customers, and I smell a class-action lawsuit brewing in the mind of some tort-king on just that point. The underlying problem, of course, is lack of usable last-mile competition; see also my running rant about Verizon-inspired state laws *forbidding* municipalities to charter monopoly transport-only fiber providers, renting to all comers on non-discriminatory terms, which is the only practical way I can see to fix any of this. Cheers, -- jra

----- Original Message -----

From: "JC Dill" <jcdill.lists@gmail.com>

...

see also my running rant about Verizon-inspired state laws *forbidding* municipalities to charter monopoly transport-only fiber providers, renting to all comers on non-discriminatory terms, which is the only practical way I can see to fix any of this.

The problem is that this should have been addressed 5-10 years ago, when there *were* alternative ISPs who could have provided competition. Now that Comcast has a monopoly on cable, and fiber is so bleeping expensive to install, at best we might get *one* alternative to Comcast, and a duopoly is really no better (for consumers, for the marketplace) than a monopoly.

I believe you misunderstood my assertion. Many local municipalities are doing the trenching themselves (well, generally, subbing it out to a contractor), and then offering the customers out to all IAP comers -- you meet-me in my fibernoc, and we'll cross connect every customer you sell to you. Lots of *other* municipalities would dearly love to do this, but state laws (lobbied for, in many places, by Verizontal) make this *illegal*. Wonder why Verizon would want to do *that*... See http://money.cnn.com/video/technology/2010/03/15/tech_tt_fiber_fios.cnnmoney... and also http://www.freepress.net/files/mb_telco_lies.pdf And ORA's Mike Loukides: http://radar.oreilly.com/2010/03/google-fiber-and-the-fcc-natio.html and a whole lot more here: http://www.ftthcouncil.org/en Those links from the consumer-level piece I wrote on this earlier this year: http://baylink.pitas.com/#LASTMILE Cheers, -- jra

On Thu, Dec 16, 2010 at 8:02 AM, Jared Mauch <jared@puck.nether.net> wrote:

On Dec 16, 2010, at 1:16 AM, JC Dill wrote:

...

On 15/12/10 9:29 PM, Jay Ashworth wrote:

...

The underlying problem, of course, is lack of usable last-mile

competition;

I agree.

It exists where there is an ROI on investment. Capital markets haven't been friendly to network build since the dot-bomb, and for some reason localities are more willing to give tax-incentive financing to malls and stadiums rather than incenting over-builders.

...

...

see also my running rant about Verizon-inspired state laws *forbidding* municipalities to charter monopoly transport-only fiber providers, renting to all comers on non-discriminatory terms, which is the only practical way I can see to fix any of this.

The problem is that this should have been addressed 5-10 years ago, when

there *were* alternative ISPs who could have provided competition. Now that

Comcast has a monopoly on cable, and fiber is so bleeping expensive to install,

at best we might get *one* alternative to Comcast, and a duopoly is really no

better (for consumers, for the marketplace) than a monopoly.

Funny thing about competition is that there are losers as well as winners. DSL competition didn't lose by regulation, it lost (nationally) by cheaper, more elastic bandwidth available on other media and JC's previously-noted fickle and lazy consumers. Where there is competition, the little guy gets an easy low percentage (10-25%) of penetration based solely on not being the incumbent, but churn is high as soon as sign-up incentives expire and they get on a downward spiral of catering to complainers. Magic phrases are traded on dslreports and any retention-packages get spread across the entire customer base. Where there isn't market- sustainable competition, there is no actual legislated monopoly but rather ignorant local boards.

This is why I suggested it might take regulatory action, or changes in state laws.

Also engage locality first, as Jared indicates. The problem in going to the fed is that power will be skewed to the larger entities. Competitive providers breathed a sign of relief when Verizontal lost their attempts to get statewide television franchising and had to deal locality-by-locality, just like the small guys did. Would be worse if there was a single federal entity to buy off now that corporate campaign funding is both anonymous and unlimited.

If I want to start up a coop, or convince my local county/state they should be a neutral provider of conduits/dark fiber as roads are rebuilt, etc.. there are various barriers. Even if the cost would be nominal. I scaled-up some quotes to be an area-wide effort for fiber down every public road ROW, and came back with $100mil. (you private road types need to shell out your own cash for that leg).

The barriers to doing this as a project are well known. Even if you don't like ars, they have decent articles on these topics:

http://arstechnica.com/tech-policy/news/2010/01/municipal-fiber-needs-more-f...

http://arstechnica.com/tech-policy/news/2009/06/monticello-appeals-court-win...

http://arstechnica.com/old/content/2008/07/telco-wont-install-fiber-sues-to-...

Similar to the above, I could not even get Comcast to give me a quote to build to my area. AT&T ... good luck getting any data from them. I can tell they are filling in the gaps based on the trenching/boring going on, but there's no good way to motivate them. And even if I decided to drop $10k to install a bunch of POTS service for 1 month to force a build, who knows if that build would bring the right level of service. (As the POTS is regulated with a low install fee).

The incentives are clearly skewed here, but without that $100mil, reaching the 125k properties (111k residences) in my local area may be tough. (Note: there may be actual cost savings by not running down *every* public road, but using public road mileage and property counts seemed like a good method without actually designing the final fiber plant).

My notes are here:

http://puck.nether.net/~jared/blog/?p=84

The reply I received from my elected reps:

"Additionally, offering a millage to build a network for the general public may violate recent provisions within the Michigan Telecommunication Act."

- Jared

In a country where government-supplied healthcare is viewed as evil, how can people honestly expect the less-important telecommunications to be allowed to be "government run" as neutral municipal networks? Any unbundling of local HFC or FTTP loops will be slow and problematic.

On 12/16/10 9:51 AM, Craig L Uebringer wrote:

Funny thing about competition is that there are losers as well as winners. DSL competition didn't lose by regulation, it lost (nationally) by cheaper, more elastic bandwidth available on other media and JC's previously-noted fickle and lazy consumers.

Apparently, you've never owned or run an ISP in the past dozen years.... Pacific Bell Telephone v. LinkLine, 07-512 It lost *precisely* by regulation: Google "Tauzin-Dingell". We used to offer up to 7 Mbps bidirectional DSL long before cable or the Bells offered anything in that range. We had our own DSLAMs. How exactly do you compete when the Incumbent charges us $80 per month wholesale for UNE lines that they sell $10 per month retail? http://www.techdirt.com/articles/20061228/181255.shtml http://www.dslreports.com/shownews/ATT-10-DSL-Today-84904 Note that's only $10 for "new" customers (that is, *our* customers). And that's just the tip of the iceberg: http://www.cybertelecom.org/broadband/dslnaked.htm org.law.rutgers.edu/publications/lawjournal/issues/38_1/Sholinsky.pdf ...

From: JC Dill Sent: Wednesday, December 15, 2010 9:13 PM Cc: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style

Sure, Comcast's customers are also paying Comcast. But Comcast wants to get paid from the content provider. I think they are betting that in the long run it's easier to make money from content providers (and have the content providers charge customers or advertisers as necessary to make a profit) than to make money from the end consumer. And I think they are right about this "easier" part. I think that they will succeed at pressuring big content providers to play by Comcast's rules and shift the cost of running Comcast's network from consumers to content providers.

jc

There are two different innovation paths according to who is paying. If the customer is paying, innovation is driven by the interest of the customer. If the provider is paying, innovation is driven by the interest of the provider. If the customer pays the cost of the transport, a provider with better transport efficiency / quality ratio wins. It spurs innovation where we get better quality product with a better transport efficiency. If there are three competing content services in the market offering basically the same quality product, the one with the better transport efficiency is going to win customers. Or in some cases the customer might choose to sacrifice some quality for transport efficiency. The market eventually settles on what the customers in the aggregate decide is their willingness to trade price for performance. If the provider pays the cost of the transport, a provider might effectively subsidize the transport cost of a bloated content distribution mechanism. It won't make any difference to the last mile delivery network either way. Either way they get the same amount of money. If provider pays the freight, there might be some company with an absolutely killer technology that can stream much higher quality stuff with less bandwidth usage but if the customer doesn't see the benefit, that in and of itself isn't enough to drive eyeballs to that content. If that content transport method did save the customer money, the eyeballs would move in that direction. Having the provider pay the cost stifles technological advancement. It facilitates a "deep pocket" established company creating a barrier of adoption to a startup who might have a more efficient product but the user doesn't get any direct benefit so they don't adopt it. Having the user pay gives an incentive to develop technologies that reduce the network burden. Having the provider pay distorts innovation. In the end, having the end user pay the cost for the product they are consuming results in better, faster, cheaper (yes, you can have all three). Externalizing those costs through subsidies by outside parties throws things out of balance and drives innovation in a way that benefits the provider, not the consumer.

From: JC Dill Sent: Wednesday, December 15, 2010 10:20 PM To: NANOG list Subject: Re: Some truth about Comcast - WikiLeaks style

On 15/12/10 10:05 PM, George Bonser wrote:

...

If the customer pays the cost of the transport, a provider with

better

...

transport efficiency / quality ratio wins.

This (and everything that followed) assumes the customer has a choice of providers. For most customers who already have Comcast, they don't have any choice for similar broadband services (speeds). So open market principles don't come into play, and Comcast knows it.

jc

While that is true, it was less true today than it was yesterday and will be even less true tomorrow. The "captive" audience the cable providers have had is shrinking. Even in markets where Comcast might have a monopoly on TV they don't necessarily have a monopoly on a broadband path to the residence for other services. While they might be able to keep out other "cable" companies, it is going to be hard for them to keep the phone company out. They already have a path to the home. I found U-verse and FiOS coverage map here: http://www.dslreports.com/gmaps/uverse but I can't vouch for the accuracy except in my local area. I believe that as these cable agreements expire, it is going to be more difficult to get a monopoly. Where there is already separate television and telephone infrastructure, you will see the "cable" company and the telephone companies competing for triple-play services.

Interesting point. I'd also like to point out that putting the cost on the content providers rather than the network may raise the cost of the content service, but only to those that want that service. In effect, if the transport provider is paying for the bandwidth generated by a content provider, in effect we have another service bundled to all services offered, which increases the cost to people using Internet service but not necessarily accessing that content. Kind of the same reason TV channels aren't a la carte. Sincerely, Brian A . Rettke RHCT, CCDP, CCNP, CCIP Network Engineer, CableONE Internet Services -----Original Message----- From: George Bonser [mailto:gbonser@seven.com] Sent: Wednesday, December 15, 2010 11:41 PM To: JC Dill; NANOG list Subject: RE: Some truth about Comcast - WikiLeaks style

From: JC Dill Sent: Wednesday, December 15, 2010 10:20 PM To: NANOG list Subject: Re: Some truth about Comcast - WikiLeaks style

On 15/12/10 10:05 PM, George Bonser wrote:

...

If the customer pays the cost of the transport, a provider with

better

...

transport efficiency / quality ratio wins.

This (and everything that followed) assumes the customer has a choice of providers. For most customers who already have Comcast, they don't have any choice for similar broadband services (speeds). So open market principles don't come into play, and Comcast knows it.

No, you misunderstood. It doesn't matter if you have only one internet service provider. If the end customer foots the bill, the incentive for innovation is for the *content* provider to strike a balance between quality and cost that the customers want. If the *content* provider foots the bill, innovation is driven in a way that the content providers want. Lets say I have foo.com and bar.com that offer video services and I am on Comcast. If Comcast meters my bandwidth usage and foo.com has good quality with a lower bandwidth use, I use foo. In the other model, if the content providers subsidize the bill, bar.com might be completely bloated but they have deep pockets and can pay the subsidy, they drive foo.com out of business and Comcast still has a congested network.

----- Original Message -----

From: "Brian Rettke" <Brian.Rettke@cableone.biz>

Interesting point. I'd also like to point out that putting the cost on the content providers rather than the network may raise the cost of the content service, but only to those that want that service. In effect, if the transport provider is paying for the bandwidth generated by a content provider, in effect we have another service bundled to all services offered, which increases the cost to people using Internet service but not necessarily accessing that content. Kind of the same reason TV channels aren't a la carte.

Having worked for a small cable TV network in the 90s, I have some insights into why cable systems don't sell most channels alacarte. 1) The accounting goes pear-shaped pretty quickly, or at least, it did in the 80s when that practice got started -- having to account for each individual subscriber pushed the complexity up, in much the same way that flat rate telecom services are popular equally because customers prefer them, and because the *cost of keeping track* becomes >delta. 2) New networks prefer it, and the fact that it happens makes the creation of new cable networks practical -- you don't have to go around and sell your idea to people retail; you sell it to CATV systems (well, really, multi-system operators) *once* -- generally at something like the Western Show -- and they buy it and give it to *all* of their subs as part of a tier. Makes it much easier to achieve critical mass. And finally, 3) the increased complexity of having *everything* alacarte increases the cognitive load on new subscribers to the point where they probably will consider other alternatives -- it's just too many decisions to make when you're trying to sign up. Additionally, it makes marketing harder: there isn't a real "base price, nicely equipped" to point to. In the current tiered approach, a very small group of people inside the cable system is charged with picking the channels, and putting them in the tiers, and they're the only ones who ought to have to care about that, in my mostly humble opinion. The percentage of people who want channel by channel control over their cable service, I think, is roughly akin to the percentage of people who root their Android phone so they can play with the apps and the controls that you can't get without doing that; ie: minuscule. (I actually mistyped "minusclue", but that's what those people are *not*; our only real blindspot as geeks is realizing that we're exceptional -- that most people really couldn't give a damn.) Cheers, -- jra

On Fri, Dec 17, 2010 at 12:26 AM, Jay Ashworth <jra@baylink.com> wrote:

the 80s when that practice got started -- having to account for each individual subscriber pushed the complexity up, in much the same way that flat rate telecom services are popular equally because customers prefer them, and because the *cost of keeping track* becomes >delta.

Having personally and solely designed and written a toll billing system from scratch that directly exchanged billing and settlement data (and end-user data) with hundreds of ILECs, I can tell you a number of things I learned: 1) billing is only as hard as you (or your vendor) make it 2) if your company can't figure out how to bill for a new product or service, blame the billing people, not the product 3) keeping up with taxes and fees consume a lot more resources than calculating the net bills themselves; so adding products is really trivial compared to dealing with every pissant local government that decides to apply a different taxing method to your HBO (or your telephone calls) This is not to say the folks that handle billing at cable companies are equally capable, but if they had legitimate competitors, they would figure out how to run many parts of their businesses more efficiently. Imagine if Wal-Mart was the only game in town that had bar code readers at the cash registers, and every other grocery chain had to look up every item and punch in the price to check you out. Other stores would quickly improve their technology or find themselves out of business.

2) New networks prefer it, and the fact that it happens makes the creation of new cable networks practical -- you don't have to go around and sell your idea to people retail; you sell it to CATV systems (well,

My understanding is that networks/media giants like it because they can force cable companies to carry 11 irrelevant channels to get the Disney Channel that your kids want. Would enough people really ask for G4TV to make producing and syndicating shows for that channel cost-effective? I don't know the answer, but my suspicion is that people who really just want CSN, E!, or the Golf Channel are subsidizing G4 viewers. I wanted BBCA a few years ago, but my cable provider required that I buy 30 other channels I did not want or had never even heard of to get BBCA, so I didn't subscribe to it. I do not know if a la carte channel selection would be good for me, as a consumer, or not. I do think the reasons the industry does not want to offer that to end-users are disingenuous. -- Jeff S Wheeler <jsw@inconcepts.biz> Sr Network Operator  /  Innovative Network Concepts

On 17/12/10 4:54 AM, Carlos Martinez-Cagnazzo wrote:

I do believe that video over the Internet is about to change the cable business in a very deep and possibly traumatic way.

+1 It's clear that this is a major driving factor in the Comcast/L3/Netflix peering/transit issue. Comcast is obviously looking for ways to fill the looming hole in their revenue chart as consumers turn off Cable and get their TV/video entertainment delivered via the internet. jc

On Fri, 17 Dec 2010, Jay Ashworth wrote:

The more I look at this, the more it looks like "pharmaceuticals bought from Canada are cheaper than ones purchased in America -- and they will be *just as long* as only a minority of Americans buy them there. As soon as *everyone* in America is buying their drugs cross-border, the prices will go right back up to what they were paying here."

This is what's gonna happen with Comcast, too; if their customers drop CATV, then they're going to have to raise their prices -- and the cable networks themselves will have *no* way to collect revenue; the cable systems being their collection agent network.

This Can't End Well.

Why not? As people shift from watching broadcast channels to streaming content and look to shut off their cable TV service, but keep internet, the cable co's are just going to have to raise internet prices to compensate. I can see a future where you buy internet from the cable co and they give you the basic cable TV channel lineup at "no charge" but in reality, you're paying for the cable internet what you used to pay for both cable internet and TV. The people I see this being a problem for are HBO/Showtime/Stars etc. unless they can hop on with the streaming providers or make that move themselves. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Jon Lewis <jlewis@lewis.org> writes:

...

This Can't End Well.

Why not? As people shift from watching broadcast channels to streaming content and look to shut off their cable TV service, but keep internet, the cable co's are just going to have to raise internet prices to compensate. I can see a future where you buy internet from the cable co and they give you the basic cable TV channel lineup at "no charge" but in reality, you're paying for the cable internet what you used to pay for both cable internet and TV.

Here in NoVA (Comcast former Adelpha territory), the future is now. I used to have internet-only service (there is little on TV that I care about). A bit over a year and a half ago, we added basic cable to the service. Total additional cost per month to go from Internet-only to Internet-plus-TV-bundle (same speed) was about $4. -r

On 12/18/10 7:27 PM, Kevin Oberman wrote:

...

From: "Robert E. Seastrom"<rs@seastrom.com> ... I can see a future where you buy internet from

...

the cable co and they give you the basic cable TV channel lineup at "no charge" but in reality, you're paying for the cable internet what you used to pay for both cable internet and TV.

Here in NoVA (Comcast former Adelpha territory), the future is now.

I used to have internet-only service (there is little on TV that I care about). A bit over a year and a half ago, we added basic cable to the service. Total additional cost per month to go from Internet-only to Internet-plus-TV-bundle (same speed) was about $4.

Hmmm. Better than the situation in my Comcast area. Internet w/o any cable costs MORE than basic cable (i.e. over the air + PEG). ...

Likewise, here in Michigan I helped a brother setup Comcast, and discovered that the charge for Internet + Basic Cable was about $2 per month *cheaper* than Internet-only.

Jay Ashworth wrote:

individual subscriber pushed the complexity up, in much the same way that flat rate telecom services are popular equally because customers prefer them, and because the *cost of keeping track* becomes >delta.

Can someone then please explain me why the hell in many other countries flatrate telecom service (I refer to flatrate local calls) does not exist or has been phased out. In the Netherlands they phased it out in the mid to late 80s. I am sure the then government owned telecom rats saw increased revenue coming real soon now due to increased modem usage. (still pissed at ridiculously and unnecessarily high phonebills...) It seems to me that at least in that case the cost of keeping track was far less than the increased revenue that metered (if that's the right word) local calls would provide. Regards, Jeroen -- http://goldmark.org/jeff/stupid-disclaimers/ http://linuxmafia.com/~rick/faq/plural-of-virus.html

On 15/12/10 10:40 PM, George Bonser wrote:

...

From: JC Dill Sent: Wednesday, December 15, 2010 10:20 PM To: NANOG list Subject: Re: Some truth about Comcast - WikiLeaks style

On 15/12/10 10:05 PM, George Bonser wrote:

...

If the customer pays the cost of the transport, a provider with better transport efficiency / quality ratio wins.

This (and everything that followed) assumes the customer has a choice of providers. For most customers who already have Comcast, they don't have any choice for similar broadband services (speeds). So open market principles don't come into play, and Comcast knows it. No, you misunderstood. It doesn't matter if you have only one internet service provider. If the end customer foots the bill, the incentive for innovation is for the *content* provider to strike a balance between quality and cost that the customers want. If the *content* provider foots the bill, innovation is driven in a way that the content providers want.

The customer *always* foots the bill in the end. It's just a matter of how many intermediaries there are between the bill-paying customer and the underlying service they are paying for. Customers clearly prefer to have the true costs of services hidden and obfuscated. Take a look at the byzantine way we pay for health care in the US today, versus how we paid for health care 50 years ago. Then take a look at the industry that has sprung up to wring ever more dollars out of consumers by insulating them from the true costs of health care. Repeat for the cost of body work on your car (paid for with insurance, with the "quality" (and thus cost) of repair being ever escalated because the consumer doesn't see the direct cost of the increased repair), the quality of food production (massive poultry houses where birds are routinely fed antibiotics and infected eggs lead to nationwide recalls) etc. Consumers are too insulated from the production and true costs, and don't realize how the market consolidation is taking away their choices AND producing ever lower quality of goods and services. Why should internet access be any different? There was a story on NPR the other day where the talking head spoke about how "consumers overwhelmingly want a do-not-track system". Hello?! Consumers also don't want spam. Can you point to a SINGLE case where CAN-SPAM actually stopped a significant amount of spam? The reason consumers have functioning email mailboxes isn't because of legislation stopping spam, it's because of ISPs implementing ever increasingly effective anti-spam techniques. Anyone who thinks a "do not track" legislation can have any possible measurable effect on how websites track users is simply ignorant about the magnitude of the problem, and how companies will simply outsource (ultimately to overseas companies) their "customer tracking" services to avoid needing to comply with any US laws. And how can the consumer know if their "do not track" request is being honored anyway? It's not like they get a popup every time a website tracks their activities. What customers *really* want, and what they gladly accept as long as it saves them a few pennies, are miles apart. (Which is why so many people blindly give their data to Facebook etc.) This is why I think the direction Comcast is going is ultimately going to win in the marketplace. Do I *want* to see Comcast win? No! But I think it's an inevitable trend. Customers are lazy. Customers are cheap. They will - en masse - support the lowest cost solution that *appears* to give them something of value, even when it's really not in their best interest. jc

On Wed, Dec 15, 2010 at 10:40 PM, George Bonser <gbonser@seven.com> wrote:

...

From: JC Dill Sent: Wednesday, December 15, 2010 10:20 PM To: NANOG list Subject: Re: Some truth about Comcast - WikiLeaks style

  On 15/12/10 10:05 PM, George Bonser wrote:

...

If the customer pays the cost of the transport, a provider with

better

...

transport efficiency / quality ratio wins.

This (and everything that followed) assumes the customer has a choice of providers.  For most customers who already have Comcast, they don't have any choice for similar broadband services (speeds).  So open market principles don't come into play, and Comcast knows it.

No, you misunderstood.  It doesn't matter if you have only one internet service provider.  If the end customer foots the bill, the incentive for innovation is for the *content* provider to strike a balance between quality and cost that the customers want.  If the *content* provider foots the bill, innovation is driven in a way that the content providers want.

Lets say I have foo.com and bar.com that offer video services and I am on Comcast.  If Comcast meters my bandwidth usage and foo.com has good quality with a lower bandwidth use, I use foo.  In the other model, if the content providers subsidize the bill, bar.com might be completely bloated but they have deep pockets and can pay the subsidy, they drive foo.com out of business and Comcast still has a congested network.

http://techcrunch.com/2010/12/15/yahoo-video-no-longer-accepts-video-uploads... You may find that simply fewer content providers decide it's worth it to play in that space, under those conditions, which results in fewer choices for the consumer, and something closer to a monopoly on the available content to be consumed. People *were* happy with only having three national TV networks to choose from for their major content in the US, right? bar.com doesn't have to drive foo.com out of business; they just have to outlast them in the war of attrition driven by the monopoly holder, until bar.com decides it's no longer worth providing that content anymore. end game--one monopoly access provider, and one giant content source--and a huge barrier to entry keeping anyone else from providing an alternative view of the world. Matt (speaking only for myself, and definitely not for any companies named foo, bar, or any other combination of letters. Or punctuation marks of any sort.)

I disagree with this theory. If customers pay comcast for bytes then eventually the upstream (L3) will want some of that revenue. That revenue will be passed onto the provider as a lower bill. This encourages Netflix to send more bytes, because if they do Comcast and L3 get paid more and Netflix's bill goes down. The income stream is now completely dependent on how much money the customer can pay. Sure other services (Hulu, Youtube, etc) could step in and offer lower usage but customers are MUCH more likely to pick based on available content then an abstract "we use N bytes per month fewer then that guy". Also, if income is dependent on sending more bytes, provider usage will creep up as far as they can get push it while still keeping customers. In comparison, does the average person pick their mobile provider based on how many minutes they get or if they get coverage at their house. Keep in mind that all providers are within ~10% of each other on pricing. Where do we see costs in mobile going? How much is a text message these days? That is what happens when the user pays for bytes. If the provider pays for transport then the amount of data going over wires is dependent on that providers customer base. The cost of transport now scales directly with size of business. Netflix or Hulu are now directly responsible for their costs which motivates them to be efficient. Comcast can now charge its customers only for upkeep of its network and use the income they get as an "end point delivery network" to offset customer cost. Comcast's cost, which are upkeep and expansion of its physical network, now scale proportionally with its customer base. So in this model, customers pay for the laying of the wire to their house and the upkeep of that wire, which is a 1:1 ratio for the consumer/comcast. Providers (Netflix) pay per byte to the transport providers. Transport providers make thier money off providing transport. Income and cost of doing business for Netflix is directly tied to their subscriber base so it is easy for them to balance their income. Of course, I am not an economist and could be entirely wrong. There are certainly other HUGE political factors, but I think in theory we would all be happier if the system worked by someone paying for a postage stamp then COD. On Thu, Dec 16, 2010 at 1:05 AM, George Bonser <gbonser@seven.com> wrote:

...

From: JC Dill Sent: Wednesday, December 15, 2010 9:13 PM Cc: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style

Sure, Comcast's customers are also paying Comcast. But Comcast wants to get paid from the content provider. I think they are betting that in the long run it's easier to make money from content providers (and have the content providers charge customers or advertisers as necessary to make a profit) than to make money from the end consumer. And I think they are right about this "easier" part. I think that they will succeed at pressuring big content providers to play by Comcast's rules and shift the cost of running Comcast's network from consumers to content providers.

jc

There are two different innovation paths according to who is paying. If the customer is paying, innovation is driven by the interest of the customer. If the provider is paying, innovation is driven by the interest of the provider.

If the customer pays the cost of the transport, a provider with better transport efficiency / quality ratio wins. It spurs innovation where we get better quality product with a better transport efficiency. If there are three competing content services in the market offering basically the same quality product, the one with the better transport efficiency is going to win customers. Or in some cases the customer might choose to sacrifice some quality for transport efficiency. The market eventually settles on what the customers in the aggregate decide is their willingness to trade price for performance.

If the provider pays the cost of the transport, a provider might effectively subsidize the transport cost of a bloated content distribution mechanism. It won't make any difference to the last mile delivery network either way. Either way they get the same amount of money. If provider pays the freight, there might be some company with an absolutely killer technology that can stream much higher quality stuff with less bandwidth usage but if the customer doesn't see the benefit, that in and of itself isn't enough to drive eyeballs to that content. If that content transport method did save the customer money, the eyeballs would move in that direction.

Having the provider pay the cost stifles technological advancement. It facilitates a "deep pocket" established company creating a barrier of adoption to a startup who might have a more efficient product but the user doesn't get any direct benefit so they don't adopt it. Having the user pay gives an incentive to develop technologies that reduce the network burden. Having the provider pay distorts innovation.

In the end, having the end user pay the cost for the product they are consuming results in better, faster, cheaper (yes, you can have all three). Externalizing those costs through subsidies by outside parties throws things out of balance and drives innovation in a way that benefits the provider, not the consumer.

If ponies are being handed out, count me in. Sure, market forces can do lots of strange things, for example, see our current position. Pretty much any scheme breaks terribly when there is a monopoly, since the only company involved gets to remove the relationship between cost and profit. On Thu, Dec 16, 2010 at 10:41 AM, JC Dill <jcdill.lists@gmail.com> wrote:

On 16/12/10 7:17 AM, Mikel Waxler wrote:

...

I disagree with this theory.

If customers pay comcast for bytes then eventually the upstream (L3) will want some of that revenue.

And I want a pony.

What the upstream "wants" and what market forces will decide could be very different. And as customers "want" lower internet access costs, if Comcast can collect more money from upstreams then they can lower the rates for customers, gain more customers, and be in an even better position to squeeze money out of upstreams.

jc

How well did the lawsuits against Microsoft's monopoly work to reduce their ability to use their monopoly to manipulate the market?

jc

Why don't you ask the folks over at The Technical Committee (http://www.thetc.org), since they monitor Microsoft compliancy for the DOJ. Randy

On 16/12/10 8:52 AM, Jack Bates wrote:

On 12/16/2010 9:17 AM, Mikel Waxler wrote:

...

Comcast can now charge its customers only for upkeep of its network and use the income they get as an "end point delivery network" to offset customer cost. Comcast's cost, which are upkeep and expansion of its physical network, now scale proportionally with its customer base.

The problem with your layout is that, as a netflix user, I pay more to netflix so that you can have their service over comcast, and my provider doesn't get income from the netflix streams as it is sub 100k users (so I still have to pay for my provider's upgrades to handle the netflix which percentage wise will be higher than comcast due to less ideal bandwidth discounts and the locality which may even drive up the overall percentage of netflix streams per customer base).

Problem? For Comcast, none of this is a problem. (Do you see the problem now?) Again, I predict that things ARE heading in this direction, and that market forces and the current regulatory climate encourages it. Dire news for small providers. Saying you "want" it to be different[1] won't change anything. I don't know what the solution is (if there is a solution) but so far all I see are people complaining "but if that happens, it's bad for me and for others". Yes, it's bad. What are you going to do to stop it? If Comcast can continue to force other networks to pay it to carry data to Comcast's users, it will create a tidal wave of momentum in their favor for lowering rates and pushing other eyeball networks aside, buying them up or just taking over their territory and customers. jc [1] I want a pony, etc.

On Wed, 15 Dec 2010, JC Dill wrote:

Sure, Comcast's customers are also paying Comcast. But Comcast wants to get paid from the content provider. I think they are betting that in the long run it's easier to make money from content providers (and have the content providers charge customers or advertisers as necessary to make a profit) than to make money from the end consumer. And I think they are right about this "easier" part. I think that they will succeed at pressuring big content providers to play by Comcast's rules and shift the cost of running Comcast's network from consumers to content providers.

Personally, I'd like to see any provider (content or otherwise) tell Comcast (as things stand today) to pound sand when asked to enter into such a 'paid peering' arrangement with them. As others have said, the fact that Comcast is holding their customers hostage to squeeze money out of providers who often have no direct connectivity to Comcast is particularly troubling. I don't use Comcast for Internet access, but I've been considering kicking their overpriced cable TV service to the curb for some time now... jms

On Wednesday, December 15, 2010 05:47:09 pm Adam Rothschild wrote:

What we have here is Comcast holding its users captive, plain and simple. They have established an ecosystem where, to reach them, one must pay to play, otherwise there's a good chance that packets are discarded. [snip] Folk in content/hosting should find this all more than a little bit scary.

I'm surprised no one here has thought of the obvious thing content providers can do to communicate to the customers of the providers who artificially throttle traffic from 'freeloading' content providers. In the web server configuration, detect what network is accessing the page. If it's a provider who is trying to coerce content provider payment, tell the eyeball up front that that's the case, and give a pointer to the place on the FCC website (or the FCC phone number) where they can lodge a complaint. If it gets ugly, simply don't serve content to those eyeballs. In other words, a content provider boycott of eyeball networks that want to try to play hardball. If you get enough content providers to band together to do this, the customers of those eyeball networks will make a difference. Hrmph, all you really have to do is get google or facebook to boycott an eyeball network. IOW, if there's no content to see, there's no need for an 'Internet' connection.

On 12/16/2010 10:54 AM, Mikel Waxler wrote:

But in that scheme, Comcast looses in the long run, when the FCC gets around to them, but Netflix looses customers immediately.

" I pay Netflix 10$ a month and they wont let me use their service cause I am on Comcast? I am taking my money to Hulu!"

Sure netflix is "right" but by the time it matters they are out of business.

Surely serving a "bumper" video at the beginning - "Comcast is trying to charge you more for Netflix - see http://www.netflix.com/comcastripoff/" - would be enough? --Patrick

On 12/15/2010 05:09 AM, ML wrote:

...

According to: http://en.wikipedia.org/wiki/Comcast "Comcast has 15.930 million high-speed internet customers"

If a 10G port for transit is paid by comcast $30/Mbit/s monthly that's 0.19 cent/internet customer/month for a new 10G port to properly desaturate this particular link.

Did I compute something wrong?

Laurent

Assuming that I did my math right.

It's actually 1.9 cents/month/per customer.

Assuming they pay $30/meg...

Probably preaching to the choir here but there are a lot more costs than that involved. It's all right having the bandwidth at transit points, but you've got to be able to get the bandwidth to the customers locations. With no idea of what Comcast's distribution is like for all we know the graph could be one transit point in one area of the country and indicative of poor localised behaviour rather than centralised. Virgin Media were notorious in various cities in the UK for over-saturating the local network. Out in the towns and smaller cities you'd be okay and have no problem saturating a 20Mb line, but often whole areas of London, Manchester and the like would suffer high latency, packet loss and so on during 'peak' hours because they would over sell their infrastructure (12am-10am fine, then steadily worse until unusable come the evening). They only seemed to add more capacity to the areas when enough people complained. IMO two network graphs are next to useless out of context. Paul

Ay 10 Gig levels bandwidth should be much much cheaper than $30 /Mbit. -- Justin Wilson <j2sw@mtin.net> Aol & Yahoo IM: j2sw http://www.mtin.net/blog ­ xISP News http://www.twitter.com/j2sw ­ Follow me on Twitter Wisp Consulting ­ Tower Climbing ­ Network Support From: Jon Lewis <jlewis@lewis.org> Date: Wed, 15 Dec 2010 11:46:13 -0500 (EST) To: <nanog@nanog.org> Subject: Re: Some truth about Comcast - WikiLeaks style On Wed, 15 Dec 2010, Laurent GUERBY wrote:

If a 10G port for transit is paid by comcast $30/Mbit/s monthly that's 0.19 cent/internet customer/month for a new 10G port to properly desaturate this particular link.

Did I compute something wrong?

At that bandwidth level, isn't $30/mbit roughly an order of magnitude higher than people are actually paying? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route Senior Network Engineer | therefore you are Atlantic Net | _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

On 12/15/2010 1:13 PM, Jeffrey Lyon wrote:

They can't be paying more than a couple of dollars per Mbps.

$10 tops for any provider than can hand off a 10GE pipe; and at full-rate multiple 10GE, you can expect it to be less than $5. Jack

You mean it is not a settlement free peering agreement? (sorry top post, following trend) ~J

-----Original Message----- From: Jeffrey Lyon [mailto:jeffrey.lyon@blacklotus.net] Sent: Wednesday, December 15, 2010 11:26 AM To: Jack Bates Cc: nanog@nanog.org Subject: Re: Some truth about Comcast - WikiLeaks style

...

From Tata? I'd eat my own hand if they were paying more than $1-2 across the board.

Jeff

On Wed, Dec 15, 2010 at 2:17 PM, Jack Bates <jbates@brightok.net> wrote:

...

On 12/15/2010 1:13 PM, Jeffrey Lyon wrote:

...

They can't be paying more than a couple of dollars per Mbps.

$10 tops for any provider than can hand off a 10GE pipe; and at full- rate multiple 10GE, you can expect it to be less than $5.

Jack

-- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions

On Tue, Dec 14, 2010 at 11:24:45AM -0500, Craig L Uebringer wrote:

Yeah, the 30 day looks like a classic uptick in traffic toward the holidays. Some bellhead beancounter maybe took out capacity in the summer lull and ignored the engineers. Or they just have stupidly-slow install intervals. Same crap I've seen on loads of provider networks.

Except that they seem to be busy actively turning down other capacity, and forcing extra traffic through their Tata ports by blocking other paths with BGP no-export communities. For example, we've been observing Comcast turning down some of their Global Crossing capacity in recent days, causing new congestion during peak traffic times. I've even seen people contact the various NOCs involved, and they've been told explicitly and by multiple parties that Comcast is intentionally turning down extra capacity and running their existing ports hot. Everybody who deals with interconnection capacity in this industry knows what's going on, but the graphs and interconnection details are all under NDA, so it takes an inside source secretly leaking graphs to the public to expose this kind of activity. Even then you'll still have people who claim that it proves nothing because the graphs can't be positively associated to a specific customer port, but realistically these kinds of leaks are probably the best public info you'll ever see. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

On Tue, Dec 14, 2010 at 03:39:07PM -0600, Aaron Wendel wrote:

To what end? And who's calling the shots there these days? Comcast has been nothing but shady for the last couple years. Spoofing resets, The L3 issue, etc. What's the speculation on the end game?

I believe Comcast has made clear their position that they feel content providers should be paying them for access to their customers. I've seen them repeatedly state that they feel networks who send them too much traffic are "abusing their network". It isn't a ratios argument in the classic sense, between two peers trying to maintain a fair balance of costs and benefits, it's that they object to ANY content provider being able to deliver to their customers without paying them for access. They do this by trying to enforce ratios which are well beyond what their actual end users are routing, and as in the case of Level 3, they leverage that position to claim that other networks should be paying them under threat of blocking uncongested access to their customers. I would say their short term goal is to make people who currently won't peer with them do so, so they can become transit free. This has been seen time and time again, as they move networks who they want to peer with but who will not peer with them into "congested transit" bucket. A while back it was SAVVIS, now it is Tata, but the pattern is clear and repetitive. Note that this only extends to a certain point though, as in the case of Global Crossing, who they claim is a settlement free peer, but who they have recently started pressuring and intentionally congesting because of ratio imbalances. Their long term goal seems to be to force content networks to pay them for direct transit or on-net connectivity, by removing the available capacity from other paths. If you are a content network, and you can't reach them in a reliable fashion via "The Internet", your only choice may be to buy from Comcast directly. This is obviously not the first time that networks have used this strategy, there are several prominent examples in recent history of others using this exact same technique. But this is definitely one of the worst examples in the US of a major eyeball network using access to their customers (who may have little or no choice in their broadband access) to force other networks to pay them, and IMHO it needs to be called out publicly whenever possible. -- Richard A Steenbergen <ras@e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

...

...

To what end? And who's calling the shots there these days? Comcast has been nothing but shady for the last couple years. Spoofing resets, The L3 issue, etc. What's the speculation on the end game? I believe Comcast has made clear their position that they feel content

On Tue, Dec 14, 2010 at 03:39:07PM -0600, Aaron Wendel wrote: providers should be paying them for access to their customers. The Internet would offer lesser value by allowing access providers to hold their customers hostage. Clearly, such providers are not acting in

On 12/14/10 2:38 PM, Richard A Steenbergen wrote: their customer's interests when inhibiting access to desired and legitimate content. What is net neutrality expected to mean? Providers should charge a fair price for bandwidth offered, not over sell the bandwidth, and not constrain bandwidth below advertised rates. Congestion pricing rewards bad practices that leads to the congestion. -Doug

I just see this as a natural progression of what happens of a single player with a captive audience due to mergers and attrition. They know their customers aren't going anywhere. The only way to "fix" it would be to go back to the days when there were a bunch of competing local providers.

Wait -- you mean competition that is healthy and fair?! The FCC would not hear of this. -- Alex, remembering the days of 8000 ISP's with substantially better customer service than is available today

On Dec 14, 2010, at 6:59 PM, Seth Mattinen wrote:

I just see this as a natural progression of what happens of a single player with a captive audience due to mergers and attrition. They know their customers aren't going anywhere. The only way to "fix" it would be to go back to the days when there were a bunch of competing local providers.

This requires one or more of the following: o regulatory action o last mile regulation or competitive access o subsidies for new players o massive capital outlays o state laws changed in various markets o reformation of USF o changes at NTIA o changes at USDA (RUS) I'll once again use my example of the verizon assets going to fairpoint. it shows that the costs are significant. I can get a 10G across an ocean for cheaper than I can get one delivered over a 1 mile distance in a neighborhood. I do believe that FTTH will eventually become the solution to all the edge network ills, but at the same time, replacing that costs a lot of money. Take a look at this article from 2008 - http://bits.blogs.nytimes.com/2008/08/19/a-bear-speaks-why-verizons-pricey-f... "Here is how Mr. Moffett looks at the costs of the plan that Verizon has announced for FiOS. Through 2010 the company will pay an average of $817 to run the fiber past the 19 million homes, on poles or under the ground. It will also incur $172 per home passed in other costs related to the video infrastructure. He assumes that 40 percent of the customers passed will buy at least one FiOS service. If you allocate the cost of running the fiber past the homes that don’t buy FiOS to those that do, that makes the cost of building the network $2,473 per home. (That cost would be less if more than 40 percent of the potential customers sign up. Or it could be higher, if sales don’t achieve the 40 percent level.)" If you are willing to pay $2500 to have service installed, I'm sure the incumbents would be jumping at you. Instead, these are often regulated, last I recall in Michigan it was $42/line, even if they had to trench a quarter mile to reach you.. or if they just tested the existing copper to your home. This masks the actual costs. - Jared

----- Original Message -----

From: "George Bonser" <gbonser@seven.com> If monopolies are needed in order to get service to an area, make them "last mile" wire monopolies that provide no content of their own and allow the content providers (Comcast, Verizon, AT&T, etc.) provide service over the infrastructure on a competitive basis. Content monopolies tied to the infrastructure are bad for everyone and as existing monopoly agreements expire, more competition is entering the market. I would possibly compromise by saying a company willing to install the infrastructure could get a one-time monopoly for some period of time, after which the infrastructure is spun off as a separate company and opened up to competitive access.

That's the magic answer, right there, yes: fiber last-mile is a natural monopoly, for a whole host of practicality reasons. So, if we could repeal all the laws Verizon's FiOS division has gotten passed forbidding municipalities from building last mile fiber, and renting it to all comers on non-discriminatory terms, as you suggest, and encourage them to do so -- as I strongly suspect is Google's planned end-game -- then we might see some more sanity in the IAP business. I'd like to see a Jesus-load and a half more geographic locality of reference on the backbone too -- my RoadRunner Tampa packets to FiOS Tampa really ought not to have to go via *Dallas* on a regular basis -- but I guess that part's a lost cause. Cheers, -- jra

On Tue, Dec 14, 2010 at 04:38:27PM -0600, Richard A Steenbergen wrote:

I believe Comcast has made clear their position that they feel content providers should be paying them for access to their customers. I've seen them repeatedly state that they feel networks who send them too much traffic are "abusing their network".

That's rich, given the enormous quantity of spam sourced from Comcast's network over the last decade. (And yes, it's ongoing: 162 unique sources in the last hour noted at one small observation point.) Now I realize that SMTP abuse isn't exactly the most bandwidth-chewing problem. However, it's a surface indicator of underlying security issues, which in this particular case can be summarized as "one heck of a lot of zombies". Given that those systems are known-hostile and under the control of adversaries, it's certain that they're doing all kinds of other things that chew up a lot more bandwidth than the spam does. So maybe instead of engaging in brinkmanship with other network providers or spending engineering time trying to monetize DNS queries, Comcast should try solving this seven-year-old problem and *then* reassess whether or not the pipes are fat enough. ---rsk

On Wed, Dec 15, 2010 at 07:14:01PM -0500, Jon Lewis wrote:

On Wed, 15 Dec 2010, Rich Kulawiec wrote:

...

That's rich, given the enormous quantity of spam sourced from Comcast's network over the last decade. (And yes, it's ongoing: 162 unique sources in the last hour noted at one small observation point.)

Spam is irrelevant. In this context, abuse = sending large amounts of data to Comcast customers (at their request) without paying at the Comcast toll booth.

Yes, I know; I did read that in context and understand the point the original author was making. I probably should have made that clear.

...

Now I realize that SMTP abuse isn't exactly the most bandwidth-chewing problem. However, it's a surface indicator of underlying security issues, which in this particular case can be summarized as "one heck of a lot of zombies". Given that those systems are known-hostile and under the control of adversaries, it's certain that they're doing all kinds of other things that chew up a lot more bandwidth than the spam does.

It might even "improve" their ratios if they stopped those zombies from sendig spam, participating in DDoS's, etc. After all, that's outgoing traffic, and the less they send, the worse the ratio gets for networks sending data to Comcast.

True enough. But its continued presence, *seven years* after it was well-known to be a serious problem, tells us that Comcast either (a) can't or (b) won't run its network properly. So given this prima facie evidence of either (a) systemic, chronic incompetence or (b) systemic, chronic negligence, I think it's reasonable to wonder how many other aspects of their operation are just as horribly broken, and what the impact of that on their ability to carry steadily-increasing traffic might be. ---rsk