The pertinent question is what time period Level 3 was looking at / averaging when writing the blog post. Even if Comcast and Level 3 are not congested right at this moment, they were most definitely congested several years following their landmark agreement. A better question would be why that is/was. Drive Slow, Paul Wall On Fri, May 9, 2014 at 12:27 PM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:

Hi Jeff – I noticed the question posed here so thought I’d respond, perhaps at risk of stirring up a hornet’s nest given how long the last thread was. ;-) Anyway… there’s no congestion between Comcast and Level 3 connections, and we’re working collaboratively with Level 3. Given these facts, we have no reason to believe that Comcast is on their list.

- Jason Comcast

On 5/8/14, 1:18 PM, "=JeffH" <Jeff.Hodges@KingsMountain.com<mailto:Jeff.Hodges@KingsMountain.com>> wrote:

Level 3 accuses five unnamed US ISPs of abusing their market power in peering http://gigaom.com/2014/05/05/level-3-accuses-five-unnamed-us-isps-of-abusing...

"...I’d love to see Cogent, Google and other providers release their data next, so even if the FCC doesn’t want to pursue this, a growing cry of consumer outrage could push the agency to do something about a very real and difficult problem that’s crippling access to video content on 5 U.S. broadband networks. Level 3 didn’t name names, but based on the deals Netflix has signed and the complaints it has made about AT&T, I’m confident that AT&T, Verizon and Comcast are among the five. "

This is a lot of hand waving and self justification to attempt to validate the practice of [Access Network] trying to charge 3rd party entities to deliver the content that [Access Network]'s paying customers have requested over the service they already pay for, instead of [Access Network] having to themselves pay for the bandwidth because they know their customers can't leave them, and they know they have a big enough market presence that they can rent seek with impunity. Why pay for transit connectivity expansion, when it financially benefits you to instead let the links run over full, and charge the world individually for uncongested access to your captive customers? -Blake -Blake -Blake On Sat, May 10, 2014 at 10:04 AM, Rick Astley <jnanog@gmail.com> wrote:

That was an interesting read but it's not the whole story. Skip to the TL;DR if you'd like but I'll attempt to explain what happened. What he isn't saying is the roles of the companies involved have changed over the last 10 years. Mostly gone are the days that content providers and access networks each just gave a middleman/transit provider money to reach each other. "Content provider" has expanded to become "content delivery network" and "access network" has expanded their role to offer transit as well. If these networks have a large amount of traffic between them and are able to reach each other in multiple locations nationally what is the technical reason a 3rd party transit network is required instead of a direct peering relationship? From a purely technical perspective content and access at that scale can peer directly cutting out the middle man.

The reality is an increasingly directly peered Internet doesn't sit well if you are in the business of being the middle man. Now if you will, why do transit companies themselves charge content companies to deliver bits? How is it fair to be in the business of charging companies to receive their bits and hand them to a settlement free peer on the hook to deliver them, but not fair for content to just bypass the transit company and enter a paid peering agreement with the company delivering the bits? In this case paid peering is mutually beneficial to both companies involved and is typically cheaper for the content company than it would cost to send that traffic over transit.

What we have is a major shift in the market over the last 10 or so years. So why are these large nationally connected "access" networks charging Level 3 for ports? That's the elephant in the room here and to understand that you have to go back to where (to my knowledge) this dispute first went public. The most comprehensive description I have seen to date is the following Youtube video: https://www.youtube.com/watch?v=tR1sLLOYxnY

I recommend the video before continuing. "Level 3" is really both Level 3 transit and Level 3 CDN. Level 3 has already had a long standing precedent of justifying the right of an ISP to charge for content delivery. So what happens when Level 3 greatly expands their content delivery business and sends traffic to other ISP's over settlement free ports? The large access networks say "hey, content delivery is a billable service, you should know" and they ask Level 3 CDN for compensation. The middleman networks protest and say "Charging for content delivery is only OK if we do it, but not when you do it!" and their justification for this claim is made on the basis that unlike access networks they a) Have a large network and b) send a full table of prefixes.

So lets look at the first claim. Are the transit networks large? Yes, but especially in the case of North American traffic destined for North America they are typically smaller overall than the largest access networks who arguably have the lions share of equipment tasked with delivering the bits beyond just the colo. The 2nd claim is mostly a strawman and this is why. Middlemen still carry traffic not destined to directly connected peers but how they bill for it is largely based on volume of traffic, not the number of prefixes exchanged. The big content providers and the big access networks make up a majority of the traffic on the Internet even if they don't make up a majority of the prefixes.

TL;DR So the reason the ports are maxed out is the market has changed, access networks have attempted to change peering agreements to match the existing market conditions but the middleman networks are arguing they should be exempt from the long standing tradition of charging for content delivery they themselves helped to establish. Some middleman networks have responded by refusing payment to access networks for delivery and as a result, the paths have not been upgraded and remain congested.

End of TL;DR

The next part is (even) more opinion than fact so you are forgiven if you stop here. My opinion is this is a peering dispute more than something that should fall under net neutrality. If content companies sent letters to "middlmen" that said "In your statements to the public you made the case that content delivery to ISP's should be settlement free so we have decided to take your offer and refuse any further payment to you from here forward" how would they handle it? Likely those companies would not only find themselves congested but depeered.

A bunch of people say charging at both ends is double dipping but really modern access networks are now at least partly filling the role of transit as well as last mile delivery. Where "content" "transit" and "access" all have a presence in the same colo, paying more money to send traffic through transit first instead of just directly to access because of some dated definition of what the roles of those companies are supposed to be doesn't make sense to me. Hijacking NN to attempt to bring litigation into the matter to protect an old business model from a changing market makes even less sense. Seeing Level 3 publish half truths in what looks like an attempt to mislead the public on the matter is disappointing. I would expect it from maybe Cogent but I have higher expectations of Level 3.

Broadband providers obviously aren't without some blame in the matter either. One of them is allowing customer satisfaction to be so low that they are easy targets for misinformation as most the comments I have seen on the matter to date are more emotional than rational. They have other mistakes too but for the purpose of keeping this brief and because some of them have been heavily documented elsewhere I'll save them for another day.

On Thu, May 8, 2014 at 1:18 PM, =JeffH <Jeff.Hodges@kingsmountain.com>wrote:

...

Observations of an Internet Middleman (Level3) http://blog.level3.com/global-connectivity/observations- internet-middleman/

See also...

Level 3 accuses five unnamed US ISPs of abusing their market power in peering http://gigaom.com/2014/05/05/level-3-accuses-five-unnamed- us-isps-of-abusing-their-market-power-in-peering/

"... I’d love to see Cogent, Google and other providers release their data next, so even if the FCC doesn’t want to pursue this, a growing cry of consumer outrage could push the agency to do something about a very real and difficult problem that’s crippling access to video content on 5 U.S. broadband networks. Level 3 didn’t name names, but based on the deals Netflix has signed and the complaints it has made about AT&T, I’m confident that AT&T, Verizon and Comcast are among the five. "

=JeffH

The UK only does this with BT OpenReach since they were the telco monopoly that originated as a government entity. Virgin Media (well all the people who now form Virgin Media) built and operates their own fiber/HFC access networks, the same as MSOs in the US, and does not offer wholesale access and isn't treated as a utility. There are areas in the UK Virgin serves where the wholesale network does not, and areas where they offer much faster speeds, which is the same exact scenario as we have here. Just because Verizon isn't using VDSL/VDSL2 or hasn't brought FIOS to your area isn't Comcast's fault. The newer OpenReach wholesale fiber network is also partially subsidized by the government. I'm all for wholesale broadband access, but I wouldn't paint the situation in the UK as vastly different than here. We had the same thing the UK does now 10+ years ago with the CLECs and DSL providers like Covad, etc. but the regulations changed and dried up access. TWC did wholesale access during the same time; Earthlink Cable had quite a few customers back in the day through the arrangement, but it was complicated and ultimately your Internet pipe all still went through TWC. Phil On 5/10/14, 2:42 PM, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

Nice discussion about history & motivations. Not completely correct, but it's always fun to argue over history, and over motivations, since both are open to intepretation.

Personally, I am interested in the future, and specifically in market-driven solutions to our problems. Call me a capitalist if you like, but I believe in a functioning market, we can get a very good approximation of "fair".

If Company A and Company B have a mutual customer, and that customer needs both companies to perform a task, the market will find a way to make those two companies work together. Either that, or the customer will replace A or B, whichever the customer feels is underperforming, with Company C.

We have that situation today. Streaming Company wants to send End User of Broadband Company some content. If Streaming Company sucks - not enough titles, lousy customer service, high price, poor performance, etc., etc. - End User is free to select Streaming Company 2. And contrary to popular belief, there are plenty of "Streaming Company 2s" available. Besides NF, there is Hulu, Amazon, iTunes, iPlayer, etc. They might have different models, but they all allow you to access streaming content, so choice is available.

And here is where we get into the problem. Should End User believe Broadband Company sucks, they frequently cannot choose Broadband Company 2. I know I cannot, my choices are Comcast @ 100 Mbps or Verizon at 1.1 (yes, one-point-one) Mbps. So when Streaming Company sucks, but they suck because Broadband company is doing something I do not like, I cannot "vote with my wallet" and pick Broadband Company 2. I have no choice but to pick Streaming Company 2, even if I think the problem is Broadband Company's fault. (To be clear, I am not a NF subscriber - any more - and so this is not a NF/CC thing, I'm just talking generalities.)

Put more succinctly, there is no functioning market. therefore there cannot be a market-based solution.

Personally, I view that as about the most Un-American, Un-Capitalistic thing there is.

Lots of people have suggested a simple, if very difficult, fix to this problem. Make the underlying physical infrastructure a regulated monopoly, i.e. a Utility. Then allow anyone to run services over that physical infrastructure.

This is not pipe dream. The UK does it today. People there pick ISPs based on service, price, features, etc., not on "who paid off my local PUC".

And before anyone brings up the whole "the UK is more dense than the US", I preemptively call BS. There is more choice, faster speeds, and lower prices in the middle of no-where UK than downtown manhattan. Please just leave that argument where it belongs, in the dung heap.

Why can we not do something similar in the US? because the companies who own the lines have enough money to pay enough lobbyists to avoid even the promises they do make. (If anyone on this list is un-aware of things like the telcos promising ubiquitous high-speed BB years ago and never delivering, but never giving back their tax breaks or monopoly positions, you should be ashamed of yourselves.)

But hey, a guy can dream, right?

In the mean time, let's stop pretending that 'oh, L3 paid CC so they must be best friends'. L3 paid because They Had No Choice, and maybe because they see some long-term strategic benefit (e.g. they can charge others more later).

This is not a functioning market. This is a few players with Market Power charging Rents, which any first year econ major will explain is a _very_very_very_ bad place for the market to be.

-- TTFN, patrick

I agree with your summary. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On May 10, 2014, at 3:14 PM, Michael Conlen <mike@conlen.org> wrote:

If we ignore why and how the few high speed options exist for a moment and accept that it's "the way it is," then it seems reasonable that the place to put regulation is on them. At the same time cutting out middlemen is generally good for everyone but the middlemen.

My current opinion then is to let ISPs cut out the middlemen but ensure that services which don't pay fees get reasonable access; regulate peering and transit agreements (not just for access providers but across the board). ISPs should be responsible to keep their links congestion free and have fair and reasonable terms to connect to their networks. They can sell direct access to their network to anyone as long as they aren't selling QoS.

Comcast and Verizon can sell direct access to content providers but they cannot degrade service as leverage in negotiations.

That set of regulations would be utterly impossible to meaningfully enforce because so much of it depends on subjective evaluation. The various law firms involved (Comcast, AT&T, Verizon, et al.) would have a field day playing in the gray areas of any such set of rules, most likely creating a situation of exactly the opposite of what is intended.

A side effect would be that if peering agreements must be public and there are stated terms for various types of peering many of the silky peering games that get played and the silky peering disagreements that cause problems would be more difficult.

More likely, costs would go up for everyone for everything and the game wouldn’t change by all that much.

We could finally answer the age old question, "is company X a 'tier 1'. “

Since nobody has a real definition for “tier 1”, it’s a fairly meaningless question to begin with. (Yes, I am familiar with the alleged “does not pay for transit” definition, but I’ll point out that a completely disconnected network doesn’t pay for transit, either, but I doubt anyone would think they are a tier 1.) Owen

-- Mike

...

On May 10, 2014, at 14:42, "Patrick W. Gilmore" <patrick@ianai.net> wrote:

Nice discussion about history & motivations. Not completely correct, but it's always fun to argue over history, and over motivations, since both are open to intepretation.

Personally, I am interested in the future, and specifically in market-driven solutions to our problems. Call me a capitalist if you like, but I believe in a functioning market, we can get a very good approximation of "fair".

If Company A and Company B have a mutual customer, and that customer needs both companies to perform a task, the market will find a way to make those two companies work together. Either that, or the customer will replace A or B, whichever the customer feels is underperforming, with Company C.

We have that situation today. Streaming Company wants to send End User of Broadband Company some content. If Streaming Company sucks - not enough titles, lousy customer service, high price, poor performance, etc., etc. - End User is free to select Streaming Company 2. And contrary to popular belief, there are plenty of "Streaming Company 2s" available. Besides NF, there is Hulu, Amazon, iTunes, iPlayer, etc. They might have different models, but they all allow you to access streaming content, so choice is available.

And here is where we get into the problem. Should End User believe Broadband Company sucks, they frequently cannot choose Broadband Company 2. I know I cannot, my choices are Comcast @ 100 Mbps or Verizon at 1.1 (yes, one-point-one) Mbps. So when Streaming Company sucks, but they suck because Broadband company is doing something I do not like, I cannot "vote with my wallet" and pick Broadband Company 2. I have no choice but to pick Streaming Company 2, even if I think the problem is Broadband Company's fault. (To be clear, I am not a NF subscriber - any more - and so this is not a NF/CC thing, I'm just talking generalities.)

Put more succinctly, there is no functioning market. therefore there cannot be a market-based solution.

Personally, I view that as about the most Un-American, Un-Capitalistic thing there is.

Lots of people have suggested a simple, if very difficult, fix to this problem. Make the underlying physical infrastructure a regulated monopoly, i.e. a Utility. Then allow anyone to run services over that physical infrastructure.

This is not pipe dream. The UK does it today. People there pick ISPs based on service, price, features, etc., not on "who paid off my local PUC".

And before anyone brings up the whole "the UK is more dense than the US", I preemptively call BS. There is more choice, faster speeds, and lower prices in the middle of no-where UK than downtown manhattan. Please just leave that argument where it belongs, in the dung heap.

Why can we not do something similar in the US? because the companies who own the lines have enough money to pay enough lobbyists to avoid even the promises they do make. (If anyone on this list is un-aware of things like the telcos promising ubiquitous high-speed BB years ago and never delivering, but never giving back their tax breaks or monopoly positions, you should be ashamed of yourselves.)

But hey, a guy can dream, right?

In the mean time, let's stop pretending that 'oh, L3 paid CC so they must be best friends'. L3 paid because They Had No Choice, and maybe because they see some long-term strategic benefit (e.g. they can charge others more later).

This is not a functioning market. This is a few players with Market Power charging Rents, which any first year econ major will explain is a _very_very_very_ bad place for the market to be.

-- TTFN, patrick

++1 Andrew Fried andrew.fried@gmail.com On 5/10/14, 2:42 PM, Patrick W. Gilmore wrote:

Nice discussion about history & motivations. Not completely correct, but it's always fun to argue over history, and over motivations, since both are open to intepretation.

Personally, I am interested in the future, and specifically in market-driven solutions to our problems. Call me a capitalist if you like, but I believe in a functioning market, we can get a very good approximation of "fair".

If Company A and Company B have a mutual customer, and that customer needs both companies to perform a task, the market will find a way to make those two companies work together. Either that, or the customer will replace A or B, whichever the customer feels is underperforming, with Company C.

We have that situation today. Streaming Company wants to send End User of Broadband Company some content. If Streaming Company sucks - not enough titles, lousy customer service, high price, poor performance, etc., etc. - End User is free to select Streaming Company 2. And contrary to popular belief, there are plenty of "Streaming Company 2s" available. Besides NF, there is Hulu, Amazon, iTunes, iPlayer, etc. They might have different models, but they all allow you to access streaming content, so choice is available.

And here is where we get into the problem. Should End User believe Broadband Company sucks, they frequently cannot choose Broadband Company 2. I know I cannot, my choices are Comcast @ 100 Mbps or Verizon at 1.1 (yes, one-point-one) Mbps. So when Streaming Company sucks, but they suck because Broadband company is doing something I do not like, I cannot "vote with my wallet" and pick Broadband Company 2. I have no choice but to pick Streaming Company 2, even if I think the problem is Broadband Company's fault. (To be clear, I am not a NF subscriber - any more - and so this is not a NF/CC thing, I'm just talking generalities.)

Put more succinctly, there is no functioning market. therefore there cannot be a market-based solution.

Personally, I view that as about the most Un-American, Un-Capitalistic thing there is.

Lots of people have suggested a simple, if very difficult, fix to this problem. Make the underlying physical infrastructure a regulated monopoly, i.e. a Utility. Then allow anyone to run services over that physical infrastructure.

This is not pipe dream. The UK does it today. People there pick ISPs based on service, price, features, etc., not on "who paid off my local PUC".

And before anyone brings up the whole "the UK is more dense than the US", I preemptively call BS. There is more choice, faster speeds, and lower prices in the middle of no-where UK than downtown manhattan. Please just leave that argument where it belongs, in the dung heap.

Why can we not do something similar in the US? because the companies who own the lines have enough money to pay enough lobbyists to avoid even the promises they do make. (If anyone on this list is un-aware of things like the telcos promising ubiquitous high-speed BB years ago and never delivering, but never giving back their tax breaks or monopoly positions, you should be ashamed of yourselves.)

But hey, a guy can dream, right?

In the mean time, let's stop pretending that 'oh, L3 paid CC so they must be best friends'. L3 paid because They Had No Choice, and maybe because they see some long-term strategic benefit (e.g. they can charge others more later).

This is not a functioning market. This is a few players with Market Power charging Rents, which any first year econ major will explain is a _very_very_very_ bad place for the market to be.

On May 10, 2014 at 22:34 randy@psg.com (Randy Bush) wrote:

imiho think vi hart has it down simply and understandable by a lay person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my friends in last mile providers disagree. i take that as a good sign.

Yeah, well, for extra credit integrate Akamai into that story. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

On 10/05/2014 22:34, Randy Bush wrote:

imiho think vi hart has it down simply and understandable by a lay person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my friends in last mile providers disagree. i take that as a good sign.

From a financial point of view, the content providers will say that access

From the proverbial helicopter viewpoint, we are walking towards a situation where the short-term business actions of the individual companies involved in the industry is going to lead towards customers being hurt and

Vi's analogy is wrong on a subtle but important point. In the analogy, the delivery company needs to get a bunch of new trucks to handle the delivery but as the customer is paying for each delivery instances, the delivery company's costs are covered by increased end-user charges. In the net neutrality debate, the last mile service providers are in a position where they need to upgrade their access networks, but the end-user pricing is not necessarily keeping pace. There are lot of ways to argue this point, depending on whether you're the user, the access provider or the content provider. providers need to charge their end users in a way which reflects their usage requirements because let's face it, it's the users that are pulling the traffic - they're not sending traffic to arbitrary IP addresses just for the fun of it. The end users will say that they're only going to pay market rate for their services, and they won't care whether this covers their costs or not. The access providers will say that they're only upgrading to deal with the additional requirements of the larger content providers, particularly the CDNs and the video streaming services, and that the going market rate doesn't allow them to charge the end users more. Besides, it's a whole pile easier to chase a small number of companies for a large amount of money than it is to chase a large number of customer for a small amount of money. Even better, if you chase the the content sources for cash, you can do this without increasing customer prices which means you can stay more competitive in the sales market. So from a business perspective it makes lots of sense to deprioritise the large companies that don't pay in favour of the ones that do. Those who pay get better service for their customers; seems fair, right? this means that the likely long-term outcome is more regulation and legislative control imposed on the industry. It is another tragedy of the commons. Nick

On 12/05/2014 15:27, Clayton Zekelman wrote:

I think that's where the biggest gulf exists. It doesn't seem fair. It seems like extortion. The last mile access guys are the gatekeepers to the end user, with little competition.

that is the core problem: lack of competition. Net neutrality is a kludge to deal with a specific type of failure in the market. Nick

On Mon, 12 May 2014 15:02:28 +0200, Nick Hilliard said:

a small amount of money. Even better, if you chase the the content sources for cash, you can do this without increasing customer prices which means you can stay more competitive in the sales market.

Thank you, I needed my morning chuckle. :) We know which last-mile providers are doing this. Over what percentage of their territory is there any *realistic* competition?

On May 12, 2014, at 6:02 AM, Nick Hilliard <nick@foobar.org> wrote:

On 10/05/2014 22:34, Randy Bush wrote:

...

imiho think vi hart has it down simply and understandable by a lay person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my friends in last mile providers disagree. i take that as a good sign.

Vi's analogy is wrong on a subtle but important point. In the analogy, the delivery company needs to get a bunch of new trucks to handle the delivery but as the customer is paying for each delivery instances, the delivery company's costs are covered by increased end-user charges.

Two words nuke your suggestion here: Amazon Prime Owen

That's true for any given network topology up to the limit of capacity. On cable plant, node splitting does incur significant costs. May cable companies who built their fiber optic network with 200-500 customers per RF node are now finding they have to reduce that density. They don't have enough feeder fiber in their network to accommodate. They have a few choices: 1) overlash more fiber, 2) use more WDM, 3) build more pop sites. The incremental cost of delivering that one package above what the truck will hold is a doozie. It requires the delivery company to buy bigger trucks, or buy more trucks and run smaller routes. At 10:30 AM 12/05/2014, joel jaeggli wrote:

Once you build the capacity to reach every delivery point every-day it's maybe not enough to hope that people utilize that facility . decreasing the cost per package requires higher unit volume. The incremental cost of delivering the second package is much lower than the first.

--- Clayton Zekelman Managed Network Systems Inc. (MNSi) 3363 Tecumseh Rd. E Windsor, Ontario N8W 1H4 tel. 519-985-8410 fax. 519-985-8409

On 5/12/14, 10:07 AM, "Owen DeLong" <owen@delong.com> wrote:

On May 12, 2014, at 6:02 AM, Nick Hilliard <nick@foobar.org> wrote:

...

On 10/05/2014 22:34, Randy Bush wrote:

...

imiho think vi hart has it down simply and understandable by a lay person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my friends in last mile providers disagree. i take that as a good sign.

Vi's analogy is wrong on a subtle but important point. In the analogy, the delivery company needs to get a bunch of new trucks to handle the delivery but as the customer is paying for each delivery instances, the delivery company's costs are covered by increased end-user charges.

Two words nuke your suggestion here: Amazon Prime

Amazon Prime isn’t a flat-rate delivery service for the delivery company, else it’d be called FedUPS Prime. It’s a flat rate shipping subscription for *Amazon*, and is likely a loss leader to ensure better stickiness of Amazon’s potential customers. They may have a great deal of negotiating leverage on their delivery partners to reduce their shipping costs, and the sheer volume of Amazon warehouses mean that they can take advantage of proximity to reduce costs further (like a CDN), but I haven’t seen anything implying that they’ve been successful in negotiating a contract that is insensitive to the *amount* of items being shipped. Wes George This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

On 5/12/14, 10:55 AM, "Owen DeLong" <owen@delong.com<mailto:owen@delong.com>> wrote: Who cares? It’s insensitive from the end-customer perspective. Same as what I pay to Comcast is insensitive to my usage. Amazon hasn’t negotiated insensitive pricing with their shipping companies, just as Comcast hasn’t negotiated insensitive pricing for infrastructure upgrades. Seems to me that the analogy holds. Interesting analogy. To digress a bit. UPS/FedEx/UPS generally serve as the package delivery network for Amazon Prime shipments, though sometimes they do so via smaller regional package delivery networks. Also Amazon seems to be experimenting with direct delivery in some areas in order to provide a level of delivery quality they don’t feel they can get via 3rd party (same day delivery). As you describe, the end users (consumers of Amazon Prime) are insensitive to the shipping cost due to the flat fee the consumer paid to Amazon. This dynamic appears to have contributed to a 2013 holiday shipping season that saw package delays (packages sent exceeded capacity to deliver on time), which appears to have played a role in FedEx’s recent announcement of a plan to charge shippers by size, rather than purely by weight. Some analysis I read was that this may prompt companies like Amazon to get more efficient in what they send to package delivery networks; packages will become smaller (think fewer of those inflatable plastic bags inside). Jason

On 05/12/2014 12:48 PM, Livingood, Jason wrote:

Also Amazon seems to be experimenting with direct delivery in some areas in order to provide a level of delivery quality they don’t feel they can get via 3rd party

Someone wake me up when RFC 1149 gets updated to "IP Datagrams on Drone Carriers"

--As of May 12, 2014 3:02:28 PM +0200, Nick Hilliard is alleged to have said:

On 10/05/2014 22:34, Randy Bush wrote:

...

imiho think vi hart has it down simply and understandable by a lay person. <http://vihart.com/net-neutrality-in-the-us-now-what/>. my friends in last mile providers disagree. i take that as a good sign.

Vi's analogy is wrong on a subtle but important point. In the analogy, the delivery company needs to get a bunch of new trucks to handle the delivery but as the customer is paying for each delivery instances, the delivery company's costs are covered by increased end-user charges.

In the net neutrality debate, the last mile service providers are in a position where they need to upgrade their access networks, but the end-user pricing is not necessarily keeping pace.

--As for the rest, it is mine. So the fact that the USA has higher prices than many other countries, for slower service, and those prices are rising (mine went up three times in the past year, including them starting to charge rent for a cable modem I bought when I signed up, for the same service) doesn't mean anything? Or the fact that they are one of the most profitable market segments in the country? They have the money. They have the ability to get more money. *They see no reason to spend money making customers happy.* They can make more profit without it. Daniel T. Staal --------------------------------------------------------------- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. ---------------------------------------------------------------

On Wednesday, May 14, 2014 09:04:11 AM Jean-Francois Mezei wrote:

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

That depends on your point-of-view and/or interpretation of "growth". In the last several years, networks have been adding more capacity than they have customers. I suppose, for a number of operators now, growth is in bandwidth, which is non- linear to customer growth. Mark.

So they seek new sources of revenues, and/or attempt to thwart

...

competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers. I'm not endorsing this; just pointing out that you two are actually in agreement here. -- Hugo On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote:

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

...

On 14-05-13 22:50, Daniel Staal wrote:

They have the money. They have the ability to get more money. *They see

...

no reason to spend money making customers happy.* They can make more profit without it.

There is the issue of control over the market. But also the pressure from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

...

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

...

When you combine this with threath to large incumbents's media and media distribution endeavours by the likes of Netflix (and cat videos on Youtube), large incumbents start thinking about how they will be able to continue to grow revenus/profits when customers will shift spending to vspecialty channels/cableTV to Netflix and customer growth will not compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

...

So they seek new sources of revenues, and/or attempt to thwart competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

...

The current trend is to "if you can't fight them, jon them" where cablecos start to include the Netflix app into their proprietary set-top boxes. The idea is that you at least make the customer continue to use your box and your remote control which makes it easier for them to switch between netflix and legacy TV.

True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing

...

to add the Netflix app onto their proprietary STBs also play peering capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

Respectfully, this is a highly inaccurate "sound bite" - Kevin 215-313-1083

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

...

So they seek new sources of revenues, and/or attempt to thwart

...

competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

...

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

...

On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote:

...

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

On 14-05-13 22:50, Daniel Staal wrote:

They have the money. They have the ability to get more money. *They see

...

no reason to spend money making customers happy.* They can make more profit without it.

There is the issue of control over the market. But also the pressure from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

...

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

...

When you combine this with threath to large incumbents's media and media distribution endeavours by the likes of Netflix (and cat videos on Youtube), large incumbents start thinking about how they will be able to continue to grow revenus/profits when customers will shift spending to vspecialty channels/cableTV to Netflix and customer growth will not compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

...

So they seek new sources of revenues, and/or attempt to thwart competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

...

The current trend is to "if you can't fight them, jon them" where cablecos start to include the Netflix app into their proprietary set-top boxes. The idea is that you at least make the customer continue to use your box and your remote control which makes it easier for them to switch between netflix and legacy TV. True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing

...

to add the Netflix app onto their proprietary STBs also play peering capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

I don’t disagree. However, given the choice between Comcast and broadband services in NL, Chatanooga, or Seoul, just to name a few, Comcast loses badly. Choosing between Comcast and a legacy Telco is like choosing between legionnaire’s disease and SARS. Owen On May 14, 2014, at 5:15 PM, Jared Mauch <jared@puck.nether.net> wrote:

Owen,

I've seen a vast difference between Comcast and others in the "marketplace". Right now, if I had the choice between Comcast and a "legacy" telco, I would pick Comcast hands-down for:

a) performance b) IPv6 support c) willingness to work on issues

- Jared

On May 14, 2014, at 5:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

...

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

...

So they seek new sources of revenues, and/or attempt to thwart

...

competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

...

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

...

On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote:

...

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

On 14-05-13 22:50, Daniel Staal wrote:

They have the money. They have the ability to get more money. *They see > no reason to spend money making customers happy.* They can make more > profit without it.

There is the issue of control over the market. But also the pressure from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

...

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

...

When you combine this with threath to large incumbents's media and media distribution endeavours by the likes of Netflix (and cat videos on Youtube), large incumbents start thinking about how they will be able to continue to grow revenus/profits when customers will shift spending to vspecialty channels/cableTV to Netflix and customer growth will not compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

...

So they seek new sources of revenues, and/or attempt to thwart competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

...

The current trend is to "if you can't fight them, jon them" where cablecos start to include the Netflix app into their proprietary set-top boxes. The idea is that you at least make the customer continue to use your box and your remote control which makes it easier for them to switch between netflix and legacy TV. True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing

...

to add the Netflix app onto their proprietary STBs also play peering capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

On May 15, 2014, at 10:18 AM, Jean-Francois Mezei <jfmezei_nanog@vaxination.ca> wrote:

On 14-05-15 10:26, Owen DeLong wrote:

...

Choosing between Comcast and a legacy Telco is like choosing between legionnaire’s disease and SARS.

Twisted pair is certantly "legacy".

Is there a feeling that coax cable/DOSCIS is also "legacy" in terms of current capacity/speeds ? Or is that technology still considered viable against FTTH ?

I realise that business practices make north american incumbents undesirable compared to the rest of the world, especially Comcast's dirty tricks with Netflix as an example.

But in terms of the last mile technology and wiring (for instance, homes per HFC node) sre north american cavlecos up to par with the rest of the world ?

I am not speaking specifically about any one company here. In North America, very few places have any level of FTTH. If you are in a rural area with USF subsidies, you are more likely to have FTTH than many urban areas. Co-ax, or if you’re somewhat lucky, HFC is about the best last mile technology available to most US subscribers. In states where some city invested in municipal FTTH on an open-access basis, the incumbent $CABLECOS and $TELCOS have fought hard to push legislation making it illegal for other cities in the state to do the same. The state of broadband networks in the US in general can best be described as pathetic and/or apathetic when it comes to the consumer’s interest. Lilly Tomlin summed this up very well in a number of her early comedy sketches where she pretended to be a telephone company operator. Her catch phrase was “We don’t care. We don’t have to. We’re the phone company!” Further, it appears that several of the $CABLECOS and $TELCOS will actually attempt to quash their more vocal opponents by discussing public comments they make on a personal basis with said opponents employer and using them as a “negotiating tactic”. Personally, I think this is one of the most underhanded and lowest forms of an act of desperation to try and squash public debate. To be very clear… This statement is absolutely not targeted at any one company. There were several. Owen

Upgrades/buildout are happening every day. They are continuous to keep ahead of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, Ookla, etc What is not well known is that Comcast has been an existing commercial transit business for 15+ years (with over 8000 commercial fiber customers). Comcast also has over 40 balanced peers with plenty of capacity, and some of the largest Internet companies as customers. - Kevin 215-313-1083

On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com> wrote:

Oh, please do explicate on how this is inaccurate…

Owen

...

On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

...

So they seek new sources of revenues, and/or attempt to thwart

...

competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

...

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

...

...

On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote:

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

On 14-05-13 22:50, Daniel Staal wrote:

They have the money. They have the ability to get more money. *They see > no reason to spend money making customers happy.* They can make more > profit without it.

There is the issue of control over the market. But also the pressure from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

...

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

...

When you combine this with threath to large incumbents's media and media distribution endeavours by the likes of Netflix (and cat videos on Youtube), large incumbents start thinking about how they will be able to continue to grow revenus/profits when customers will shift spending to vspecialty channels/cableTV to Netflix and customer growth will not compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

...

So they seek new sources of revenues, and/or attempt to thwart competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

...

The current trend is to "if you can't fight them, jon them" where cablecos start to include the Netflix app into their proprietary set-top boxes. The idea is that you at least make the customer continue to use your box and your remote control which makes it easier for them to switch between netflix and legacy TV. True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing

...

to add the Netflix app onto their proprietary STBs also play peering capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

There is no gaming on measurements and disputes are isolated and temporary with issues not unique over the history of the internet. I think all the same rhetorical quotes continue to be reused - Kevin

On May 15, 2014, at 11:43 AM, "Scott Berkman" <scott@sberkman.net> wrote:

Unfortunately these build-outs are primarily in subscriber facing bandwidth and number of headend locations (to add more customers to the network). These peering point/transit connection issues have been going on for a long time, evidenced by Level 3 coming out with this post. Comcast is also suspiciously absent from public exchanges (TelX's TIE would be one example) while many of their competitors participate for the benefit of the Internet as a whole and their customers.

Measured broadband is also a game, because its very easy for large providers to give priority to (or otherwise "help") known speed test and similar sites, giving customers a false impression of their available capacity or performance. We've all seen cases where customers have some amazing result on their favorite test site, and then real world performance can't even come close.

That said, if Comcast does or is making efforts to finally resolve this, more power to them and congratulations to their customers. Unfortunately trying to brute-force the industry and external content providers tells a very different story. Where is Comcast's official blog post showing evidence as to where they do ensure their peering and or transit to the largest Tier 1 providers are not congested? Instead all we see are policy arguments about who should pay for what, while users continue to suffer.

This is really similar to when TV providers have spats with content owners, and the result is the end users missing out on something they are paying for. It is good for related industries and the large players in each to keep working with each other in open ways to keep pricing reasonable (as opposed to working together in hiding to price fix), but it is not OK to do so by throwing tantrums and making everyone involved suffer.

-Scott

...

On 05/15/2014 10:57 AM, McElearney, Kevin wrote: Upgrades/buildout are happening every day. They are continuous to keep ahead of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, Ookla, etc

What is not well known is that Comcast has been an existing commercial transit business for 15+ years (with over 8000 commercial fiber customers). Comcast also has over 40 balanced peers with plenty of capacity, and some of the largest Internet companies as customers.

- Kevin

215-313-1083

...

On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com> wrote:

Oh, please do explicate on how this is inaccurate…

Owen

...

On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

> So they seek new sources of revenues, and/or attempt to thwart >> competition any way they can. No to the first. Yes to the second. If they were seeking new sources of > revenue, they'd be massively expanding into un/der served markets and > aggressively growing over the top services (which are fat margin). Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

>> On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote: >> >> On 2014-05-14 02:04, Jean-Francois Mezei wrote: >> >> On 14-05-13 22:50, Daniel Staal wrote: >> >> They have the money. They have the ability to get more money. *They see >>> no reason to spend money making customers happy.* They can make more >>> profit without it. >> There is the issue of control over the market. But also the pressure >> from shareholders for continued growth. > > Yes. That is true. Except that it's not. > > How do service providers grow? Let's explore that: > > What is growth for a transit provider? > > More (new) access network(s) (connections). > More bandwidth across backbone pipes. > > > What is growth for access network? > More subscribers. > > Except that the incumbent carriers have shown they have no interest in > providing decent bandwidth to anywhere but the most profitable rate > centers. I'd say about 2/3 of the USA is served with quite terrible access. > > > > >> The problem with the internet is that while it had promises of wild >> growth in the 90s and 00s, once penetration reaches a certain level, >> growth stabilizes. > Penetration is ABYSMAL sir. Huge swaths of underserved americans exist. > > > >> When you combine this with threath to large incumbents's media and media >> distribution endeavours by the likes of Netflix (and cat videos on >> Youtube), large incumbents start thinking about how they will be able to >> continue to grow revenus/profits when customers will shift spending to >> vspecialty channels/cableTV to Netflix and customer growth will not >> compensate. > Except they aren't. Even in the most profitable rate centers, they've > declined to really invest in the networks. They aren't a real business. You > have to remember that. They have regulatory capture, natural/defacto > monopoly etc etc. They don't operate in the real world of > risk/reward/profit/loss/uncertainty like any other real business has to. > > > >> So they seek new sources of revenues, and/or attempt to thwart >> competition any way they can. > No to the first. Yes to the second. If they were seeking new sources of > revenue, they'd be massively expanding into un/der served markets and > aggressively growing over the top services (which are fat margin). They did > a bit of an advertising campaign of "smart home" offerings, but that seems > to have never grown beyond a pilot. > > > >> The current trend is to "if you can't fight them, jon them" where >> cablecos start to include the Netflix app into their proprietary set-top >> boxes. The idea is that you at least make the customer continue to use >> your box and your remote control which makes it easier for them to >> switch between netflix and legacy TV. > True. I don't know why one of the cablecos hasn't licensed roku, added > cable card and made that available as a "hip/cool" set top box offering and > charge another 10.00 a month on top of the standard dvr rental. > > > > Would be interesting to see if those cable companies that are agreeing >> to add the Netflix app onto their proprietary STBs also play peering >> capacity games to degrade the service or not. > So how is the content delivered? Is it over the internet? Or is it over > the cable plant, from cable headends?

This is a smart group. If if that was true I think every internet site / service one visits from home would be a negatively impacted. That is not the case As I said before, Comcast also has over 40 balanced peers with plenty of capacity. Wholesale $$ are very small, highly competitive and only "skin in the game" to promote efficiencies - Kevin

On May 15, 2014, at 12:01 PM, "Jared Mauch" <jared@puck.nether.net> wrote:

...

On May 15, 2014, at 11:50 AM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

There is no gaming on measurements and disputes are isolated and temporary with issues not unique over the history of the internet. I think all the same rhetorical quotes continue to be reused

Kevin,

in the past most issues were transient for a few months as both sides got complaints, but while at RIPE earlier this week someone commented to me: there's no one provider you can buy access from to get a packet-loss free connection to all their other business partners/customers. This hurts the entire marketplace when there is persistent congestion.

Some of these issues are related to (as Craig called them) "Hypergiants" (OTT) but others are due to providers having poor capital models so they don't have "budget" for upgrading unless someone pays for that upgrade, vs seeing their existing customer base as that source for the capital.

As an engineer, I'm hopeful that those responsible for budgeting will do the right thing. As a greedy capitalist, please pay me more $$$. It does feel a bit like tic-tac-toe with zero players in wargames though, the only way to win is to not play [games].

- Jared

On 5/15/14, 12:49 PM, "arvindersingh@mail2tor.com" <arvindersingh@mail2tor.com> wrote:

I have two issues with the comments:

2. You mention that all packets treated equally - no games. Why does AS7922 assign the speed test different DSCP from regular internet connection?

I have no idea what you are talking about. Our Internet traffic, including to speedtest web sites, is all best effort class data. Do you have more specific information? Jason

On Fri, May 16, 2014 at 7:56 AM, Vinny Abello <vinny@abellohome.net> wrote:

I think he's questioning why packets from speedtest.comcast.net have CS1 if everything is supposedly equal, and what that is used for. A quick Wireshark shows that to be true right now running to your Plainfield, NJ speedtest site, and my network peers directly with Comcast.

are you measuring inside the (for this) comcast network or after your cable-modem? I recall that the cable-modem(s) often (by docsis config) impose some qos markings on the lan-side of the connection. I think they can do the same on the WAN side for traffic leaving your site to the tubes... but you probably can't measure that as easily as with wireshark on your pc. -chris (also, is there some other equipment between your wiresharker and the cable-modem? could that equipment be re-marking/marking as well?)

Jason, In your first reply you mention a lot of "we're all good, we comply, we don't do x etc" However you seem to have forgotten to reply to question #1 that Arvinder asked. (#2 you were able to reply) http://comcrust.com/ is already four years old it would seem enough time to get an upgrade in place. Our tata upgrades are usually in place after a couple of weeks. Thanks, Bas On Fri, May 16, 2014 at 4:58 PM, Livingood, Jason < Jason_Livingood@cable.comcast.com> wrote:

On 5/16/14, 7:56 AM, "Vinny Abello" <vinny@abellohome.net> wrote:

...

I think he's questioning why packets from speedtest.comcast.net have CS1 if everything is supposedly equal, and what that is used for. A quick Wireshark shows that to be true right now running to your Plainfield, NJ speedtest site, and my network peers directly with Comcast.

I'm kind of curious too. What is the purpose of this? Is it the traditional purpose of CS1 to be less than best effort or something else? If this is the case it seems Comcast would be purposely putting themselves at a disadvantage in speed tests when congestion is involved... or is this possibly on purpose to make peering problems look even worse during congestion?

Ah! That makes sense now. CS1 is used internally to mark best effort Internet traffic. This has often caused confusion when folks see our markings. If folks want to send me any data off-list that you think merits further investigation, let me know (never know if something someplace is an honest config error).

Jason

Blake Dunlap <ikiris@gmail.com> wrote:

And the "unbalanced" peers / transit?

Surely it is too much to expect a service provider to actually provide service even if it is not entirely fair and balanced. It's not like, you know, anyone was paying them to provide a service ... [...rewind...] <Kevin_McElearney@cable.comcast.com> wrote:

This is a smart group.

Well, smart enough to at least try to see it for what it actually is. Telling us we're smart and then expecting us to swallow a load doesn't quite seem to work, judging from the last few responses you've had. Some of us are actually businesspeople, so we understand the issues from multiple dimensions. Including historical ones, where there are both examples of Monopolies Gone Wild! (Spring Break Edition!) and also Government Regulation Gone Overboard. If you'd like to say that you're trying to leverage as much revenue as possible by taking advantage of new trends (i.e. cord cutting) in a customer base that's at least partially without other reasonable options, while keeping investment costs as low as possible, well, then, we have the potential for an honest conversation. But if you're going to tell us about how you've managed to acquire transit customers, that feels like the start of a dishonest discussion because basically most of us here wouldn't buy transit from a cable company unless it was the only available option, or there was some other distorting reason - such as congestion - that caused such an arrangement to be needed. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On 5/15/14, 11:58 AM, Joe Greco wrote:

2) Netflix purchases 5Mbps "fast lane"

I appreciate Joe's use of quotation marks here. A lot of the dialog has included this 'fast lane' terminology, yet all of us know there's no 'fast lane' being constructed, rather just varying degrees of _slow_ applied to existing traffic. It's a shame the use of 'fast lane' is ubiquitous in this argument. If the local distribution networks would like to actually build something fast, then this would be a different story. -Ryan Brooks

That link is broken and insists that I install a windows upgrade for Flash on my Mac. Owen On May 15, 2014, at 10:17 AM, Paul Ferguson <fergdawgster@mykolab.com> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

On 5/15/2014 10:06 AM, Ryan Brooks wrote:

...

It's a shame the use of 'fast lane' is ubiquitous in this argument. If the local distribution networks would like to actually build something fast, then this would be a different story.

Okay, then call it the "faster lane" or the "uncongested lane" or something that actually reflects bias and preferential treatment. It's a done deal now:

http://www.washingtonpost.com/blogs/the-switch/wp/2014/05/15/fcc-approves-pl...

FYI,

- - ferg

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlN09roACgkQKJasdVTchbJPlgEAtBpp0TKNZcIdrDkVP75Tni7a O9PvcnKZdaPNuNUpOb0A/RQ5hvrqPAu/QLSp8dPbcDSO5Zad8Z3JG67UfI6yaeJH =DhnX -----END PGP SIGNATURE-----

That link is broken and insists that I install a windows upgrade for = Flash on my Mac.

Try http://arstechnica.com/tech-policy/2014/05/fcc-votes-for-internet-fast-lanes... ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Its not really that complex, if you think about it having 10000s of 'movieco' with the same priority is the status quo. At the end of the day the QoS mechanics in DOCSIS are pretty straightforward and rely on service flows, while service flows can have equal priority I doubt most operators will sell more than a few (perhaps just one) top priority in a given a category. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Thu, May 15, 2014 at 1:22 PM, Christopher Morrow <morrowc.lists@gmail.com

wrote:

On Thu, May 15, 2014 at 1:06 PM, Ryan Brooks <ryan@hack.net> wrote:

...

On 5/15/14, 11:58 AM, Joe Greco wrote:

...

2) Netflix purchases 5Mbps "fast lane"

I appreciate Joe's use of quotation marks here. A lot of the dialog has included this 'fast lane' terminology, yet all of us know there's no 'fast lane' being constructed, rather just varying degrees of _slow_ applied to existing traffic.

please correct me if I'm wrong, but 'fast lane' really is (in this example): 'cableco' port from 'moviecompany' has 'qos' marking configuration to set all 'moviecompany' traffic (from this port!) to some priority level.

customer-port to 'cableco' has 'qos' handling/queuing that will ensure '5mbps' of 'moviecompany' is always going to get down the link to the customer, regardless of the other traffic the customer is requesting.

right? (presume that in the rest of the 'cableco' network is protecting 'moviecompany' traffic as well, of course)

So, when there are 1 'moviecompany' things to prioritize and deliver that's cool... but what about when there are 10? 100? 1000? doesn't the queuing get complicated? what if the 'cableco' customer with 10mbps link has 3 people in the location all streaming from 3 different 'moviecompany' organizations which have paid for 'fastlane' services?

3 x 5 == 15 ... not 10. How will 'cableco' manage this when their 100gbps inter-metro links are seeing +100gbps if 'fastlane' traffic and 'fastlane' traffic can't make it to the local metro from the remote one?

This all seems much, much more complicated and expensive than just building out networking, which they will have to do in the end anyway, right? Only with 'fastlanes' there's extra capacity management and configuration and testing and ... all on top of: "Gosh, does the new umnptyfart card from routerco actually work in old routerco routers?"

This looks, to me, like nuttiness... like mutually assured destruction that the cableco folk are driving both parties into intentionally.

-chris

BTW: I didn't use a particular 'cable company' name for 'cableco', nor did I use a particular streaming media company for 'moviecompany'... Also, 'cableco' is short-hand for 'lastmile-consumer-provider-network'. Less typing was better, for me, I thought.

Chris, You're not reading what I said, nor did I make a statement anything like one of the silly things you referenced (640k ram etc). Prioritization isn't that complex and today we handle the maximum amount of complexity already since everything is the same priority right now. You're trying to make the statement that giving multiple content providers priority somehow makes connectivity unworkable for consumers as if we don't have this problem already. Consumers can easily starve themselves of bandwidth with video or any other content and almost no connections in the US have any sort of intelligent fair usage buffering provided by the service provider. This is true for both cable, telco, and other operators. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Thu, May 15, 2014 at 2:01 PM, Christopher Morrow <morrowc.lists@gmail.com

wrote:

On Thu, May 15, 2014 at 1:48 PM, Scott Helms <khelms@zcorum.com> wrote:

...

Its not really that complex, if you think about it having 10000s of 'movieco' with the same priority is the status quo. At the end of the day the QoS mechanics in DOCSIS are pretty straightforward and rely on service flows, while service flows can have equal priority I doubt most operators will sell more than a few (perhaps just one) top priority in a given a category.

yes, there will only ever be 5 computers. or you couldn't possibly need more than 640kb of ram..... or more than 4billion 'ip addresses'.

I don't think you have to get to more than 10 or 20 of the stated examples before things get dicey ... Once a set of customers experience (and can measure) the effect, they'll back their complaints up to 'moviecompany' and some set of contract penalties will kick in, I suspect.

Sure, if there is only one it's not a problem, but there are already not just one...

...

Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Thu, May 15, 2014 at 1:22 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

...

On Thu, May 15, 2014 at 1:06 PM, Ryan Brooks <ryan@hack.net> wrote:

...

On 5/15/14, 11:58 AM, Joe Greco wrote:

...

2) Netflix purchases 5Mbps "fast lane"

I appreciate Joe's use of quotation marks here. A lot of the dialog has included this 'fast lane' terminology, yet all of us know there's no 'fast lane' being constructed, rather just varying degrees of _slow_ applied to existing traffic.

please correct me if I'm wrong, but 'fast lane' really is (in this example): 'cableco' port from 'moviecompany' has 'qos' marking configuration to set all 'moviecompany' traffic (from this port!) to some priority level.

customer-port to 'cableco' has 'qos' handling/queuing that will ensure '5mbps' of 'moviecompany' is always going to get down the link to the customer, regardless of the other traffic the customer is requesting.

right? (presume that in the rest of the 'cableco' network is protecting 'moviecompany' traffic as well, of course)

So, when there are 1 'moviecompany' things to prioritize and deliver that's cool... but what about when there are 10? 100? 1000? doesn't the queuing get complicated? what if the 'cableco' customer with 10mbps link has 3 people in the location all streaming from 3 different 'moviecompany' organizations which have paid for 'fastlane' services?

3 x 5 == 15 ... not 10. How will 'cableco' manage this when their 100gbps inter-metro links are seeing +100gbps if 'fastlane' traffic and 'fastlane' traffic can't make it to the local metro from the remote one?

This all seems much, much more complicated and expensive than just building out networking, which they will have to do in the end anyway, right? Only with 'fastlanes' there's extra capacity management and configuration and testing and ... all on top of: "Gosh, does the new umnptyfart card from routerco actually work in old routerco routers?"

This looks, to me, like nuttiness... like mutually assured destruction that the cableco folk are driving both parties into intentionally.

-chris

BTW: I didn't use a particular 'cable company' name for 'cableco', nor did I use a particular streaming media company for 'moviecompany'... Also, 'cableco' is short-hand for 'lastmile-consumer-provider-network'. Less typing was better, for me, I thought.

On Thu, May 15, 2014 at 1:06 PM, Ryan Brooks <ryan@hack.net> wrote:

...

On 5/15/14, 11:58 AM, Joe Greco wrote:

...

2) Netflix purchases 5Mbps "fast lane"

I appreciate Joe's use of quotation marks here. A lot of the dialog has included this 'fast lane' terminology, yet all of us know there's no 'fast lane' being constructed, rather just varying degrees of _slow_ applied to existing traffic.

please correct me if I'm wrong, but 'fast lane' really is (in this example): 'cableco' port from 'moviecompany' has 'qos' marking configuration to set all 'moviecompany' traffic (from this port!) to some priority level.

I think that's a possibility, but that we're actually talking at a less-technical level. [...]

3 x 5 == 15 ... not 10. How will 'cableco' manage this when their 100gbps inter-metro links are seeing +100gbps if 'fastlane' traffic and 'fastlane' traffic can't make it to the local metro from the remote one?

#whocares You've made a technical implementation issue out of a mostly non-tech issue.

This all seems much, much more complicated and expensive than just building out networking, which they will have to do in the end anyway, right? Only with 'fastlanes' there's extra capacity management and configuration and testing and ... all on top of: "Gosh, does the new umnptyfart card from routerco actually work in old routerco routers?"

I certainly agree. This isn't a technical issue though. A majority of the people on this list should appreciate the costs associated with building and maintaining networks, and there are lots of them to be sure. This is about other aspects of the business.

This looks, to me, like nuttiness... like mutually assured destruction that the cableco folk are driving both parties into intentionally.

No. I don't actually believe that. Businesses are in the habit of making money. There's a reasonably strong desire to remain in business and hopefully make a profit. To that end, in the capitalist model, competition serves to lower prices and increase quality to levels that the average consumer finds acceptable. A monopoly or duopoly environment distorts that; in a market with a constrained number of providers, the conventional capitalistic model can perform poorly or even fail entirely - as an example, consider the LCD price fixing scandal last decade, where prices ended up artificially high. The current situation is worse; the telcos and cablecos have a bunch of incentives to prevent cannibalizing their existing profitable pay TV product lines... which are seeing competition from the likes of Netflix. And there's even some legitimacy there: if all of those customers suddenly dropped their pay TV service and went to Netflix, the whole economic underpinnings of a cable TV company could be thrown into disarray. Because those pay TV subscribers are in some way contributing to covering the opex and capex of the cable TV distribution network. That'd also be damaging to the last mile IP connectivity, heh. But it's hard to have an honest discussion about all of this when those involved are so busy trying to spin things in their favor, and to keep the status quo, etc. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On 5/15/14, 4:16 PM, "Scott Berkman" <scott@sberkman.net<mailto:scott@sberkman.net>> wrote: Everyone knows Comcast uses (or used) Sandvine for shaping (unless they've finished building a new probably internal solution, I'm sure this is another secret we'll only have rumors to work with, ). Comcast turned off Sandvine’s active traffic management system at the end of 2008; I know because it was my job to do it (and I had nothing to do with the decision to turn on the Sandvine system). ;-) FCC Chairman Kevin Martin required the turn down of that system by EOY 2008. Here is the letter to the FCC confirming that transition was completed on January 9, 2009: http://downloads.comcast.net/docs/comcast-nm-transition-notification.pdf. It was replaced with a protocol-agnosting congestion management system (active only in the DOCSIS network). That system was disclosed here: http://downloads.comcast.net/docs/Attachment_B_Future_Practices.pdf http://tools.ietf.org/html/rfc6057 http://downloads.comcast.net/docs/IETF%2072%20-%20TANA%20BoF%20-%20ISP%20Req... http://downloads.comcast.net/docs/Comcast-IETF-P2Pi-20080528.pdf http://downloads.comcast.net/docs/ietf-p2pi-comcast-20080509.pdf There is no other active traffic management system (other than what any ISP has for DDoS protection/mitigation), period. I'm no longer in a position to test this for reporting to the FCC as suggested, but in a previous life we were able to prove it enough for the Comcast customer getting the short end of the stick to stop yelling at us and get a new provider, which of course made everyone involved happier. We used to have a “positive” traffic shaping system called PowerBoost. That enabled customers to boost above their advertised or provisioned rates for brief periods. That system seemed to cause more customer confusion than it was worth and PowerBoost was eliminated across all of our tiers of service. Sometimes tools to notice traffic shaping noticed PowerBoost and it was sometimes hard to explain that we were shaping traffic *up* in capacity rather than down, but I digress. I'd love to see a case study published by Comcast on how that project went and what the impacts to the network and bottom line were. We documented every step of the way on our Network Management page at http://networkmanagement.comcast.net/. You may also be interested to read Alissa Cooper’s September 2013 PhD thesis, which touches on this system on some level at http://www.alissacooper.com/files/Thesis.pdf. There is also a good paper by the Broadband Internet Technical Advisory Group (BITAG) on this topic at http://www.bitag.org/documents/BITAG_-_Congestion_Management_Report.pdf. We at Comcast comply with all of the BITAG recommendations in that paper. Jason

Guys, I'm already pretty far off the reservation and will not respond to trolling. I think most ISPs are starting to avoid participation here for the same reason. I'm going to stop for a while. - Kevin On May 15, 2014, at 12:42 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote: Yes, you've got "some of the largest Internet companies as customers". Because you told them "if you don't pay us, we'll throttle you". Then you throttled them. I'm sorry, not a winning argument. Nick On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com<mailto:Kevin_McElearney@cable.comcast.com>> wrote: Upgrades/buildout are happening every day. They are continuous to keep ahead of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, Ookla, etc What is not well known is that Comcast has been an existing commercial transit business for 15+ years (with over 8000 commercial fiber customers). Comcast also has over 40 balanced peers with plenty of capacity, and some of the largest Internet companies as customers. - Kevin 215-313-1083<tel:215-313-1083>

On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com<mailto:owen@delong.com>> wrote:

Oh, please do explicate on how this is inaccurate…

Owen

...

On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com<mailto:Kevin_McElearney@cable.comcast.com>> wrote:

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083<tel:215-313-1083>

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com<mailto:owen@delong.com>> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>> wrote:

...

...

So they seek new sources of revenues, and/or attempt to thwart

...

competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

...

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

...

...

On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org<mailto:charles@thefnf.org>> wrote:

On 2014-05-14 02:04, Jean-Francois Mezei wrote:

On 14-05-13 22:50, Daniel Staal wrote:

They have the money. They have the ability to get more money. *They see > no reason to spend money making customers happy.* They can make more > profit without it.

There is the issue of control over the market. But also the pressure from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

...

The problem with the internet is that while it had promises of wild growth in the 90s and 00s, once penetration reaches a certain level, growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

...

When you combine this with threath to large incumbents's media and media distribution endeavours by the likes of Netflix (and cat videos on Youtube), large incumbents start thinking about how they will be able to continue to grow revenus/profits when customers will shift spending to vspecialty channels/cableTV to Netflix and customer growth will not compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

...

So they seek new sources of revenues, and/or attempt to thwart competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

...

The current trend is to "if you can't fight them, jon them" where cablecos start to include the Netflix app into their proprietary set-top boxes. The idea is that you at least make the customer continue to use your box and your remote control which makes it easier for them to switch between netflix and legacy TV. True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing

...

to add the Netflix app onto their proprietary STBs also play peering capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

To be fair, I have no evidence that Comcast demanded money in advance. As far as I can tell, Level 3, Cogent and Comcast all agree on the rest though, Comcast's peering filled up. Both Level 3 and Cogent offered/requested to upgrade. Then at least Cogent (IIRC?) offered to upgrade *and pay Comcast to upgrade*. All of these offers were ignored or rejected, and at this point I think all parties agree that ransom was demanded by Comcast. Then, just weeks after Verizon's successful litigation against Net Neutrality, Netflix agreed to pay Comcast directly. I'm not sure if the removal of Netflix is enough to resolve the congestion on the Level 3 and Cogent links, but I'm not sure how one could explain the above situation other than deliberate throttling in an attempt to extort money. Nick On Thu, May 15, 2014 at 12:51 PM, <arvindersingh@mail2tor.com> wrote:

Yes Kevin, this is understood - but valid observation from Nick.

Can you pls answer my question first? Very curious.

Arvinder

...

Guys, I'm already pretty far off the reservation and will not respond to trolling. I think most ISPs are starting to avoid participation here for the same reason. I'm going to stop for a while.

- Kevin

On May 15, 2014, at 12:42 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote:

Yes, you've got "some of the largest Internet companies as customers". Because you told them "if you don't pay us, we'll throttle you". Then you throttled them. I'm sorry, not a winning argument. Nick

On Thu, May 15, 2014 at 10:57 AM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com<mailto: Kevin_McElearney@cable.comcast.com>> wrote: Upgrades/buildout are happening every day. They are continuous to keep ahead of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, Ookla, etc

What is not well known is that Comcast has been an existing commercial transit business for 15+ years (with over 8000 commercial fiber customers). Comcast also has over 40 balanced peers with plenty of capacity, and some of the largest Internet companies as customers.

- Kevin

215-313-1083<tel:215-313-1083>

...

On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com<mailto:owen@delong.com>> wrote:

Oh, please do explicate on how this is inaccurate…

Owen

...

On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com<mailto: Kevin_McElearney@cable.comcast.com>> wrote:

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083<tel:215-313-1083>

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com<mailto:owen@delong.com>> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>> wrote:

...

> > So they seek new sources of revenues, and/or attempt to thwart >> competition any way they can. No to the first. Yes to the second. If they were seeking new sources of > revenue, they'd be massively expanding into un/der served markets > and > aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

>> On Wed, May 14, 2014 at 7:23 AM, >> <charles@thefnf.org<mailto:charles@thefnf.org>> wrote: >> >> On 2014-05-14 02:04, Jean-Francois Mezei wrote: >> >> On 14-05-13 22:50, Daniel Staal wrote: >> >> They have the money. They have the ability to get more money. >> *They see >>> no reason to spend money making customers happy.* They can make >>> more >>> profit without it. >> >> There is the issue of control over the market. But also the >> pressure >> from shareholders for continued growth. > > > Yes. That is true. Except that it's not. > > How do service providers grow? Let's explore that: > > What is growth for a transit provider? > > More (new) access network(s) (connections). > More bandwidth across backbone pipes. > > > What is growth for access network? > More subscribers. > > Except that the incumbent carriers have shown they have no interest > in > providing decent bandwidth to anywhere but the most profitable rate > centers. I'd say about 2/3 of the USA is served with quite terrible > access. > > > > >> The problem with the internet is that while it had promises of wild >> growth in the 90s and 00s, once penetration reaches a certain >> level, >> growth stabilizes. > > Penetration is ABYSMAL sir. Huge swaths of underserved americans > exist. > > > >> When you combine this with threath to large incumbents's media and >> media >> distribution endeavours by the likes of Netflix (and cat videos on >> Youtube), large incumbents start thinking about how they will be >> able to >> continue to grow revenus/profits when customers will shift spending >> to >> vspecialty channels/cableTV to Netflix and customer growth will not >> compensate. > > Except they aren't. Even in the most profitable rate centers, > they've > declined to really invest in the networks. They aren't a real > business. You > have to remember that. They have regulatory capture, natural/defacto > monopoly etc etc. They don't operate in the real world of > risk/reward/profit/loss/uncertainty like any other real business has > to. > > > >> So they seek new sources of revenues, and/or attempt to thwart >> competition any way they can. > > No to the first. Yes to the second. If they were seeking new sources > of > revenue, they'd be massively expanding into un/der served markets > and > aggressively growing over the top services (which are fat margin). > They did > a bit of an advertising campaign of "smart home" offerings, but that > seems > to have never grown beyond a pilot. > > > >> The current trend is to "if you can't fight them, jon them" where >> cablecos start to include the Netflix app into their proprietary >> set-top >> boxes. The idea is that you at least make the customer continue to >> use >> your box and your remote control which makes it easier for them to >> switch between netflix and legacy TV. > True. I don't know why one of the cablecos hasn't licensed roku, > added > cable card and made that available as a "hip/cool" set top box > offering and > charge another 10.00 a month on top of the standard dvr rental. > > > > Would be interesting to see if those cable companies that are > agreeing >> to add the Netflix app onto their proprietary STBs also play >> peering >> capacity games to degrade the service or not. > > So how is the content delivered? Is it over the internet? Or is it > over > the cable plant, from cable headends?

On 5/15/14, 12:43 PM, "Nick B" <nick@pelagiris.org> wrote:

Yes, you've got "some of the largest Internet companies as customers². Because you told them "if you don't pay us, we'll throttle you". Then you throttled them. I'm sorry, not a winning argument. Nick

That is categorically untrue, however nice a soundbite it may be. If you or anyone else truly believes we are throttling someone then I encourage you to file a formal complaint with the FCC. According to their Open Internet rules that we are bound to through at least 2018 (IIRC) we may not discriminate on traffic in that way, so there is a clear rule and a clear process for complaints. Jason

On 5/15/14, 1:28 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote: By "categorically untrue" do you mean "FCC's open internet rules allow us to refuse to upgrade full peers"? Throttling is taking, say, a link from 10G and applying policy to constrain it to 1G, for example. What if a peer wants to go from a balanced relationship to 10,000:1, well outside of the policy binding the relationship? Should we just unquestionably toss out our published policy – which is consistent with other networks – and ignore expectations for other peers? Jason

So by extension, if you enter an agreement and promise to remain balanced you can just willfully throw that out and abuse the heck out of it? Where does it end? Why even bother having peering policies at all then? To use an analogy, if you and I agree to buy a car together and agree to switch off who uses it every other day, can I just say "forget our agreement – I’m just going to drive the car myself every single day – its all mine”? And as you say, “interestingly enough only Comcast and Verizon are having this problem” someone else might say “interestingly enough one content distributor is at the center of all of these issues.” I’m frankly surprised that no one is stepping back to try to understand what was and is driving those changes. Jason On 5/15/14, 1:43 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote: Yes, throttling an entire ISP by refusing to upgrade peering is clearly a way to avoid technically throttling. Interestingly enough only Comcast and Verizon are having this problem, though I'm sure now that you have set an example others will follow. Nick

Jason, like Kevin, thank you very much for opening up to us. It is not every day that someone so close to the issues posts with insight.

From what we see here in India, it is true only Comcast and Verizon are access networks with peering problems. We are able to reach Cox, RCN, Charter, Sonoma Interconnect, other without congestion from AS 6453 "Tata".

Please can you explain what it is about your network design or management that causes the choke? Arvinder

So by extension, if you enter an agreement and promise to remain balanced you can just willfully throw that out and abuse the heck out of it? Where does it end? Why even bother having peering policies at all then?

To use an analogy, if you and I agree to buy a car together and agree to switch off who uses it every other day, can I just say "forget our agreement – I’m just going to drive the car myself every single day – its all mine”?

And as you say, “interestingly enough only Comcast and Verizon are having this problem” someone else might say “interestingly enough one content distributor is at the center of all of these issues.” I’m frankly surprised that no one is stepping back to try to understand what was and is driving those changes.

Jason

On 5/15/14, 1:43 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote:

Yes, throttling an entire ISP by refusing to upgrade peering is clearly a way to avoid technically throttling. Interestingly enough only Comcast and Verizon are having this problem, though I'm sure now that you have set an example others will follow. Nick

On 5/15/14, 3:05 PM, "Joe Greco" <jgreco@ns.sol.net> wrote:

"Hi I'm an Internet company. I don't actually know what the next big thing next year will be but I promise that I won't host it on my network and cause our traffic to become lopsided."

Wow. Is that what you're saying?

Of course not. JL

Their existing agreements notwithstanding, I believe the problem many have with Comcast's balanced ratio requirement is that they have 10s of millions of customers, all or almost all of whom are sold "unbalanced" services. In addition the majority of their customers are end users, who are also going to bias toward heavily inbound patterns (which is one of the reasons for the asymmetric connections in the first place). As primarily an eyeball network with a token (8000 quoted) number of transit customers it does not seem reasonable for them to expect balanced ratios on peering links. They are, effectively by their own choice of market, always going to have a heavily inbound traffic ratio. It seems to me that requiring anything else is basically a way to give the finger to a potential peer while claiming to be neutral. I find it hard to believe that Comcast would be running many balanced links (peering or transit) at all, except perhaps to other consumer ISPs. In today's environment there are inevitably going to be heavily inbound and heavily outbound networks. Content networks don't have any problem with SFI despite their ratio. Eyeball networks do. Both are in the position they are because of the line of business they have respectively chosen. But the eyeball network is the only one that is explicitly and exclusively paid *to carry traffic*. IMO if the content network is willing to bring their content, for free, to the eyeball network's edge, this is to the benefit of the eyeball network more than content, in the absence of other "factors". In this case that factor appears to me to be "ad-hoc oligopoly". If customers had options and an easy path to switch, they would not tolerate this behaviour when they can switch to a competitor who provides good service for the bits they request. Content would gain a lot of leverage in this situation as they could help "educate" customers on alternatives, automatically and without paying a support agent. Of course we should be careful not to let the opposite situation occur either... Keenan ________________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Scott Helms <khelms@zcorum.com> Sent: May 15, 2014 12:54 PM To: Joe Greco Cc: nanog@nanog.org Subject: Re: Observations of an Internet Middleman (Level3) (was: RIP On Thu, May 15, 2014 at 3:05 PM, Joe Greco <jgreco@ns.sol.net> wrote:

...

So by extension, if you enter an agreement and promise to remain balanced y= ou can just willfully throw that out and abuse the heck out of it? Where do= es it end? Why even bother having peering policies at all then?

It doesn't strike you as a ridiculous promise to extract from someone?

You could certainly say its ridiculous, but it is (and has been) the basis for almost all peering arrangements in North America for several decades in my personal experience. I believe that the practice came from the telco world when large telephone companies would exchange traffic without billing each other so long as the traffic was relatively balanced. You can imagine AT&T and Sprint exchange toll traffic and so long as things we're fairly close there wasn't a big imbalance of traffic to worry the financial folks over and thus having to do exact accounting on each minute, which was technically challenging 30 years ago. "Hi I'm an Internet company. I don't actually know what the next big

thing next year will be but I promise that I won't host it on my network and cause our traffic to become lopsided."

Wow. Is that what you're saying?

That's not what happened. What happened is that Netflix went to Level 3 who already had a peering arrangement with Comcast which was built around normal (roughly) balanced peering. It had been in place for years before Netflix signed with Level 3 and worked, and was contracted this way, around relatively balanced traffic. Once Netflix started sending most of their traffic destined to Comcast end user through Level 3 things got out of balance. Netflix still has a contract with Cogent (I believe that is the correct one) or other provider that had previously been handling the bulk of the Comcast directed traffic, but the Level 3 connection was cheaper for Netflix. If anyone actually acted in bad faith it was, IMO, Level 3.

...

To use an analogy, if you and I agree to buy a car together and agree to sw= itch off who uses it every other day, can I just say "forget our agreement = =96 I=92m just going to drive the car myself every single day =96 its all m= ine=94?

Seems like a poor analogy since I'm pretty sure both parties on a peering can use the port at the same time.

His point was you can't simply change a contract without having both parties involved. Level 3 tried to do just that.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

All the talk about ratios is a red herring… The real issue boils down to this: 1. The access (eyeball) networks don’t want to bear the cost of delivering what they promised to their customers. 2. This is because when they built their business models, they didn’t expect their customers to use nearly as much of their promised bandwidth as they are now using. Most of the models were constructed around the idea that a customer receiving, say 27mbps down/7mbps up would use all of that bandwidth in short bursts and mostly use less than a megabit. 3. New services have been developed (streaming video, et al.) which have created an increasing demand from customers for more of the bandwidth they were sold. 4. Instead of raising the prices to the access network customers or accepting that the lavish profits that they eyeball networks had been pocketing were no more, the access networks are trying to slough off the costs of delivering that higher fraction of what they sold onto someone else. 5. The content providers looked like an easy target with the advantage that: A. Some of them appear to have deep pockets. B. They are the competition for many of the access network’s other lines of business, so increasing their costs helps make them less competitive. C. Consumers are emotional about price increases. Content providers look at it as a business problem and perform a mathematical analysis. If their customer satisfaction impact costs more than paying the extortion from the access networks, they’ll pay it. In reality, if the $ACCESS_PROVIDERS wanted to satisfy their customers, they’d be aggressively seeking to peer with content providers in as many locations as possible. They might (reasonably) require content providers to build out to additional locations to keep their long-haul costs down (It’s reasonable, IMHO, for a content provider not to want to carry multiple gigabits of traffic from a content provider clear across the country for free. If $CONTENT_PROVIDER wants to access California customers of $ACCESS_PROVIDER, then it’s reasonable for $ACCESS_PROVIDER to insist that $CONTENT_PROVIDER peer in California for delivering those bits.) Neither side of this issue has completely clean hands. Both have been trying to take as much of the money on the table for themselves with limited regard for serving the consumer. The Access Networks have done a far worse job of serving the consumer than the content providers and that’s a big part of what is driving the current backlash. As a general rule, access customers don’t select the provider they love the most, they select the one they think sucks the least. I think the recent FCC NPRM is a bit optimistic in that it expects the $ACCESS_PROVIDERS to act in good faith. If they do, it will likely turn out to be a limited victory for the $ACCESS_PROVIDERS. However, I don’t expect the $ACCESS_PROVIDERS to live within that limited victory. Assuming the NRPM becomes rule and then withstands the likely legal challenges, I expect they will, as usual, play in the gray areas of the ruling as much as they think they legally can and push the edges as far as possible to try and extort every dollar they can from $CONTENT_PROVIDERS with this so-called fast-lane (which we all know is just preferential peering and/or QoS[1] tuning). I suspect they will likely push this far enough that over the next several years, things will get progressively worse until the FCC finally decides that they have to move from section 706 to Title II. OTOH, if I’m wrong and the $ACCESS_PROVIDERS suddenly start behaving like civilized companies, develop a sudden concern for their customers’ experiences, and start unimaginably acting in good faith, the proposed rule wouldn’t be so bad for $CONTENT_PROVIDERS, $CONSUMERS, or $ACCESS_PROVIDERS. Of course, you can already see the $ACCESS_PROVIDERS laying the groundwork to try and mount a legal challenge against the FCC’s authority to use rule 706. Sadly, some of this groundwork is being laid by FCC commissioners. Said commissioners clearly have no interest in representing the people’s interest and are strictly there as mouth-pieces for some of the big players in the industry. Owen [1] QoS — A deceptive name if ever there was one. QoS is not about Quality of Service, it’s about screwing over network users by choice rather than by chance when you haven’t built an adequate network.

Social media is not a big driver of symmetrical traffic here in the US or internationally. Broadband suffers here for a number of reasons, mainly topological and population density, in comparison to places like Japan, parts (but certainly not all) of Europe, and South Korea. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 11:02 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:

On Friday, May 16, 2014 03:54:33 PM Owen DeLong wrote:

...

customers. 2. This is because when they built their business models, they didn’t expect their customers to use nearly as much of their promised bandwidth as they are now using. Most of the models were constructed around the idea that a customer receiving, say 27mbps down/7mbps up would use all of that bandwidth in short bursts and mostly use less than a megabit.

And in general, models have assumed, for a long time, that customer demand patterns are largely asymmetric.

While that is true a lot of the time (especially for eyeball networks), it is less so now due to social media. Social media forces the use of symmetric bandwidth (like FTTH), putting even more demand on the network, and making the gist of this thread an even bigger issue, if you discount the fact, of course, that Broadband in the U.S. currently sucks for a developed market.

Mark.

Mark, Bandwidth use trends are actually increasingly asymmetical because of the popularity of OTT video. Social media, even with video uploading, simply doesn't generate that much traffic per session. "During peak period, Real-Time Entertainment traffic is by far the most dominant traffic category, accounting for almost half of the downstream bytes on the network. As observed in past reports, Social Networking applications continue to be very well represented on the mobile network. This speaks to their popularity with subscribers as these applications typically generate far less traffic than those that stream audio and video." https://www.sandvine.com/downloads/general/global-internet-phenomena/2013/sa... Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 11:26 AM, Mark Tinka <mark.tinka@seacom.mu> wrote:

On Friday, May 16, 2014 05:08:33 PM Scott Helms wrote:

...

Social media is not a big driver of symmetrical traffic here in the US or internationally. Broadband suffers here for a number of reasons, mainly topological and population density, in comparison to places like Japan, parts (but certainly not all) of Europe, and South Korea.

It might not be (now), but if symmetrical bandwidth will go in on the back of teenagers wanting to upload videos about their lives, the meer fact that the bandwidth is there means someone will find bigger and better use for it, than social media.

We saw this when we deployed FTTH in Malaysia, back in '09.

Mark.

Michael, No, its not too much to ask and any end user who has that kind of requirement can order a business service to get symmetrical service but the reality is that symmetrical service costs more and the vast majority of customers don't use the upstream capacity they have today. I have personal insight into about half a million devices and the percentage of people who bump up against their upstream rate is less than 0.2%. I have the ability to get data on another 10 million and the last time I checked their rates were similar. This kind of question has been asked of operators since long before cable companies could offer internet service. What happens if everyone in an area use their telephone (cellular or land line) at the same time? A fast busy or recorded "All circuits are busy message." Over subscription is a fact of economics in virtually everything we do. By this logic restaurants should be massively over built so that there is never a waiting line, highways should always be a speed limit ride, and all of these things would cost much more money than they do today. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Sun, Apr 27, 2014 at 8:21 PM, Michael Thomas <mike@mtcc.com> wrote:

Scott Helms wrote:

...

Mark,

Bandwidth use trends are actually increasingly asymmetical because of the popularity of OTT video.

Until my other half decides to upload a video.

Is it too much to ask for a bucket of bits that I can use in whichever direction happens to be needed at the moment?

Mike

Lazlo, You're correct that some applications are being restricted, but AFAIK in North America they are all being restricted for quite valid network management reasons. While back in the day I ran Sendmail and sometimes qmail on my home connection I was also responsible with my mail server and more importantly the world was different. The threat from an open relay or mail server with a compromise is much higher, in part because the speeds are higher, but also because the attackers are more sophisticated and the hardware the mail server is running on is much more powerful. P2P is _not_ being blocked legally anywhere and if you believe that it is then you should complain to the FCC in the US or the CRTC in Canada. Running a DNS or NTP server that's open to the Internet on a home connection should NOT be allowed. I'm sorry if you're one of the few people who can run those services effectively and safely (just like SMTP) but the vast majority of customers can't and in most cases they aren't running them intentionally. I won't get into marketing, that's not what I do and I agree that unlimited seems to mean something other than the way I understand it but that's no different from unlimited telephone service, all you can eat buffets, or just about anywhere else you can see the word "unlimited" or all in marketing. I'd also like to see much more competition in the market and that's one the things I work to accomplish. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 2:38 PM, Laszlo Hanyecz <laszlo@heliacal.net> wrote:

I'd just like to point out that a lot of people are in fact using their upstream capability, and the operators always throw a fit and try to cut off specific applications to force it back into the idle state. For example P2P things like torrents and most recently the open NTP and DNS servers. How about SMTP? Not sure about you guys but my local broadband ISP has cut me off and told me that my 'unlimited internet' is in fact limited. The reality is that those people who are not using it (99.8%?) are just being ripped off - paying for something they were told they need, thinking that it's there when they want it, then getting cut off when they actually try to use it.

It's not like whining about it here will change anything, but the prices are severely distorted. Triple play packages are designed to force people to pay for stuff they don't need or want - distorting the price of a service hoping to recover it elsewhere, then if the gamble doesn't pan out, the customer loses again. The whole model is based on people buying stuff that they won't actually come to collect, so then you can sell it an infinite number of times. The people who do try to collect what was sold to them literally end up getting called names and cut off - terms like "excessive bandwidth user" and "network abuser" are used to describe paying customers. With regard to the peering disputes, it's hardly surprising that their business partners are treated with the same attitude as their customers. Besides, if you cut off the customers and peers who are causing that saturation, then the existing peering links can support an infinite number of idle subscribers. The next phase is usage-based-billing which is kind of like having to pay a fine for using it, so they can artificially push the price point lower and hopefully get some more idle customers. That will help get the demand down and keep the infrastructure nice and idle. When you're paying for every cat video maybe you realize you can live without it instead.

Everyone has been trained so well, they don't even flinch anymore when they hear about "over subscription", and they apologize for the people who are doing it to them. The restaurant analogy is incorrect - you can go to the restaurant next door if a place is busy, thus they have pressure to increase their capacity if they want to sell more meals. With broadband you can't go anywhere else, (for most people) there's only one restaurant, and there's a week long waiting list. If you don't like it, you're probably an abuser or excessive eater anyway.

-Laszlo

On May 16, 2014, at 5:34 PM, Scott Helms <khelms@zcorum.com> wrote:

...

Michael,

No, its not too much to ask and any end user who has that kind of requirement can order a business service to get symmetrical service but the reality is that symmetrical service costs more and the vast majority of customers don't use the upstream capacity they have today. I have personal insight into about half a million devices and the percentage of people who bump up against their upstream rate is less than 0.2%. I have the ability to get data on another 10 million and the last time I checked their rates were similar.

This kind of question has been asked of operators since long before cable companies could offer internet service. What happens if everyone in an area use their telephone (cellular or land line) at the same time? A fast busy or recorded "All circuits are busy message." Over subscription is a fact of economics in virtually everything we do. By this logic restaurants should be massively over built so that there is never a waiting line, highways should always be a speed limit ride, and all of these things would cost much more money than they do today.

Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Sun, Apr 27, 2014 at 8:21 PM, Michael Thomas <mike@mtcc.com> wrote:

...

Scott Helms wrote:

...

Mark,

Bandwidth use trends are actually increasingly asymmetical because of the popularity of OTT video.

Until my other half decides to upload a video.

Is it too much to ask for a bucket of bits that I can use in whichever direction happens to be needed at the moment?

Mike

Mike, In my experience you're not alone, just in a really tiny group. As I said I have direct eyeballs on ~500k devices and the ability to see another 10 million anytime I want and the percentage of people who cap their upstream in both of those sample groups for more than 15 minutes (over the last 3 years) is about 0.2%. Interestingly if a customer does it once they have about a 70% chance of doing it regularly. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 3:46 PM, Michael Thomas <mike@mtcc.com> wrote:

Scott Helms wrote:

...

Michael,

No, its not too much to ask and any end user who has that kind of requirement can order a business service to get symmetrical service but the reality is that symmetrical service costs more and the vast majority of customers don't use the upstream capacity they have today. I have personal insight into about half a million devices and the percentage of people who bump up against their upstream rate is less than 0.2%. I have the ability to get data on another 10 million and the last time I checked their rates were similar.

I've just been on the losing end of yet another piece of why crappy upstream bandwidth sucks: Mavericks seems to have decided that my other half's imovie library really, really ought to be uploaded to iCloud (without asking, ftw).

I can and should be pissed at Apple for doing such a wrongheaded thing, but the fact is that my upstream bandwidth was saturated for hours and days and it was extremely difficult to figure out why. I doubt I'm alone.

Better upstream bandwidth would have at least made the pain period shorter.

Mike

I think you will, all of those things have been around for a long time (well, except for pervasive video calls, which I think is vapor) and none generate the kind of traffic it takes to congest a decent link. Most of the DOCSIS systems I've worked with are running at least 6 mbps upstreams and many are well into the double digits. My current connection (tested this morning) is about 22 mbps. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 4:06 PM, Michael Thomas <mike@mtcc.com> wrote:

Scott Helms wrote:

...

Mike,

In my experience you're not alone, just in a really tiny group. As I said I have direct eyeballs on ~500k devices and the ability to see another 10 million anytime I want and the percentage of people who cap their upstream in both of those sample groups for more than 15 minutes (over the last 3 years) is about 0.2%. Interestingly if a customer does it once they have about a 70% chance of doing it regularly.

Well, given Sling, Dropbox, iCloud, pervasive video calls (you have heard about webrtc, yes? 24/7 babycams!), youtube, etc, etc, I won't be a "tiny group" for long.

Mike

Michael, I didn't claim Webrtc is vapor, I claim that pervasive video calling is vapor. Further, even if that prediction is wrong pervasive video calling isn't enough even if 100% of users adopt it to swing the need for symmetrical bandwidth. An average Skype/Google Hangout/Apple is less than 400 kbps at peak and averages something like 150 kbps. http://www.digitalsociety.org/2010/08/iphone-facetime-bandwidth-gets-measure... Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 4:22 PM, Michael Thomas <mike@mtcc.com> wrote:

Scott Helms wrote:

...

I think you will, all of those things have been around for a long time (well, except for pervasive video calls, which I think is vapor) and none generate the kind of traffic it takes to congest a decent link. Most of the DOCSIS systems I've worked with are running at least 6 mbps upstreams and many are well into the double digits. My current connection (tested this morning) is about 22 mbps.

Um, no it's not vapor. Webrtc is quite real, and the barrier to implementation for any random web site is weeks, not years as was the case before.

I just saw this that you wrote:

...

1) Very few consumers are walking around with a HD or 4K camera

today.

In the US, we just surpassed 1/2 of the population who have that capability, iirc. They call them phones nowadays.

Mike

...

Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Fri, May 16, 2014 at 4:06 PM, Michael Thomas <mike@mtcc.com <mailto: mike@mtcc.com>> wrote:

Scott Helms wrote:

Mike,

In my experience you're not alone, just in a really tiny group. As I said I have direct eyeballs on ~500k devices and the ability to see another 10 million anytime I want and the percentage of people who cap their upstream in both of those sample groups for more than 15 minutes (over the last 3 years) is about 0.2%. Interestingly if a customer does it once they have about a 70% chance of doing it regularly.

Well, given Sling, Dropbox, iCloud, pervasive video calls (you have heard about webrtc, yes? 24/7 babycams!), youtube, etc, etc, I won't be a "tiny group" for long.

Mike

On May 16, 2014, at 1:06 PM, Michael Thomas <mike@mtcc.com> wrote:

Scott Helms wrote:

...

Mike, In my experience you're not alone, just in a really tiny group. As I said I have direct eyeballs on ~500k devices and the ability to see another 10 million anytime I want and the percentage of people who cap their upstream in both of those sample groups for more than 15 minutes (over the last 3 years) is about 0.2%. Interestingly if a customer does it once they have about a 70% chance of doing it regularly.

Well, given Sling, Dropbox, iCloud, pervasive video calls (you have heard about webrtc, yes? 24/7 babycams!), youtube, etc, etc, I won't be a "tiny group" for long.

Mike

Yes… Scott is making what I consider a classic mistake. Attempting to define the future in terms of the limitations that users have adapted to from the past. Eventually, users do realize that the limitations are no longer necessary and then they won’t accept them. Unfortunately, this takes far longer than is desirable. Fortunately, this gives proactive and innovative service providers the opportunity to adapt the technology and remove those limitations before the users care. Unfortunately, it also gives other service providers the opportunity to try and hold users back even after they care. IIRC, there are awards in both categories. Owen

On Friday, May 16, 2014 05:45:06 PM Scott Helms wrote:

Bandwidth use trends are actually increasingly asymmetical because of the popularity of OTT video.

Social media, even with video uploading, simply doesn't generate that much traffic per session.

Our experience showed that there is a direct co-relation between the lack of traffic in the upstream direction and poor upload bandwidth (primarily, due to asymmetric tech. such as ADSL), e.g., because of the ADSL I have at home (512Kbps up, 4Mbps down), I generally do not send very large e-mails when working from home; nor do I use my laptop for remote router/switch updates as the software images are a nightmare to upload. And yes, there is a larger proportion of downstream traffic than there is upstream traffic pretty much most of the time (even with symmetric links). However, with symmetry, upstream traffic will increase significantly as customers realize it is now available. One of the use-cases we thought about when deploying an FTTH backbone was having remote PVR's. So rather than record and save linear Tv programming on the STB, record and save it in the network. This could only be done with symmetric bandwidth. Mark.

----- Original Message -----

From: "Mark Tinka" <mark.tinka@seacom.mu>

While that is true a lot of the time (especially for eyeball networks), it is less so now due to social media. Social media forces the use of symmetric bandwidth (like FTTH), putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3. :-) Could you expand a bit, Mark on "Social media forces the use of symmetric bandwidth"? Which social media platform is it that you think has a) symmetrical flows that b) are big enough to figure into transit symmetry? Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

Blake, None of those applications come close to causing symmetrical traffic patterns and for many/most networks the upstream connectivity has greatly improved. Anything related to voice is no more than 80 kbps per line, even if the SIP traffic isn't trunked (less if it is because the signaling data is shared). Document sharing is not being impinged, on my residential account right now I've uploaded about 30 documents this morning including large PDFs and Power Point presentations. Off site back up is one use that could drive traffic, but I don't believe that the limiting factor is bandwidth. We looked at getting into that business and from what we saw the limiting factor was that most residential and SOHO accounts didn't want to pay enough to cover your storage & management costs. In our analysis the impact of bandwidth on the consumer side adoption was basically zero. There is no expectation that back ups run instantly. Having said all of that, even if hosted back up became wildly popular would not change the balance of power because OTT video is both larger, especially for HD streams, and used much more frequently. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 11:53 AM, Blake Hudson <blake@ispn.net> wrote:

Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

...

----- Original Message -----

...

From: "Mark Tinka" <mark.tinka@seacom.mu> While that is true a lot of the time (especially for eyeball networks), it is less so now due to social media. Social media forces the use of symmetric bandwidth (like FTTH), putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on "Social media forces the use of symmetric bandwidth"? Which social media platform is it that you think has a) symmetrical flows that b) are big enough to figure into transit symmetry?

Cheers, -- jra

Applications like Skype and Facetime (especially conference calls) would be one example where an application benefits from symmetric (or asymmetric in favor of higher upload speed) connectivity. Cloud office applications like storage of documents, email, and IVR telephony also benefit from symmetrical connectivity. Off-site backup software is another great example. Most residential connections are ill suited for this. I believe these applications (and derivatives) would be more popular today if the connectivity was available.

--Blake

Blake, I might agree with your premise if weren't for a couple of items. 1) Very few consumers are walking around with a HD or 4K camera today. 2) Most consumers who want to share video wouldn't know how to host it themselves, which isn't an insurmountable issue but is a big barrier to entry especially given the number of NAT'ed connections. I think this is much more of a problem than available bandwidth. 3) Most consumers who want to share videos seem to be satisfied with sharing via one of the cloud services whether that be YouTube (which was created originally for that use), Vimeo, or one of the other legions of services like DropBox. 4) Finally, upstream bandwidth has increased on many/most operators. I just ran the FCC's speedtest (mLab not Ookla) and got 22 mbps on my residential cable internet service. I subscribe to one of the major MSOs for a normal residential package. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 12:38 PM, Blake Hudson <blake@ispn.net> wrote:

Certainly video is one of the most bandwidth intensive applications. I don't deny that a < 1 Mbps video call is both less common and consumes less bandwidth than an 8Mbps HD stream. However, if Americans had access to symmetric connections capable of reliably making HD video calls (they don't, in my experience), we might be seeing video calls as a common occurrence and not a novelty. I think the state of usage is a reflection on the technology available.

If the capability was available at an affordable price to residential consumers, we might see those consumers stream movies or send videos from their home or mobile devices via their internet connection directly to the recipient rather than through a centralized source like Disney, NetFlix, Youtube, etc. Video sharing sites (like youtube, vimeo, etc) primary reason for existence is due to the inability of the site's users to distribute content themselves. One of the hurdles to overcome in video sharing is the lack of availability in affordable internet connectivity that is capable of sending video at reasonable (greater than real time) speeds.

--Blake

Scott Helms wrote the following on 5/16/2014 11:02 AM:

...

Blake,

None of those applications come close to causing symmetrical traffic patterns and for many/most networks the upstream connectivity has greatly improved. Anything related to voice is no more than 80 kbps per line, even if the SIP traffic isn't trunked (less if it is because the signaling data is shared). Document sharing is not being impinged, on my residential account right now I've uploaded about 30 documents this morning including large PDFs and Power Point presentations.

Off site back up is one use that could drive traffic, but I don't believe that the limiting factor is bandwidth. We looked at getting into that business and from what we saw the limiting factor was that most residential and SOHO accounts didn't want to pay enough to cover your storage & management costs. In our analysis the impact of bandwidth on the consumer side adoption was basically zero. There is no expectation that back ups run instantly. Having said all of that, even if hosted back up became wildly popular would not change the balance of power because OTT video is both larger, especially for HD streams, and used much more frequently.

Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Fri, May 16, 2014 at 11:53 AM, Blake Hudson <blake@ispn.net <mailto: blake@ispn.net>> wrote:

Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

----- Original Message -----

From: "Mark Tinka" <mark.tinka@seacom.mu <mailto:mark.tinka@seacom.mu>> While that is true a lot of the time (especially for eyeball networks), it is less so now due to social media. Social media forces the use of symmetric bandwidth (like FTTH), putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on "Social media forces the use of symmetric bandwidth"? Which social media platform is it that you think has a) symmetrical flows that b) are big enough to figure into transit symmetry?

Cheers, -- jra

Applications like Skype and Facetime (especially conference calls) would be one example where an application benefits from symmetric (or asymmetric in favor of higher upload speed) connectivity. Cloud office applications like storage of documents, email, and IVR telephony also benefit from symmetrical connectivity. Off-site backup software is another great example. Most residential connections are ill suited for this. I believe these applications (and derivatives) would be more popular today if the connectivity was available.

--Blake

Blake, You're absolutely correct. The world adapts to the reality that we find ourselves in via normal market mechanics. The problem with proposing that connectivity for residential customers should be more symmetrical is that its expensive, which is why we as operators didn't roll it out that way to start. We also don't see consumer demand for symmetrical connections and with the decline in peer to peer file sharing we've actually seen a decrease the ratio of used upstream bandwidth (though not a decrease in absolute terms). I would like to deliver symmetrical bandwidth to all consumers just so those few customers who need it today would have lower bills but trying to justify that to our CFO without being able to point to an increase in revenue either because of more revenue per sub or more subs is a very tough task. I don't believe my situation is uncommon. Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 1:20 PM, Blake Hudson <blake@ispn.net> wrote:

Thanks for the insight Scott. I appreciate the experience and point of view you're adding to this discussion (not just the responses to me). While I might be playing the devil's advocate here a bit, I think one could argue each of the points you've made below.

I do feel that general usage patterns are a reflection of the technologies that have traditionally been available to consumers. New uses and applications would be available to overcome hurdles if the technologies had developed to be symmetrical. I'm not saying that the asymmetrical choice was a bad one, but it was not without consequences. If residential ISPs sell asymmetric connections for decades, how can the ISP expect that application developers would not take this into account when developing applications? I don't think my application would be very successful if it required X Mbps and half of my market did not meet this requirement. Of course content/service providers are going to tailor their services based around their market.

--Blake

Scott Helms wrote the following on 5/16/2014 12:06 PM:

...

Blake,

I might agree with your premise if weren't for a couple of items.

1) Very few consumers are walking around with a HD or 4K camera today.

2) Most consumers who want to share video wouldn't know how to host it themselves, which isn't an insurmountable issue but is a big barrier to entry especially given the number of NAT'ed connections. I think this is much more of a problem than available bandwidth.

3) Most consumers who want to share videos seem to be satisfied with sharing via one of the cloud services whether that be YouTube (which was created originally for that use), Vimeo, or one of the other legions of services like DropBox.

4) Finally, upstream bandwidth has increased on many/most operators. I just ran the FCC's speedtest (mLab not Ookla) and got 22 mbps on my residential cable internet service. I subscribe to one of the major MSOs for a normal residential package.

Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Fri, May 16, 2014 at 12:38 PM, Blake Hudson <blake@ispn.net <mailto: blake@ispn.net>> wrote:

Certainly video is one of the most bandwidth intensive applications. I don't deny that a < 1 Mbps video call is both less common and consumes less bandwidth than an 8Mbps HD stream. However, if Americans had access to symmetric connections capable of reliably making HD video calls (they don't, in my experience), we might be seeing video calls as a common occurrence and not a novelty. I think the state of usage is a reflection on the technology available.

If the capability was available at an affordable price to residential consumers, we might see those consumers stream movies or send videos from their home or mobile devices via their internet connection directly to the recipient rather than through a centralized source like Disney, NetFlix, Youtube, etc. Video sharing sites (like youtube, vimeo, etc) primary reason for existence is due to the inability of the site's users to distribute content themselves. One of the hurdles to overcome in video sharing is the lack of availability in affordable internet connectivity that is capable of sending video at reasonable (greater than real time) speeds.

--Blake

Scott Helms wrote the following on 5/16/2014 11:02 AM:

Blake,

None of those applications come close to causing symmetrical traffic patterns and for many/most networks the upstream connectivity has greatly improved. Anything related to voice is no more than 80 kbps per line, even if the SIP traffic isn't trunked (less if it is because the signaling data is shared). Document sharing is not being impinged, on my residential account right now I've uploaded about 30 documents this morning including large PDFs and Power Point presentations.

Off site back up is one use that could drive traffic, but I don't believe that the limiting factor is bandwidth. We looked at getting into that business and from what we saw the limiting factor was that most residential and SOHO accounts didn't want to pay enough to cover your storage & management costs. In our analysis the impact of bandwidth on the consumer side adoption was basically zero. There is no expectation that back ups run instantly. Having said all of that, even if hosted back up became wildly popular would not change the balance of power because OTT video is both larger, especially for HD streams, and used much more frequently.

Scott Helms Vice President of Technology ZCorum (678) 507-5000 <tel:%28678%29%20507-5000> -------------------------------- http://twitter.com/kscotthelms --------------------------------

On Fri, May 16, 2014 at 11:53 AM, Blake Hudson <blake@ispn.net <mailto:blake@ispn.net> <mailto:blake@ispn.net <mailto:blake@ispn.net>>> wrote:

Jay Ashworth wrote the following on 5/16/2014 10:35 AM:

----- Original Message -----

From: "Mark Tinka" <mark.tinka@seacom.mu <mailto:mark.tinka@seacom.mu> <mailto:mark.tinka@seacom.mu <mailto:mark.tinka@seacom.mu>>> While that is true a lot of the time (especially for eyeball networks), it is less so now due to social media. Social media forces the use of symmetric bandwidth (like FTTH), putting even more demand on the network,

Oh yes; clearly, Twitter will be the end of L3.

:-)

Could you expand a bit, Mark on "Social media forces the use of symmetric bandwidth"? Which social media platform is it that you think has a) symmetrical flows that b) are big enough to figure into transit symmetry?

Cheers, -- jra

Applications like Skype and Facetime (especially conference calls) would be one example where an application benefits from symmetric (or asymmetric in favor of higher upload speed) connectivity. Cloud office applications like storage of documents, email, and IVR telephony also benefit from symmetrical connectivity. Off-site backup software is another great example. Most residential connections are ill suited for this. I believe these applications (and derivatives) would be more popular today if the connectivity was available.

--Blake

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote:

in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect peering ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question. Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

On Fri, May 16, 2014 at 3:11 PM, Blake Hudson <blake@ispn.net> wrote:

Christopher Morrow wrote the following on 5/16/2014 1:52 PM:

...

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote:

...

in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect peering ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

I agree about the term being passe ...and that it never applied to ISPs ...and that peering is about cost reduction, reliability, and performance.

ok.

It seems to me that many CDNs or content providers want to setup peering relationships and are willing to do so at a cost to them in order to bypass "the internet middle men". But I mention traffic ratios because some folks

'the internet middle men' - is really, it seems to me, 'people I have no business relationship with'. There's also no way to control the capacity planning process with these middle-men, right? Some AS in the middle of my 3-AS-way conversation isn't someone I can capacity plan with :( -chris

in this discussion seem to be using it as justification for not peering. But hey, why peer at little or no cost if they can instead hold out and possibly peer at a negative cost?

--Blake

On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow < morrowc.lists@gmail.com> wrote:

...

in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote: peering

...

ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

Traffic asymmetry across peering connections was what lit the fuse on this whole powder keg, if I understand correctly; at the point the traffic went asymmetric, the refusals to augment capacity kicked in, and congestion became a problem. I've seen the same thing; pretty much every rejection is based on ratio issues, even when offering to cold-potato haul the traffic to the home market for the users. If the refusals hinged on any other clause of the peering requirements, you'd be right; but at the moment, that's the flag networks are waving around as their speedbump-du-jour. So, it may be very "1990", but unfortunately that seems to be the year many people in the industry are mentally stuck in. :( Matt

On Fri, May 16, 2014 at 3:15 PM, Matthew Petach <mpetach@netflight.com> wrote:

On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

...

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote:

...

in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect peering ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

Traffic asymmetry across peering connections was what lit the fuse on this whole powder keg, if I understand correctly; at the point the traffic went asymmetric, the refusals to augment capacity kicked in, and congestion became a problem.

Is it that? or is it that planning at some ISP pair had a '6 months to upgrade' regularly penciled in, then 'all of a sudden' their links were filling up faster than every 6months and... now they are 1x or 2x upgrade cycles behind? I imagine that up to a point upgrading a router that does only 'peering' (SFP) is 'easy', but at some step function of upgrades on the edge ports you need to provision more backhaul and more core and probably upgrade the link types and the chassis and ... At some ISPs this process involves more than 1 dude/group. So coordination and budget issues and scheduling ... become a bit harder. Adjusting to the new reality of 'you need to plan for pipe filling more often, increase upgrade cycle crank speed!' seems like at least one problem, to me at least. It's really hard to tell what's upsetting people about this whole topic :( There's a mix of 'my access link blows' to 'isps should peer better and for freer' and a bunch of other stuff all mixed in the middle :(

I've seen the same thing; pretty much every rejection is based on ratio issues, even when offering to cold-potato haul the traffic to the home market for the users.

yes, welp... it's often rough to get folk who want to think in terms of apples to suddenly thing in terms of the new best fruit 'acai berry'. Especially at large and entrenched organizations.

If the refusals hinged on any other clause of the peering requirements, you'd be right; but at the moment, that's the flag networks are waving around as their speedbump-du-jour.

sure, it's also super easy for them to do this, see entrenched org comment above. it seems to me that the point of peering is not <stalin voice>'ratio or bust'</stalin voice> but 'mutual benefit'. If a skewed ratio of 100:1 in a local market still is cheaper than 'backhaul that traffic from LHR to SFO' there's mutual benefit and a reason to peer. I understand that this is a bit of a rosy landscape I'm painting, but...

So, it may be very "1990", but unfortunately that seems to be the year many people in the industry are mentally stuck in. :(

oh, entrenched. I see. thanks! -chris

Traffic Symmetry is a distraction that the $ACCESS_PROVIDERS would like us to focus on. The reality is that $ACCESS_PROVIDERS want us to focus on that so that we don’t see what is really going on which is a battle to deeper (or avoid increasing peering capacity with) networks they think they can force to pay them more money. This is an age old tactic and it isn’t unique to $ACCESS_PROVIDERS. The larger $BACKBONE_PROVIDERS did it in the past, too. The first one was a railroad company turned telecom. Then came the remnants of PSI. Today, it’s the largest $ACCESS_PROVIDERS. Usually, this just results in harm to both sides and eventually a loss of subscribers. The $ACCESS_PROVIDERS have an advantage in the latter as they mostly avoid loss of subscribers through the fact that the subscribers don’t have anywhere else that they can usefully go. Owen On May 16, 2014, at 12:15 PM, Matthew Petach <mpetach@netflight.com> wrote:

On Fri, May 16, 2014 at 11:52 AM, Christopher Morrow < morrowc.lists@gmail.com> wrote:

...

...

in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect

On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote: peering

...

ratios to be symmetric

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

Traffic asymmetry across peering connections was what lit the fuse on this whole powder keg, if I understand correctly; at the point the traffic went asymmetric, the refusals to augment capacity kicked in, and congestion became a problem.

I've seen the same thing; pretty much every rejection is based on ratio issues, even when offering to cold-potato haul the traffic to the home market for the users.

If the refusals hinged on any other clause of the peering requirements, you'd be right; but at the moment, that's the flag networks are waving around as their speedbump-du-jour. So, it may be very "1990", but unfortunately that seems to be the year many people in the industry are mentally stuck in. :(

Matt

On Friday, May 16, 2014 08:52:31 PM Christopher Morrow wrote:

is 'symmetric traffic ratios' even relevant though? Peering is about offsetting costs, right? it might not be important that the ratio be 1:1 or 2:1... or even 10:1, if it's going to cost you 20x to get the traffic over longer/transit/etc paths... or if you have to build into some horrific location(s) to access the content in question.

Harping on symmetric ratios seems very 1990... and not particularly germaine to the conversation at hand.

Agree. We don't have a ratio requirement, for example. We have a "if it makes sense" requirement. I'm forced to peer with certain African providers in London and Amsterdam because they don't want to peer in Africa, where we are literally are an x-connect away from each other. And the reasons are not even because either of us is larger or smaller than the other... it's just legacy thinking and we're the new guy that has grown rapidly. Now we both have to pay for traffic to get sent to Europe and back. How nice... Mark.

I'm forced to peer with certain African providers in London and Amsterdam because they don't want to peer in Africa, where we are literally are an x-connect away from each other. And the reasons are not even because either of us is larger or smaller than the other... it's just legacy thinking and we're the new guy that has grown rapidly.

Now we both have to pay for traffic to get sent to Europe and back. How nice...

which is amusing given you have massive east coast to europe fiber capacity. randy

Blake, I'm not sure what the relationship between what an access network sells has to do with how their peering is done. I realize that everyone's favorite target is Comcast right now, but would anyone bat an eye over AT&T making the same requirement since they have much more in the way of transit traffic? I don't think anyone forced Level 3 into their peering agreement with Comcast and it was (roughly) symmetrical for years before Level 3 was contracted by Netflix. Shouldn't Level 3 gone to Comcast and told them they needed to change their peering or get a different contract? Why was Cogent able to maintain (roughly) symmetrical traffic with Comcast when they were the primary path for Netflix to Comcast users? Scott Helms Vice President of Technology ZCorum (678) 507-5000 -------------------------------- http://twitter.com/kscotthelms -------------------------------- On Fri, May 16, 2014 at 2:47 PM, Blake Hudson <blake@ispn.net> wrote:

Oh, I'm not proposing symmetrical connectivity at all. I'm just supporting the argument that in the context of this discussion I think it's silly for a residential ISP to purport themselves to be a neutral carrier of traffic and expect peering ratios to be symmetric when the overwhelming majority of what they're selling (and have been selling for over a decade) is asymmetric connectivity. Their traffic imbalance is, arguably, their own doing.

How residential ISPs recoup costs (or simply increase revenue/profit) is another question entirely. I think the most insightful comment in this discussion was made by Mr. Rick Astley (I assume a pseudonym), when he states that ISPs have several options to increase revenue A) Increase price of their product, B) Implement usage restrictions, or C) Charge someone else/Make someone else your customer. I think he successfully argues that option C may be the best. As we've seen, the wireless market in the US went for option B. We've yet to see where the wireline market will go.

Of course, the market would ideally keep ISPs' demands for revenue/profit in check and we'd all reach a satisfactory solution. One of the arguments, one I happen to support, in this thread is that there is not a free market for internet connectivity in many parts of the US. If there was, I believe Comcast would be focusing on how to provide a balance between the best product at the lowest cost and not on how they can monetize their paying customers in order to increase profits. I appreciate honesty; When a service provider advertises X Mbps Internet speeds, I expect they can deliver on their claims (to the whole Internet, and not just the portions of it they've decided). I understand congestion, overselling, etc. But choosing which portions of the internet work well and which don't is a lot more like censorship than service.

--Blake

On Fri, May 16, 2014 at 12:14 PM, James R Cutler < james.cutler@consultant.com> wrote:

All this talk about symmetry and asymmetry is interesting.

Has anyone actually quantified how much congestion is due to buffer bloat which is, in turn, exacerbated by asymmetric connections?

James R. Cutler James.cutler@consultant.com PGP keys at http://pgp.mit.edu

I think you might have the cart before the horse. If there's no congestion on a peering link, buffering doesn't come into play, at least not within the transport infrastructure. We're not talking congestion on the last mile side, we're looking at congestion on the interconnect links between networks, typically 10G or 100G ports. Unless you're running those links near or at capacity, buffering should be a complete non-issue. And if you're running those links at capacity, then the congestion is due to too much traffic, period, not to the size of buffers involved on either side of the link. ^_^; Thanks! Matt

Duh. On 5/16/14, 1:54 PM, "Matt Palmer" <mpalmer@hezmatt.org> wrote:

On Fri, May 16, 2014 at 01:47:53PM -0500, Blake Hudson wrote:

...

Mr. Rick Astley (I assume a pseudonym)

Why would you assume that? Mr. Astley has long been a champion of solid network engineering, and even net neutrality... he's long said that he's "Never gonna drop a route, never gonna fill a link, never gonna turn around and congest you."

- Matt

On Friday, May 16, 2014 08:47:53 PM Blake Hudson wrote:

How residential ISPs recoup costs (or simply increase revenue/profit) is another question entirely. I think the most insightful comment in this discussion was made by Mr. Rick Astley (I assume a pseudonym), when he states that ISPs have several options to increase revenue A) Increase price of their product, B) Implement usage restrictions, or C) Charge someone else/Make someone else your customer. I think he successfully argues that option C may be the best. As we've seen, the wireless market in the US went for option B. We've yet to see where the wireline market will go.

Some of the operators, here in Africa, who are venturing into FTTH, are continuing on with their data caps. I suppose if you're primarily a mobile carrier who uses data caps to charge for access (and makes lots of money in the process for, pretty much, selling nothing), the model becomes attractive regardless of the medium. New providers who are not encumbered by this type of thinking, obviously, have a more flexible and forward- thinking business model. Mark.

On Friday, May 16, 2014 05:35:39 PM Jay Ashworth wrote:

Could you expand a bit, Mark on "Social media forces the use of symmetric bandwidth"? Which social media platform is it that you think has a) symmetrical flows that b) are big enough to figure into transit symmetry?

What we saw with FTTH deployments is that customers uploaded more videos and photos to Youtube, Facebook, MySpace, e.t.c. They didn't do this on ADSL as much (it's too frustrating). When that caught on, customers started buying online backup services - synchronizing backups of their home or office computers to remote backup infrastructure. Again, they never did this with ADSL. What we learned: don't take it for granted that you will always know what your customers (or the content providers who serve them) will do with the bandwidth. If they have it, expect the worst, and plan for it as best you can. Mark.

On Friday, May 16, 2014 09:44:55 PM Scott Helms wrote:

I don't think that anyone disputes that when you improve the upstream you do get an uptick in usage in that direction. What I take issue with is the notion that the upstream is anything like downstream even when the capacity is there.

Certainly not - what I'm saying is that there can be a lot more upstream utilization than we are typically seeing today, if that stopper is unblocked. We can, then, take it from there... Mark.

I agree, and those peers should be then paid for the bits that your customers are requesting that they send through you if you cannot maintain a balanced peer relationship with them. It's shameful that access networks are attempting to not pay for their leeching of mass amounts of data in clear violation of standard expectations for balanced peering agreements. Oh... you meant something else? -Blake On Thu, May 15, 2014 at 12:34 PM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote:

On 5/15/14, 1:28 PM, "Nick B" <nick@pelagiris.org<mailto:nick@pelagiris.org>> wrote:

By "categorically untrue" do you mean "FCC's open internet rules allow us to refuse to upgrade full peers"?

Throttling is taking, say, a link from 10G and applying policy to constrain it to 1G, for example. What if a peer wants to go from a balanced relationship to 10,000:1, well outside of the policy binding the relationship? Should we just unquestionably toss out our published policy – which is consistent with other networks – and ignore expectations for other peers?

Jason

If traffic is unbalanced, what determines who is the payer and who is the payee? Apparently whoever can hold on to their customers better while performance is shit. On Thu, May 15, 2014 at 1:28 PM, Blake Dunlap <ikiris@gmail.com> wrote:

I agree, and those peers should be then paid for the bits that your customers are requesting that they send through you if you cannot maintain a balanced peer relationship with them. It's shameful that access networks are attempting to not pay for their leeching of mass amounts of data in clear violation of standard expectations for balanced peering agreements.

Oh... you meant something else?

-Blake

...

On 5/15/14, 1:28 PM, "Nick B" <nick@pelagiris.org<mailto: nick@pelagiris.org>> wrote:

By "categorically untrue" do you mean "FCC's open internet rules allow us to refuse to upgrade full peers"?

Throttling is taking, say, a link from 10G and applying policy to constrain it to 1G, for example. What if a peer wants to go from a balanced relationship to 10,000:1, well outside of the policy binding the relationship? Should we just unquestionably toss out our published policy – which is consistent with other networks – and ignore expectations for other

On Thu, May 15, 2014 at 12:34 PM, Livingood, Jason <Jason_Livingood@cable.comcast.com> wrote: peers?

...

Jason

Jason I think it is important to consider that you are operating your AS 7922 to serve a global Internet. In US, there is not a lot of choke because all the big Internet property - Google, Facebook, Microsoft, Amazon - pay toll to reach Comcast Broadband customer. If they do not pay u, there is not a competitive provider to pay. The routes to Comcast from "peers" are very congested. If u don't pay Comcast, you get sent over throttled Tata link... how is that "open" internet? In my home country, the routes to Comcast are poor bc dominant provider, AS 6453 Tata, is vendor Comcast refuses to upgrade so it can play games. Thank u again for helping provide information, but pls try to keep it accurate. Regards- Arvinder Singh

On 5/15/14, 12:43 PM, "Nick B" <nick@pelagiris.org> wrote:

...

Yes, you've got "some of the largest Internet companies as customers². Because you told them "if you don't pay us, we'll throttle you". Then you throttled them. I'm sorry, not a winning argument. Nick

That is categorically untrue, however nice a soundbite it may be. If you or anyone else truly believes we are throttling someone then I encourage you to file a formal complaint with the FCC. According to their Open Internet rules that we are bound to through at least 2018 (IIRC) we may not discriminate on traffic in that way, so there is a clear rule and a clear process for complaints.

Jason

On May 15, 2014, at 7:57 AM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

Upgrades/buildout are happening every day. They are continuous to keep ahead of demand and publicly measured by SamKnows (FCC measuring broadband), Akamai, Ookla, etc

I didn’t say they weren’t doing any upgrades/buildouts. I will say that the copper capabilities in my neighborhood are so far behind demand(s) that it is abysmal. There hasn’t been significant maintenance to the $TELCO copper plant in my neighborhood since it was installed in 1960.

What is not well known is that Comcast has been an existing commercial transit business for 15+ years (with over 8000 commercial fiber customers). Comcast also has over 40 balanced peers with plenty of capacity, and some of the largest Internet companies as customers.

I’ve been asked by my employer to stop picking on specific large ISPs. However, my experiences with $CABLECO have been as described. The infrastructure in my neighborhood was horrible and did not improve at all until I ordered business class service from them. I’ve seen nothing to indicate that there is any significant effort to improve customer satisfaction, but lots of things to indicate that they are trying to leverage as much revenue out of as little investment as possible. Owen

- Kevin

215-313-1083

...

On May 15, 2014, at 10:19 AM, "Owen DeLong" <owen@delong.com> wrote:

Oh, please do explicate on how this is inaccurate…

Owen

...

On May 14, 2014, at 2:14 PM, McElearney, Kevin <Kevin_McElearney@cable.comcast.com> wrote:

Respectfully, this is a highly inaccurate "sound bite"

- Kevin

215-313-1083

...

On May 14, 2014, at 3:05 PM, "Owen DeLong" <owen@delong.com> wrote:

Yes, the more accurate statement would be aggressively seeking new ways to monetize the existing infrastructure without investing in upgrades or additional buildout any more than absolutely necessary.

Owen

On May 14, 2014, at 8:02 AM, Hugo Slabbert <hugo@slabnet.com> wrote:

...

...

So they seek new sources of revenues, and/or attempt to thwart > competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of

...

revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin).

Sure they are (seeking new sources of revenue). They're not necessarily creating new products or services, i.e. actually adding any value, but they are finding ways to extract additional revenue from the same pipes, e.g. through paid peering with content providers.

I'm not endorsing this; just pointing out that you two are actually in agreement here.

-- Hugo

...

> On Wed, May 14, 2014 at 7:23 AM, <charles@thefnf.org> wrote: > > On 2014-05-14 02:04, Jean-Francois Mezei wrote: > > On 14-05-13 22:50, Daniel Staal wrote: > > They have the money. They have the ability to get more money. *They see >> no reason to spend money making customers happy.* They can make more >> profit without it. > > There is the issue of control over the market. But also the pressure > from shareholders for continued growth.

Yes. That is true. Except that it's not.

How do service providers grow? Let's explore that:

What is growth for a transit provider?

More (new) access network(s) (connections). More bandwidth across backbone pipes.

What is growth for access network? More subscribers.

Except that the incumbent carriers have shown they have no interest in providing decent bandwidth to anywhere but the most profitable rate centers. I'd say about 2/3 of the USA is served with quite terrible access.

> The problem with the internet is that while it had promises of wild > growth in the 90s and 00s, once penetration reaches a certain level, > growth stabilizes.

Penetration is ABYSMAL sir. Huge swaths of underserved americans exist.

> When you combine this with threath to large incumbents's media and media > distribution endeavours by the likes of Netflix (and cat videos on > Youtube), large incumbents start thinking about how they will be able to > continue to grow revenus/profits when customers will shift spending to > vspecialty channels/cableTV to Netflix and customer growth will not > compensate.

Except they aren't. Even in the most profitable rate centers, they've declined to really invest in the networks. They aren't a real business. You have to remember that. They have regulatory capture, natural/defacto monopoly etc etc. They don't operate in the real world of risk/reward/profit/loss/uncertainty like any other real business has to.

> So they seek new sources of revenues, and/or attempt to thwart > competition any way they can.

No to the first. Yes to the second. If they were seeking new sources of revenue, they'd be massively expanding into un/der served markets and aggressively growing over the top services (which are fat margin). They did a bit of an advertising campaign of "smart home" offerings, but that seems to have never grown beyond a pilot.

> The current trend is to "if you can't fight them, jon them" where > cablecos start to include the Netflix app into their proprietary set-top > boxes. The idea is that you at least make the customer continue to use > your box and your remote control which makes it easier for them to > switch between netflix and legacy TV. True. I don't know why one of the cablecos hasn't licensed roku, added cable card and made that available as a "hip/cool" set top box offering and charge another 10.00 a month on top of the standard dvr rental.

Would be interesting to see if those cable companies that are agreeing > to add the Netflix app onto their proprietary STBs also play peering > capacity games to degrade the service or not.

So how is the content delivered? Is it over the internet? Or is it over the cable plant, from cable headends?

It is important to consider bias and factual accuracy of the material. George Ou was working for Comcast and AT&T as a lobbyist at the time he produced the Youtube video. Drive Slow, Paul Wall On Sat, May 10, 2014 at 3:04 PM, Rick Astley <jnanog@gmail.com> wrote:

That was an interesting read but it's not the whole story. Skip to the TL;DR if you'd like but I'll attempt to explain what happened. What he isn't saying is the roles of the companies involved have changed over the last 10 years. Mostly gone are the days that content providers and access networks each just gave a middleman/transit provider money to reach each other. "Content provider" has expanded to become "content delivery network" and "access network" has expanded their role to offer transit as well. If these networks have a large amount of traffic between them and are able to reach each other in multiple locations nationally what is the technical reason a 3rd party transit network is required instead of a direct peering relationship? From a purely technical perspective content and access at that scale can peer directly cutting out the middle man.

The reality is an increasingly directly peered Internet doesn't sit well if you are in the business of being the middle man. Now if you will, why do transit companies themselves charge content companies to deliver bits? How is it fair to be in the business of charging companies to receive their bits and hand them to a settlement free peer on the hook to deliver them, but not fair for content to just bypass the transit company and enter a paid peering agreement with the company delivering the bits? In this case paid peering is mutually beneficial to both companies involved and is typically cheaper for the content company than it would cost to send that traffic over transit.

What we have is a major shift in the market over the last 10 or so years. So why are these large nationally connected "access" networks charging Level 3 for ports? That's the elephant in the room here and to understand that you have to go back to where (to my knowledge) this dispute first went public. The most comprehensive description I have seen to date is the following Youtube video: https://www.youtube.com/watch?v=tR1sLLOYxnY

I recommend the video before continuing. "Level 3" is really both Level 3 transit and Level 3 CDN. Level 3 has already had a long standing precedent of justifying the right of an ISP to charge for content delivery. So what happens when Level 3 greatly expands their content delivery business and sends traffic to other ISP's over settlement free ports? The large access networks say "hey, content delivery is a billable service, you should know" and they ask Level 3 CDN for compensation. The middleman networks protest and say "Charging for content delivery is only OK if we do it, but not when you do it!" and their justification for this claim is made on the basis that unlike access networks they a) Have a large network and b) send a full table of prefixes.

So lets look at the first claim. Are the transit networks large? Yes, but especially in the case of North American traffic destined for North America they are typically smaller overall than the largest access networks who arguably have the lions share of equipment tasked with delivering the bits beyond just the colo. The 2nd claim is mostly a strawman and this is why. Middlemen still carry traffic not destined to directly connected peers but how they bill for it is largely based on volume of traffic, not the number of prefixes exchanged. The big content providers and the big access networks make up a majority of the traffic on the Internet even if they don't make up a majority of the prefixes.

TL;DR So the reason the ports are maxed out is the market has changed, access networks have attempted to change peering agreements to match the existing market conditions but the middleman networks are arguing they should be exempt from the long standing tradition of charging for content delivery they themselves helped to establish. Some middleman networks have responded by refusing payment to access networks for delivery and as a result, the paths have not been upgraded and remain congested.

End of TL;DR

The next part is (even) more opinion than fact so you are forgiven if you stop here. My opinion is this is a peering dispute more than something that should fall under net neutrality. If content companies sent letters to "middlmen" that said "In your statements to the public you made the case that content delivery to ISP's should be settlement free so we have decided to take your offer and refuse any further payment to you from here forward" how would they handle it? Likely those companies would not only find themselves congested but depeered.

A bunch of people say charging at both ends is double dipping but really modern access networks are now at least partly filling the role of transit as well as last mile delivery. Where "content" "transit" and "access" all have a presence in the same colo, paying more money to send traffic through transit first instead of just directly to access because of some dated definition of what the roles of those companies are supposed to be doesn't make sense to me. Hijacking NN to attempt to bring litigation into the matter to protect an old business model from a changing market makes even less sense. Seeing Level 3 publish half truths in what looks like an attempt to mislead the public on the matter is disappointing. I would expect it from maybe Cogent but I have higher expectations of Level 3.

Broadband providers obviously aren't without some blame in the matter either. One of them is allowing customer satisfaction to be so low that they are easy targets for misinformation as most the comments I have seen on the matter to date are more emotional than rational. They have other mistakes too but for the purpose of keeping this brief and because some of them have been heavily documented elsewhere I'll save them for another day.

On Thu, May 8, 2014 at 1:18 PM, =JeffH <Jeff.Hodges@kingsmountain.com>wrote:

...

Observations of an Internet Middleman (Level3) http://blog.level3.com/global-connectivity/observations- internet-middleman/

See also...

Level 3 accuses five unnamed US ISPs of abusing their market power in peering http://gigaom.com/2014/05/05/level-3-accuses-five-unnamed- us-isps-of-abusing-their-market-power-in-peering/

"... I’d love to see Cogent, Google and other providers release their data next, so even if the FCC doesn’t want to pursue this, a growing cry of consumer outrage could push the agency to do something about a very real and difficult problem that’s crippling access to video content on 5 U.S. broadband networks. Level 3 didn’t name names, but based on the deals Netflix has signed and the complaints it has made about AT&T, I’m confident that AT&T, Verizon and Comcast are among the five. "

=JeffH

On Sat, May 10, 2014 at 8:04 AM, Rick Astley <jnanog@gmail.com> wrote:

[...] The reality is an increasingly directly peered Internet doesn't sit well if you are in the business of being the middle man. Now if you will, why do transit companies themselves charge content companies to deliver bits? How is it fair to be in the business of charging companies to receive their bits and hand them to a settlement free peer on the hook to deliver them, but not fair for content to just bypass the transit company and enter a paid peering agreement with the company delivering the bits? In this case paid peering is mutually beneficial to both companies involved and is typically cheaper for the content company than it would cost to send that traffic over transit.

What you're missing is that the transit provider is selling full routes. The access network is selling paid peering, which is a tiny fraction of the routes. If I pay transit provider X $10/mb (i know, not realistic, but it makes my math work) to reach the entire internet, it might seem reasonable to pay access network C $5/mb to hand traffic to them, and bypass the transit provider, avoiding potentially congested links. But then access network A decides they want to cut out the middleman as well--so they do the same thing, run their ports to transit provider X hot; to avoid that, I can pay the cheap price of $4/mb to reach them. Now access networks F and D want to do the same thing; their prices for their routes are $4 and $5/mb, respectively. Finally, little access provider T wants in at $2/mb for their routes. So, at the end of the week, I *had* been paying $10/mb to send traffic through transit to reach the whole rest of the internet. Now, I'm paying $5+$4+$4+$5+$2, or $30, and I don't have a full set of routes, so I've still got to keep paying the transit provider as well at $10. Depending on port counts, locations, and commit volumes, your "typically cheaper for the content company than it would cost to send that traffic over transit" has flown completely out the window. It could even end up being many times more expensive to handle the traffic that way. In order for the costs to work out, you'd really need to apply a formula along the lines of C(n) <= T(n) * C(t) where T(n) =fraction of traffic volume destined for access network X C(t)=cost of transit (ie, full routes, reachability to the entire internet) C(n)=cost of paid peering to access network X So, if you're an access network and want to charge for paid peering, and you represent 1/20th of my traffic, there's no reason for me to pay more than 1/20th of my transit cost for your routes; otherwise, it's more cost effective for me as a business to continue to pay a transit provider. I'm constantly amazed at how access networks think they can charge 2/3 the price of full transit for just their routes when they represent less than 1/10th of the overall traffic volume. The math just doesn't work out. It's nothing about being tier 1, or bigger than someone else; it's just math, pure and simple. Matt (currently not being paid by anyone for my time or thoughts, so take what I'm saying as purely my own thoughts on the matter, nothing more)

On Wednesday, May 14, 2014 11:27:57 AM Roland Dobbins wrote:

Are there any real-world models out there for revenue-sharing between app/content providers and access networks which would eliminate or reduce 'paid peering' (an alternate way to think of it is as 'delimited transit', another oxymoron like 'paid peering', but with a slightly different emphasis) monetary exchanges?

I think so, but none will cop to it publicly :-). Mark.