Verizon Public Policy on Netflix
This is Brett Glass; I have been alerted to some of the responses to my message (which was cross-posted by a third party) and have temporarily joined the list to chime in. The following is my response to his message, edited slightly to include some new information. Dave Temkin wrote:
First and foremost, we built our CDN, Open Connect,
"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set of policies for direct connection to ISPs, and for placing servers in ISPs' facilities, that is as favorable as possible in every way to Netflix. It costs Netflix as little as possible and the ISP as much as possible.
with the intention to deploy it as widely as possible in order to save ISPs who are delivering our traffic money
It does not, in fact, appear to save ISPs money. Note that Comcast asked for, and was given, additional payments even after it did all of the things that are part of Netflix' "Open Connect" program. Netflix, exercising inappropriate market power, has not offered smaller ISPs such as my own the same amount per customer. In fact, it has offered us no money at all -- even though our costs per Netflix customer are higher. Netflix thus discriminates against and threatens smaller ISPs, and by doing so, harms broadband competition.
and improve our mutual customer experience. This goes for ISPs large and small, domestic and international, big endian and little endian. We've never demanded payment from an ISP nor have we ever charged for an Open Connect Appliance.
The power and bandwidth consumed by an "Open Connect Appliance" (which is really just a hosted Netflix server) are a substantial expense for any ISP. Especially because the server is not a cache; it is stocked with content whether it is used or not and therefore wastes bandwidth on content and formats that will never be used even once.
When we first launched almost three years ago, we set a lower boundary for receiving a Netflix Open Connect Appliance (which are always free) at 5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies of how we pre-load our appliances (more on that below).
Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts. If every one of those streams at 1 Mbps at the same time, which is highly unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded simply by this requirement if not by others (such as the requirement that the ISP alone pay for a dedicated connection to one of Netflix' relatively few "peering points").
We explicitly call our "cache" an Appliance because it's not a demand driven transparent or flow-through cache like the Akamai or Google caches. We do this because we know what's going to be popular the next day or even week and push a manifest to the Appliance to tell it what to download (usually in the middle of the night, but this is configurable by the ISP).
What Netflix does not say here is that (a) it can only somewhat predict what will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple copies of each video to its server in different formats, rather than transcoding locally or saving bandwidth on lesser used formats via caching; and (c) its server consumes large amounts of energy and bandwidth. A cache can be much more efficient and can be owned and managed by the ISP.
The benefit of this architecture is that a single Appliance can get 70+% offload on a network, and three appliances clustered together can get 90+% offload, while consuming approximately 500 watts of power, using 4U of rack space, and serving 14Gbps per appliance.
To put this in perspective: LARIAT builds its own caches which consume as little as 20 watts and can saturate a 10 Gbps Ethernet port. The Netflix servers are large, bloated power hogs compared to a well designed cache.
The downside of this architecture is that it requires significant bandwidth to fill; in some ISPs cases significantly more than they consume at peak viewing time. This is why our solution may not work well for some small ISPs and we instead suggest peering, which has 100% offload.
Because it requires expensive bandwidth that's dedicated solely to Netflix, "peering" (as Netflix calls it; it's really just a dedicated link) has 0%, not 100%, offload. The ISP is paying for all of the bandwidth, and it cannot be used for anything else.
We've put a lot of effort into localizing our peering infrastructure worldwide. As you can see from this map (sorry for the image), we're in 49 locations around the world with the significant bulk of them in the US (blue pins = 1 location, red pins = >1 location in a metro) - more detailed version at <http://goo.gl/eDHpHU>http://goo.gl/eDHpHU and in our PeeringDB record ( <http://as2906.peeringdb.com>http://as2906.peeringdb.com) :
Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet "crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is, if anyplace can make the claim, the "center" of the entire Internet). Netflix' small and relatively sparse network of "peering" points does not include either of those locations. (I've noted, just today, that they have only recently listed a presence at a private data center outside of Denver in Englewood, Colorado -- not one of the major Denver exchanges, such as 1850 Pearl, to which we're already connected at great expense.) To get to it, we would have to spend several thousand dollars per month on an expensive connection, with no reimbursement of this expense from Netflix. If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer. --Brett Glass
Brett, You've previously stated: https://www.grc.com/sn/sn-457.htm "$20 per mbps per month" "1.25 gigabits of bandwidth coming in" Math: $20/mb/month * 1250mb = $25,000month If netflix is 1/3 of bandwidth... saving 1/3 of $25,000 -=> $8,000/month. (OK, Keep 100mbps for Netflix to pre-populate, 100mbps is 30TB/month) (Now I'm curious how many GB/month Netflix pre-populates, hmmm) How would "4U of rent" and 500W($50) electricity *not* save money? If it's a money thing, then you have a price... How much would Netflix have to pay you for you to consider? Can you elaborate on: It costs Netflix as little as possible and the ISP as much as possible. If your ISP isn't tall enough for Netflix, Akamai has a lower barrier of entry. Have you let Akamai give you a local cache? why or why not? Steven Tardy, maybe I'm missing/overlooking something... On Sat, Jul 12, 2014 at 8:22 PM, <nanog@brettglass.com> wrote:
This is Brett Glass; I have been alerted to some of the responses to my message (which was cross-posted by a third party) and have temporarily joined the list to chime in. The following is my response to his message, edited slightly to include some new information.
Dave Temkin wrote:
First and foremost, we built our CDN, Open Connect,
"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set of policies for direct connection to ISPs, and for placing servers in ISPs' facilities, that is as favorable as possible in every way to Netflix. It costs Netflix as little as possible and the ISP as much as possible.
with the intention to deploy it as widely as possible in order to save ISPs who are delivering our traffic money
It does not, in fact, appear to save ISPs money. Note that Comcast asked for, and was given, additional payments even after it did all of the things that are part of Netflix' "Open Connect" program. Netflix, exercising inappropriate market power, has not offered smaller ISPs such as my own the same amount per customer. In fact, it has offered us no money at all -- even though our costs per Netflix customer are higher. Netflix thus discriminates against and threatens smaller ISPs, and by doing so, harms broadband competition.
and improve our mutual customer experience. This goes for ISPs large and small, domestic and international, big endian and little endian. We've never demanded payment from an ISP nor have we ever charged for an Open Connect Appliance.
The power and bandwidth consumed by an "Open Connect Appliance" (which is really just a hosted Netflix server) are a substantial expense for any ISP. Especially because the server is not a cache; it is stocked with content whether it is used or not and therefore wastes bandwidth on content and formats that will never be used even once.
When we first launched almost three years ago, we set a lower boundary for receiving a Netflix Open Connect Appliance (which are always free) at 5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies of how we pre-load our appliances (more on that below).
Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts. If every one of those streams at 1 Mbps at the same time, which is highly unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded simply by this requirement if not by others (such as the requirement that the ISP alone pay for a dedicated connection to one of Netflix' relatively few "peering points").
We explicitly call our "cache" an Appliance because it's not a demand driven transparent or flow-through cache like the Akamai or Google caches. We do this because we know what's going to be popular the next day or even week and push a manifest to the Appliance to tell it what to download (usually in the middle of the night, but this is configurable by the ISP).
What Netflix does not say here is that (a) it can only somewhat predict what will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple copies of each video to its server in different formats, rather than transcoding locally or saving bandwidth on lesser used formats via caching; and (c) its server consumes large amounts of energy and bandwidth. A cache can be much more efficient and can be owned and managed by the ISP.
The benefit of this architecture is that a single Appliance can get 70+% offload on a network, and three appliances clustered together can get 90+% offload, while consuming approximately 500 watts of power, using 4U of rack space, and serving 14Gbps per appliance.
To put this in perspective: LARIAT builds its own caches which consume as little as 20 watts and can saturate a 10 Gbps Ethernet port. The Netflix servers are large, bloated power hogs compared to a well designed cache.
The downside of this architecture is that it requires significant bandwidth to fill; in some ISPs cases significantly more than they consume at peak viewing time. This is why our solution may not work well for some small ISPs and we instead suggest peering, which has 100% offload.
Because it requires expensive bandwidth that's dedicated solely to Netflix, "peering" (as Netflix calls it; it's really just a dedicated link) has 0%, not 100%, offload. The ISP is paying for all of the bandwidth, and it cannot be used for anything else.
We've put a lot of effort into localizing our peering infrastructure worldwide. As you can see from this map (sorry for the image), we're in 49 locations around the world with the significant bulk of them in the US (blue pins = 1 location, red pins = >1 location in a metro) - more detailed version at <http://goo.gl/eDHpHU>http://goo.gl/eDHpHU and in our PeeringDB record ( <http://as2906.peeringdb.com>http://as2906.peeringdb.com) :
Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet "crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is, if anyplace can make the claim, the "center" of the entire Internet).
Netflix' small and relatively sparse network of "peering" points does not include either of those locations. (I've noted, just today, that they have only recently listed a presence at a private data center outside of Denver in Englewood, Colorado -- not one of the major Denver exchanges, such as 1850 Pearl, to which we're already connected at great expense.) To get to it, we would have to spend several thousand dollars per month on an expensive connection, with no reimbursement of this expense from Netflix.
If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer.
--Brett Glass
On 13 July 2014 06:39, Steven Tardy <sjt5atra@gmail.com> wrote:
(OK, Keep 100mbps for Netflix to pre-populate, 100mbps is 30TB/month) (Now I'm curious how many GB/month Netflix pre-populates, hmmm)
Shame Netflix can't fill their appliances using really cheap, bulk, one-way satellite bandwidth which is useless for most other Internet applications. Then their traffic wouldn't use any of your real, paid for, transit. Of course siting a dish would be another expense with hosting one of their boxes, but if it made the on-going costs go away... Aled
iBeam tried to do that. If only they had used something else than Windows Server and other Microsoft products to do the caching. On Sun, Jul 13, 2014 at 7:45 AM, Aled Morris <aledm@qix.co.uk> wrote:
On 13 July 2014 06:39, Steven Tardy <sjt5atra@gmail.com> wrote:
(OK, Keep 100mbps for Netflix to pre-populate, 100mbps is 30TB/month) (Now I'm curious how many GB/month Netflix pre-populates, hmmm)
Shame Netflix can't fill their appliances using really cheap, bulk, one-way satellite bandwidth which is useless for most other Internet applications. Then their traffic wouldn't use any of your real, paid for, transit.
Of course siting a dish would be another expense with hosting one of their boxes, but if it made the on-going costs go away...
Aled
-- Ulf Zimmermann, 1525 Pacific Ave., Alameda, CA-94501, #: 510-396-1764 You can find my resume at: http://www.Alameda.net/~ulf/resume.html
Just an observation: I've been on the internet since dirt was rocks. It seems to me that one theme which has come up over and over and over is that some new-ish technology demands more bandwidth than whatever it was people were doing previously and as it popularizes people begin fighting. In the early 80s it was downloading the host table, "could people please try NOT to all download via a script at exactly midnight!!!" Then it was free software in the eighties, did WSMR et al really have a RIGHT to become a magnet for such popular program downloads?! And graphic connection to remote super-computer centers. Could the images please be generated locally and downloaded "off hours" (whatever "off hours" meant on the internet) or even shipped via tape etc rather than all these real-time graphical displays running???!!! Hey, the BACKBONE was 56kb. Then Usenet, and images, particularly, oh, explicit images because OMG imagine if our administration found out our link was slow because students (pick a powerless political class to pick on and declare THEIR use wasteful) were downloading...um...you know. And games OMG games. I remember sitting in an asst provost's office in the 80s being lectured about how email was a complete and total waste of the university's resources! Computers were for COMPUTING (he had a phd in physics which is where that was coming from.) And the public getting on the internet (ahem.) On and on. Now it's video streaming. And then the bandwidth catches up and it's no big deal anymore. And then everyone stops arguing about it and goes on to the next thing to argue about. Probably will be something in the realm of this "Internet of Things" idea, too many people conversing with their toaster-ovens. My comment has always been the same: There are two kinds of people in this world: Those who try to figure out how bake more bread, and those who herd people into bread lines. I've always tried to be the sort of person who tries to figure out how to bake more bread. This too shall pass. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On 13/07/2014 01:22, nanog@brettglass.com wrote:
"Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set of policies for direct connection to ISPs, and for placing servers in ISPs' facilities, that is as favorable as possible in every way to Netflix. It costs Netflix as little as possible and the ISP as much as possible.
it's a local cache server and of course it costs Netflix as little as possible because they're a business not a charity. The access service provider gets better performance, gets to choose where in their network makes most sense to have the device installed, and gets to save on transit / long-hauling costs. The overall traffic figures aren't changed in any material way, so it's not obvious why you claim that "it costs [...] the ISP as much as possible", because it just doesn't. For local content cache services of this type, some service providers are large enough to be able to financially muscle the content providers into paying for hosting / local connectivity fees, etc. This is nearly always a function of whether that sort of a deal can be strong-armed or not. The reverse is also true: the content providers will nearly always decline to pay hosting and connectivity fees if they feel they can get away with it. Both situations are a reflection of the relative importance of each business to each other. Nick
----- Original Message -----
From: nanog@brettglass.com
This is Brett Glass; I have been alerted to some of the responses to my message (which was cross-posted by a third party) and have temporarily joined the list to chime in. The following is my response to his message, edited slightly to include some new information.
Well, they were actually responses to *my* message, which made a fundamental point which you carefully don't address here at all, amongst what our British counterparts would probably term your whinging. :-)
If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer.
Bandwidth is expensive. Given. You made the wrong gamble on how asymmetrical your customers connections would *really* be. But that doesn't make that traffic *not be* -- as your brothers in the telco arm would phrase it -- "at your customers' instance", rather than, as your arguments all assume, at Netflix's. About 80% of so of the responses I've seen here agree that's a reasonable view of the situation... so we'll for the moment assume that you didn't address it because you *can't* address it. Care to differ? Cheers, -- jra [ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ] -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
Jay Ashworth wrote: [ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ] Jay, Quite agree with you on this stuff. I used to spend a good part of my time working with municipalities on planning fiber builds - so VZ's behavior on those matters leave a pretty bad taste in my mouth too. But.. that's kind of a different issue, wouldn't you say? Am I obtuse or does it all boil down to: 1. If both Netflix customers, and Netflix all connected to a single network - customers would be paying for their access connections, and Netflix would be paying for a pipe big enough to handle the aggregate demand. 2. The issue is that customers connect to one network (actually multiple networks, but lets stick with Verizon for now), and pay Verizon; Netflix buys aggregate capacity into other networks; with one or more transit networks in the middle. 3. Somebody has to pay for what's in the middle (ports into transit networks, bandwidth across them). Those are additional costs, that wouldn't exist if everyone were connected to the same network. 4. Both parties can make reasonable claims about why the other guys should pay. 5. Verizon and Comcast are big enough to say "Netflix pays" - with Netflix making a visible stink about it. 6. Netflix is important enough to end users, that Netflix can tell the little guys "you pay." And yes, they're making it a little easier by providing the CDN boxes. 7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff. Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. .... Yogi Berra
On Jul 14, 2014, at 9:46 AM, Miles Fidelman <mfidelman@meetinghouse.net> wrote:
7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff.
I think here is where many of us may disagree. While the current (public) dispute between Verizon and Netflix is fun for everyone to point fingers at saying "look here, there is a problem", the market also "mostly works". Verizon and Netflix seem to have reached (per press reports) an agreement and the largest problem today is the lack of ability to turn-up these ports quickly. Some market players move at light-speed, others at more glacial paces. I've been paying close attention to this for a variety of reasons. I've heard stories of some incumbents taking double-digit months to provision these types of services to correct congestion. I'm expecting the resolution time-scale to be much longer than was seen with the Comcast <-> Netflix connections. it would not surprise me if it took 18 months to provision these ports. (I recall phoning AT&T once asking for 100m service at a commercial address and it took a swat-team of people on the phone to tell me they would be 4x/mo what I was paying.. I politely told them they were too expensive and to not schedule a 8 person conference call for a basic service level). - Jared
I recall phoning AT&T once asking for 100m service at a commercial address and it took a swat-team of people on the phone to tell me they would be 4x/mo what I was paying.. I politely told them they were too expensive and to not schedule a 8 person conference call for a basic service level.
i remember calling a very large telco to order a ds3 (yes, it was a while ago). they transferred me to engineering to see if they had capacity. engineering told me to hang on, and then i heard a lot of rustling of paper. i knew that even if i could get the circuit i would not want it. randy
----- Original Message -----
From: "Miles Fidelman" <mfidelman@meetinghouse.net>
Jay Ashworth wrote:
[ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ]
Jay, Quite agree with you on this stuff. I used to spend a good part of my time working with municipalities on planning fiber builds - so VZ's behavior on those matters leave a pretty bad taste in my mouth too. But.. that's kind of a different issue, wouldn't you say?
Certainly. Just full disclosure: I'm as motivated to reply to this as I am *because* I already have a hard-on for VZN. :-)
Am I obtuse or does it all boil down to:
1. If both Netflix customers, and Netflix all connected to a single network - customers would be paying for their access connections, and Netflix would be paying for a pipe big enough to handle the aggregate demand.
Correct.
2. The issue is that customers connect to one network (actually multiple networks, but lets stick with Verizon for now), and pay Verizon; Netflix buys aggregate capacity into other networks; with one or more transit networks in the middle.
3. Somebody has to pay for what's in the middle (ports into transit networks, bandwidth across them). Those are additional costs, that wouldn't exist if everyone were connected to the same network.
4. Both parties can make reasonable claims about why the other guys should pay.
There's argument about whether VZN's claims are reasonable, and I tend to fall on the "they are not, even though I don't like VZN anyway" side; this thread was as much a sanity check as anything.
5. Verizon and Comcast are big enough to say "Netflix pays" - with Netflix making a visible stink about it.
Yup.
6. Netflix is important enough to end users, that Netflix can tell the little guys "you pay." And yes, they're making it a little easier by providing the CDN boxes.
Fair amount easier I would say, but I don't think we have enough empirical evidence either way, at least not in this thread.
7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff.
I hope that it doesn't come to that. Regulation is horrible. Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
On Jul 14, 2014, at 06:46 , Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Jay Ashworth wrote:
[ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ]
Jay, Quite agree with you on this stuff. I used to spend a good part of my time working with municipalities on planning fiber builds - so VZ's behavior on those matters leave a pretty bad taste in my mouth too. But.. that's kind of a different issue, wouldn't you say?
Am I obtuse or does it all boil down to:
1. If both Netflix customers, and Netflix all connected to a single network - customers would be paying for their access connections, and Netflix would be paying for a pipe big enough to handle the aggregate demand.
2. The issue is that customers connect to one network (actually multiple networks, but lets stick with Verizon for now), and pay Verizon; Netflix buys aggregate capacity into other networks; with one or more transit networks in the middle.
Well, there are multiple possibilities here... A: CUST<->ACCESS_NETWORK<->TRANSIT_A<->TRANSIT_N<->NETFLIX B: CUST<->ACCESS_NETWORK<->TRANSIT<->NETFLIX C: CUST<->ACCESS_NETWORK<->NETFLIX In case A, it's pretty obvious that CUST $->ACCESS_NETWORK$->TRANSIT_A and NETFLIX$->TRANSIT_N It's not entirely clear what the economics would be between TRANSIT_A<->TRANSIT_N, but most likely settlement free peering. In case B, it's fairly obvious CUST $->ACCESS_NETWORK and it's less clear wehter: B1: ACCESS_NETWORK$->TRANSIT<-$NETFLIX (transit double-dip) B2: ACCESS_NETWORK$->TRANSIT and Transit is settlement free with Netflix (Access pays transit) B3: TRANSIT<-$NETFLIX and Access is settlement free with Transit (Netflix pays transit) I'm sure in the real world there are likely examples of all three scenarios. In case C, we arrive at what I think most of the argument is actually about. Obviuosly, CUST$->ACCESS_NETWORK. The question is whether there should also be ACCESS_NETWORK<-$NETFLIX, which is what Brett is claiming should happen and what at least one very large ACCESS_NETWORK has been able to achieve at least temporarily. In my opinion, this case is a case of Access Double Dip where the access network is being paid by both the customer and the supplier for the same delivery. As I said, this would be like paying for a product from $BOX_STORE and having $BOX_STORE bill me for shipping, and pay $CARRIER for deliver only to have $CARRIER show up at my door asking for even more money before they will fork over my package.
3. Somebody has to pay for what's in the middle (ports into transit networks, bandwidth across them). Those are additional costs, that wouldn't exist if everyone were connected to the same network.
I don't think that's really part of the argument here.
4. Both parties can make reasonable claims about why the other guys should pay.
Not really, IMHO. (See above and below)
5. $LARGE_ACCESS_NETWORKs are big enough to say "Netflix pays" - with Netflix making a visible stink about it.
LARGE_ACCESS_NETWORK may be able to force Netflix to pay, but that's not the same as saying Netflix _SHOULD_ pay. It's more like recognizing that market power and a large customer base can often force an economic decision that is contrary to what _SHOULD_ happen by any other rational evaluation.
6. Netflix is important enough to end users, that Netflix can tell the little guys "you pay." And yes, they're making it a little easier by providing the CDN boxes.
Perhaps, but that's not really what is happening here if you look at it in more detail. I don't deny that Netflix _COULD_ do this, just as $LARGE_ACCESS_NETWORKs _HAVE_ apparently done this to Netflix. However, so far, Netflix seems to be trying as hard as they can to provide cost-effective alternatives for ISPs to accept their bits in a variety of ways and allowing the ISP to choose which solution works best for them. True, Netflix hasn't built out every single distant corner of the universe with their peering network, but I would say that by any reasonable view of the situation, they have aggressively built quite a network over a large fraction of their service geography and to their credit, they are continuing to aggressively expand that network. To the best of my knowledge (and I'm sure Dave will correct me if I am wrong), Netflix would prefer to deliver bits settlement free directly to as many ACCESS_PROVIDERS as possible, because it saves Netflix from paying transit costs and it saves ACCESS_PROVIDERS from paying additional circuit or transit costs and it provides a better customer experience all around. In cases where Netflix' network does not geographically overlap $ACCESS_PROVIDER's network, then one or both will need to cover the cost of bridging that distance, whether with an IP transit relationship, a circuit, or some other mechanism. In most cases, since Netflix is in a high percentage of the major peering centers, most $ACCESS_PROVIDERS already have to build into one of those centers in order to reach many other things, so it is reasonable for them to connect to Netflix at that same point. In other cases, they use a transit provider to reach those centers as well, and so likely they will use the same transit to reach Netflix. In virtually every case, they're going to reach Netflix the same way they reach the majority of other top sites on the internet. In such a case, it makes sense that $ACCESS_PROVIDER pays for their unique geographic situation. It doesn't make sense to expect Netflix to subsidize their choice of geography. It seems to me that other than $LARGE_ACCESS_PROVIDERS' public statements trying to extort money from $CONTENT_PROVIDERS and Brett's posts in this conversation, the vast majority of people in this thread have overwhelmingly agreed with this point of view.
7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff.
Probably true, but I will point out that one of the main reasons that the Internet has become such a cost-effective alternative even for voice traffic vs. the PSTN is the lack of said formal policies and regulations about settlements. Of course, you did say "reasonably balanced" which I don't think is a term that could be rationally applied to the ITU settlement rules for the PSTN. Owen
Brett, Why would Netflix pay your ISP? You are, Brett, a tiny ISP. Only 200 customers. That's barely a /24 of IP addresses. What will happen instead is that your customers will pay to subsidize the network of larger ISPs who do have that marketing power. This is the true risk. Of Netflix is also paying money to the ISP, how do we know the true cost of the connection? A big ISP can fight their way into a position where no other ISP can compete.
Hello This is so simple. ISP offers xxMbps and should deliver that to the customer. Dear customer. If you cannot stream full quality, upgrade . Dear ISP stop promising xxMbps if you advertise a port cap lower than theoretical port bandwidth. Basically fraud. On Jul 16, 2014 7:19 PM, "Owen DeLong" <owen@delong.com> wrote:
On Jul 14, 2014, at 06:46 , Miles Fidelman <mfidelman@meetinghouse.net> wrote:
Jay Ashworth wrote:
[ As you might imagine, this is a bit of a hobby horse for me; Verizon's behavior about municipally owned fiber, and it's attempts to convert post- Sandy customers in NYS from regulated copper to unregulated FiOS service leave a pretty bad taste in my mouth about VZN. ]
Jay, Quite agree with you on this stuff. I used to spend a good part of my time working with municipalities on planning fiber builds - so VZ's behavior on those matters leave a pretty bad taste in my mouth too. But.. that's kind of a different issue, wouldn't you say?
Am I obtuse or does it all boil down to:
1. If both Netflix customers, and Netflix all connected to a single network - customers would be paying for their access connections, and Netflix would be paying for a pipe big enough to handle the aggregate demand.
2. The issue is that customers connect to one network (actually multiple networks, but lets stick with Verizon for now), and pay Verizon; Netflix buys aggregate capacity into other networks; with one or more transit networks in the middle.
Well, there are multiple possibilities here...
A: CUST<->ACCESS_NETWORK<->TRANSIT_A<->TRANSIT_N<->NETFLIX B: CUST<->ACCESS_NETWORK<->TRANSIT<->NETFLIX C: CUST<->ACCESS_NETWORK<->NETFLIX
In case A, it's pretty obvious that CUST $->ACCESS_NETWORK$->TRANSIT_A and NETFLIX$->TRANSIT_N It's not entirely clear what the economics would be between TRANSIT_A<->TRANSIT_N, but most likely settlement free peering.
In case B, it's fairly obvious CUST $->ACCESS_NETWORK and it's less clear wehter: B1: ACCESS_NETWORK$->TRANSIT<-$NETFLIX (transit double-dip) B2: ACCESS_NETWORK$->TRANSIT and Transit is settlement free with Netflix (Access pays transit) B3: TRANSIT<-$NETFLIX and Access is settlement free with Transit (Netflix pays transit)
I'm sure in the real world there are likely examples of all three scenarios.
In case C, we arrive at what I think most of the argument is actually about. Obviuosly, CUST$->ACCESS_NETWORK. The question is whether there should also be ACCESS_NETWORK<-$NETFLIX, which is what Brett is claiming should happen and what at least one very large ACCESS_NETWORK has been able to achieve at least temporarily. In my opinion, this case is a case of Access Double Dip where the access network is being paid by both the customer and the supplier for the same delivery.
As I said, this would be like paying for a product from $BOX_STORE and having $BOX_STORE bill me for shipping, and pay $CARRIER for deliver only to have $CARRIER show up at my door asking for even more money before they will fork over my package.
3. Somebody has to pay for what's in the middle (ports into transit networks, bandwidth across them). Those are additional costs, that wouldn't exist if everyone were connected to the same network.
I don't think that's really part of the argument here.
4. Both parties can make reasonable claims about why the other guys should pay.
Not really, IMHO. (See above and below)
5. $LARGE_ACCESS_NETWORKs are big enough to say "Netflix pays" - with Netflix making a visible stink about it.
LARGE_ACCESS_NETWORK may be able to force Netflix to pay, but that's not the same as saying Netflix _SHOULD_ pay. It's more like recognizing that market power and a large customer base can often force an economic decision that is contrary to what _SHOULD_ happen by any other rational evaluation.
6. Netflix is important enough to end users, that Netflix can tell the little guys "you pay." And yes, they're making it a little easier by providing the CDN boxes.
Perhaps, but that's not really what is happening here if you look at it in more detail. I don't deny that Netflix _COULD_ do this, just as $LARGE_ACCESS_NETWORKs _HAVE_ apparently done this to Netflix. However, so far, Netflix seems to be trying as hard as they can to provide cost-effective alternatives for ISPs to accept their bits in a variety of ways and allowing the ISP to choose which solution works best for them.
True, Netflix hasn't built out every single distant corner of the universe with their peering network, but I would say that by any reasonable view of the situation, they have aggressively built quite a network over a large fraction of their service geography and to their credit, they are continuing to aggressively expand that network.
To the best of my knowledge (and I'm sure Dave will correct me if I am wrong), Netflix would prefer to deliver bits settlement free directly to as many ACCESS_PROVIDERS as possible, because it saves Netflix from paying transit costs and it saves ACCESS_PROVIDERS from paying additional circuit or transit costs and it provides a better customer experience all around.
In cases where Netflix' network does not geographically overlap $ACCESS_PROVIDER's network, then one or both will need to cover the cost of bridging that distance, whether with an IP transit relationship, a circuit, or some other mechanism. In most cases, since Netflix is in a high percentage of the major peering centers, most $ACCESS_PROVIDERS already have to build into one of those centers in order to reach many other things, so it is reasonable for them to connect to Netflix at that same point. In other cases, they use a transit provider to reach those centers as well, and so likely they will use the same transit to reach Netflix. In virtually every case, they're going to reach Netflix the same way they reach the majority of other top sites on the internet. In such a case, it makes sense that $ACCESS_PROVIDER pays for their unique geographic situation. It doesn't make sense to expect Netflix to subsidize their choice of geography.
It seems to me that other than $LARGE_ACCESS_PROVIDERS' public statements trying to extort money from $CONTENT_PROVIDERS and Brett's posts in this conversation, the vast majority of people in this thread have overwhelmingly agreed with this point of view.
7. In the absence of some reasonably balanced formal policies and regulations about settlements - we're going to keep seeing this kind of stuff.
Probably true, but I will point out that one of the main reasons that the Internet has become such a cost-effective alternative even for voice traffic vs. the PSTN is the lack of said formal policies and regulations about settlements. Of course, you did say "reasonably balanced" which I don't think is a term that could be rationally applied to the ITU settlement rules for the PSTN.
Owen
If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer.
I noted most of the discussion seems to point to Internet bandwidth as a cost factor to ISPs, but I wonder what's the impact of Netflix on access network costs ? They might be harder to measure or directly correlate to streaming usage, but for non-wired networks (which is usually the case in rural networks), this impact sounds more harmful to me than uplink costs. Rubens
Let's just dispel this, internet bandwidth is not a very significant cost for access networks when compared to moving the data internally and maintaining the last mile access. That being said, incremental usage can drive huge capex, almost always in the very expensive last mile. Most of our cost (as a cable provider) on a per-bit basis is between the head-end and the customer, or between the head-end and the regional pop. The main driver here should be obvious, the bigger the pipe on the same route, the cheaper the bits... A cable carrying 300 kbit/sec costs just as much to maintain and install as a cable carrying 300 gbit/sec on the outside plant side of the equation, and that is where the real cost is. John -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Rubens Kuhl Sent: Monday, July 14, 2014 10:07 AM To: Nanog Subject: Re: Verizon Public Policy on Netflix
If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer.
I noted most of the discussion seems to point to Internet bandwidth as a cost factor to ISPs, but I wonder what's the impact of Netflix on access network costs ? They might be harder to measure or directly correlate to streaming usage, but for non-wired networks (which is usually the case in rural networks), this impact sounds more harmful to me than uplink costs. Rubens
If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer.
I noted most of the discussion seems to point to Internet bandwidth as a cost factor to ISPs, but I wonder what's the impact of Netflix on access network costs ? They might be harder to measure or directly correlate to streaming usage, but for non-wired networks (which is usually the case in rural networks), this impact sounds more harmful to me than uplink costs. if your customer buys 20, needs 6 and gets 4 I guess that's problem, if
On 7/14/14 10:06 AM, Rubens Kuhl wrote: the customer buys 2 and needs 4 that's a different one... It's politically inconvenient to assign blame to third parties for the provisioned capacity of the last mile network.
Rubens
participants (18)
-
Aled Morris -
Barry Shein -
Jared Mauch -
Jay Ashworth -
joel jaeggli -
John Osmon -
John van Oppen -
Matthew Petach -
mcfbbqroast . -
Miles Fidelman -
nanog＠brettglass.com -
Nick Hilliard -
Owen DeLong -
Randy Bush -
Rogan Schlassa -
Rubens Kuhl -
Steven Tardy -
Ulf Zimmermann