July 2014 - Test - lists.nanog.org

Verizon Public Policy on Netflix
by nanog＠brettglass.com 17 Jul '14

17 Jul '14

This is Brett Glass; I have been alerted to some of the responses to my message (which was cross-posted by a third party) and have temporarily joined the list to chime in. The following is my response to his message, edited slightly to include some new information. Dave Temkin wrote: >First and foremost, we built our CDN, Open Connect, "Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set of policies for direct connection to ISPs, and for placing servers in ISPs' facilities, that is as favorable as possible in every way to Netflix. It costs Netflix as little as possible and the ISP as much as possible. >with the intention to >deploy it as widely as possible in order to save ISPs who are delivering >our traffic money It does not, in fact, appear to save ISPs money. Note that Comcast asked for, and was given, additional payments even after it did all of the things that are part of Netflix' "Open Connect" program. Netflix, exercising inappropriate market power, has not offered smaller ISPs such as my own the same amount per customer. In fact, it has offered us no money at all -- even though our costs per Netflix customer are higher. Netflix thus discriminates against and threatens smaller ISPs, and by doing so, harms broadband competition. >and improve our mutual customer experience. This goes for >ISPs large and small, domestic and international, big endian and little >endian. We've never demanded payment from an ISP nor have we ever charged >for an Open Connect Appliance. The power and bandwidth consumed by an "Open Connect Appliance" (which is really just a hosted Netflix server) are a substantial expense for any ISP. Especially because the server is not a cache; it is stocked with content whether it is used or not and therefore wastes bandwidth on content and formats that will never be used even once. >When we first launched almost three years ago, we set a lower boundary for >receiving a Netflix Open Connect Appliance (which are always free) at >5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies >of how we pre-load our appliances (more on that below). Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts. If every one of those streams at 1 Mbps at the same time, which is highly unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded simply by this requirement if not by others (such as the requirement that the ISP alone pay for a dedicated connection to one of Netflix' relatively few "peering points"). >We explicitly call our "cache" an Appliance because it's not a demand >driven transparent or flow-through cache like the Akamai or Google caches. >We do this because we know what's going to be popular the next day or even >week and push a manifest to the Appliance to tell it what to download >(usually in the middle of the night, but this is configurable by the ISP). What Netflix does not say here is that (a) it can only somewhat predict what will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple copies of each video to its server in different formats, rather than transcoding locally or saving bandwidth on lesser used formats via caching; and (c) its server consumes large amounts of energy and bandwidth. A cache can be much more efficient and can be owned and managed by the ISP. >The benefit of this architecture is that a single Appliance can get 70+% >offload on a network, and three appliances clustered together can get 90+% >offload, while consuming approximately 500 watts of power, using 4U of rack >space, and serving 14Gbps per appliance. To put this in perspective: LARIAT builds its own caches which consume as little as 20 watts and can saturate a 10 Gbps Ethernet port. The Netflix servers are large, bloated power hogs compared to a well designed cache. >The downside of this architecture >is that it requires significant bandwidth to fill; in some ISPs cases >significantly more than they consume at peak viewing time. This is why our >solution may not work well for some small ISPs and we instead suggest >peering, which has 100% offload. Because it requires expensive bandwidth that's dedicated solely to Netflix, "peering" (as Netflix calls it; it's really just a dedicated link) has 0%, not 100%, offload. The ISP is paying for all of the bandwidth, and it cannot be used for anything else. >We've put a lot of effort into localizing our peering infrastructure >worldwide. As you can see from this map (sorry for the image), we're in 49 >locations around the world with the significant bulk of them in the US >(blue pins = 1 location, red pins = >1 location in a metro) - more detailed >version at <http://goo.gl/eDHpHU>http://goo.gl/eDHpHU and in our PeeringDB record ( ><http://as2906.peeringdb.com>http://as2906.peeringdb.com) : Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet "crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is, if anyplace can make the claim, the "center" of the entire Internet). Netflix' small and relatively sparse network of "peering" points does not include either of those locations. (I've noted, just today, that they have only recently listed a presence at a private data center outside of Denver in Englewood, Colorado -- not one of the major Denver exchanges, such as 1850 Pearl, to which we're already connected at great expense.) To get to it, we would have to spend several thousand dollars per month on an expensive connection, with no reimbursement of this expense from Netflix. If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer. --Brett Glass

18 18

Re: Verizon Public Policy on Netflix
by nanog＠brettglass.com 16 Jul '14

16 Jul '14

At 11:17 AM 7/13/2014, Todd Lyons wrote: >Because that Netflix box is not an on-demand cache, it gets a bunch of >shows pushed to it that may or may not be watched by any of Brett's >customers. Then the bandwidth he must use to preload that box is >large, much larger than the sum of the streams his customers do watch. Yes. Especially since Netflix insists upon sending multiple copies of each video -- not only in different formats but in different resolutions -- to the server. >I did agree with the comment later in the email that making content >freely cached is a non-starter because that content could be copied >too easily. The content could be copied just as easily from a Netflix server if it were stolen from the ISP's office. However, by far the most likely place for illicit copies to be made is at the client end, because it can be done in a private location and attract no attention at all. However, if there is any concern about either a Netflix server OR an ISP's cache being used to obtain illicit copies of the video, the solution is simple. This is a trivial problem to solve. Send and store the streams in encrypted form, passing a decryption key to the user via a separate, secured channel such as an HTTPS session. Then, it is not possible to obtain usable copies of the content by stealing either a Netflix server OR an ISP-owned cache. Problem solved. --Brett Glass

3 2

Re: Net Neutrality...
by Brett Glass 16 Jul '14

16 Jul '14

At 05:06 PM 7/15/2014, Rubens Kuhl wrote: >Do you see Connect America Fund, the successor to Universal Service Fund, >as a threat to US rural WISPs or as the possible solution for them ? It's a major threat to rural WISPs and all competitive ISPs. Here's why. The FCC is demanding that ISPs become "Eligible Telecommunications Carriers," or ETCs, before they can receive money from it. An ETC is a telephone company which is regulated under the mountain of regulations, requirements, and red tape of Title II of the Telecomm Act. It has to report to both state regulatory agencies AND the FCC. It's a classification that doesn't fit ISPs at all, but they would have to subject themselves to this heavy-handed regulation before they could get a dime from the fund. The FCC just announced a "rural broadband experiment" in which it will fund ETCs, but not pure-play ISPs, to build out rural broadband; see http://www.fcc.gov/document/rural-broadband-experiments-order As part of this experiment, the FCC will pay telephone companies to overbuild us, even though the residents of the areas in question already have service. This is because, as far as the regulators are concerned, if they do not have their regulatory hooks in us, we don't exist and any service we provide does not count. The "experiment" also requires participants to tie up large amounts of money in escrow accounts so that they can obtain "letters of credit" guaranteeing performance. All of this is, alas, the regulators' way of attempting to destroy those whom they cannot regulate. IMHO, the USF is outmoded and should be disbanded. --Brett Glass

6 9

NANOG 62 - Baltimore - Call For Presentations is Open!
by Greg Dendy 16 Jul '14

16 Jul '14

NANOG Community- Thanks for helping to make NANOG 61 in Bellevue such a smashing success as the most attended meeting ever, on the 20th anniversary of the first meeting! NANOG will hold its 62nd meeting in Baltimore, MD on October 6-8, 2014, hosted by EdgeConneX. The NANOG Program Committee is now seeking proposals for presentations, panels, tutorials, tracks sessions, and keynote materials for the NANOG 62 program. We invite presentations highlighting issues relating to technology already deployed or soon-to-be deployed in the Internet, . Vendors are encouraged to work with operators to present real-world deployment experiences with the vendor's products and interoperability. Key dates to track if you wish to submit a presentation: * Presentation Abstracts and Draft Slides Due: August 4, 2014 * Topic List Posted: August 18, 2014 * Slides Due: September 1, 2014 * Agenda Published: September 15, 2014 NANOG 62 submissions are welcome on the P<http://pc.nanog.org/>rogram Committee Site<http://pc.nanog.org/> or email me if you have questions. Looking forward to seeing everyone in Baltimore! Thanks, Greg Dendy Chair, Program Committee North American Network Operator Group (NANOG)

1 1

Re: Net Neutrality...
by Brett Glass 15 Jul '14

15 Jul '14

At 02:16 PM 7/15/2014, Joly MacFie wrote: >And, in my experience, one needs around double or more of the listed bandwidth for a robust streaming connection. This is only true if the connection is of poor quality and dropped packets lead to regular 50% cuts in the data rate. Most users (and the FCC!) do not understand that, due to the Van Jacobson AIMD algorithm, quality matters far more than quantity when a service is delivered via TCP. --Brett Glass

1 0

Net Neutrality FCC COMMENTS OF THE INTERNET ASSOCIATION
by William Allen Simpson 15 Jul '14

15 Jul '14

http://internetassociation.org/wp-content/uploads/2014/07/Comments.pdf Really good, for those of us with the patience to ponder it. I tried writing my own FCC response, and was flummoxed by the difficulty. Official comment period ends today.

2 1

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 15 Jul '14

15 Jul '14

At 09:30 AM 7/15/2014, Baldur Norddahl wrote: >If that is the case, how would peering with Netflix help you any? It would not, and that is the point. Netflix' "peering" scheme (again, I take issue with the use of the term) doesn't help ISPs with high backhaul costs. Measures to reduce the amount of bandwidth that Netflix wastes, via uncached unicast streaming, would. But (and this is the point of the message which started this thread) they are sitting pretty as a monopoly and do not feel a need to work with ISPs to solve this problem. It's frustrating and is causing us to look for workarounds -- including going as far as to found a competing streaming service that is more ISP-friendly. >I took a look at your plans at http://www.lariat.net/rates.html. You use >the Netflix brand in your advertising (in the flyer) We don't "use" their brand, but do mention them as an example of a company that provides streaming media. (We also mention YouTube, Hulu, and Amazon Prime.) It's natural for them to be on that list because they have such a large market share that they qualify as a monopoly. They are attempting to leverage their market power against ISPs instead of working with us, which is a shame. Again, a customer of a small rural ISP ought to be every bit as valuable to them as a Comcast customer. We should receive at least the amount per customer that Comcast receives, especially because our costs are higher. >but none of your plans >are actually fast enough to provide Netflix service (up to 6 Mbps per >stream for Super HD). Netflix itself claims that you need only half a megabit to stream. (Whether that claim is accurate is another matter, but that is what they themselves say.) >Selling 1 Mbps is just not going to do it going forward, not even in rural areas. Unfortunately, due to the cost of backhaul (which the FCC is doing nothing about; it has refused to deal with the problem of anticompetitive price gouging on Special Access lines), that's what we can offer. The FCC has also failed to release enough spectrum (Shannon's Law) to allow us to provide much more to the average user; we have to budget access point bandwidth carefully. We do what we can and price as best we can. Most of our customers, given a choice of possible levels of service, choose 1 Mbps and in fact are satisfied with that because the quality is high. Remember, due to Van Jacobson's algorithm, a 10 Mbps TCP session that drops packets slows down (by a factor of 2 for each dropped packet!) to a net throughput of less than 1 Mbps very quickly. So, we concentrate on quality and our customers have a very good experience. Usually better than with cable modem connections with much higher claimed speeds. We're used to doing a lot with a little and watching every penny. But Netflix doesn't have the same attitude. It wastes bandwidth. Rural ISPs and their customers cannot afford to cover the cost of that waste. >I can say how we solve the backhaul problem. We only lease dark fiber and >then put our own 10 Gbps equipment on it. We can upgrade that any day to >40G, 100G or whatever we need, without any additional rent for the fiber. Nice if you can do that. We have not been able to obtain affordable dark fiber in our area. >Given your expertise seems to be wireless links, you could also backhaul >using Ubiquiti Airfiber: http://www.ubnt.com/airfiber/airfiber5/ That Ubiquiti radio reaches at most one mile reliably due to rain fade. Most of our links go much farther. Wireless is our specialty and we do know our options; we use carefully selected and engineered microwave and millimeter wave links throughout our network. Being a WISP is not easy; it employs every skill I've acquired throughout my entire life and is constantly challenging me to improve and learn more. --Brett Glass

4 3

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 15 Jul '14

15 Jul '14

At 12:18 PM 7/15/2014, Rubens Kuhl wrote: >If you are picky enough to prefer other radios that cost more on Mbps/$, >that's your call, We need reliability. That particular radio wouldn't cut it. As I've mentioned, users can get away with much less bandwidth if the quality is high, so going for a less reliable radio with a high nominal speed does not actually save money. Also, that 5 GHz radio is a "spectrum spammer" and hence is a bad neighbor. After 25 years of doing wireless, one learns what really works and what is a false economy. Believe me, we've learned some hard and expensive lessons. --Brett Glass

3 2

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 15 Jul '14

15 Jul '14

At 11:40 AM 7/15/2014, Rubens Kuhl wrote: >Read again. You answered thinking about AirFiber 24, while he mentioned >AirFiber 5, which goes much longer. Ah. I assumed that you were talking about the 24 GHz version, because we rejected the 5 GHz radio the moment we scanned the data sheet. It does not meet our standards for antenna gain or spectral efficiency. The 5 GHz band is in heavy use in our area (not only by us, but by many others). Such a radio simply couldn't survive in our RF environment. And even if by some miracle it could, the 5 GHz band is far too valuable for us to devote so much spectrum to a single backhaul. We use other bands and better equipment for high capacity point-to-point links. --Brett Glass

2 1

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 15 Jul '14

15 Jul '14

Netflix's arrangement isn't "peeering." (They call it that, misleadingly, as a way of attempting to characterize the connection as one that doesn't require money to change hands.) ISPs peer to connect their mutual Internet customers. Netflix is not an ISP, so it cannot be said to be "peering." It's merely establishing a dedicated link to an ISP while trying to avoid paying the ISP for the resources used. But regardless of the financial arrangements, such a connection doesn't require an ASN or BGP. In fact, it doesn't even require a registered IP address at either end! A simple Ethernet connection (or a leased line of any kind, in fact; it could just as well be a virtual circuit) and a static route would work just fine. --Brett Glass At 09:35 PM 7/14/2014, Mike Lyon wrote: >So if Netflix was at 1850 Pearl, you wouldn't be able to peer with >them anyways cuz u have no ASN?

7 7