- Test - lists.nanog.org

Re: Net Neutrality...
by Larry Sheldon 17 Jul '14

17 Jul '14

On 7/16/2014 1:57 PM, Doug Barton wrote: > On 07/16/2014 08:45 AM, Eric Brunner-Williams wrote: >> On 7/16/14 7:50 AM, Fred Baker (fred) wrote: >>> Relevant article by former FCC Chair >>> >>> http://www.washingtonpost.com/posteverything/wp/2014/07/14/this-is-why-the-… >>> >>> >> >> It reads like a hit piece (by a Republican "free markets" ideologue) on >> a (Progressive) Democratic primary candidate for Lt. Governor of New >> York, not like a reasoned case by an informed policy analyst. > > Errr, I didn't see anything about any LTG candidates in that piece, what > did I miss? I'm also curious about what it is that you think is > misstated or overblown in that piece that would lead you to believe that > it's a "hit piece." His use of phrases like "by a Republican "free markets" ideologue" pretty much nailed the value of his remarks for me. -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)

1 0

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 17 Jul '14

17 Jul '14

At 07:47 PM 7/14/2014, Matthew Petach wrote: >And as long as they're happy with their single upstream >connectivity picture, more power to them. You're assuming that the only way to be multi-homed is to have an ASN. That's not correct. ARIN's fees are discriminatory; a small ISP must pay a much higher percentage of its revenues than a large one for IPs, ASNs, etc. Clever small ISPs find ways to work around that, and it makes them more competitive. --Brett Glass

7 10

Re: Inevitable death, was Re: Verizon Public Policy on Netflix
by Brett Glass 17 Jul '14

17 Jul '14

Mike: An ASN is, literally, just a number. One that's used by a very awkward and primitive routing system that requires constant babysitting and tweaking and, after lo these many years, still doesn't deliver the security or robustness it should. Obtaining this token number (and a bunch of IP addresses which is no different, qualitatively, from what I already have) would be a large expense that would not produce any additional value for my customers but could force me to raise their fees -- something which I absolutely do not want to do. Perhaps it's best to think of it this way: I'm outsourcing some backbone routing functions to my upstreams, which (generously) aren't charging me anything extra to do it. In my opinion, that's a good business move. As for "peering:" the definition is pretty well established. ISPs do it; content providers at the edge do not. Netflix is fighting a war of semantics and politics with ISPs. It is trying to cling to every least penny it receives and spend none of it on the resources it consumes or on making its delivery of content more efficient. We have been in conversations with it in which we've asked only for it to be equitable and pay us the same amount per customer as it pays other ISPs, such as Comcast (since, after all, they should be just as valuable to it). It has refused to do even that much. That's why talks have, for the moment, broken down and we are looking at other solutions. --Brett Glass At 09:58 PM 7/14/2014, Mike Lyon wrote: >So we are splitting hairs with what "peering" means? And I am sure >Netflix (or any other content / network / CDN provider) would be >more than happy to statically route to you? Doubtful. > >Dude, put your big boy pants on, get an ASN, get some IP space, Â >I am a smaller ISP than you I am sure and I have both. It's not >rocket science. How are other networks suppose to take you >seriously if you don't have an ASN? > >-Mike

8 7

Re: Verizon Public Policy on Netflix
by nanog＠brettglass.com 17 Jul '14

17 Jul '14

At 11:39 PM 7/12/2014, Steven Tardy wrote: >How would "4U of rent" and 500W($50) electricity *not* save money? Because, on top of that, we'd have huge bandwidth expenses. And Netflix would refuse to cover any of that out of the billions in fees it's collecting from subscribers. We can't raise our prices (that would not only cost us customers but be unfair to many of them; it would be forcing the non-Netflix users to subsidize Netflix). We simply need Netflix to pay at least some of its freight. >If your ISP isn't tall enough for Netflix, Akamai has a lower barrier of entry. >Have you let Akamai give you a local cache? why or why not? Akamai refused to do so when we approached them. The Akamai rep was rather rude and dismissive about it; we were too small to be worthy of their attention. It's important to note that the growth of rural ISPs is limited by population. Even if we did not have rapacious cable and telephone monopolies to compete with, our size is naturally limited by the number of possible customers. Each of those customers is every bit as valuable as an urban customer, but Netflix won't even give us the SAME amount per customer it gives Comcast, much less more (it costs more to serve each one). And Netflix is particularly out of line because it is insisting that we pay huge bandwidth bills for an exclusive connection just to it. It is also wasting our existing bandwidth by refusing to allow caching. If Netflix continues on its current course, ALL ISPs -- not just rural ones, will eventually be forced to rebel. And it will not be pretty. Our best hope, unless Netflix changes its ways, is for a competitor to come along which has more ISP-friendly practices. Such a competitor could easily destroy Netflix via better relations with ISPs... and better performance and lower costs due to caching at the ISP. --Brett Glass

10 10

Re: Verizon Public Policy on Netflix
by Brett Glass 17 Jul '14

17 Jul '14

At 10:25 AM 7/13/2014, Charles Gucker wrote: >ALL ISPs are in the business of providing access to >the Internet. If you feel the need to rebel, then I suggest you >look at creative ways to increase revenue from your customers, My customers do not want me to "creatively" find ways to extract additional money from them so as to cover expenses that Netflix should be covering. Nor do they want me to subsidize Netflix subscribers from the fees from non-Netflix subscribers. They want to pay a fair price for their Internet that does not include paying ransom to third parties. We currently provide that: we guarantee each subscriber a certain minimum capacity to the Internet exchange at 1850 Pearl Street in Denver (to which Netflix does not directly connect) with a certain maximum duty cycle. But we can't guarantee the performance of a specific third party service such as Netflix. If Netflix wants us to do that, it is going to have to pay us, as it pays Comcast. That's only fair, because we would be doing something special just for it -- something which costs money. If Netflix tries to use its market power to harm ISPs, or to smear us via nasty on-screen messages as it has been smearing Verizon, ISPs have no choice but to react. One way we could do this -- and I'm strongly considering it -- is to start up a competing streaming service that IS friendly to ISPs. It would use the minimum possible amount of bandwidth, make proper use of caching, and -- most importantly -- actually PAY Internet service providers, instead of sapping their resources, by allowing them to sell it and keep a portion of the fee. This would provide an automatic, direct, per-customer reimbursement to the ISP for the cost of bandwidth. ISPs would sign on so fast that such a service could BURY Netflix in short order. --Brett Glass

23 28

Verizon Public Policy on Netflix
by nanog＠brettglass.com 17 Jul '14

17 Jul '14

This is Brett Glass; I have been alerted to some of the responses to my message (which was cross-posted by a third party) and have temporarily joined the list to chime in. The following is my response to his message, edited slightly to include some new information. Dave Temkin wrote: >First and foremost, we built our CDN, Open Connect, "Open Connect" is not, in fact, a CDN. Nor is it "peering." It is merely a set of policies for direct connection to ISPs, and for placing servers in ISPs' facilities, that is as favorable as possible in every way to Netflix. It costs Netflix as little as possible and the ISP as much as possible. >with the intention to >deploy it as widely as possible in order to save ISPs who are delivering >our traffic money It does not, in fact, appear to save ISPs money. Note that Comcast asked for, and was given, additional payments even after it did all of the things that are part of Netflix' "Open Connect" program. Netflix, exercising inappropriate market power, has not offered smaller ISPs such as my own the same amount per customer. In fact, it has offered us no money at all -- even though our costs per Netflix customer are higher. Netflix thus discriminates against and threatens smaller ISPs, and by doing so, harms broadband competition. >and improve our mutual customer experience. This goes for >ISPs large and small, domestic and international, big endian and little >endian. We've never demanded payment from an ISP nor have we ever charged >for an Open Connect Appliance. The power and bandwidth consumed by an "Open Connect Appliance" (which is really just a hosted Netflix server) are a substantial expense for any ISP. Especially because the server is not a cache; it is stocked with content whether it is used or not and therefore wastes bandwidth on content and formats that will never be used even once. >When we first launched almost three years ago, we set a lower boundary for >receiving a Netflix Open Connect Appliance (which are always free) at >5Gbps. Since then we've softened that limit to 3.5Gbps due to efficiencies >of how we pre-load our appliances (more on that below). Most small ISPs (the average in the US, in fact) have 1,000 to 2,000 accounts. If every one of those streams at 1 Mbps at the same time, which is highly unlikely, this still does not reach 3.5 Gbps. Therefore, most ISPs are excluded simply by this requirement if not by others (such as the requirement that the ISP alone pay for a dedicated connection to one of Netflix' relatively few "peering points"). >We explicitly call our "cache" an Appliance because it's not a demand >driven transparent or flow-through cache like the Akamai or Google caches. >We do this because we know what's going to be popular the next day or even >week and push a manifest to the Appliance to tell it what to download >(usually in the middle of the night, but this is configurable by the ISP). What Netflix does not say here is that (a) it can only somewhat predict what will be in demand or go "viral;" (b) it wastes bandwidth by sending multiple copies of each video to its server in different formats, rather than transcoding locally or saving bandwidth on lesser used formats via caching; and (c) its server consumes large amounts of energy and bandwidth. A cache can be much more efficient and can be owned and managed by the ISP. >The benefit of this architecture is that a single Appliance can get 70+% >offload on a network, and three appliances clustered together can get 90+% >offload, while consuming approximately 500 watts of power, using 4U of rack >space, and serving 14Gbps per appliance. To put this in perspective: LARIAT builds its own caches which consume as little as 20 watts and can saturate a 10 Gbps Ethernet port. The Netflix servers are large, bloated power hogs compared to a well designed cache. >The downside of this architecture >is that it requires significant bandwidth to fill; in some ISPs cases >significantly more than they consume at peak viewing time. This is why our >solution may not work well for some small ISPs and we instead suggest >peering, which has 100% offload. Because it requires expensive bandwidth that's dedicated solely to Netflix, "peering" (as Netflix calls it; it's really just a dedicated link) has 0%, not 100%, offload. The ISP is paying for all of the bandwidth, and it cannot be used for anything else. >We've put a lot of effort into localizing our peering infrastructure >worldwide. As you can see from this map (sorry for the image), we're in 49 >locations around the world with the significant bulk of them in the US >(blue pins = 1 location, red pins = >1 location in a metro) - more detailed >version at <http://goo.gl/eDHpHU>http://goo.gl/eDHpHU and in our PeeringDB record ( ><http://as2906.peeringdb.com>http://as2906.peeringdb.com) : Our ISP connects to the Internet in Cheyenne, Wyoming (a major Internet "crossroads;" it's where I-80 meets I-25) and Denver, Colorado (which is, if anyplace can make the claim, the "center" of the entire Internet). Netflix' small and relatively sparse network of "peering" points does not include either of those locations. (I've noted, just today, that they have only recently listed a presence at a private data center outside of Denver in Englewood, Colorado -- not one of the major Denver exchanges, such as 1850 Pearl, to which we're already connected at great expense.) To get to it, we would have to spend several thousand dollars per month on an expensive connection, with no reimbursement of this expense from Netflix. If Netflix were a good citizen, it would (a) let ISPs cache content; (b) pay them equitably for direct connections (smaller and more remote ISPs have higher costs per customer and should get MORE per account than Comcast, rather than receiving nothing); and (c) work with ISPs to develop updated technology that makes streaming more efficient. Bandwidth is expensive, and unicast streaming without caching is by far the most inefficient conceivable way of delivering "fat" content to the consumer. --Brett Glass

18 18

Re: Verizon Public Policy on Netflix
by nanog＠brettglass.com 16 Jul '14

16 Jul '14

At 11:17 AM 7/13/2014, Todd Lyons wrote: >Because that Netflix box is not an on-demand cache, it gets a bunch of >shows pushed to it that may or may not be watched by any of Brett's >customers. Then the bandwidth he must use to preload that box is >large, much larger than the sum of the streams his customers do watch. Yes. Especially since Netflix insists upon sending multiple copies of each video -- not only in different formats but in different resolutions -- to the server. >I did agree with the comment later in the email that making content >freely cached is a non-starter because that content could be copied >too easily. The content could be copied just as easily from a Netflix server if it were stolen from the ISP's office. However, by far the most likely place for illicit copies to be made is at the client end, because it can be done in a private location and attract no attention at all. However, if there is any concern about either a Netflix server OR an ISP's cache being used to obtain illicit copies of the video, the solution is simple. This is a trivial problem to solve. Send and store the streams in encrypted form, passing a decryption key to the user via a separate, secured channel such as an HTTPS session. Then, it is not possible to obtain usable copies of the content by stealing either a Netflix server OR an ISP-owned cache. Problem solved. --Brett Glass

3 2

Re: Net Neutrality...
by Brett Glass 16 Jul '14

16 Jul '14

At 05:06 PM 7/15/2014, Rubens Kuhl wrote: >Do you see Connect America Fund, the successor to Universal Service Fund, >as a threat to US rural WISPs or as the possible solution for them ? It's a major threat to rural WISPs and all competitive ISPs. Here's why. The FCC is demanding that ISPs become "Eligible Telecommunications Carriers," or ETCs, before they can receive money from it. An ETC is a telephone company which is regulated under the mountain of regulations, requirements, and red tape of Title II of the Telecomm Act. It has to report to both state regulatory agencies AND the FCC. It's a classification that doesn't fit ISPs at all, but they would have to subject themselves to this heavy-handed regulation before they could get a dime from the fund. The FCC just announced a "rural broadband experiment" in which it will fund ETCs, but not pure-play ISPs, to build out rural broadband; see http://www.fcc.gov/document/rural-broadband-experiments-order As part of this experiment, the FCC will pay telephone companies to overbuild us, even though the residents of the areas in question already have service. This is because, as far as the regulators are concerned, if they do not have their regulatory hooks in us, we don't exist and any service we provide does not count. The "experiment" also requires participants to tie up large amounts of money in escrow accounts so that they can obtain "letters of credit" guaranteeing performance. All of this is, alas, the regulators' way of attempting to destroy those whom they cannot regulate. IMHO, the USF is outmoded and should be disbanded. --Brett Glass

6 9

NANOG 62 - Baltimore - Call For Presentations is Open!
by Greg Dendy 16 Jul '14

16 Jul '14

NANOG Community- Thanks for helping to make NANOG 61 in Bellevue such a smashing success as the most attended meeting ever, on the 20th anniversary of the first meeting! NANOG will hold its 62nd meeting in Baltimore, MD on October 6-8, 2014, hosted by EdgeConneX. The NANOG Program Committee is now seeking proposals for presentations, panels, tutorials, tracks sessions, and keynote materials for the NANOG 62 program. We invite presentations highlighting issues relating to technology already deployed or soon-to-be deployed in the Internet, . Vendors are encouraged to work with operators to present real-world deployment experiences with the vendor's products and interoperability. Key dates to track if you wish to submit a presentation: * Presentation Abstracts and Draft Slides Due: August 4, 2014 * Topic List Posted: August 18, 2014 * Slides Due: September 1, 2014 * Agenda Published: September 15, 2014 NANOG 62 submissions are welcome on the P<http://pc.nanog.org/>rogram Committee Site<http://pc.nanog.org/> or email me if you have questions. Looking forward to seeing everyone in Baltimore! Thanks, Greg Dendy Chair, Program Committee North American Network Operator Group (NANOG)

1 1

Re: Net Neutrality...
by Brett Glass 15 Jul '14

15 Jul '14

At 02:16 PM 7/15/2014, Joly MacFie wrote: >And, in my experience, one needs around double or more of the listed bandwidth for a robust streaming connection. This is only true if the connection is of poor quality and dropped packets lead to regular 50% cuts in the data rate. Most users (and the FCC!) do not understand that, due to the Van Jacobson AIMD algorithm, quality matters far more than quantity when a service is delivered via TCP. --Brett Glass

1 0