$CheapTransitProvider > $EyeballNetwork's because there is so much
Here is a quote I made in the other thread around the same time you were sending this: "I also think the practice of paying an intermediary ISP a per Mbps rate in order to get to a last mile ISP over a settlement free agreement is also a bit disingenuous in cases where the amount of traffic is sufficient enough to fill multiple links. Theoretically there are many times where the intermediary ISP can hand off the traffic to a last mile ISP in exactly the same building they received it in so they have very few of the costs of actually delivering the traffic yet are the only party receiving money from the content provider for delivery. This arrangement makes sense when the traffic to the last mile ISP is a percentage of one link but after enough links are involved the intermediary ISP is serving no real other purpose than as a loophole used to circumvent paid peering fees (right or wrong)." I think we are in agreement that $EyeballNetwork's customers pay it for internet access and $ContentProvider should pay for their own pipes. But where we diverge is with $CheapTransitProvider. At least for the purpose of traffic following the path of $ContentProvider traffic involved the only real purpose of the relationship with $CheapTransitProvider is a loophole to get around paying $EyeballNetwork. They are able to charge ridiculously low delivery prices because traffic is only on their network for just long enough to say it touched and should now be considered settlement free. It's little more than a cheap trick and it makes them sort of the Cash4Gold of the Internet. I can completely understand why $EyeballNetwork would tell $CheapTransitProvider they no longer choose to have a settlement free agreement and they must buy future ports. On Sun, Apr 27, 2014 at 11:52 PM, Hugo Slabbert <hslabbert@stargate.ca>wrote:
Apologies that I dropped offlist as I was out for the day. I think the bulk of my thoughts on this have already been covered by others since, including e.g. Matt's poor grandmother and her phone dilemma in the "What Net Neutrality should and should not cover" thread.
Basically I think we're on the same page for the most part, with maybe some misunderstandings between us.
I covered this scenario in more detail in my post "What Net Neutrality should and should not cover" but if you expand on the assumption that paying for an internet connection also pays for the direct connection of every party who you exchange traffic with then you have a scenario where only half the people connected to the Internet should have to pay at all for their connection because any scenario where people simply buy their own pipe would be considered "double billing".
I don't think anyone on the Netflix^H^H^H^H^H^H $ContentProvider side of this was saying that $ContentProvider should get everything handed to them on a silver platter. $ContentProvider pays for transit sufficient to handle the traffic that their customers request. $EyeballNetwork's customers pay it for internet access, i.e. to deliver the content that they request, e.g. from $ContentProvider. That covers both directions here. Links between $ContentProvider's transit provider and $EyeballNetwork were getting congested, and $EyeballNetwork refuses to upgrade capacity. Where we were getting into the double-dip was $EyeballNetwork saying to $ContentProvide: "Hey, we know you already pay for transit, but you're gonna have to pay us as well if you want us to actually accept the traffic our customers requested".
The alternate arrangement between $ContentProvider and $EyeballNetwork seems to be private peering, where again it would seem to be fair for each side to bring the needed transport and ports to peering points. In recent history, though, it seems that $EyeballNetwork came out ahead in that agreement somehow. Now, Tore brought up a good point on paid peering in cases where e.g. $EyeballNetwork is already exchanging traffic with $ContentProvider through existing peering or below their CDR on existing transit, and indeed it seems that was the case for $EyeballNetwork via peering with $CheapTransitProvider that $ContentProvider was using. But it seems that $EyeballNetwork was having a pissing match with $CheapTransitProvider and refusing to upgrade ports.
"Okay", says $ContentProvider. "How about we just peer directly." "Sounds great," says $EyeballNetwork. "Since we have to allocate capacity for this discrete from our existing peering capacity, you'll need to foot the bill for that." "Huh?" says $ContentProvider. "This could have been fixed by you increasing your peering capacity to match the traffic volume your users are requesting, but you didn't want to do that because of your tiff with $CheapTransitProvider. Tell me again why we're paying for your side of this *in addition* to our own when we're only going this route because of a decision *you* made?" "Because you need to reach our customers, and we're the only path to them, so we have leverage." *blank stare* "So you're willing to give your customers crappy service because your customers don't have alternate options and you think we need this more than you do?" "That's a possibility." "I hate you." "I know; sign here please."
But, again, this is outside looking in. For now, I'll pick up a copy of Bill Norton's Internet Peering book as per Bob's suggestion, for some light Sunday night reading.
Cheers,
-- Hugo
________________________________ From: Rick Astley <jnanog@gmail.com> Sent: Sunday, April 27, 2014 8:45 AM To: Hugo Slabbert Cc: nanog@nanog.org Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post
If it were through a switch at the exchange it would be on each of them to individually upgrade their capacity to it but at the capacities they are at it they are beyond what would make sense financially to go over an exchange switch so they would connect directly instead. It's likely more along the lines of needing several 100G ports as Netflix is over 30% of peak usage traffic in North America:
"Netflix (31.6%) holds its ground as the leading downstream application in North America and together with YouTube (18.6%) accounts for over 50% of downstream traffic on fixed networks." (source https://www.sandvine.com/trends/global-internet-phenomena/ )
That amount of data is massive scale. I don't see it as double dipping because each party is buying the pipe they are using. I am buying a 15Mbps pipe to my home but just because we are communicating over the Internet doesn't mean the money I am paying covers the cost of your connection too. You must still buy your own pipe in the same way Netflix would. I covered this scenario in more detail in my post "What Net Neutrality should and should not cover" but if you expand on the assumption that paying for an internet connection also pays for the direct connection of every party who you exchange traffic with then you have a scenario where only half the people connected to the Internet should have to pay at all for their connection because any scenario where people simply buy their own pipe would be considered "double billing".
The cost for residential broadband is high enough in the US without a policy like that in place. If there is one policy that would keep poor families from being able to afford broadband it would be that one.
On Sun, Apr 27, 2014 at 2:58 AM, Hugo Slabbert <hslabbert@stargate.ca <mailto:hslabbert@stargate.ca>> wrote:
...but if that point of congestion is the links between Netflix and Comcast...
Which, from the outside, does appear to have been the case.
...then Netflix would be on the hook to ensure they have enough capacity to Comcast to get the data at least gets TO the Comcast network.
Which I don't believe was a problem? Again, outside looking in, but the appearances seemed to indicate that Comcast was refusing to upgrade capacity/ports, whereas I didn't see anything indicating that Netflix was doing the same. So:
I have gear; you have gear. I upgrade or add ports on my side; you upgrade or add ports on your side.
The argument at hand is if Comcast permitted to charge them for the links to get to their network or should they be free/settlement free. I think it should be OK to charge for those links as long as its a fair market rate and the price doesn't basically amount to extortion.
Are we talking here about transport between Netflix's POPs and Comcast's? I definitely don't expect Comcast to foot the bill for transport between the two, and if Netflix was asking for that I'm with you that would be out of line. If there are existing exchange points, though, would it not be reasonable to expect each side to up their capacity at those points?
Once that traffic is given directly to comcast no other party receives payment for delivering it so there is no double billing.
The "double-dip" reference was to charging both the content provider and the ISP's own customer to deliver the same bits. If the traffic from Netflix was via Netflix's transit provider and Comcast then again was looking to bill Netflix to accept the traffic, we'd hit double billing.
I guess that's the question here: If additional transport directly been POPs of the two parties was needed, somebody has to pay for the links. Releases around the deal seemed to indicate that the peering was happening at IXs (haven't checked this thoroughly), so at that point it would seem reasonable for each party to handle their own capacity to the peering points and call it even. No?
-- Hugo
________________________________ From: Rick Astley <jnanog@gmail.com<mailto:jnanog@gmail.com>> Sent: Saturday, April 26, 2014 11:23 PM To: Hugo Slabbert Cc: nanog@nanog.org<mailto:nanog@nanog.org> Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post
How is this *not* Comcast's problem? If my users are requesting more traffic than I banked on, how is it not my responsibility to ensure I have capacity to handle that? I have gear; you have gear. I upgrade or add ports on my side; you upgrade or add ports on your side. Am I missing something?
Sort of yes, it's Comcasts problem to upgrade subscriber lines but if that point of congestion is the links between Netflix and Comcast then Netflix would be on the hook to ensure they have enough capacity to Comcast to get the data at least gets TO the Comcast network. The argument at hand is if Comcast permitted to charge them for the links to get to their network or should they be free/settlement free. I think it should be OK to charge for those links as long as its a fair market rate and the price doesn't basically amount to extortion. Sadly the numbers are not public so I couldn't tell you one way or the other aside from I disagree with the position Netflix seems to be taking that they simply must be free. Once that traffic is given directly to comcast no other party receives payment for delivering it so there is no double billing.
This diagram best describes the relationship (ignoring pricing): http://www.digitalsociety.org/files/gou/free-and-paid-peering.png
"Content provider" would be Netflix and Comcast would be Broadband ISP 1.
On Sun, Apr 27, 2014 at 1:56 AM, Hugo Slabbert <hslabbert@stargate.ca <mailto:hslabbert@stargate.ca><mailto:hslabbert@stargate.ca<mailto: hslabbert@stargate.ca>>> wrote: Okay, I'm not as seasoned as a big chunk of this list, but please correct me if I'm wrong in finding this article a crock of crap. With Comcast/Netflix being in the mix and by association Cogent in the background of that there's obviously room for some heated opinions, but here goes anyway...
A long, long time ago when the Internet was young and few, if any had thought to make a profit off it, an unofficial system developed among the network providers who carried the traffic: You carry my traffic and I'll carry yours and we don't need money to change hands. This system has collapsed under modern realities.
I wasn't aware that settlement-free peering had "collapsed". Not saying it's the "only way", but "she ain't dead yet".
Seltzer uses that to set up balanced ratios as the secret sauce that makes settlement-free peering viable: "The old system made sense when the amount of traffic each network was sending to the other was roughly equivalent."
...and since Netflix sends Comcast more than it gets, therefor Netflix needs to buck up: "Of course Netflix should pay network providers in order to get the huge amounts of bandwidth they require in order to reach their customers with sufficient quality."
But this isn't talking about transit; this is about Comcast as an edge network in this context and Netflix as a content provider sending to Comcast users the traffic that they requested. Is there really anything more nuanced here than:
1. Comcast sells connectivity to their end users and sizes their network according to an oversubscription ratio they're happy with. (Nothing wrong here; oversubscription is a fact of life). 2. Bandwidth-heavy applications like Netflix enter the market. 3. Comcast's customers start using these bandwidth-heavy applications and suck in more data than Comcast was betting on. 4. Comcast has to upgrade connectivity, e.g. at peering points with the heavy inbound traffic sources, accordingly in order to satisfy their customers' usage.
How is this *not* Comcast's problem? If my users are requesting more traffic than I banked on, how is it not my responsibility to ensure I have capacity to handle that? I have gear; you have gear. I upgrade or add ports on my side; you upgrade or add ports on your side. Am I missing something?
Overall it seems like a bad (and very public) precedent & shift towards double dipping, and the pay-for-play bits in the bastardized "Open Internet" rules don't help on that front. Now, Comcast is free to leverage their customers as bargaining chips to try to extract payments, and Randy's line of encouraging his competitors to do this sort thing seems fitting here. Basically this doesn't harm me directly at this point. Considering the lack of broadband options for large parts of the US, though, it seems that end users are getting the short end of the stick without any real recourse while that plays out.
-- Hugo
________________________________________ From: NANOG <nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org
<mailto:nanog-bounces@nanog.org<mailto:nanog-bounces@nanog.org>>> on behalf of Larry Sheldon <LarrySheldon@cox.net<mailto:LarrySheldon@cox.net <mailto:LarrySheldon@cox.net<mailto:LarrySheldon@cox.net>>> Sent: Saturday, April 26, 2014 4:58 PM To: nanog@nanog.org<mailto:nanog@nanog.org><mailto:nanog@nanog.org<mailto: nanog@nanog.org>> Subject: Re: The FCC is planning new net neutrality rules. And they could enshrine pay-for-play. - The Washington Post
h/t Suresh Ramasubramanian
FCC throws in the towel on net neutrality
http://www.zdnet.com/fcc-throws-in-the-towel-on-net-neutrality-7000028770/
Forward! On to the next windmill, Sancho! -- Requiescas in pace o email Two identifying characteristics of System Administrators: Ex turpi causa non oritur actio Infallibility, and the ability to learn from their mistakes. (Adapted from Stephen Pinker)