On 1 Oct 2021, at 16:46, Mark Tinka <mark@tinka.africa> wrote:



...

On 10/1/21 16:19, Blake Hudson wrote:

I'll never understand over how ISPs see content providers as the enemy (or a rival). The content is why ISPs have customers. Don't get upset when your customer uses the service that you sold them (in a way that is precisely in accordance with the expected usage)!

It's because infrastructure (that's us, the network operators), still don't get it.

We are no longer front & centre in the eyes of our users. They see us as an impediment... providers they must buy costly megabytes of mobile data from, providers they must call to fix broken fibre, providers they must shout at when a single CGN IPv4 address they sit behind breaks their Netflix, and so on and so on.

Users only care about the service they use their mobile phone, tablet, console or laptop for. They don't care how many customers their ISP has, whether the ISP is a small mom & pop or some global behemoth, or whether the ISP's CEO is was on the cover of TIME magazine last week.

As my American friend used to say, "They just want their MTV".

In the late 90's and early 2000's, when content folk wanted to work with us, infrastructure folk, to grow their businesses, we just saw easy, free money to tax toward our shiny new Lamborghinis and beach side holiday villas. Well, guess whom we are now begging for seats on their submarine cable build projects, community funding programs, and caches to be installed in our not-so-huge data centres, all for free?

The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own global backbones is because of this nonsense that SK Broadband is trying to pull with Netflix. At some point, the content folk will get fed up, and go build it themselves. What an opportunity infrastructure cost itself!

Akamai have also quietly been building their own backbone. Wonder why.

No doubt Netflix have someone either thinking about the same, or putting a plan into motion.

The bad news now, is, there are plenty of many, small, local and regional ISP's who are willing to do whatever it takes to work with the content providers. All that's required is some network, a half-decent data centre and an exchange point. Gone are the days where customers clamored to sign up with Big Telco.

If anyone wonders why "infrastructure is dead", well, this is why.

21 years later, and we still don't get it! No wonder the mobile companies are watching their slow death, from the rosy days of billions from basic SMS, to the perils of 5G investments for diddly return.

Wake me up when all this is over. I'll be in my wine stupor until then. I couldn’t agree more.

Mark.

Cheers. Darwin-.

In the old days, postal services used to charge the recipient of a letter to deliver the letter. Then stamps were invented, and postal services charged the sender of the letter, and the recipent got free delivery. Now there is free-shipping, and pre-paid return envelopes for DVDs. Of course, the shipping isn't really "Free." Its built into the cost of goods sold. There is no universal, fixed, unchangable way of allocating business costs.

I think instances where the end ISP is peered directly with Netflix and demands more money is not valid at all. That should be normal cost of doing business to increase capacity as the consumer demand grows. The topic of interest is instances where the ISP is not directly peered with Netflix and uses upstream providers and those providers are trying to make content providers absorb the cost of increasing peering capacity for services that traverse their infrastructure. One could make the argument that Tier1's should never be the choke point as they should be keeping up with the times and be proactively increasing capacity. One could also note that it's 2021 and Cogent and Hurricane Electric are still not peered. On Fri, Oct 1, 2021 at 10:47 AM Blake Hudson <blake@ispn.net> wrote:

On 10/1/2021 11:23 AM, Sean Donelan wrote:

...

In the old days, postal services used to charge the recipient of a letter to deliver the letter. Then stamps were invented, and postal services charged the sender of the letter, and the recipent got free delivery.

Now there is free-shipping, and pre-paid return envelopes for DVDs.

Of course, the shipping isn't really "Free." Its built into the cost of goods sold.

There is no universal, fixed, unchangable way of allocating business costs.

True. But when the sender has already paid the stamp to their courier to deliver the bits on their leg of the journey, and the recipient has already paid a stamp to deliver the bits on their leg of the journey, what case does the recipient's courier have to demand additional payment from the sender (lest the package get lost)? The stamp has already been paid. TWICE! Withholding service until additional payments are made just smells like extortion.

The bottom line problem is that we have allowed vertical integration to allow the natural monopoly that exists in last mile infrastructure in most locations to be leveraged into an effective full-stack monopoly for those same players. Lack of competition in the last-mile/eyeball space has allowed larger monopoly eyeball providers to leverage double-payment for the same traffic, extorting eyeballs that have no alternative to pay them to deliver the content while simultaneously turning around and extorting the content providers themselves to be allowed to reach “their customers”. While monopoly $CABLECO/$DSLTELCO/$GPONPROVIDER can generally get away making access problematic for a handful of content providers for short periods of time, no content provider can really absorb the losses associated with allowing that situation to continue, thus giving the content providers leverage. Unfortunately for the content providers, newer laws and lower costs to provide higher bandwidth are making WISPs a viable competitor in both rural and metropolitan settings. Consumers are catching on to this and telling the more traditional ISPs that they now have a choice and the ISPs will have to up their game to keep their business, so finally progress is being made, which is removing that leverage from the previously monopoly providers on both the consumer and the content provider side of that equation. Obviously, ISPs don’t like this and the monopoly ones being not in the communications business, but rather operating as a law firm with some switching infrastructure will attempt to use legislation, PUC, and courts as weapons to try and preserve the status quo. I’m hopeful that the Korean courts will see SK’s shakedown for what it is and toss it in summary judgment (or whatever the Korean equivalent is) with costs and attorneys’ fees awarded to Netflix. I’m hoping that this will also send a clear message to other ISPs contemplating such extortive behavior. OTOH, I admit I will watch in amusement if SK’s customers trying to play Netflix videos are only able to watch in 480p after viewing a brief video explaining that the video they are about to watch is degraded courtesy of SK’s business practices (obviously with appropriate details). Owen

On Oct 1, 2021, at 11:24 , Joshua Pool via NANOG <nanog@nanog.org> wrote:

I think instances where the end ISP is peered directly with Netflix and demands more money is not valid at all. That should be normal cost of doing business to increase capacity as the consumer demand grows. The topic of interest is instances where the ISP is not directly peered with Netflix and uses upstream providers and those providers are trying to make content providers absorb the cost of increasing peering capacity for services that traverse their infrastructure. One could make the argument that Tier1's should never be the choke point as they should be keeping up with the times and be proactively increasing capacity. One could also note that it's 2021 and Cogent and Hurricane Electric are still not peered.

On Fri, Oct 1, 2021 at 10:47 AM Blake Hudson <blake@ispn.net <mailto:blake@ispn.net>> wrote:

On 10/1/2021 11:23 AM, Sean Donelan wrote:

...

In the old days, postal services used to charge the recipient of a letter to deliver the letter. Then stamps were invented, and postal services charged the sender of the letter, and the recipent got free delivery.

Now there is free-shipping, and pre-paid return envelopes for DVDs.

Of course, the shipping isn't really "Free." Its built into the cost of goods sold.

There is no universal, fixed, unchangable way of allocating business costs.

True. But when the sender has already paid the stamp to their courier to deliver the bits on their leg of the journey, and the recipient has already paid a stamp to deliver the bits on their leg of the journey, what case does the recipient's courier have to demand additional payment from the sender (lest the package get lost)? The stamp has already been paid. TWICE! Withholding service until additional payments are made just smells like extortion.

On 01/10/2021 17:05, Laura Smith via NANOG wrote:

- $certain_large_cdn publishes routes on route server ? Nope.

Many (most?) route servers provide little control over who your routes are advertised toward. This can be fun where DDoS is concerned. I've used some that did have deny-list controls for ASNs, fail to consistently apply those rules. Again, that was a 'fun' surprise.

- $certain_large_cdn willing to establish direct peering session ? Nope.

There is a non-zero cost to peering. Many CDNs are happy to send cache boxes/setup peering sessions for small peers, but their definition of "small" will no doubt vary between CDNs, based on their perspective & business costs. Some networks may fall well below each individual network's thresholds, and indeed some networks may have different thresholds between countries. It's never as simple as "why don't you peer???????" Regards, -- Tom

On Fri, Oct 1, 2021 at 9:08 AM Laura Smith via NANOG <nanog@nanog.org> wrote:

...

The bad news now, is, there are plenty of many, small, local and regional ISP's who are willing to do whatever it takes to work with the content providers. All that's required is some network, a half-decent data centre and an exchange point. Gone are the days where customers clamored to sign up with Big Telco.

Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps you could answer me this riddle.....

- PoP in one of your "half-decent data centres" ... tick. - Connnection to one of your "exchange point" ... tick. - $certain_large_cdn present on said "exchange point" ... tick.

And yet .....

- $certain_large_cdn publishes routes on route server ? Nope. - $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where the large networks see it beneath them to peer with (or publish routes for the benefit of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off their nose to spite their face ?

Having worked on building out a global content delivery network, I can hopefully shed some light here. Let's imagine a scenario where you have a CDN node attached to a large peering location. At that peering location, you have Nx100G direct peering links to large eyeball networks, and Nx10G direct peering links to midsized eyeball networks. You have a historical connection into the peering switch as well, which you've kept around and upgraded as time went by. Life is generally good. You've got your own backbone between your locations, and you capacity engineer your infrastructure so that if your peering sessions go down at that site, users are redirected to the next closest CDN node. Then one day, Bad Juju(tm) happens. The router that your Nx100G links with Large Eyeball Network A terminate on goes down, doesn't matter if it's on their end or your end, those direct sessions go down. But instead of your traffic failing over to site B instead, it continues flowing--but now it's going through the route server sessions, across the now grossly-undersized connection into the public peering switch, and *nobody* is happy about that. You tried putting import and export lists into the route server, but they don't seem to be taking effect, and the phone is ringing off the hook, so you simply stop exporting routes into the route server, and suddenly traffic is now failing over to site B for all the eyeball requests from Large Eyeball Network A, and you get to explain to your boss how those sessions with the route servers caused the failover plan to fail badly. Decision is made that route server peering sessions are too far removed from direct control to be safe, and the only way to effectively prevent this scenario from happening again is to stop announcing routes to the route servers. You're a smaller network; you're caught up in this, because you'd been getting routes from the route server, and now you're not. You request direct peering; unfortunately, the ports on the existing line cards on the CDN edge routers are full, and it's going to be months before the next capacity upgrade is planned; and those line cards are all 100G cards, with no 10G ports planned. It's decided that letting that traffic get handled by the transit ports is cheaper than trying to get some additional 10G peering port capacity at the site. And so, your request for direct peering is denied; there's just no place to plug you in on the router on the other side, unless you can drum up enough traffic to justify a 100G peering link. Establishing direct peering across the peering fabric becomes the only possible point of commonality, then; but that requires adding explicit neighbor configuration into the router, unlike the route-server mediated prefix exchange; and now you're still running into limitations on the edge routers--though in this case it's not physical port limitations, it's neighbor adjacency limits on the routers. The more neighbors you configure on an edge router, the longer the configs are, and depending on how the neighbors are established, the more processing power and memory it takes on the router. At a certain point, there's just not enough to go around, and the CDN makes the decision to not add any new BGP neighbors at that site. A few years down the line, when hardware is added/upgraded, they may revisit that decision, or it may persist, even after the upgrade, due simply to corporate inertia--nobody goes back to revisit the decision, and see if it still makes sense or not. Sometimes it's just a case of limited humans; without good automation tools in place, adding BGP adjacencies takes a ticket being created in a request system, and then another more highly skilled human going into the router to configure the neighbor. If you've got a limited number of skilled humans available (and every company has fewer of those than they really need, but that's always going to be the case), the decision generally gets made to focus their attention on efforts that bring in more revenue--and configuring public peering sessions generally doesn't fall into that bucket. :( And sometimes, financial considerations come into play; the CDN network has a default upstream transit provider to catch the traffic that isn't handled by direct peering, and the transit provider gives a really nice price break if the traffic makes it up into the next tier; so, CDN that's just *below* the cheaper tier looks for traffic it can shunt onto transit, to nudge the volume over the line, to qualify for the lower per-mbit price point--and the easiest traffic to use to nudge them over is the smaller public peering neighbor traffic. And so, your request for direct peering is looked at, and your traffic volume is deemed to be the perfect fit to help fill out the transit commitment needed to bump them into the cheaper tier. In a perfect world, these would all be solvable issues, and exchanging traffic directly with the CDN across the public switches would make sense for everybody involved. Unfortunately, we live in an imperfect world, with route server filters that don't react quickly and with the level of precision needed, we have limitations on how many BGP adjacencies routers can handle, we have limited port counts on line cards, we have limited numbers of humans to make changes to router configurations, and we have economic pressures to meet volume thresholds to get better pricing that all end up creating the scenario you find yourself in; a completely reasonable request for direct peering nonetheless is ignored, or denied, due to one or more of the reasons I've discussed. Thanks! Matt

On 10/1/21 18:05, Laura Smith wrote:

Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps you could answer me this riddle.....

- PoP in one of your "half-decent data centres" ... tick. - Connnection to one of your "exchange point" ... tick. - $certain_large_cdn present on said "exchange point" ... tick.

And yet .....

- $certain_large_cdn publishes routes on route server ? Nope. - $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where the large networks see it beneath them to peer with (or publish routes for the benefit of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off their nose to spite their face ?

Yes, this is a rather painful one, and it has been on the rise in the last few years as the major content folk struggle to keep up with peering requirements, and everything that goes along with maintaining those relationships. They also seem to have a number of complex operational requirements between their own backbones, their own PoP's, partner PoP's, transit links, e.t.c. There is no easy answer for this one, apart from doing the leg-work to find out who the best person at the other end to speak to is. I'd recommend working with the data centre and exchange point operators to make meaningful introductions. You may not end up where you want, but you certainly increase your chances of doing so. Mark.

On 10/10/21 21:08, Doug Barton wrote:

Except that Facebook, Microsoft, and Amazon all caved to SK's demands:

"The popularity of the hit series "Squid Game" and other offerings have underscored Netflix's status as the country's second-largest data traffic generator after Google's YouTube, but the two are the only ones to not pay network usage fees, which other content providers such as Amazon, Apple and Facebook are paying, SK said."

Which has emboldened SK to go after the bigger fish.

Prior to the popularity of "House Of Cards", Netflix would have bent over and taken it without any lube. Heck, they signed away plenty of rights around the world to several networks for "House Of Cards", purely because they didn't know how well their own in-house production would succeed. Fast-forward, it's 2021 now. Other players in BigContent that haven't yet found their leverage, will do.

One incentive I haven't seen anyone mention is that ISPs don't want to charge customers what it really costs to provide them access. If you're the only one in your market that is doing that, no one is going to sign up because your pricing would be so far out of line with your competition.

Isn't this the curse of a service people consider to be a basic utility for life to occur? Unlike water and power, nearly anyone can start an ISP, and further feed the race to the bottom.

Given that issue, I have some sympathy for eyeball networks wanting to charge content providers for the increased capacity that is needed to bring in their content. The cost would be passed on to the content provider's customers...

But eyeballs are already paying you a monthly fee for 100Mbps of service (for example). So they should pay a surcharge, over-and-above that, that determines how they can use that 100Mbps? Seems overly odd, to me.

(in the same way that corporations don't pay taxes, their customers do),...

Many a company pays corporate tax, which is separate from the income tax they pay for compensation to their staff. Of course, YMMV depending on where you live.

so the people on that ISP who are creating the increased demand would be (indirectly) paying for the increased capacity. That's actually fairer for the other customers who aren't Netflix subscribers.

The reason that Netflix doesn't want to do it is the same reason that ISPs don't want to charge their customers what it really costs to provide them access.

So what rat hole does this lead us down into? People who want to stream Youtube should pay their ISP for that? People who want to spend unmentionable hours on Linkedin should be their ISP for that? People who want to gawk over Samsung's web site because they love it so much, should pay their ISP for that? Hey, maybe you're right. Maybe that's the model that is needed. After all, when we go to a rave, the entry fee is just the entry fee. You still need to fork out more cash to actually buy drinks, food or engage in some kind of entertainment that may be taking place inside that walled garden you paid a cover charge to be a part of. I don't know... Mark.

----- On Oct 10, 2021, at 2:42 PM, Doug Barton dougb@dougbarton.us wrote: Hi,

And for the record, not only have I never worked for an ISP, I was saying all the way back in the late '90s that the oversubscription business model (which almost always includes punishing users who actually use their bandwidth) is inherently unfair to the customers, and when the Internet becomes more pervasive in daily life will come back to bite them in the ass. I was laughed at for being hopelessly naive, not understanding how the bandwidth business works, etc.

I have worked for ISPs. And I remember the late 90s. Bandwidth was $35/mbit on average, at least for the outfit where I was. Consumers paid roughly $40 for their DSL connections, which at the time went up to 2Mbit depending on the age of the copper and distance to the DSLAM. Consumer connections were oversubscribed, on average, 1:35 to 1:50. B2B connections got a better deal, 1:10 to 1:15. It was simply not feasible to offer 1:1 bandwidth and still make a profit, unless you're charging fees the average consumer cannot afford. Especially considering that the average user doesn't even need or use that much bandwidth. It's a recurring discussion. People demand more bandwidth without considering whether or not they need it. End-users, business subs, and host-owners at large enterprises where I worked. The last ones are the funniest: entire racks using no more than 100mbit/s and hostowners are demanding an upgrade from 10G to 25G bEcaUse LaTenCy. The last consumer ISP I worked at had a very small subset of users that really needed bandwidth: the "download dudes" who were 24/7 leeching news servers, and the inevitable gamers that complained about the latency due to the links being full as a result of said leechers. In that case, a carefully implemented shaping of tcp/119 did the trick. Thanks, Sabri

On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka <mark@tinka.africa> wrote:

However, in an era where content is making a push to get as close to the eyeballs as possible, kit getting cheaper and faster because of merchant silicon, and abundance of aggregated capacity at exchange points, can we leverage the shorter, faster links to change the model?

Mark.

I think it would be absolutely *stunning* for content providers to turn the model on its head; use a bittorrent like model for caching and serving content out of subscribers homes at recalcitrant ISPs, so that data doesn't come from outside, it comes out of the mesh within the eyeball network, with no clear place for the ISP to stick a $$$ bill to. Imagine you've got a movie; you slice it into 1,000 encrypted chunks; you make part of your license agreement for customers a requirement that they will allow you to use up to 20GB of disk space on their computer and to serve up data chunks into the network in return for a slightly cheaper monthly subscription cost to your service. You put 1 slice of that movie on each of 1,000 customers in a network; then you replicate that across the next thousand customers, and again for the next thousand, until you've got enough replicas of each shard to handle a particular household going offline. Your library is still safe from piracy, no household has more than 1/1000th of a movie locally, and they don't have the key to decrypt it anyhow; but they've got 1/1000th of 4,0000 different movies, and when someone in that ISP wants to watch the movie, the chunks are being fetched from other households within the eyeball network. The content provider would have shard servers in region, able to serve up any missing shards that can't be fetched locally within the ISP--but the idea would be that as the number of subscribers within an ISP goes up, instead of the ISP seeing a large, single-point-source increase in traffic, what they see is an overall increase in east-west traffic among their users. Because the "serving of shards to others" happens primarily while the user is actively streaming content, you have a natural bell curve; during peak streaming times, you have more nodes active to serve up shards, handling the increased demand; at lower demand times, when fewer people are active, and there's fewer home-nodes to serve shards, the content network's shard servers can pick up the slack...but that'll generally only happen during lower traffic times, when the traffic won't be competing and potentially causing pain for the ISP. Really, it seems like a win-win scenario. I'm confident we'll see a content network come out with a model like this within the next 5 years, at which point the notion of blackmailing content networks for additional $$$s will be a moot point, because the content will be distributed and embedded within every major eyeball network already, whether they like it or not, on their customer's devices. Let's check back in 2026, and see if someone's become fantastically successful doing this or not. ;) Thanks! Matt

On Oct 11, 2021, at 13:57 , Matthew Walster <matthew@walster.org> wrote:

On Mon, 11 Oct 2021 at 21:05, Matthew Petach <mpetach@netflight.com <mailto:mpetach@netflight.com>> wrote: I think it would be absolutely *stunning* for content providers to turn the model on its head; use a bittorrent like model for caching and serving content out of subscribers homes at recalcitrant ISPs, so that data doesn't come from outside, it comes out of the mesh within the eyeball network, with no clear place for the ISP to stick a $$$ bill to.

Ignoring for the moment that P2P is inherently difficult to stream with (you're usually downloading chunks in parallel, and with devices like Smart TVs etc you don't really have the storage to do so anyway) there's also the problem that things like BitTorrent don't know network topology and therefore only really increases the cross-sectional bandwidth required.

A 4K 2 hour movie is about 40GB. Most modern smart TVs around 32GB of RAM and can probably devote about 20GB of that to buffering a stream, so yeah, that should actually be doable. While torrent-like distribution isn’t particularly good for the eyeball provider, it can be good for getting content to eyeballs under some circumstances regardless of how bad it is for said network. Unfortunately, it’s not good at knowing how bad it’s being for the network and it’s also not good at detecting the circumstances when it’s good for the end user or not.

Not to mention that it has been tried before, and didn't work then either.

Yep. Owen

On 10/12/21 14:20, Jason Iannone wrote:

Isn't this a problem with legacy peering agreements in today's internet? The same thing happened between Netflix, Level3, and Verizon a few years ago. The legacy concept of settlement-free peering is based on traffic forwarding parity. If what I forward to you roughly matches what you forward to me, we can agree that we have no reason to charge each other for access. The concept works fine when content and eyeballs are evenly distributed between providers. This doesn't work in today's divergent content and eyeball networks.

If Netflix agreed to settlement-free peering under the legacy definition, then as far as the letter of the law goes, Netflix is in the wrong.

Indeed. Traffic ratios to determine peering partners(hips) is, I think, archaic, and a cop out for not having to deal with peering requests. There are many networks with whom peering would add value, despite not matching traffic ratios. As you say, BigContent are such networks. Specifically for us, geographic network scope is the most important, as I don't want to help you build your backbone on the back of mine, for free. There may be some volume requirements, but certainly not made on the basis of directional ratios. Mark.

On 10/11/21 22:57, Matthew Walster wrote:

Ignoring for the moment that P2P is inherently difficult to stream with (you're usually downloading chunks in parallel, and with devices like Smart TVs etc you don't really have the storage to do so anyway) there's also the problem that things like BitTorrent don't know network topology and therefore only really increases the cross-sectional bandwidth required.

Not to mention that it has been tried before, and didn't work then either.

Yeah, and people also want to click a title and start watching immediately. Someone can correct me if I'm wrong, but the way I know BitTorrent to work is the file is downloaded to disk, unarchived and then listed as ready to watch. It also assumes the device has all the necessary apps and codecs needed to render the file. On the other hand, BitTorrent could just make an Apple TV/PS4/PS5/Xbox/whatever-device-you-use app as well. But I doubt that will work, unless someone can think up a clever way to modify BitTorrent to suit today's network architectures. Mark.

On Oct 11, 2021, at 13:05 , Matthew Petach <mpetach@netflight.com> wrote:

On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka <mark@tinka.africa> wrote: However, in an era where content is making a push to get as close to the eyeballs as possible, kit getting cheaper and faster because of merchant silicon, and abundance of aggregated capacity at exchange points, can we leverage the shorter, faster links to change the model?

Mark.

I think it would be absolutely *stunning* for content providers to turn the model on its head; use a bittorrent like model for caching and serving content out of subscribers homes at recalcitrant ISPs, so that data doesn't come from outside, it comes out of the mesh within the eyeball network, with no clear place for the ISP to stick a $$$ bill to.

I worked for a company that tried this model several years back. It did not go well. Users were not happy. Eyeball ISPs were very not happy (and started actively mitigating users who participated through a variety of nefarious means), and content providers got a bit hot under the collar about it (the experiment was a content aggregator for lack of a better term).

Imagine you've got a movie; you slice it into 1,000 encrypted chunks; you make part of your license agreement for customers a requirement that they will allow you to use up to 20GB of disk space on their computer and to serve up data chunks into the network in return for a slightly cheaper monthly subscription cost to your service. You put 1 slice of that movie on each of 1,000 customers in a network; then you replicate that across the next thousand customers, and again for the next thousand, until you've got enough replicas of each shard to handle a particular household going offline. Your library is still safe from piracy, no household has more than 1/1000th of a movie locally, and they don't have the key to decrypt it anyhow; but they've got 1/1000th of 4,0000 different movies, and when someone in that ISP wants to watch the movie, the chunks are being fetched from other households within the eyeball network. The content provider would have shard servers in region, able to serve up any missing shards that can't be fetched locally within the ISP--but the idea would be that as the number of subscribers within an ISP goes up, instead of the ISP seeing a large, single-point-source increase in traffic, what they see is an overall increase in east-west traffic among their users.

That’s a pretty close description of exactly what we did.

Because the "serving of shards to others" happens primarily while the user is actively streaming content, you have a natural bell curve; during peak streaming times, you have more nodes active to serve up shards, handling the increased demand; at lower demand times, when fewer people are active, and there's fewer home-nodes to serve shards, the content network's shard servers can pick up the slack...but that'll generally only happen during lower traffic times, when the traffic won't be competing and potentially causing pain for the ISP.

It’s a beautiful theory. We expected this to be the case, too. Turns out, not so much. There’s an awful lot of long tail out there that is poorly accounted for in this. Yes, it does help with some of the most popular content, but there are pitfalls there, too.

Really, it seems like a win-win scenario.

We thought so too, until we learned otherwise. Today, with more symmetrical connections and larger uplinks, it might be more feasible. Back then (Around 2005), it was a good idea on paper.

I'm confident we'll see a content network come out with a model like this within the next 5 years, at which point the notion of blackmailing content networks for additional $$$s will be a moot point, because the content will be distributed and embedded within every major eyeball network already, whether they like it or not, on their customer's devices.

You’d be surprised at the number of different ways eyeball providers have to denature such an effort. I’m sorry to rain on your parade, but it’s not a new idea. I was brought in as the operations guy once the software engineers realized that they needed someone who could, you know, keep stuff running and not just write code and test it in production. This model resulted in a frank and uncomfortable conversation between me and the developer of what was internally being called the “overlay network” code. We went through a number of different scenarios where he had made incorrect assumptions about how the internet worked, especially at the eyeball end and there were a dozen or so that we simply couldn’t find ways to work around. Networks are a bit different now and some content providers have people far more clever than I working on things like this, so who knows… It might succeed by 2026, but when we tried it in 2005, we couldn’t make it work well enough to avoid annoying anyone involved.

Let's check back in 2026, and see if someone's become fantastically successful doing this or not. ;)

I’ll look forward to it. Owen

I think it would be absolutely *stunning* for content providers to turn the model on its head; use a bittorrent like model for caching and serving content out of subscribers homes at recalcitrant ISPs, so that data doesn't come from outside, it comes out of the mesh within the eyeball network, with no clear place for the ISP to stick a $$$ bill to.

I'm familiar with some work and ideas that have gone into such a thing, and I'm personally very much against it for non-technical reasons. Given how far the law lags behind technology, the last thing anyone should be ok with is a 3rd party storing bits on ANYTHING in their house, or transmitting those bits from a network connection that is registered to them. On Mon, Oct 11, 2021 at 4:06 PM Matthew Petach <mpetach@netflight.com> wrote:

On Mon, Oct 11, 2021 at 1:01 AM Mark Tinka <mark@tinka.africa> wrote:

...

However, in an era where content is making a push to get as close to the eyeballs as possible, kit getting cheaper and faster because of merchant silicon, and abundance of aggregated capacity at exchange points, can we leverage the shorter, faster links to change the model?

Mark.

I think it would be absolutely *stunning* for content providers to turn the model on its head; use a bittorrent like model for caching and serving content out of subscribers homes at recalcitrant ISPs, so that data doesn't come from outside, it comes out of the mesh within the eyeball network, with no clear place for the ISP to stick a $$$ bill to.

Imagine you've got a movie; you slice it into 1,000 encrypted chunks; you make part of your license agreement for customers a requirement that they will allow you to use up to 20GB of disk space on their computer and to serve up data chunks into the network in return for a slightly cheaper monthly subscription cost to your service. You put 1 slice of that movie on each of 1,000 customers in a network; then you replicate that across the next thousand customers, and again for the next thousand, until you've got enough replicas of each shard to handle a particular household going offline. Your library is still safe from piracy, no household has more than 1/1000th of a movie locally, and they don't have the key to decrypt it anyhow; but they've got 1/1000th of 4,0000 different movies, and when someone in that ISP wants to watch the movie, the chunks are being fetched from other households within the eyeball network. The content provider would have shard servers in region, able to serve up any missing shards that can't be fetched locally within the ISP--but the idea would be that as the number of subscribers within an ISP goes up, instead of the ISP seeing a large, single-point-source increase in traffic, what they see is an overall increase in east-west traffic among their users.

Because the "serving of shards to others" happens primarily while the user is actively streaming content, you have a natural bell curve; during peak streaming times, you have more nodes active to serve up shards, handling the increased demand; at lower demand times, when fewer people are active, and there's fewer home-nodes to serve shards, the content network's shard servers can pick up the slack...but that'll generally only happen during lower traffic times, when the traffic won't be competing and potentially causing pain for the ISP.

Really, it seems like a win-win scenario.

I'm confident we'll see a content network come out with a model like this within the next 5 years, at which point the notion of blackmailing content networks for additional $$$s will be a moot point, because the content will be distributed and embedded within every major eyeball network already, whether they like it or not, on their customer's devices.

Let's check back in 2026, and see if someone's become fantastically successful doing this or not. ;)

Thanks!

Matt

On 10/12/21 9:15 PM, Matthew Petach wrote:

So, I take it you steadfastly block *all* cookies from being stored or transmitted from your browser at home? ------------------------------------------------------------------\

I used to when Firefox had the "ask me every time" for cookies. They got rid of that, so now I clear them out all the time.  Many times a day and every time I close the browser... :) Then I found out about Mozilla Location Services, how they made it so we can't block that and realized they only blocked others and not themselves from feasting on our data. https://en.wikipedia.org/wiki/Mozilla_Location_Services https://location.services.mozilla.com Bastards! scott

On 10/13/21 2:39 AM, Doug Barton wrote:

On the cookie issue, I have had very good luck with this in Firefox:

https://addons.mozilla.org/en-US/firefox/addon/cookie-autodelete/ ---------------------------------------------------------------------------------------------

Nice, I have the settings to delete all history and cookies when I close the browser as well as remove them all the time while I am using it. I don't want to leave Firefox because of NoScript.  That stops a lot of snooping.  Too bad it doesn't work for other browsers like Vivaldi.  I would switch in a heartbeat because the dirty stuff Mozilla Location Services does is ugly. scott

hope this helps,

Doug

On 10/12/21 6:26 AM, scott wrote:

...

On 10/12/21 9:15 PM, Matthew Petach wrote:

...

So, I take it you steadfastly block *all* cookies from being stored or transmitted from your browser at home? ------------------------------------------------------------------\

I used to when Firefox had the "ask me every time" for cookies. They got rid of that, so now I clear them out all the time. Many times a day and every time I close the browser... :)

Then I found out about Mozilla Location Services, how they made it so we can't block that and realized they only blocked others and not themselves from feasting on our data.

https://en.wikipedia.org/wiki/Mozilla_Location_Services

https://location.services.mozilla.com

Bastards!

scott

----- On Oct 11, 2021, at 12:58 AM, Mark Tinka mark@tinka.africa wrote: Hi,

However, in an era where content is making a push to get as close to the eyeballs as possible, kit getting cheaper and faster because of merchant silicon, and abundance of aggregated capacity at exchange points, can we leverage the shorter, faster links to change the model?

Yes, let's go back to 2003. The ISP I worked for at that time was one of the first in the country (if not the first) to host Akamai's caching servers. Ten years later I worked on a project where Akamai caching was embedded in subscriber management routers. It was announced, but never productized. This concept would have brought caching as close to the subscriber as possible. Today, with the widespread use of HTTPS, something like this is just not feasible. Thanks, Sabri

* dougb@dougbarton.us (Doug Barton) [Sun 10 Oct 2021, 23:44 CEST]:

First, I'm not saying "should." I'm saying that given the market economics, having the content providers who use "a lot" of bandwidth do something to offset those costs to the ISPs might be the best/least bad option. Whether "something" is a local cache box, peering, money, or <other> is something I think that the market should determine.

Sounds like you think SK should be paying Netflix for bringing their content all the way from the US to the Korean peninsula. That's some expensive wet cable being used there. -- Niels.

On 10/10/21 23:42, Doug Barton wrote:

I didn't say income tax. Corporate taxes are considered an expense by the corporation paying them. Like all other expenses, they are factored into the cost of goods/services sold.

Ah, yes, agreed. I thought you meant something else...

First, I'm not saying "should." I'm saying that given the market economics, having the content providers who use "a lot" of bandwidth do something to offset those costs to the ISPs might be the best/least bad option. Whether "something" is a local cache box, peering, money, or <other> is something I think that the market should determine.

But all the major content providers do this already. They come to exchange points. They provide caches. What more do we want them to do? Content providers that don't do these things don't generally tend to be popular, in which case, we don't have to worry about them flooding backbone links. I am almost sure Netflix have some degree of presence in South Korea. What I'm not sure about is what else SK wants them to do beyond that.

And to answer Matthew's question, I don't know what "a lot" is. I think the market should determine that as well.

Hehe, free markets determine the fundamental principles through price and competition, which is why we are in this mess to begin with. Unless you want "government" to make the determination :-).

And for the record, not only have I never worked for an ISP, I was saying all the way back in the late '90s that the oversubscription business model (which almost always includes punishing users who actually use their bandwidth) is inherently unfair to the customers, and when the Internet becomes more pervasive in daily life will come back to bite them in the ass. I was laughed at for being hopelessly naive, not understanding how the bandwidth business works, etc.

Totally agreed. However, we are sort of forced to use this model because of the underlying technology. Whenever a finite resource has to be shared amongst several people, there has to be some way to manage that sharing. Maybe if Internet services were circuit-switched, we wouldn't have this problem. But then again, we wouldn't have an Internet like we do today either. Mark.

On Sun, Oct 10, 2021 at 2:44 PM Doug Barton <dougb@dougbarton.us> wrote:

[some snipping below]

Also just to be clear, these are my own opinions, not necessarily shared by any current or former employers.

On 10/10/21 12:31 PM, Mark Tinka wrote:

...

On 10/10/21 21:08, Doug Barton wrote

...

Given that issue, I have some sympathy for eyeball networks wanting to charge content providers for the increased capacity that is needed to bring in their content. The cost would be passed on to the content provider's customers...

But eyeballs are already paying you a monthly fee for 100Mbps of service (for example). So they should pay a surcharge, over-and-above that, that determines how they can use that 100Mbps? Seems overly odd, to me.

Yes, I get that. But as you pointed out here and in other comments, the ISP market is based entirely on undercutting competitors (with a lot of gambling thrown in, as Matthew pointed out).

[...]

...

So what rat hole does this lead us down into? People who want to stream Youtube should pay their ISP for that? People who want to spend unmentionable hours on Linkedin should be their ISP for that? People who want to gawk over Samsung's web site because they love it so much, should pay their ISP for that?

First, I'm not saying "should." I'm saying that given the market economics, having the content providers who use "a lot" of bandwidth do something to offset those costs to the ISPs might be the best/least bad option. Whether "something" is a local cache box, peering, money, or <other> is something I think that the market should determine.

Going back to the fact that it's not the content providers "using" a lot of bandwidth, it's the eyeball customer *requesting* a lot of bandwidth, I think the best approach is for the content providers to help manage traffic levels by lowering bit rates towards eyeball networks that are feeling strained by their users. Instead of a 4K stream, drop it to 480 or 240; the eyeball network should be happy at the reduced strain the resulting stream puts on their network. The content network can even point out they're being a good Network Citizen by putting up a brief banner at the top of the stream saying "reducing bit rate to relieve stress on your ISPs network". That way, the happy customer knows that the content provider is doing their part to help their ISP stay profitable...I mean, doing their part to help the Internet run better.

And to answer Matthew's question, I don't know what "a lot" is. I think the market should determine that as well.

The market *is* determining that at the moment...but not in the direction people expect. Instead, it's creating a new market for intermediaries; imagine you're an eyeball network that happens to have peering with SKB, and largely inbound traffic flows. Wouldn't it make sense for you to reach out to a player like Netflix, and offer to host content cache boxes that happen to only answer requests coming from SKB IP space, at a price well below what SKB was going to charge the content provider? As the eyeball network, you'd see your traffic ratios balance out as the cache traffic filled your under-utilized outbound port capacity, and you'd get a bit of additional revenue you otherwise wouldn't get. As the content provider, you're serving your customers for a lower price than SKB wants to charge, and without giving into SKB's extortion tactics. It's a win-win-lose situation, in which the content provider wins, the eyeball network that has a peering relationship with SKB wins, and the only loser is SKB, which doesn't get the additional revenue it was looking for, and actually helps funnel money to a competitor that they otherwise wouldn't have gotten. I'm pretty sure this is going to start happening more and more, as ISPs realize that putting content caches into their IP space to serve not only their own customers, but also customers of selected peers can be a source of good leverage in the market. Matt

* mark@tinka.africa (Mark Tinka) [Mon 11 Oct 2021, 17:18 CEST]:

To be fair, Jane + Thatho don't care about video resolution.

I don't think that's being entirely fair. Netflix in plenty places differentiates its subscriptions based partly on video resolution: https://help.netflix.com/en/node/24926/us Some people will definitely care enough to sign up for a more expensive tier. -- Niels.

Going back to the fact that it's not the content providers "using" a lot of bandwidth, it's the eyeball customer *requesting* a lot of bandwidth, I think the best approach is for the content providers to help manage traffic levels by lowering bit rates towards eyeball networks that are feeling strained by their users.

This is the model that has pissed me off for decades… Somehow the cellular carrier networks have been able to force phone application and content providers to do things like limit the maximum size of file that can be downloaded over the cellular network and force you to download certain content over wifi. Since I maintain a backup handset anyway and it is rarely utilized, I let it accumulate rollover data until I want to do a large download. Then I turn on its hotspot and the other phone uses that wifi to download the large file I wasn’t allowed to download over the cellular network due to exactly this stupid kind of limitation.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network should be happy at the reduced strain the resulting stream puts on their network.

So your solution is to make the content provider punish the eyeball user and deliver a poor user experience in order to let the crappy eyeball network off the hook? I don’t think that’s a good solution. It makes the content provider look bad to the end user and it shifts the burden from the ISP that got paid to deliver content they are failing to deliver onto the content provider that is trying to live up to their agreement with their user. IIRC, Netflix charges extra for a 4K level subscription these days, so an end user that paid for 4K service and got 240p because their ISP managed to force Netflix into a lower bitrate would likely be pretty annoyed at both Netflix and the ISP if they understood the situation.

The content network can even point out they're being a good Network Citizen by putting up a brief banner at the top of the stream saying "reducing bit rate to relieve stress on your ISPs network". That way, the happy customer knows that the content provider is doing their part to help their ISP stay profitable...I mean, doing their part to help the Internet run better.

Yeah, I’d be calling $CONTENT_PROVIDER and asking what I need to do to get the full rate service I paid for. I’d also be calling $ISP (or switching $ISP if possible) to one that didn’t have those issues .

I'm pretty sure this is going to start happening more and more, as ISPs realize that putting content caches into their IP space to serve not only their own customers, but also customers of selected peers can be a source of good leverage in the market.

Agreed… Caching close to the edge makes complete sense, especially with aggregated caching models through CDNs. Owen

On Mon, Oct 11, 2021 at 10:09 AM Michael Thomas <mike@mtcc.com> wrote:

On 10/11/21 12:49 AM, Matthew Petach wrote:

Instead of a 4K stream, drop it to 480 or 240; the eyeball network should be happy at the reduced strain the resulting stream puts on their network.

As a consumer paying for my 4k stream, I know who I'm calling when it drops to 480 and it ain't Netflix. The eyeballs are most definitely not happy.

Mike

I apologize for that. I was tired after two back-to-back days of board meetings, and I missed putting a clear sarcasm marker on that last line about "the eyeball networks should be happy at the reduced strain..." :( There should have been a clear ;-P at the end of the line to make it unmistakeable I was poking a very sharp stick at the eyeball networks and what it takes to actually make them happy. ^_^; Yes--the end consumers really shouldn't be the hostage in this battle, being moved about the chess board by either side, whether by their ISP trying to squeeze more money out of the content side, or by the content side trying to force more complaints into the service desk of the ISP. I mean, imagine this scenario for any other utility. Pacific Gas and Electric calling up Hoover Dam to say "hey, we're going to need to charge you some additional money this month." Hoover Dam: "...what?" PGE: "well, you're sending a lot more electricity to our customers this month, and we're going to have to upgrade our power lines to handle it; and since you're the one sending the electricity, you should pay for part of the costs." Hoover Dam: "...we're only sending enough electricity to meet the demands YOUR customers are placing on the grid. If they want to run their air conditioners all summer long, you need to charge them enough to cover your costs for it." Drat. My analogy just ran out, because I realize the dollars already flow the other way, and the hydroelectric station would just laugh at PG&E and threaten to raise the cost of the electricity simply for having to listen to their BS. ^_^; You can run the same scenario with your municipal water company, and imagine how it would play out if the municipality that put the pipes in to every home tried to charge the water supplier more because homes were taking longer showers. It's just such a fundamentally broken model, we laugh at it in any other industry. :( Again, I'm sorry for being tired and missing the explicit sarcasm indicator--not just for you, but for others who also responded to that paragraph. ^_^; Thanks! Matt

On 10/11/21 02:58, Owen DeLong wrote:

That’s irrelevant to what he is saying.

What he’s saying (and he’s 100% correct) is that any tax a corporation pays is collected from their customers one way or another.

A corporation has no other source of income with which to pay its taxes beyond those revenues collected from customers.

Of course I should probably expect this from someone who thinks IPv4 shortages can be avoided by rationing IPv4 addresses.

There you go again, getting overly excited trying to divert the topic to your keen area of interest - IPv4 in Africa. I'll spell it out here so you are clear and have zero doubt: I do not respect you - for what you represent on my continent, amongst other things. So ignore me, because I am ignoring you. But if you don't ignore me, that's your problem too, not mine. Mark.

On Sun, Oct 10, 2021 at 12:12 PM Doug Barton <dougb@dougbarton.us> wrote:

On 10/1/21 7:45 AM, Mark Tinka wrote:

...

The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their own global backbones is because of this nonsense that SK Broadband is trying to pull with Netflix. At some point, the content folk will get fed up, and go build it themselves. What an opportunity infrastructure cost itself!

Except that Facebook, Microsoft, and Amazon all caved to SK's demands:

I will note that my $previous_employer was a top-10 web content provider that did *not* pay SK Broadband. Not all the content providers caved to SKB.

One incentive I haven't seen anyone mention is that ISPs don't want to charge customers what it really costs to provide them access. If you're the only one in your market that is doing that, no one is going to sign up because your pricing would be so far out of line with your competition.

That's a problem with your (collective) business model, then. If you sell something for less than it costs to make, it's called a loss-leader; and while you can do it for a little while, you'll get very little sympathy if people take advantage of it to drain your coffers. If you sell a service for less than it costs to provide, simply based on the hopes that people won't actually *use* it, that's called "gambling", and I have very little sympathy for businesses that gamble and lose.

Given that issue, I have some sympathy for eyeball networks wanting to charge content providers for the increased capacity that is needed to bring in their content. The cost would be passed on to the content provider's customers (in the same way that corporations don't pay taxes, their customers do), so the people on that ISP who are creating the increased demand would be (indirectly) paying for the increased capacity. That's actually fairer for the other customers who aren't Netflix subscribers.

That argument makes no sense whatsoever. What if instead of a single content provider, the extra traffic was generated by 10,000 small websites, each adding 1/10,000th of the volume of a single content provider? The cumulative impact on the eyeball network to handle the increased traffic is the same whether it comes from one content provider or from 10,000 separate smaller websites. Why should it be OK to go after the one content provider, but not go after the 10,000 smaller websites? At one point does your argument break down, and can you defend why that break point makes sense? Why is it OK to go after one, two, three, four content providers, but not to go after every website that is contributing to the increased traffic volume the eyeball network is handling? Seriously. Make your case. At what point do you draw that line, and say "we can charge content sites if there's less than 5 of them, but not if there's more than 10,000 of them?" How do you defend the choice of where you drew that arbitrary line? The reason that Netflix doesn't want to do it is the same reason that

ISPs don't want to charge their customers what it really costs to provide them access.

ISPs who don't charge enough to cover their costs are gambling, and hoping they get lucky. When they don't get lucky, and they lose their bet, they shouldn't get to make up for it by trying to strong-arm others to make up the difference. if you decide that "sender pays" is a fair model for the Internet to follow, then it needs to be applied equally, not just cherry-picking a few companies to extort, but leaving everyone else alone. As it stands, what you're arguing for is completely arbitrary and unfair. Matt

On 10/10/21 12:57 PM, Mark Tinka wrote:

On 10/10/21 21:33, Matthew Petach wrote:

...

If you sell a service for less than it costs to provide, simply based on the hopes that people won't actually *use* it, that's called "gambling", and I have very little sympathy for businesses that gamble and lose.

You arrived at the crux of the issue, quickly, which was the basis of my initial response last week - infrastructure is dying. And we simply aren't motivated enough to figure it out.

When you spend 25+ years sitting in a chair waiting for the phone to ring or the door to open, for someone to ask, "How much for 5Mbps?", your misfortune will never be your own fault.

Isn't that what Erlang numbers are all about? My suspicion is that after about 100Mbs most people wouldn't notice the difference in most cases. My ISP is about 25Mbs on a good day (DSL) and it serves our needs fine and have never run into bandwidth constraints. Maybe if we were streaming 4k all of the time it might be different, but frankly the difference for 4k isn't all that big. It's sort of like phone screen resolution: at some point it just doesn't matter and becomes marketing hype. Mike

On Sunday, 10 October, 2021 14:21, Mark Tinka wrote:

They are looking at the aggregate Gbps or Tbps of traffic that BigContent is seeking to deliver across their network, for "no $$".

This is blatantly incorrect. The bits were payed for by the requestor. BigContent does not "send bits" to non-requestors. The Internet is Point-to-Point, not a Broadcast medium. If the seller (the network operator) cannot provide the service which they have sold, they should be imprisoned for the remainder of their natural lives at hard labour. This sort of behaviour by the network operator is a Criminal Activity called FRAUD (based on Fraudulent Misrepresentation of Material Fact) and is, in fact, a Criminal Conspiracy. -- You can tell a politician is lying because it's lips are moving. The only good politician is a dead politician.

On Oct 10, 2021, at 13:21 , Mark Tinka <mark@tinka.africa> wrote:

On 10/10/21 22:13, Michael Thomas wrote:

...

Isn't that what Erlang numbers are all about? My suspicion is that after about 100Mbs most people wouldn't notice the difference in most cases. My ISP is about 25Mbs on a good day (DSL) and it serves our needs fine and have never run into bandwidth constraints. Maybe if we were streaming 4k all of the time it might be different, but frankly the difference for 4k isn't all that big. It's sort of like phone screen resolution: at some point it just doesn't matter and becomes marketing hype.

The ISP looking to charge BigContent for increased link saturation isn't looking at the individual 100Mbps links they have sold to their downstream customers.

They are looking at the aggregate Gbps or Tbps of traffic that BigContent is seeking to deliver across their network, for "no $$".

Which is the kind of ignorant view of the situation that creates this problem in the first place. It’s not for “no $$”, it’s for all the $$ they got from all those 100Mbps links that they are delivering those Tbps of traffic to. If the aggregate $$ they are collecting is insufficient, then they have priced their service incorrectly and should either re-evaluate, or go bankrupt and sell to someone that knows how to run a business. Owen

"at some point it just doesn't matter and becomes marketing hype." There is A LOT of hype over increasing broadband speeds, so much so to the point where immense oversubscription is the only practical way forward, then people piss and moan that ISPs didn't build enough to keep up with non-existent (at the time) demand. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Michael Thomas" <mike@mtcc.com> To: nanog@nanog.org Sent: Sunday, October 10, 2021 3:13:50 PM Subject: Re: S.Korea broadband firm sues Netflix after traffic surge On 10/10/21 12:57 PM, Mark Tinka wrote: On 10/10/21 21:33, Matthew Petach wrote: <blockquote> If you sell a service for less than it costs to provide, simply based on the hopes that people won't actually *use* it, that's called "gambling", and I have very little sympathy for businesses that gamble and lose. You arrived at the crux of the issue, quickly, which was the basis of my initial response last week - infrastructure is dying. And we simply aren't motivated enough to figure it out. When you spend 25+ years sitting in a chair waiting for the phone to ring or the door to open, for someone to ask, "How much for 5Mbps?", your misfortune will never be your own fault. </blockquote> Isn't that what Erlang numbers are all about? My suspicion is that after about 100Mbs most people wouldn't notice the difference in most cases. My ISP is about 25Mbs on a good day (DSL) and it serves our needs fine and have never run into bandwidth constraints. Maybe if we were streaming 4k all of the time it might be different, but frankly the difference for 4k isn't all that big. It's sort of like phone screen resolution: at some point it just doesn't matter and becomes marketing hype. Mike

On 10/10/21 22:10, Geoff Huston wrote:

I have to agree with Doug Barton's earlier observation is that the base problem is that the ISPs are using a flawed business model and they don't want to charge their customers what it really costs to provide them with high speed access, nor do they want to fund additional back-end capacity in their network without some form of offset revenue stream.

I think ISP's do want to charge their customers what it actually costs to provide them with a service, but they can't because many ISP's business models are based purely on undercutting their nearest competitor. I might be naive and hopeful to think that operators will have a blood handshake to set prices where customers can't wag the tail. Mark.

Netflix has programs for which many ISPs - even smaller are able to build a cache system. This may help the ISP who filed suit here - That being said - Our Consultancy has helped a number of smaller ISPs build using the Open Connect options - however for many they cannot justify the want from Netflix to have the minimum of 5Gbps of peak Netflix traffic let alone the 1.2Gbps of inbound traffic daily during the 12 hour update windows. A move like this may help wake up Netflix to making these options a bit nicer for the smaller boys Glenn

On Oct 11, 2021, at 00:01 , Mark Tinka <mark@tinka.africa> wrote:

On 10/11/21 00:31, Geoff Huston wrote:

...

In many environments, the words we use to describe this form of price setting are generally prefixed by the adjective “illegal” :-)

Indeed - colluding is generally frowned upon, in which case we are doomed to the current model, and may the best man win.

Ultimately, many ISP's and telco's will die. Consolidation will occur, but the "big operator" will no longer be as fat as they used to be. Focus on hauling bits around with no frills will be a good model, especially if you can keep the team lean. The chances of having large monopolies that do okay but stifle the market, being chased by struggling ISP's that favour passion + frills, is what is likely to happen, over the next decade or two.

Mark.

Ideally, a regulatory framework which prohibits vertical integration and thus prevents the natural monopoly of last mile physical infrastructure from being leveraged into a monopoly on higher-layer services would significantly improve the current situation, making room for a strong competing market with low barrier to entry for services while the last mile infrastructure was managed by a regulated utility or the run by the local municipality. Owen

On Sun, Oct 10, 2021 at 8:11 PM Doug Barton <dougb@dougbarton.us> wrote:

On 10/1/21 7:45 AM, Mark Tinka wrote:

The reason that Netflix doesn't want to do it is the same reason that ISPs don't want to charge their customers what it really costs to provide them access.

SO: In my part of the world where end user internet services are largely based on data caps, this should be good news to the ISP as the faster users burn the data on streaming, the sooner they renew their subscription. What is an ISP without content by the way? would one blame the likes of facebook for trying to run their own pipe thereby further contributing to internet defragmentation. Regard -- ------------------------------------------------------------------------ *Seun Ojedeji,Mobile: +2348035233535* Bringing another down does not take you up - think about your action!

On 10/1/21 07:19, Blake Hudson wrote:

It's about time Netflix played chicken with one of these ISPs and stopped offering service  (or offered limited service) to the ISPs that try to extort them and other content providers: Sorry, your service provider does not believe in net neutrality and has imposed limitations on your Netflix experience. For a better Netflix experience, consider exploring one of these other nearby internet providers: x, y, z.

Netflix actually did pretty much exactly that with Verizon back in 2014, displaying a message that read "The Verizon network is crowded right now." https://techcrunch.com/2014/06/05/netflix-error-blames-verizon-for-playback-... -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

Imagine it's 2008 and your AP is pushing out 3 mbps. Customers are all happy. Suddenly, Netflix demands 10x what you're offering. Customers are not happy. Customers don't understand. People don't understand. There are a million cogs in the machine and if the path of least resistance is to turn left, an ISP is going to turn left. Josh Luthman 24/7 Help Desk: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Oct 1, 2021 at 10:19 AM Blake Hudson <blake@ispn.net> wrote:

On 10/1/2021 8:48 AM, Sean Donelan wrote:

...

South Korean Internet service provider SK Broadband has sued Netflix to pay for costs from increased network traffic and maintenance work because of a surge of viewers to the U.S. firm's content, an SK spokesperson said on Friday. [...] Last year, Netflix had brought its own lawsuit on whether it had any obligation to pay SK for network usage, arguing Netflix's duty ends with creating content and leaving it accessible. It said SK's expenses were incurred while fulfilling its contractual obligations to Internet users, and delivery in the Internet world is "free of charge as a principle", according to court documents. [...]

https://www.reuters.com/business/media-telecom/skorea-broadband-firm-sues-ne...

...

I'll never understand over how ISPs see content providers as the enemy (or a rival). The content is why ISPs have customers. Don't get upset when your customer uses the service that you sold them (in a way that is precisely in accordance with the expected usage)!

Netflix, as an example, has even been willing to bear most of the cost with peering or bringing servers to ISPs to reduce the ISP's costs and improve the ISP customer's experience. It's about time Netflix played chicken with one of these ISPs and stopped offering service (or offered limited service) to the ISPs that try to extort them and other content providers: Sorry, your service provider does not believe in net neutrality and has imposed limitations on your Netflix experience. For a better Netflix experience, consider exploring one of these other nearby internet providers: x, y, z.

On 10/18/21 07:02, Josh Luthman wrote:

Netflix, as an example, has even been willing to bear most of the cost with peering or bringing servers to ISPs to reduce the ISP's costs and improve the ISP customer's experience.

Netflix doesn't do those things because it cares about the ISP's costs and the ISP customers' experience. Netflix does these things because Netflix cares about Netflix's costs and Netflix's customers' experience.

It's about time Netflix played chicken with one of these ISPs and stopped offering service  (or offered limited service) to the ISPs that try to extort them and other content providers:

Then Netflix would risk losing those customers, especially if the ISP in question is a cable company or offers its own video streaming services. Also, by peering and bringing servers to ISPs, Netflix improves its customers' experience and reduces Netflix's costs because they no longer need to pay a transit provider to deliver content.

Sorry, your service provider does not believe in net neutrality and has imposed limitations on your Netflix experience.

They actually did pretty much exactly that with Verizon back in 2014. https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-ov... -- Jay Hennigan - jay@west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV

Vs. an ISP that is causing the problem or trying to run a protection racket against content providers, I think it wouldn’t be hard for the content provider to supply appropriate messaging inserted at the front end of playback to explain the situation to their mutual customers. Instead of the typical FBI notice, imagine the movie starting with an ad that explains how the ISP is trying to increase consumer costs by forcing Netflix to pass along additional fees paid to the ISP to deliver content the customer has already paid said same ISP to deliver.

Wouldn't be hard, but doubtful it would be effective. Consumers already get the same message on a few TV channels during the annual carriage dispute-a-palooza, with both sides telling them to call the other one to complain. It clearly doesn't work. Outside of our sphere, nobody cares about this stuff. They just want their thing to work. On Mon, Oct 18, 2021 at 9:37 PM Owen DeLong via NANOG <nanog@nanog.org> wrote:

...

On Oct 18, 2021, at 14:48 , Jay Hennigan <jay@west.net> wrote:

On 10/18/21 07:02, Josh Luthman wrote:

...

Netflix, as an example, has even been willing to bear most of the cost with peering or bringing servers to ISPs to reduce the ISP's costs and improve the ISP customer's experience.

Netflix doesn't do those things because it cares about the ISP's costs and the ISP customers' experience.

Netflix does these things because Netflix cares about Netflix's costs and Netflix's customers' experience.

Of course, that doesn’t change the fact that it does lower the ISP’s costs and improve the ISP customers’ experience.

...

...

It's about time Netflix played chicken with one of these ISPs and stopped offering service (or offered limited service) to the ISPs that try to extort them and other content providers:

Then Netflix would risk losing those customers, especially if the ISP in question is a cable company or offers its own video streaming services.

Vs. an ISP that is causing the problem or trying to run a protection racket against content providers, I think it wouldn’t be hard for the content provider to supply appropriate messaging inserted at the front end of playback to explain the situation to their mutual customers. Instead of the typical FBI notice, imagine the movie starting with an ad that explains how the ISP is trying to increase consumer costs by forcing Netflix to pass along additional fees paid to the ISP to deliver content the customer has already paid said same ISP to deliver.

Somehow, I don’t see the ISP doing well against such a PR onslaught.

...

Also, by peering and bringing servers to ISPs, Netflix improves its customers' experience and reduces Netflix's costs because they no longer need to pay a transit provider to deliver content.

Where the ISP in question isn’t trying to force them to pay transit costs within said eyeball network, sure. But in SK’s case, it looks like they’re trying to force Netflix to pay to reach their eyeballs, even though the eyeballs in question are already paying them to deliver Netflix (and other) content.

...

...

Sorry, your service provider does not believe in net neutrality and has imposed limitations on your Netflix experience.

They actually did pretty much exactly that with Verizon back in 2014.

https://www.cnet.com/tech/home-entertainment/netflix-takes-aim-at-verizon-ov...

It appears to have worked out fairly well for them, too.

Owen