On Fri, Oct 1, 2021 at 9:08 AM Laura Smith via NANOG <nanog@nanog.org> wrote:

> The bad news now, is, there are plenty of many, small, local
> and regional ISP's who are willing to do whatever it takes to
> work with the content providers. All that's required is some
> network, a half-decent data centre and an exchange point. Gone
> are the days where customers clamored to sign up with Big
> Telco.

Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps you could answer me this riddle.....

- PoP in one of your "half-decent data centres" ... tick.
- Connnection to one of your "exchange point" ... tick.
- $certain_large_cdn present on said "exchange point" ... tick.

And yet .....

- $certain_large_cdn publishes routes on route server ? Nope.
- $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where the large networks see it beneath them to peer with (or publish routes for the benefit of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off their nose to spite their face ?

Having worked on building out a global content delivery network,

I can hopefully shed some light here.

Let's imagine a scenario where you have a CDN node attached to

a large peering location.

At that peering location, you have Nx100G direct peering links to

large eyeball networks, and Nx10G direct peering links to midsized

eyeball networks.

You have a historical connection into the peering switch as well,

which you've kept around and upgraded as time went by. Life

is generally good. You've got your own backbone between

your locations, and you capacity engineer your infrastructure

so that if your peering sessions go down at that site, users

are redirected to the next closest CDN node.

Then one day, Bad Juju(tm) happens. The router that your

Nx100G links with Large Eyeball Network A terminate on

goes down, doesn't matter if it's on their end or your end,

those direct sessions go down. But instead of your traffic

failing over to site B instead, it continues flowing--but now

it's going through the route server sessions, across the

now grossly-undersized connection into the public peering

switch, and *nobody* is happy about that. You tried putting

import and export lists into the route server, but they don't

seem to be taking effect, and the phone is ringing off the

hook, so you simply stop exporting routes into the route

server, and suddenly traffic is now failing over to site B

for all the eyeball requests from Large Eyeball Network A,

and you get to explain to your boss how those sessions

with the route servers caused the failover plan to fail badly.

Decision is made that route server peering sessions are

too far removed from direct control to be safe, and the

only way to effectively prevent this scenario from happening

again is to stop announcing routes to the route servers.

You're a smaller network; you're caught up in this, because you'd

been getting routes from the route server, and now you're not.

You request direct peering; unfortunately, the ports on the existing

line cards on the CDN edge routers are full, and it's going to be

months before the next capacity upgrade is planned; and those

line cards are all 100G cards, with no 10G ports planned. It's

decided that letting that traffic get handled by the transit ports is

cheaper than trying to get some additional 10G peering port capacity

at the site. And so, your request for direct peering is denied; there's

just no place to plug you in on the router on the other side, unless

you can drum up enough traffic to justify a 100G peering link.

Establishing direct peering across the peering fabric becomes

the only possible point of commonality, then; but that requires

adding explicit neighbor configuration into the router, unlike

the route-server mediated prefix exchange; and now you're

still running into limitations on the edge routers--though in this

case it's not physical port limitations, it's neighbor adjacency

limits on the routers. The more neighbors you configure on an

edge router, the longer the configs are, and depending on how

the neighbors are established, the more processing power and

memory it takes on the router. At a certain point, there's just not

enough to go around, and the CDN makes the decision to not add

any new BGP neighbors at that site. A few years down the line,

when hardware is added/upgraded, they may revisit that decision,

or it may persist, even after the upgrade, due simply to corporate

inertia--nobody goes back to revisit the decision, and see if it still

makes sense or not.

Sometimes it's just a case of limited humans; without good automation

tools in place, adding BGP adjacencies takes a ticket being created

in a request system, and then another more highly skilled human

going into the router to configure the neighbor. If you've got a limited

number of skilled humans available (and every company has fewer

of those than they really need, but that's always going to be the case),

the decision generally gets made to focus their attention on efforts

that bring in more revenue--and configuring public peering sessions

generally doesn't fall into that bucket. :(

And sometimes, financial considerations come into play; the CDN

network has a default upstream transit provider to catch the traffic

that isn't handled by direct peering, and the transit provider gives

a really nice price break if the traffic makes it up into the next tier;

so, CDN that's just *below* the cheaper tier looks for traffic it can

shunt onto transit, to nudge the volume over the line, to qualify for

the lower per-mbit price point--and the easiest traffic to use to nudge

them over is the smaller public peering neighbor traffic. And so, your

request for direct peering is looked at, and your traffic volume is

deemed to be the perfect fit to help fill out the transit commitment

needed to bump them into the cheaper tier.

In a perfect world, these would all be solvable issues, and exchanging

traffic directly with the CDN across the public switches would make

sense for everybody involved.

Unfortunately, we live in an imperfect world, with route server filters

that don't react quickly and with the level of precision needed, we

have limitations on how many BGP adjacencies routers can handle,

we have limited port counts on line cards, we have limited numbers

of humans to make changes to router configurations, and we have

economic pressures to meet volume thresholds to get better pricing

that all end up creating the scenario you find yourself in; a completely

reasonable request for direct peering nonetheless is ignored, or

denied, due to one or more of the reasons I've discussed.

Thanks!

Matt