Greetings Jason, Thank you for your kind feedback, we have CM planned for today to do the needed change. Kind regards, Mara ________________________________ From: NANOG <nanog-bounces@nanog.org> on behalf of Mike Hammett <nanog@ics-il.net> Sent: Wednesday, January 30, 2019 8:41 AM To: Jason Lixfeld Cc: North American Network Operators' Group Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY A lot of huge companies apparently find it tough to find the $75k to hire one more peering person. Not all, though. For many, everything just runs like clockwork. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ________________________________ From: "Jason Lixfeld" <jason+nanog@lixfeld.ca> To: "North American Network Operators' Group" <nanog@nanog.org> Sent: Wednesday, January 30, 2019 7:52:09 AM Subject: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY Hi, In late October 2018, DE-CIX announced that they would be renumbering their IPv4 address block in New York between 01-28-19 and 01-30-19. This was followed by numerous reminders in months, weeks and even days leading up to the renumbering activity. The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are still using the old IPs. If three months has gone by and the numerous reminders that have been sent have resulted in these organizations still living on the old IP space, it seems to me that there may be some sort of a disconnect between who receives the notifications from IXPs and how they are filtered upstream. I’m hopeful that the eyeballs who read this list are some of those folks who should have received the notifications from DE-CIX, or can at least filter the info back downstream to whoever can perform the renumbering activity. Thanks.

Akamai is working on doing our part. Apologies. Sent from my iCar

On Jan 30, 2019, at 12:02 PM, Mehmet Akcin <mehmet@akcin.net> wrote:

Pinged my contacts in each

...

On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld <jason+nanog@lixfeld.ca> wrote: Hi,

In late October 2018, DE-CIX announced that they would be renumbering their IPv4 address block in New York between 01-28-19 and 01-30-19.

This was followed by numerous reminders in months, weeks and even days leading up to the renumbering activity.

The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are still using the old IPs.

If three months has gone by and the numerous reminders that have been sent have resulted in these organizations still living on the old IP space, it seems to me that there may be some sort of a disconnect between who receives the notifications from IXPs and how they are filtered upstream.

I’m hopeful that the eyeballs who read this list are some of those folks who should have received the notifications from DE-CIX, or can at least filter the info back downstream to whoever can perform the renumbering activity.

Thanks.

-- Mehmet +1-424-298-1903

Hi all, Thanks for your support! This helps us getting all peers on the new IPv4 space. Our looking glass shows which peers already have changed the IP settings (see section “BGP session established”) and which peers are still working on it (see section “BGP sessions down”): https://lg.de-cix.net/routeservers/rs1_nyc_ipv4#sessions-up Best regards, Thomas From: NANOG <nanog-bounces@nanog.org> on behalf of James Stankiewicz <stankiewicz@njedge.net> Date: Wednesday, 30. January 2019 at 19:32 To: Jared Mauch <jared@puck.nether.net> Cc: North American Network Operators' Group <nanog@nanog.org> Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY Microsoft an Edgecast has not yet made there changes. On Wed, Jan 30, 2019 at 12:20 PM Jared Mauch <jared@puck.nether.net> wrote: Akamai is working on doing our part. Apologies. Sent from my iCar On Jan 30, 2019, at 12:02 PM, Mehmet Akcin <mehmet@akcin.net> wrote: Pinged my contacts in each On Wed, Jan 30, 2019 at 05:52 Jason Lixfeld <jason+nanog@lixfeld.ca> wrote: Hi, In late October 2018, DE-CIX announced that they would be renumbering their IPv4 address block in New York between 01-28-19 and 01-30-19. This was followed by numerous reminders in months, weeks and even days leading up to the renumbering activity. The renumbering activity has come and gone, but LinkedIn, Amazon and Akamai are still using the old IPs. If three months has gone by and the numerous reminders that have been sent have resulted in these organizations still living on the old IP space, it seems to me that there may be some sort of a disconnect between who receives the notifications from IXPs and how they are filtered upstream. I’m hopeful that the eyeballs who read this list are some of those folks who should have received the notifications from DE-CIX, or can at least filter the info back downstream to whoever can perform the renumbering activity. Thanks. -- Mehmet +1-424-298-1903 -- Jim Stankiewicz Principal Network Architect NJEdge W:855.832.EDGE(3343) c:201.306.4409

On Thu, Jan 30, 2019 at 23:10 AM Ren Provo <ren.provo@gmail.com> wrote:

You probably should remove sessions with networks explicitly *not* participating in route servers versus displaying them on a global shame list.

And so it begins — yet another discussion on what does the word "responsibility" really mean. Given that e.g. the peering facility in Amazon, according to an adjacent NANOG ML thread, is in deep deep trouble since Nov 2018, just shutting down sessions with all of the entries in that shame list is likely to cause huge disruption and disappoinment. -- Töma

On Wed, 30 Jan 2019 23:55:40 +0000, "i3D.net - Martijn Schmidt" said:

Here: all networks that didn't already change their peering IP are not yet connected to the updated route-server. Some networks are not connected to any route-server. Therefore, those networks did not yet change their peering IP.

I think you can see what's wrong with that statement.. it does not follow. That has nothing to do with peering department resources, but everything to do with the chosen peering strategy.

Under what conditions would somebody be present at the exchange and not talking to the route server *at all* before the IP change?

Hi Ren et al., Thanks for pointing out that some peers do not use the route servers. This group can be subdivided in a group of peers not sending any IP prefixes to the route servers while maintaining a route server BGP session, and a group of peers not even connecting to the route server. The latter do not show up in the "BGP session established" section even if they have applied the required IPv4 changes. Best regards, Thomas On 31.01.19, 02:32, "valdis.kletnieks@vt.edu" <valdis.kletnieks@vt.edu> wrote: On Wed, 30 Jan 2019 23:55:40 +0000, "i3D.net - Martijn Schmidt" said: > Here: all networks that didn't already change their peering IP are not > yet connected to the updated route-server. Some networks are not > connected to any route-server. Therefore, those networks did not yet > change their peering IP. > > I think you can see what's wrong with that statement.. it does not > follow. That has nothing to do with peering department resources, but > everything to do with the chosen peering strategy. Under what conditions would somebody be present at the exchange and not talking to the route server *at all* before the IP change?

On 31/1/19 7:08 pm, Mark Tinka wrote:

I believe most exchange points maintain both route servers and route collectors.

Generally, most peers will connect to the RS, but not all. As you mention, some may connect but not send any routes.

However, I believe all peers will connect to the RC. Of course, I can envisage scenarios where even this could be selectively done. But the reasons for (not) connecting to the RC are very different from those for (not) connecting to the RS.

Back when I looked more deeply at this (mid-2014 when I was redoing the AS15169 policy around this area) I saw things rather differently, and my impression is little has changed since. Even in exchanges that strongly encourage their use route collectors were much less connected to than route servers, and few exchanges had them in the first place. Part of the problem with advertising on route servers is many clients, including networks that should know better often treat those routes as a higher priority than is sensible, in some cases equal or higher than a PNI link in the same city. Yes we could have used prepends, but that's not necessarily effective and affects everyone for the actions of a few, and is why the routes AS15169 advertised on route servers reduced back then.

Do people not know how to use local pref and MED to prefer PNI over route server? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mark Tinka" <mark.tinka@seacom.mu> To: nanog@nanog.org Sent: Thursday, January 31, 2019 6:20:42 AM Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY On 31/Jan/19 12:04, Julien Goodwin wrote:

Even in exchanges that strongly encourage their use route collectors were much less connected to than route servers, and few exchanges had them in the first place.

We, for example, connect to RS's more selectively. We are more liberal about RC's since they do not have an impact on our forwarding paradigm, and it helps the exchange point know what's happening across their fabric. But yes, I do imagine that interest level of connecting to either an RS or RC could vary, particularly the larger of a network you are.

Part of the problem with advertising on route servers is many clients, including networks that should know better often treat those routes as a higher priority than is sensible, in some cases equal or higher than a PNI link in the same city.

Well, there are a number of peers that do not have a linear peering relationship for all routes available at an exchange point, i.e., they don't see those routes both via the RS and bi-lateral sessions. For many networks, RS is the true source and bi-lateral sessions are not even considered. We may not always peer with an RS, but we will always have bi-lateral sessions... even when we have sessions to the RS. Mark.

Not all routes are created equal. If you have a PNI and an IX connection of equal capacity, obviously the IX connection will fill up first given that there is more opportunity there. Also, there are more moving parts in an IX (and accompanying route servers), thus more to go wrong. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mark Tinka" <mark.tinka@seacom.mu> To: "Mike Hammett" <nanog@ics-il.net> Cc: nanog@nanog.org Sent: Thursday, January 31, 2019 7:09:54 AM Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY On 31/Jan/19 14:59, Mike Hammett wrote: Do people not know how to use local pref and MED to prefer PNI over route server? We don't particularly care how the routes are learned. Routes are routes. Our motivation for or against peering with an RS is granular policy control, and the level of trust we can put in the stability of the same over time. Mark.

A prefix is a prefix. A route is a prefix plus a next-hop. Your next hop for your PNI is different than your IX. I don't believe I advocated running IX links hot. Financially, as an IX operator, I'd prefer that people ran all their bits over an IX and that all links were best kept below 10% utilization. ;-) Obviously I know that's not good engineering or fiscally responsible on the network's behalf. Just going to the extreme to support my point. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Mark Tinka" <mark.tinka@seacom.mu> To: "Mike Hammett" <nanog@ics-il.net> Cc: nanog@nanog.org Sent: Thursday, January 31, 2019 8:14:44 AM Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY On 31/Jan/19 15:54, Mike Hammett wrote: Not all routes are created equal. If you have a PNI and an IX connection of equal capacity, obviously the IX connection will fill up first given that there is more opportunity there. I think you meant to say not all "paths" are equal. Routes are routes. Where they lead to is another matter. The presence of a PNI does not preclude good governance of an exchange point link. If you are going to (willingly or otherwise) ignore the health of your public peering links over your private ones (or vice versa), then I wish upon you all the hell you'll face that comes with taking that position. Our policy is simple - 50% utilized, you upgrade. Doesn't matter what type of link it is; WDM Transport, IP, peering (public or private), Metro, core backbone, protection paths, e.t.c. Choosing to let your public peering links run hot because your "major" peers are taken care of by the private links is irresponsible. Do a lot of networks do it; hell yes, and for reasons you'd not think are obvious. <blockquote> Also, there are more moving parts in an IX (and accompanying route servers), thus more to go wrong. </blockquote> Agreed, but that's not the crux of this thread (even though it's one of the reasons we do not relay solely on RS's). Mark.

On Jan 30, 2019, at 17:32 , valdis.kletnieks@vt.edu wrote:

On Wed, 30 Jan 2019 23:55:40 +0000, "i3D.net - Martijn Schmidt" said:

...

Here: all networks that didn't already change their peering IP are not yet connected to the updated route-server. Some networks are not connected to any route-server. Therefore, those networks did not yet change their peering IP.

I think you can see what's wrong with that statement.. it does not follow. That has nothing to do with peering department resources, but everything to do with the chosen peering strategy.

Under what conditions would somebody be present at the exchange and not talking to the route server *at all* before the IP change?

Route servers are a double-edged sword for many networks. There are a number of reasons that one might choose not to peer with route servers at an exchange point, even if you are willing to peer with every single individual peer at the exchange. It would be difficult for me to go into specific details without violating NDAs from former employers, but it really doesn’t take all that much imagination. Consider the following questions: 1. What information does one get from a direct peering that is removed by a route server? 2. How does the AS PATH change if you are peering with a route server? 3. What tools are available for measuring results of individual peering sessions vs. sorting out individual next-hops learned from a common peering session? Owen

What do IXes do (or can do) to enforce the completion of a renumbering? For example, Chicago Equinix announced a renumbering beginning of 2018, and while 99% of our peers have renumbered, we still have an albeit small handful who have not. (I will not name names.) That situation didn't get nearly the "media attention" that DE-CIX New York did, if any. The "end of migration period" was May 1st 2018; here we are. What's the next step, if any? On 1/30/19 4:05 PM, Thomas King wrote:

Hi all,

Thanks for your support! This helps us getting all peers on the new IPv4 space.

Our looking glass shows which peers already have changed the IP settings (see section “BGP session established”) and which peers are still working on it (see section “BGP sessions down”):

https://lg.de-cix.net/routeservers/rs1_nyc_ipv4#sessions-up

On 1/2/19 1:31 pm, Niels Bakker wrote:

* bryan@shout.net (Bryan Holloway) [Fri 01 Feb 2019, 02:00 CET]:

...

What do IXes do (or can do) to enforce the completion of a renumbering?

- Be ready to move ports to a quarantine VLAN when they haven't renumbered in time, despite those previously mentioned personal communications, so you can reuse the IP address space elsewhere.

Or just ACL ARP traffic for that subnet (assuming your equipment allows you to configure such a filter, most kit can probably program such an ACL, although perhaps not to configure it in the OS)