Packet loss and latency between Akamai and NTT in Miami
Any contacts with either Akamai or NTT here ? This is kind of important as this is affecting three of our RPKI publication servers (servers which I have de-priorized in Route53 to prevent any issues for RPs) I have a ticket open with Akamai but I'm not directly an NTT customer so any help is appreciated. A sample MTR report, see between hops 7 and 8. Funnily enough this is periodic and has a cycle of between 15 and 25 minutes. %% START MTR TCP IPV4 en 20240517-16:10 Start: 2024-05-17T16:10:01+0000 HOST: rpki-fe-45-79-203-193.rrdp. Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.204.6.21 0.0% 20 0.5 0.2 0.1 0.5 0.1 2.|-- 10.204.35.59 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.35.60 3.|-- 10.204.64.38 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.64.37 4.|-- lo0-0.gw3.atl1.us.linode. 0.0% 20 19.9 4.1 0.4 19.9 6.1 lo0-0.gw4.atl1.us.linode.com 5.|-- ae45.r12.atl01.ien.netarc 0.0% 20 0.5 0.5 0.4 0.7 0.1 ae48.r11.atl01.ien.netarch.akamai.com 6.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 5.2 2.3 0.4 14.8 3.7 ae0.r11.atl01.ien.netarch.akamai.com 7.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 3.3 11.4 0.9 38.5 11.0 ae-2.r25.atlnga05.us.bb.gin.ntt.net 8.|-- ae-1.r22.miamfl02.us.bb.g 20.0% 20 7271. 1816. 0.8 7271. 3236.6 ae-2.r25.atlnga05.us.bb.gin.ntt.net 9.|-- ae-1.r22.miamfl02.us.bb.g 5.0% 20 7201. 1684. 13.3 7256. 2970.2 ae-0.r23.miamfl02.us.bb.gin.ntt.net 10.|-- ae-11.a00.saplbr02.br.bb. 0.0% 20 123.2 206.8 13.1 3035. 667.7 ae-0.r23.miamfl02.us.bb.gin.ntt.net 11.|-- ae1-1326.gw1.nu.registro. 0.0% 20 129.2 125.2 116.7 139.5 5.8 ae-11.a00.saplbr02.br.bb.gin.ntt.net 12.|-- xe-0-1-2-0.core1.nu.regis 0.0% 20 117.9 174.0 117.6 1133. 225.9 ae1-1326.gw1.nu.registro.br Thanks Carlos -- -- ========================= Carlos M. Martinez-Cagnazzo http://cagnazzo.me =========================
On the measurement that you provided the packet loss doesn't carry through to the final hop, so surely it's more likely what you're looking at here is control plane rate limiting rather than true packet loss? See slide 33 onwards on https://archive.nanog.org/sites/default/files/10_Roisman_Traceroute.pdf On Fri, May 17, 2024, 18:42 Carlos Martinez-Cagnazzo <carlosm3011@gmail.com> wrote:
Any contacts with either Akamai or NTT here ?
This is kind of important as this is affecting three of our RPKI publication servers (servers which I have de-priorized in Route53 to prevent any issues for RPs)
I have a ticket open with Akamai but I'm not directly an NTT customer so any help is appreciated.
A sample MTR report, see between hops 7 and 8. Funnily enough this is periodic and has a cycle of between 15 and 25 minutes.
%% START MTR TCP IPV4 en 20240517-16:10 Start: 2024-05-17T16:10:01+0000 HOST: rpki-fe-45-79-203-193.rrdp. Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.204.6.21 0.0% 20 0.5 0.2 0.1 0.5 0.1 2.|-- 10.204.35.59 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.35.60 3.|-- 10.204.64.38 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.64.37 4.|-- lo0-0.gw3.atl1.us.linode. 0.0% 20 19.9 4.1 0.4 19.9 6.1 lo0-0.gw4.atl1.us.linode.com 5.|-- ae45.r12.atl01.ien.netarc 0.0% 20 0.5 0.5 0.4 0.7 0.1 ae48.r11.atl01.ien.netarch.akamai.com 6.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 5.2 2.3 0.4 14.8 3.7 ae0.r11.atl01.ien.netarch.akamai.com 7.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 3.3 11.4 0.9 38.5 11.0 ae-2.r25.atlnga05.us.bb.gin.ntt.net 8.|-- ae-1.r22.miamfl02.us.bb.g 20.0% 20 7271. 1816. 0.8 7271. 3236.6 ae-2.r25.atlnga05.us.bb.gin.ntt.net 9.|-- ae-1.r22.miamfl02.us.bb.g 5.0% 20 7201. 1684. 13.3 7256. 2970.2 ae-0.r23.miamfl02.us.bb.gin.ntt.net 10.|-- ae-11.a00.saplbr02.br.bb. 0.0% 20 123.2 206.8 13.1 3035. 667.7 ae-0.r23.miamfl02.us.bb.gin.ntt.net 11.|-- ae1-1326.gw1.nu.registro. 0.0% 20 129.2 125.2 116.7 139.5 5.8 ae-11.a00.saplbr02.br.bb.gin.ntt.net 12.|-- xe-0-1-2-0.core1.nu.regis 0.0% 20 117.9 174.0 117.6 1133. 225.9 ae1-1326.gw1.nu.registro.br
Thanks
Carlos
-- -- ========================= Carlos M. Martinez-Cagnazzo http://cagnazzo.me =========================
Responding off list. -Aaron May 17, 2024 at 12:42 PM by carlosm3011@gmail.com:
Any contacts with either Akamai or NTT here ?
This is kind of important as this is affecting three of our RPKI publication servers (servers which I have de-priorized in Route53 to prevent any issues for RPs)
I have a ticket open with Akamai but I'm not directly an NTT customer so any help is appreciated.
A sample MTR report, see between hops 7 and 8. Funnily enough this is periodic and has a cycle of between 15 and 25 minutes.
%% START MTR TCP IPV4 en 20240517-16:10 Start: 2024-05-17T16:10:01+0000 HOST: rpki-fe-45-79-203-193.rrdp. Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.204.6.21 0.0% 20 0.5 0.2 0.1 0.5 0.1 2.|-- 10.204.35.59 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.35.60 3.|-- 10.204.64.38 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.64.37 4.|-- lo0-0.gw3.atl1.us.linode. 0.0% 20 19.9 4.1 0.4 19.9 6.1 lo0-0.gw4.atl1.us.linode.com 5.|-- ae45.r12.atl01.ien.netarc 0.0% 20 0.5 0.5 0.4 0.7 0.1 ae48.r11.atl01.ien.netarch.akamai.com 6.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 5.2 2.3 0.4 14.8 3.7 ae0.r11.atl01.ien.netarch.akamai.com 7.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 3.3 11.4 0.9 38.5 11.0 ae-2.r25.atlnga05.us.bb.gin.ntt.net 8.|-- ae-1.r22.miamfl02.us.bb.g 20.0% 20 7271. 1816. 0.8 7271. 3236.6 ae-2.r25.atlnga05.us.bb.gin.ntt.net 9.|-- ae-1.r22.miamfl02.us.bb.g 5.0% 20 7201. 1684. 13.3 7256. 2970.2 ae-0.r23.miamfl02.us.bb.gin.ntt.net 10.|-- ae-11.a00.saplbr02.br.bb. 0.0% 20 123.2 206.8 13.1 3035. 667.7 ae-0.r23.miamfl02.us.bb.gin.ntt.net 11.|-- ae1-1326.gw1.nu.registro. 0.0% 20 129.2 125.2 116.7 139.5 5.8 ae-11.a00.saplbr02.br.bb.gin.ntt.net 12.|-- xe-0-1-2-0.core1.nu.regis 0.0% 20 117.9 174.0 117.6 1133. 225.9 ae1-1326.gw1.nu.registro.br
Thanks
Carlos
-- -- ========================= Carlos M. Martinez-Cagnazzo http://cagnazzo.me =========================
Hi, just to give some closure here: The issue was fixed, connectivity has been back to normal for approximately 36 hours already. I don't know exactly who fixed it or where exactly the problem was located. I guess it was in the end some sort of collective effort. Thanks to all those who offered help, I believe these kind of situations is where the value of the *NOG communities is actually seen and appreciated. /Carlos On Fri, May 17, 2024 at 2:40 PM Carlos Martinez-Cagnazzo <carlosm3011@gmail.com> wrote:
Any contacts with either Akamai or NTT here ?
This is kind of important as this is affecting three of our RPKI publication servers (servers which I have de-priorized in Route53 to prevent any issues for RPs)
I have a ticket open with Akamai but I'm not directly an NTT customer so any help is appreciated.
A sample MTR report, see between hops 7 and 8. Funnily enough this is periodic and has a cycle of between 15 and 25 minutes.
%% START MTR TCP IPV4 en 20240517-16:10 Start: 2024-05-17T16:10:01+0000 HOST: rpki-fe-45-79-203-193.rrdp. Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.204.6.21 0.0% 20 0.5 0.2 0.1 0.5 0.1 2.|-- 10.204.35.59 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.35.60 3.|-- 10.204.64.38 0.0% 20 0.3 0.3 0.2 0.6 0.1 10.204.64.37 4.|-- lo0-0.gw3.atl1.us.linode. 0.0% 20 19.9 4.1 0.4 19.9 6.1 lo0-0.gw4.atl1.us.linode.com 5.|-- ae45.r12.atl01.ien.netarc 0.0% 20 0.5 0.5 0.4 0.7 0.1 ae48.r11.atl01.ien.netarch.akamai.com 6.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 5.2 2.3 0.4 14.8 3.7 ae0.r11.atl01.ien.netarch.akamai.com 7.|-- ae-41.a03.atlnga05.us.bb. 0.0% 20 3.3 11.4 0.9 38.5 11.0 ae-2.r25.atlnga05.us.bb.gin.ntt.net 8.|-- ae-1.r22.miamfl02.us.bb.g 20.0% 20 7271. 1816. 0.8 7271. 3236.6 ae-2.r25.atlnga05.us.bb.gin.ntt.net 9.|-- ae-1.r22.miamfl02.us.bb.g 5.0% 20 7201. 1684. 13.3 7256. 2970.2 ae-0.r23.miamfl02.us.bb.gin.ntt.net 10.|-- ae-11.a00.saplbr02.br.bb. 0.0% 20 123.2 206.8 13.1 3035. 667.7 ae-0.r23.miamfl02.us.bb.gin.ntt.net 11.|-- ae1-1326.gw1.nu.registro. 0.0% 20 129.2 125.2 116.7 139.5 5.8 ae-11.a00.saplbr02.br.bb.gin.ntt.net 12.|-- xe-0-1-2-0.core1.nu.regis 0.0% 20 117.9 174.0 117.6 1133. 225.9 ae1-1326.gw1.nu.registro.br
Thanks
Carlos
-- -- ========================= Carlos M. Martinez-Cagnazzo http://cagnazzo.me =========================
-- -- ========================= Carlos M. Martinez-Cagnazzo http://cagnazzo.me =========================
participants (3)
-
Aaron Atac -
Ben Cartwright-Cox -
Carlos Martinez-Cagnazzo