High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi, Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
Hi, wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss. Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo? Hi, Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0 2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0 3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7 4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5 5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9 9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0 10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0 Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations. On 04.09.2015 at 23:24 Jürgen Jaritsch wrote:
Hi,
wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.
Best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
We're seeing issues between the US and northwest Europe (UK / Ireland), that started around 40 minutes ago. They are fairly unrelated services (AWS, Linode, L3VPN, commercial IP transit)...so I'm assuming there's some kind of larger outage going on? Oliver ------------------------------------- Oliver Garraux Check out my blog: blog.garraux.net Follow me on Twitter: twitter.com/olivergarraux On Fri, Sep 4, 2015 at 2:33 PM, Fred Hollis <fred@web2objects.com> wrote:
1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0 2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0 3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7 4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5 5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9 9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0 10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0
Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations.
On 04.09.2015 at 23:24 Jürgen Jaritsch wrote:
Hi,
wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.
Best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
Hi, We see the same hier from server pingdom servers towards Amsterdam, The Netherlands. Last hops without loss in the traceroutes were fist: • 4 xe-8-0-0.bar1.Tampa1.Level3.net (4.53.172.1) 57.509 ms 57.970 ms 58.201 ms • 4 ae52.edge1.Washington4.Level3.net (4.53.112.25) 2.254 ms 2.453 ms 2.633 ms and last few I saw 10min ago was: • 4 xe-11-0-3.bar2.LasVegas1.Level3.net (205.129.18.249) 1.928 ms 2.211 ms 2.484 ms -- Kind regards, Peter Tavenier
On 04 Sep 2015, at 23:41, Oliver Garraux <oliver@g.garraux.net> wrote:
We're seeing issues between the US and northwest Europe (UK / Ireland), that started around 40 minutes ago. They are fairly unrelated services (AWS, Linode, L3VPN, commercial IP transit)...so I'm assuming there's some kind of larger outage going on?
Oliver
-------------------------------------
Oliver Garraux Check out my blog: blog.garraux.net Follow me on Twitter: twitter.com/olivergarraux
On Fri, Sep 4, 2015 at 2:33 PM, Fred Hollis <fred@web2objects.com> wrote:
1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0 2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0 3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7 4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5 5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9 9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0 10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0
Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations.
On 04.09.2015 at 23:24 Jürgen Jaritsch wrote:
Hi,
wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.
Best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
Hi, I do see one of our offices down ... but ftw.nj.nyi.net is reachable without any issue for us (but the path differ from yours): Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 47 0.5 5.4 0.4 80.3 17.4 2. cs70.nyi.net 0.0% 47 0.9 6.7 0.8 118.0 21.4 3. cs70.cs80.v.ewr.nyinternet.net 0.0% 47 2.6 8.1 2.0 144.8 27.0 4. 96.47.77.142.static.nyinternet.net 0.0% 47 2.1 2.1 2.0 3.1 0.1 5. ftw.nj.nyi.net 0.0% 47 2.0 2.0 1.9 2.2 0.1 One other thing we do see: looks like Level3 is broken globally ... we can't send traffic from Europe via Level3 to the Asian region: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-03.0v-00-03.anx04.vie.at.anexia-it.com 0.0% 101 0.3 0.4 0.3 2.2 0.2 2. cr-04.0v-08-71.anx03.vie.at.anexia-it.com 0.0% 101 0.8 0.8 0.4 17.8 2.0 3. win-b4-link.telia.net 0.0% 101 0.5 1.6 0.5 29.1 4.0 4. level-ic-1573273-wien-b4.c.telia.net 0.0% 101 0.5 3.6 0.4 85.3 12.1 5. ??? 6. 4.69.152.144 99.0% 100 307.4 307.4 307.4 307.4 0.0 7. 4.53.208.102 89.9% 100 253.8 255.7 251.3 265.8 3.9 8. TenGE4-2.br01.tok02.pccwbtn.net 87.9% 100 524.0 406.4 366.9 536.7 66.9 9. cr-01.0v-00-08.anx11.tyo.jp.anexia-it.com 91.9% 100 368.1 371.0 365.4 382.9 5.5 10. anx-lg-jp-tyo01.anexia-it.com 96.0% 100 368.9 368.6 366.9 369.3 1.1 We'll start a ticket with Level3 support ... hopefully they will share some information (in the next days ... ). Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:34 An: nanog@nanog.org Betreff: Re: High latency/packetloss in nyc/nj for cogent/level3/zayo? 1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0 2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0 3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7 4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5 5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9 9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0 10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0 Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations. On 04.09.2015 at 23:24 Jürgen Jaritsch wrote:
Hi,
wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.
Best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
Surprise, surprise ... the cleaning staff stopped his worked and connected back in the correct cord ... Looks like everything went back to normal .... Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jürgen Jaritsch Gesendet: Freitag, 04. September 2015 23:45 An: Fred Hollis <fred@web2objects.com>; nanog@nanog.org Betreff: AW: High latency/packetloss in nyc/nj for cogent/level3/zayo? Hi, I do see one of our offices down ... but ftw.nj.nyi.net is reachable without any issue for us (but the path differ from yours): Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 47 0.5 5.4 0.4 80.3 17.4 2. cs70.nyi.net 0.0% 47 0.9 6.7 0.8 118.0 21.4 3. cs70.cs80.v.ewr.nyinternet.net 0.0% 47 2.6 8.1 2.0 144.8 27.0 4. 96.47.77.142.static.nyinternet.net 0.0% 47 2.1 2.1 2.0 3.1 0.1 5. ftw.nj.nyi.net 0.0% 47 2.0 2.0 1.9 2.2 0.1 One other thing we do see: looks like Level3 is broken globally ... we can't send traffic from Europe via Level3 to the Asian region: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-03.0v-00-03.anx04.vie.at.anexia-it.com 0.0% 101 0.3 0.4 0.3 2.2 0.2 2. cr-04.0v-08-71.anx03.vie.at.anexia-it.com 0.0% 101 0.8 0.8 0.4 17.8 2.0 3. win-b4-link.telia.net 0.0% 101 0.5 1.6 0.5 29.1 4.0 4. level-ic-1573273-wien-b4.c.telia.net 0.0% 101 0.5 3.6 0.4 85.3 12.1 5. ??? 6. 4.69.152.144 99.0% 100 307.4 307.4 307.4 307.4 0.0 7. 4.53.208.102 89.9% 100 253.8 255.7 251.3 265.8 3.9 8. TenGE4-2.br01.tok02.pccwbtn.net 87.9% 100 524.0 406.4 366.9 536.7 66.9 9. cr-01.0v-00-08.anx11.tyo.jp.anexia-it.com 91.9% 100 368.1 371.0 365.4 382.9 5.5 10. anx-lg-jp-tyo01.anexia-it.com 96.0% 100 368.9 368.6 366.9 369.3 1.1 We'll start a ticket with Level3 support ... hopefully they will share some information (in the next days ... ). Best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:34 An: nanog@nanog.org Betreff: Re: High latency/packetloss in nyc/nj for cogent/level3/zayo? 1.|-- hosted-by-i3d.net 0.0% 10 8.1 17.3 0.3 144.6 45.0 2.|-- 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 10 0.3 2.1 0.2 9.4 3.0 3.|-- 40ge.cr1-cr0.smartdc.rtd.i3d.net 0.0% 10 0.3 7.3 0.3 13.3 5.7 4.|-- ae51.edge4.London1.Level3.net 0.0% 10 10.6 14.2 7.6 30.4 7.5 5.|-- 4.69.156.9 90.0% 10 224.7 224.7 224.7 224.7 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 8.|-- cs20.cs90.v.ewr.nyinternet.net 80.0% 10 169.4 168.8 168.1 169.4 0.9 9.|-- 96.47.77.134.static.nyinternet.net 90.0% 10 167.7 167.7 167.7 167.7 0.0 10.|-- ftw.nj.nyi.net 90.0% 10 169.4 169.4 169.4 169.4 0.0 Having this to almost every network located in NYC/NJ that is going through the said three carrier from many locations. On 04.09.2015 at 23:24 Jürgen Jaritsch wrote:
Hi,
wer're working with Telia and Hurricane in NYC and we only see some latency flaps in the HE network .... flapping from 0.3 to ~15ms. Nothing really bad. No visible packet loss.
Best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Fred Hollis Gesendet: Freitag, 04. September 2015 23:18 An: nanog@nanog.org Betreff: High latency/packetloss in nyc/nj for cogent/level3/zayo?
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
On 4 Sep 2015, at 22:18, Fred Hollis <fred@web2objects.com> wrote:
Hi,
Anyone also experiencing really high lancy and packetloss 80%+ in nyc/nj area for cogent/level3/zayo?
From London, yes to a general problem, but not sure if it’s specific to NYC, just East Coast in general. Start: Fri Sep 4 22:31:02 2015 BST HOST: severn Loss% Snt Last Avg Best Wrst StDev 1.|-- gw1.thn.exonetric.net 0.0% 10 7.7 0.9 0.2 7.7 2.4 2.|-- e1-3-2.a00.londen03.uk.ra.gin.ntt.net 0.0% 10 0.6 0.8 0.5 1.4 0.0 3.|-- be3028.ccr21.lon02.atlas.cogentco.com 0.0% 10 0.8 0.9 0.8 1.2 0.0 4.|-- be2328.ccr21.lon01.atlas.cogentco.com 0.0% 10 1.0 0.9 0.8 1.2 0.0 5.|-- be2178.ccr42.lon13.atlas.cogentco.com 0.0% 10 1.0 1.0 0.9 1.1 0.0 6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 7.|-- be2268.ccr41.iad02.atlas.cogentco.com 80.0% 10 147.2 145.7 144.2 147.2 2.0 8.|-- be2657.ccr42.dca01.atlas.cogentco.com 90.0% 10 145.4 145.4 145.4 145.4 0.0 9.|-- be2149.ccr42.jfk02.atlas.cogentco.com 90.0% 10 145.0 145.0 145.0 145.0 0.0 10.|-- be2149.ccr42.dca01.atlas.cogentco.com 80.0% 10 154.7 152.4 150.1 154.7 3.2 11.|-- be2113.ccr42.atl01.atlas.cogentco.com 80.0% 10 163.3 162.6 161.8 163.3 1.0 12.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 13.|-- ??? 100.0 8 0.0 0.0 0.0 0.0 0.0 14.|-- te0-0-1-0.agr12.dfw01.atlas.cogentco.com 71.4% 7 182.0 181.9 181.7 182.0 0.0 15.|-- te0-0-2-3.nr11.b000821-1.dfw01.atlas.cogentco.com 71.4% 7 183.3 182.8 182.2 183.3 0.0 16.|-- ??? 100.0 7 0.0 0.0 0.0 0.0 0.0
participants (5)
-
Fred Hollis -
Jürgen Jaritsch -
Mark Blackman -
Oliver Garraux -
Peter Tavenier