Level3 routing issue US west coast
Hi, does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ??? Trace from NYC is also broken: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1 At 10:24 UTC+2 it was even more broken: Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ??? best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at<mailto:jj@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at<mailto:jj@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Hi Joseph, in the meantime I have ~20 verified paths which are affected and Level3 is simply not competent enough to reroute/drop the affected path ... FYI: my private ticket # is 9446435 ### There is also a new global ticket available: Network Event Detail Network Event Summary: Multiple devices were unreachable in Europe impacting IP services. Event Ticket ID: 9446797 Market Area Affected: Multiple Markets in Europe Ticket Create Date: 7/10/15 11:15:39 AM GMT Impacted For: 12 minutes Event Status: Active Time Since Last Update: 1 hour 39 minutes 7/10/15 11:26:39 AM GMT The IP NOC reported multiple devices were simultaneously unreachable in Europe impacting IP services. The devices recovered without any intervention from Level 3 after an interruption lasting twelve minutes. Due to the possibility of additional impacted, the IP NOC will monitor services until the completion of multiple ongoing events in the region. A final update will be se ### best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Joseph Jenkins Gesendet: Freitag, 10. Juli 2015 14:55 An: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at<mailto:jj@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
On Friday, July 10, 2015, Joseph Jenkins <joe@breathe-underwater.com> wrote:
Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
I have seen this several times with level3. They confirm packets are dropping and sevice is degraded yet they refuse to take tactical corrective action for hours and hours. Makes me furious. CB
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at <javascript:;>> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at <javascript:;><mailto:jj@anexia.at <javascript:;>> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Hi, sitting here and watching the packet loss coming and going :(. It changes every 10-25min. Looks like an massive issue in San Jose - routers out there sometimes have an latency from 5-6 SECONDS ... best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Ca By Gesendet: Freitag, 10. Juli 2015 15:42 An: Joseph Jenkins Cc: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast On Friday, July 10, 2015, Joseph Jenkins <joe@breathe-underwater.com> wrote:
Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
I have seen this several times with level3. They confirm packets are dropping and sevice is degraded yet they refuse to take tactical corrective action for hours and hours. Makes me furious. CB
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at <javascript:;>> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at <javascript:;><mailto:jj@anexia.at <javascript:;>> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they did nothing to fix the issue: ### Event Case ID: 9446216 Location: Los Angeles, CA Impacted For: 10 hours 52 minutes ETR: Unknown Bridge: N/A 08:52 GMT - Event Conclusion Summary Start Time: 07/09 19:48 GMT Stop Time: 07/10 06:40 GMT Root Cause: A packet loss issue in Los Angeles, CA. Fix Action: The packet loss issue resolved before any action was taken. Summary: The IP NOC is currently investigated a packet loss issue in Los Angeles, CA that was impacting IP services. The packet loss issue resolved before any action was taken and the IP NOC deemed services are stable after monitoring was concluded. 07:53 GMT - The IP NOC is currently monitoring services after the packet loss issue resolved before any action was taken. An update will be sent when traffic is deemed stable or if status changes. 06:43 GMT - The IP NOC is currently investigating a packet loss issue in Los Angeles, CA that is impacting IP services. Troubleshooting efforts are ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed hourly unless otherwise noted. ### best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: Jürgen Jaritsch Gesendet: Freitag, 10. Juli 2015 15:47 An: 'Ca By'; Joseph Jenkins Cc: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast Hi, sitting here and watching the packet loss coming and going :(. It changes every 10-25min. Looks like an massive issue in San Jose - routers out there sometimes have an latency from 5-6 SECONDS ... best regards Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Ca By Gesendet: Freitag, 10. Juli 2015 15:42 An: Joseph Jenkins Cc: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast On Friday, July 10, 2015, Joseph Jenkins <joe@breathe-underwater.com> wrote:
Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
I have seen this several times with level3. They confirm packets are dropping and sevice is degraded yet they refuse to take tactical corrective action for hours and hours. Makes me furious. CB
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at <javascript:;>> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at <javascript:;><mailto:jj@anexia.at <javascript:;>> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move. Kind regards, Jens
Hi, No SLA broken cause A- and B-End were not directly our circuits ... but it helps a lot to place some new orders ... at other partners :). best regards Jürgen Jaritsch -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jens Hoffmann Gesendet: Freitag, 10. Juli 2015 17:16 An: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move. Kind regards, Jens
We took them down yesterday, and attempted to bring them back up midnight PST, and still massive packet loss. so they remain down for now. On Fri, Jul 10, 2015 at 9:44 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
Hi,
No SLA broken cause A- and B-End were not directly our circuits ... but it helps a lot to place some new orders ... at other partners :).
best regards
Jürgen Jaritsch
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jens Hoffmann Gesendet: Freitag, 10. Juli 2015 17:16 An: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast
Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move.
Kind regards, Jens
While I can¹t say with any degree of certainty it's related, it¹s somewhat coincidental that one of one of their west coast customers (Daybreak Games / SOE) has been under a fairly hefty DDoS since mid-week. From what I recall see Daybreak/SOE only uses Level3. (Lots to talk about in that case.. They¹ve invaded his life.. Not sure I¹d react much better, albeit privately..) http://fortune.com/2015/07/10/john-smedley-vs-hackers/ http://eq2wire.com/2015/07/09/daybreak-ceo-to-convicted-lizard-squad-hacker -im-coming-for-you/ On 7/10/15, 11:05 AM, "Mr. NPP" <mr.npp@nopatentpending.com> wrote:
We took them down yesterday, and attempted to bring them back up midnight PST, and still massive packet loss. so they remain down for now.
On Fri, Jul 10, 2015 at 9:44 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
Hi,
No SLA broken cause A- and B-End were not directly our circuits ... but it helps a lot to place some new orders ... at other partners :).
best regards
Jürgen Jaritsch
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jens Hoffmann Gesendet: Freitag, 10. Juli 2015 17:16 An: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast
Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move.
Kind regards, Jens
One the DDoS targets was PCCW and their ports were congested ... this was the official explanation we got. Lots of discussion starts from here .... Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Donn Lasher Gesendet: Sonntag, 12. Juli 2015 18:13 An: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast? While I can¹t say with any degree of certainty it's related, it¹s somewhat coincidental that one of one of their west coast customers (Daybreak Games / SOE) has been under a fairly hefty DDoS since mid-week. From what I recall see Daybreak/SOE only uses Level3. (Lots to talk about in that case.. They¹ve invaded his life.. Not sure I¹d react much better, albeit privately..) http://fortune.com/2015/07/10/john-smedley-vs-hackers/ http://eq2wire.com/2015/07/09/daybreak-ceo-to-convicted-lizard-squad-hacker -im-coming-for-you/ On 7/10/15, 11:05 AM, "Mr. NPP" <mr.npp@nopatentpending.com> wrote:
We took them down yesterday, and attempted to bring them back up midnight PST, and still massive packet loss. so they remain down for now.
On Fri, Jul 10, 2015 at 9:44 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
Hi,
No SLA broken cause A- and B-End were not directly our circuits ... but it helps a lot to place some new orders ... at other partners :).
best regards
Jürgen Jaritsch
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jens Hoffmann Gesendet: Freitag, 10. Juli 2015 17:16 An: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast
Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move.
Kind regards, Jens
On Mon, Jul 13, 2015 at 4:14 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
One the DDoS targets was PCCW and their ports were congested ... this was the official explanation we got.
Lots of discussion starts from here ....
Can it be somehow related to the DDoS on Telegram (AS62041, AS59930)? 200Gbps SYN flood was what they said on twitter. I don't see 3491 as an upstream for either ASN any more. On a side note 3356 became upstream for 62041 about a week ago http://www.inmediahk.net/files/imagecache/w456/column_images/113252.png (the tweet has been deleted)
No idea about the final target ... I heard so much wrong information in the past 3 days ... Level3 didn't investigate in my packet loss report because there was another incident ongoing and so they thought my report was related to this issue. Even when I updated them AFTER they reported "solved" for the first issue they did nothing ... Other involved companies did nothing because of 100% incompetence ... It's simply frustrating .. :( Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: Yang Yu [mailto:yang.yu.list@gmail.com] Gesendet: Sonntag, 12. Juli 2015 23:42 An: Jürgen Jaritsch Cc: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast? On Mon, Jul 13, 2015 at 4:14 AM, Jürgen Jaritsch <jj@anexia.at> wrote:
One the DDoS targets was PCCW and their ports were congested ... this was the official explanation we got.
Lots of discussion starts from here ....
Can it be somehow related to the DDoS on Telegram (AS62041, AS59930)? 200Gbps SYN flood was what they said on twitter. I don't see 3491 as an upstream for either ASN any more. On a side note 3356 became upstream for 62041 about a week ago http://www.inmediahk.net/files/imagecache/w456/column_images/113252.png (the tweet has been deleted)
Level3 is broken again ... Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 178.255.154.17 63.6% 12 0.2 0.2 0.2 0.3 0.0 2. ge-6-14.car2.Prague1.Level3.net 0.0% 12 58.1 123.3 0.4 338.6 117.2 3. ??? 4. 4.53.208.102 90.9% 12 732.2 732.2 732.2 732.2 0.0 5. TenGE0-4-0-16.br02.hkg15.pccwbtn.net 81.8% 12 871.8 867.2 862.6 871.8 6.5 6. TenGE0-4-0-16.br02.hkg15.pccwbtn.net 90.0% 11 860.8 860.8 860.8 860.8 0.0 7. ? 80.0% 11 881.6 877.8 874.0 881.6 5.4 8. ??? Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-04.0v-00-03.anx04.vie.at.anexia-it.com 0.0% 9 0.9 2.1 0.4 14.8 4.8 2. cr-04.0v-08-71.anx03.vie.at.anexia-it.com 0.0% 9 8.9 2.2 0.5 8.9 3.5 3. win-b4-link.telia.net 0.0% 8 0.5 0.6 0.5 1.3 0.3 4. level-ic-1573273-wien-b4.c.telia.net 0.0% 8 0.5 0.5 0.5 0.7 0.1 5. ae-4-90.edge1.SanJose3.Level3.net 62.5% 8 160.7 159.6 158.7 160.7 1.0 6. ae-4-90.edge1.SanJose3.Level3.net 50.0% 8 158.7 158.7 158.5 159.0 0.2 7. ??? 8. TenGE1-3.br01.seo01.pccwbtn.net 57.1% 8 870.6 871.0 867.4 875.0 3.8 9. ??? 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 85.7% 8 873.1 873.1 873.1 873.1 0.0 11. ??? 12. 61.250.89.2 80.0% 6 893.1 893.1 893.1 893.1 0.0 13. ??? Jürgen Jaritsch Head of Network & Infrastructure ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-300 Telefax: +43-5-0556-500 E-Mail: jj@anexia.at Web: http://www.anexia.at Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 -----Ursprüngliche Nachricht----- Von: Jürgen Jaritsch Gesendet: Freitag, 10. Juli 2015 18:45 An: Jens Hoffmann; nanog@nanog.org Betreff: AW: Level3 routing issue US west coast Hi, No SLA broken cause A- and B-End were not directly our circuits ... but it helps a lot to place some new orders ... at other partners :). best regards Jürgen Jaritsch -----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Jens Hoffmann Gesendet: Freitag, 10. Juli 2015 17:16 An: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast Hi,
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they >did nothing to fix the issue:
Any SLA broken? Probably not, that would be a reason to move. Kind regards, Jens
On 7/10/15 7:54 AM, Jürgen Jaritsch wrote:
Wow .... Level3 responded to me that they had an issue last night .... but they simply did nothing ... for at least 10 hours they did nothing to fix the issue:
It's more likely that there's a certain amount of nonsense and a lot of loose ends in the ticketing system; than that the issue resolved itself without intervention. You might reasonably conclude that the corrective action and the ticket are unrelated at least as far as the noc is concerned.
###
Event Case ID: 9446216 Location: Los Angeles, CA Impacted For: 10 hours 52 minutes ETR: Unknown Bridge: N/A
08:52 GMT - Event Conclusion Summary
Start Time: 07/09 19:48 GMT Stop Time: 07/10 06:40 GMT
Root Cause: A packet loss issue in Los Angeles, CA.
Fix Action: The packet loss issue resolved before any action was taken. Summary: The IP NOC is currently investigated a packet loss issue in Los Angeles, CA that was impacting IP services. The packet loss issue resolved before any action was taken and the IP NOC deemed services are stable after monitoring was concluded.
07:53 GMT - The IP NOC is currently monitoring services after the packet loss issue resolved before any action was taken. An update will be sent when traffic is deemed stable or if status changes.
06:43 GMT - The IP NOC is currently investigating a packet loss issue in Los Angeles, CA that is impacting IP services. Troubleshooting efforts are ongoing with no estimated time to restore services available at this point. Please be advised that updates for this event will be relayed hourly unless otherwise noted.
###
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: Jürgen Jaritsch Gesendet: Freitag, 10. Juli 2015 15:47 An: 'Ca By'; Joseph Jenkins Cc: nanog@nanog.org Betreff: AW: Level3 routing issue US west coast
Hi,
sitting here and watching the packet loss coming and going :(.
It changes every 10-25min. Looks like an massive issue in San Jose - routers out there sometimes have an latency from 5-6 SECONDS ...
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at Web: http://www.anexia.at
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
-----Ursprüngliche Nachricht----- Von: NANOG [mailto:nanog-bounces@nanog.org] Im Auftrag von Ca By Gesendet: Freitag, 10. Juli 2015 15:42 An: Joseph Jenkins Cc: nanog@nanog.org Betreff: Re: Level3 routing issue US west coast
On Friday, July 10, 2015, Joseph Jenkins <joe@breathe-underwater.com> wrote:
Level3 had an issue with one of their core routers in Los Angeles last night(7pm Pacific) and early this morning(1am Pacific). Last update to my trouble ticket had the issue still being reviewed by engineering, but that a core router was dropping packets.
I have seen this several times with level3. They confirm packets are dropping and sevice is degraded yet they refuse to take tactical corrective action for hours and hours.
Makes me furious.
CB
On Jul 10, 2015, at 3:59 AM, Jürgen Jaritsch <jj@anexia.at <javascript:;>> wrote:
Hi,
does anyone else experience issues with the Level3 network at the US west coast? We see lots of broken paths like this:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 231 0.6 0.5 0.2 18.1 1.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 231 0.5 9.9 0.3 361.1 40.1 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 230 6.5 7.7 6.3 49.7 5.3 4. win-b4-link.telia.net 0.0% 230 6.6 6.8 6.4 20.2 1.5 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 230 6.6 9.3 6.3 69.1 9.6 6. ae-2-70.edge1.SanJose3.Level3.net 38.4% 230 164.8 165.0 164.5 194.9 2.6 7. ae-2-70.edge1.SanJose3.Level3.net 45.9% 230 164.7 164.8 164.5 174.1 0.9 8. 4.53.208.102 34.3% 230 634.9 310.7 168.5 680.1 199.7 9. TenGE5-4.br01.seo01.pccwbtn.net 34.1% 230 412.0 455.2 304.9 954.6 203.4 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 40.6% 230 323.4 441.4 323.1 822.0 182.1 11. 211.115.201.92 38.4% 230 289.8 412.9 289.6 846.7 185.4 12. 61.250.89.2 35.8% 230 290.6 439.4 290.2 804.3 205.1 13. ???
Trace from NYC is also broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. cr-01.0v-00-05.anx32.nyc.us.anexia-it.com 0.0% 30 0.4 4.3 0.4 57.8 13.3 2. nyk-b5-link.telia.net 0.0% 30 0.3 0.4 0.3 0.9 0.1 3. ??? 4. ae-3-80.edge1.SanJose3.Level3.net 17.2% 30 71.7 73.2 71.7 98.7 5.5 5. ae-3-80.edge1.SanJose3.Level3.net 0.0% 30 71.8 71.8 71.7 72.0 0.1 6. 4.53.208.102 31.0% 30 569.6 250.7 70.5 579.3 231.2 7. 63.218.250.73 31.0% 30 672.6 355.5 178.0 672.6 232.1
At 10:24 UTC+2 it was even more broken:
Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. er-01.0v-00-03.anx01.klu.at.anexia-it.com 0.0% 326 0.4 0.5 0.3 39.7 2.2 2. cr-01.0v-08-06.anx01.klu.at.anexia-it.com 0.0% 326 0.5 6.7 0.3 198.1 26.3 3. cr-04.01-01-04.anx03.vie.at.anexia-it.com 0.0% 326 6.6 7.6 6.4 43.6 4.5 4. win-b4-link.telia.net 0.0% 326 6.7 7.4 6.3 43.1 3.2 5. level-ic-1573273-wien-b4.c.telia.net 0.0% 326 6.9 9.2 6.3 73.2 10.1 6. ae-1-60.edge5.LosAngeles1.Level3.net 62.6% 326 164.7 165.5 164.5 176.7 1.5 ae-2-70.edge1.SanJose3.Level3.net 7. ae-1-60.edge5.LosAngeles1.Level3.net 63.1% 326 164.8 165.8 164.6 204.2 3.9 ae-2-70.edge1.SanJose3.Level3.net 8. 205.129.5.70 74.2% 326 799.9 487.2 169.0 799.9 305.7 4.53.208.102 9. TenGE5-4.br01.seo01.pccwbtn.net 77.2% 326 1359. 701.0 308.7 3716. 510.6 10. sejong-telecom.ge5-3.br01.seo01.pccwbtn.net 75.1% 326 960.4 643.0 323.4 960.4 307.6 11. 211.115.201.92 68.9% 326 925.3 674.2 289.8 932.3 296.6 12. 61.250.89.2 72.9% 326 928.5 637.2 291.9 928.5 304.3 13. ???
best regards
Jürgen Jaritsch Head of Network & Infrastructure
ANEXIA Internetdienstleistungs GmbH
Telefon: +43-5-0556-300 Telefax: +43-5-0556-500
E-Mail: jj@anexia.at <javascript:;><mailto:jj@anexia.at <javascript:;>> Web: http://www.anexia.at<http://www.anexia.at/>
Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601
participants (10)
-
Ca By -
Christopher Morrow -
Donn Lasher -
Jens Hoffmann -
Joe Hamelin -
joel jaeggli -
Joseph Jenkins -
Jürgen Jaritsch -
Mr. NPP -
Yang Yu