Re: [CloudOps] [outages] Major loss/latency on Twelve99 around NY/SJO
Fwiw, still ongoing I raised an AWS ticket and they’re looking into it. If there’s any friendly AWS Net Engineers on list, your help removing Twelve99 from route would be appreciated. Cross-posting to Nanog for wider reach.
On 15 Aug 2024, at 09:44, Matt Almgren <malmgren@backblaze.com> wrote:
There's a planned SJC maintenance happening now. Expected end time is 2am PST.
On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
It’s a good week for the Internet. Third outages list post this week :(
Seeing major loss on Twelve99 around SJO:
UK to AWS USW2:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.14) -> 216.147.3.209 (216.147.3.209) 2024-08-15T09:29:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 9 1.1 1.2 1.0 1.7 0.2 2. lo0-0.bng2.ixn-lon.zen.net.uk <http://lo0-0.bng2.ixn-lon.zen.net.uk/> 0.0% 9 5.5 6.8 5.4 11.2 1.9 3. lag-5.p2.ixn-lon.zen.net.uk <http://lag-5.p2.ixn-lon.zen.net.uk/> 0.0% 8 5.5 6.0 5.2 6.5 0.5 4. ldn-b13-link.ip.twelve99.net <http://ldn-b13-link.ip.twelve99.net/> 25.0% 8 5.8 5.5 5.0 6.0 0.4 5. ldn-bb2-link.ip.twelve99.net <http://ldn-bb2-link.ip.twelve99.net/> 0.0% 8 6.6 6.8 5.9 7.5 0.7 6. nyk-bb1-link.ip.twelve99.net <http://nyk-bb1-link.ip.twelve99.net/> 0.0% 8 75.2 75.2 74.5 75.9 0.5 7. sjo-b23-link.ip.twelve99.net <http://sjo-b23-link.ip.twelve99.net/> 57.1% 8 296.2 312.5 296.2 343.7 27.0 8. (waiting for reply) 9. 15.230.28.67 62.5% 8 294.4 295.7 294.4 296.8 1.2 10. 54.240.242.119 25.0% 8 337.1 305.7 294.6 337.1 16.2 11. (waiting for reply) 12. 52.93.15.36 57.1% 8 398.2 400.8 395.4 408.8 7.1 13. 52.93.15.31 42.9% 8 388.7 395.5 388.7 403.3 6.2 14. 52.93.14.34 28.6% 8 431.8 432.2 393.2 488.2 35.8 15. 52.93.246.177 42.9% 8 397.7 399.3 397.7 401.1 1.4 16. 52.93.246.81 25.0% 8 393.4 396.9 393.2 400.6 3.0 17. (waiting for reply) 18. (waiting for reply)19. (waiting for re 28.6% 8 394.3 397.5 394.3 400.2 3.0
AWS USW2 to UK:
Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 87 0.2 0.1 0.1 0.2 0.0 2. ec2-44-233-118-55.us-west-2.compute.amazonaws.com <http://ec2-44-233-118-55.us-west-2.compute.amazonaws.com/> 0.0% 87 7.3 11.9 1.0 66.9 16.4 ec2-34-221-151-47.us-west-2.compute.amazonaws.com <http://ec2-34-221-151-47.us-west-2.compute.amazonaws.com/> ec2-44-233-117-187.us-west-2.compute.amazonaws.com <http://ec2-44-233-117-187.us-west-2.compute.amazonaws.com/> 3. 240.0.212.4 0.0% 87 0.5 0.4 0.3 1.7 0.2 240.0.72.7 4. 242.2.110.95 0.0% 87 0.8 1.8 0.4 20.6 3.4 5. 240.2.144.13 0.0% 87 5.8 5.8 5.7 5.9 0.1 6. amazon-ic-375418.ip.twelve99-cust.net <http://amazon-ic-375418.ip.twelve99-cust.net/> 0.0% 87 5.7 5.6 5.6 6.1 0.1 7. port-b4-link.ip.twelve99.net <http://port-b4-link.ip.twelve99.net/> 0.0% 87 6.2 6.1 5.9 6.5 0.1 8. port-b3-link.ip.twelve99.net <http://port-b3-link.ip.twelve99.net/> 0.0% 86 5.6 6.3 5.6 18.5 1.8 9. palo-b24-link.ip.twelve99.net <http://palo-b24-link.ip.twelve99.net/> 0.0% 86 21.4 21.3 21.1 21.6 0.1 10. nyk-bb2-link.ip.twelve99.net <http://nyk-bb2-link.ip.twelve99.net/> 19.8% 86 165.3 172.2 165.3 206.1 4.7 11. ldn-bb1-link.ip.twelve99.net <http://ldn-bb1-link.ip.twelve99.net/> 68.2% 86 391.5 401.2 387.3 411.8 6.3 12. ldn-b3-link.ip.twelve99.net <http://ldn-b3-link.ip.twelve99.net/> 57.0% 86 397.1 401.4 395.5 409.0 2.7 13. zeninternet-ic-363994.ip.twelve99-cust.net <http://zeninternet-ic-363994.ip.twelve99-cust.net/> 40.7% 86 397.0 412.3 397.0 473.3 18.0 14. lag-1.bng2.ixn-lon.zen.net.uk <http://lag-1.bng2.ixn-lon.zen.net.uk/> 63.5% 86 387.2 402.5 382.5 422.8 9.0 15. 62-3-100-30.dsl.in-addr.zen.co.uk <http://62-3-100-30.dsl.in-addr.zen.co.uk/> 55.8% 86 391.6 404.5 391.6 412.5 6.1 16. (waiting for reply) _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.
See traffic on Twelve99 re-routed via LAX now. Seems to be stable, for the moment: My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.219) -> 216.147.3.209 (216.147.3.209) 2024-08-15T11:29:35+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 68 0.7 1.0 0.7 1.9 0.2 2. lo0-0.bng2.ixn-lon.zen.net.uk 0.0% 67 12.3 8.4 5.1 43.1 6.1 3. lag-5.p2.ixn-lon.zen.net.uk 0.0% 67 5.6 5.9 5.4 7.8 0.4 4. ldn-b13-link.ip.twelve99.net 45.5% 67 5.6 5.5 4.9 6.1 0.3 5. ldn-bb2-link.ip.twelve99.net 0.0% 67 6.1 6.3 5.4 8.9 0.6 6. nyk-bb1-link.ip.twelve99.net 0.0% 67 74.7 75.0 74.1 77.0 0.5 7. rest-bb1-link.ip.twelve99.net 1.5% 67 81.5 81.5 80.8 83.9 0.5 8. lax-b22-link.ip.twelve99.net 0.0% 67 137.5 138.4 136.7 175.7 5.4 9. (waiting for reply) 10. (waiting for reply) 11. 15.230.16.155 0.0% 67 141.6 143.9 140.3 167.9 5.4 12. (waiting for reply) 13. 52.93.14.118 0.0% 67 216.6 178.8 165.8 267.3 20.9 14. 52.93.14.105 0.0% 67 171.5 171.4 170.2 175.7 1.0 15. 52.93.15.28 0.0% 67 169.2 179.0 169.0 238.9 15.5 16. 52.93.246.191 0.0% 67 166.1 167.8 165.3 194.0 4.4 17. 52.93.246.81 0.0% 67 172.3 172.4 171.3 179.7 1.4 18. (waiting for reply) 19. (waiting for reply) 20. (waiting for reply) 21. (waiting for reply) 22. (waiting for reply) 23. (waiting for reply) 24. 216.147.3.209 0.0% 67 166.4 167.1 166.0 169.1 0.6
On 15 Aug 2024, at 11:09, Phil Lavin <phil.lavin@vonage.com> wrote:
Fwiw, still ongoing
I raised an AWS ticket and they’re looking into it. If there’s any friendly AWS Net Engineers on list, your help removing Twelve99 from route would be appreciated. Cross-posting to Nanog for wider reach.
On 15 Aug 2024, at 09:44, Matt Almgren <malmgren@backblaze.com> wrote:
There's a planned SJC maintenance happening now. Expected end time is 2am PST.
On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <outages@outages.org <mailto:outages@outages.org>> wrote:
It’s a good week for the Internet. Third outages list post this week :(
Seeing major loss on Twelve99 around SJO:
UK to AWS USW2:
My traceroute [v0.95] VONMVPCVXYVFD (172.20.0.14) -> 216.147.3.209 (216.147.3.209) 2024-08-15T09:29:13+0100 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 172.20.0.1 0.0% 9 1.1 1.2 1.0 1.7 0.2 2. lo0-0.bng2.ixn-lon.zen.net.uk <http://lo0-0.bng2.ixn-lon.zen.net.uk/> 0.0% 9 5.5 6.8 5.4 11.2 1.9 3. lag-5.p2.ixn-lon.zen.net.uk <http://lag-5.p2.ixn-lon.zen.net.uk/> 0.0% 8 5.5 6.0 5.2 6.5 0.5 4. ldn-b13-link.ip.twelve99.net <http://ldn-b13-link.ip.twelve99.net/> 25.0% 8 5.8 5.5 5.0 6.0 0.4 5. ldn-bb2-link.ip.twelve99.net <http://ldn-bb2-link.ip.twelve99.net/> 0.0% 8 6.6 6.8 5.9 7.5 0.7 6. nyk-bb1-link.ip.twelve99.net <http://nyk-bb1-link.ip.twelve99.net/> 0.0% 8 75.2 75.2 74.5 75.9 0.5 7. sjo-b23-link.ip.twelve99.net <http://sjo-b23-link.ip.twelve99.net/> 57.1% 8 296.2 312.5 296.2 343.7 27.0 8. (waiting for reply) 9. 15.230.28.67 62.5% 8 294.4 295.7 294.4 296.8 1.2 10. 54.240.242.119 25.0% 8 337.1 305.7 294.6 337.1 16.2 11. (waiting for reply) 12. 52.93.15.36 57.1% 8 398.2 400.8 395.4 408.8 7.1 13. 52.93.15.31 42.9% 8 388.7 395.5 388.7 403.3 6.2 14. 52.93.14.34 28.6% 8 431.8 432.2 393.2 488.2 35.8 15. 52.93.246.177 42.9% 8 397.7 399.3 397.7 401.1 1.4 16. 52.93.246.81 25.0% 8 393.4 396.9 393.2 400.6 3.0 17. (waiting for reply) 18. (waiting for reply)19. (waiting for re 28.6% 8 394.3 397.5 394.3 400.2 3.0
AWS USW2 to UK:
Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-96-50-153.us-west-2.compute.internal 0.0% 87 0.2 0.1 0.1 0.2 0.0 2. ec2-44-233-118-55.us-west-2.compute.amazonaws.com <http://ec2-44-233-118-55.us-west-2.compute.amazonaws.com/> 0.0% 87 7.3 11.9 1.0 66.9 16.4 ec2-34-221-151-47.us-west-2.compute.amazonaws.com <http://ec2-34-221-151-47.us-west-2.compute.amazonaws.com/> ec2-44-233-117-187.us-west-2.compute.amazonaws.com <http://ec2-44-233-117-187.us-west-2.compute.amazonaws.com/> 3. 240.0.212.4 0.0% 87 0.5 0.4 0.3 1.7 0.2 240.0.72.7 4. 242.2.110.95 0.0% 87 0.8 1.8 0.4 20.6 3.4 5. 240.2.144.13 0.0% 87 5.8 5.8 5.7 5.9 0.1 6. amazon-ic-375418.ip.twelve99-cust.net <http://amazon-ic-375418.ip.twelve99-cust.net/> 0.0% 87 5.7 5.6 5.6 6.1 0.1 7. port-b4-link.ip.twelve99.net <http://port-b4-link.ip.twelve99.net/> 0.0% 87 6.2 6.1 5.9 6.5 0.1 8. port-b3-link.ip.twelve99.net <http://port-b3-link.ip.twelve99.net/> 0.0% 86 5.6 6.3 5.6 18.5 1.8 9. palo-b24-link.ip.twelve99.net <http://palo-b24-link.ip.twelve99.net/> 0.0% 86 21.4 21.3 21.1 21.6 0.1 10. nyk-bb2-link.ip.twelve99.net <http://nyk-bb2-link.ip.twelve99.net/> 19.8% 86 165.3 172.2 165.3 206.1 4.7 11. ldn-bb1-link.ip.twelve99.net <http://ldn-bb1-link.ip.twelve99.net/> 68.2% 86 391.5 401.2 387.3 411.8 6.3 12. ldn-b3-link.ip.twelve99.net <http://ldn-b3-link.ip.twelve99.net/> 57.0% 86 397.1 401.4 395.5 409.0 2.7 13. zeninternet-ic-363994.ip.twelve99-cust.net <http://zeninternet-ic-363994.ip.twelve99-cust.net/> 40.7% 86 397.0 412.3 397.0 473.3 18.0 14. lag-1.bng2.ixn-lon.zen.net.uk <http://lag-1.bng2.ixn-lon.zen.net.uk/> 63.5% 86 387.2 402.5 382.5 422.8 9.0 15. 62-3-100-30.dsl.in-addr.zen.co.uk <http://62-3-100-30.dsl.in-addr.zen.co.uk/> 55.8% 86 391.6 404.5 391.6 412.5 6.1 16. (waiting for reply) _______________________________________________ Outages mailing list Outages@outages.org <mailto:Outages@outages.org> https://puck.nether.net/mailman/listinfo/outages
This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.
participants (1)
-
Phil Lavin