See traffic on Twelve99 re-routed via LAX now. Seems to be stable, for the moment:

                                    My traceroute  [v0.95]
VONMVPCVXYVFD (172.20.0.219) -> 216.147.3.209 (216.147.3.209)         2024-08-15T11:29:35+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                      Packets               Pings
 Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.20.0.1                                       0.0%    68    0.7   1.0   0.7   1.9   0.2
 2. lo0-0.bng2.ixn-lon.zen.net.uk                    0.0%    67   12.3   8.4   5.1  43.1   6.1
 3. lag-5.p2.ixn-lon.zen.net.uk                      0.0%    67    5.6   5.9   5.4   7.8   0.4
 4. ldn-b13-link.ip.twelve99.net                    45.5%    67    5.6   5.5   4.9   6.1   0.3
 5. ldn-bb2-link.ip.twelve99.net                     0.0%    67    6.1   6.3   5.4   8.9   0.6
 6. nyk-bb1-link.ip.twelve99.net                     0.0%    67   74.7  75.0  74.1  77.0   0.5
 7. rest-bb1-link.ip.twelve99.net                    1.5%    67   81.5  81.5  80.8  83.9   0.5
 8. lax-b22-link.ip.twelve99.net                     0.0%    67  137.5 138.4 136.7 175.7   5.4
 9. (waiting for reply)
10. (waiting for reply)
11. 15.230.16.155                                    0.0%    67  141.6 143.9 140.3 167.9   5.4
12. (waiting for reply)
13. 52.93.14.118                                     0.0%    67  216.6 178.8 165.8 267.3  20.9
14. 52.93.14.105                                     0.0%    67  171.5 171.4 170.2 175.7   1.0
15. 52.93.15.28                                      0.0%    67  169.2 179.0 169.0 238.9  15.5
16. 52.93.246.191                                    0.0%    67  166.1 167.8 165.3 194.0   4.4
17. 52.93.246.81                                     0.0%    67  172.3 172.4 171.3 179.7   1.4
18. (waiting for reply)
19. (waiting for reply)
20. (waiting for reply)
21. (waiting for reply)
22. (waiting for reply)
23. (waiting for reply)
24. 216.147.3.209                                    0.0%    67  166.4 167.1 166.0 169.1   0.6


On 15 Aug 2024, at 11:09, Phil Lavin <phil.lavin@vonage.com> wrote:

Fwiw, still ongoing

I raised an AWS ticket and they’re looking into it. If there’s any friendly AWS Net Engineers on list, your help removing Twelve99 from route would be appreciated. Cross-posting to Nanog for wider reach.


On 15 Aug 2024, at 09:44, Matt Almgren <malmgren@backblaze.com> wrote:

There's a planned SJC maintenance happening now. Expected end time is 2am PST. 

On Thu, Aug 15, 2024, 1:38 AM Phil Lavin via Outages <outages@outages.org> wrote:
It’s a good week for the Internet. Third outages list post this week :(

Seeing major loss on Twelve99 around SJO:

UK to AWS USW2:

                                My traceroute  [v0.95]
VONMVPCVXYVFD (172.20.0.14) -> 216.147.3.209 (216.147.3.209) 2024-08-15T09:29:13+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                             Packets               Pings
 Host                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 172.20.0.1                              0.0%     9    1.1   1.2   1.0   1.7   0.2
 2. lo0-0.bng2.ixn-lon.zen.net.uk           0.0%     9    5.5   6.8   5.4  11.2   1.9
 3. lag-5.p2.ixn-lon.zen.net.uk             0.0%     8    5.5   6.0   5.2   6.5   0.5
 4. ldn-b13-link.ip.twelve99.net           25.0%     8    5.8   5.5   5.0   6.0   0.4
 5. ldn-bb2-link.ip.twelve99.net            0.0%     8    6.6   6.8   5.9   7.5   0.7
 6. nyk-bb1-link.ip.twelve99.net            0.0%     8   75.2  75.2  74.5  75.9   0.5
 7. sjo-b23-link.ip.twelve99.net           57.1%     8  296.2 312.5 296.2 343.7  27.0
 8. (waiting for reply)
 9. 15.230.28.67                           62.5%     8  294.4 295.7 294.4 296.8   1.2
10. 54.240.242.119                         25.0%     8  337.1 305.7 294.6 337.1  16.2
11. (waiting for reply)
12. 52.93.15.36                            57.1%     8  398.2 400.8 395.4 408.8   7.1
13. 52.93.15.31                            42.9%     8  388.7 395.5 388.7 403.3   6.2
14. 52.93.14.34                            28.6%     8  431.8 432.2 393.2 488.2  35.8
15. 52.93.246.177                          42.9%     8  397.7 399.3 397.7 401.1   1.4
16. 52.93.246.81                           25.0%     8  393.4 396.9 393.2 400.6   3.0
17. (waiting for reply)
18. (waiting for reply)19. (waiting for re 28.6%     8  394.3 397.5 394.3 400.2   3.0

AWS USW2 to UK:

 Host                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ip-10-96-50-153.us-west-2.compute.internal               0.0%    87    0.2   0.1   0.1   0.2   0.0
 2. ec2-44-233-118-55.us-west-2.compute.amazonaws.com        0.0%    87    7.3  11.9   1.0  66.9  16.4
    ec2-34-221-151-47.us-west-2.compute.amazonaws.com
    ec2-44-233-117-187.us-west-2.compute.amazonaws.com
 3. 240.0.212.4                                              0.0%    87    0.5   0.4   0.3   1.7   0.2
    240.0.72.7
 4. 242.2.110.95                                             0.0%    87    0.8   1.8   0.4  20.6   3.4
 5. 240.2.144.13                                             0.0%    87    5.8   5.8   5.7   5.9   0.1
 6. amazon-ic-375418.ip.twelve99-cust.net                    0.0%    87    5.7   5.6   5.6   6.1   0.1
 7. port-b4-link.ip.twelve99.net                             0.0%    87    6.2   6.1   5.9   6.5   0.1
 8. port-b3-link.ip.twelve99.net                             0.0%    86    5.6   6.3   5.6  18.5   1.8
 9. palo-b24-link.ip.twelve99.net                            0.0%    86   21.4  21.3  21.1  21.6   0.1
10. nyk-bb2-link.ip.twelve99.net                            19.8%    86  165.3 172.2 165.3 206.1   4.7
11. ldn-bb1-link.ip.twelve99.net                            68.2%    86  391.5 401.2 387.3 411.8   6.3
12. ldn-b3-link.ip.twelve99.net                             57.0%    86  397.1 401.4 395.5 409.0   2.7
13. zeninternet-ic-363994.ip.twelve99-cust.net              40.7%    86  397.0 412.3 397.0 473.3  18.0
14. lag-1.bng2.ixn-lon.zen.net.uk                           63.5%    86  387.2 402.5 382.5 422.8   9.0
15. 62-3-100-30.dsl.in-addr.zen.co.uk                       55.8%    86  391.6 404.5 391.6 412.5   6.1
16. (waiting for reply)
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

This email, including its contents and any attachment(s), may contain confidential and/or proprietary information and is solely for the review and use of the intended recipient(s). If you have received this email in error, please notify the sender and permanently delete this email, its content, and any attachment(s). Any disclosure, copying, or taking of any action in reliance on an email received in error is strictly prohibited.