Thanks, Phil. I got a response back from Angola Cables at 1248hrs UTC to say it should be fixed, so the times line up. Mark. On 5/25/23 18:23, Phil Lavin wrote:
Thanks for the efforts, Mark.
Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected prefixes from Angola Cables in LAX and Singapore.
That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: https://twitter.com/DougMadory/status/1661723901519380481
On 25 May 2023, at 17:13, Mark Tinka <mark@tinka.africa> wrote:
Angola Cable say they have fixed the issue.
Can you please check again and verify? Thanks.
Mark.
On 5/25/23 14:36, Phil Lavin wrote:
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) ->phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12.37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16.core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17.core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18.core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19.ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20.vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22.http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil