AS37468 (Angola Cables) - Route Leak?
Cross-posting from outages list: Hey Folks, Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69). Traffic was transiting via AS37468 (Angola Cables) in Coresite IX. Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX. I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently. My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2 I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks? Phil
Thanks for the efforts, Mark. Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected prefixes from Angola Cables in LAX and Singapore. That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: https://twitter.com/DougMadory/status/1661723901519380481 <https://twitter.com/DougMadory/status/1661723901519380481>
Hi everyone, I'm Frederico Oliveira, Head of Engineering at Angola Cables. Let me explain the root cause: We had an issue into our routing automatic tool, which daily performs some changes into our VRFs. During this activity, we had an anomalous situation and the routes were leaked between the VRFs(Peering and Transit). Our sincere apologies for the incovenience, please make sure we are working to avoid this kind of situation. Att, Frederico Oliveira
participants (3)
-
Frederico Oliveira -
Mark Tinka -
Phil Lavin