AS37468 (Angola Cables) - Route Leak?
Cross-posting from outages list: Hey Folks, Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69). Traffic was transiting via AS37468 (Angola Cables) in Coresite IX. Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX. I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently. My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2 I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks? Phil
I've reached out to some known folk at Angola Cables. Will let you know if I hear back. Mark. On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) ->phil.lavin.me.uk <http://lavin.me.uk> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12.37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16.core6.par.hetzner.com <http://core6.par.hetzner.com> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17.core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18.core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19.ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20.vmh02.lavin.me.uk <http://vmh02.lavin.me.uk> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22.http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
Angola Cable say they have fixed the issue. Can you please check again and verify? Thanks. Mark. On 5/25/23 14:36, Phil Lavin wrote:
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) ->phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12.37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16.core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17.core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18.core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19.ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20.vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22.http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
Thanks for the efforts, Mark. Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected prefixes from Angola Cables in LAX and Singapore. That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: https://twitter.com/DougMadory/status/1661723901519380481 <https://twitter.com/DougMadory/status/1661723901519380481>
On 25 May 2023, at 17:13, Mark Tinka <mark@tinka.africa> wrote:
Angola Cable say they have fixed the issue.
Can you please check again and verify? Thanks.
Mark.
On 5/25/23 14:36, Phil Lavin wrote:
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa <mailto:mark@tinka.africa>> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
Thanks, Phil. I got a response back from Angola Cables at 1248hrs UTC to say it should be fixed, so the times line up. Mark. On 5/25/23 18:23, Phil Lavin wrote:
Thanks for the efforts, Mark.
Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected prefixes from Angola Cables in LAX and Singapore.
That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: https://twitter.com/DougMadory/status/1661723901519380481
On 25 May 2023, at 17:13, Mark Tinka <mark@tinka.africa> wrote:
Angola Cable say they have fixed the issue.
Can you please check again and verify? Thanks.
Mark.
On 5/25/23 14:36, Phil Lavin wrote:
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) ->phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12.37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16.core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17.core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18.core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19.ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20.vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22.http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
Hi everyone, I'm Frederico Oliveira, Head of Engineering at Angola Cables. Let me explain the root cause: We had an issue into our routing automatic tool, which daily performs some changes into our VRFs. During this activity, we had an anomalous situation and the routes were leaked between the VRFs(Peering and Transit). Our sincere apologies for the incovenience, please make sure we are working to avoid this kind of situation. Att, Frederico Oliveira
On 25 May 2023, at 13:23, Phil Lavin via NANOG <nanog@nanog.org> wrote:
Thanks for the efforts, Mark.
Tricky for me to check as the one known/testable traffic path was fixed by AWS around 12:27 UTC - they rejected prefixes from Angola Cables in LAX and Singapore.
That said, I see from the Kentik folks that it appears to have been resolved at 12:42 UTC: https://twitter.com/DougMadory/status/1661723901519380481
On 25 May 2023, at 17:13, Mark Tinka <mark@tinka.africa <mailto:mark@tinka.africa>> wrote:
Angola Cable say they have fixed the issue.
Can you please check again and verify? Thanks.
Mark.
On 5/25/23 14:36, Phil Lavin wrote:
Thanks, Mark
On 25 May 2023, at 13:34, Mark Tinka <mark@tinka.africa <mailto:mark@tinka.africa>> wrote:
I've reached out to some known folk at Angola Cables. Will let you know if I hear back.
Mark.
On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) -> phil.lavin.me.uk <http://lavin.me.uk/> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com/> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12. 37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com/> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16. core6.par.hetzner.com <http://core6.par.hetzner.com/> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17. core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com/> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18. core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com/> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19. ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com/> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20. vmh02.lavin.me.uk <http://vmh02.lavin.me.uk/> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22. http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk/> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
I've reached out to some known folk at Angola Cables. Will let you know if I hear back. Mark. On 5/25/23 14:01, Phil Lavin via NANOG wrote:
Cross-posting from outages list:
Hey Folks,
Seeing massive packet loss on routes from AWS to Hetzner today. First was AWS USW2 -> Hetzner (e.g. 88.99.88.69).
Traffic was transiting via AS37468 (Angola Cables) in Coresite IX.
Now got loss from AWS apse1/apse2. Traffic transiting via AS37468 on Equinix IX.
I e-mailed Hetzner NOC but they don’t seem entirely bothered, currently.
My traceroute [v0.94] jcasc-rtc01 (10.36.21.252) ->phil.lavin.me.uk <http://lavin.me.uk> 2023-05-25T11:50:13+0000 Keys: Help Display mode Restart statistics Order of fields quit Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. ip-10-36-17-199.ap-southeast- 0.0% 18 0.1 0.6 0.1 7.7 1.8 2. ec2-18-141-171-15.ap-southeas 0.0% 18 4.3 17.5 0.9 95.7 24.0 ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com <http://ec2-18-141-171-1.ap-southeast-1.compute.amazonaws.com> 3. (waiting for reply) 4. (waiting for reply) 5. (waiting for reply) 6. (waiting for reply) 7. (waiting for reply) 8. 100.65.10.161 0.0% 17 0.6 1.4 0.3 9.6 2.4 100.65.11.65 9. 150.222.108.75 0.0% 17 2.1 3.0 1.0 13.1 3.3 52.93.10.74 10. 150.222.108.82 0.0% 17 2.2 2.4 1.3 10.2 2.1 11. 52.93.11.115 0.0% 17 5.8 2.4 1.2 9.7 2.5 52.93.10.185 12.37468-sg1-ix.equinix.com <http://37468-sg1-ix.equinix.com> 68.8% 17 268.6 279.0 262.4 312.8 19.8 13. 102.219.127.3 93.8% 17 420.0 420.0 420.0 420.0 0.0 14. (waiting for reply) 15. 195.66.227.209 62.5% 17 467.9 456.9 440.3 488.2 18.5 16.core6.par.hetzner.com <http://core6.par.hetzner.com> 75.0% 17 455.6 464.1 455.6 486.3 14.8 17.core11.nbg1.hetzner.com <http://core11.nbg1.hetzner.com> 75.0% 17 459.5 467.5 445.9 497.3 21.8 18.core23.fsn1.hetzner.com <http://core23.fsn1.hetzner.com> 68.8% 17 478.4 441.0 390.0 478.4 32.0 19.ex9k1.dc1.fsn1.hetzner.com <http://ex9k1.dc1.fsn1.hetzner.com> 87.5% 17 460.4 454.2 448.1 460.4 8.7 20.vmh02.lavin.me.uk <http://vmh02.lavin.me.uk> 75.0% 17 433.3 424.2 382.7 441.9 27.9 21. (waiting for reply) 22.http-lb-01.lavin.me.uk <http://http-lb-01.lavin.me.uk> 75.0% 17 436.8 438.6 385.0 493.3 44.2
I’ve been out of the BGP game for a few years now. Anybody have visibility of the extent of the leaks?
Phil
participants (3)
-
Frederico Oliveira -
Mark Tinka -
Phil Lavin