Hello, We are currently experiencing massive packet loss from Amazon EU-West 1. This page http://ec2-reachability.amazonaws.com/ will show most of eu-west-1 as down but actually it is packet loss of 90+ %. I have found that if I shutdown our link to Hurricane Electric the problem disappears. Also it is only an IPv4 problem as IPv6 is fine (also going through Hurricane Electric). A traceroute shows everything ok until the first hop inside the Amazon network. The peering is apparently at AMS-IX. Right now we are ok because I am keeping the HE.Net connection down, but I managed to find another NLNOG Ring node that has the same problem: gigabit@casablanca01:~$ ping 52.16.0.2 PING 52.16.0.2 (52.16.0.2) 56(84) bytes of data. 64 bytes from 52.16.0.2: icmp_seq=2 ttl=244 time=35.1 ms 64 bytes from 52.16.0.2: icmp_seq=34 ttl=244 time=35.2 ms 64 bytes from 52.16.0.2: icmp_seq=44 ttl=244 time=35.2 ms 64 bytes from 52.16.0.2: icmp_seq=52 ttl=244 time=35.2 ms ^C --- 52.16.0.2 ping statistics --- 72 packets transmitted, 4 received, 94% packet loss, time 71228ms rtt min/avg/max/mdev = 35.170/35.220/35.251/0.232 ms That IP is one from eu-west-1 taken from the ec2-reachability page. When I ping it with the HE.Net link down there is 0% packet loss. With it up we are seeing something like the above. However I notice that Casablanca is going out through Telia not HE. The return path could very well be via HE of course, there is no way to know that. From all the RING nodes I checked only Casablanca01 seems to experience the problem, so it appears it is not very wide spread. Anyone know how to get in contact with Amazon and have them debug this issue? Regards, Baldur
HE is having issues in Europe. I spoke with them an hour ago, they said they were aware of the issue but didn't have an ETA. Based on the traces we saw issues with connections coming into London. Original Message From: baldur.norddahl@gmail.com Sent: April 23, 2017 6:52 AM To: nanog@nanog.org Subject: Amazon EU-West 1 trouble Hello, We are currently experiencing massive packet loss from Amazon EU-West 1. This page http://ec2-reachability.amazonaws.com/ will show most of eu-west-1 as down but actually it is packet loss of 90+ %. I have found that if I shutdown our link to Hurricane Electric the problem disappears. Also it is only an IPv4 problem as IPv6 is fine (also going through Hurricane Electric). A traceroute shows everything ok until the first hop inside the Amazon network. The peering is apparently at AMS-IX. Right now we are ok because I am keeping the HE.Net connection down, but I managed to find another NLNOG Ring node that has the same problem: gigabit@casablanca01:~$ ping 52.16.0.2 PING 52.16.0.2 (52.16.0.2) 56(84) bytes of data. 64 bytes from 52.16.0.2: icmp_seq=2 ttl=244 time=35.1 ms 64 bytes from 52.16.0.2: icmp_seq=34 ttl=244 time=35.2 ms 64 bytes from 52.16.0.2: icmp_seq=44 ttl=244 time=35.2 ms 64 bytes from 52.16.0.2: icmp_seq=52 ttl=244 time=35.2 ms ^C --- 52.16.0.2 ping statistics --- 72 packets transmitted, 4 received, 94% packet loss, time 71228ms rtt min/avg/max/mdev = 35.170/35.220/35.251/0.232 ms That IP is one from eu-west-1 taken from the ec2-reachability page. When I ping it with the HE.Net link down there is 0% packet loss. With it up we are seeing something like the above. However I notice that Casablanca is going out through Telia not HE. The return path could very well be via HE of course, there is no way to know that. From all the RING nodes I checked only Casablanca01 seems to experience the problem, so it appears it is not very wide spread. Anyone know how to get in contact with Amazon and have them debug this issue? Regards, Baldur
On Sun, Apr 23, 2017 at 12:52:08PM +0200, Baldur Norddahl wrote:
We are currently experiencing massive packet loss from Amazon EU-West 1. This page http://ec2-reachability.amazonaws.com/ will show most of eu-west-1 as down but actually it is packet loss of 90+ %.
I have found that if I shutdown our link to Hurricane Electric the problem disappears. Also it is only an IPv4 problem as IPv6 is fine (also going through Hurricane Electric).
Perhaps related to https://puck.nether.net/pipermail/outages/2017-April/010340.html Kind regards, Job
Issue resolved as 07:32 EST. On Sun, Apr 23, 2017 at 7:00 AM, Job Snijders <job@instituut.net> wrote:
We are currently experiencing massive packet loss from Amazon EU-West 1. This page http://ec2-reachability.amazonaws.com/ will show most of eu-west-1 as down but actually it is packet loss of 90+ %.
I have found that if I shutdown our link to Hurricane Electric the
On Sun, Apr 23, 2017 at 12:52:08PM +0200, Baldur Norddahl wrote: problem
disappears. Also it is only an IPv4 problem as IPv6 is fine (also going through Hurricane Electric).
Perhaps related to https://puck.nether.net/pipermail/outages/2017-April/010340.html
Kind regards,
Job
participants (3)
-
Baldur Norddahl -
Dovid Bender -
Job Snijders