AWS hosted sites like slack unreachable
Hi Nanog, We are seeing this weird issue in one part of the network. Customers in one public subnet are not able to reach certain websites suddenly which are hosted in AWS like slack.com, bill.com.. We changed the subnet to new one and issue resolved, after 48 hours, we have the same issue again. We are not AWS customer, so can't call them, but what are our options? Thanks, Margi
You didn’t specify anything that would be useful to narrow down the issue (i.e. location, asn, error codes, etc) - We had a somewhat similar issue at DET-IX with routes to us-east-1 and us-east-2 seeing a lot of packet loss, but AWS eventually just de-peered the exchange entirely since it was an issue with their equipment.
On Jul 20, 2023, at 5:17 PM, Margi Varia via NANOG <nanog@nanog.org> wrote:
Hi Nanog,
We are seeing this weird issue in one part of the network. Customers in one public subnet are not able to reach certain websites suddenly which are hosted in AWS like slack.com <http://slack.com/>, bill.com <http://bill.com/>..
We changed the subnet to new one and issue resolved, after 48 hours, we have the same issue again. We are not AWS customer, so can't call them, but what are our options?
Thanks, Margi
We have seen this in our consulting business with a large number of smaller ISPs both FISP and WISPS Often it is due to traffic leaving the network they believe to be an attack. If you let them know the Network Blocks, ASN, etc in an email to abuse@amazonaws.com they are very responsive. I would suggest running a simple netflow and see what might be going outbound to them as well. There is a good chance you will see an outlier or two in the netflow should it be an abuse issue. I hope that helps Glenn S. Kelley, I am a Connectivity.Engineer Text and Voice Direct: 740-206-9624 a Division of CreatingNet.Works IMPORTANT: The contents of this email and any attachments are confidential. They are intended for the named recipient(s) only. If you have received this email by mistake, please notify Glenn Kelley, the sender, immediately and do not disclose the contents to anyone or make copies thereof. On Thu, Jul 20, 2023 at 5:32 PM Daniel Marks via NANOG <nanog@nanog.org> wrote:
You didn’t specify anything that would be useful to narrow down the issue (i.e. location, asn, error codes, etc) - We had a somewhat similar issue at DET-IX with routes to us-east-1 and us-east-2 seeing a lot of packet loss, but AWS eventually just de-peered the exchange entirely since it was an issue with their equipment.
On Jul 20, 2023, at 5:17 PM, Margi Varia via NANOG <nanog@nanog.org> wrote:
Hi Nanog,
We are seeing this weird issue in one part of the network. Customers in one public subnet are not able to reach certain websites suddenly which are hosted in AWS like slack.com, bill.com..
We changed the subnet to new one and issue resolved, after 48 hours, we have the same issue again. We are not AWS customer, so can't call them, but what are our options?
Thanks, Margi
Margi, I ran into that years ago with AWS. I had a service provider clearing calls for me, and they were hosted on AWS. Kept pushing my service provider to open tickets with AWS. The issue would resolve for a day, then return, etc..... There was no permanent resolution offered by AWS. The issue kept re-emerging. I wasn't a paying customer of AWS, so I had to find another solution. The solution that I was forced to use was to set up a proxy on another network. I built a virtual server (I used DigitalOcean), set it up to proxy that specific traffic, and I had to bounce all the traffic off of that proxy to get in/out of AWS. Keep that solution in your back pocket if you don't get this cleared up. Good Luck, Pete Stage2 "Survivor Island" Bronze Medal Winner On 7/20/23 17:31, Daniel Marks via NANOG wrote:
You didn’t specify anything that would be useful to narrow down the issue (i.e. location, asn, error codes, etc) - We had a somewhat similar issue at DET-IX with routes to us-east-1 and us-east-2 seeing a lot of packet loss, but AWS eventually just de-peered the exchange entirely since it was an issue with their equipment.
On Jul 20, 2023, at 5:17 PM, Margi Varia via NANOG <nanog@nanog.org> wrote:
Hi Nanog,
We are seeing this weird issue in one part of the network. Customers in one public subnet are not able to reach certain websites suddenly which are hosted in AWS likeslack.com <http://slack.com/>,bill.com <http://bill.com/>..
We changed the subnet to new one and issue resolved, after 48 hours, we have the same issue again. We are not AWS customer, so can't call them, but what are our options?
Thanks, Margi
participants (4)
-
d＠nielmarks.com -
Glenn Kelley -
Margi Varia -
Pete Rohrman