Connectivity issue between Verizon and Amazon EC2
I'm short some important details on this one, but hopefully can fill in more shortly. We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits. The issue is reproducible on both our production Gigabit circuit as well as a consumer grade Verizion FIOS line. Speeds are normal (10MB/sec plus) via non-Verizon circuits we've tested. Source IP's are in the 198.102.62.0/24 range and destination on the EC2 side is 54.197.239.228. I'm not sure in which availability zone the latter IP sits, but hope to find out shortly. MTR traceroute details are as follows: Host Loss% Snt Drop Avg Best Wrst StDev 1. 198.102.62.253 0.0% 526 0 0.2 0.2 0.5 0.0 2. 152.179.250.141 0.0% 526 0 14.1 7.0 19.4 3.6 3. 140.222.225.135 37.5% 526 197 7.7 6.8 35.8 1.9 4. 129.250.8.85 0.0% 526 0 8.1 7.4 11.7 0.3 5. 129.250.2.229 10.3% 525 54 11.4 7.1 85.7 9.6 6. 129.250.2.169 41.5% 525 218 63.0 45.5 130.7 10.3 7. 129.250.2.154 0.2% 525 1 59.9 44.5 69.0 4.0 8. ??? 9. 54.240.229.96 7.8% 525 41 76.6 71.3 119.9 8.6 54.240.229.104 54.240.229.106 10. 54.240.229.2 6.9% 525 36 74.7 71.6 109.1 4.9 54.240.229.4 54.240.229.20 54.240.229.8 54.240.229.14 54.240.228.254 54.240.229.16 54.240.229.10 11. 54.240.229.174 5.5% 525 29 76.0 71.7 109.0 7.3 54.240.229.162 54.240.229.160 54.240.229.170 54.240.229.172 54.240.229.168 54.240.229.164 12. 54.240.228.167 94.5% 525 495 76.4 71.7 126.0 11.6 54.240.228.169 54.240.228.165 54.240.228.163 13. 72.21.220.108 5.1% 525 27 75.2 71.3 112.6 6.8 205.251.244.12 72.21.220.8 205.251.244.64 72.21.220.96 205.251.244.8 72.21.220.6 205.251.244.4 14. 72.21.220.45 9.0% 525 47 74.0 71.6 199.5 8.5 72.21.220.149 72.21.220.29 72.21.220.125 72.21.220.37 72.21.220.61 72.21.220.2 72.21.220.69 15. 72.21.222.33 10.5% 525 55 73.4 71.5 87.1 1.5 205.251.245.65 72.21.222.149 72.21.222.35 72.21.220.29 72.21.222.131 72.21.222.147 72.21.220.37 16. 205.251.245.65 93.9% 525 492 73.1 72.2 76.2 1.2 72.21.222.35 72.21.222.131 17. ??? 18. ??? 19. 216.182.224.79 13.5% 524 71 77.9 72.4 101.2 5.4 216.182.224.81 216.182.224.95 216.182.224.77 20. 216.182.224.81 94.1% 524 492 77.9 72.8 93.0 6.3 216.182.224.95 216.182.224.77 21. ??? The 140.222.225.135 shows up in the traceroutes via our Verizon Business FIOS line as well. Will be opening a ticket with both Verizon and AWS to assist, but hoping someone out there can take a look or chime in. Feel free to reply off list. Thanks, Ray
On Jul 22, 2014, at 10:31 AM, Ray Van Dolson <rvandolson@esri.com> wrote:
We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
Have you tried dorking around with your MTU to see if that makes a difference? ---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Equo ne credite, Teucri. -- Laocoön
On Tue, Jul 22, 2014 at 10:41:25AM +0700, Roland Dobbins wrote:
On Jul 22, 2014, at 10:31 AM, Ray Van Dolson <rvandolson@esri.com> wrote:
We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
Have you tried dorking around with your MTU to see if that makes a difference?
Not in a position to easily test that tonight (PDT) but will do so tomorrow. Ray
On Mon, Jul 21, 2014 at 08:47:36PM -0700, Ray Van Dolson wrote:
On Tue, Jul 22, 2014 at 10:41:25AM +0700, Roland Dobbins wrote:
On Jul 22, 2014, at 10:31 AM, Ray Van Dolson <rvandolson@esri.com> wrote:
We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
Have you tried dorking around with your MTU to see if that makes a difference?
Not in a position to easily test that tonight (PDT) but will do so tomorrow.
Monkeyed with the MTU -- 500, 100, 1200 -- no real observed difference and still seeing lots of retransmits. Ray
I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route. I would provide an mtr, however my network configuration is something mtr doesn't support. Cheers! -Tim On Jul 21, 2014 8:44 PM, "Roland Dobbins" <rdobbins@arbor.net> wrote:
On Jul 22, 2014, at 10:31 AM, Ray Van Dolson <rvandolson@esri.com> wrote:
We're seeing poor performance (very slow download speeds -- <
100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
Have you tried dorking around with your MTU to see if that makes a
difference?
---------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Equo ne credite, Teucri.
-- Laocoön
Realized I sent the reply to Roland. Apologies. Here it is in full: #### I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route. I would provide an mtr, however my network configuration is something mtr doesn't support. Cheers! -Tim On Jul 21, 2014 8:34 PM, "Ray Van Dolson" <rvandolson@esri.com> wrote:
I'm short some important details on this one, but hopefully can fill in more shortly.
We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits. The issue is reproducible on both our production Gigabit circuit as well as a consumer grade Verizion FIOS line.
Speeds are normal (10MB/sec plus) via non-Verizon circuits we've tested.
Source IP's are in the 198.102.62.0/24 range and destination on the EC2 side is 54.197.239.228. I'm not sure in which availability zone the latter IP sits, but hope to find out shortly.
MTR traceroute details are as follows:
Host Loss% Snt Drop Avg Best Wrst StDev 1. 198.102.62.253 0.0% 526 0 0.2 0.2 0.5 0.0 2. 152.179.250.141 0.0% 526 0 14.1 7.0 19.4 3.6 3. 140.222.225.135 37.5% 526 197 7.7 6.8 35.8 1.9 4. 129.250.8.85 0.0% 526 0 8.1 7.4 11.7 0.3 5. 129.250.2.229 10.3% 525 54 11.4 7.1 85.7 9.6 6. 129.250.2.169 41.5% 525 218 63.0 45.5 130.7 10.3 7. 129.250.2.154 0.2% 525 1 59.9 44.5 69.0 4.0 8. ??? 9. 54.240.229.96 7.8% 525 41 76.6 71.3 119.9 8.6 54.240.229.104 54.240.229.106 10. 54.240.229.2 6.9% 525 36 74.7 71.6 109.1 4.9 54.240.229.4 54.240.229.20 54.240.229.8 54.240.229.14 54.240.228.254 54.240.229.16 54.240.229.10 11. 54.240.229.174 5.5% 525 29 76.0 71.7 109.0 7.3 54.240.229.162 54.240.229.160 54.240.229.170 54.240.229.172 54.240.229.168 54.240.229.164 12. 54.240.228.167 94.5% 525 495 76.4 71.7 126.0 11.6 54.240.228.169 54.240.228.165 54.240.228.163 13. 72.21.220.108 5.1% 525 27 75.2 71.3 112.6 6.8 205.251.244.12 72.21.220.8 205.251.244.64 72.21.220.96 205.251.244.8 72.21.220.6 205.251.244.4 14. 72.21.220.45 9.0% 525 47 74.0 71.6 199.5 8.5 72.21.220.149 72.21.220.29 72.21.220.125 72.21.220.37 72.21.220.61 72.21.220.2 72.21.220.69 15. 72.21.222.33 10.5% 525 55 73.4 71.5 87.1 1.5 205.251.245.65 72.21.222.149 72.21.222.35 72.21.220.29 72.21.222.131 72.21.222.147 72.21.220.37 16. 205.251.245.65 93.9% 525 492 73.1 72.2 76.2 1.2 72.21.222.35 72.21.222.131 17. ??? 18. ??? 19. 216.182.224.79 13.5% 524 71 77.9 72.4 101.2 5.4 216.182.224.81 216.182.224.95 216.182.224.77 20. 216.182.224.81 94.1% 524 492 77.9 72.8 93.0 6.3 216.182.224.95 216.182.224.77 21. ???
The 140.222.225.135 shows up in the traceroutes via our Verizon Business FIOS line as well.
Will be opening a ticket with both Verizon and AWS to assist, but hoping someone out there can take a look or chime in. Feel free to reply off list.
Thanks, Ray
Others appear to have repoted this. Seems like Verizon is pointing at AWS: https://forums.aws.amazon.com/thread.jspa?messageID=558094 Ray On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
Realized I sent the reply to Roland. Apologies.
Here it is in full:
####
I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route.
I would provide an mtr, however my network configuration is something mtr doesn't support.
Cheers! -Tim
On Jul 21, 2014 8:34 PM, "Ray Van Dolson" <rvandolson@esri.com> wrote:
I'm short some important details on this one, but hopefully can fill in more shortly.
We're seeing poor performance (very slow download speeds -- < 100KB/sec) to certain EC2 instances via our Verizon hosted circuits. The issue is reproducible on both our production Gigabit circuit as well as a consumer grade Verizion FIOS line.
Speeds are normal (10MB/sec plus) via non-Verizon circuits we've tested.
Source IP's are in the 198.102.62.0/24 range and destination on the EC2 side is 54.197.239.228. I'm not sure in which availability zone the latter IP sits, but hope to find out shortly.
MTR traceroute details are as follows:
Host Loss% Snt Drop Avg Best Wrst StDev 1. 198.102.62.253 0.0% 526 0 0.2 0.2 0.5 0.0 2. 152.179.250.141 0.0% 526 0 14.1 7.0 19.4 3.6 3. 140.222.225.135 37.5% 526 197 7.7 6.8 35.8 1.9 4. 129.250.8.85 0.0% 526 0 8.1 7.4 11.7 0.3 5. 129.250.2.229 10.3% 525 54 11.4 7.1 85.7 9.6 6. 129.250.2.169 41.5% 525 218 63.0 45.5 130.7 10.3 7. 129.250.2.154 0.2% 525 1 59.9 44.5 69.0 4.0 8. ??? 9. 54.240.229.96 7.8% 525 41 76.6 71.3 119.9 8.6 54.240.229.104 54.240.229.106 10. 54.240.229.2 6.9% 525 36 74.7 71.6 109.1 4.9 54.240.229.4 54.240.229.20 54.240.229.8 54.240.229.14 54.240.228.254 54.240.229.16 54.240.229.10 11. 54.240.229.174 5.5% 525 29 76.0 71.7 109.0 7.3 54.240.229.162 54.240.229.160 54.240.229.170 54.240.229.172 54.240.229.168 54.240.229.164 12. 54.240.228.167 94.5% 525 495 76.4 71.7 126.0 11.6 54.240.228.169 54.240.228.165 54.240.228.163 13. 72.21.220.108 5.1% 525 27 75.2 71.3 112.6 6.8 205.251.244.12 72.21.220.8 205.251.244.64 72.21.220.96 205.251.244.8 72.21.220.6 205.251.244.4 14. 72.21.220.45 9.0% 525 47 74.0 71.6 199.5 8.5 72.21.220.149 72.21.220.29 72.21.220.125 72.21.220.37 72.21.220.61 72.21.220.2 72.21.220.69 15. 72.21.222.33 10.5% 525 55 73.4 71.5 87.1 1.5 205.251.245.65 72.21.222.149 72.21.222.35 72.21.220.29 72.21.222.131 72.21.222.147 72.21.220.37 16. 205.251.245.65 93.9% 525 492 73.1 72.2 76.2 1.2 72.21.222.35 72.21.222.131 17. ??? 18. ??? 19. 216.182.224.79 13.5% 524 71 77.9 72.4 101.2 5.4 216.182.224.81 216.182.224.95 216.182.224.77 20. 216.182.224.81 94.1% 524 492 77.9 72.8 93.0 6.3 216.182.224.95 216.182.224.77 21. ???
The 140.222.225.135 shows up in the traceroutes via our Verizon Business FIOS line as well.
Will be opening a ticket with both Verizon and AWS to assist, but hoping someone out there can take a look or chime in. Feel free to reply off list.
Thanks, Ray
On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
Others appear to be having similar issues. Seems like Verizon is pointing at AWS:
https://forums.aws.amazon.com/thread.jspa?messageID=558094
Ray
On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
Realized I sent the reply to Roland. Apologies.
Here it is in full:
####
I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route.
I would provide an mtr, however my network configuration is something mtr doesn't support.
Cheers! -Tim
Update on this: - We have a ticket open with both AWS and Verizon. - AWS has responded and felt the issue was with Verizon, but notified their network team and asked them to investigate further. - Nothing back from Verizon yet (anyone here have a Verizon NOC contact?) In the interim, the issuer persists. Thanks, Ray
On Tue, Jul 22, 2014 at 05:29:55PM -0700, Ray Van Dolson wrote:
On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
Others appear to be having similar issues. Seems like Verizon is pointing at AWS:
https://forums.aws.amazon.com/thread.jspa?messageID=558094
Ray
On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
Realized I sent the reply to Roland. Apologies.
Here it is in full:
####
I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route.
I would provide an mtr, however my network configuration is something mtr doesn't support.
Cheers! -Tim
Update on this:
- We have a ticket open with both AWS and Verizon. - AWS has responded and felt the issue was with Verizon, but notified their network team and asked them to investigate further. - Nothing back from Verizon yet (anyone here have a Verizon NOC contact?)
In the interim, the issue persists.
Further update -- Verizon indicates that the issue is related to saturation on a peering link between themselves and NTT. Verizon is pointing to the NTT side as the source of the saturation / congestion. We don't have a direct customer relationship with NTT so am hoping someone on this list may be able to pass this information along or investigate on our behalf. Ray
On Tue, Jul 22, 2014 at 7:40 PM, Ray Van Dolson <rvandolson@esri.com> wrote:
[...] Further update -- Verizon indicates that the issue is related to saturation on a peering link between themselves and NTT. Verizon is pointing to the NTT side as the source of the saturation / congestion.
So, Verizon is saying that Level3 into them is congested, NTT into them is congested...sounds like there might be a bit of a trend happening here. I wonder if Verizon is having trouble provisioning sufficient capacity to support all of their customers? If so, it would kinda suck to be stuck being their customer at the moment if you can't get to places you want to reach. :(
We don't have a direct customer relationship with NTT so am hoping someone on this list may be able to pass this information along or investigate on our behalf.
Ray
I'm sure there's NTT folks watching the thread go past, but it's unlikely they'd be in a position to say anything in a public forum like this one way or the other. ^_^; Matt
On Jul 23, 2014, at 3:23 AM, Matthew Petach <mpetach@netflight.com> wrote:
We don't have a direct customer relationship with NTT so am hoping someone on this list may be able to pass this information along or investigate on our behalf.
Ray
I'm sure there's NTT folks watching the thread go past, but it's unlikely they'd be in a position to say anything in a public forum like this one way or the other. ^_^;
Is there anything to be said that adds anything to what is already a well established situation regarding Verizon vs. much of the Internet? -dorian
On Jul 23, 2014 12:34 AM, "Dorian Kim" <dorian@blackrose.org> wrote:
On Jul 23, 2014, at 3:23 AM, Matthew Petach <mpetach@netflight.com> wrote:
We don't have a direct customer relationship with NTT so am hoping someone on this list may be able to pass this information along or investigate on our behalf.
Ray
I'm sure there's NTT folks watching the thread go past, but it's unlikely they'd be in a position to say anything in a public forum like this one way or the other. ^_^;
Is there anything to be said that adds anything to what is already a well established situation regarding Verizon vs. much of the Internet?
-dorian
Fooled me once shame on you. Fooled me twice... Dont by service from companies that allow peering wars to happen at paying customers expense (verzon, cogent, ...)
On Wed, 23 Jul 2014 06:31:06 -0700, Ca By said:
Fooled me once shame on you. Fooled me twice... Dont by service from companies that allow peering wars to happen at paying customers expense (verzon, cogent, ...)
There's one coax coming into my domicile, and the owner of the other end of the cable isn't ripping me off *too* much, and the service pretty much works as advertised. There's one pair of twisted copper coming to a punchdown block just outside, and neither the owner of the copper or their competitors seem to be motivated to provide a DSL package I consider really acceptable. Now, there's at least one DSL provider that has an offering that I could deal with if the owner of the cable was in fact unable to deliver the service, but under these conditions I'm not feeling very motivated to change providers to make a political statement....
Am 23.07.2014 09:23, schrieb Matthew Petach:
So, Verizon is saying that Level3 into them is congested, NTT into them is congested...sounds like there might be a bit of a trend happening here.
They (Verizon) should issue a list of those which are _not_ congested. I guess the list would be rather short... *SCNR* -- Fredy Kuenzler --------------------- Fiber7. No Limits. https://www.fiber7.ch --------------------- Init7 (Switzerland) Ltd. AS13030 St.-Georgen-Strasse 70 CH-8400 Winterthur Skype: flyingpotato Phone: +41 44 315 4400 Fax: +41 44 315 4401 Twitter: @init7 / @kuenzler http://www.init7.net/
On Tue, Jul 22, 2014 at 07:40:26PM -0700, Ray Van Dolson wrote:
On Tue, Jul 22, 2014 at 05:29:55PM -0700, Ray Van Dolson wrote:
On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
Others appear to be having similar issues. Seems like Verizon is pointing at AWS:
https://forums.aws.amazon.com/thread.jspa?messageID=558094
Ray
On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
Realized I sent the reply to Roland. Apologies.
Here it is in full:
####
I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's Fremont 2 location (Linode). Looks to be deep within Amanzon's network based on changes in latency in a simple trace route.
I would provide an mtr, however my network configuration is something mtr doesn't support.
Cheers! -Tim
Update on this:
- We have a ticket open with both AWS and Verizon. - AWS has responded and felt the issue was with Verizon, but notified their network team and asked them to investigate further. - Nothing back from Verizon yet (anyone here have a Verizon NOC contact?)
In the interim, the issue persists.
Further update -- Verizon indicates that the issue is related to saturation on a peering link between themselves and NTT. Verizon is pointing to the NTT side as the source of the saturation / congestion.
We don't have a direct customer relationship with NTT so am hoping someone on this list may be able to pass this information along or investigate on our behalf.
Ray
To close the loop on this one, Amazon made a change for us and shifted their peering point from NTT Ashburn to NTT Dallas. This helped out tremendously, and although we still see times where things are "slower", it's at least not 100KB/sec slow. :) Appreciate all the responses received. Ray
participants (8)
-
Ca By -
Dorian Kim -
Fredy Kuenzler -
Matthew Petach -
Ray Van Dolson -
Roland Dobbins -
Tim Heckman -
Valdis.Kletnieks＠vt.edu