Algar looking Glass: http://lg.ctbc.com.br/lg/ I found the response from the GC engineer. It won't mean much cut&pasted verbatim and out of context, so I'll just summarize. Basically, I was seeing vastly different response times to given hosts in the same subnet on CTBC's network. My source ip was the same. Traceroute revealed to me the packets taking different paths, after hitting a particular GLBX router. The GC engineer said CTBC is a customer that hangs off of this particular router, and that traffic was hitting an interface and hair-pinning back out to the customer's segment. He pointed to possible congestion on CTBCs switching fabric as the cause for the varied response times (conceivable) and different routes (um...ok maybe some kind of load-balancing at play). I managed to call CTBC (Algar) and confirmed they were experiencing congestion issues and they had a scheduled circuit capacity upgrade with GLBX in a few weeks. As a network admin, i find it sometimes sadly humorous how we can poke and prod at a problem, go through various carrier support channels and scratch our heads for hours/days when all it would normally take is a RO userid on a couple routers in the path to figure things out. that's not too much to ask, right? ;) -matt
CC: vinny@abellohome.net; nanog@nanog.org From: the76posse@gmail.com Subject: Re: Connectivity to Brazil Date: Wed, 2 Feb 2011 08:07:14 -0500 To: gourmetcisco@hotmail.com
That thread detail would be very interesting to me. Thanks for the heads up.
Sent from my iPhone
On Feb 2, 2011, at 7:14 AM, Matt Disuko <gourmetcisco@hotmail.com> wrote:
Very interesting. I have had similar issues with connectivity to my Brazil office, and oddly enough it involved GBLX and CTBC (now called Algar Telecom). I also vastly divergent paths to a couple hosts in the same subnet. I ended up communicating with GBLX via email (who were actually really great in corresponding with me)...the engineer pointed to some sort of link capacity issue...i'll dig up the thread...
Date: Wed, 2 Feb 2011 01:21:09 -0500 From: vinny@abellohome.net Subject: Re: Connectivity to Brazil To: the76posse@gmail.com CC: nanog@nanog.org
We saw similar issues with IKE through Global Crossing (as odd as that sounds) out of the NYC market at the same time. We routed around them and problem solved. Still scratching our heads on that one... In my experiences, GLBX has numerous odd issues to the point where it's become a bad joke anytime something breaks with connectivity... we blame them. It's kind of not funny though because it's almost always true. Taking them out of the equation usually fixes the problem. One of our customers who is frequently affected by GBLX problems jumps to the (often correct) conclusion that they are causing problems. :-/
-Vinny
On Feb 1, 2011, at 3:57 PM, Steve Danelli wrote:
Thanks for the response.
Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess)
Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist
Thanks to all for their feedback so far.
SD
On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks@vt.edu wrote:
On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:
Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.
Has IKE been known to work to that location before? Or is this something new? My first guess is some chucklehead banana-eater at the service provider either fat-fingered the firewall config, or semi-intentionally blocked it because it was "traffic on a protocol/port number they didn't understand so it must be evil".
Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:
Anyone have any insight on to what may be occurring?
The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying to do some load-balancing across 2 paths, and it's using the source IP as a major part of the selector function ("route to round-robin interface source-IP mod N" or similar?).
The other possibility is your two traceroutes happened to catch a routing flap in progress (obviously not the case if the two routes are remaining stable).
Sorry I can't be more helpful than that...