OMG !!!...l know this sounds weird and you wouldn't believe me...Yes, am really stuck out here in the UK and it's so devastating at the moment i really need your help l wish l could cal but l don't have access to phone at the moment , I really need your help to get myself out of this mess. The hotel management has been kind to let have an access to a library to get across to anybody to help me since my luggage was also taken away.Hope to hear from you soon ___ Kelvin On Tue, Feb 1, 2011 at 3:10 PM, isabel dias <isabeldias1@yahoo.com> wrote:

who are you?

________________________________ From: Steve Danelli <the76posse@gmail.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Tue, February 1, 2011 1:54:47 PM Subject: Connectivity to Brazil

Hello all!!!  First post here

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:

For dest  x.y.x.52

  16 ms    3 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-0-1-0.er1.lga5.us.above.net [64.125.27.61]   12 ms    3 ms    13 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  118 ms  118 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   136 ms  137 ms  136 ms ctbc-multimidia-data-net-s-a.gigabitethernet1-2.ar5.gru1.gblx.net [207.138.94.102]   157 ms  136 ms  138 ms  xe-3-2-0-0.core-b.ula001.ctbc.com.br [201.48.44.165]   132 ms  132 ms  141 ms  ge-3-0-0-0.core-b.spo511.ctbc.com.br [201.48.44.14]   135 ms  133 ms  134 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

For dest  x.y.x.53

  3 ms    2 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-1-1-0.er1.lga5.us.above.net [64.125.26.162]   19 ms    3 ms    12 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  117 ms  117 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   117 ms  117 ms  118 ms  64.209.106.170   118 ms  118 ms  118 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

Anyone have any insight on to what may be occurring?

Sent from my iPhone

-- Cheers, Kevin ================================================================  :: :: Kevin Hodle | http://www.linkedin.com/in/kevinhodle  :: :: PGP Key ID  | fingerprint  :: :: 0x803F24BE  | 1094 FB06 837F 2FAB C86B E4BE 4680 3679 803F 24BE "Elegance is not a dispensable luxury but a factor that decides between success and failure. " -Edsgar Dijkstra ================================================================

I just need you to help me with some cash just till i get back home On Tue, Feb 1, 2011 at 3:21 PM, Steve Danelli <the76posse@gmail.com> wrote:

Isabel - I am a network engineer working for a small financial firm outside Philadelphia.

Can you offer any insight?

On Feb 1, 2011, at 9:10 AM, isabel dias <isabeldias1@yahoo.com> wrote:

...

who are you?

From: Steve Danelli <the76posse@gmail.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Tue, February 1, 2011 1:54:47 PM Subject: Connectivity to Brazil

Hello all!!!  First post here

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:

For dest  x.y.x.52

  16 ms    3 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-0-1-0.er1.lga5.us.above.net [64.125.27.61]   12 ms    3 ms    13 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  118 ms  118 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   136 ms  137 ms  136 ms  ctbc-multimidia-data-net-s-a.gigabitethernet1-2.ar5.gru1.gblx.net [207.138.94.102]   157 ms  136 ms  138 ms  xe-3-2-0-0.core-b.ula001.ctbc.com.br [201.48.44.165]   132 ms  132 ms  141 ms  ge-3-0-0-0.core-b.spo511.ctbc.com.br [201.48.44.14]   135 ms  133 ms  134 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

For dest  x.y.x.53

  3 ms    2 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-1-1-0.er1.lga5.us.above.net [64.125.26.162]   19 ms    3 ms    12 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  117 ms  117 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   117 ms  117 ms  118 ms  64.209.106.170   118 ms  118 ms  118 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

Anyone have any insight on to what may be occurring?

Sent from my iPhone

-- Cheers, Kevin ================================================================  :: :: Kevin Hodle | http://www.linkedin.com/in/kevinhodle  :: :: PGP Key ID  | fingerprint  :: :: 0x803F24BE  | 1094 FB06 837F 2FAB C86B E4BE 4680 3679 803F 24BE "Elegance is not a dispensable luxury but a factor that decides between success and failure. " -Edsgar Dijkstra ================================================================

normally address to service providers that sit behind a block of addresses and are able to provide visibility from their networks ,  looking glasses are as usefull but if you need someone to be as cleaver as the ones on training at Holloway just be more specific as there are plenty of options around the world and also in the UK ......:-) ________________________________ From: isabel dias <isabeldias1@yahoo.com> To: Kevin Hodle <kevin.hodle@gmail.com>; Steve Danelli <the76posse@gmail.com> Cc: "nanog@nanog.org" <nanog@nanog.org> Sent: Tue, February 1, 2011 4:16:42 PM Subject: Re: Connectivity to Brazil I still don't know where you are and the simulation you are doing .....can you be more specific ? ________________________________ From: Kevin Hodle <kevin.hodle@gmail.com> To: Steve Danelli <the76posse@gmail.com> Cc: isabel dias <isabeldias1@yahoo.com>; "nanog@nanog.org" <nanog@nanog.org> Sent: Tue, February 1, 2011 2:27:41 PM Subject: Re: Connectivity to Brazil I just need you to help me with some cash just till i get back home On Tue, Feb 1, 2011 at 3:21 PM, Steve Danelli <the76posse@gmail.com> wrote:

Isabel - I am a network engineer working for a small financial firm outside Philadelphia.

Can you offer any insight?

On Feb 1, 2011, at 9:10 AM, isabel dias <isabeldias1@yahoo.com> wrote:

...

who are you?

From: Steve Danelli <the76posse@gmail.com> To: "nanog@nanog.org" <nanog@nanog.org> Sent: Tue, February 1, 2011 1:54:47 PM Subject: Connectivity to Brazil

Hello all!!!  First post here

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our

...

Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the

...

IPSEC tunnel between our gateways.

Also something else is awry, for two given hosts on the same subnet (x.y.z.52

...

and x.y.z.53), they take two wildly divergent paths:

For dest  x.y.x.52

  16 ms    3 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-0-1-0.er1.lga5.us.above.net [64.125.27.61]   12 ms    3 ms    13 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  118 ms  118 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   136 ms  137 ms  136 ms  ctbc-multimidia-data-net-s-a.gigabitethernet1-2.ar5.gru1.gblx.net [207.138.94.102]   157 ms  136 ms  138 ms  xe-3-2-0-0.core-b.ula001.ctbc.com.br [201.48.44.165]   132 ms  132 ms  141 ms  ge-3-0-0-0.core-b.spo511.ctbc.com.br [201.48.44.14]   135 ms  133 ms  134 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

For dest  x.y.x.53

  3 ms    2 ms    3 ms  xe-2-3-0.cr1.lga5.us.above.net [64.125.31.34]   2 ms    2 ms    2 ms  xe-1-1-0.er1.lga5.us.above.net [64.125.26.162]   19 ms    3 ms    12 ms  e2-4-10000m.ar9.nyc1.gblx.net [208.178.58.197]   117 ms  117 ms  117 ms  te3-4-10g.asr1.gru1.gblx.net [67.16.142.238]   117 ms  117 ms  118 ms  64.209.106.170   118 ms  118 ms  118 ms  ae1-0.edge-a.spo511.ctbc.com.br [201.48.44.93]

Anyone have any insight on to what may be occurring?

Sent from my iPhone

-- Cheers, Kevin ================================================================  :: :: Kevin Hodle | http://www.linkedin.com/in/kevinhodle  :: :: PGP Key ID  | fingerprint  :: :: 0x803F24BE  | 1094 FB06 837F 2FAB C86B E4BE 4680 3679 803F 24BE "Elegance is not a dispensable luxury but a factor that decides between success and failure. " -Edsgar Dijkstra ================================================================

Thanks for the response. Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess) Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist Thanks to all for their feedback so far. SD On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks@vt.edu wrote:

On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:

...

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.

Has IKE been known to work to that location before? Or is this something new? My first guess is some chucklehead banana-eater at the service provider either fat-fingered the firewall config, or semi-intentionally blocked it because it was "traffic on a protocol/port number they didn't understand so it must be evil".

...

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:

...

Anyone have any insight on to what may be occurring?

The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying to do some load-balancing across 2 paths, and it's using the source IP as a major part of the selector function ("route to round-robin interface source-IP mod N" or similar?).

The other possibility is your two traceroutes happened to catch a routing flap in progress (obviously not the case if the two routes are remaining stable).

Sorry I can't be more helpful than that...

Very interesting. I have had similar issues with connectivity to my Brazil office, and oddly enough it involved GBLX and CTBC (now called Algar Telecom). I also vastly divergent paths to a couple hosts in the same subnet. I ended up communicating with GBLX via email (who were actually really great in corresponding with me)...the engineer pointed to some sort of link capacity issue...i'll dig up the thread...

Date: Wed, 2 Feb 2011 01:21:09 -0500 From: vinny@abellohome.net Subject: Re: Connectivity to Brazil To: the76posse@gmail.com CC: nanog@nanog.org

We saw similar issues with IKE through Global Crossing (as odd as that sounds) out of the NYC market at the same time. We routed around them and problem solved. Still scratching our heads on that one... In my experiences, GLBX has numerous odd issues to the point where it's become a bad joke anytime something breaks with connectivity... we blame them. It's kind of not funny though because it's almost always true. Taking them out of the equation usually fixes the problem. One of our customers who is frequently affected by GBLX problems jumps to the (often correct) conclusion that they are causing problems. :-/

-Vinny

On Feb 1, 2011, at 3:57 PM, Steve Danelli wrote:

...

Thanks for the response.

Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess)

Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist

Thanks to all for their feedback so far.

SD

On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks@vt.edu wrote:

...

On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:

...

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.

Has IKE been known to work to that location before? Or is this something new? My first guess is some chucklehead banana-eater at the service provider either fat-fingered the firewall config, or semi-intentionally blocked it because it was "traffic on a protocol/port number they didn't understand so it must be evil".

...

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:

...

Anyone have any insight on to what may be occurring?

The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying to do some load-balancing across 2 paths, and it's using the source IP as a major part of the selector function ("route to round-robin interface source-IP mod N" or similar?).

The other possibility is your two traceroutes happened to catch a routing flap in progress (obviously not the case if the two routes are remaining stable).

Sorry I can't be more helpful than that...

Thanks Vinny - how did you route around? There seems to be one path from the US to Brazil via GBLX and CTBC. Are you leveraging leased connectivity? Thanks for the info!! SD Sent from my iPhone On Feb 2, 2011, at 1:21 AM, Vinny Abello <vinny@abellohome.net> wrote:

We saw similar issues with IKE through Global Crossing (as odd as that sounds) out of the NYC market at the same time. We routed around them and problem solved. Still scratching our heads on that one... In my experiences, GLBX has numerous odd issues to the point where it's become a bad joke anytime something breaks with connectivity... we blame them. It's kind of not funny though because it's almost always true. Taking them out of the equation usually fixes the problem. One of our customers who is frequently affected by GBLX problems jumps to the (often correct) conclusion that they are causing problems. :-/

-Vinny

On Feb 1, 2011, at 3:57 PM, Steve Danelli wrote:

...

Thanks for the response.

Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess)

Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist

Thanks to all for their feedback so far.

SD

On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks@vt.edu wrote:

...

On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:

...

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways.

Has IKE been known to work to that location before? Or is this something new? My first guess is some chucklehead banana-eater at the service provider either fat-fingered the firewall config, or semi-intentionally blocked it because it was "traffic on a protocol/port number they didn't understand so it must be evil".

...

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths:

...

Anyone have any insight on to what may be occurring?

The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying to do some load-balancing across 2 paths, and it's using the source IP as a major part of the selector function ("route to round-robin interface source-IP mod N" or similar?).

The other possibility is your two traceroutes happened to catch a routing flap in progress (obviously not the case if the two routes are remaining stable).

Sorry I can't be more helpful than that...

Very simply. :) We chose to stop accepting prefixes from and announcing prefixes to them. You could attempt this in more elaborate and less forceful ways if you're in the right position, but we encounter issues like this too much and it affects critical clients that cannot afford any downtime, and we have plenty of other transit connections. We are in a position where we have direct connectivity with them (which based on our track record won't be much longer), so it might be much easier for us. Otherwise you'd have to hope your upstream is the one connected to them and has communities available to tinker with to withdraw your tagged prefixes from being announced to them, and just change the local preference or however you prefer to do it on the inbound routes from your upstream, or better yet filter on as-path. -Vinny On 2/2/2011 8:08 AM, Steve Danelli wrote:

Thanks Vinny - how did you route around? There seems to be one path from the US to Brazil via GBLX and CTBC. Are you leveraging leased connectivity? Thanks for the info!!

SD

Sent from my iPhone

On Feb 2, 2011, at 1:21 AM, Vinny Abello <vinny@abellohome.net> wrote:

...

We saw similar issues with IKE through Global Crossing (as odd as that sounds) out of the NYC market at the same time. We routed around them and problem solved. Still scratching our heads on that one... In my experiences, GLBX has numerous odd issues to the point where it's become a bad joke anytime something breaks with connectivity... we blame them. It's kind of not funny though because it's almost always true. Taking them out of the equation usually fixes the problem. One of our customers who is frequently affected by GBLX problems jumps to the (often correct) conclusion that they are causing problems. :-/

-Vinny

On Feb 1, 2011, at 3:57 PM, Steve Danelli wrote:

...

Thanks for the response.

Ike had worked great up until Monday. The provider did a local test and our box saw the Ike packets so it appears to lie somewhere upstream. (GLBX may be a good guess)

Also - the paths are stable and we are sourcing from the same ip - very strange behaivor. Hope someone from GLBX or CTBC (or someone who had experienced an issue like this) can assist

Thanks to all for their feedback so far.

SD

On Feb 1, 2011, at 3:19 PM, Valdis.Kletnieks@vt.edu wrote:

...

On Tue, 01 Feb 2011 08:54:47 EST, Steve Danelli said:

...

Some carrier, somewhere between us and the service provider is selectively dropping the IKE packets originating from our VPN gateway and destined for our Brazil gateway. Other traffic is able to pass, as are the IKE packets coming back from Brazil to us. This is effectively preventing us from establishing the IPSEC tunnel between our gateways. Has IKE been known to work to that location before? Or is this something new? My first guess is some chucklehead banana-eater at the service provider either fat-fingered the firewall config, or semi-intentionally blocked it because it was "traffic on a protocol/port number they didn't understand so it must be evil".

...

Also something else is awry, for two given hosts on the same subnet (x.y.z.52 and x.y.z.53), they take two wildly divergent paths: Anyone have any insight on to what may be occurring? The paths appear to diverge at 67.16.142.238. I wonder if that's gear trying to do some load-balancing across 2 paths, and it's using the source IP as a major part of the selector function ("route to round-robin interface source-IP mod N" or similar?).

The other possibility is your two traceroutes happened to catch a routing flap in progress (obviously not the case if the two routes are remaining stable).

Sorry I can't be more helpful than that...