cogent update suppression, and routing loops
hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them? i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent. example: http://ryry.foursquare.com/image/0e0K1K0t0W2M it's been a bit of a frustrating experience talking to their noc to demonstrate it, but i'm able to duplicate it on demand. even pushing routes using their communities to offload the circuit takes forever to propagate even on their own looking-glasses. thx ryan
First time I'm seeing it, and I've been a Cogent client for quite a while. Have you tried getting in touch with their NOC yet? They're one of the most responsive in the industry. On 10/3/2014 午前 01:03, ryanL wrote:
hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them?
i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent.
example: http://ryry.foursquare.com/image/0e0K1K0t0W2M
it's been a bit of a frustrating experience talking to their noc to demonstrate it, but i'm able to duplicate it on demand. even pushing routes using their communities to offload the circuit takes forever to propagate even on their own looking-glasses.
thx
ryan
as stated, yep. i was on the phone with them for over three hours yesterday. On Thu, Oct 2, 2014 at 9:08 AM, Paul S. <contact@winterei.se> wrote:
First time I'm seeing it, and I've been a Cogent client for quite a while.
Have you tried getting in touch with their NOC yet? They're one of the most responsive in the industry.
On 10/3/2014 午前 01:03, ryanL wrote:
hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them?
i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent.
example: http://ryry.foursquare.com/image/0e0K1K0t0W2M
it's been a bit of a frustrating experience talking to their noc to demonstrate it, but i'm able to duplicate it on demand. even pushing routes using their communities to offload the circuit takes forever to propagate even on their own looking-glasses.
thx
ryan
On Thu, 2 Oct 2014, ryanL wrote:
hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them?
i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent.
Perhaps related, I made an as-path prepend change on a route advertised to Cogent yesterday and noticed it took about 10 minutes for that change to propagate throughout Cogent's network and be visible on route-views. A moment later, I did the same thing with another carrier, and got the expected nearly instant gratification. Perhaps they've got a bunch of routers configured with minimum-advertisement-interval to batch the BGP updates and if you're unlucky with the timing, it can take a while for routes/changes to percolate through their network? Imagine driving down a long road, where every traffic light turns red just before you get to the intersection. Wait a minute here, wait a minute there... ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
I've had the exact same thing happen. Recently, I removed a bunch of /24's that were member of a larger /20. We use to advertise the /24's to certain carriers for traffic engineering. I saw the exact same issue you're describing. I chocked it up to Slow BGP in their core. For instance, I removed the route from my session with them in Orlando. It would get removed from Orlando, But still exist in Jacksonville. And basically route loop coming in from the north. It'd land in JAX, And get forwarded to MCO. Which would then forward it back to JAX. 20-30 seconds later, it would get removed from JAX, and I'd see the same behavior between JAX and ATL. It took a 4-5 minutes in my experience for the route to finally leave the Cogent network. I lessened the blow by making Cogent the first peer I'd remove the /24 from. Leaving the other /24's out there via the other carriers. So only those that chose cogent as the most direct route felt the unintentional blackhole. Other carriers removed almost instantly as expected. Nick Olsen Network Operations (855) FLSPEED x106 ---------------------------------------- From: "ryanL" <ryan.landry@gmail.com> Sent: Thursday, October 02, 2014 12:07 PM To: "North American Network Operators Group" <nanog@nanog.org> Subject: cogent update suppression, and routing loops hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them? i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent. example: http://ryry.foursquare.com/image/0e0K1K0t0W2M it's been a bit of a frustrating experience talking to their noc to demonstrate it, but i'm able to duplicate it on demand. even pushing routes using their communities to offload the circuit takes forever to propagate even on their own looking-glasses. thx ryan
On Thu, Oct 02, 2014 at 09:03:15AM -0700, ryanL wrote:
(who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent.
Maybe running into some non-standard BGP Advertisement intervals inside Cogent's network? Might be running a lot of sub-ASes inside the confederation, and having to wait multiple advertisement intervals for full propagation. https://routingfreak.wordpress.com/tag/minimum-route-advertisement-interval/ -- Brandon Ewing (nicotine@warningg.com)
circling back on this, i guess my case with cogent has been escalated to vp engineering, and i've had a few people reply on and off list citing the same problems. i encourage you to open up cases to help demonstrate further examples (ie: it's not just me!) thx everyone. ryan On Thu, Oct 2, 2014 at 9:03 AM, ryanL <ryan.landry@gmail.com> wrote:
hi. relatively new cogent customer. is what i've stated in my subject line kinda standard fare with them?
i've discovered that when i advertise a /24 from inside a larger /22 to XO, (who peers with cogent), and then pull the /24 some time later, that cogent holds onto the /24 and then bounces packets around in their network a bunch of times for upwards of 8-10 minutes until they finally yank it. this effectively blackholes traffic to my /24 for anyone that is using a path thru cogent.
example: http://ryry.foursquare.com/image/0e0K1K0t0W2M
it's been a bit of a frustrating experience talking to their noc to demonstrate it, but i'm able to duplicate it on demand. even pushing routes using their communities to offload the circuit takes forever to propagate even on their own looking-glasses.
thx
ryan
participants (5)
-
Brandon Ewing -
Jon Lewis -
Nick Olsen -
Paul S. -
ryanL