Throw your coffee at them! Just my two pence ;) ...James -----BEGIN GEEK CODE BLOCK----- Version: 3.1 GIT/MU/U dpu s: a--> C++>$ U+> L++> B-> P+> E?> W+++>$ N K W++ O M++>$ V- PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+> DI D+++ G+ e(+++++) h--(++) r++ z++ ------END GEEK CODE BLOCK------

On Mon, Aug 17, 2009 at 03:37:07PM -0600, randal k wrote:

Yep, we started seeing this right around 12:20pm MST. We saw it from a customer's rapidly-flapping BGP peer. We told them to configure bgp maxas-limit, but apparently CRS1s don't have that command.

Anybody have a handy route-map that will deny anything with a as-path longer than say 15-20? ;-)

Been a while since I had to throw this on cisco, but I since it lacks sane repeat constraint, you have to either choose to iterate over your acceptable space or deny on the longer-than-acceptable. For the latter, ^[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_[0-9]+_([0-9]+_)+ clobbers 15 ASNs and longer. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Dear Colleagues, Sorry for the late response. The problem was due to faulty firmware on one of our Alaxala routers. We resolved the problem the same day (Aug. 18) by downgrading firmware. For more details, please see Alaxala page here (English): http://www.alaxala.com/en/information/20090827.html If you have filters etc. blocking AS9354, please remove them. Thank you, Nick Boyadjiev AS9354 Nagoya JAPAN

On Mon, 17 Aug 2009 18:40:39 -0400, Jared Mauch <jared@puck.nether.net> wrote:

Is there some significant barrier to people getting recent code on the devices that is not impacted by this and the other fun bgp 'attacks' that can happen?

In a word: YES. Any respectable ISP will not load code that has not been extensively tested. Failure to do so can, and WILL, lead to even greater impact outages. (we've all made that mistake. Once.) Unless you do millions with Cisco and can therefore get custom IOS builds, you won't get a newer version with *just* the intended bug fixed. Their maint "rebuilds" end up with multiple "fixes" and all too often, previous fixes reverted. (I stopped counting the number of times I had to bitch at them to refix the SNMP DLCI interface counters on the 7401... "we don't test frame relay on the 7401" -- sure, that's eons ago, but nothing has changed over there.) And then there's the question of support... again, any respectable ISP maintains maint contracts with their vendors. But, things tend to fall through the cracks... contracts expire, people forget to list all the equipment, vendors drop support for various hardware and software, etc. You've obviously not gone to Cisco for any "non-contract" software updates. It's faster to bribe someone with an active service contract or use google. Also... Never underestimate the power of Lazy! --Ricky

Confidence level: Boots in lab."

One could argue that certain things haven't actually changed that much ;-). Marko. -- Marko CCIE #18427 (SP) My network blog: http://cisco.markom.info/

From: Brett Watson <brett@the-watsons.org> Date: Mon, 17 Aug 2009 19:11:06 -0700

On Aug 17, 2009, at 5:17 PM, Paul Ferguson wrote:

...

I recall Cisco code bugs that were fixed in semi- real-time, and quotes from tli: "Code still warm from compiler. Confidence level: Boots in lab."

IETF Dallas, 1995 I think. MCI Reston engg and Cisco (Ravi and others) in the terminal room cutting a new image of IS-IS for us because we'd tripped on the 24-day timer bug wherein all adjacencies would drop. Loaded on production routers that evening to fix the problem... good times :)

I remember back in 1993 or 1994 getting a note from Tony Li chastising us for running BGP4 code that was over a WEEK old. How could we possibly expect things to work with code that far out of date? The amazing part is that BGP4 not only worked, but was very stable less then a month later when NSFnet shut down an se had to go to BGP4 to peer with everyone (like MCI, Sprint, NASA, and UUnet). Yep, it booted in the lab, all right. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751

Ivan- Thanks for posting this how-to on excessive as prepends. I have a couple of questions that some of the less BGP savvy out their may find helpfull 1. In my enviornment, we are not doing full routes. We have partial routes from AS209 and then fail to AS7263. Is their any advantage for someone like me to do this, as we are not providing any IP transit so we are not passing the route table to anyone else? 2. When I run the "sh ip bgp quote-regexp "_([0-9]+)_\1_\1_\1_\1_ \1_" | begin Network" I am seeing many many ASes that would be filtered by this access-list. What happens to those networks, are they unreachable from my AS, or do I just route those networks to my upstream provider and let them deal with it? 3. This last question is a little OT, but relates to your access list In the event of some kind if DOS attack coming from one of a few AS numbers (in this case we will use 14793), what is the feesability of using ip as-path access-list 100 deny _([0-9]+)_\1_\1_\1_\1_ ip as-path access-list 100 deny 14793 ip as-path access-list 100 permit .* Would this have any affect at all, or would my pipe from my upstream still be congested with garbage traffic ? Thanks Dylan Ebner -----Original Message----- From: Ivan Pepelnjak [mailto:ip@ioshints.info] Sent: Tuesday, August 18, 2009 2:37 AM To: 'randal k'; 'Adam Hebert' Cc: nanog@nanog.org Subject: RE: Anyone else seeing "(invalid or corrupt AS path) 3 bytes E01100" ?

Anybody have a handy route-map that will deny anything with a as-path longer than say 15-20? ;-)

http://wiki.nil.com/Filter_excessively_prepended_BGP_paths Ivan http://www.ioshints.info/about http://blog.ioshints.info/

Ivan- This helps vey much. Thanks Dylan Ebner -----Original Message----- From: Ivan Pepelnjak [mailto:ip@ioshints.info] Sent: Tuesday, August 18, 2009 1:58 PM To: Dylan Ebner; 'randal k'; 'Adam Hebert' Cc: nanog@nanog.org Subject: RE: Anyone else seeing "(invalid or corrupt AS path) 3 bytes E01100" ?

Ivan- Thanks for posting this how-to on excessive as prepends. I have a couple of questions that some of the less BGP savvy out their may find helpfull

1. In my enviornment, we are not doing full routes. We have partial routes from AS209 and then fail to AS7263. Is their any advantage for someone like me to do this, as we are not providing any IP transit so we are not passing the route table to anyone else?

You could do it to protect your BGP table, but as you're not a transit AS, it does not make much sense.

2. When I run the "sh ip bgp quote-regexp "_([0-9]+)_\1_\1_\1_\1_ \1_" | begin Network" I am seeing many many ASes that would be filtered by this access-list.

Obviously a lot of people are prepend-happy.

What happens to those networks, are they unreachable from my AS, or do I just route those networks to my upstream provider and let them deal with it?

If I understood correctly, you're using a default route toward AS7263, which means that anything that is not in your BGP table (and consequently your IP routing table) will be sent out of your AS via the default route. If you're getting the paths you're filtering from AS209 that means that more traffic will go to AS7263.

3. This last question is a little OT, but relates to your access list In the event of some kind if DOS attack coming from one of a few AS numbers (in this case we will use 14793), what is the feesability of using ip as-path access-list 100 deny _([0-9]+)_\1_\1_\1_\1_ ip as-path access-list 100 deny 14793 ip as-path access-list 100 permit .*

Would this have any affect at all, or would my pipe from my upstream still be congested with garbage traffic ?

No. You cannot influence the inbound traffic apart from not advertising some of your prefixes to some of your neighbors or giving them hints with BGP communities or AS-path prepending. Whatever you do with BGP on your routers influences only the paths the outbound traffic is taking. What you'd actually need is remote-triggered black hole. Search the Nanog archives for RTBH, you'll find a number of links in a message from Frank Bulk sent a few days ago. Hope this helps Ivan http://www.ioshints.info/about http://blog.ioshints.info/

On Tue, Aug 18, 2009 at 09:37:22AM +0200, Ivan Pepelnjak wrote:

...

Anybody have a handy route-map that will deny anything with a as-path longer than say 15-20? ;-)

http://wiki.nil.com/Filter_excessively_prepended_BGP_paths

It will still be a while before we see unbroken 4byte AS behavior (that whole 'fix the teardown on a anyone sneezing' problem). But like with stale bogon filters, I expect folks inclined to use this to drop it in and forget about it. So it would be wise to adjust the recommended filter to anticipate a 2byteAS view allowing multiple instances of AS-TRANS; there's likely a more elegant approach, but the quick step of explicitly allowing _(23465_)+ before you deny _([0-9]+)_\1_\1_\1_\1_ Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE