Yep, soBGP or S-BGP could have prevented this. But that seems to be a bridge too far right now. This seems to have been a route leak from AS9318, not a false origin announcement, so not sure if soBGP or s-BGP would help here.

--Ricardo

On Sun, 8 Jul 2007, Marcus H. Sachs wrote:

If we had routing registries that were accurate and authoritative, then soBGP/S-BGP would have something to verify a route change against. It should not matter if last Friday's event was a leak or a false announcement - with some sort of verification system we could mitigate errors, intentional or accidental.

either way, in this case (and a number of other public incidents/outages in the last 3 years) simple prefix-list application would have resolved the issue. Cogent leak via (i think) turk-telecom NY-Edison leak 9918 leak this-leak all would have been prevented with the most simple of steps: "prefix filter your customers". While S*BGP seem like they may offer additional protections and additional knobs to be used for protecting 'us' from 'them', the very basics are obviously not being done so added complexity is not going to really help :( Or, perhaps its not that its not going to help its just not going to get done because even prefix-lists are 'too hard', apparently. -Chris

On Mon, 09 Jul 2007 02:18:25 -0000, "Chris L. Morrow" said:

While S*BGP seem like they may offer additional protections and additional knobs to be used for protecting 'us' from 'them', the very basics are obviously not being done so added complexity is not going to really help :( Or, perhaps its not that its not going to help its just not going to get done because even prefix-lists are 'too hard', apparently.

"Wow, prefix-lists are *hard*" -- BGP Barbie.. You'd think that by now, we as an industry could do better than that. (Yes, I know the jury is still out on what really happened at L3-Hanaro. Doesn't change the fact that we collectively shoot ourselves in the foot because providers will believe the most implausible things from their neighbors, like announcements for 128/1 ;)

On Jul 9, 2007, at 10:47 AM, Valdis.Kletnieks@vt.edu wrote:

On Mon, 09 Jul 2007 02:18:25 -0000, "Chris L. Morrow" said:

...

While S*BGP seem like they may offer additional protections and additional knobs to be used for protecting 'us' from 'them', the very basics are obviously not being done so added complexity is not going to really help :( Or, perhaps its not that its not going to help its just not going to get done because even prefix-lists are 'too hard', apparently.

"Wow, prefix-lists are *hard*" -- BGP Barbie..

You'd think that by now, we as an industry could do better than that.

I agree that we need something better but nobody has shown me a better system than prefix lists and irr that actually *works*. The simple truth is that prefix lists ARE hard to manage. There are a lot of folks that have complex relationships or don't see why they should register their routes. Some people lack tools and automation to make it work or to manage their networks. It would be nice to see everyone filter routes, including those from even transit and large peers. I don't think we will be able to ignore this forever. I also do not see the status quo changing soon either.

* Valdis Kletnieks:

(Yes, I know the jury is still out on what really happened at L3-Hanaro. Doesn't change the fact that we collectively shoot ourselves in the foot because providers will believe the most implausible things from their neighbors, like announcements for 128/1 ;)

Well, if L3 creates its filters based on RADB entries (which is still considered a RR, isn't it?), they will accept a 213/8 announcement. 8-( 128/1 isn't too far away, I fear. -- Florian Weimer <fweimer@bfk.de> BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99

On Sun, 8 Jul 2007, Lixia Zhang wrote:

I agree here. So the mostly needed thing is some verification system.

I keep on coming back to "any proposal that requires universal adoption and deployment is fatally flawed". As far as "needing a verification system", is there something deeply problematic about filtering your customers? It's a fine example of thinking globally and acting locally.

Wonder what people think about this: http://www.nanog.org/mtg-0706/osterweil.html something one can start using *now*. There was virtually no comment after the presentation at last NANOG.

Perhaps that's because there wasn't much to say about it? I think DNS is already badly overloaded, and adding still more random crap into it just isn't on my list. Beyond that, the phrase "leverages concepts from PGP's Web of Trust" makes my hair stand on end. Perhaps the author(s) are comfortable with using a pile of disassociated islands 'with low reliability for confirming prefix ownership -- I'd rather apply my efforts to something that doesn't boil down to the equivalent of an educated judgement call. cheers! ========================================================================== "A cat spends her life conflicted between a deep, passionate and profound desire for fish and an equally deep, passionate and profound desire to avoid getting wet. This is the defining metaphor of my life right now."

Andy Dills wrote:

On Mon, 9 Jul 2007, Cat Okita wrote:

...

As far as "needing a verification system", is there something deeply problematic about filtering your customers? It's a fine example of thinking globally and acting locally.

That's what I'm curious about...this boils down to L3 not properly filtering Hanaro.

Having recently turned up some L3 connectivity, I was happy to discover that they can use any of the routing registries as the source of their prefix filters. They told me the prefix filters are automatically constructed based on the RR of your choice...update the RR, and their filters will update that night, no need to bug them. Yay for that, wish everybody worked that way.

But...why wasn't Hanaro being filtered? If the filters are being automatically generated, I would think they would just filter all of their peers regardless of number of prefixes etc.

Handily it's possible to query Level3's filters through a whois interface, for example: whois -h filtergen.level3.net RIPE::AS25577 For some reason this doesn't work: whois -h filtergen.level3.net APNIC::AS9318 But this does, presumably this one is reading of an internal Level3 db: whois -h filtergen.level3.net AS9318 List of prefixes returned is: Prefix list for policy AS9318 = LEVEL3::AS9318 61.98.32.0/19 61.98.64.0/20 61.98.96.0/20 124.111.0.0/16 210.93.131.0/24 210.93.132.0/22 211.33.96.0/20 211.49.96.0/20 211.49.144.0/20 211.49.192.0/19 211.49.240.0/20 211.59.160.0/19 211.59.208.0/20 211.59.224.0/20 211.110.160.0/19 211.110.224.0/20 211.186.0.0/19 211.186.32.0/20 211.186.64.0/19 211.186.96.0/20 211.186.144.0/20 211.186.160.0/19 211.243.144.0/20 211.243.224.0/19 211.244.0.0/20 211.244.32.0/19 211.244.64.0/20 211.244.176.0/20 211.245.48.0/20 218.49.114.0/24 218.234.87.0/24 Sam

following existing BCPs with currently-deployed techniques/functionality/features would have prevented the issue described in the post.

knowing that level(3) is one of the most serious deployments of irr-based route filters and other prudent practices, perhaps we should wait for a post mortem from level(3) before jumping to conclusions? randy

On Mon, Jul 09, 2007 at 02:31:10PM +0800, Randy Bush wrote:

...

following existing BCPs with currently-deployed techniques/functionality/features would have prevented the issue described in the post.

knowing that level(3) is one of the most serious deployments of irr-based route filters and other prudent practices, perhaps we should wait for a post mortem from level(3) before jumping to conclusions?

randy

Level3's filter implmentation is indeed well-done, however, the fact remains that the IRR (which I use and endorse) has no linkage to any other source of information for purposes of validation. It's fundamentally garbage in, garbage out. Say some ISP has a provisioning tool which updates their router configs and the IRR in one fell swoop. If the provisioner makes a typo the IRR will gladly accept the entry for, say, 12/8, and the upstream will rebuild their filters with that entry automatically and you get the same result. There's no magic bullet in updating BGP if a fundamental, verifiable data model is not accepted and agreed upon. Tony

On Tue, 10 Jul 2007, Randy Bush wrote:

the space of routing data validation is large, we can explore it at our leisure, and we have been for some years. but my point was that it is silly to indulge in conjecturbation on the cause of the recent event and excoriate l(3), hanaro, or john curran's grandmother until we have heard from the folk who have actual data.

If companies thought it was in their self-interest, they might actually share that actual data. However history has shown over and over again that companies generally avoid any public discussion about their problems until they are overwhelmed. http://tech.monstersandcritics.com/news/article_1327791.php/Yahoo_outage_cau... If you wait for the companies to reveal the data, you will probably have a long wait. WorldCom still hasn't released its official investigative report into why its national frame networks failed for nearly a week in 1999.

On Sun, 8 Jul 2007, Steven M. Bellovin wrote:

...

I put up a diary at the Storm Center (http://isc.sans.org/diary.html?storyid=3112) that summarizes what we know about the Yahoo outage on Friday. If anybody has any additional info they want to share or comments about the write-up please let me know.

In other words, it was yet another BGP screw-up that secured routing could have prevented.

Or using route registeries and filters, or any of the other dozen ideas suggested over the last decade.

Any clue about the root cause, i.e., malice or accident?

Does it matter? You are screwed either way.

On Sun, 8 Jul 2007, Steven M. Bellovin wrote:

...

...

Any clue about the root cause, i.e., malice or accident?

Does it matter? You are screwed either way.

It tells us what we need to do to prevent such things from happening in the future. For example, most misconfigurations could be blocked if all routers matched prefixes against originating ASNs, and it doesn't matter much if the assertion is digitally signed or not -- all that matters is that the check is done against some authoritative database run, say, by the RIRs. (No, that's not quite the right solution, but it serves to illustrate my point.) That's completely inadequate against an attacker.

The bad guys will (almost) always say Oops, it was an accident while being very clever at deliberatly bypassing every safety feature you can design into a system. The foolish guys will (almost) always say Oops, I didn't know while also being very clever at accidently bypassing every safety feature you can design into a system. Unfortunately engineering can't rely on human intentions. Both the evil and the foolish have the same result. The hope is the foolish will give up before bypassing the last step, so you keep adding more steps to stop the fool. The hope is the evilish will go after something easier before bypassing the last step, so you keep adding more steps to stop the evil (sic). As always evil or foolish gals do the same thing as evil or foolish guys. Its not just IP addresses that exhibit misrouting. But it only occassionaly effects the important or famous enough to attract attention. http://blog.oregonlive.com/siliconforest/2007/06/rivalry_between_qwest_comca...