On 5/9/2010 23:17, Scott Howard wrote:

Made it to Slashdot too - http://tech.slashdot.org/story/10/05/10/0056228/The-Status-of-Routing-Reform...

As usual I wouldn't recommend reading the comments unless you want your eyes to bleed...

I go back a step or two--I try very hard not to read AP. Which pretty much takes me out of reading anything but the comics in the "news"papers. And a lot of the comics have become so politicized that I don't read many of them any more. -- Somebody should have said: A democracy is two wolves and a lamb voting on what to have for dinner. Freedom under a constitutional republic is a well armed lamb contesting the vote. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml

On Mon, May 10, 2010 at 2:40 AM, Randy Bush <randy@psg.com> wrote:

...

http://skunkpost.com/news.sp?newsId=2327

same article as the one bellovin linked yesterday.  but this version has author attribution.  thanks.

a couple choice quotes: "Spokesmen at AT&T Inc. and Verizon Communications Inc., two of the largest, world-spanning carriers of Internet traffic, said they were unable to find anyone at their companies who could discuss the issue of routing reform." guess they didn't find rexford's successor @7018 or schiller @701? oh, well actually these folks don't do telephone networking so I guess it makes perfect sense :( "Pieter Poll, the chief technology officer at Qwest Communications International Inc., says that he would support some simple mechanisms to validate data routes, but he argues that fundamental reform isn't necessary. Hijackings are typically corrected quickly enough that they don't pose a major threat, he argues." qwest customers may want to take note here..."quickly enough" is how much of your business lost exactly? -chris :(

On Mon, May 10, 2010 at 11:30 AM, Jack Bates <jbates@brightok.net> wrote:

They filter my routes. Not sure what he's talking about.

just guessing but they probably don't filter 'peer' (SFP) routes, so if say 2914 accepted a route from their customer ConEdison that included a subnet of yours... boom goes your reachability :(

All of the major providers I have worked with have required proof of 'ownership' of address space or an LoA from the registered holder of that space before they would allow advertisements from me, which are then filtered. Is this not the norm? I can understand if they are talking about an operator making a mistake, but the article seems to imply that anyone running BGP can bring down the Internet... I think any competent provider can easily eliminate this threat from customers. Are there any types of penalties if an ISP is found to not be taking adequate precautions, other than the possible threat of losing business? -----Original Message----- From: Franck Martin [mailto:franck@genius.com] Sent: Monday, May 10, 2010 9:48 AM To: nanog@nanog.org Subject: Re: Securing the BGP or controlling it? APNIC allows you to put your BGP data in the whois, so like this you have a third party verification tool on who is peering with who.

Ziad, I agree, its unfortunate that so many people no longer require route registration. Not that it would solve all the issues. Tom School, Todd Underwood and I present some work we did looking @ this @ nanog in LA a while back. Unfortunately we could never find time to take it to the next steps. -jim Sent from my BlackBerry device on the Rogers Wireless Network -----Original Message----- From: Zaid Ali <zaid@zaidali.com> Date: Mon, 10 May 2010 10:32:47 To: Thomas Magill<tmagill@providecommerce.com>; Franck Martin<franck@genius.com>; <nanog@nanog.org> Subject: Re: Securing the BGP or controlling it? What we need (as operators) is to get better at ensuring that advertisements are coming from the valid owner of said address space. What we don't need is a separate governance model which I worry this article is trying to imply. I still use RADB but I hear not every peer/provider checks there anymore? This is hearsay so interested in other opinions. As far as the mistakes pointed out in this article one can be assured that these things are bound to happen. The youtube situation could have been prevented if the peer opening a filter (and responsible for announcing out) had reach to a system where the other peer's advertisement can be verified. I don't think leaning on competency is a good way to go about solving this problem, we need a system or model in place to ensure we have a trust and verification system. Zaid On 5/10/10 9:54 AM, "Thomas Magill" <tmagill@providecommerce.com> wrote:

All of the major providers I have worked with have required proof of 'ownership' of address space or an LoA from the registered holder of that space before they would allow advertisements from me, which are then filtered. Is this not the norm? I can understand if they are talking about an operator making a mistake, but the article seems to imply that anyone running BGP can bring down the Internet... I think any competent provider can easily eliminate this threat from customers. Are there any types of penalties if an ISP is found to not be taking adequate precautions, other than the possible threat of losing business?

-----Original Message----- From: Franck Martin [mailto:franck@genius.com] Sent: Monday, May 10, 2010 9:48 AM To: nanog@nanog.org Subject: Re: Securing the BGP or controlling it?

APNIC allows you to put your BGP data in the whois, so like this you have a third party verification tool on who is peering with who.

On 10/05/2010 16:29, Christopher Morrow wrote:

qwest customers may want to take note here..."quickly enough" is how much of your business lost exactly?

this is a matter of risk analysis. No secure routing means we'll continue to see the occasional high profile outage which is dealt with very quickly. Secure routing is going to introduce significant complexities into the inter-domain routing system. Complexities lead to greater pilot error, and pilot error leads to outages. So while we may have fixed the 2 hour youtube externally derived problem which we get once every couple of years, it's probably going to come at the cost of having N hours worth of outages per year per ASN, because someone's mucked up their configuration, or has let their cert expire, or whatever. My gut instinct tells me that secure routing and the rpki venture well into the realm of negative returns. But I would be really interested to see some proper risk analysis in this area done by someone with clue. Nick

On 10/05/2010 17:00, Aaron Glenn wrote:

my gut says things would do well to begin with simply making an effort at maintaining usable irr data and automagically generating sane filters. why don't people do that again? I hope I'm not naively misunderstanding a primary use of irr data in front of 10,000 of my closest friends...

There are a lot of problems associated with using IRRDB filters for inbound prefix filtering. - some clients announce lots of prefixes. This can make inbound prefix filtering difficult in some situations. pixiedust:/home/nick> grep '>' pakistani-telecom.bgpdump.txt | wc -l 967 - there are some endemic data reliability problems with the IRRDBs, exacerbated by the fact that on most of the widely-used IRRDBs, there is no link between the RIR and the IRRDB, which means that anyone can register any address space. whois.ripe.net doesn't allow this, but lots of other IRRDBs do. - the ripe whois server software does not support server-side as-set expansion. This is a really serious problem if you're expanding large ASNs. - there is very little client software. At least irrtoolset compiles these days, but its front-end is very primitive. rpsltool provides some really nice templating functionality, but doesn't implement large sections of the rpsl standards. Nick

On 10/05/2010 17:58, Jared Mauch wrote:

On May 10, 2010, at 12:48 PM, Nick Hilliard wrote:

...

- there are some endemic data reliability problems with the IRRDBs, exacerbated by the fact that on most of the widely-used IRRDBs, there is no link between the RIR and the IRRDB, which means that anyone can register any address space. whois.ripe.net doesn't allow this, but lots of other IRRDBs do.

Certainly this is a function that you can petition your local RIR to do, have you made a proposal to them?

RIPE does this automatically. But I have no idea how this sort of thing would be implemented between an RIR like ARIN and an IRRDB like whois.radb.net.

...

- the ripe whois server software does not support server-side as-set expansion. This is a really serious problem if you're expanding large ASNs.

Have you asked them to include this?

I've enquired informally and was left with the impression that it would be difficult; the RIPE DB code is troublesome, and there are line protocol differences between the ripe server and the merit server which would make parsing an interesting proposition.

I certainly agree the tools here are suboptimal, but is that the the reason to throw the baby out with the bathwater?

Not at all - I use prefix filtering in anger, and it works very well in its place.

Who is going to be the provider that turns away business because their customer is unwilling to register their routes in a klunky-toolset?

Lots. They'll certainly take on the business, but I know of several well-known names who provide service in Dublin and who won't accept your prefixes unless they are registered in an IRRDB.

What improvements to the toolset should go back to the community to improve filtering?

If you're offering to hack code, great - email me offline :-) Nick

On May 10, 2010, at 10:48 AM, Nick Hilliard wrote:

There are a lot of problems associated with using IRRDB filters for inbound prefix filtering.

We used them over 15 years ago near ubiquitously and stopped mostly because: 1) there was nothing akin to route refresh so you had to bounce best routes or reset sessions to trigger readvertisement after policy updates. This made unscheduled a pain and required some turning of the steam valves. 2) traditional ACLs were used for routing policy specification and weren't incrementally updatable, which was a huge pita. 3) IRRs were insecure to update, no one ever deletes anything from IRRs, and some folks even proxy register IRR objects based on BGP routing table entries. 4) customers complained they had to maintain them (ohh, wait, we told them if they wanted to be routed they had no choice) Regarding 1, we have route refresh and inherent soft-reconfig today. Regarding 2, pretty much all implementations support this today (although it will be a pita to maintain near exact prefix list and ACL entries for a customer down the road when used for both routing policies and ingress anti-spoofing). Regarding 3, RPKI should help here quite a bit, either used directly, or enabling IRR object population - the RIRs that run IRRs and other other folks are helping with secure update mechanisms as well. Regarding 4 - they didn't scream as loud about the policy as they did when things broke because of the absence of it, that I know firsthand.

- some clients announce lots of prefixes. This can make inbound prefix filtering difficult in some situations.

pixiedust:/home/nick> grep '>' pakistani-telecom.bgpdump.txt | wc -l 967

Yes, this needs to be automated, clearly.

- there are some endemic data reliability problems with the IRRDBs, exacerbated by the fact that on most of the widely-used IRRDBs, there is no link between the RIR and the IRRDB, which means that anyone can register any address space. whois.ripe.net doesn't allow this, but lots of other IRRDBs do.

See 3 above.

- the ripe whois server software does not support server-side as-set expansion. This is a really serious problem if you're expanding large ASNs.

Do it yourself for now and file a feature request..

- there is very little client software. At least irrtoolset compiles these days, but its front-end is very primitive. rpsltool provides some really nice templating functionality, but doesn't implement large sections of the rpsl standards.

Agreed, we need to do work here. -danny

On 10/05/2010 20:20, Randy Bush wrote:

if something like those happen again, we are gonna be spending a lot of time explaining our selves to people who wear funny clothes, and telling them why it is not going to happen again if they let us keep our jobs.

Yes, I have observed that people who wear funny clothes with blood constriction devices wrapped tightly around their necks seem to be concerned primarily with ass covering theatre. Risk analysis is ass covering without the theatre. You collect data, make a judgement based on that data, and if it turns out that the judgement says that signed bgp updates constitute more of a stability risk to network operations than the occasional shock problem, then you point these people with odd dress sense towards the conclusions of this risk analysis report, having made sure that the conclusions are printed in a 48pt font, with no more than 2 syllables per word, preferably with a filled circle preceding each sentence. It may well be that they will ignore the risk analysis and be more concerned with the theatre than with data; this happens all the time, an excellent example being airport security, where security theatre seems to be considered much more important than actual security. Or it could go the other way, where risk analysis dictates that sensible precautions be taken, but they are thrown out for other reasons. A good example here is road safety, where it would be sensible to speed limit all cars to 50km/h, and ban motorbikes and bull-bars; but instead we substantially choose to ignore the risk and accept an attrition rate of 80,000 people every year between Europe and the US. Nick

Dear Danny; On May 11, 2010, at 10:13 AM, Danny McPherson wrote:

On May 11, 2010, at 7:32 AM, Nick Hilliard wrote:

...

Risk analysis is ass covering without the theatre. You collect data, make a judgement based on that data, and if it turns out that the judgement says that signed bgp updates constitute more of a stability risk to network operations than the occasional shock problem

So apply the risk management analogy here. We all know that pretty much anyone can assert reachability for anyone else's address space inter-domain on the Internet, in particular the closer you get to 'the core' the easier this gets. We also know that route "leaks" commonly occur that result in outages and the potential for intercept or other nefarious activity. Additionally, we know that deaggregation, and similar events result in wide-scale systemic effects. We also know that topologically localized events occur that can impact our reachability, whether we're party to the actual fault or not. We have a slew of empirical data to support all of these things, some more high profile than others, with route leaks likely occurring at the highest frequency (every single day).

I would suspect that the probability of fire effecting your network availability is very low, as you can fail over to a new facility. OTOH, if you have a route hijack (intentional or not) failover to a new facility with that address space isn't going to help, and hijacks can be topologically localized - the same applies for DDoS. Yet I suspect your organization has invested reasonably in fire suppression systems, but the asset that matters most that enables the substrate of some applications and services that you care about - the availability of your address space within the global routing system, has no safeguards whatsoever, and can be impacted from anywhere in the world.

I'd also venture a guess that we've had more routing issues that have resulted in network downtime of critical sites than we have had fires (if someone disproves that _nice dinner on me!).

But there is also recovery time, which you don't mention in your bet. If the building I am sitting in right now were to burn down to the ground, the client I am at would be affected for months and months. Yes, they have backups, and redundancy, but this is their HQ. If they (say) fat finger their BGP, well, it would be bad, but if they fix it this afternoon, everything will go back to normal shortly thereafter. So, sure, network outages may be more frequent than catastrophic fires, but that doesn't mean that the aggregated duration of disruption from network outages is greater than the aggregated disruption duration from fires. Regards Marshall

We've got empirical data, we understand the vulnerability and the risk (probability of a threat being used). Put that in your risk management equation and consider what assets are most vulnerable to your organization - I'd venture it's something to do with network, and if routing ain't working, network ain't working...

-danny

On May 10, 2010, at 9:52 AM, Nick Hilliard wrote:

this is a matter of risk analysis. No secure routing means we'll continue to see the occasional high profile outage which is dealt with very quickly.

If 3 weeks (e.g., the recent 'i root w/China incident) is "very quickly" then we're operating on different timescales.

My gut instinct tells me that secure routing and the rpki venture well into the realm of negative returns.

I believe 'sucks less' falls into the realm of positive, so here we disagree. -danny

On 5/10/2010 15:31, Anton Kapela wrote:

On May 9, 2010, at 11:39 PM, Franck Martin wrote:

...

http://skunkpost.com/news.sp?newsId=2327

"Just how fragile is the internet?"

Rhetoric, much?

Interestingly, the article misses interception and other non-outage potentials due to (sub) prefix hijacking.

At the risk of seeming to be a conspiracy theorist, I am worried that with "Central Authority" we might not have "hijacking" but "rerouting for inspection and correction". -- Somebody should have said: A democracy is two wolves and a lamb voting on what to have for dinner. Freedom under a constitutional republic is a well armed lamb contesting the vote. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml

On May 10, 2010, at 2:52 PM, Larry Sheldon wrote:

At the risk of seeming to be a conspiracy theorist, I am worried that with "Central Authority" we might not have "hijacking" but "rerouting for inspection and correction".

Building a database (i.e,. RPKI) aligned with the Internet number resource allocation hierarchy attesting to who's authorized to originate what route announcements and telling you how to configure your routers are two fundamentally different things. If that database doesn't exist it's tough to discriminate between legitimate and malicious or erroneous announcements - irrespective of how you discriminate. If it does exist, and you use it, anyone that can rub two packets together is surely going to employ preferences that first consider organizational and local objectives, then potentially national, and then some global inputs. This basically helps people to make more informed decisions, methinks. -danny

On 5/10/2010 17:04, Randy Bush wrote:

...

Interestingly, the article misses interception and other non-outage potentials due to (sub) prefix hijacking.

you seem to be entering the world of attacks. the AP article's point was fat fingers.

Interesting. I took it as a set up of why we NEED a Central Authority. *shrug* -- We'll see I guess, but probably not in time. -- Somebody should have said: A democracy is two wolves and a lamb voting on what to have for dinner. Freedom under a constitutional republic is a well armed lamb contesting the vote. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml