Randy Epstein wrote:

My point was that even with a license, accidents still occur.

My point is that without a license more accidents will occur.

Vendors currently do train their customers and certify them.

A lot of companies dont send their personel to training lessons because of the costs. The vendor primarily trains how to _implement_ a BGP policy on their equipment and not neccessarily how to develop a good peering and filter policy. The "youtube ip hijacking" case _may_ be a result of route redistribution from an internal routing protocol to BGP without any route filters applied. Every decent BGP engineer knows that this is a very bad idea.

LIRs don't and cannot know all the gear out there and configurations from network to network vary.

They dont need to. They could/should ensure that people running ASNs have a good knowledge about how BGP works. Not how to _implement_ a BGP policy on a vendor device. This truly is up to the vendors and ISPs.

This doesn't stop route leaks, nor would this protect us from intentional mischief.

True, but it will help reducing incidents which will have a huge impact on the live and economy of a lot of people. The "youtube IP hijacking" was only a minor nuisance in relation to what can happen if other prefixes are "hijacked" or just leak due to clueless personal. -- Arnd

...

You _need_ a license to drive a car, fly a plane etc. but until now you dont need to show that youre skilled enough to run a border router. Good idea? I dont think so.

My point was that even with a license, accidents still occur.

Even with a licence and testing, airline crashes still occur, commercial airline pilots still arrive at work drunk or die of heart-attacks behind the wheel of the airplane. But, due to a lot of effort in making better educational material available for pilots, including better flight simulators and better simulator scenarios, flying is a lot safer than it was in 1958. Not to mention the great effort that is put into post-mortem studies of airline crashes, the open sharing of information around the world, and the steady incremental improvement of best-practices and educational materials. The Internet operations "profession" could do all of that without any need for laws, licenses, inspections, or whatever. In fact, if you look around you, the Internet ops profession actually *DOES* do a lot of the same stuff that the airline industry does and things ARE getting better when you measure the impact per user or per connected device. The net is a lot bigger than it was 10 years ago, and far fewer incidents happen that have wide impact. In fact, it is not even clear that this YouTube incident counts as having wide impact. How many people were impacted by the YouTube outage compared to the Asian Tsunami/landslide of 2005? You don't need a Rogers Commission (Challenger disaster) or a 9-11 Commission set up by the President to solve these problems. For all the moaning and complaining that hit this list and the blogosphere, lots of people actually are studying the root cause of this disaster and taking action to mitigate such events in the future. It should be no surprise that the most important such mitigation events are not related to installing more BGP filters, but in making sure that outages/anomalies are promptly detected and promptly escalated to the RIGHT people in the operations team who can fix or mitigate them.

I am not against training personnel, but your solution doesn't resolve either of the above for the most part.

Training is a form of education, and education is a necessary prelude to action. You would be a fool to just accept someone's advice from this list and run out to implement it RIGHT NOW. Better to study it, try it in the lab. Figure out what it does, why it does it. Think about how to monitor it and manage it. Write up a business case to see if you really can justify this action to management. Then document it and do it when everybody understands the problem and the solution. Alex wrote such a brilliant message summarizing the discussion to date that I'm thinking we should reshape the mailing list committee into a kind NTSB http://www.ntsb.gov/ for the Internet that would solicit comments, compile incident reports and produce best practice documents. That kind of thing might be valuable enough that somebody would pay NANOG to do it. --Michael Dillon

Since the US has no jurisdiction over 17557, other than for the US govt. to force ISPs to refuse to accept any advertisements with 17557 or any other AS that didn't meet some regulatory requirements in the path, how would you propose that the regulatory environment you envision work? American Airlines isn't the right straw-man here, Pakistan International Airlines is. The only reason THEY meet anyone else's standards is that they wouldn't be allowed to use the airspace or land if they didn't.

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Dave Pooser Sent: Tuesday, February 26, 2008 10:15 AM To: nanog@merit.edu Subject: Re: YouTube IP Hijacking

...

But, due to a lot of effort in making better educational material available for pilots, including better flight simulators and better simulator scenarios, flying is a lot safer than it was in 1958.

At the risk of being a stereotypical American liberal, I'll point out two significant reasons flying is safer than it used to be in the US are Federal regulation and post-accident lawsuits. If there were an organization like the FAA that had the power to "ground" AS17557 until their network engineers completed a week's refresher course, there'd be significantly better change management techniques in play. If YouTube were currently suing Pakistani Telecom for eighty-seven gazillion dollars-- and were widely considered a lock to win their lawsuit-- suddenly a whole lot of other ISPs would magically find the training budget to make sure THEIR engineers didn't expose THEM to that sort of liability.

Pilots don't spend dozens of hours in simulators because it's fun, they do it to get/keep their license. American Airlines doesn't spend millions of dollars on pilot (and ground crew) education because they're run by philanthropists, they do it because screwups could cost them orders of magnitude more money. The Internet lacks any such enforcement mechanisms. How many people do you think have lost their jobs for this latest incident? What are the odds that the responsible party lost a penny in revenue or in fines?

When there is no financial or regulatory pressure to avoid screwups, avoiding screwups ceases to be a priority at Layer 8 or Layer 9. And then you have incidents like this, where the operational solutions are widely agreed upon and the political obstacles are widely agreed to be insurmountable. And we wait for the next incident. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com

On Tue, 26 Feb 2008, Tomas L. Byrnes wrote:

(first quoting Dave Pooser -- quote order changed by scg)

...

At the risk of being a stereotypical American liberal, I'll point out two significant reasons flying is safer than it used to be in the US are Federal regulation and post-accident lawsuits. If there were an organization like the FAA that had the power to "ground" AS17557 until their network engineers completed a week's refresher course, there'd be significantly better change management techniques in play. If YouTube were currently suing Pakistani Telecom for eighty-seven gazillion dollars-- and were widely considered a lock to win their lawsuit-- suddenly a whole lot of other ISPs would magically find the training budget to make sure THEIR engineers didn't expose THEM to that sort of liability.

Since the US has no jurisdiction over 17557, other than for the US govt. to force ISPs to refuse to accept any advertisements with 17557 or any other AS that didn't meet some regulatory requirements in the path, how would you propose that the regulatory environment you envision work?

American Airlines isn't the right straw-man here, Pakistan International Airlines is. The only reason THEY meet anyone else's standards is that they wouldn't be allowed to use the airspace or land if they didn't.

From a technical perspective, this is pretty cut and dried. Networks should be careful what they announce, but sometimes aren't. Upstream

I sent Tomas some private mail complaining about some of the things he was posting yesterday, but I think Dave's posting was spot on and Tomas's follow-up is adding an important point. As far as I can piece together from what's been reported and argued here, there were three responsible parties: The Pakistani Government who ordered YouTube blocked, Pakistan Telecom who implemented a lawful order but overshot their government's jurisdiction, and PCCW who accepted the announcements and passed them on to the world. providers should be careful what they accept, but sometimes aren't. Systems and policies to improve filtering sometimes cause more problems than they solve, especially when relying on a central source for authentication, and those costs are borne by the party trying to be responsible. Intentional leaks are harder to guard against than unintentional ones. Those hit hard by route leaks generally aren't the party responsible for the leak, so incentives to be careful are lacking. But this case also brings up a bunch of interesting policy and legal questions, which I'm less or not at all qualified to answer. This was a legally required routing announcement in Pakistan, and there was presumably a desire that other Pakistani ISPs be able to see the announcement. What if any responsibility do those following a lawful order have to keep the results of that order from being seen outside of their government's jurisdiction? What legal responsibility did PCCW have here, and in what countries? Given that they've got network infrastructure in the United States and around the world, they're presumably vulnerable to lawsuits in the US and elsewhere if Hong Kong law isn't sufficient. How will Google respond? Route leaks happen from time to time. Usually they're of relatively little consequence, and people clean them up and get back to work. I don't know how much revenue YouTube brings in over the course of a couple of hours, but it wouldn't surprise me if they could claim to have lost millions of dollars. PCCW has deep pockets, and Google has lots of lawyers. Will Google sue? If not, will it be because they think they don't have a case, because they value other relationships they have with PCCW, or because they're worried about establishing a precedent that would make them liable for their own engineers' errors? If Google did sue, would that lead to some BGP certification requirements for ISPs to get liability insurance? If such an insurance requirement didn't affect ISPs like Pakistan Telecom, would having it become a requirement for the international ISPs that tend to provide international transit be sufficient? (And then, of course, the really scary questions: What would such a certification process look like, and how many of us would be able to pass?) -Steve

On Feb 26, 2008, at 5:02 PM, Danny McPherson wrote:

The fact is that employment of explicit inter-domain prefix filtering seems to only be deteriorating from where it was 15 years ago is telling, and I think folks have become lazy and accepting, even as more and more critical infrastructure and services require an available and accurate routing system.

IMO, the onus is on the operators to step up...

Darn well said (the whole post). Some of us *used* the available infrastructure almost to a fault about 15+ years ago (the early days of Sprint, MCI, ANSNet, etc) but it has slowly gone down hill since. It takes work to keep it all up to date but it appears that it would save a lot of pain (and very long threads on nanog) if we did. I've never been able to figure out the incentive program to use it though. "Stopping the pain" doesn't seem to do it :) (I know, I know, it's all about the money). -b