Hi folks, I am trying to compile data on which providers are currently supporting BGP Flowspec at their edge, if there are any at all. The few providers I've reached out to have indicated they do not support this and have no intention of supporting this any time in the near future. I'm also curious why something so useful as to have the ability to advertise flow specification information in NLRI and distribute filtering information is taking so long to gain a foothold in the industry... Stefan Fouant: NeuStar, Inc. Principal Network Engineer 46000 Center Oak Plaza Sterling, VA 20166 [ T ] +1 571 434 5656 [ M ] +1 202 210 2075 [ E ] stefan.fouant@neustar.biz [ W ] www.neustar.biz

Fouant, Stefan wrote:

Hi folks,

I am trying to compile data on which providers are currently supporting BGP Flowspec at their edge, if there are any at all. The few providers I've reached out to have indicated they do not support this and have no intention of supporting this any time in the near future. I'm also curious why something so useful as to have the ability to advertise flow specification information in NLRI and distribute filtering information is taking so long to gain a foothold in the industry...

See ipv6 :)

On Apr 10, 2009, at 3:41 PM, Scott Doty wrote:

George William Herbert wrote:

...

Scott Doty wrote:

...

(Personally, I can think of a "MAE-Clueless" episode that was worse than this, but that was in the 90's...)

The gas main strike out front of the building in Santa Clara?

Or something else?

-george william herbert gherbert@retro.com

No, it was when an AS took their full bgp feed & fed it into their igp (which used RIP, iirc), which generated (de-aggregated) routes into /24's, which they then announced back into bgp...

That was Vinny Bono of FLIX, the Fat man Little man Internet eXchange, as7007. Happened in 1997, IIRC. He used a Bay Networks router to redistribute BGP on one card into RIPv1 on another card, stripping the CIDR notations off each prefix, making them classful, and stripping the AS Path. This means, for instance, 96.0.0.0 was a /8, not a /24. It also means He then re-redistributed RIP into BGP on a third card, which then originated each route from as7007. I have it on most excellent authority (the "Fat man" himself) that this was not possible on ciscos. Wonder if it is now ... ? Anyway, I did not know people were calling this the "MAE-Clueless" incident. I've always called it the "7007 incident". In fact, some people still have as7007 filtered.

iirc, part of the chaos than ensued was due to a router bug, so that the routes "stuck around" in global views, even after the AS killed their announcements, and even after physically disconnecting from their provider.

That was Sprint, as7007's transit provider. Sprint only did AS Path filtering, and as every single prefix was ^7007$, they all passed the filter. Vinny literally unplugged the router, no power, no fiber, no copper, but the prefixes were still bouncing around the 'Net for hours. Sprint kept the routes around for a long time as their routers would not honor withdrawals - or so the rumors said. The rumors also claimed the IOS version was named "$FOO-sean". Sean Doran was CTO of Sprint's Internet company at the time, and he supposedly specifically asked for the 'feature' of ignoring withdrawals to lower CPU on their AGS+s. I have absolutely no way of confirming this as I haven't spoken to Sean in years & years, and wouldn't even know where to find him any more. The most interesting rumor I heard is that Sprint had to shut down every single router simultaneously to clear the routes out of their network. Personally I think that's probably a bit exaggerated, but who knows?

We told our customers "the Internet is broken, please try again later"...which was acceptable back then. (But I doubt we would get away with just that nowadays... ;-) )

Really? That's what some broadband providers say nearly daily. -- TTFN, patrick

Jo¢ wrote:

I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers "can" be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno

The nice thing about the outdoors is how much of it there is.

Just my 2¢

An easy way to describe what your saying is "Security by obscurity is not security" On Apr 11, 2009, at 8:31 AM, Joe Greco wrote:

...

Jo¢ wrote:

...

I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers "can" be keyed. For those of you arguing that this is not enough, I would say at least it’s a start. Yes if enough time goes by anything can happen, but how can one argue an ATM machince that has (at times) thousands of dollars stands out 24/7 without more immediate wealth. Perhaps I am missing something here, do the Cops stake out those areas? dunno

The nice thing about the outdoors is how much of it there is.

Cute, but a lot of people seem to be wondering this, so a better answer is deserved.

The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious.

http://www.youtube.com/watch?v=4P8WM8ZZDHk

There are all sorts of strategies for attacking ATM's, and being susceptible to a sledgehammer, crowbar, or truck smashing into the unit shouldn't be hard to understand.

Most data centers have security that is designed to keep honest people out of places that they shouldn't be. Think that "security guard" at the front will stop someone from running off with something valuable? Maybe. Have you considered following the emergency fire exits instead? Running out the loading dock? Etc?

Physical security is extremely difficult, and defending against a determined, knowledgeable, and appropriately resourced attacker out to get *you* is a losing battle, every time.

Think about a door. You can close your bathroom door and set the privacy lock, but any adult with a solid shoulder can break that door, or with a pin (or flathead or whatever your particular knob uses) can stick it in and trigger the unlock. Your front door is more solid, but if it's wood, and not reinforced, I'll give my steel-toed boots better than even odds against it. What? You have a commercial hollow steel door? Ok, that beats all of that, let me go get my big crowbar, a little bending will let me win. Something more solid? Ram it with a truck. You got a freakin' bank vault door? Explosives, torches, etc. Fort Knox? Bring a large enough army, you'll still get in.

Notice a pattern? For any given level of protection, countermeasures are available. Your house is best "secured" by making changes that make it appear ordinary and non-attractive. That means that a burglar is going to look at your house, say "nah," and move on to your neighbor's house, where your neighbor left the garage open.

But if I were a burglar and I really wanted in your house? There's not that much you could really do to stop me. It's just a matter of how well prepared I am, how well I plan.

So. Now. Fiber.

Here's the thing, now. First off, there usually isn't a financial motivation to attack fiber optic infrastructure. ATM's get some protection because without locks, criminals would just open them and take the cash. Having locks doesn't stop that, it just makes it harder. However, the financial incentive for attacking a fiber line is low. Glass is cheap. We see attacks against copper because copper is valuable, and yet we cannot realistically guard the zillions of miles of copper that is all around.

Next. Repair crews need to be able to access the manholes. This is a multifaceted problem. First off, since there are so many manholes to protect, and there are so many crews who might potentially need to access them, you're probably stuck with a "standardized key" approach if you want to lock them. While this offers some protection against the average person gaining unauthorized access, it does nothing to prevent "inside job" attacks (and I'll note that this looks suspiciously like an "inside job" of some sort). Further, any locking mechanism can make it more difficult to gain access when you really need access; some manholes are not opened for years or even decades at a time. What happens when the locks are rusted shut? Is the mechanism weak enough that it can be forced open, or is it tolerable to have to wait extra hours while a crew finds a way to open it? Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete. Are you going to protect the concrete too? If not, what prevents me from simply breaking away the concrete around the manhole cover rim (admittedly a lot of work) and just discarding the whole thing?

Wait. I just want to *break* the cable? Screw all that. Get me a backhoe. I'll just eyeball the direction I think the cable's going, and start digging until I snag something.

Start to see the problems?

I'm not saying that security is a bad thing, just a tricky thing.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e- mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Joe Greco wrote:

My point was more the inverse, which is that a determined, equipped, and knowledgeable attacker is a very difficult thing to defend against.

"The Untold Story of the World's Biggest Diamond Heist" published recently in Wired was a good read on that subject: http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Which brings me to a new point: if we accept that "security by obscurity is not security," then, what (practical thing) IS security?

Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of "security by obscurity" so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either. I think the problem is that the notion of "security by obscurity isn't security" was originally meant to convey to software vendors "don't rely on closed source to hide your bugs" and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have. Mike

One thing that is missing here is before we can define "security" we need to define the "threat" and the "obstruction" the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, "If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them." So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the "Threat" to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, "where do we go from here?" It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com -----Original Message----- From: Joe Greco [mailto:jgreco@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area

Joe Greco wrote:

...

My point was more the inverse, which is that a determined, equipped,

...

and knowledgeable attacker is a very difficult thing to defend against.

"The Untold Story of the World's Biggest Diamond Heist" published recently in Wired was a good read on that subject:

http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

...

Which brings me to a new point: if we accept that "security by obscurity is not security," then, what (practical thing) IS security?

Obscurity as a principle works just fine provided the given token is obscure enough.

Of course, but I said "if we accept that". It was a challenge for the previous poster. ;-)

Ideally there are layers of "security by obscurity" so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either.

I think the problem is that the notion of "security by obscurity isn't

security" was originally meant to convey to software vendors "don't rely on closed source to hide your bugs" and has since been mistakenly

applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have.

That's really it, and bringing us back to the fiber discussion, we are forced, generally, to rely on obscurity. In general, talk to a hundred people on the street, few of them are likely to be able to tell you how fiber gets from one city to another, or that a single fiber may be carrying immense amounts of traffic. Most people expect that it just all works somehow. The fact that it's buried means that it is sufficiently inaccessible to most people. It will still be vulnerable to certain risks, including backhoes, anything else that disrupts the ground (freight derailments, earthquakes, etc), but those are all more or less natural hazards that you protect against with redundancy. The guy who has technical specifics about your fiber network, and who picks your vulnerable points and hits you with a hacksaw, that's just always going to be much more complex to defend against. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Apr 13, 2009, at 11:12 AM, Mikael Abrahamsson wrote:

...

Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases.

It doesn't stop it, it just makes it slightly harder, and they'll go after another point.

IMHO, I think manhole locks would only serve to HEIGHTEN the threat, not minimize it. Flag this under the whole "obscurity" category, but think about this - if you're a vandal itching to do something stupid, and you see a bunch of manhole covers and a couple of them have locks on them, which ones are you going to target? The ones with the locks, of course. Why? Because by the very existence of the locks, it implies there's something of considerable value beyond the lock. -Andy

I guess the next generation fiber networks will need to be installed with tunnel boring machines and just not surface anywhere except the endpoints :) After all, undersea cables get along just fine without convenient access along their length... On Mon, Apr 13, 2009 at 12:12 PM, Mikael Abrahamsson <swmike@swm.pp.se>wrote:

On Mon, 13 Apr 2009, Dylan Ebner wrote:

Manhole locks are just going to stop vandalism, and I think the threat

...

to obstruction calculation just doesn't add up for that small level of isolated cases.

It doesn't stop it, it just makes it slightly harder, and they'll go after another point.

<http://swm.pp.se/bayarea.jpg>

This is the bay area as well... How long do you need to spend with a torch to cut thru that? A couple of minutes?

There is absolutely no way you can stop a determined attacker, and it would increase cost a lot more than it's worth. Time is better spent stopping the few people who actually do these kinds of things, same way as it's not worth it for regular people to wear body armour all the time, just in case they might get shot, or have parachutes and emergency exits that work in mid-flight on commercial airliners. The various police agencies and the NTSB cost less in a cost/benefit analysis.

-- Mikael Abrahamsson email: swmike@swm.pp.se

It all comes down to money... It will cost them lots of it to get power and some type of readers installed to monitor manhole access... There has always been a lack of security on the telco side, this incident just brings it to light... In my town many of the verizon fios boxes are not locked and the wiring frame boxes for pots line neither.. Its all of a matter of how much cash they wanna throw at it... Sent on the Now Network� from my Sprint® BlackBerry -----Original Message----- From: "Dylan Ebner" <dylan.ebner@crlmed.com> Date: Mon, 13 Apr 2009 09:57:30 To: <nanog@nanog.org> Subject: RE: Fiber cut in SF area One thing that is missing here is before we can define "security" we need to define the "threat" and the "obstruction" the security creates. With an ATM machine, the threat is someone comes and steals the machine for the cash. The majority of the assailants in an ATM case are not interested in the access passwords, so that is not viewed as a threat by the bank. Then bank then says, "If we set really complicated passwords, our repair guys (or contractors) will not be able to fix them." So setting hard passwords is an obstruction. This happens every day, in every IT department in the world. So lets define the "Threat" to the fiber network? We know it isn't monetary as their isn't much value in selling cut sections of fiber. So that leaves out your typical ATM theif. That leaves us with directed attack, revenge or pure vandalism. In a directed attack or revenge scenario, which is what this case looks like, how are manhole locks going to help? If it is was the fiber union, wouldn't they already have the keys anyway? If this was some kind of terrorism scenario wouldn't they also have the resources to get the keys, either by getting employed by the phone company or the fiber union or any one of the other thousand companies that would need those keys? Manhole locks are just going to stop vandalism, and I think the threat to obstruction calculation just doesn't add up for that small level of isolated cases. Here in Qwest territory, manhole locks would be disasterours for repair times. We have had times when our MOE network has an outage and Qwest cannot fix the problem because their repair guys don't have the keys to their own buildings. Seriously. Their own buildings. Ultimately, what really needs to be addresses is the redundancy problem. And this needs to be addresses by everyone who was affected, not just ATT and Verizon, etc. A few years ago we had a site go down when a sprint DS-3 was cut. This was a major wake-up call for us because we had 2 t-1's for the site and they were suppose to have path divergence. And they did, up to the qwest CO where they handed off the circuit to sprint. In the end, we built in workflow redundancies so if any site goes down, we can still operate at near 100% capacity. My point is, it is getting harder and harder to gurantee path divergence and sometimes the redundancies need to be built into the workflow instead of IT. But that does't mean we cannot try. I remember during Katrima a datacenter in downtown New Orleans managed to stay online for the duration of disaster. These guys were on the ball and it paid off for them. In the end, as much as I like to blame the phone companies when we have problems, I also have to take some level of responsibility. And with each of these types of incidents we learn. For everyone affected, you now know even though you have two carriers, you do not have path divergence. And for everyone who colos at an affected Datacenter and get's your service from that center, you know they don't have divergence. So we need to ask ourselves, "where do we go from here?" It will be easier to get more divergence than secure all the manholes in the country. Dylan Ebner, Network Engineer Consulting Radiologists, Ltd. 1221 Nicollet Mall, Minneapolis, MN 55403 ph. 612.573.2236 fax. 612.573.2250 dylan.ebner@crlmed.com www.consultingradiologists.com -----Original Message----- From: Joe Greco [mailto:jgreco@ns.sol.net] Sent: Sunday, April 12, 2009 7:12 AM To: Mike Lewinski Cc: nanog@nanog.org Subject: Re: Fiber cut in SF area

Joe Greco wrote:

...

My point was more the inverse, which is that a determined, equipped,

...

and knowledgeable attacker is a very difficult thing to defend against.

"The Untold Story of the World's Biggest Diamond Heist" published recently in Wired was a good read on that subject:

http://www.wired.com/politics/law/magazine/17-04/ff_diamonds

Thanks, *excellent* example.

...

Which brings me to a new point: if we accept that "security by obscurity is not security," then, what (practical thing) IS security?

Obscurity as a principle works just fine provided the given token is obscure enough.

Of course, but I said "if we accept that". It was a challenge for the previous poster. ;-)

Ideally there are layers of "security by obscurity" so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either.

I think the problem is that the notion of "security by obscurity isn't

security" was originally meant to convey to software vendors "don't rely on closed source to hide your bugs" and has since been mistakenly

applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have.

That's really it, and bringing us back to the fiber discussion, we are forced, generally, to rely on obscurity. In general, talk to a hundred people on the street, few of them are likely to be able to tell you how fiber gets from one city to another, or that a single fiber may be carrying immense amounts of traffic. Most people expect that it just all works somehow. The fact that it's buried means that it is sufficiently inaccessible to most people. It will still be vulnerable to certain risks, including backhoes, anything else that disrupts the ground (freight derailments, earthquakes, etc), but those are all more or less natural hazards that you protect against with redundancy. The guy who has technical specifics about your fiber network, and who picks your vulnerable points and hits you with a hacksaw, that's just always going to be much more complex to defend against. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Matthew Petach writes:

"protected rings" are a technology of the past. Don't count on your vendor to provide "redundancy" for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc.

This completely ignores the grooming problem. About five years ago, we had a major WebEx outage caused by our diverse path routed fibers both being groomed into the same new cable / new path. We had the contracts. We paid the money. We got the data. We got updates to the data. The updates said we were still fine and all good. The new data lied. Downtown SJ backhoe hit damaged the cable, and took down 1 of our 2 links. As nobody was sure what was in it they failed to notify us that they were about to chop the rest of it to repair the bundle. So, about an hour after we lost the first leg, we went dark, and there was no coming back until the splices were all done. (typically, while the whole operations team was out at an offisite teambuilding effort. pagers go beep beep beep, and everyone hops back in the cars...) We ran it up the flagpole to CEO level of the fiber vendor (aggregator) and fiber physical plant owner (big 4 ISP), as we were paying $$$ for bandwidth and were a Highly Visible Client, and were told that they'd been making a best effort and couldn't guarantee any better in the future, no matter how much we paid or who we sued. They were very apologetic, but insisted that best effort means just that. The only way to be sure? Own your own fiber. Use a microwave link backup. You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business. -george william herbert gherbert@retro.com

Matthew Petach wrote:

...

George William Herbert <gherbert@retro.com> wrote: Matthew Petach writes:

...

"protected rings" are a technology of the past. Don't count on your vendor to provide "redundancy" for you. Get two unprotected runs for half the cost each, from two different providers, and verify the path separation and diversity yourself with GIS data from the two providers; handle the failover yourself. That way, you *know* what your risks and potential impact scenarios are. It adds a bit of initial planning overhead, but in the long run, it generally costs a similar amount for two unprotected runs as it does to get a protected run, and you can plan your survival scenarios *much* better, including surviving things like one provider going under, work stoppages at one provider, etc.

This completely ignores the grooming problem.

Not completely; it just gives you teeth for exiting your contract earlier and finding a more responsible provider to go with who won't violate the terms of the contract and re-groom you without proper notification.

That's a post-facto financial recovery / liability limitation technique, not a high availability / hardening technique...

I'll admit I'm somewhat simplifying the scenario, in that I also insist on no single point of failure, so even an entire site going dark doesn't completely knock out service; those who have been around since the early days will remember my email to NANOG about the gas main cut in Santa Clara that knocked a good chunk of the area's connectivity out, *not* because the fiber was damaged, but because the fire marshall insisted that all active electrical devices be powered off (including all UPSes) until the gas in the area had dissipated. Ever since then, I've just acknowledged you can't keep a single site always up and running; there *will* be events that require it to be powered down, and part of my planning process accounts for that, as much as possible, via BCP planning.

I was less than a mile away from that, I remember it well. My corner cube even faced in that direction. I heard the noise then the net went poof. One of those "Oh, that's not good at all" combinations.

Now, I'll be the first to admit it's a different game if you're providing last-mile access to single-homed customers. But sitting on the content provider side of the fence, it's entirely possible to build your infrastructure such that having 3 or more OC192s cut at random places has no impact on your ability to carry traffic and continue functioning.

...

You have to get out of the game the fiber owners are playing. They can't even keep score for themselves, much less accurately for the rest of us. If you count on them playing fair or right, they're going to break your heart and your business.

You simply count on them not playing entirely fair, and penalize them when they don't; and you have enough parallel contracts with different providers at different sites that outages don't take you completely offline.

The problem with grooming is that in many cases, due to provider consolidation and fiber vendor consolidation and cable swap and so forth, you end up with parallel contracts with different providers at different sites that all end up going through one fiber link anyways. I had (at another site) separate vendors with fiber going northbound and southbound out of the two diverse sites. Both directions from both sites got groomed without notification. Slightly later, the northbound fiber was Then rerouted a bit up the road, into a southbound bundle (same one as our now-groomed southbound link), south to another datacenter then north again via another path. To improve route reduncancy northbound overall, for the providers' overall customer links. And the shared link south of us was what got backhoed. This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully. But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc).... There may be 4 or 5 nines, but the 1 at the end has your name on it. -george william herbert gherbert@retro.com

Rofl Matt, I was recently laid off from my job for 'economic' reasons, what you say is deadly accurate. Bravo! :) On Mon, Apr 13, 2009 at 7:01 PM, Matthew Petach <mpetach@netflight.com>wrote:

On 4/13/09, George William Herbert <gherbert@retro.com> wrote:

...

Matthew Petach wrote:

...

...

George William Herbert <gherbert@retro.com> wrote: Matthew Petach writes:

[much material snipped in the interests of saving precious electron resources...]

...

This was all in one geographical area. Diversity out of area will get you around single points like that, if you know the overall topology of the fiber networks around the US and chose locations carefully.

But even that won't protect you against common mode vendor hardware failures, or a largescale BGP outage, or the routing chaos that comes with a very serious regional net outage (exchange points, major undersea cable cuts, etc)....

There may be 4 or 5 nines, but the 1 at the end has your name on it.

Ultimately, I think a .sig line I saw years back summed it up very succinctly:

"Earth is a single point of failure."

Below that, you're right, we're all just quibbling about which digits to put to the right of the decimal point. If the entire west coast of the US drops into the ocean, yes, having my data backed up on different continents will help; but I'll be swimming with the sharks at that point, and won't really be able to care much, so the extent of my disaster planning tends to peter out around the point where entire states disappear, and most definitely doesn't even wander into the realm of entire continents getting cut off, or the planet getting incinerated in a massive solar flare.

Fundamentally, though, I think it's actually good we have outages periodically; they help keep us employed. When networks run too smoothly, management tends to look upon us as unnecessary overhead that can be trimmed back during the next round of layoffs. The more they realize we're the only bulwark against the impending forces of chaos you mentioned above, the less likely they are to trim us off the payroll.

Matt

Note--tongue was firmly planted in cheek; no slight was intended against those who may have lost jobs recently; post was intended for humourous consumption only; any resemblence to useful content was purely coincidental and not condoned by any present or past employer. Repeated exposure may be habit forming. Do not read while operating heavy machinery.

-- Respectfully, Chris Hart George Carlin<http://www.brainyquote.com/quotes/authors/g/george_carlin.html> - "Frisbeetarianism is the belief that when you die, your soul goes up on the roof and gets stu...

True enough Jorge, however, we need full-orbed perspective here....it's not merely beating a dead horse; as far as topic goes, it is purely edification in the nth degree, manner, fashion. This is the lingua franca of this forum, and those who chose to read it, or not. Not merely pointed dialogue or geek speaks for the consummate net head ideologue. After all, iron sharpens iron. Demagoguery gives rise to elitism. No demonization here. You're ok. :-) Cheerio, Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Tuesday, April 14, 2009 9:21 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area

"Earth is a single point of failure."

On top of that, one basic principle of telecommunications: No matter how much diversity and path redundancy, tons of concrete or titanium sealed fiber vaults you have, in the data exchange between points A and B there will be always two single points of failure: A and B. IMHO, this thread is getting way off topic, boring and useless. Fiber cut is over, there will be many more, move on ... Cheers Jorge ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.

Cool enough. :-) Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fe, New Mexico 87502 Bus. Ph.: 505.827.2851 "We move the information that moves your world." -----Original Message----- From: Jorge Amodio [mailto:jmamodio@gmail.com] Sent: Tuesday, April 14, 2009 11:31 AM To: nanog@nanog.org Subject: Re: Fiber cut in SF area

True enough Jorge, however, we need full-orbed perspective here....it's not merely beating a dead horse; as far as topic goes, it is purely edification in the nth degree, manner, fashion. This is the lingua franca of this forum, and those who chose to read it, or not.  Not merely pointed dialogue or geek speaks for the consummate net head ideologue. After all, iron sharpens iron. Demagoguery gives rise to elitism. No demonization here. You're ok. :-)

I know, I don't mind the dialogue but IMHO besides trying to define which is the best way to seal a manhole, I'd rather see a more constructive discussion from an operational perspective. I really doubt that the big guys who own the fibers will make a rational decision about how to build their networks reading NANOG when the underlaying problem is not just technical or operational. For example, based on the experience with this outage, what's was out, how many users were affected, how the network operator's community handled the issue, what information was available, what kind of communications we used, what we did wrong, what we did right. BTW, now I know where to get a good padlock for my shack :-) Cheers Jorge ______________________________________________________________________ This inbound email has been scanned by the MessageLabs Email Security System. ______________________________________________________________________ Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.

Peter Beckman [mailto:beckman@angryox.com] wrote:

Sent: Monday, April 13, 2009 11:19 AM To: Dylan Ebner Cc: nanog@nanog.org Subject: RE: Fiber cut in SF area

On Mon, 13 Apr 2009, Dylan Ebner wrote:

...

It will be easier to get more divergence than secure all the manholes in the country.

I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution.

The only thing missing from your plan was a cost analysis. Cost of each, plus operational costs, * however many of each type. How much would that be? Then amortize that out to our bills. Extra credit: would you pay for it? Chris

Hi Peter, You wrote:

So, let's see. I'm pulling numbers out of my butt here,

<snip>

Total cost...is about $3000 per mile for equipment

<snip>

It could run on an overhead monorail

<snip>

Network it all

<snip>

Confickr-type domains to make sure

I get the feeling you haven't deployed or operated large networks. You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions. Joel is right: the thing about the outdoors is there's a lot of it. The cost over time investment of copper and fiber communucations networks, power transmission networks, cable transmission networks is pretty well documented elsewhere. Google around a little for them. The investment is tremendous. All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list.

...

Then amortize that out to our bills. Extra credit: would you pay for it?

Assuming bills in the hundreds of thousands of dollars per month, maybe to the millions of dollars, and then figure out what an outage costs you according to the SLAs.

Then figure out how much a breach and subsequent fiber cut costs you in SLA payouts or credits, multiply by 25%, and that's your budget. If the proposed system is less, why wouldn't you do it?

SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi-path diversity. Here come the microwave towers!

The idea is inspired by the way Google does their datacenters -- use cheap, off-the-shelf hardware, network it together in smart ways, make it energy efficient, ... profit!

Works great inside four walls.

Anyone want to invest? Maybe I should start the business.

Nahh, I already have a web cam on my Smarties orb. What else do I really need? Chris

This all implies that the majority of fiber is in "tunnels" that can be monitored. In my experience, almost none of it is in tunnels. In NYC, it's usually buried in conduits directly under the street, with no access, except through the man holes which are located about every 500 feet. In LA, a large amount of the fiber is direct bored under the streets, with access from hand holes and splice boxes located in the grassy areas between the street and the side walks. Along train tracks, the fiber is buried in conduits which are direct buried in the direct along side the train tracks, with hand holes every 1000 feet or so. In any of these scenarios, especially in the third, where the fiber might run through a rural area with no road access and no cellphone coverage. Simply walk through the woods to the train tracks, put open a hand hole and snip, snip, snip, fiber cut. Shane Ronan On Apr 13, 2009, at 5:54 PM, Peter Beckman wrote:

On Mon, 13 Apr 2009, chris.ranch@nokia.com wrote:

...

I get the feeling you haven't deployed or operated large networks.

Nope.

...

You never did say what the multiplier was. How many miles or detection nodes there were. Think millions. The number that popped into my head when thinking of active detection measures for the physical network is $billions.

It depends on where you want to deploy it and how many miles you want to protect. I was thinking along the lines of $1.5 million for 1000 miles of tunnel, equipment only. It assumes existing maintenance crews would replace sensors that break or go offline, and that those expenses already exist.

...

All for a couple of minutes advanced notice of an outage? Would it reduce the risk? No. Would it reduce the MTBF or MTTR? No. Of all outages, how often does this scenario (or one that would trigger your alarm) occur? I'm sure it's down on the list.

What if you had 5 minutes of advanced notice that something was happening in or near one of your Tunnels that served hundreds of thousands of people and businesses and critical infrastructure? Could you get someone on site to stop it? Maybe. Is it worth it? Maybe.

Given my inexperience with large networks, maybe fiber cuts and outages due to vandals, backhoes and other physical disruptions are just what we hear about in the news, and that it isn't worth the expense to monitor for those outages. If so, my idea seems kind of silly.

...

SLA's account for force de majure (including sabotage), so I really doubt there will be any credits. In fact, there will likely be an uptick on spending as those who really need nines build multi-provider multi- path diversity. Here come the microwave towers!

*laugh* Thank goodness for standardized GIS data. :-)

--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

On Mon, 13 Apr 2009 09:18:04 -0500 Stephen Sprunk <stephen@sprunk.org> wrote:

Mike Lewinski wrote:

...

Joe Greco wrote:

...

Which brings me to a new point: if we accept that "security by obscurity is not security," then, what (practical thing) IS security?

Obscurity as a principle works just fine provided the given token is obscure enough. Ideally there are layers of "security by obscurity" so compromise of any one token isn't enough by itself: my strong ssh password (1 layer of obscurity) is protected by the ssh server key (2nd layer) that is only accessible via vpn which has it's own encryption key (3rd layer). The loss of my password alone doesn't get anyone anything. The compromise of either the VPN or server ssh key (without already having direct access to those systems) doesn't get them my password either.

I think the problem is that the notion of "security by obscurity isn't security" was originally meant to convey to software vendors "don't rely on closed source to hide your bugs" and has since been mistakenly applied beyond that narrow context. In most of our applications, some form of obscurity is all we really have.

The accepted standard is that a system is secure iff you can disclose _all_ of the details of how the system works to an attacker _except_ the private key and they still cannot get in -- and that is true of most open-standard or open-source encryption/security products due to extensive peer review and iterative improvements. What "security by obscurity" refers to are systems so weak that their workings cannot be exposed because then the keys will not be needed, which is true of most closed-source systems. It does _not_ refer to keeping your private keys secret.

Correct. Open source and open standards are (some) ways to achieve that goal. They're not the only ones, nor are they sufficient. (Consider WEP as a glaring example of a failure of a standards process.) On the other hand, I was once told by someone from NSA that they design all of their gear on the assumption that Serial #1 of any new crypto device is delivered to the Kremlin. This principle, as applied to cryptography, was set out by Kerckhoffs in 1883; see http://www.petitcolas.net/fabien/kerckhoffs/ for details.

Key management is considered to be an entirely different problem. If you do not keep your private keys secure, no security system will be able to help you.

Yes. One friend of mine likens insecurity to entropy: you can't destroy it, but you can move it around. For example, cryptography lets you trade the insecurity of the link for the insecurity of the key, on the assumption that you can more easily protect a few keys than many kilometers of wire/fiber/radio. --Steve Bellovin, http://www.cs.columbia.edu/~smb

On Monday 13 April 2009 11:06:55 Roy wrote:

A friend mentioned at dinner yesterday that he spotted several AT&T trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.

:-) Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all.

Yes, they could create a solution for this that will cost money, or they could just take out the welding specs and go to town for a fraction of the price. This type of stuff is typical of incident response... Fix the bleeding and create a long term solution that won't be as big of an impact. Regards, James Pleger e: jpleger@gmail.com g: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9D7141C9 On Apr 13, 2009, at 8:49 AM, Church, Charles wrote:

Wouldn't some authentication system be more useful than trying to lock all the manholes? Picture a system maybe using RFID or some other radio system where you walk up to manhole, wave your 'wand' (like a Mobil Speedpass), you hear a couple beeps, and you're cleared to open the manhole. Without authenticating, you can still get in, but the NOCs at local utilities and telcos are notified, maybe police as well. If you can tie access to a particular person's ID, I doubt that person will misuse it. Of course, this requires power and battery backup. On the other hand, maybe it's time to put the blame on the unions. If the saboteur is found to be a union member, maybe penalize the entire union somehow, since they're acting like a terrorist group at that point.

Chuck

-----Original Message----- From: Lamar Owen [mailto:lowen@pari.edu] Sent: Monday, April 13, 2009 11:22 AM To: nanog@nanog.org Subject: Re: Cart and Horse

On Monday 13 April 2009 11:06:55 Roy wrote:

...

A friend mentioned at dinner yesterday that he spotted several AT&T trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.

:-)

Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all.

On 4/13/09, Lamar Owen <lowen@pari.edu> wrote:

On Monday 13 April 2009 11:06:55 Roy wrote:

...

A friend mentioned at dinner yesterday that he spotted several AT&T trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.

:-)

Sounds like a cutting torch or portable chop saw will become standard service equipment for them after all.

*heh* Just in case the next vandals slice the fiber, then weld the manhole covers shut on the way out? I guess the only thing worse would be for the vandals to have a truckload of quick-drying cement with them; slice the fiber, dump quick-drying cement into the vault, pop the lid on, tamp thermite in the gap around the rim and flash weld it shut. Talk about creating an extended outage scenario. ^_^;

On Apr 13, 2009, at 11:59 AM, Robert Glover wrote:

This bears investigating. I live 3 blocks away. Looks like I'm going on a stroll after work tonight.

Bobby Glover Director of Information Services South Valley Interet (AS4307) ----- Original Message ----- From: "Roy" <r.engehausen@gmail.com> To: "nanog" <nanog@merit.edu> Sent: Monday, April 13, 2009 8:06 AM Subject: Cart and Horse

...

A friend mentioned at dinner yesterday that he spotted several AT&T trucks next to manholes in the area affected by the fiber cut. They were busy welding the manhole covers to their rims.

Yeah, I would have loved to be on the wall during that conversation: "So, how can we lock people out of the manholes?" "We could put locks on them?" "No, someone could just cut the locks" <starts laughing>" We could weld them shut" <still laughing> <pointed eared boss>"Good idea, do it" <stops laughing, serious look>"Really sir?" "Yes, make it happen" <all nervously look at each other> "Uh, okay..."

On Sat, 11 Apr 2009, Joe Greco wrote:

Public key crypto is, pretty much by definition, reliant on the obscurity of private keys in order to make it work.

In security terms, public key crypto is not "security by obscurity", as the obscurity part is related to how the method works, and the key is secret. So "openssh" is definitely not "security by obscurity", as anyone with programming knowledge can find out exactly how everything works, and the only thing that is a secret is the private key generated. -- Mikael Abrahamsson email: swmike@swm.pp.se

On Sat, Apr 11, 2009 at 11:10 AM, Florian Weimer <fw@deneb.enyo.de> wrote:

* Joe Greco:

...

The ATM machine is somewhat protected for the extremely obvious reason that it has cash in it, but an ATM is hardly impervious.

http://www.youtube.com/watch?v=4P8WM8ZZDHk

Heh.  Once you install ATMs into solid walls, the attacks get a tad more interesting.  In some places of the world, gas detectors are almost mandatory because criminals pump gas into the machine, ignite it, and hope that the explosion blows a hole into the machine without damaging the money (which seems to work fairly well if you use the right gas at the right concentration).

also, there is the fact that some very large percentage of ATM machines were installed with the same admin passwd setup. I recall ~1.5 yrs ago some news about this, and that essentially banks send out the ATM machines with a stock passwd (sometimes the default which is documented in easily google-able documents) per bank (BoFA uses passwd123, Citi uses passwd456 ....) I'm not sure that the manholes == atm discussion is valid, but in the end the same thing is prone to happen to the manholes, there isn't going to be a unique key per manhole, at best it'll be 1/region or 1/manhole-owner. In the end that key is compromised as soon as the decision is made :( Also keep in mind that keyed locks don't really provide much protection, since anyone can order lockpicks over the interwebs these days, even to states where ownership is apparently illegal :( -Chris

On Sat, 11 Apr 2009, Christopher Morrow wrote:

I'm not sure that the manholes == atm discussion is valid, but in the end the same thing is prone to happen to the manholes, there isn't going to be a unique key per manhole, at best it'll be 1/region or 1/manhole-owner. In the end that key is compromised as soon as the decision is made :( Also keep in mind that keyed locks don't really provide much protection, since anyone can order lockpicks over the interwebs these days, even to states where ownership is apparently illegal :(

Too bad there isn't 1Password for manhole covers. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

On Saturday 11 April 2009 08:31:55 Joe Greco wrote:

...

Speaking of that, a manhole cover is typically protecting some hole, accessway, or vault that's made out of concrete.

An oxyacetylene torch or a plasma cutter will slice through regular steel manhole covers in minutes.

Yes, but we were discussing locked covers, which (given the underlying assumptions of this discussion) might be a bit heavier. Further, it would be vaguely suspicious and more noticeable for a "road crew" or "power company" truck to be deploying such gear, might draw more attention.

The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather.

More expense. :-)

One way of making a site more resistant to 'inside job' issues is with SCIF- like controls (see http://en.wikipedia.org/wiki/Sensitive_Compartmented_Information_Facility ) and using combination locks such as the Sargent and Greenleaf 8077AD for control, and the S&G 833 superpadlock for security (see http://www.sargentandgreenleaf.com/PL-833.php ). The tech would have the 833's key, and the area supervisor the combination. The 8077AD's combination is very easily changed in the field, and could be changed frequently. The key to this method's success is that the keyholder to the 833 cannot have the combination, and the holder of the combination cannot have an 833 key. Requires a certain atmosphere of distrust, unfortunately. And slows repairs way down, especially if the 833's key is lost....

Certainly it is *possible* to do it, but given the other variables, does it make *sense*? Consider what I was saying about just going to town with a backhoe. You have a lot to protect. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Christopher Morrow <morrowc.lists@gmail.com> writes:

and I also would ask.. what's the cost/risk here? 'We' lost at best ~1day for some folks in the outage, nothing global and nothing earth-shattering... This has happened (this sort of thing) 1 time in how many years? Expending $$ and time and people to go 'put padlocks on manhole covers' seems like spending in the wrong place...

as long as the west's ideological opponents want terror rather than panic, and also to inflict long term losses rather than short term losses, that's true. in this light you can hopefully understand why bollards to protect internet exchanges against truck bombs are not only penny wise pound foolish (since the manholes a half mile away won't be hardened or monitored or even locked) but also completely wrongheaded (since terrorists need publicity which means they need their victims to be fully able to communicate.) -- Paul Vixie

On Mon, 13 Apr 2009 14:39:23 EDT, Izaac said:

Do you realize that you're putting trust in the sane action of parties who conclude their reasoning process with destruction and murder?

And how is that different from a US general plotting destruction and the killing of enemy troops during an offensive? And yet we usually trust our generals and call them "sane".

On Sat, 11 Apr 2009, Lamar Owen wrote:

The locking covers I have seen here put the lock(s) on the inside cover cam jackscrew (holes through the jackscrew close to the inside cover seal rod nut), rather than on the outside cover, thus keeping the padlocks out of the weather.

I'm starting to wonder what makes more sense -- locking down thousands of miles of underground tunnel with mil-spec expensive locks that ideally keep unauthorized people out, OR simple motion and or video cameras in the tunnels themselves which relay their access back to a central facility, along with a video feed of sorts, to help identify who is there, whether approved or not. With locks, you know they gained access after the fact and that your locking wasn't sufficient enough. With active monitoring of the area where the cables live, you at least know the moment someone goes in, and have some lead time (and maybe a video) to do something to prevent it, or catch them in the act. Unfortunately, that kind of monitoring is also expensive and complex. I wonder what the cost of the outage was, and how much it might cost to monitor it? Would it be worth $2,000 per site per year? A great webcam, with day/night capability, and a cell phone, in a locked box, with a solar panel, on top of a pole, near the site. Sure, if you know it's there, taking it out is easy, but someone will still know something is wrong when it goes dark or the picture changes significantly. Are there some low-cost, highly-effective ways that the tunnels which carry our precious data and communications can at least be monitored remotely? Waiting for someone to cut a cable and then deploying a crew seems reactive, whereas knowing the moment someone goes INTO the tunnel is proactive, whether the person(s) are there to do some normal maintenance or something malicious. Beckman I suppose rats and other rodents could cause such a system to be too annoying to pay attention to. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------