Fiber cut - response in seconds?
http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR2009053002... Not sure if I fully believe the article. Responding to a fiber cut in seconds? I suppose it's possible if $TLA had people monitoring the construction from across the street, and they were in communication with the NOC.
I sent this to all of our transport people to.. Was quite curious as to what they'd use for this. However, they are the federal government - so anything is possible. -----Original Message----- From: Charles Wyble [mailto:charles@thewybles.com] Sent: Monday, June 01, 2009 2:41 PM To: nanog@nanog.org Subject: Fiber cut - response in seconds? http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR200905 3002114_pf.html Not sure if I fully believe the article. Responding to a fiber cut in seconds? I suppose it's possible if $TLA had people monitoring the construction from across the street, and they were in communication with the NOC.
It's pretty trivial if know where all the construction projects on your path are... I've seen this happen on a university campus several times. no black helicopters were involved. joel Charles Wyble wrote:
http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR2009053002...
Not sure if I fully believe the article. Responding to a fiber cut in seconds?
I suppose it's possible if $TLA had people monitoring the construction from across the street, and they were in communication with the NOC.
Joel Jaeggli wrote:
It's pretty trivial if know where all the construction projects on your path are...
How so? Setup OTDR traces and watch them?
I've seen this happen on a university campus several times. no black helicopters were involved.
Care to expand on the methodology used? A campus network is a lot different then a major metro area.
I'm not sure why this sounds so surprising or impressive... given g$vt budgets. Monitoring software using a pair of fibers in your bundle. OTDR or similar digital diagnostics. You detect a loss, you figure out how many feet away it is. You look at your map. A simpler way to do it (if you don't mind burning lots of fiber pairs) would be to loop up a pair of fibers (or add a reflectance source every 1000 ft or so -- spliced into the cable). You can figure out to within a thousand feet once you know WHICH set of loops has died. Given it almost always involved construction crews, you drive until you see backhoes for your final approximation. If I were the gov't I'd have originally opted for #2, and then moved to #1. "Seconds" is just a function of how far away the responding agency's personnel ( monitoring the loop ) were from the cut. Obviously we are talking about a few miles tops. Plenty of people used to have a single pair in each bundle for "testing". Its relatively trivial to make that a test pair live. This is all predicated on you actually keeping your toplogy up-to-date. Deepak Jain AiNET Charles Wyble wrote:
Joel Jaeggli wrote:
It's pretty trivial if know where all the construction projects on your path are...
How so? Setup OTDR traces and watch them?
I've seen this happen on a university campus several times. no black helicopters were involved.
Care to expand on the methodology used? A campus network is a lot different then a major metro area.
Charles Wyble wrote:
Joel Jaeggli wrote:
It's pretty trivial if know where all the construction projects on your path are...
How so? Setup OTDR traces and watch them?
When you lose link on every pair in a bundle, but don't lose any of the buildings you're serving via diverse paths, you have a pretty good idea what happened. Knowing which of the three construction projects on that path is likely to be digging a trench is a facilities issue.
I've seen this happen on a university campus several times. no black helicopters were involved.
Care to expand on the methodology used? A campus network is a lot different then a major metro area.
Given the location the guys in the blacks suvs likely have at least situational awareness of all of the contruction projects in their immediate vicinity. they don't have to monitor everyone's cable, just their own and near instantaneous response implies proximity so it may well be more akin to a campus network.
Joel Jaeggli wrote:
Charles Wyble wrote:
Joel Jaeggli wrote:
It's pretty trivial if know where all the construction projects on your path are... How so? Setup OTDR traces and watch them?
When you lose link on every pair in a bundle, but don't lose any of the buildings you're serving via diverse paths, you have a pretty good idea what happened. Knowing which of the three construction projects on that path is likely to be digging a trench is a facilities issue.
Right. So why the "near instant" response time. If it's a diverse path, one would imagine that they could respond in a few hours or a day and not have any impact. The fact that they are so closely monitoring the construction and wanting to fix it that fast seems a bit over the top for redundant systems.
I've seen this happen on a university campus several times. no black helicopters were involved. Care to expand on the methodology used? A campus network is a lot different then a major metro area.
Given the location the guys in the blacks suvs likely have at least situational awareness of all of the contruction projects in their immediate vicinity.
One would hope. Though given the archaic nature of many govt systems, that could involve a lot of manual paper pulling... or are the bid/reward/permit systems all automated on the east coast? :) they don't have to monitor everyone's cable, just
their own and near instantaneous response implies proximity so it may well be more akin to a campus network.
True.
The fact that they are so closely monitoring the construction and wanting to fix it that fast seems a bit over the top for redundant systems.
Even despite what we saw recently in the SF bay area? If black helicopters are involved, I suspect this is about par on the paranoia scale.
Right. So why the "near instant" response time. If it's a diverse path, one would imagine that they could respond in a few hours or a day and not have any impact.
Just a guess, but: A cut cable is one thing. A cut cable in which people wearing different suits and driving a different brand of SUV might splice in a fiber tap is something altogether different. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
I do feel this might be the last post from Mr Pooser. :) Your on to them it seems. ;) A very interesting idea. I imagine it wouldn't be hard for foreign actors to get access to the data feed of construction, observe for signs of a cut and then splice in a tap. Though wouldn't that tap be found via the real response team? Dave Pooser wrote:
Right. So why the "near instant" response time. If it's a diverse path, one would imagine that they could respond in a few hours or a day and not have any impact.
Just a guess, but: A cut cable is one thing. A cut cable in which people wearing different suits and driving a different brand of SUV might splice in a fiber tap is something altogether different.
Charles Wyble wrote:
I do feel this might be the last post from Mr Pooser. :)
Your on to them it seems. ;)
A very interesting idea. I imagine it wouldn't be hard for foreign actors to get access to the data feed of construction, observe for signs of a cut and then splice in a tap.
Though wouldn't that tap be found via the real response team?
No. And here's why: If you're a naughty foreign intelligence team, and you know your stuff, you already know where some of the cables you'd really like a tap on are buried. When you hear of a construction project that might damage one, you set up your innocuous white panel truck somewhere else, near a suitable manhole. When the construction guy with a backhoe chops the cable (and you may well slip him some money to do so), *then* you put your tap in, elsewhere, with your actions covered by the downtime at the construction site. That's why the guys in the SUVs are in such a hurry, because they want to close the window of time in which someone can be tapping the cable elsewhere. At least that's what I heard. I read it somewhere on the internet. Definitely. Not at all a sneaky person. No sir. Dave W At least I'm in Britain. *Slightly* harder for the NSA to make me disappear ;-)
No. And here's why: If you're a naughty foreign intelligence team, and you know your stuff, you already know where some of the cables you'd really like a tap on are buried. When you hear of a construction project that might damage one, you set up your innocuous white panel truck somewhere else, near a suitable manhole. When the construction guy with a backhoe chops the cable (and you may well slip him some money to do so), *then* you put your tap in, elsewhere, with your actions covered by the downtime at the construction site. That's why the guys in the SUVs are in such a hurry, because they want to close the window of time in which someone can be tapping the cable elsewhere.
At least that's what I heard. I read it somewhere on the internet. Definitely. Not at all a sneaky person. No sir.
And if you were a naughty foreign intelligence team installing a tap, or a bend, or whatever in the fiber contemporaneously with a known cut, you could also reamplify and dispersion compensate for the slight amount of affect your work is having so that when its tested later, the OTDR is blind to your work. Ah, the fun of Paranoia, Inc. Deepak Jain AiNET
It would also be cheaper to add an additional layer of security with encryption vs. roving teams of gun toting manhole watchers. YMMV, Best! Marty On 6/2/09, Deepak Jain <deepak@ai.net> wrote:
No. And here's why: If you're a naughty foreign intelligence team, and you know your stuff, you already know where some of the cables you'd really like a tap on are buried. When you hear of a construction project that might damage one, you set up your innocuous white panel truck somewhere else, near a suitable manhole. When the construction guy with a backhoe chops the cable (and you may well slip him some money to do so), *then* you put your tap in, elsewhere, with your actions covered by the downtime at the construction site. That's why the guys in the SUVs are in such a hurry, because they want to close the window of time in which someone can be tapping the cable elsewhere.
At least that's what I heard. I read it somewhere on the internet. Definitely. Not at all a sneaky person. No sir.
And if you were a naughty foreign intelligence team installing a tap, or a bend, or whatever in the fiber contemporaneously with a known cut, you could also reamplify and dispersion compensate for the slight amount of affect your work is having so that when its tested later, the OTDR is blind to your work.
Ah, the fun of Paranoia, Inc.
Deepak Jain AiNET
-- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
Cheaper? To quote sneakers.... were the united states govt. we don't do that sort of thing. Martin Hannigan wrote:
It would also be cheaper to add an additional layer of security with encryption vs. roving teams of gun toting manhole watchers.
YMMV,
Best!
Marty
On 6/2/09, Deepak Jain <deepak@ai.net> wrote:
No. And here's why: If you're a naughty foreign intelligence team, and you know your stuff, you already know where some of the cables you'd really like a tap on are buried. When you hear of a construction project that might damage one, you set up your innocuous white panel truck somewhere else, near a suitable manhole. When the construction guy with a backhoe chops the cable (and you may well slip him some money to do so), *then* you put your tap in, elsewhere, with your actions covered by the downtime at the construction site. That's why the guys in the SUVs are in such a hurry, because they want to close the window of time in which someone can be tapping the cable elsewhere.
At least that's what I heard. I read it somewhere on the internet. Definitely. Not at all a sneaky person. No sir. And if you were a naughty foreign intelligence team installing a tap, or a bend, or whatever in the fiber contemporaneously with a known cut, you could also reamplify and dispersion compensate for the slight amount of affect your work is having so that when its tested later, the OTDR is blind to your work.
Ah, the fun of Paranoia, Inc.
Deepak Jain AiNET
On Tue, 02 Jun 2009 13:54:44 EDT, Martin Hannigan said:
It would also be cheaper to add an additional layer of security with encryption vs. roving teams of gun toting manhole watchers.
Even if encrypted, you can probably do an amazing amount of traffic analysis to tell when something is afoot. Ask any pizzeria near State Dept or Pentagon. ;) (That, plus it's easier to break an encryption if you have gigabytes of data to work with, than if you don't have any data to work with...)
link-layer encryption for sonet/atm quite resistant to traffic analysis... The pipe is full of pdus whether you're using them or not. Valdis.Kletnieks@vt.edu wrote:
On Tue, 02 Jun 2009 13:54:44 EDT, Martin Hannigan said:
It would also be cheaper to add an additional layer of security with encryption vs. roving teams of gun toting manhole watchers.
Even if encrypted, you can probably do an amazing amount of traffic analysis to tell when something is afoot. Ask any pizzeria near State Dept or Pentagon. ;)
(That, plus it's easier to break an encryption if you have gigabytes of data to work with, than if you don't have any data to work with...)
On Tue, Jun 2, 2009 at 7:50 AM, Dave Wilson <richard.wilson@senokian.com> wrote:
No. And here's why: If you're a naughty foreign intelligence team, and you know your stuff, you already know where some of the cables you'd really like a tap on are buried. When you hear of a construction project that might damage one, you set up your innocuous white panel truck somewhere else, near a suitable manhole. When the construction guy with a backhoe chops the cable (and you may well slip him some money to do so), *then* you put your tap in, elsewhere, with your actions covered by the downtime at the construction site. That's why the guys in the SUVs are in such a hurry, because they want to close the window of time in which someone can be tapping the cable elsewhere.
Sounds like a lot of work to me. Wouldn't it be easier to just find the carrier neutral colo facilities where all the peering/transit between major networks happens, and pay them money to put up a fake wall that you can colo your optical taps behind? Drive Slow, and remember, don't open any doors that say "This Is Not An Exit", Paul Wall
Sounds like a lot of work to me. Wouldn't it be easier to just find the carrier neutral colo facilities where all the peering/transit between major networks happens, and pay them money to put up a fake wall that you can colo your optical taps behind?
Yeah.... it's not like that's ever gonna happen! :)
Drive Slow, and remember, don't open any doors that say "This Is Not An Exit",
ROFL
On Mon, 1 Jun 2009, Charles Wyble wrote:
Right. So why the "near instant" response time.
Extra budgets, job creation. Knowing ahead of time where and when work is going to be done (easily found out), have someone around the corner at a Starbucks so they can jump into action if/when something goes down. Just because you have a redundant path doesn't mean you shouldn't get the broken path repaired ASAP. Maybe there are only two paths. If the other goes down, and something happens and the Gov't can't mobilize in time, something bad happens. It's a perfect storm to be sure, but when you have the lives of 300 million people at stake, I appreciate the diligence. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
Joel Jaeggli wrote:
Given the location the guys in the blacks suvs likely have at least situational awareness of all of the contruction projects in their immediate vicinity.
This has to be the most backwards way of dealing with this problem. They know exactly where the construction is taking place - the plans are filed with the local municipality and all the relevant agencies have access. Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place? If these cables are THAT important, I'd think it would be critical to keep them from getting cut in the first place, rather than rushing out to fix them "within 24 hours". They could send guys out in white jumpsuits and hard hats and the backhoe operators would just assume it was normal bureaucracy at work (oops, we forgot to mark those cables on your map) rather than sooper sekrit black fiber that no one is supposed to know about - until they cut into it and the black SUVs show up and then they DO know about it - more than they need to know. jc
Elmar K. Bins wrote:
jcdill.lists@gmail.com (JC Dill) wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
*snicker*
You ever been to a construction site?
Yes. We have a number here to call "Before You Dig" and they send people out to mark where underground utilities are. It would be trivially easy for one more set of jump-suited and hard-hat-wearing people to show up during this phase of the project and mark one more line. For the most part the construction teams don't know and don't care who is marking the lines or who is responsible for each, they just want the lines marked (location and type of line - gas, electric, telco) so they can avoid cutting them. In this way the marking team would be "undercover" and the previously unmarked/unmapped line would be No Big Deal. When an unmarked line is cut and black SUVs show up (the opposite of "undercover"), the line becomes A Big Deal which is the opposite of what is intended. jc
In my experience they are required not only to mark the line, but to identify it with the initials of the owner. On Jun 2, 2009, at 10:44 AM, JC Dill wrote:
Elmar K. Bins wrote:
jcdill.lists@gmail.com (JC Dill) wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
*snicker*
You ever been to a construction site?
Yes. We have a number here to call "Before You Dig" and they send people out to mark where underground utilities are. It would be trivially easy for one more set of jump-suited and hard-hat-wearing people to show up during this phase of the project and mark one more line. For the most part the construction teams don't know and don't care who is marking the lines or who is responsible for each, they just want the lines marked (location and type of line - gas, electric, telco) so they can avoid cutting them. In this way the marking team would be "undercover" and the previously unmarked/ unmapped line would be No Big Deal. When an unmarked line is cut and black SUVs show up (the opposite of "undercover"), the line becomes A Big Deal which is the opposite of what is intended.
jc
sronan@fattoc.com (Shane Ronan) wrote:
In my experience they are required not only to mark the line, but to identify it with the initials of the owner.
Hell yeah - but that's not the point I wanted to make. For any given construction project, the main goal is to build something without destroying something else (unless it's planned to be destroyed). Unfortunately, this goal has to be broken into easy tasks for the people executing the work. And what leaks to them is "dig a hole". They definitely don't care whether they _will_ hit something. They do care after they hit something... (sometimes they'll try to cover up like someone did here; after cutting a whole bunch of fibre trunks, they decided to fill the just-dug hole with a ton of concrete...)
They usually hand out tin foil hats to the dig crew. A clear give away and easy to spot too. Next? On 6/2/09, JC Dill <jcdill.lists@gmail.com> wrote:
Elmar K. Bins wrote:
jcdill.lists@gmail.com (JC Dill) wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
*snicker*
You ever been to a construction site?
Yes. We have a number here to call "Before You Dig" and they send people out to mark where underground utilities are. It would be trivially easy for one more set of jump-suited and hard-hat-wearing people to show up during this phase of the project and mark one more line. For the most part the construction teams don't know and don't care who is marking the lines or who is responsible for each, they just want the lines marked (location and type of line - gas, electric, telco) so they can avoid cutting them. In this way the marking team would be "undercover" and the previously unmarked/unmapped line would be No Big Deal. When an unmarked line is cut and black SUVs show up (the opposite of "undercover"), the line becomes A Big Deal which is the opposite of what is intended.
jc
-- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
They usually hand out tin foil hats to the dig crew. A clear give away and easy to spot too. Next? On 6/2/09, JC Dill <jcdill.lists@gmail.com> wrote:
Elmar K. Bins wrote:
jcdill.lists@gmail.com (JC Dill) wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
*snicker*
You ever been to a construction site?
Yes. We have a number here to call "Before You Dig" and they send people out to mark where underground utilities are. It would be trivially easy for one more set of jump-suited and hard-hat-wearing people to show up during this phase of the project and mark one more line. For the most part the construction teams don't know and don't care who is marking the lines or who is responsible for each, they just want the lines marked (location and type of line - gas, electric, telco) so they can avoid cutting them. In this way the marking team would be "undercover" and the previously unmarked/unmapped line would be No Big Deal. When an unmarked line is cut and black SUVs show up (the opposite of "undercover"), the line becomes A Big Deal which is the opposite of what is intended.
jc
-- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
On Tue, 2 Jun 2009, JC Dill wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
Because if they DON'T hit the line, it is still a secret. Then again, if they DO hit the line, it's pretty obvious what the line is for and at least one place it runs. I wonder if the Gov't schedules a move of the line once it's operational security is comprimised by an accidental cut. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
On Tue, Jun 2, 2009 at 11:19 AM, Peter Beckman <beckman@angryox.com> wrote:
On Tue, 2 Jun 2009, JC Dill wrote:
Why do they "watch" and "monitor" rather than proactively go out and say "watch out, there's an unmarked cable here" and keep them from cutting the cable in the first place?
Because if they DON'T hit the line, it is still a secret.
Then again, if they DO hit the line, it's pretty obvious what the line is for and at least one place it runs. I wonder if the Gov't schedules a move of the line once it's operational security is comprimised by an accidental cut.
putting fiber in the ground isn't a quiet task...
-----Original Message----- From: Charles Wyble [mailto:charles@thewybles.com] Sent: Monday, June 01, 2009 7:10 PM To: nanog@nanog.org Subject: Re: Fiber cut - response in seconds?
Joel Jaeggli wrote:
It's pretty trivial if know where all the construction projects on your path are...
How so? Setup OTDR traces and watch them?
I've seen this happen on a university campus several times. no black helicopters were involved.
Care to expand on the methodology used? A campus network is a lot different then a major metro area.
Something like Fiber SenSys (http://www.fibersensys.com/) is probably used. Measures miniscule changes in light levels to tell whether or not fiber has been tampered with. As for the response in "seconds", I would have to say that the suits were parked right there watching, assuming the story is true. Not sure if anyone has ever tried to get anywhere in Tysons Corner during roadside construction (or during an afternoon drizzle for that matter), but I can guarantee you that it would be impossible without someone already being stationed onsite.
In a message written on Mon, Jun 01, 2009 at 03:40:31PM -0700, Charles Wyble wrote:
http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR2009053002...
Not sure if I fully believe the article. Responding to a fiber cut in seconds?
Folks who dig call "Miss Utility" (in Virginia, anyway) befor they dig to have folks come out and spray paint where everything is lcoated. On the back end, folks with cables in the ground subscribe to a feed of address information to know if they should go out and mark cables. I have no doubt the men in black SUV's have a feed of this data, and thus know when someone is going to be digging near their cable. Indeed, I can think of at least two instances where I was out surveying fiber digs where black SUV's seemed to be across the street the entire time. With the location having features like a metro tunnel under a US Army "classified" microwave tower it would not surprise me that they have someone in the area watching. I suspect they were waiting nearby, and when it went down went in not to tell folks they cut something, but rather to tell them that they cut nothing. Wink wink. Nudge nudge. -- Leo Bicknell - bicknell@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
On Mon, Jun 1, 2009 at 6:40 PM, Charles Wyble <charles@thewybles.com> wrote:
http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR2009053002...
Not sure if I fully believe the article. Responding to a fiber cut in seconds?
I suppose it's possible if $TLA had people monitoring the construction from across the street, and they were in communication with the NOC.
Dig Safe, Miss Utility, etc. notify potential dig impacted entities when activity is occurring around their assets and coordinate the marking of the utilities and start of construction in proximity to the targeted dig zone. This is why calling the state utility locator services is the law (everywhere that I'm aware of). The government isn't exempt from these notifications FWIW. The programs may have a slight tweak in the national capitol area. http://www.ncs.gov/ Best, -M< -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
On Jun 2, 2009, at 9:19 AM, Martin Hannigan wrote:
On Mon, Jun 1, 2009 at 6:40 PM, Charles Wyble <charles@thewybles.com> wrote:
http://www.washingtonpost.com/wp-dyn/content/article/2009/05/30/AR2009053002...
Not sure if I fully believe the article. Responding to a fiber cut in seconds?
I suppose it's possible if $TLA had people monitoring the construction from across the street, and they were in communication with the NOC.
Dig Safe, Miss Utility, etc. notify potential dig impacted entities when activity is occurring around their assets and coordinate the marking of the utilities and start of construction in proximity to the targeted dig zone. This is why calling the state utility locator services is the law (everywhere that I'm aware of). The government isn't exempt from these notifications FWIW. The programs may have a slight tweak in the national capitol area.
What you're likely interested in is TSP: http://tsp.ncs.gov/ This is something that is placed on your service when it's ordered and alters the design and engineering of the services. - Jared
participants (18)
-
Charles Wyble -
Christopher Morrow -
Dave Pooser -
Dave Wilson -
Deepak Jain -
Elmar K. Bins -
Eric Van Tol -
Jared Mauch -
Jason Fesler -
JC Dill -
Joel Jaeggli -
Leo Bicknell -
Martin Hannigan -
Paul Wall -
Peter Beckman -
Shane Ronan -
Valdis.Kletnieks＠vt.edu -
Warren Bailey