Outside plant protection, fiber cuts, interwebz down oh noes!
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute.... 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
On Thu, Apr 9, 2009 at 18:04, Charles Wyble <charles@thewybles.com> wrote:
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
I'll pipe in with this: No amount of money can deter a determined entity. If there is a will, there is a way, etc. Want to protect your "outside" plant, then make it resilient network-wise. There use to be a time when dual paths was acceptable, I (personally) think that quad paths should be the norm. -Jim P.
Hold on. Who says this sabotage? These incidents happen all the time without sabotage being involved. A ship sank off the coast of Pakistan and took out both international cables serving the country ... We had the undersea earthquake that seven seven cables in the Taiwan straits. The truth is that physical diversity is an ideal, not a reality. I have seen lots of accidents that took multiple operators and seriously disrupted in a given locality. The only difference here is that in the Heart of Geek Territory. Hence the Natives are restless ... Roderick S. Beck Director of European Sales Hibernia Atlantic -----Original Message----- From: Charles Wyble [mailto:charles@thewybles.com] Sent: Thu 4/9/2009 11:04 PM To: nanog@nanog.org Subject: Outside plant protection, fiber cuts, interwebz down oh noes! Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute.... 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
I didn't say it was sabatoage... It would appear
that this was a deliberate act
I tried to be very careful to say that it appears to have been sabatoage, but that it's not confirmed. Also this isn't the middle of the ocean, but cable underground. That usually doesn't get cut unless it's by a back hoe. And speaking of unions.... construction crews charge lots of money to work in the middle of the night, so it's usually avoided. :) Rod Beck wrote:
Hold on. Who says this sabotage?
These incidents happen all the time without sabotage being involved. A ship sank off the coast of Pakistan and took out both international cables serving the country ...
We had the undersea earthquake that seven seven cables in the Taiwan straits.
The truth is that physical diversity is an ideal, not a reality.
I have seen lots of accidents that took multiple operators and seriously disrupted in a given locality.
The only difference here is that in the Heart of Geek Territory. Hence the Natives are restless ...
Roderick S. Beck Director of European Sales Hibernia Atlantic
-----Original Message----- From: Charles Wyble [mailto:charles@thewybles.com] Sent: Thu 4/9/2009 11:04 PM To: nanog@nanog.org Subject: Outside plant protection, fiber cuts, interwebz down oh noes!
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
On Thu, Apr 9, 2009 at 7:00 PM, Charles Wyble <charles@thewybles.com> wrote:
I tried to be very careful to say that it appears to have been sabatoage, but that it's not confirmed.
T is offering a 6-figure bounty already for anyone with info.. I'd say it's pretty safe to assume.. -jamie
On Thu, Apr 09, 2009 at 03:04:16PM -0700, Charles Wyble wrote:
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
I think we'd only be speculating with no actual data surrounding the vaults the bundles traversed. That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry. I would also love it if bacon was healthy for me and didn't make my cholesterol 280. The bay area is also particularly unique in the sense that there aren't many available paths to run fiber. There are mountains on one side and the bay on the other. Your available diverse paths are "the left and right side of the tracks," and as a coworker pointed out the left has been full since 1996. -r
Ravi Pina wrote:
That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair? My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things. jc
On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote:
Ravi Pina wrote:
That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair?
Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing.
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things.
I wasn't thinking Exodus/C&W/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? -r
Not to turn this into an ethical typ discussion but this arguement would have to assume you could sue the telco not the 'vandal' due to a loss of life if it occured, and that, that dollar amt would be greater then 'securing' all cables. The cost to fix all pintos' gas tanks was only $11 per car unit and it was gambled, though they lost it was cheeper then the lawsuits, I'm betting the while fewer units, its order of magnatitudes more then 11$ per unit to 'secure' access points with a lot less certain negative lawsuit outcomes. Sent from my BlackBerry device on the Rogers Wireless Network -----Original Message----- From: Ravi Pina <ravi@cow.org> Date: Fri, 10 Apr 2009 01:51:16 To: JC Dill<jcdill.lists@gmail.com> Cc: nanog@nanog.org<nanog@nanog.org> Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote:
Ravi Pina wrote:
That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair?
Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing.
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things.
I wasn't thinking Exodus/C&W/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? -r
deleskie@gmail.com wrote:
Not to turn this into an ethical typ discussion but this arguement would have to assume you could sue the telco not the 'vandal' due to a loss of life if it occured, and that, that dollar amt would be greater then 'securing' all cables.
Internet lawyering is a different mailing list... joel
The cost to fix all pintos' gas tanks was only $11 per car unit and it was gambled, though they lost it was cheeper then the lawsuits, I'm betting the while fewer units, its order of magnatitudes more then 11$ per unit to 'secure' access points with a lot less certain negative lawsuit outcomes. Sent from my BlackBerry device on the Rogers Wireless Network
-----Original Message----- From: Ravi Pina <ravi@cow.org>
Date: Fri, 10 Apr 2009 01:51:16 To: JC Dill<jcdill.lists@gmail.com> Cc: nanog@nanog.org<nanog@nanog.org> Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote:
Ravi Pina wrote:
That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair?
Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing.
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things.
I wasn't thinking Exodus/C&W/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now.
Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through?
-r
On Fri, Apr 10, 2009 at 2:02 AM, <deleskie@gmail.com> wrote:
Not to turn this into an ethical typ discussion but this....
Maybe it's an ethical issue, with an ethical solution. Random news article from google:
Workers are seeking to preserve the health care benefit packages, said Libby Sayre, area director for District 9 of the union, which covers California, Nevada and Hawaii. Union leaders say members' health care costs would more than triple under AT&T's current proposal.
Sayre said AT&T posted a $12.9 billion profit last year and added there are few indications the company will be hit hard by the recession. She added its chief executive officer, Randall Stephenson, earns more than $10 million a year.
The article goes further to explain that there are 90,000 workers affected. The arithmetic here is pretty easy. So the operational plan is to try harder not to screw over your employees, could be. -Randy Fischer
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've really got ask if this thread has run it's course. Given the nature of earlier discussions of off-topic issues, I think we've pretty much jumped the shark with people's personal anecdotes of how to disable fiber connectivity. - - ferg - ---------- Forwarded message ---------- From: Ravi Pina <ravi@cow.org> Date: Thu, Apr 9, 2009 at 10:51 PM Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! To: JC Dill <jcdill.lists@gmail.com> Cc: "nanog@nanog.org" <nanog@nanog.org> On Thu, Apr 09, 2009 at 10:22:41PM -0700, JC Dill wrote:
Ravi Pina wrote:
That said one would *hope* vault access is not trivial and there are mechanisms in place to alert of unauthorized, unlawful entry.
I regularly drove on these roads when these lines were being put in up-and-down the SF Peninsula. There are 4 manhole covers every 1/4 mile or so that provide access to this fiber. Do the math. Multiply by the number of miles of fiber runs across the world, and the number of access points per mile on each run. Exactly how do you plan to make "vault access non-trivial" and yet make the access as easy as it needs to be for routine maintenance and repair?
Having never been in a vault or know how to get in one other than apparently lifting a manhole cover I can't possible answer that with anything more than guessing.
My guess is that it is probably less expensive in the long run to leave them unprotected and just fix the problems when they occur than to try to "secure" the vaults and deal with the costs and extended outage delays when access it "secured" and it takes longer to get into a vault to fix things.
I wasn't thinking Exodus/C&W/SAVVIS/Whoever level security, but considering communications cables traverse such sites it is hardly unreasonable to think they could implement some alarm that is centrally monitored by a NOC. I'm guessing *anything* is better than what appears to be the *nothing* that is in place now. Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? - -r -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFJ3uJ/q1pz9mNUZTMRAoRhAJ9m7GTv719RlXUrR6vuGigwpuhJSwCg+sc5 KLrSxYR/cRu1IJOjjM4Go0c= =x059 -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
Ravi Pina wrote:
Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through? Remember the exploding Ford Pinto?
On Fri, 10 Apr 2009 01:51:16 EDT, Ravi Pina said:
Also not to get sensationalist, but less expensive than a life that could be lost if an emergency call can't be put through?
The alarm that goes off saying the lid got opened is only 2 minutes before the big red alarm that says you just lost 5 OC-768s. So the link is *still* going to drop even as you're on the 911 call to try to explain to them where your manhole is, the cops *still* won't catch anybody (the perps may be gone before you hang up on the 911 call), and you're taking 2 minutes off a 10-hour outage. A lot of expense for not a lot of improvement.
----- Original Message ----- From: "Ravi Pina" <ravi@cow.org> To: "Charles Wyble" <charles@thewybles.com> Cc: <nanog@nanog.org> Sent: Thursday, April 09, 2009 5:41 PM Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes!
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
I can comment on this, I live three blocks from the scene of the cut. The manholes themselves sit along railroad tracks and an overpass. At 2am, it's a very dark area, and there is very little traffic at that time. It would be an ideal area to perform this type of vandalism. On a side note, when I was passing the area this morning at around 10am PDT, there were two fiber-trailers working in two separate manholes. My company (AS4307) fell off the map from about 2am until roughly 10:42pm when one of our upstreams (AS20115) finally came back. Our primary (AS174) came back about 11:30pm. Of course during the majority of the outage, we also had no land-lines or cell-phones, so were effectively isolated. Bobby Glover Director of Information Services South Valley Internet
On a side note, when I was passing the area this morning at around 10am PDT, there were two fiber-trailers working in two separate manholes.
This is probably the result of having to splice in a new section of fiber, since it would probably have been difficult to splice the ends of the cut section back together.
On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote:
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
This was supposedly an inside job, and I even heard the cabinets were locked. How do you stop an employee with the key from opening a lock? (See #2.)
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
Possibly, and yes.
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Probably, and who knows? How much did this cost the telcos involved? Probably nearly nothing. How much would it cost them to do what you suggest in #2? Probably 1,000,000 times nearly nothing, _at_least_. Guess what the telcos involved will choose? Hell, you would too in their place. -- TTFN, patrick
Charles Wyble wrote:
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? Some people do lock their vaults/pits/manholes. But, to be honest, I'm not sure it helps a lot. How many passersby would stop someone appearing to be in a phone company/telco high-vis vest using bolt cutters - telling them the lock had seized?
(I can also think of quite a few options which don't require opening a lid, but here's not the place to discuss!)
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
Alarms mean power. Adding power to hundreds of km of a route to every pit/manhole would cost a lot - it's underground and often quite wet. Better to provide diverse route protection for the same cost - then you protect against accidental external aggression. Maybe you could do something neat with fibre and some of the active monitoring stuff to detect pit openning passively, but you'd want it to be pretty good and reliable. Lots of false alarms lead to NOCs not caring.
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry? Obscurity and that most people are blissfully unaware of manholes and other street furniture. Locking is certainly possible but I'm not convinced it adds a LOT (see above).
Accidental external aggression is far more likely. Backhoe fade and equipment failure is a bigger problem than vandalism. MMC
Its all risk and cost. You possibly couldn't have spent enough to stop this event. The outside plant wasn't at fault, highly motivated and informed individuals were. Pretty much a non issue, IMHO. Best, Martin On 4/9/09, Charles Wyble <charles@thewybles.com> wrote:
Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all.
There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results.
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
-- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants
On Apr 9, 2009, at 6:04 PM, Charles Wyble wrote:
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple. And, yes, you can get lockable manhole covers. They aren't cheap. McGuard make a popular one. (Yes, yes...why would I possibly know any of this.....I'm a fire marshal in a small town as a part time gig, so I have to deal with this kind of thing on a reasonably regular basis) Daryl
On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote:
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple.
Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
You forgot the clip board. Without the clip board, no one will believe it. J -----Original Message----- From: Andy Ringsmuth [mailto:andyring@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote:
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple.
Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy
I know it's fun to have these sort of discussions...... however, here in Toronto anyway all of the splicers, construction people and other contractors all know each other enough to be able to spot somebody thats not auposed to be there. The city inspectors are cruising all day looking for health and safety violations, traffic inspectors are looking for issues, and the"cop Maffia" is making sure you have a pay duty cop. Unless you were incredibly lucky, a rogue crew at work In a chamber would be caught very quickly. On 13-Apr-09, at 9:07 AM, "Jamie Bowden" <jamie@photon.com> wrote:
You forgot the clip board. Without the clip board, no one will believe it.
J
-----Original Message----- From: Andy Ringsmuth [mailto:andyring@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog@nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes!
On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote:
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
Your understanding is incorrect. I'm an average sized guy and I can pull a manhole cover with one hand on the right tool. It might take 2 hands if it hasn't been opened recently and has lots of pebbles and dirt jammed in around it. It's like everything else: if you know how to do it, and you have the right tool, it's simple.
Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task.
Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it.
-Andy
Charles Wyble wrote:
So allow me to think out loud for a minute....
1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation?
Cost, both in implementing (what are likely to be easily-circumvented) physical protection mechanisms and the cost of dealing with those when on-site doing installation and maintenance.
2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction?
An alarm did go off, the moment the fiber was cut. In the old days, the alarm was gas pressure reduction on the coax followed by loss of signal... now it is loss of the optical carrier. It turns out that the absolute minimum in false alarms is to ignore things bumping into the manhole or falling into the vault and to alarm immediately if the fiber is tampered with, which is exactly what happened. A little semi-automated OTDR and you could tell which manhole it is without driving down to the CO, too.
3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?
I see individuals raising manhole covers and going down to do maintenance on their own all the time. Glass is cheap enough that the right solution to this problem is route diversity. An alarm goes off when one path is cut, but you have another path that is still running. Now it takes twice as many people to do the cutting. And if you really care, you can back that path up with other technology like microwave radio. But it all comes down to cost. ADSL and POTS subscribers in Santa Cruz County are willing to pay AT&T money for service that doesn't have sufficient route diversity along Monterey Highway. As long as it is more profitable to run the network that way, that's how it will be run. And people who care, *can* back this up. My home ADSL was down but my 90 Mbps home microwave link was running fine, and my VoIP was unaffected as well. My bank couldn't process transactions (Frame Relay was down) but the gas station next door could (VSAT was up). A few years ago it was the other way around when Galaxy 4 went belly-up. Either one of those *could* pay a few extra dollars a month and have both... and if that becomes financially worthwhile, maybe they will. But they can't expect their race-to-the-cheapest telco or ISP to do it for them without specific contractual agreements to that effect, and frankly a 14-hour outage just isn't enough lost business to pay for it. (If it was, I'd have a lot easier time signing people up as customers of my south SF bay area/north monterey bay area wireless ISP) Matthew Kaufman
participants (23)
-
Aaron Glenn -
Andy Ringsmuth -
Charles Wyble -
Dan Armstrong -
Daryl G. Jurbala -
deleskie＠gmail.com -
Jamie Bowden -
jamie rishaw -
JC Dill -
Jim Popovitch -
Joel Jaeggli -
Martin Hannigan -
Matthew Kaufman -
Matthew Moyle-Croft -
Mike Lewinski -
Patrick W. Gilmore -
Paul Ferguson -
Randy Fischer -
Ravi Pina -
Robert Glover -
Rod Beck -
Shane Ronan -
Valdis.Kletnieks＠vt.edu