- NANOG - lists.nanog.org

Fibre Cut in North Carolina
by Myke Lyons 14 Apr '09

14 Apr '09

Does anyone have any more information on the cut AT&T cut in Chapel Hill? .myke

2 1

Re: Fiber cut in SF area
by Scott Doty 14 Apr '09

14 Apr '09

David Edwards wrote: > At 12:55 PM 4/9/2009, you wrote: >> >From the news coverage it appears to be in the general area of >> http://cow.org/r/?545c >> >> -r > > Interesting. The report I got from a vendor was that it is Above.net > with a fiber cut in Redwood City which is affecting a circuit of mine > between 200 Paul in SF and PAIX in Palo Alto, which is a ways from > south San Jose. http://www.kcbs.com/Phone-Outage-Likely-Caused-by-Vandals/4174734 ''Police say that at 1:20 a.m., four to five fiber optic cables located beneath a manhole were cut and severed on Monterey Highway, north of Blossom Hill Road.'' ''In San Carlos, vandals struck a second time along Old County Road at the edge of San Carlos and Redwood City'' I also heard on KCBS: The cuts were in 4 manholes in San Carlos, and they said it was "seven cables". (Not sure if that means the same 7 cables were cut 4 times, or what...) I also heard: there were 4 cables cut in the South SJ manhole. A lot of comms (incl. 911) are out for Santa Cruz County, as well as South Santa Clara Country, including Gilroy and Morgan Hill. Just now, from their web stream, they refer to this as "an act of sabotage". On interview was with an "info-worker" in Morgan Hill, and for her, this was "the end of the world". (Personally, I can think of a "MAE-Clueless" episode that was worse than this, but that was in the 90's...) Finally -- and I'm not a lawyer -- I want to note that killing 911 to a city can get you tried for murder in California, if someone dies as a result, if I understand the law correctly. Better days, -Scott

39 62

RE: Fiber cut in SF area
by Skywing 14 Apr '09

14 Apr '09

Apologies for continuing this thread, but -- I don't understand this preoccupation with "early warning" systems on access to said manhole. What's the point? There are two possibilities here: 1) Someone goes down there and breaks something. You *already* know when this happens, because of your normal link monitoring. 2) There's a false positive (i.e. nothing malicious is done). >From where I stand, these seem like ways to spend money in order to increase the reporting noise. Or am I missing something? Irregardless, it would be wise to focus on the *common* causes of outages. The things that happen and cause customers pain every day, due to more mundane occurrances like backhoes. Regardless of whether it's a hacksaw or a backhoe that takes out a cable, the customer is still down. Simple economics seem to dictate that the most attention should be devoted to the problems where you get the most "bang for your buck" - i.e. not movie theatre plot scenarios that happen once in many blue moons when there are so many other, far too common (and yet mundane) causes of outages. - S -----Original Message----- From: Peter Beckman <beckman(a)angryox.com> Sent: Monday, April 13, 2009 11:19 To: Dylan Ebner <dylan.ebner(a)crlmed.com> Cc: nanog(a)nanog.org <nanog(a)nanog.org> Subject: RE: Fiber cut in SF area On Mon, 13 Apr 2009, Dylan Ebner wrote: > It will be easier to get more divergence than secure all the manholes in > the country. I still think skipping the securing of manholes and access points in favor of active monitoring with offsite access is a better solution. You can't keep people out, especially since these manholes and tunnels are designed FOR human access. But a better job can be done of monitoring and knowing what is going on in the tunnels and access points from a remote location. Cheap: light sensor + cell phone = knowing exactly when and where the amount of light in the tunnel changes. Detects unauthorized intrusions. Make sure to detect all visible and IR spectrum, should someone very determined use night vision and IR lights to disable the sensor. Mid-Range: Webcam + cell phone = SEEING what is going on plus everything above. High-end: Webcam + cell phone + wifi or wimax backup both watching the entrance and the tunnels. James Bond: Lasers. Active monitoring of each site makes sure each one is online. Pros: * Knowing immediately that there is a change in environment in your tunnels. * Knowing who or at least THAT something is in there * Being able to proactively mitigate attempts * Availability of Arduino, SIM card adapters, and sophisticated sensor and camera equipment at low cost Cons: * Cell provider outage or spectrum blocker removes live notifications * False positives are problematic and can lower monitoring thresholds * Initial expense of deployment of monitoring systems Farmers use tiny embedded devices on their farms to monitor moisture, rain, etc. in multiple locations to customize irrigation and to help avoid loss of crops. These devices communicate with themselves, eventually getting back to a main listening post which relays the information to the farmer's computers. Tiny, embedded, networked devices that monitor the environment in the tunnels that run our fiber to help avoid loss of critical communications services seems to be a good idea. Cheap, disposable devices that can communicate with each other as well as back to some HQ is a way to at least know about problems of access before they happen. No keys to lose, no technology keeping people out and causing repair problems. Some other things that could detect access problems: * Pressure sensors (maybe an open manhole causes a detectable change in air pressure in the tunnel) * Temperature sensors (placed near access points, detects welding and thermite use) * Audio monitor (can help determine if an alert is just a rat squealing or people talking -- could even be automated to detect certain types of noises) * IR (heat) motion detection, as long as giant rats/rodents aren't a problem * Humidity sensors (sell the data to weatherbug!) One last thought inspired by the guy who posted about pouring quick-set concrete in to slow repair. Get some heavy-duty bags, about 10 feet long and large enough to fill the space in the tunnel. More heavily secure the fiber runs directly around the access space, then inflate two bags on either side of the access point. Easily deflated, these devices also have an electronic device which can notify HQ that they are being deflated or the pressure inside is changing (indicating pushing or manipulation). That way you only need to put these bags at access points, not throughout the whole tunnel. Kinda low-tech, but could be effective. No keys needed, could be inflated/deflated quickly, and you still get notification back to a monitoring point. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman(a)angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

1 0

RE: Fiber cut in SF area
by David Barak 14 Apr '09

14 Apr '09

--- On Mon, 4/13/09, chris.ranch(a)nokia.com <chris.ranch(a)nokia.com> wrote: >> From: Peter Beckman >> Subject: RE: Fiber cut in SF area > > Total cost...is about $3000 per mile for > equipment > I get the feeling you haven't deployed or operated large > networks. You never did say what the multiplier > was. How many miles or detection nodes there > were. Think millions. The number that popped > into my head when thinking of active detection measures for > the physical network is $billions. AT&T: 888,000 route miles(1). Verizon: 485,000 route miles(2). If we assume that 1/4 of AT&T and Verizon's route-miles are in the US(3), this would mean a capital expense of $666M and $364M respectively, not including any costs incurred for maintenance, monitoring, repair, false positive etc. In addition, as has been noted, this system wouldn't PREVENT a failure, it would just give you some warning that a failure may be coming, probably by a matter of minutes. In the words of Randy Bush, "I encourage my competitors to do this." David Barak Need Geek Rock? Try The Franchise: http://www.listentothefranchise.com 1) http://www.att.com/gen/press-room?pid=4800&cdvn=news&newsarticleid=26554 2) http://mediumbusiness.verizon.com/about/network.aspx 3) I believe this to be an underestimate.

5 5

Re: Fiber cut in SF area
by Peter Lothberg 14 Apr '09

14 Apr '09

There are three solutions to the problem; A: Put a armed soldier every 150ft on the fiber path. B: Make the infrstructure so redundant that cutting things just makes you tired, but nothing hapens. C: Do nothing. As the society becomes more and more dependent on the infrastructure for electronic communication, my suggestion to policy makers has been that it should be easier to imprison all the government officials of a contry than knocking out it's infrastrcture. -P

3 4

Re: Fiber cut in SF area
by Shane Ronan 14 Apr '09

14 Apr '09

But you are ignoring the cost of designing, procuring, installing, monitoring, maintaining such a solution for the THOUSANDS of man holes and hand holes in even a small fiber network. The reality is, the types of outages that these things would protect against (intentional damage to the physical fiber) just don't happen often enough to warrant the cost. These types of solutions don't protect against back hoes digging up the fiber, as even if they gave a few minutes of advanced notice, the average telco can't get someone to respond to a site in an hour let alone minutes. On Apr 13, 2009, at 9:05 PM, Peter Beckman wrote: > On Mon, 13 Apr 2009, Shane Ronan wrote: > >> This all implies that the majority of fiber is in "tunnels" that >> can be monitored. In my experience, almost none of it is in tunnels. >> >> In NYC, it's usually buried in conduits directly under the street, >> with no access, except through the man holes which are located >> about every 500 feet. >> >> In LA, a large amount of the fiber is direct bored under the >> streets, with access from hand holes and splice boxes located in >> the grassy areas between the street and the side walks. >> >> Along train tracks, the fiber is buried in conduits which are >> direct buried in the direct along side the train tracks, with hand >> holes every 1000 feet or so. >> >> In any of these scenarios, especially in the third, where the fiber >> might run through a rural area with no road access and no cellphone >> coverage. Simply walk through the woods to the train tracks, put >> open a hand hole and snip, snip, snip, fiber cut. > > I'm sure more malicious fiber cuts would result in heightened > security. > If you can put your hand in it, you could put a sensor in it. It > wouldn't > work everywhere, but it could work even in conduit or just simply > inside > access points. > > A device the size of your fist or smaller could do the monitoring, and > would fit in most access points I would guess. > > You can't protect it all, and obviously you can't put a camera at > every > access point (well, maybe you can). You can't stop a determined > person > from doing anything (like promote networked smart sensors for fiber > runs, > or setting a small explosion inside an access point). And maybe > environmental monitoring of these areas just won't do anything to > help. > But who knows. > > Beckman > --------------------------------------------------------------------------- > Peter Beckman > Internet Guy > beckman(a)angryox.com http://www.angryox.com/ > ---------------------------------------------------------------------------

1 0

RE: Fiber cut in SF area
by Scott Weeks 14 Apr '09

14 Apr '09

--- beckman(a)angryox.com wrote: >> I still think skipping the securing of manholes and access >> points in favor of active monitoring with offsite access is a >> better solution. > > The only thing missing from your plan was a cost analysis. Cost of each, > plus operational costs, * however many of each type. How much would that > be? So, let's see. I'm pulling numbers out of my butt here, but basing it on non-quantity-discounted hardware available off the shelf. ----------------------------------------- Manpower to design, build, maintain, train folks and monitor in the NOC. Costs of EMS, its maintenance. blah, blah, blah... scott

3 3

Re: Fiber cut in SF area
by Stefan Molnar 14 Apr '09

14 Apr '09

"But that would not be NEBS Complient" -PHB I have thought of air horns in my colo cage when a tech of mine messes up. ------Original Message------ From: Nathan Ward To: nanog list Subject: Re: Fiber cut in SF area Sent: Apr 13, 2009 4:55 PM On 14/04/2009, at 11:35 AM, David Barak wrote: > In addition, as has been noted, this system wouldn't PREVENT a > failure, it would just give you some warning that a failure may be > coming, probably by a matter of minutes. Some statistics about the effectiveness of car alarms and unmonitored house alarms would probably be useful here. Whack a $5 12v horn on it, and my bet is that it'd become a deterrent pretty quickly. -- Nathan Ward

1 0

RIM Mail Admin Contact
by David Prude 13 Apr '09

13 Apr '09

Hello, If there is anyone from RIM who would be willing to contact me off list I would be most appreciative. Thank you, -David Prude -- David Prude System Administrator Murphy & Durieu (212)618-0320

1 0

Verizon BGP Contact
by Ozar 13 Apr '09

13 Apr '09

Could someone from Verizon contact me off list? We are having some problems with a new turn up with 2 Gig Links, and tech support has not been much help over the last few days in trying to get this resolved. Thanks, Brian

1 0