- Test - lists.nanog.org

Outside plant protection, fiber cuts, interwebz down oh noes!
by Charles Wyble 13 Apr '09

13 Apr '09

Seriously though I want to start some discussion around outside plant protection. This isn't the middle of the ocean or desert after all. There were multiple fiber cuts in a major metropolitan area, resulting in the loss of critical infrastructure necessary to many peoples daily lives (though twitter stayed up so it's all good). :) It would appear that this was a deliberate act by one or more individuals, who seemed to have a very good idea of where to strike which resulted in a low cost, low effort attack that yielded significant results. So allow me to think out loud for a minute.... 1) Why wasn't the fiber protected by some sort of hardened/locked conduit? Is this possible? Does it add extensive cost or hamper normal operation? 2) Why didn't an alarm go off that someone had entered the area? It was after business hours, presumably not in response to a trouble ticket, and as such a highly suspicious action. Does it make sense for these access portals to have some sort of alarm? I mean there is fiber running through and as such it could carry the signaling. Would this be a massive cost addition during construction? 3) From what I understand it's not trivial to raise a manhole cover. Most likely can't be done by one person. Can they be locked? Or were the carriers simply relying on obscurity/barrier to entry?

23 25

Visualizing and confirming network anomalies - Newsight
by Dave Choffnes 13 Apr '09

13 Apr '09

Hi folks, We've just released a new tool that allows you to see network problems from end-user machines, currently BitTorrent users. (Think of this as putting P2P traffic to good use.) The interface that we've provided is still very beta, but we'd appreciate any (constructive) feedback. A little more detail: we've deployed software called NEWS (network early warning system) that uses P2P users' natural traffic to detect sudden, unexpected drops in network performance -- this allows our 25k BitTorrent users to corroborate network problems with each other. Our tool, called Newsight, lets you see summaries of their reported network anomalies. You can browse a timeline of anomalies grouped by country, AS or prefix. You can also confirm known network problems, helping to create a public database of outages and their explanations. You can play with Newsight here: http://aqualab.cs.northwestern.edu/projects/news/newsight.html Cheers, Dave (on behalf of the Aqualab research group) -- David Choffnes PhD Candidate, Computer Science Northwestern University

1 0

Re: Outside plant protection, fiber cuts, interwebz down oh noes!
by joel.mercado＠verizon.net 13 Apr '09

13 Apr '09

I agree 100 percent.... The clipboard makes it official... ------Original Message------ From: Jamie Bowden To: Andy Ringsmuth Cc: nanog(a)nanog.org Subject: RE: Outside plant protection, fiber cuts, interwebz down oh noes! Sent: Apr 13, 2009 9:07 AM You forgot the clip board. Without the clip board, no one will believe it. J -----Original Message----- From: Andy Ringsmuth [mailto:andyring@inebraska.com] Sent: Friday, April 10, 2009 1:52 PM To: Daryl G. Jurbala Cc: nanog(a)nanog.org Subject: Re: Outside plant protection, fiber cuts, interwebz down oh noes! On Apr 10, 2009, at 12:37 PM, Daryl G. Jurbala wrote: >> >> 3) From what I understand it's not trivial to raise a manhole >> cover. Most likely can't be done by one person. Can they be locked? >> Or were the carriers simply relying on obscurity/barrier to entry? > > > Your understanding is incorrect. I'm an average sized guy and I can > pull a manhole cover with one hand on the right tool. It might take > 2 hands if it hasn't been opened recently and has lots of pebbles > and dirt jammed in around it. It's like everything else: if you > know how to do it, and you have the right tool, it's simple. Agreed. Manhole covers are very simple to remove. I don't even need any tools. I've removed countless manhole covers to retrieve balls, frisbees, etc., with nothing more than my bare hands. It's a pretty trivial task. Think about it. All anyone would need to do is pull up to the manhole, set a few orange cones around it, put on an orange vest and a hard hat, and crawl on in with your wire cutters and bolt cutter. Guaranteed NO ONE will even question it. -Andy Sent on the Now Network� from my Sprint® BlackBerry

1 0

Re: Fiber cut in SF area
by Roger Marquis 12 Apr '09

12 Apr '09

Jorge Amodio wrote: > s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. /DARPA/ARPA/ may be splitting hairs. According to http://www.livinginternet.com/i/ii_roberts.htm "DARPA head Charlie Hertzfeld promised IPTO Director Bob Taylor a million dollars to build a distributed communications network". And apologies WRT /BBM/BBN/. Guess it was really has been a while now (given the 4 and 5 figure checks to BBN I signed back in the day). Sean Donelan wrote: > On Sat, 11 Apr 2009, Roger Marquis wrote: >> The real problem is route redundancy. This is what the original contract >> from DARPA to BBM, to create the Internet, was about! "The net" was >> created to enable communications bttn point A and point B in this exact >> scenario. > > Uh, not exactly. There was diversity in this case, but there was also > N+1 breaks. Outside of a few counties in the Bay Area, the rest of the > country's telecommunication system was unaffected. So in that sense the > system worked as designed. > > Read the original DARPA papers, they were not about making sure grandma > could still make a phone call. Apparently even some network operators don't yet grasp the significance of this event. > Why didn't the "man in the street" pharmacy have its own backup plans? I assume they, as most of us, believed the government was taking care of the country's critical infrastructure. Interesting how well this illustrates the growing importance of the Internet vis-a-vis other communications channels. Roger Marquis

2 1

RE: Fiber cut in SF area
by Carlos Alcantar 12 Apr '09

12 Apr '09

I know as far as att/sbc/pacbell a lot of the time they run the ring within the same conduit to at least have hardware protection on the circuit I'm sure it's the same with other providers. -carlos -----Original Message----- From: Roy [mailto:r.engehausen@gmail.com] Sent: Saturday, April 11, 2009 6:02 PM To: nanog(a)nanog.org Subject: Re: Fiber cut in SF area Sean Donelan wrote: > ,,,, > Uh, not exactly. There was diversity in this case, but there was also > N+1 breaks. Outside of a few counties in the Bay Area, the rest of > the country's telecommunication system was unaffected. So in that > sense the system worked as designed. > .... About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.

1 0

RE: Fiber cut in SF area
by Roger Marquis 12 Apr '09

12 Apr '09

Jo? wrote: > I'm confussed, but please pardon the ignorance. > All the data centers we have are at minimum keys to access > data areas. Not that every area of fiber should have such, but > at least should they? Manhole covers "can" be keyed. For those of > you arguing that this is not enough, I would say at least it?s a start. That is an option, but it doesn't address the real problem. The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! "The net" was created to enable communications bttn point A and point B in this exact scenario. No one should be surprised that ATT would cut-corners on critical infrastructure. The good news is that this incident will likely result in increased Federal scrutiny if not regulation. We know how spectacularly energy and banking deregulation failed. Is that mistake being repeated with telecommunications? The bad news is that some of the $16M/yr ATT spends lobbying Congress (for things like fighting number portability and getting a free pass on illegal domestic surveillance) will likely be redirected to ask for money to "fix" the problem they created. This assumes ATT is as badly managed, and the US FCC and DHS are better managed, than has been the case for the last 8 years. Time will tell. For a good "man in the street" perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: http://www.jivamedia.com/askdrdawn/askdrdawn.php http://geekspeak.org/ Roger Marquis

6 8

BGP FlowSpec support on provider networks
by Fouant, Stefan 11 Apr '09

11 Apr '09

Hi folks, I am trying to compile data on which providers are currently supporting BGP Flowspec at their edge, if there are any at all. The few providers I've reached out to have indicated they do not support this and have no intention of supporting this any time in the near future. I'm also curious why something so useful as to have the ability to advertise flow specification information in NLRI and distribute filtering information is taking so long to gain a foothold in the industry... Sorry for the repost but wanted to make sure this got it's own thread. Thanks, Stefan Fouant: NeuStar, Inc. Principal Network Engineer 46000 Center Oak Plaza Sterling, VA 20166 [ T ] +1 571 434 5656 [ M ] +1 202 210 2075 [ E ] stefan.fouant(a)neustar.biz [ W ] www.neustar.biz

5 5

Re: [OT] Re: Fiber cut in SF area
by Brandon Butterworth 11 Apr '09

11 Apr '09

> You can cut the concrete, too, for that matter, with oxyacetylene, as long as > you wear certain protective gear. We have a few vault covers here that are > concrete covering the largest vaults we have. You need more than a manhole > hook to get one of those covers up. And when you think you have it safely burried someone drives a tunnel boring machine through it - http://www.flickr.com/photos/23919135@N00/3426407496/ brandon

1 0

Re: Outside plant protection, fiber cuts, interwebz down oh noes!
by Robert Glover 11 Apr '09

11 Apr '09

*SNIP* located ten feet down a manhole on Monterey Highway, north of Blossom Hill Road in San Jose. Authorities also found two other locations where fiber optic cables were similarly cut -- near Hayes Avenue and Cottle Road in San Jose *SNIP* Just for clarification, these locations are one in the same. And as an update, the splicing operations are done at this location. (I live 3 blocks away The multitudes of crews that were at the site were gone when I drove by at 3:30PM PST today. The splice trailers were still there around 9:30AM PST working on cable. Bobby Glover Director of Information Services South Valley Internet (AS4307)

1 0

Re: BGP FlowSpec support on provider networks
by Richard A Steenbergen 11 Apr '09

11 Apr '09

> I am trying to compile data on which providers are currently > supporting BGP Flowspec at their edge, if there are any at all. The > few providers I've reached out to have indicated they do not support > this and have no intention of supporting this any time in the near > future. I'm also curious why something so useful as to have the > ability to advertise flow specification information in NLRI and > distribute filtering information is taking so long to gain a foothold > in the industry... <shamelessplug> nLayer has offered flowspec support to customers for many years now. </shamelessplug> It's really quite simple to use and support too, if you happen to have a largely Juniper based network that is. I'm not aware of any other router vendor who currently supports it, but the loss is entirely theirs. We do have a fair bit of Crisco in the network, with Juniper primarily in the core and peering/transit edge, so we use a route-server to feed the flowspec routes into the Juniper core. Customers set up an ebgp multihop session with it, and can announce flowspec routes (or standard blackhole via bgp communities) for anything in their register prefix-list. It's also quite handy for distributing internal use packet filters too. As for why something so (insanely) useful is slow to be adopted... There are a few reasons, but mostly: 1) A healthy dose of inter-vendor politics. 2) A silly religious belief that bgp shouldn't be used to carry firewall information, and that some other protocol should be invented instead. I think the counter-argument to that is the ability to do inter-provider filtering is precisely why it should be done via bgp. and the success of the current implementation proves how well it works. 3) Another large network who shall remain nameless once used a third party flowspec speaking piece of software which shall also remain nameless, and managed to blackhole their entire network for a noticeable amount of time. As with anything combining the words "network wide protocol" and "packet filter", a healthy amount of user discretion is advised. -- Richard A Steenbergen <ras(a)e-gerbil.net> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)

1 0