Re: Sabotage investigation of fiber cuts in Northwest
From: Valdis.Kletnieks@vt.edu Date: Sun, 02 Nov 2003 22:12:20 -0500
On Sun, 02 Nov 2003 21:37:30 EST, "Robert M. Enger" <enger@comcast.net> said:
You'd think after three previous disruptions, that Qwest would have enabled some form of redundancy.
Redundancy hell. How about a *PADLOCK*?
You mean that these places aren't even locked? Who has (had) the key? That'd be the first place I looked. Oh well... Back to lurk mode. --------------------------------------------------------------------- Gregory Hicks | Principal Systems Engineer Cadence Design Systems | Direct: 408.576.3609 555 River Oaks Pkwy M/S 6B1 | Fax: 408.894.3479 San Jose, CA 95134 | Internet: ghicks@cadence.com Never attribute to malice that which is adequately explained by ignorance or stupidity. Asking the wrong questions is the leading cause of wrong answers "The best we can hope for concerning the people at large is that they be properly armed." --Alexander Hamilton
On Sun, 02 Nov 2003 19:38:09 PST, Gregory Hicks said:
You mean that these places aren't even locked? Who has (had) the key?
The article says: "While the FBI scrambles to find a suspect, KIRO Team 7 Investigators went to check out security measures at the Qwest routing station. We walked straight through an unlocked gate, a wide-open door, and then mapped the interior of the building with our hidden camera. Nobody asked for ID or questioned our motives. KIRO Team 7 Investigators then headed to Qwest Corporate Headquarters in downtown Seattle. Ironically, it was lots tougher getting in there." Either the KIRO guys are lying through their teeth, or somebody dropped the ball BIG time. The bio of the guy who wrote it is here; http://www.kirotv.com/station/1868106/detail.html Either that's fibs too, or the guy is credible. Draw your own conclusions. :)
On Sun, 2 Nov 2003 Valdis.Kletnieks@vt.edu wrote:
"While the FBI scrambles to find a suspect, KIRO Team 7 Investigators went to check out security measures at the Qwest routing station. We walked straight through an unlocked gate, a wide-open door, and then mapped the interior of the building with our hidden camera. Nobody asked for ID or questioned our motives. KIRO Team 7 Investigators then headed to Qwest Corporate Headquarters in downtown Seattle. Ironically, it was lots tougher getting in there."
Either the KIRO guys are lying through their teeth, or somebody dropped the ball BIG time. The bio of the guy who wrote it is here;
I wonder has he ever tried walking into the transmission hut next to the tower of a major broadcast television or radio station? Usually when the revolution arrives, the first thing you take over is the television and radio outlets. The reality is there are a lot of weak points everywhere. Remember, part of the Internet design is the assumption that individual points of failure exist everywhere; the goal is to avoid single points of failure. There were (and are) alternate communication paths in the region, and several people pointed out during the last couple of attacks their data centers and Internet connections kept working even while the telephone and cell phones didn't. The quesiton isn't so much how someone cut a fiber strand, but why the failure of a single fiber strand had such an impact on the telephone service in the region.
On Sun, 2 Nov 2003, Sean Donelan wrote:
On Sun, 2 Nov 2003 Valdis.Kletnieks@vt.edu wrote:
Either the KIRO guys are lying through their teeth, or somebody dropped the ball BIG time. The bio of the guy who wrote it is here;
I wonder has he ever tried walking into the transmission hut next to the tower of a major broadcast television or radio station? Usually when the revolution arrives, the first thing you take over is the television and radio outlets.
"The revolution will NOT be televised The revolution will not be brought to you by Xerox In 4 parts without commercial interruptions" I'm fairly certain that the telco huts or CO's have to accomodate multiple groups having access, so I'd bet that a padlock probably is a tough sell :( Its very interesting that the 'critical infrastructure' has seemingly loose security on such vital parts.
lets not even begin to talk about American Towers Inc cell tower and relay facilities :) :( same combo on about 60 towers I know of security comes down to the "human condition" its a bitch to remember all those combo's, keep them updated, or install wiggle wire card readers, bad readers lost cards, etc. like current and voltage, we take the path of least resistence. On Mon, Nov 03, 2003 at 04:53:05AM +0000, Christopher L. Morrow wrote:
On Sun, 2 Nov 2003, Sean Donelan wrote:
On Sun, 2 Nov 2003 Valdis.Kletnieks@vt.edu wrote:
Either the KIRO guys are lying through their teeth, or somebody dropped the ball BIG time. The bio of the guy who wrote it is here;
I wonder has he ever tried walking into the transmission hut next to the tower of a major broadcast television or radio station? Usually when the revolution arrives, the first thing you take over is the television and radio outlets.
"The revolution will NOT be televised The revolution will not be brought to you by Xerox In 4 parts without commercial interruptions"
I'm fairly certain that the telco huts or CO's have to accomodate multiple groups having access, so I'd bet that a padlock probably is a tough sell :( Its very interesting that the 'critical infrastructure' has seemingly loose security on such vital parts.
On Sun, 2 Nov 2003, John Brown (CV) wrote:
lets not even begin to talk about American Towers Inc cell tower and relay facilities :) :(
same combo on about 60 towers I know of
just like padlocks at Fairfax County Public Schools... all are key #1345 (or where when I went through that system) I assume they do similar things in most similar situations in the telco world.
I'm fairly certain that the telco huts or CO's have to accomodate multiple groups having access, so I'd bet that a padlock probably is a tough sell :( Its very interesting that the 'critical infrastructure' has seemingly loose security on such vital parts.
Actually padlocks are quite common. When multiple organizations need entrance into a single gated area, its standard practice to have each of them put a padlock onto a string, separated by only one or two links of chain. When you want access you just unlock your padlock. Low tech but works pretty well considering the weak point in a chain-link fence is usually the chain-link, at least where a serious saboteur is concerned. We are collocated in about a hundred ROW huts and the security is usually aimed at preventing casual vandalism. -vb
The quesiton isn't so much how someone cut a fiber strand, but why the failure of a single fiber strand had such an impact on the telephone service in the region.
I'd be willing to bet it wasn't a single "strand". More likely the press or whoever got it wrong and it was an entire cable or maybe just a tube. -vb
Not having seen the entire cut, I would have to imagin the entire bundle was cut and the poor splicers had their hands full. -Henry "Vincent J. Bono" <vbono@vinny.org> wrote:
The quesiton isn't so much how someone cut a fiber strand, but why the failure of a single fiber strand had such an impact on the telephone service in the region.
I'd be willing to bet it wasn't a single "strand". More likely the press or whoever got it wrong and it was an entire cable or maybe just a tube. -vb
----- Original Message ----- From: Henry Linneweh To: Vincent J. Bono ; nanog@merit.edu Cc: Sean Donelan Sent: Monday, November 03, 2003 6:02 AM Subject: Re: Sabotage investigation of fiber cuts in Northwest
Not having seen the entire cut, I would have to imagin the entire bundle was cut and the poor splicers had their hands full.
From experience, I can say that its quite easy to sabatoge a fiber run. The perfect example - a few years ago when I was a network admin, the whole NOC where the bulk of our T1s were went out suddenly one morning. We discovered that less then a block away a fiber seeking backhoe dug right through the fibers - both the primary *and* secondary fibers - because Verizon burried them both in the same trench rather then run them separate routes. So, the supposed redundancy went right out the window.
The phone companies really aren't helping the situation one bit by doing stuff like this. -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.sosdg.org The AHBL - http://www.ahbl.org
What you describe is a folded ring and is indicative of either a temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems. Douglas S. Peeples Technology Assurance Labs -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Bruns Sent: Monday, November 03, 2003 7:39 AM To: Henry Linneweh; Vincent J. Bono; nanog@merit.edu Cc: Sean Donelan Subject: Re: Sabotage investigation of fiber cuts in Northwest ----- Original Message ----- From: Henry Linneweh To: Vincent J. Bono ; nanog@merit.edu Cc: Sean Donelan Sent: Monday, November 03, 2003 6:02 AM Subject: Re: Sabotage investigation of fiber cuts in Northwest
Not having seen the entire cut, I would have to imagin the entire bundle was cut and the poor splicers had their hands full.
From experience, I can say that its quite easy to sabatoge a fiber run. The perfect example - a few years ago when I was a network admin, the whole NOC where the bulk of our T1s were went out suddenly one morning. We discovered that less then a block away a fiber seeking backhoe dug right through the fibers - both the primary *and* secondary fibers - because Verizon burried them both in the same trench rather then run them separate routes. So, the supposed redundancy went right out the window.
The phone companies really aren't helping the situation one bit by doing stuff like this. -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.sosdg.org The AHBL - http://www.ahbl.org
On Mon, 3 Nov 2003 08:15:11 -0500 "Douglas S. Peeples" wrote:
What you describe is a folded ring and is indicative of either a = temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems. =20
Douglas S. Peeples Technology Assurance Labs
Are you sure? On or about August 5th, 2003 in Augusta, ME, while cross examining a Verizon employee expert witness in testimony for Maine PUC Docket 2002-243, I got him to say muliple times under oath that 90% of Verizon's fiber rings in the state of Maine are at least partially collapsed, what you call a folded ring... That is, only 10% of Verizon's fiber rings are fully redundant.[1] I don't have any proof of this but suspect that Maine is average and Verizon is average for a US RBOC. I suspect that 90% of US RBOC fiber rings are at least partially collapsed... regards, fletcher [1] I Am Not A Lawyer; the above is my best recollection of the testimony. Documents in regards to Docket 2002-243 are on line off of the Maine PUC web site (http://www.state.me.us/mpuc/homepage.htm) under the virtual case file (http://mpuc.informe.org.) Search by docket number 2002243. For some reason, the testimony is not there. I will ask a PUC lawyer why that is and see if it can be added.
On Mon, 3 Nov 2003 08:15:11 -0500 "Douglas S. Peeples" wrote:
What you describe is a folded ring and is indicative of either a = temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems. =20
Douglas S. Peeples Technology Assurance Labs
Are you sure? On or about August 5th, 2003 in Augusta, ME, while cross examining a Verizon employee expert witness in testimony for Maine PUC Docket 2002-243, I got him to say muliple times under oath that 90% of Verizon's fiber rings in the state of Maine are at least partially collapsed, what you call a folded ring... That is, only 10% of Verizon's fiber rings are fully redundant.[1]
I don't have any proof of this but suspect that Maine is average and Verizon is average for a US RBOC. I suspect that 90% of US RBOC fiber rings are at least partially collapsed...
Are you sure he wasn't talking about customer-buildings? I bet that 10% is to their COs and most of the 90% don't pay to have redundant paths to their building. If your business case is not sufficient for VZW or another company to build redundantly, they will build folded rings. Deepak Jain AiNET
On Mon, 3 Nov 2003 12:13:45 -0500 "Deepak Jain" wrote:
Are you sure he wasn't talking about customer-buildings? I bet that 10% is to their COs and most of the 90% don't pay to have redundant paths to their building. If your business case is not sufficient for VZW or another company to build redundantly, they will build folded rings.
No, specifically he was talking about IOF (Inter-Office Facility). Best place to see this is Page 56-57, lines 56.12 through 57.17. Testimony of Mr. Albert, being cross examined by Nick Whichester of Mid-Maine: MR. ALBERT: 12 .... The interrogatory that was asked was of your fiber 13 systems, how many of them are configured completely with the 14 full fiber route in the survivable fashion. Our response to 15 that interrogatory is, of all our IOF system, 10 percent of 16 them are completely configured that way. The other 90 have 17 got some portion of overlap. 18 MR. WINCHESTER: Collapsed network, collapsed 19 ring? 20 MR. ALBERT: Some portion of the routing is a 21 collapsed ring. That might only just be a section. It 22 might only be, you know, 10,000 feet, but some portion of 23 them are collapsed rings. When you look at our maintenance 24 spare quantities for IOF, they are nowhere near, you know, 25 what you would need to throw every single working OC-48 that 1 we've got out there in the network. We've got 192 OC-48s in 2 Maine. There was one interrogatory we answered with the 3 snapshot count. With the quantities that we have, 4 maintenance spares for IOF, we still don't have the ability 5 when there is a failure to reroute on a fiber basis The upshot as I understand it is that Verizon in Maine responds to fiber cuts by a very manual process of finding the engineer responsible for the region, having an engineer go into the office, consult a number of paper maps and then figure out an alternate route. They then dispatch technicians to the various COes to string jumpers to route around the outage. This process is described in the testimony. The document makes fascinating reading and is available off the Maine PUC web site under the "transcripts" section (http://www.state.me.us/mpuc/misctranscripts/2002-243%20080503.htm). As I said before, I don't think Maine is unique to Verizon nor Verizon unique to RBOCs. I think the fact that some of you find the above information news is fascinating in and of itself... regards, fletcher
Please tell me what phone companies you've been working with. As a rule, the ones I've experienced build whatever is the path of least resistance and often do stupid telco tricks like folded rings and single entries into buildings unless you stand over them with a bull-whip and insist that they do better. I'd love to know of a telco that does this right without having to stand over them. Owen --On Monday, November 3, 2003 8:15 AM -0500 "Douglas S. Peeples" <dpeeples@talabs.com> wrote:
What you describe is a folded ring and is indicative of either a temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems.
Douglas S. Peeples Technology Assurance Labs
-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Bruns Sent: Monday, November 03, 2003 7:39 AM To: Henry Linneweh; Vincent J. Bono; nanog@merit.edu Cc: Sean Donelan Subject: Re: Sabotage investigation of fiber cuts in Northwest
----- Original Message ----- From: Henry Linneweh To: Vincent J. Bono ; nanog@merit.edu Cc: Sean Donelan Sent: Monday, November 03, 2003 6:02 AM Subject: Re: Sabotage investigation of fiber cuts in Northwest
Not having seen the entire cut, I would have to imagine the entire bundle was cut and the poor splicers had their hands full.
From experience, I can say that its quite easy to sabatoge a fiber run. The perfect example - a few years ago when I was a network admin, the whole NOC where the bulk of our T1s were went out suddenly one morning. We discovered that less then a block away a fiber seeking backhoe dug right through the fibers - both the primary *and* secondary fibers - because Verizon burried them both in the same trench rather then run them separate routes. So, the supposed redundancy went right out the window.
The phone companies really aren't helping the situation one bit by doing stuff like this. -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.sosdg.org
The AHBL - http://www.ahbl.org
-- If it wasn't signed, it probably didn't come from me.
www.telcove.com They are running a DS3 'through' our building, enters one side and exits the other. They refused to run a spur but are adding a loop for us.
I'd love to know of a telco that does this right without having to stand over them.
Ray Burkholder ray@oneunified.net http://www.oneunified.net 704 576 5101 -- Scanned for viruses and dangerous content at http://www.oneunified.net and is believed to be clean.
I tend to agree, fiber rings when built out correctly have subtending rings to handle redundancy with extremely low delay times 50ms at worse -Henry "Douglas S. Peeples" <dpeeples@talabs.com> wrote: What you describe is a folded ring and is indicative of either a temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems. Douglas S. Peeples Technology Assurance Labs -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Brian Bruns Sent: Monday, November 03, 2003 7:39 AM To: Henry Linneweh; Vincent J. Bono; nanog@merit.edu Cc: Sean Donelan Subject: Re: Sabotage investigation of fiber cuts in Northwest ----- Original Message ----- From: Henry Linneweh To: Vincent J. Bono ; nanog@merit.edu Cc: Sean Donelan Sent: Monday, November 03, 2003 6:02 AM Subject: Re: Sabotage investigation of fiber cuts in Northwest
Not having seen the entire cut, I would have to imagin the entire bundle was cut and the poor splicers had their hands full.
From experience, I can say that its quite easy to sabatoge a fiber run. The perfect example - a few years ago when I was a network admin, the whole NOC where the bulk of our T1s were went out suddenly one morning. We discovered that less then a block away a fiber seeking backhoe dug right through the fibers - both the primary *and* secondary fibers - because Verizon burried them both in the same trench rather then run them separate routes. So, the supposed redundancy went right out the window.
The phone companies really aren't helping the situation one bit by doing stuff like this. -------------------------- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.sosdg.org The AHBL - http://www.ahbl.org
On Mon, Nov 03, 2003 at 08:15:11AM -0500, Douglas S. Peeples wrote:
What you describe is a folded ring and is indicative of either a temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems.
LATA and ILEC boundaries, along with fiscal measures tend to conspire against rural areas. The result is bad network design or a temporary solution that is going to be in place for decades. By example: I can't imagine Taos, New Mexico needing more than an OC-48 to service all of its voice and data needs over the next few years. The ILEC had a reason (and funding) to build fiber to the city -- but who will pickup the tab for the redundant build? The revenue from the services on a single OC-48 won't pay for it...
OC-48 doesn't have to do with the redundant build. physical fiber path does. thus, who will pay for the redundant fiber build. maybe that should come from the $700 Million the ILEC promised the state in new investment money, in exchange for relaxed regulation. rural or not, capitalism will hinder redundancy unless the shareholders or the insurance companies say otherwise. heck right in ABQ, NM you can kill several LD providers with just 2 fiber cuts. Take them off net for ABQ. On Mon, Nov 03, 2003 at 12:28:40PM -0700, John Osmon wrote:
On Mon, Nov 03, 2003 at 08:15:11AM -0500, Douglas S. Peeples wrote:
What you describe is a folded ring and is indicative of either a temporary solution or bad network design. As a rule, phone companies and capacity suppliers build very robust systems.
LATA and ILEC boundaries, along with fiscal measures tend to conspire against rural areas. The result is bad network design or a temporary solution that is going to be in place for decades.
By example: I can't imagine Taos, New Mexico needing more than an OC-48 to service all of its voice and data needs over the next few years. The ILEC had a reason (and funding) to build fiber to the city -- but who will pickup the tab for the redundant build? The revenue from the services on a single OC-48 won't pay for it...
On Mon, 3 Nov 2003 19:53:00 -0700 " John Brown (CV)" wrote:
rural or not, capitalism will hinder redundancy unless the shareholders or the insurance companies say otherwise.
Lack of capitalism killed telco redundancy. The telephone company had no competitive reason to build it and the regulators don't understand the issue enough to enforce it. Therefore, the telco management (and engineers?) coasted. We compete to some extent with the incumbent. Our shareholders care an awfully lot about redundancy. Senior management has promised that our networks are redundant in the vast majority of cases and any existing lack of redundancy will be removed in short order. We buy fiber from the telco, CLECs and the power company. The telco could get redundancy cheaply from bothering to buy fiber from other sources, but they have a real "not invented here" mentality which reduces the quality of their services. If they can't justify the capital costs to pull fiber, they don't have fiber... at least in Maine. I believe in the long run, our shareholders will see a better return on investment than telco shareholders caused by issues like this. If so, capitialism works in this case. regards, fletcher
On Mon, 3 Nov 2003, John Brown (CV) wrote: rural or not, capitalism will hinder redundancy unless the shareholders or the insurance companies say otherwise. YM, capitalism will foster redundancy? It does from where I sit.. matto --mghali@snark.net------------------------------------------<darwin>< Flowers on the razor wire/I know you're here/We are few/And far between/I was thinking about her skin/Love is a many splintered thing/Don't be afraid now/Just walk on in. #include <disclaim.h>
You'd think after three previous disruptions, that Qwest would have enabled some form of redundancy.
Redundancy hell. How about a *PADLOCK*?
You mean that these places aren't even locked? Who has (had) the key? That'd be the first place I looked.
The most amazing things can be found on certain northern cross-country fiber routes in areas where cellphones don't work - they thought about everything putting hundred thousand dollar doors and locks to prevent those who are not supposed to get into the huts from getting there... Excellence to the nines. Of course, since no one wants to carry keys to those super secure entrances, the same time of cobination keyholders that S&D and some others use to attach cabinet keys to the back of the cabinets themselves had been placed right by those super secure doors. Needless to say, it did not take long for every combination locked to be popped, keys taken out and super-secure doors opened. Alex
Based on my knowledge of fiber routes in Western Fairfax and Loudoun County and also from my NASA / US Navy days, there is a whole lot of security through obscurity in the physical infrastructure. On Monday, November 3, 2003, at 08:43 AM, Alex Yuriev wrote:
You'd think after three previous disruptions, that Qwest would have enabled some form of redundancy.
Redundancy hell. How about a *PADLOCK*?
You mean that these places aren't even locked? Who has (had) the key? That'd be the first place I looked.
The most amazing things can be found on certain northern cross-country fiber routes in areas where cellphones don't work - they thought about everything putting hundred thousand dollar doors and locks to prevent those who are not supposed to get into the huts from getting there... Excellence to the nines. Of course, since no one wants to carry keys to those super secure entrances, the same time of cobination keyholders that S&D and some others use to attach cabinet keys to the back of the cabinets themselves had been placed right by those super secure doors. Needless to say, it did not take long for every combination locked to be popped, keys taken out and super-secure doors opened.
Alex
Regards Marshall Eubanks T.M. Eubanks e-mail : marshall.eubanks@telesuite.com http://www.telesuite.com
participants (17)
-
Alex Yuriev -
Brian Bruns -
Christopher L. Morrow -
Deepak Jain -
Douglas S. Peeples -
fkittred＠gwi.net -
Gregory Hicks -
Henry Linneweh -
John Brown (CV) -
John Osmon -
just me -
Marshall Eubanks -
Owen DeLong -
Ray Burkholder -
Sean Donelan -
Valdis.Kletnieks＠vt.edu -
Vincent J. Bono