Recently, Paul Vixie wrote...

...

P.S. what is the deal with the Santa Clara POP anyway? I've seen gas main explosions, circuits cut by a blowtorch (gee? maybe they're related), "missing" fiber, "lost" circuits, and other shenangans.

The story may be growing with the telling, but I was told that a gas main ruptured, with or without a small explosion (depends on who you ask), and the fire marshall came in, smelled natural fumes all over, and ordered that power be cut to the surrounding buildings, presumably to prevent explosions due to sparks and such. WilTel ultimately brought in a portable generator and put this wonderful new spark generator out in the parking lot, closer to the natural gas fumes than anything inside the building would have been.

*sigh* I'd kinda hoped the issue would die down, but since it won't, I guess I'll actually speak up. I was driving to work, running a bit late at around 11:10 am, and turned west on Walsh Avenue from San Tomas. I pulled into the left turn lane leading into the business park, and noticed the driveway was blocked with orange cones and a backhoe digging. A woman in a dark, Pennsylvania green lincoln towncar pulled up into the left turn lane behind me. When the left turn arrow turned green, I made a u-turn on Walsh, and entered the business park via a secondary driveway closer to San Tomas. I glanced in my rear-view mirror, and saw that the woman in the green car was stopped midway through the intersection, blocking oncoming traffic, waiting for the construction crew to move the cones to let her into the parking lot via that driveway; I guess she figured the earth rotated around her, so why shouldn't everyone else change to fit? So, as I pulled into the parking lot in front of work, I looked over at the entrance, and saw that the construction workers were attempting to be nice, and were moving the cones out of the way, and trying to back up the backhoe so she could go past, and stop blocking the traffic that was now getting very irate. :-) As the worker in the backhoe was pulling the bucket out of the hole, he must have hit the main, because there was a very loud "kawoosh" and a blast of white condesation headed skyward. The construction crew started running away from the hole, and I headed into work to alert my coworkers to the problem. One of my coworkers called PG&E while a second one called 911. PG&E immediately said they were dispatching a crew to the site, and also told us to begin shutting down any exterior equipment that was likely to cause sparks. By this time, the smell of natural gas was nearly overpowering outside, and was becoming more so indoors. We began shutting down all the air conditioners on the roof, and began prepping the site for evacuation. Within about 15 minutes, the fire dept. had arrived, as had PG&E, and began roping off the area, and had us evacuate the building. We used cellular phones to call back in, and changed the outgoing messages for the NOC and tech support ACD queues to alert customers that we had been evacuated, and could not currently come to the phones. :-) PG&E began working on isolating the section of pipe, but within 15 minutes of their arrival, a spark triggered a fireball that roasted the dump truck, the backhoe, the tree nearby, the signs, and pretty much everything else in the vicinity. The one good offshoot of the fireball was that the pipe was actually burning now, and sending flames ~75 feet in a nice vertical column; the fire dept. immediately pulled hoses into place, to keep the fire contained to the single jet, and the nice folks next door at Wiltel rolled out their 75kVa generator out the BACK of the building, which was approx 150 yards from the actual gas leak, and fired it up, since there wasn't much danger of an explosion now that there was a constant flame source burning off the gas as it emerged. Since we couldn't get to our cars, we walked to lunch, and answered pages from various customers who noticed that network connectivity was spotty or nonexistent to large areas, and explained the situation. We got back to work by about 2:30pm, the fire was still going, but PG&E was working on clamping off the pipe physically a few dozen yards upstream; once the pipe was crushed, and the fire out, it only took about an hour to get clearance from the incident commander to air out the building, get santa clara electric to restore power, reset the PDU, reset the UPS's, and by 3:30 full network connectivity through our network was restored. Hopefully this clarifies the situation from a firsthand witness, and keeps people from blaming Wiltel for sparks or any other such nonsense.

...

This time I'm glad we followed WorldCom's (WilTel, whatevertheirnameisthisweek) site recommendation and used DC power for our equipment. But this POP is like working out of a telecommunications war zone.

That's one of several good reasons why CIX has installed a router up in Digital's Palo Alto facility (I'd call it a NAP, since it has rack space for rent and GIGAswitches all over the place, but "NAP" seems to indicate congressional funding so I'll choose another word... "Interconnect/CoLo" (ICL)). In addition to having both AC and DC on UPS and a generator, Palo Alto has actual coffee houses for those late night adventures.

Matt Petach