- NANOG - lists.nanog.org

NYC update, etc
by Adam Rothschild 13 Sep '01

13 Sep '01

Please excuse the operational nature of this posting, but there are a few news items I'd like to share. This message is void of political arguments and wild, unwarranted speculation, and as such, some of you might find it uninteresting. - Genuity has opened a SEV 0 ("multiple sites down"; see NOC tour video for explanation of varying severity levels) ticket for NYC. For more info: finger ticket-1222256(a)tickets.bbnplanet.com - 111 8th Avenue and 60 Hudson Street are operating normally on utility power. Both of these locations are mostly unmanned, and obtaining physical access may be tough. - 25 Broadway, NYC (Telehouse) is without utility power, and has been running off generator since around 4pm yesterday. They expect their current fuel supply to last 20+ hours. Additional fuel has been secured, though transporting it into the city might be difficult. There are techs currently on site. - Unconfirmed rumors of 34 Exchange Place in Jersey City, NJ (home to numerous important financial co's and Exodus) being evacuated. - Reports indicate that there were power/HVAC issues at 32 Old Slip, NYC, now resolved. Multiple dialup providers are reporting that all their PRI's are down. - There have been talks of evacuating 300 Boulevard East in Weehawken, NJ (Level3, EXDS, etc), but I'm not sure if any action's been taken. - I took some digital photos on the Jersey City waterfront, in hopes of giving folks a better idea of what's going on. They're viewable at <http://www.latency.net/~asr/wtc/>. I'll try to update this directory tree periodically. If you're interested in mirroring it, drop me a line off-list. - Mitch Halmu <mitch(a)netside.net> and <measl(a)mfn.org> remain subscribed to this list, despite posts made that are not only in violation of this list's charter, but are outright sickening. FYI, -adam

5 4

www.nyc.gov back online
by Sean Donelan 13 Sep '01

13 Sep '01

The hostmaster at www.nyc.gov sent me a note it is up and operating again.

1 0

Re: Verizon damage and service disruptions
by Frank Coluccio 13 Sep '01

13 Sep '01

re: the NYSE not having tridundancy, and the Broad Street CO Power uncertainties: Several things. The Pearl Street Central Office [which sits adjacent to the Brooklyn Bridge on the East Side of City Hall] is capable and does provide back up to both the Broad Street and West Street Central Offices on a special assembly basis. This, in addition to supporting its own subscribers, many of whom are residential and small business with routes running into the Chinatown area. Some firms near the NY Stock Exchange [I know of two on Broad Street] had elected long ago to have tri-dundancy built into their local access networks, by adding Pearl Street to the mix. So, the NYSE has to tread lightly when they look to why they don't have the same. All they had to do is place an order. This does raise an interesting question, though, concerning why their other local providers (of which, I believe, there are at least four) can't pick up some of the slack. As for the Broad Street Central Office having some power problems, I would remind all here that the Seaport Substation [which is just south of the Pearl Street CO] is also having its difficulties at this time. I believe that this substation failed yesterday under the load when 7WTC collapsed. You might recall that the Seaport substation was the cause of another of the Wall Street's major disasters back in the early 1990s. A substantial part of the financial district down Water Street and Broad Street are fed directly from this substation. Major re-grooming of feeders in this area was done back then when the failure occurred, in order to shore up switching to alternate substations in the event that it failed again. But this situation is different, because now it is the Seaport station itself that the West Side grid would depend on, and neither is working at this time. Most of what is working at this time in the area is actually on generator. It's going to take a lot of doing to support trading when the Stock Exchanges are re-opened on Friday, or Monday at the latest. Another thing that I remain wary of is the integrity of many of the underground duct bank systems supporting newly- run fiber routes in the area. Some of these sit atop older ducts and are only a foot or two beneath street asphalt in many places, despite standards to the contrary, and they run up and down Church Street, which becomes Tinity Place. These will not be tested fully until the downtown trading floors and data centers are reopened in a couple of days. Some rings in the area are now showing failed links on remote surveillance systems in offsite data centers, but it is not known whether this is due to carriers taking their nodes down temporarily, or due to underground problems, or due to anomalies at the customers own premises that are now almost entirely unpersonneled below 14th Street and working on generator. -FAC > > First, 1 Verizon employee and 3 Genuity employees are still missing, my > prayers, and I'm sure everyone else's prayers go out to their families. > No one wants to distract or impair the continuing rescue efforts in the area. > > > As Verizon continues its assessment of the damage, it is starting to > look worse than original reports. Verizon hasn't filed an outage > report with the FCC, or the FCC has delayed its release. So this is > based on Verizon's news conference and news reports. The reports > aren't very clear in all the details. > > The walls of the 140 West ST CO adjacent to the WTO Building 7 were breached > when Building 7 collapsed, and the building has flooded and the equipment > is covered with debris. 140 West St has no power. Verizon needs to pump > out the building because it is 5 stories below ground. > > Also mentioned is the Broad Street CO, which is experiencing intermittent > power problems. The Broad Street CO supplies 80% of the New York Stock > Exchange lines, and the West Steet CO serves as the alternate CO supplying > 20% of the NYSE lines. > > The CO's supply 200,000 voice lines and 3 million separate private lines. > Although the NYSE had backup facilities, it looks like the disaster in > NYC is disrupting service from both CO's. The telecommunication directors > of the exchanges have been meeting during the day. > > According to news reports, Verizon left it unclear why a backup facility > somewhere else wouldn't be an adequate substitute for Wall Street's > financial community and stock trading. According to one news report, > NYSE Chairman Grasso acknowledged that work is needed to come up with > a better telecommunications plan in the future. > > > >

2 1

Stranded US citizens in Germany - offer of assistence
by Dave Morton 13 Sep '01

13 Sep '01

Folks, Apologies for this rather belated offer. Simply didn't occur to me previously. This morning I saw a TV report about the very large number of US travellers stranded here in Europe and elsewhere around the world. I, for one, can offer accommodation and any other other help needed in particular here in the Munich or surrounding areas. I live half way between Munich city and the airport. I know a large number of US bound flights had to return here. The general accommodation situation may not currently be bad - I do not know. As many of you may know, the October festival is due to start Sat. week although there are talks in progress to cancel it entirely given the current situation. In any case, there may or may not be an accommodation shortage depending on a number of factors. Given that transatlantic flights show no sign of being allowed in the following days I can offer those of you have either relatives, friends or colleagues here or in the area that they contact me immediately. I can comfortably house up to six people. I will also gladly pick folks and baggage up as needed. Home phone and cell numbers are noted below. Please feel free to pass this information along to anyone you know is stranded and may need assistance. Best r. Dave -- Dave Morton Am Heckenacker 8 85652 Pliening, Munich Germany Tel: +49 8121 976 651 Fax: +49 8121 976 652 Mobile: +49 178 445 6208 E-mail: dave(a)manbitesdog.net

1 0

Re: Rumor Control, Indeed (was: Re: Rumor Control: Re: 60 hudson may collapse..
by Perry E. Metzger 13 Sep '01

13 Sep '01

Frank Coluccio <fcoluccio(a)dticonsulting.com> writes: > Like 140 West Street (adjacent to where 7 WTC once stood), 60H is a > brick shit house. 60 Hudson is not properly built for its role, actually. It is not a windowless telecom building -- it was originally a western union office building that slowly evolved into its current role. Most of the nearby telecom buildings are indeed windowless and designed to take punishment, but 60 is not. I recall on many occasions pointing it out to friends and noting you could see racked equipment through its numerous windows. It may (or may not) be perfectly true that it is safe, but its design has nothing to do with that safety -- only its position during this incident. Perry

1 0

Re: Re: Rumor Control, Indeed (was: Re: Rumor Control: Re: 60 hudson may collapse..
by Frank Coluccio 13 Sep '01

13 Sep '01

On the construction of 60 Hudson, Perry Piermont wrote: >>It is not a windowless telecom building -- it was originally a western union office building that slowly evolved into its current role.<< Good point, 60 Hudson is not windowless <except, in parts, where the windows have been plugged up ;> But from my recall of its details it was built to the same 'fortress-like' standards as 32 Sixth Avenue and other classical telco-grade buildings of the era. But true, its design precedes windowless COs by about thirty years ;) > > Frank Coluccio <fcoluccio(a)dticonsulting.com> writes: > > Like 140 West Street (adjacent to where 7 WTC once stood), 60H is a > > brick shit house. > > 60 Hudson is not properly built for its role, actually. It is not a > windowless telecom building -- it was originally a western union > office building that slowly evolved into its current role. Most of > the nearby telecom buildings are indeed windowless and designed to > take punishment, but 60 is not. I recall on many occasions pointing > it out to friends and noting you could see racked equipment through > its numerous windows. > > It may (or may not) be perfectly true that it is safe, but its design > has nothing to do with that safety -- only its position during this > incident. > > Perry >

1 0

Verizon damage and service disruptions
by Sean Donelan 13 Sep '01

13 Sep '01

First, 1 Verizon employee and 3 Genuity employees are still missing, my prayers, and I'm sure everyone else's prayers go out to their families. No one wants to distract or impair the continuing rescue efforts in the area. As Verizon continues its assessment of the damage, it is starting to look worse than original reports. Verizon hasn't filed an outage report with the FCC, or the FCC has delayed its release. So this is based on Verizon's news conference and news reports. The reports aren't very clear in all the details. The walls of the 140 West ST CO adjacent to the WTO Building 7 were breached when Building 7 collapsed, and the building has flooded and the equipment is covered with debris. 140 West St has no power. Verizon needs to pump out the building because it is 5 stories below ground. Also mentioned is the Broad Street CO, which is experiencing intermittent power problems. The Broad Street CO supplies 80% of the New York Stock Exchange lines, and the West Steet CO serves as the alternate CO supplying 20% of the NYSE lines. The CO's supply 200,000 voice lines and 3 million separate private lines. Although the NYSE had backup facilities, it looks like the disaster in NYC is disrupting service from both CO's. The telecommunication directors of the exchanges have been meeting during the day. According to news reports, Verizon left it unclear why a backup facility somewhere else wouldn't be an adequate substitute for Wall Street's financial community and stock trading. According to one news report, NYSE Chairman Grasso acknowledged that work is needed to come up with a better telecommunications plan in the future.

1 0

Rumor Control, Indeed (was: Re: Rumor Control: Re: 60 hudson may collapse..
by Frank Coluccio 13 Sep '01

13 Sep '01

Sean, All, agreed, we have to hold down the rumors. If someone posts a "I heard so and so," the least you should do is cite the source. re: The 60 Hudson location, Like 140 West Street (adjacent to where 7 WTC once stood), 60H is a brick shit house. It's blocks away from Ground Zero, it has not been hit by significant debris (read as: everything within a half-mile radius has had some form of ash or other office artifacts dropped on it), but 60 H it isn't going anywhere from a structural standpoint. Power and spare parts? That's another story. A number of carrier and other SP tenants at 60 Hudson, btw, have their own generation and vent systems installed in this building, and their individual preparedness for this sort of contingency is on a case by case basis. Smaller SPs who have rented racks or shelves are at the mercy of those who they have rented space and power from. As relates to the Empire State Building: NY Police sniffers, as well as ESB bomb dogs, have already disproved the Empire State affair as a hoax, and folks are now able to roam freely there, once again. We are going to see a lot more of this. My niece, who works in the Brooklyn District Attorney's office, was sent home today due to a bomb scare outside the State Supreme Court building. It, too, was a hoax. What I am still not clear on is the state of 140 West Street, which was once the HQ building for NY Telephone (now Veirzon), which remains standing and operational, from all indications. The Centrex that was once on the 12 floor in the South Tower was supported by the West Street CO. 140 West withstood the same impact that befell 7 WTC, one would think. It's on the corner of West Street and Veesey, separated from what was 7 WTC by a narrow alley. If anyone knows the operational status of 140 West, please post. --FAC > > On Wed, 12 September 2001, Rich Fulton wrote: > > we are hearing that 60 hudson has been evacuated and may collapse. > > > > can anyone give confirmation/details? > > No one is answering the phones at 60 Hudson, but there is a bomb > threat at the Empire State Building and police are evacuating the > surrounding area. > > >

2 1

140 West Street info
by Scott Drassinower 13 Sep '01

13 Sep '01

Verizon gave a press conference on Wednesday afternoon (1pm) with an update on the status of their network in Lower Manhattan, including details about 140 West Street. http://newscenter.verizon.com/proactive/newsroom/release.vtml?id=61409 should work to pull up the entire release, or call 877 519-4471, PIN 2831692. "The West Street facility contains four call-processing computers serving 200,000 access lines and the equivalent of 3 million data circuits in lower Manhattan. Babbio said that while some circuits that serve the Wall Street area go through the building, the company was attempting to by-pass the West Street facility for these circuits. Babbio said the company does not have an estimate on how long it will take to bring the center fully back online. "The building has three or four major holes -- that means water and dirt have gotten in to sensitive computer gear. Babbio said the company also is pumping water from the basement so that it can restore power to the building using generators. Additional switching equipment is on standby." The replay of the conference goes into more detail about the conditions in the building and what they are worried about in terms of cables and hardware being damaged or destroyed. They take care to point out that once the building comes back online, customers still may not get service until additional repair work underground or at customer premises gets completed. A few Verizon employees (installers or repair techs) working on higher floors had not been accounted for at the time of the conference, and there was a somewhat confusing mention of a couple of Genuity employees who may have been with them. The sales and other office folk (about 500) were on lower floors (9-12 or so) of the tower and nearly all have been accounted for. -- Scott M. Drassinower scottd(a)cloud9.net Cloud 9 Internet White Plains, NY +1 914 696-4000 / 800 356-5683 http://www.cloud9.net

1 0

RE: Lack of Security
by Roeland Meyer 13 Sep '01

13 Sep '01

|> From: Dave Stewart [mailto:dbs@ntrnet.net] |> Sent: Wednesday, September 12, 2001 7:08 PM |> This is just my feeling, but I honestly believe these |> measures are only |> giving the *appearance* of security, apparently to make the |> general public |> feel better. |> |> I do agree, though, with the comments on network security - |> so many, many |> are much more lax about the security of their networks than |> airports have |> been. As an anecdote, when I came to my current job last |> year, the network |> was wide open. Since, I've placed servers behind firewalls, |> and blocked |> things like NetBIOS (you wouldn't believe the cry that went up from |> customers when I did that - they *want* to use NetBIOS |> shares between |> business offices in various cities) This is exactly the sort of thing you are crying against. There is nothing wrong with PDC shares on the Internet, provided that you know what you are doing and are willing to do the work. Albeit, it works better behind a VPN and it will be much more accessible if you set up a PPTP session. Just to let you know that I am willing to back up my words nslookup 59204364-5b49-41ac-8ada-6dd78ab2ee86._msdcs.mhsc.com. Realize also that more thna half of those boxen are SuSE Linux, running Samba.

1 0