At 22:01 -0600 3/21/98, Sean Donelan wrote:

Both good ideas. But be sure to learn and understand when, where and why the operational experiences given in these sources apply to building a NOC.

No argument, and no argument that the practices have to be current. But there are things to be learned from the past, as well. In about 1981 or so, I built the NOC for what was then the US Agriculture Department Washington Computer Center. As you point out, one must consider the specific requirements. In this case, it was with horrible fascination I realized that my client was the Federal agency with official responsibility for bull, and presumably their output, in the US government.

One difficulty in designing a NOC for an ISP is they operate across several different problem domains, telecommunications, data processing, lan, wan, etc. Be careful if you hire a consultant with expertise in one particular area. NEBS and the like have several good ideas, but they also have a built-in set of assumptions. If you, or your consultant, don't understand when the assumptions apply and don't apply, when you apply the solutions to a different problem domain, you can actually make things worse. For example, the National Electrical Code doesn't apply to communications equipment located inside a communication utility building. But the building inspector will get very grouchy if an ISP's electrical wiring doesn't meet NEC standards.

Yes. And your consultant must know local practices as well. If you are building a facility in Fairfax County, Virginia (a major technology suburb of Washington DC), avoid, in your plans or anything the building inspector may see, the term "computer room." "Communications room," "network room," etc., all prevent the problem: if they see "computer room," they will demand a mainframe-style central emergency power off control, which greatly increases electrical wiring cost. I haven't looked at this recently, but another Fairfax County practice was if you used halon, regardless if the room was also sprinklered, you had to do a live test with halon before the inspector would approve it. The refill of the halon had to be built into your budget...an extra $16,000 as I remember from about 1986. The specific isn't that important, but if you are going to have to deal with irrational requirements, you need to know that.

Also, there have been changes in 'standard' practices in the last 5 years. If your sources for designing data processing centers were written prior to 1990, I would look for a second source. The military has been updating its standards to reflect revised commercial practices, but the civilian sellers of the MILSTDs don't always show when an old practice has been superceded.

Wasn't thinking of MILSTDs as much as some of the DISA documents, and often relating to base practices. The DoD manual on grounding, bonding, and shielding is excellent. There has been some very good work on filtering alarms in radio and multiplex algorithms, for reducing the noise level of alarms triggered in response to a single component failure. I'd have to dig through my library, but IIRC this was done by GTE (Sylvania?) for European Command.

My basic problem with the original question is I don't really understand what kind of NOC the person is trying to build. Are we talking about how build a NOC staff, or a NOC building to monitor a network of 70,000 nodes?

Another big question, frequently ignored, is staff training and retention. Someone who can quickly diagnose complex problems tends not to want to answer phones. Some organizations come up with reasonably creative ways to give a growth path to NOC people, arrange rotations off the help desk, etc., to avoid losing some very valuable knowledge. Other organization simply assume their senior operations support people will leave every year or so.

Are we talking about building a NOC theatre so management/customers can view CNN on the projection TV wall when they visit, or a lights-out server/equipment room designed for limited human occupation? ...

Speaking of irrational requirements, your point about NOC theatre is well taken. I've dealt more recently with some organizations that outsourced the NOC function for state governments, and was told flatly "we are going to be visited by politicians who want to see where their money went. While the capacity planning projections are the really useful things, they simply won't believe the contractor isn't cheating them unless the place looks like something out of Star Trek." Of course, the bridge of the NCC-1701 series Enterprise(s) here show incredibly bad human factors design, but that is often ignored. Perhaps more realistic is the display over the medical bed. While one of the bar graphs has text too small to read on the TV screen, the Star Trek technical manual shows how it is labeled "Medical Insurance Remaining." Surely there is a NOC equivalent of that bar graph.