On 5 Dec 2000, Sean Donelan wrote:

Key features for a NOC: 1) Good chairs 2) Quiet 3) Adequate ice maker 4) Lots of bookshelves, file cabinets, and personal storage 5) Lots of phone lines (including conference and analog) 6) Some direct phone(s) (not through PBX, i.e. Red Phones) 7) Multiple PCs/Workstations per operator 8) Private tunes (cd player & headset) 9) CNN and The Weather Channel (really ESPN) 10) Drapes across the glass window

To which I would add from my own years in a NOC: - Low light (especially halogens -- flourescents are a killer) - Ready access to sodas, company-paid pref. (coffee is nice, but soda is necessary.) - Wide aisles, so people can -run- in an emergency - Binder containing phone numbers for telco contacts, management up to and including CEO, etc. - Kooshes or similar toys for slow grave shifts.

5) Lots of phone lines (including conference and analog)

that can call in and out and internationally randy

Sean's starting post to this thread really wasn't a question, nor is the content to follow consistent with the "NA" part of the acronym (though it well suits the "NOG" part) but considering the general s/n ratio here I figure my small bit of static with at least some useful data won't cause anyone's mail buffers to detonate. The general NOC discussion got me to thinking about some of the "details" of NOC design that I've kicked around before. I'll throw a few points in here that are often overlooked due to "practical" (read: no planning, or no budget) reasons. I'll preface this with the comment that we're talking about a NOC, and not simply a call center. Call centers are where huge numbers of people are herded into a queue to have their questions and minor problems addressed by a mass of often underpaid, under-trained "flaks." A NOC is significantly different, where one expects a much higher level of expertise of staff, a much higher average professionalism quotient (more on this in a minute), and where the criteria for success are not things like "hold time" and "abandon rate" but are "ETR reduction" and "network uptime." Regardless of the particular tasks (repair? observation? construction?) that a NOC staff must perform, there is an element of inter-personal communication that must occur. Previous notes on this thread have talked about how wireless phones (with headsets, I would assume) are a good thing, since often people move from cube to cube to get questions answered. To solve problems, very often there are several people working on the same issue from different directions, so quick and effective communications between staff members is required. I am convinced that electronic communications are not the ideal method for this quick communication. Most cubes/office spaces are not built for effective personal communications. They're built for personal privacy and adequate workspace. NOC spaces should not (IMHO) generally _not_ follow these same reasons for design, despite what NOC staff usually wish. The concept here is to have easy verbal or visual communications between members of the staff, and to have the manager able to view and understand what everyone is doing. This is often directly contradictory to what NOC staff want, which often resembles a maze of caves from which grunts and empty candy wrappers are thrown. The Suggestion: Think of NASA's Houston Mission Control Center (MCC). The room is primarily a large, open space with high ceilings. By standing up, any one member of the staff can see all other members of the staff. The people in the back of the room are on slightly raised platforms (1 foot?) so that they can see more of what the "front" row people are doing. (Anyone been to the MCC lately? Is this still the case? I know they re-modeled a few years ago.) The people in the rear (again, I assume this to be the case) are higher in the chain of authority, and can survey what is going on at all stations to get a clear idea of what is happening, and anyone in the room can engage in conversation easily with the manager without having to pick up a phone. With this in mind, the expectation of having a professional staff arises. Talking loudly on the phone with one's significant other is probably going to be noticed. Shouting The NASA control room (scaled down, of course) is an idealized version of what I think works best in situations that are fast-moving and complex. For 99.9% of the time, the NOC will be simply comfortable, but in that moment where you've lost all four cross-country strands and for some reason the out-of-band modems aren't picking up and your IGP is melting down... you'll appreciate the ability to have everyone in the same room able to get work done quickly. It might even be worth the few million you spend on the room. ;) I've often thought about a "pit" style design, where the managers sit in a depression in the middle of the room, and on all sides there are raised workstations facing towards the middle. The "big board" screens would have to be on hanging tubes directly above the manager area. I like that method since nobody then has to turn to talk to anyone else, but I dislike it because of the complexity of the design. Anyone have any examples of a NOC in this style? More minor comments on environments (network, telephone, and machine choices aside): - Sound reduction panels on ceiling/walls - Personally, I prefer having tile/raised floor in the NOC, since it allows one to slide a wheeled chair easily between multiple stations, which often is required when working on larger problems or when doing training. This is a matter of preference, though, as there are some drawbacks (noise) - Incandescent lighting instead of florescent and spot lighting instead of "wash" lighting (easier on eyes; less glare) - All members of team should be visible to the manager - Airflow should not point down directly on staff - try sitting at a workstation with a 1mph, 10-degree F cooler breeze blowing on your head for 8-10 hours - Multiple phones per workstation - Headsets are mandatory, for ease of use as well as reduction in worker's comp claims - All "big board" items should be visible without staff having to turn their heads more than 30 degrees from their normal focus point - Workstations should be labelled with numbers so easy reference can be made to locations in the room - Work areas should have room for a keyboard and a 8.5"x11" binder to be completely on the surface (i.e.: about 20" deep) so that someone can work from the operations manual and type at the same time - While it looks impressive, I've found that often people don't use or look at monitors that are above eye level. It seems from anecdotal evidence that people are more willing to use multiple horizontal monitors than glance upwards. Besides, stacking monitors starts to build "walls" around workstations. Some other NOCs or NOC-like settings that I've liked: - Akamai's facility in Boston. Nice looking, lots of $$$. Haven't checked it out for functionality. Mistah Gilmore, do you have pictures of this? Couldn't find 'em on your website. - Many major TV station newsrooms or production rooms. Of course, it's a different medium, but the same concepts: short deadlines, lots of activity, a lot of cooperative effort. PS: To answer someone's question about cordless phones, I've never seen an occasion where a cordless phone has interfered with computer equipment. Having a data center with no cordless phones is certainly painful for your ops staff and your customers. Now, cell phones I would have to experiment with since they are normally more powerful... JT

Several folks have asked me how to build a NOC for their operations people. Instead I thought I would post a few pointers to some of the larger oeprations centers with public information.

AT&T Global Network Operations Center, Bedminster, NC (where are the people?) http://www.archrecord.com/CONTEDUC/ARTICLES/11_00_1.asp

Cheyenne Mountain Operations Center (NORAD), Colorado Springs, CO (designed by the navy, built by the army, used by the air force) https://www.cheyennemountain.af.mil/cmoc/cmocimages.html

Genuity Network Operations Center, Burlington, MA (cool video John :-) http://www.genuity.com/help/noc/tour.htm

UUNET Network Operations Center, Northern Virginia (aren't there any cities in Virginia?) http://www.us.uu.net/support/noc/

Verio Network Operations Center, Dallas, TX http://home.verio.com/company/technology/noc.cfm

Key features for a NOC: 1) Good chairs 2) Quiet 3) Adequate ice maker 4) Lots of bookshelves, file cabinets, and personel storage 5) Lots of phone lines (including conference and analog) 6) Some direct phone(s) (not through PBX, i.e. Red Phones) 7) Multiple PCs/Workstations per operator 8) Private tunes (cd player & headset) 9) CNN and The Weather Channel (really ESPN) 10) Drapes across the glass window

as someone (else) who's worked in, and helped build a NOC, I'll throw in $0.015

Key features for a NOC: 1) Good chairs

while good chairs are important, it's also quite important to design a useful work space. reasonable desk space for keyboard, mouse, notepad, coffee cup, redundant coffee cup, and a paper plate with 2 slices of pizza should be the minimum standard per person. and like your network, don't build it before you design it. :-) remember that the "nasa look" is cool, but NASA spent their money designing rockets, not useful/comfortable workstations for mission control. try to keep in mind that people actually have to work there the 99% of the time they aren't just blocking the view of the cool furniture.

2) Quiet

NOCs aren't quiet, make sure that all of the NOCheads in your NOC have stereo headsets (i.e. 2 ear pieces) for their phones. providing headphones for the computers (in place of desktop speakers) is a good idea if the computers are going to be generating sound. some day some genius will figure out a way to design a headset that can be used to listen to music/computer beepage *AND* switch to a telephone headset at the push of a button and lowering of a boom microphone.

3) Adequate ice maker

adequate == large, and at least as reliable as your most sickly router. bottled/filtered water is also preferred where tap-water taste is undesirable or unreliable.

4) Lots of bookshelves, file cabinets, and personel storage

and pens/notepads. every week, open the door and throw in a couple boxes of pens and some notepads. NOC geeks like to scribble illegible notes on notepads while fixing stuff. and they like to doodle while waiting on hold to talk to $big_evil_telco.

5) Lots of phone lines (including conference and analog) 6) Some direct phone(s) (not through PBX, i.e. Red Phones)

this has been discussed ad nauseum, but different formats are nice. (i.e. PBX, POTS, cell, perhaps gsm) OH!!! don't give the alternate numbers to anyone but NOC manglement and engineers.

7) Multiple PCs/Workstations per operator

multiple monitors are far easier to deal with than multiple computers, and flat-panel LCDs are the only way to go for screens people have to look at 8 (or more) hours straight. give your geeks their choice of platform/OS as much as possible. geeks are more productive in an environment they're comfortable with.

8) Private tunes (cd player & headset)

give the geeks what they want, a private streaming mp3 server. an old throaway peecee from your IT department + $peecee_nix does really well el freebo.

9) CNN and The Weather Channel (really ESPN)

there's nothing a NOC likes better on Sundays than football, except maybe free beer and pizza while watching football.

10) Drapes across the glass window

a soundproof window would be very nice. you want your customers to see the NOC, not hear the NOC making fun of salespeople. :-) 11) give the NOC their own network, that they can manage themselves. do not put them at the mercy of your other departments for *anything* to the greatest extent possible. NOCs have special needs because of the demands that most companies put on them, and they need a handful of people dedicated to their needs. 12) toys. legos, lincoln logs, k'nex, anything with blinking lights, assorted sci-fi action figures with kung-fu grip. 12.5) a toy-box where they can put the toys when the suits come to visit 13) a BIG whiteboard. I just always liked them, and they're great for impromptu mentoring. 14) a coffee maker. the type that brews into an insulated caraf/thermos is best. 15) coffee fit for human consumption. it may become traditional for engineers to buy coffee for their NOC. (uhm, HINT ;) 16) useful procedures. a room full of people moving in different directions is a mosh pit, not a NOC. 17) a door that locks. don't give salespeople the key. don't even mark the door. put the door at the end of a long, dark, cold hallway. 18) a location near physiological needs facilities. figure that one out on your own. -- Sam Thomas Geek Mercenary