Sean's starting post to this thread really wasn't a question, nor is the content to follow consistent with the "NA" part of the acronym (though it well suits the "NOG" part) but considering the general s/n ratio here I figure my small bit of static with at least some useful data won't cause anyone's mail buffers to detonate. The general NOC discussion got me to thinking about some of the "details" of NOC design that I've kicked around before. I'll throw a few points in here that are often overlooked due to "practical" (read: no planning, or no budget) reasons. I'll preface this with the comment that we're talking about a NOC, and not simply a call center. Call centers are where huge numbers of people are herded into a queue to have their questions and minor problems addressed by a mass of often underpaid, under-trained "flaks." A NOC is significantly different, where one expects a much higher level of expertise of staff, a much higher average professionalism quotient (more on this in a minute), and where the criteria for success are not things like "hold time" and "abandon rate" but are "ETR reduction" and "network uptime." Regardless of the particular tasks (repair? observation? construction?) that a NOC staff must perform, there is an element of inter-personal communication that must occur. Previous notes on this thread have talked about how wireless phones (with headsets, I would assume) are a good thing, since often people move from cube to cube to get questions answered. To solve problems, very often there are several people working on the same issue from different directions, so quick and effective communications between staff members is required. I am convinced that electronic communications are not the ideal method for this quick communication. Most cubes/office spaces are not built for effective personal communications. They're built for personal privacy and adequate workspace. NOC spaces should not (IMHO) generally _not_ follow these same reasons for design, despite what NOC staff usually wish. The concept here is to have easy verbal or visual communications between members of the staff, and to have the manager able to view and understand what everyone is doing. This is often directly contradictory to what NOC staff want, which often resembles a maze of caves from which grunts and empty candy wrappers are thrown. The Suggestion: Think of NASA's Houston Mission Control Center (MCC). The room is primarily a large, open space with high ceilings. By standing up, any one member of the staff can see all other members of the staff. The people in the back of the room are on slightly raised platforms (1 foot?) so that they can see more of what the "front" row people are doing. (Anyone been to the MCC lately? Is this still the case? I know they re-modeled a few years ago.) The people in the rear (again, I assume this to be the case) are higher in the chain of authority, and can survey what is going on at all stations to get a clear idea of what is happening, and anyone in the room can engage in conversation easily with the manager without having to pick up a phone. With this in mind, the expectation of having a professional staff arises. Talking loudly on the phone with one's significant other is probably going to be noticed. Shouting The NASA control room (scaled down, of course) is an idealized version of what I think works best in situations that are fast-moving and complex. For 99.9% of the time, the NOC will be simply comfortable, but in that moment where you've lost all four cross-country strands and for some reason the out-of-band modems aren't picking up and your IGP is melting down... you'll appreciate the ability to have everyone in the same room able to get work done quickly. It might even be worth the few million you spend on the room. ;) I've often thought about a "pit" style design, where the managers sit in a depression in the middle of the room, and on all sides there are raised workstations facing towards the middle. The "big board" screens would have to be on hanging tubes directly above the manager area. I like that method since nobody then has to turn to talk to anyone else, but I dislike it because of the complexity of the design. Anyone have any examples of a NOC in this style? More minor comments on environments (network, telephone, and machine choices aside): - Sound reduction panels on ceiling/walls - Personally, I prefer having tile/raised floor in the NOC, since it allows one to slide a wheeled chair easily between multiple stations, which often is required when working on larger problems or when doing training. This is a matter of preference, though, as there are some drawbacks (noise) - Incandescent lighting instead of florescent and spot lighting instead of "wash" lighting (easier on eyes; less glare) - All members of team should be visible to the manager - Airflow should not point down directly on staff - try sitting at a workstation with a 1mph, 10-degree F cooler breeze blowing on your head for 8-10 hours - Multiple phones per workstation - Headsets are mandatory, for ease of use as well as reduction in worker's comp claims - All "big board" items should be visible without staff having to turn their heads more than 30 degrees from their normal focus point - Workstations should be labelled with numbers so easy reference can be made to locations in the room - Work areas should have room for a keyboard and a 8.5"x11" binder to be completely on the surface (i.e.: about 20" deep) so that someone can work from the operations manual and type at the same time - While it looks impressive, I've found that often people don't use or look at monitors that are above eye level. It seems from anecdotal evidence that people are more willing to use multiple horizontal monitors than glance upwards. Besides, stacking monitors starts to build "walls" around workstations. Some other NOCs or NOC-like settings that I've liked: - Akamai's facility in Boston. Nice looking, lots of $$$. Haven't checked it out for functionality. Mistah Gilmore, do you have pictures of this? Couldn't find 'em on your website. - Many major TV station newsrooms or production rooms. Of course, it's a different medium, but the same concepts: short deadlines, lots of activity, a lot of cooperative effort. PS: To answer someone's question about cordless phones, I've never seen an occasion where a cordless phone has interfered with computer equipment. Having a data center with no cordless phones is certainly painful for your ops staff and your customers. Now, cell phones I would have to experiment with since they are normally more powerful... JT
Several folks have asked me how to build a NOC for their operations people. Instead I thought I would post a few pointers to some of the larger oeprations centers with public information.
AT&T Global Network Operations Center, Bedminster, NC (where are the people?) http://www.archrecord.com/CONTEDUC/ARTICLES/11_00_1.asp
Cheyenne Mountain Operations Center (NORAD), Colorado Springs, CO (designed by the navy, built by the army, used by the air force) https://www.cheyennemountain.af.mil/cmoc/cmocimages.html
Genuity Network Operations Center, Burlington, MA (cool video John :-) http://www.genuity.com/help/noc/tour.htm
UUNET Network Operations Center, Northern Virginia (aren't there any cities in Virginia?) http://www.us.uu.net/support/noc/
Verio Network Operations Center, Dallas, TX http://home.verio.com/company/technology/noc.cfm
Key features for a NOC: 1) Good chairs 2) Quiet 3) Adequate ice maker 4) Lots of bookshelves, file cabinets, and personel storage 5) Lots of phone lines (including conference and analog) 6) Some direct phone(s) (not through PBX, i.e. Red Phones) 7) Multiple PCs/Workstations per operator 8) Private tunes (cd player & headset) 9) CNN and The Weather Channel (really ESPN) 10) Drapes across the glass window