Hi folks! Several people asked for my summary of information I learned in the NANOG thread "building a new NOC" - I decided to assemble the summary and post it here, since enough people seemed to want it. I learned about several contractors who do NOC buildouts. To avoid seeming like a plugger, I won't give their names here. If you want the names of the contractor list i've assembled, please e-mail me privately. There were three major vendors of furniture that seemed to provide NOC-level stuff. APC-Wrightline, SMC, and Evans were the people that seemed to make items specifically built for use in an IT center or a NOC. I don't know where they compare on price; we're going through the process with each to determine who is the best bet. Evans by far seems to make the nicest furniture, though. I wasn't all that impressed with APC. Obviously a lot of the considerations for building a NOC come down to where you are building it at. Several people consider the same criteria for NOCs as they do for datacenters: Things like earthquake protection, good power, redundant everything, HVAC, etc. Several people also mentioned that you want to be careful that the building you're building your NOC in isn't required to use union labor, since that ups the cost significantly. For our NOC, we considered the following criteria: * Breakrooms * Lighting * Sleeping areas * Televisions and projection screens * Consoles with "roaming" capability - leave any console and move to another * Access control in and out of the facility * Laptops: For those people who must leave the NOC and go to a customer immediately * Whiteboards and conference rooms, inc. conference phones * Good chairs, keyboards, and required breaks for NOC personnel I roughly use the criteria of 50-100 machines/clusters per on-duty engineer (we're a systems shop). There's at least one senior engineer on staff for each shift. This probably varies widely among NANOG NOC managers. I considered several projector manufacturers. I haven't gotten all the way down to the lower level of detail on the projector scene, but we are told by an AV guy we're retaining that we should buy something between 700 and 1400 ANSI lumens. The big, 3000-3400 lumen units will run about $15K, and will project about 100+ ft in bright sunlight - obviously not necessarily for our purposes. The less lumened units, when placed closer to your projection screens, will provide more sharpness and clarity. (Any comments from AV people?) DLP projectors are the way to go. We're ordering three projectors. I'm still looking for either software or hardware that will allow me to multiplex PC signals onto one projector, and looking for what NOCs use to control the machines the projector is on. If anyone has info on this stuff, please let me know. In my pricing sheet for CFO consumption, I included the common items as well as the expensive technology stuff. We decided to include things like blankets and pillows. Rather than buy cots (there are lots of lumpy NOC people, like myself) we chose to build bunks out of sheets of plywood, and just lay the standard bed mats you find at camp stores over them. NOCs usually have pretty good AC, so we included lots of blankets. Cold and tired engineers have a hard time working - we included a fridge, a good microwave, and nice coffee machines. I'm getting into minutiae[sic] here. Our layout was for seven operators and three senior engineers. The engineers are officed behind the NOC people, with a good view of the screens through windows in their area. There's a dedicated staircase just to this room, to avoid traffic. The room is very low-light - just enough to see where you're going. We haven't decided on lighting fixtures yet, even for the overheads, but I think we've all come to the conclusion that we're just going to install some soft track lighting and let the engineers decide what kind of task lights they want over their consoles. The consoles are arrayed in such a way that even though they curve well so that everyone can see the screen, there is some privacy in the two-monitor setup we have. I chose Sun's line of Sun Ray workstations. These are really cool machines, folks. It's basically a dumb X terminal with a smart card reader - you take your smart card out, move to another machine, and your desktop is right there with no loss of anything. You can show another engineer something just by running your card over to his workstation. This was critical - in a shifted environment, you want everyone to have their own space. NOC people are picky. The workstations provided a good mix of that stuff. Fairly cheap, too. Everyone gets their own locker, including the senior engineers, to store personal items. We have a rule that nothing personal comes into the NOC except for books and cell phones - this keeps our engineers working. There is some music service available from an MP3 library, but there isn't any audio CD stuff on the workstations. The Sun Ray stuff really only seems good for terminal serving, and it requires its own segment. We run all our applications via tunneled SSH to separate PC or SPARC servers running Linux or Solaris. This helps us so that nobody can screw up the terminal server by installing software or changing parameters - everyone pretty much gets redirected to an appropriate machine upon login where they do their sandbox stuff. One thing I did like about the Sun Ray setup is that it does voice over IP. When you plug in your smartcard, your phone and voicemail also gets redirected to that location. It remains to be seen whether we'll use this solution, but I thought it was appropriate for NANOG consumption. I didn't really skimp on ergonomics. Since there is a high turnover in NOC personnel, we included every creature comfort we reasonably could. This includes Playstation 2 and Dreamcast, DVD, and large-screen TV with DSS for the guys. They're required to take 20 minute breaks every two hours, not including any Nazi bathroom permission or anything like that. This keeps them loose and relaxed. That rule isn't violated, even if everything's falling down around our ears. Personally, I think the biggest investment we make in the NOC is in people and creature comforts. The second biggest investment is in standard operating procedures - don't make people think if they don't have to. Make sure you invest on training your folks right when they come in. This is probably old hat to lots of you, and it's old hat to me, but it doesn't hurt to mention it. The hardest thing in a NOC is to make sure that things are kept quiet so that everyone can work, and make sure that the business types don't bring people in for tours without approval and/or scheduling. You don't want people walking through your NOC when there's a problem, whether it's obvious or not. I'm going to be building a web page to contain all of this information, as well as a mailing list, over the next couple of weeks. I'll post when available. Thanks very much to Brennan Murphy @NAI, Derrick Daugherty, Matt McGuirl, the folks over at NOCbuilder, and herb@tomobiki.urusei.net for excellent commentary and help. Thanks also to everyone else who contributed privately - there were so many responses. Tim