- NANOG - lists.nanog.org

Building a new NOC: Summary
by Timothy Brown 17 Jul '01

17 Jul '01

Hi folks! Several people asked for my summary of information I learned in the NANOG thread "building a new NOC" - I decided to assemble the summary and post it here, since enough people seemed to want it. I learned about several contractors who do NOC buildouts. To avoid seeming like a plugger, I won't give their names here. If you want the names of the contractor list i've assembled, please e-mail me privately. There were three major vendors of furniture that seemed to provide NOC-level stuff. APC-Wrightline, SMC, and Evans were the people that seemed to make items specifically built for use in an IT center or a NOC. I don't know where they compare on price; we're going through the process with each to determine who is the best bet. Evans by far seems to make the nicest furniture, though. I wasn't all that impressed with APC. Obviously a lot of the considerations for building a NOC come down to where you are building it at. Several people consider the same criteria for NOCs as they do for datacenters: Things like earthquake protection, good power, redundant everything, HVAC, etc. Several people also mentioned that you want to be careful that the building you're building your NOC in isn't required to use union labor, since that ups the cost significantly. For our NOC, we considered the following criteria: * Breakrooms * Lighting * Sleeping areas * Televisions and projection screens * Consoles with "roaming" capability - leave any console and move to another * Access control in and out of the facility * Laptops: For those people who must leave the NOC and go to a customer immediately * Whiteboards and conference rooms, inc. conference phones * Good chairs, keyboards, and required breaks for NOC personnel I roughly use the criteria of 50-100 machines/clusters per on-duty engineer (we're a systems shop). There's at least one senior engineer on staff for each shift. This probably varies widely among NANOG NOC managers. I considered several projector manufacturers. I haven't gotten all the way down to the lower level of detail on the projector scene, but we are told by an AV guy we're retaining that we should buy something between 700 and 1400 ANSI lumens. The big, 3000-3400 lumen units will run about $15K, and will project about 100+ ft in bright sunlight - obviously not necessarily for our purposes. The less lumened units, when placed closer to your projection screens, will provide more sharpness and clarity. (Any comments from AV people?) DLP projectors are the way to go. We're ordering three projectors. I'm still looking for either software or hardware that will allow me to multiplex PC signals onto one projector, and looking for what NOCs use to control the machines the projector is on. If anyone has info on this stuff, please let me know. In my pricing sheet for CFO consumption, I included the common items as well as the expensive technology stuff. We decided to include things like blankets and pillows. Rather than buy cots (there are lots of lumpy NOC people, like myself) we chose to build bunks out of sheets of plywood, and just lay the standard bed mats you find at camp stores over them. NOCs usually have pretty good AC, so we included lots of blankets. Cold and tired engineers have a hard time working - we included a fridge, a good microwave, and nice coffee machines. I'm getting into minutiae[sic] here. Our layout was for seven operators and three senior engineers. The engineers are officed behind the NOC people, with a good view of the screens through windows in their area. There's a dedicated staircase just to this room, to avoid traffic. The room is very low-light - just enough to see where you're going. We haven't decided on lighting fixtures yet, even for the overheads, but I think we've all come to the conclusion that we're just going to install some soft track lighting and let the engineers decide what kind of task lights they want over their consoles. The consoles are arrayed in such a way that even though they curve well so that everyone can see the screen, there is some privacy in the two-monitor setup we have. I chose Sun's line of Sun Ray workstations. These are really cool machines, folks. It's basically a dumb X terminal with a smart card reader - you take your smart card out, move to another machine, and your desktop is right there with no loss of anything. You can show another engineer something just by running your card over to his workstation. This was critical - in a shifted environment, you want everyone to have their own space. NOC people are picky. The workstations provided a good mix of that stuff. Fairly cheap, too. Everyone gets their own locker, including the senior engineers, to store personal items. We have a rule that nothing personal comes into the NOC except for books and cell phones - this keeps our engineers working. There is some music service available from an MP3 library, but there isn't any audio CD stuff on the workstations. The Sun Ray stuff really only seems good for terminal serving, and it requires its own segment. We run all our applications via tunneled SSH to separate PC or SPARC servers running Linux or Solaris. This helps us so that nobody can screw up the terminal server by installing software or changing parameters - everyone pretty much gets redirected to an appropriate machine upon login where they do their sandbox stuff. One thing I did like about the Sun Ray setup is that it does voice over IP. When you plug in your smartcard, your phone and voicemail also gets redirected to that location. It remains to be seen whether we'll use this solution, but I thought it was appropriate for NANOG consumption. I didn't really skimp on ergonomics. Since there is a high turnover in NOC personnel, we included every creature comfort we reasonably could. This includes Playstation 2 and Dreamcast, DVD, and large-screen TV with DSS for the guys. They're required to take 20 minute breaks every two hours, not including any Nazi bathroom permission or anything like that. This keeps them loose and relaxed. That rule isn't violated, even if everything's falling down around our ears. Personally, I think the biggest investment we make in the NOC is in people and creature comforts. The second biggest investment is in standard operating procedures - don't make people think if they don't have to. Make sure you invest on training your folks right when they come in. This is probably old hat to lots of you, and it's old hat to me, but it doesn't hurt to mention it. The hardest thing in a NOC is to make sure that things are kept quiet so that everyone can work, and make sure that the business types don't bring people in for tours without approval and/or scheduling. You don't want people walking through your NOC when there's a problem, whether it's obvious or not. I'm going to be building a web page to contain all of this information, as well as a mailing list, over the next couple of weeks. I'll post when available. Thanks very much to Brennan Murphy @NAI, Derrick Daugherty, Matt McGuirl, the folks over at NOCbuilder, and herb(a)tomobiki.urusei.net for excellent commentary and help. Thanks also to everyone else who contributed privately - there were so many responses. Tim

1 0

RE: When will 128M not be enough?
by Roeland Meyer 17 Jul '01

17 Jul '01

> From: Andy Walden [mailto:andy@tigerteam.net] > Sent: Sunday, July 15, 2001 3:02 PM > A unique situation such as the CW/PSI mess would > have called for a prefix filter tweak, which, albeit, not automatic, relatively > painless on a small scale (i.e, you don't have to change hundreds of > customer routers because they don't have the talent themselves). I disagree that the CW/PSI mess is unique. My analysis leads me to think that it might be a foreshadowing. Many of the mid-tier ISPs, and some of the larger ones, are over-extended on growth and short on revenue. They were caught by surpise, just as much as the rest of us, when the dot-com market tanked. That sucking sound you hear is the rest of us being dragged in. Those of us that have enough bank-cash to hold onto and sufficient revenue to resist the pull, will be successful. Two points, the DSL market (RBOCs are moving in) is soft and the wireless ISP market is soft (I hope Ricochet survives). The point is that one cannot assume that PSInet will be the last of those. The failure-mode is exactly the same as that for WebVan, excessive (debt funded) growth, excessive burn-rate, not enough revenue, and too long to ROI (not enough cash in the bank to last until profitability). Many ISPs are slicing the baloney so thin that you can see daylight through it. No one expected the market to collapse and impact revenues the way that it has. Couple this with a market that is used to getting everything for free and you have a major problem brewing. Monetizing such a customer-base is very much harder to do, than to say.

3 2

Viewpoints on how to handle errors
by Sean Donelan 16 Jul '01

16 Jul '01

Two viewpoints on handling a major outage, and dealing with the consequences of your mistake. Handling Even Your Big Glitches Gracefully http://www.washtech.com/news/atwork/11178-1.html Nasdaq: Outage Not Likely To Recur http://www.internetweek.com/story/INW20010711S0005

1 0

Re: When will 128M not be enough?
by rs＠seastrom.com 16 Jul '01

16 Jul '01

"Majdi S. Abbas" <msa(a)samurai.sfo.dead-dog.com> writes: > On Sun, Jul 15, 2001 at 09:41:05PM -0400, Christopher A. Woodfield wrote: > > True, I prefer $30 after rebate from Circuit City... :) > > Not for a 2650, you don't. > > cisco 'fixed' that particular problem. yeah, now you have to buy it from kingston, crucial, or transcend instead of buying true-ghetto off the shelf memory. only $131.00 from kingston for the KCS-D2650/128. oh the humanity! ---rob

1 0

RE: how to handle peering fights
by David Schwartz 15 Jul '01

15 Jul '01

> I must respectfully disagree: better in my opinion is > to receive partial routes from multiple providers > (specifically, the providers' own routes), and then > keep a default route to your least-used provider. If > you discover that some particular address isn't > reachable, direct the traffic elsewhere. The real > solution is to pay attention, and make sure that > you're aware of what your providers are doing: peering > loss between providers isn't nearly as bad as complete > business failure, although they can both be as deadly. You are fixing the example, rather than the problem. > My contention is that it's substantially more > effective (both in terms of cost and results) to use > human intervention to resolve this type of problem > rather than rely on an automatic system. There were > warnings before the whole C&W/PSI fiasco, and this > seems to be the rule rather than the exception. For the C&W/PSI fiasco, yes. For other types of connectivity losses, probably not. Some may only last for a few hours, by the time you respond, the problem is over. Without full routes from at least one provider, how can you tell what traffic is affected by the fiasco even if you know about it? You often can't just take customer routes and point default away from a link with a problem like that because so much traffic takes the default route that your other link(s) would be too heavily loaded. (Of course, if your providers are large, this is less of a problem since customer routes will cover more of your traffic). Wven having full routes won't help you when your provider sends you routes it can't actually handle traffic for. Sometimes, a sick router is too sick to withdraw routes, but it can still take an awfully long time for the BGP session(s) to it to break. If you have a lot of delay-sensitive traffic, taking full routes can also result in a performance improvement. Also, taking shorter routes generally means taking more reliable routes as there are fewer points for there to be problems that might not be reflected in the BGP routes (like a sick router, overloaded link, or misconfigured filter). I wasn't trying to start an argument about how to deal with peering fights, really. I was just trying to voice the other side of the 'no need to take full routes' fight. DS

1 0

Re: Improving Robustness of Distributed Services (Re: DDoS attacks)
by Marshall Eubanks 15 Jul '01

15 Jul '01

> > >On Thu, 12 Jul 2001, Dickson, Brian wrote: > >> - What is IRC, or for that matter net-news, at its heart? A transient, >> store-and-forward, one-to-many message system. >> >> In otherwords, multicast re-implemented on unicast, in some cases poorly and >> at great cost (news). > >AFAIK, there is no implemented multicasting protocol - routing and reliable >transport - that can scale to the number of groups, senders and receivers >visible in current IRC networks. Source-based trees would require the >order of 1E6 (S,G) pairs and core/RP-based routing would be an even easier >target for the SK's than IRC already is. > >Achieving instant group messaging on a global scale with convential multicast >is unlikely. Relevant academic research continues. > >J > > This sounds like a good application for SSM (which is, BTW, can be made much "harder" than ASM). Send many to one to a nearby server, which sends out one to many to interested parties. This way, participants would not need their own S,G. The crucial thing would be to make it fractal (hierarchical) in nature. I agree that multicasting will become a target for SK; in many ways I think the IRC attacks are a premonition of things to come. Regards Marshall Eubanks

1 0

RE: DDoS attacks
by Roeland Meyer 15 Jul '01

15 Jul '01

> From: Brad [mailto:brad@americanisp.net] > Sent: Thursday, July 12, 2001 9:18 AM > > On Thu, 12 Jul 2001, Roeland Meyer wrote: > > > > From: up(a)3.am [mailto:up@3.am] > > > Sent: Thursday, July 12, 2001 7:23 AM > > > > > I can't help but believe that if even 20% of them > > > were caught and had to spend just a little time (even > hours) with the > > > cops, and had their peecees confiscated, you'd not be seeing > > > nearly the problems we are now. > > > > This is the main point, a script-kiddie hunt, with > prosecution, is the ONLY > > real deterrent. Throw some of them in hotel greybar and > remove them from > > computing, for life, and we may see some of this turn around. > > I am just concerned about our current legal systems being > able to handle such cases efficently. Well.. Perhaps I > should not use 'legal systems' and 'efficently' in the same > sentence, but you get the idea ;) Think "Kaspureff" (AlterNIC). They went after, and nailed, him with gusto and efficiency. I think that the largest problem is "selective prosecution". A couple of years ago three MHSC servers were root-kitted via the BIND interface. It took 18x7 man-hours to scrub and bare-metal recover, without the suspect backups, those three servers. Even then, we couldn't jump the FBI's $60K damages hurdle. Yet, Kaspureff, with a lot less provable damage, got caught, grilled, and chilled. For those that don't know, this case is the best and most well-known example of [arguably] deliberate DNS cache poisoning on record. If you don't remember it then you need not comment. The point is that our legal systems can move with great alacrity, given sufficient motivation.

2 1

Improving Robustness of Distributed Services (Re: DDoS attacks)
by Dickson, Brian 15 Jul '01

15 Jul '01

> * Most of DoS attacks are against IRC networks. Hence, if we can get rid of those, the health of the Internet as a whole should improve. > * Experience gathered with this approach should be useful to developers and administrators of other distributed services and protocols. To paraphrase Lilu in _The Fifth Element_, "mooolteeeecaast". Multicast. It's a huge leap, which would require development, no doubt. However, consider the following: - What is IRC, or for that matter net-news, at its heart? A transient, store-and-forward, one-to-many message system. In otherwords, multicast re-implemented on unicast, in some cases poorly and at great cost (news). - What happens to IRC when you change this to multicast? IRC channels morph to multicast groups; RPs replace IRC servers; and most importantly, the infrasture "glue" no longer has a visible IP address, and becomes much less vulnerable to attack. Multicast has *intrisnic* RPF checking (that's how it works, in fact). Attackers cut themselves off first, before any propogation occurs. The protocol(s) deals with localized 'issues'. Anyone wishing to interrupt the IRC network would have to attack the entirety of the Internet, simultaneously. (This is *not* a challenge, really.) How difficult would it be to (a) implement, and (b) to migrate the users over to the new system? I can't speak to (a), never having operated a network with a network-administered IRC server (that I know of anyway ;-)). As to (b), if an initial gateway from unicast<->multicast (think MBGP<->DVMRP for the mbone) or many exist, should be easy. Show users that the new system is more resilient, and they'll use it. More importantly, have the operators and/or upstream networks promise to support the old system for a limited time only, after successful deployment of the new system. For network admins, the fact that it's more efficent for backbone use (a bigger factor near the edge, where bottom-tier ISPs with not many resources, and lots of IRC users - fill the modem ports with bandwidth, at no cost to your upstream link!) should be a selling point in and of itself. A significant number of top-tier networks already have the necessary multicast support and peering. Not much more needs to happen, beyond client and server software, and administrative tools. Multicast. A solution looking for a problem; a problem found, at last. :-) Brian Dickson

4 3

Re: When will 128M not be enough?
by Marshall Eubanks 15 Jul '01

15 Jul '01

> > >up(a)3.am schrieb: > >> >> It looks like some recent aggregation has been helping to slow down the >> growth. Yes, us http://www.multicasttech.com/status/index.html (at ~ 102k), Telstra http://www.telstra.net/ops/bgp/index.html (at ~ 105k) and KPNQwest http://www.mcvax.org/~jhma/routing/bgp-hist.html (at ~ 103k) have had flat numbers of suffixes since about mid-June. IMHO it's way too soon to tell if this is really a new trend. Regards Marshall Eubanks >If you look at some figures (e.g. >http://www.employees.org/~tbates/cidr-report.html) it's even pretty stable >around 101K prefixes. > >128M should do if you only have two or three upstreams, soft reconfiguration >disabled and almost nothing else enabled. Does anyone have experience with >zebra/mrt/... as a route server?. I need one and would like to you for the >cheap server based solution instead of having to buy a fully fledged router. > >TIA > >-- Arnold >

1 0

Smurf Amplification and Sonoma Access
by Jon Stanley 14 Jul '01

14 Jul '01

I have a big problem with the Sonoma Access prodcut that I have not been able to figure out how to solve, hoping maybe someone here has the product and can help. Basically the problem is that the box is a smurf amplifer by default, and there seems to be no way to disable it. I have looked through both the console interface and the SonomaView GUI with no success. Being that we have a fairly significant amount of these on our network, this is a major problem. By the way, this product is the same as the Nortel Universal Edge 640, they bought Sonoma..... Any help would be appreciated....

1 0