- NANOG - lists.nanog.org

Re: Building a NOC
by Howard C. Berkowitz 23 Mar '98

23 Mar '98

>Floor drains are good. But once again, you need to have a chat with >your designer & inspector about what, why, when. Anti-backflow valves, >and floor sink traps are mandatory. Automatic trap primers seem to lead >to flooding. For example, St. Louis County recently opened a new jail. >However it was delayed three months because the week before opening when >they discovered if all the toilets were flushed at the same time on the >eighth floor, as prisoners are known to do, the sinks exploded on the >bottom floor. The plumbing designer had followed normal industry practice >for commercial buildings, but forgot jails house a user population that >is different from a normal commercial building. Hmmmm...reminiscent of a NANOG meeting? > >Like most disasters, you may not be able to prevent flooding, but you >can mitigate some of its affects. Hit the E.P.O. and unfurl the plastic >tarps you have stored & handy for just this emergency. Good point. After one flood from upper-story firefighting, the US Senate Computer Center did something along these lines I haven't seen elsewhere but wasn't a bad idea. They got some large rolls of plastic, and mounted the rolls on bars suspended from the ceiling -- sort of like giant toilet paper rolls. The leading edges of the rolls were folded over a strip of wood, and a rope tied to that. In the event of a fire/flood alarm, the operators could grab a handle at the end of the rope, and pull the plastic sheet over the top of equipment in under a minute. Far faster than unfolding tarps. > >>Putting equipment on high, rather than low, floors can protect it. > >Pick your disaster scenario, low-flying airplanes or high-flowing water. How about a lost Forest Service water bomber? No statistics available, but I would suspect water flowing down is far more common than airplanes flowing down. >>I wonder if any of the TEMPEST manuals have been declassified? THey never >>were at a terribly high level, and had some very nice general engineering >>in them. > >Not yet. Although I've been trying to figure out how to buy the old >MCAUTO (McDonnell-Douglas Automation) Data Center and turn it in to the >Mid-West high-tech co-location space. In addition to a three-story, >multi-football field long data center, and a backup power plant a small >city would envy, it is rumored to have so-called black-project rooms with >TEMPEST/EMP/EMF/RFI/TLA... shielding. InterNIC and PSI could co-locate >in the same building, and not run into each other even in the parking >lot. Is there a place that would be good for Fleming? >I've only seen one cafeterria in the building, so that could be >a problem :-). This may be the same data center that Bob Courtney, the retired security chief for IBM, talked about when he told the story of the Rustproof 360/195 in A Large Unnamed Aerospace Company in St. Louis. Apparently, they had a 195, a supercomputer for the time, in a magnificent computer room. Fire protection up the wazoo. Every kind of power and grounding anyone could think of. What no one did think of was that there were two paint shops in the same building, and the paint tanks were in one of them, with piping to the other. Guess what the overhead pipes ran over. Late one night, a pipe blew and dumped several hundred gallons of zinc chromate primer into the CPU cabinet. The resulting hardened mass gave new meaning to the term, "solid state." Moral: check for hazardous materials elsewhere in the building, and materials that might not be hazardous to people but will do the nasty to equipment.

2 1

Re: IP over SONET considered harmful?
by Sean M. Doran 23 Mar '98

23 Mar '98

| There's LQM in PPP. I would prefer to eliminate PPP. | Better yet, | what i do with splicing traffic into many channels effectively | provides IP-invisible restoration w/o loss of useful capacity. | Just route those channels into diverse physical paths :) Um, unfortunately I missed your presentation at NANOG and there is as yet no sign of the real video archives that I can find, so going by memory and your web pages, what I remember is that you are planning on using interfaces roughly comparable to STS-1 and STS-3c, and using telco WAN gear to do the work of add/drop multiplexing. I thought this was rather clever, actually, particularly given the accuracy of the prediction that non-facilities-owners would have trouble getting access to anything faster than DS3s in the near run. If you have changed your mind and are building your own fibre muxes, that would be neat to know. :) | > SDH/SONET is also a really keen way of sharing a network | > among IP and other things on the wide-area side, and | > an even keener way of having access to relatively low-speed | > customer data on the more local side. | | On customer-access side even ATM seems to be fine :) It | definitely is a huge lot cheaper than SONET gear. It is pretty easy to pluck out VCs/SPEs from SDH/SONET. The general thought was to have the equivalent of a CT3 (cisco device, takes T3 in on BNC pair, pulls out DS1s) that can extract from, say, an OC3 or OC12, DS3s, SPEs containing T1s and E1s, and perhaps even individual DS0s. (DS0s would be neat as with clever load-balancing that gives you a granularity that is competitive with the use of ATM as a fine-grained TDM system). Essentially, an ADM on a card. I don't see that as being prohibitively expensive to engineer, and ironically I think it would be easier for you than for some of your competitors to make switching work, since they may be forced to simulate a hierarchy of routers on the card. | Ciena folks told me some other compelling reasons for dropping | SONET, which i do not feel i'm at liberty to discuss here. Interesting. Aim them at me or Peter if it doesn't have to do with SONET's speed ceilings. | I also expect it :) My current favourite for framing is | 32-to-33 bit encoding, with flag being one of "malformed words", | one word header (2 bytes for payload length, 2 bytes for tag), and | 32-bit CRC at the end. I.e. the per-packet overhead is 12 bytes + 3.1% + | rounding to 4 bytes for odd-sized packets. Make all 1s and all 0s | and chess patterns to be invalid words, and loss-of-carrier becomes | easy to detect. Well, I still prefer the idea of a synchronous byte stream to a clever HDLC on steroids... :) Sean.

2 1

Re: IP over SONET considered harmful?
by Mike Trest (In Japan) 23 Mar '98

23 Mar '98

At 06:32 AM 3/21/98 -0800, Sean M. Doran wrote: >| Native IP over fiber is not a religion; it's a solid engineering. >snip< >SDH/SONET is also a really keen way of sharing a network >among IP and other things on the wide-area side, and >an even keener way of having access to relatively low-speed >customer data on the more local side. Both in US and In Japan, SONET/SDH systems are used to deliver such non-IP traffic as CATV/Video over the same physical facilities as IP traffic. Indeed, I can walk you into a top-rated commercial television station using a large Unix box with six ATM-over-SONET interfaces to drive video broadcast streams to the stations' live broadcast to millions of viewers. The same interfaces also transport TCP-IP application control data between Unix and the various off-line editing systems and the graphic development engines (MAC/W'95). You should not think that the prospect of internet based viewers are outside the planning horizion of commercial broadcast industry, even if that industry consists of predominately time critical (non-IP traffic) today. Some major players in this industry agree that the mixture of SONET/SDH with both IP and ATM traffic is "really keen". ..mike..

2 1

Re: Building a NOC
by Sean Donelan 22 Mar '98

22 Mar '98

>Yes. And your consultant must know local practices as well. If you are >building a facility in Fairfax County, Virginia (a major technology suburb >of Washington DC), avoid, in your plans or anything the building inspector >may see, the term "computer room." "Communications room," "network room," >etc., all prevent the problem: if they see "computer room," they will >demand a mainframe-style central emergency power off control, which greatly >increases electrical wiring cost. The psychology of dealing with building inspectors would fill an entire book, however this is a good example of understanding the "Why." If building inspectors and computer room designers were logical, they would realize why this requirement exists. They would also realize when the requirement didn't apply. The reason why the requirement exists is computer room wiring often doesn't meet all the safety and fire retardation requirements of normal occupancy space because the entire computer room is treated as if it was a restricted access area and fire retarding space. Hence the requirements for a seperate HVAC system, fire-rated walls, and a single disconnection means (i.e. EPO) for the entire room INCLUDING the UPS(s). On the other hand, if you followed normal occupancy wiring practices, one would expect the space could be treated as normal occupancy space by the building inspector. I know, that's not how inspectors think. I'm not sure eliminating the EPO would really be cheaper. Its not simply a matter of eliminating the EPO, but also replacing much of the raised floor cabling with conduit, plenum rated cable, and other changes. Not just power cables, but also communications cables, such as the cable connecting a disk cabinet to the CPU cabinet. And worse, from the building inspector's point of view, having some assurance someone won't install some cable next week without having it inspected. I realize in most offices there are more computers on people's desktops than in the computer room. At this point I get to critize designers for how they think. Electronic equipment is spread throughout the organization, and the infrastructure to support it can't be limited just to the computer room. But the building codes are always going to be playing catchup, so designers and inspectors have their negotiating work cut out for them. >Wasn't thinking of MILSTDs as much as some of the DISA documents, and often >relating to base practices. The DoD manual on grounding, bonding, and >shielding is excellent. This was precisely the one I was thinking about. Since the IEEE Emerald book was published in 1992, gounding, bounding and shielding practices for electronic equipment have been getting updated. There is a LOT of techno-babble floating around about computer room grounding (pun intended). Make sure you have the updated manual. There are some old ideas about practices such as isolated grounds for electronic equipment which are dangerous, and have been revised/clarified in the updates. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

2 1

Building a NOC
by John_W_Connors＠res.raytheon.com 22 Mar '98

22 Mar '98

Would anyone have information or knowledge of books, WWW Sites, or other means that would provide details on constructing a state of the art NOC to support 75000+ nodes in a LAN/MAN/WAN environment Thanks in advance JC

9 8

Re: IP over SONET considered harmful?
by Sean M. Doran 22 Mar '98

22 Mar '98

| Sounds reasonable to me. My point was that decrementing the IP TTL, | whether it be at each LSR or only at egress, should be a decision | that can be made by the provider. Well, in that case you also have to not copy the IP TTL into the tag TTL, but rather synthesize a new tag TTL, assuming the goal is not to end up with traceroute stars. I think this is insane, personally, but then Sprintlink in the past fell victim to devastating forwarding loops, both tight and loose; the latter were particularly bad, and so maybe I am "over sensitive". If you _really_ want to conceal one's network, why not just outright lie -- at your ingress LSR look for packets with low TTLs, and for ones that won't make it to their destinations, synthesize an ICMP time exceeded with your choice of source addresses. (cf the last paragraph of 2.14 in the current MPLS arch draft) Where this is not possible, you generate the lie as close as possible to the point where the ttl timeout would happen. Sean.

1 0

Re: Building a NOC
by Sean Donelan 22 Mar '98

22 Mar '98

Both good ideas. But be sure to learn and understand when, where and why the operational experiences given in these sources apply to building a NOC. One difficulty in designing a NOC for an ISP is they operate across several different problem domains, telecommunications, data processing, lan, wan, etc. Be careful if you hire a consultant with expertise in one particular area. NEBS and the like have several good ideas, but they also have a built-in set of assumptions. If you, or your consultant, don't understand when the assumptions apply and don't apply, when you apply the solutions to a different problem domain, you can actually make things worse. For example, the National Electrical Code doesn't apply to communications equipment located inside a communication utility building. But the building inspector will get very grouchy if an ISP's electrical wiring doesn't meet NEC standards. Also, there have been changes in 'standard' practices in the last 5 years. If your sources for designing data processing centers were written prior to 1990, I would look for a second source. The military has been updating its standards to reflect revised commercial practices, but the civilian sellers of the MILSTDs don't always show when an old practice has been superceded. My basic problem with the original question is I don't really understand what kind of NOC the person is trying to build. Are we talking about how build a NOC staff, or a NOC building to monitor a network of 70,000 nodes? Are we talking about building a NOC theatre so management/customers can view CNN on the projection TV wall when they visit, or a lights-out server/equipment room designed for limited human occupation? Are we trying to manage 70,000 remote systems, or just the core network connecting the sites. >Excellent observation. There was a text, _Engineering and Operations in >the Bell System_, that gave some good operational experience. With >divestiture, I'm not sure who took it over. I vaguely remember that >_Transmission Systems for Communications_ went to AT&T Bell Labs, and E&O >went to Bellcore. > >I'd also consider looking through some of the military communications >manuals. Unfortunately, I'm in the midst of a computer conversion and have >to recover some bookmark files, but you can search for Army Field Manuals >online -- they are at the TRADOC site at Ft. Monroe. -- Sean Donelan, Data Research Associates, Inc, St. Louis, MO Affiliation given for identification not representation

4 3

Re: IP over SONET considered harmful?
by Sean M. Doran 22 Mar '98

22 Mar '98

Scott Brim wrote: | For example, in order for TTL to be effective, not every node needs to | decrement it -- we just need to be sure that no loops can form where it | doesn't get decremented. Yes, but you also want to make sure that the price-per-packet is really really really close to zero in the "gap" between ttl decrementing routers, otherwise, even though a loop may never form in the non-ttl-decrementing fabric, it may form across it, for example, between the two routers on either side of the gap, or through the two routers on either side of the gap. If the "gap" is a sub-IP fabric a continent and an ocean wide, such a loop tends to be somewhat expensive. TTL decrementing as a loop mitigation feature ought to be fairly popular in such circumstances. Sean.

1 0

Re: Building a NOC
by shields＠crosslink.net 22 Mar '98

22 Mar '98

In article <v03007825b13a859bc617(a)[168.143.1.215]>, "Howard C. Berkowitz" <hcb(a)clark.net> wrote: > Of course, the bridge of the NCC-1701 series Enterprise(s) here show > incredibly bad human factors design, but that is often ignored. Perhaps > more realistic is the display over the medical bed. While one of the bar > graphs has text too small to read on the TV screen, the Star Trek technical > manual shows how it is labeled "Medical Insurance Remaining." > > Surely there is a NOC equivalent of that bar graph. "Venture Capital Remaining"? -- Shields, CrossLink.

1 0

Re: Building a NOC
by Howard C. Berkowitz 21 Mar '98

21 Mar '98

At 14:51 -0500 3/16/98, Charles Sprickman wrote: >I think I'd disagree with that. Where would all you folks be if you never >picked up a book? You wouldn't be reading this list. Seriously, while >there's not an ORA "How to Build a NOC" book with some cute animal on the >cover, there is some information out there worth reading. > >I'd suggest digging through some telco info (if you'd like to consider a >CO a NOC). Also, there should be plenty of people on this list that >work in or have even designed decent sized NOCs. You can also cheat and >go browsing as a prospective customer at someplace like Exodus or any >other colo facility that is well-managed and modern. You can actually >learn quite a bit by just looking around... Excellent observation. There was a text, _Engineering and Operations in the Bell System_, that gave some good operational experience. With divestiture, I'm not sure who took it over. I vaguely remember that _Transmission Systems for Communications_ went to AT&T Bell Labs, and E&O went to Bellcore. I'd also consider looking through some of the military communications manuals. Unfortunately, I'm in the midst of a computer conversion and have to recover some bookmark files, but you can search for Army Field Manuals online -- they are at the TRADOC site at Ft. Monroe. > >Hopefully some of the startup NSPs will answer here, as the experience >should still be fresh in their minds... > >Charles > >~~~~~~~~~ ~~~~~~~~~~~ >Charles Sprickman Internet Channel >INCH System Administration Team (212)243-5200 >spork(a)inch.com access(a)inch.com > >On Mon, 16 Mar 1998, Martin Hannigan wrote: > >> Date: Mon, 16 Mar 1998 14:14:47 -0500 (EST) >> From: Martin Hannigan <hannigan(a)xcom.net> >> To: John_W_Connors(a)res.raytheon.com >> Cc: nanog(a)merit.edu >> Subject: Re: Building a NOC >> >> > >> > Would anyone have information or knowledge of books, WWW Sites, or other >> > means that would provide details on constructing a state of the art NOC to >> > support 75000+ nodes in a LAN/MAN/WAN environment >> > >> > Thanks in advance >> > JC >> > >> > >> > >> >> No books, it's all experience. >> >> >> Martin Hannigan V:617.500.0108 >> Director - Data Networks F:617.500.0002 >> XCOM Technologies, INC. hannigan(a)xcom.net >> Cambridge, MA http://www.xcom.net >>

1 0