At 08:52 AM 3/23/98 -0500, you wrote:
Pocket protectors? Slide rules? Any other items nerd/engineers carry in their shirt pocket that falls out when they are head-first into the raised floor.
We've found socks and cigarette butts. Needless to say the perp is no longer with us :)
Like most disasters, you may not be able to prevent flooding, but you can mitigate some of its affects. Hit the E.P.O. and unfurl the plastic tarps you have stored & handy for just this emergency.
We also put drip pans which cover the entire ceiling over the data center - we were burnt 2 years ago in a pre-war office building in NYC. (avoid space with wc's above!) The pans drain outside of the machine room into a high-traffic hallway (with a drain). Any drips get noticed very quickly, and we don't have to test the water alarm.
I guess that flooding by the mains water is seriousl worth thinking about. For example we just realised that because the "mens room" (and the "ladies) lavatory cistern overflows go into the room (they are in the middle of the building) if something goes wrong we have a little flood. So any mains water installations (dishwashers, kitchens, showers) are a potential hazzard - solution a) install mains water cutoff valves with sensors in the floors b) get everyone not involved in the NOC out of the building. We are just doing both! I suppose that all the wiring could be in the ceiling although that would not help in multistorey building. Rambling a bit but what about the film Inferno with all that water stored at the top of the building. :-)) Bill
-- U-NET Ltd "Just Internet" mailto:bill@u-net.net http://www.u-net.net/ Tel 01925 484444 Fax 01925 484466
At 09:22 +0000 3/24/98, Bill Unsworth wrote:
At 08:52 AM 3/23/98 -0500, you wrote:
Pocket protectors? Slide rules? Any other items nerd/engineers carry in their shirt pocket that falls out when they are head-first into the raised floor.
We've found socks and cigarette butts. Needless to say the perp is no longer with us :)
Like most disasters, you may not be able to prevent flooding, but you can mitigate some of its affects. Hit the E.P.O. and unfurl the plastic tarps you have stored & handy for just this emergency.
We also put drip pans which cover the entire ceiling over the data center - we were burnt 2 years ago in a pre-war office building in NYC. (avoid space with wc's above!) The pans drain outside of the machine room into a high-traffic hallway (with a drain). Any drips get noticed very quickly, and we don't have to test the water alarm.
I guess that flooding by the mains water is seriousl worth thinking about. For example we just realised that because the "mens room" (and the "ladies) lavatory cistern overflows go into the room (they are in the middle of the building) if something goes wrong we have a little flood. So any mains water installations (dishwashers, kitchens, showers) are a potential hazzard - solution a) install mains water cutoff valves with sensors in the floors b) get everyone not involved in the NOC out of the building. We are just doing both! I suppose that all the wiring could be in the ceiling although that would not help in multistorey building. Rambling a bit but what about the film Inferno with all that water stored at the top of the building. :-))
Bill
The anecdotes in this discussion thread, I think, are definitely worth keeping somewhere, although I'm not sure of the venue. Some of us had been vaguely chatting in Albuquerque about a documentation part of the NANOG web page. But some non-obvious threads are surfacing in this discussion, at least non-obvious if you haven't been there. Restating a few of the less obvious ones, -- Define the NOC's function -- Define the NOC's audience, including people that will have no useful function but do have political influence that MUST be satisfied -- When considering a physical site, do a careful and paranoid threat of the planned space, the building, and its immediate environs. You will want to identify possible hazards including fire, flood, etc. -- Consider how you will get large equipment in and out of the site, especially those that might need emergency replacement. What if the building is on limited power and the elevator is down? Some buildings have to have large equipment lifted in with a crane or helicopter. Air conditioning equipment or major power supplies are examples of very hard to handle components. -- As part of the site survey, understand how electrical power and communications feeders come into the space and building. Are there alternatives for redundancy? -- Will there be a backup NOC, even consisting of a node into which staff can dial? -- Do a thorough electrical requirmements plan, and be sure the grounding system is up to current practices. When devices have multiple power supplies (e.g., Cisco 7000/7500), be sure they are plugged into circuits on different breakers. -- Think through who will be planning, installing, and inspecting signal cabling. If it's a union shop, or especially if non-union personnel will be doing any work in a union area, see if you can get the shop steward on your side. -- Be sure cellular/cordless phones will work in all your wire closets, or be sure there are voice jacks where a telephone set can be plugged in to coordinate testing. -- When planning cable runs, be sure you can access critical components that otherwise might be buried under a heavy mass of cables. Plan the runs so they will not interfere with cards sliding in and out of chassis. -- Beware of snakes, floor drains, shock hazards, etc., under raised floors. -- Consider backup facilities for critical people. If there was a major disaster that isolated the site, but still let some of its function work, where will staff sleep? Are there emergency food supplies (not a bad idea even beyond major disasters -- if people are working around the clock, it's good to have alternatives besides pizza, if that is available)? Especially if you are in earthquake or amorous rat country, think about catastrophes that can knock out the NOC but leave parts of your network working.
On 24-Mar-98 Howard C. Berkowitz wrote:
At 09:22 +0000 3/24/98, Bill Unsworth wrote:
At 08:52 AM 3/23/98 -0500, you wrote: [snip]
The notes below will be a keeper for our organization for some time. We are in the process of building a NOC. Anyone interested in hearing the progress (or lack of) thus far, let me know and I can email some of the information direct. Thanks to those who contributed to the list below, I will be using it much from now on. Morgan
The anecdotes in this discussion thread, I think, are definitely worth keeping somewhere, although I'm not sure of the venue. Some of us had been vaguely chatting in Albuquerque about a documentation part of the NANOG web page.
But some non-obvious threads are surfacing in this discussion, at least non-obvious if you haven't been there. Restating a few of the less obvious ones,
-- Define the NOC's function
-- Define the NOC's audience, including people that will have no useful function but do have political influence that MUST be satisfied
-- When considering a physical site, do a careful and paranoid threat of the planned space, the building, and its immediate environs. You will want to identify possible hazards including fire, flood, etc.
-- Consider how you will get large equipment in and out of the site, especially those that might need emergency replacement. What if the building is on limited power and the elevator is down? Some buildings have to have large equipment lifted in with a crane or helicopter. Air conditioning equipment or major power supplies are examples of very hard to handle components.
-- As part of the site survey, understand how electrical power and communications feeders come into the space and building. Are there alternatives for redundancy?
-- Will there be a backup NOC, even consisting of a node into which staff can dial?
-- Do a thorough electrical requirmements plan, and be sure the grounding system is up to current practices. When devices have multiple power supplies (e.g., Cisco 7000/7500), be sure they are plugged into circuits on different breakers.
-- Think through who will be planning, installing, and inspecting signal cabling. If it's a union shop, or especially if non-union personnel will be doing any work in a union area, see if you can get the shop steward on your side.
-- Be sure cellular/cordless phones will work in all your wire closets, or be sure there are voice jacks where a telephone set can be plugged in to coordinate testing.
-- When planning cable runs, be sure you can access critical components that otherwise might be buried under a heavy mass of cables. Plan the runs so they will not interfere with cards sliding in and out of chassis.
-- Beware of snakes, floor drains, shock hazards, etc., under raised floors.
-- Consider backup facilities for critical people. If there was a major disaster that isolated the site, but still let some of its function work, where will staff sleep? Are there emergency food supplies (not a bad idea even beyond major disasters -- if people are working around the clock, it's good to have alternatives besides pizza, if that is available)?
Especially if you are in earthquake or amorous rat country, think about catastrophes that can knock out the NOC but leave parts of your network working.
----------------------------------------- Morgan Sarges Voice Phone: 605-338-8334 blip@morgan.iw.net Fax: 605-335-3942 System Administrator Dakota Telecommunications Group, Internet Division Network Operations Center No man is an island, but some of us are long peninsulas. PGP Public Key block available upon request. ------------------------------------------
I helped build a NOC, too.. It was scrapped time and again when serious mistakes on the parts of all (planning and esp. management) were made. Consider manpower. Be nice to your NOC. Let THEM pick the times that the NOC shift will go. We started with a 3x9h shift (with overlap) and then scrapped it for a 12h model, which has taken morale down. Consider training. Dont hire people unless you have the time to train them. Serious time where other things will be put off. Assign someone to be trainer (someone clued) and get ALL other responsibilities off of him/her for the duration of the training, which, if done right, will be forever.. Don't "cut" the NOC in. Phase it in. "The NOC will be operational on x" really doesnt make sense. It will guarantee failure, unless you have an extra million laying around to hire triple the needed staff. My advice, start small, train each person, phase in the NOC.. slowly move duties over. If you have a phone system and a "NOC line", round robin calls or ACD them so x% of them go to the NOC and the rest to the "traditional" phone answerers. That's about all I can think of.. i'm sick as hell, just woke up.. 213 am.. ugh.. need.. more.. nyquil.. -jamie PS/The noc was not AIS. It was at a former employer. -- jamie rishaw (dal/efnet:gavroche) American Information Systems, Inc. Tel:312.425.7140, FAX:312.425.7240
Are signatures migrating into headers? Dirk To: blip@morgan.iw.net Date: Wed, 25 Mar 1998 02:12:48 -0600 (CST) Cc: nanog@merit.edu RFC_Violation: You saw it here first! From: jamie@dilbert.ais.net (James Rishaw) Reply-To: jamie@ais.net Organization: American Information Systems X-PGP-Fingerprint: <921C135D> C4 48 1B 26 18 7B 1F D9 BA C4 9C 7A B1 07 07 E8 X-Y: I'd rather be hated for something I am than loved for something I'm not. X-No-Archive: Yes X-Face: d=) On Wed, Mar 25, 1998 at 02:12:48AM -0600, James Rishaw wrote:
I helped build a NOC, too..
It was scrapped time and again when serious mistakes on the parts of all (planning and esp. management) were made.
Consider manpower. Be nice to your NOC. Let THEM pick the times that the NOC shift will go. We started with a 3x9h shift (with overlap) and then scrapped it for a 12h model, which has taken morale down.
Consider training. Dont hire people unless you have the time to train them. Serious time where other things will be put off. Assign someone to be trainer (someone clued) and get ALL other responsibilities off of him/her for the duration of the training, which, if done right, will be forever..
Don't "cut" the NOC in. Phase it in. "The NOC will be operational on x" really doesnt make sense. It will guarantee failure, unless you have an extra million laying around to hire triple the needed staff. My advice, start small, train each person, phase in the NOC.. slowly move duties over. If you have a phone system and a "NOC line", round robin calls or ACD them so x% of them go to the NOC and the rest to the "traditional" phone answerers.
That's about all I can think of.. i'm sick as hell, just woke up.. 213 am.. ugh..
need.. more.. nyquil..
-jamie PS/The noc was not AIS. It was at a former employer. -- jamie rishaw (dal/efnet:gavroche) American Information Systems, Inc. Tel:312.425.7140, FAX:312.425.7240
<NOOP flames="hit d"> Of course. The reason for signatures is because headers were sometimes not good enough to figure out where messages came from.. It's kinda like your appendix :) Still there, but now in "modern times" you really dont need it. Simple, small .signature for people to have your phone number, and the "cool" stuff in the headers. </NOOP> dirk@power.net wrote:
Are signatures migrating into headers?
Dirk
To: blip@morgan.iw.net Date: Wed, 25 Mar 1998 02:12:48 -0600 (CST) Cc: nanog@merit.edu RFC_Violation: You saw it here first! From: jamie@dilbert.ais.net (James Rishaw) Reply-To: jamie@ais.net Organization: American Information Systems X-PGP-Fingerprint: <921C135D> C4 48 1B 26 18 7B 1F D9 BA C4 9C 7A B1 07 07 E8 X-Y: I'd rather be hated for something I am than loved for something I'm not. X-No-Archive: Yes X-Face: d=)
-- jamie rishaw (dal/efnet:gavroche) American Information Systems, Inc. Tel:312.425.7140, FAX:312.425.7240
On Wed, Mar 25, 1998 at 07:10:41AM -0800, dirk@power.net wrote:
RFC_Violation: You saw it here first! X-Y: I'd rather be hated for something I am than loved for something I'm not. X-No-Archive: Yes X-Face: d=)
Are signatures migrating into headers?
Gee, Dirk... I didn't even notice, until you pointed it out. Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "Two words: Darth Doogie." -- Jason Colby, Tampa Bay, Florida on alt.fan.heinlein +1 813 790 7592 Managing Editor, Top Of The Key sports e-zine ------------ http://www.totk.com
participants (6)
-
Bill Unsworth -
dirk＠power.net -
Howard C. Berkowitz -
jamie＠dilbert.ais.net -
Jay R. Ashworth -
Morgan Sarges