We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling. We have a few ideas, all of which have really bad downsides: - 4day, 10hour rotation. 2-hour overlap. - Downside is overlap, and the fact that nobody will have a regular schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh. - M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras". - Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap. Anyhow,, we're looking for information on how other NOCs around the world do it. We're trying to give good coverage without killing/burning out the NOC team members. TIA. Jamie Rishaw
- Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap.
How many of your NOC people willing to work that schedule? i worked 7 months of having tues/wed off. It sucked, becuase they allways called me even when i was OFF!. Now that i have weekends off, they only call me when its real important (Not my definition of Important ... :(). Scott
On Oct 2, Jamie Rishaw <jamie@arpa.com> wrote:
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
We have a few ideas, all of which have really bad downsides:
- 4day, 10hour rotation. 2-hour overlap. - Downside is overlap, and the fact that nobody will have a regular schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.
I understand AGIS used to do that. AFAIK, the two (yes, two) people they had on that rotation quit long ago.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras".
You're gonna want good people on weekends, 'cause it's harder to find a clueful telco person during non-business hours.
- Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap.
You're probably gonna have overlap at some point; either that or have somebody working extra hours every week.
Anyhow,, we're looking for information on how other NOCs around the world do it. We're trying to give good coverage without killing/burning out the NOC team members.
Can't be done. *grin* At a previous job, we had eight-hour shifts, no formal overlap; for quite a while we'd have one person in the NOC at any given time. Each week, one of us would work six days. There wasn't any overtime. It's a good thing we all liked working there; in fact, once we stopped enjoying the job everybody who was in the NOC at that time quit. What I'm trying at Priori is four-hour shifts. People can work eight or sometimes even twelve hours in a row if they want, but it's all in four-hour blocks. Since we're expecting that at least some of our part-timers will be students, that gives us a lot of flexibility to deal with their schedules. We don't yet have all the shifts covered, so Justin, Michael, and myself are covering as needed, and I don't yet have a good feeling as to how well this is going to work. Luckily, since we're a startup, it won't be too hard to move to something else as needed. The most important thing, IMHO, is to make sure that the schedule you have set is something that your staff can live with -- at that previous job I mentioned we were all completely insane and actually didn't mind working a couple day shifts, having a day and a half off, and then coming in for night shifts, and rotating that around every couple of weeks -- except for one full-time student, we preferred that to being stuck with the same hours every day of every week. ********************************************************* J.D. Falk voice: +1-650-482-2840 Supervisor, Network Operations fax: +1-650-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." *********************************************************
At 05:34 PM 10/2/97 -0400, Jamie Rishaw wrote:
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
Ahhh, a subject near and dear to all NOC managers. Tip 1) Talk to the people involved, give them a voice in the decision. They will be much less prone to gripe later on if they helped define the schedule. Tip 2) Overlap is *good* in a 7x24 team. No way can a group function as a team if they never have any face to face time. I was, and still am, a firm proponent of this. Seeing NOC people hang around and talk is a good thing; later on when pressure mounts and tempers flare, its much better they be friends. Tip 3) Fixed schedules are better than fluctuating. People need some stability to arrange their lives around. Tip 4) Double up all shifts otherwise, when your weekend guys wife goes into labor early (on New Years Eve), you aren't scrambling for coverage. It happens. To do 7x24 comfortably, you'll need 12 people and a supervisor. We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three days off in a row, including one weekend day. We have our staff meetings/ training on Wednesday afternoons. We overlap the split weeks with Senior/Supervisory staff who work M-F and tie the two ends of the week together. The only ones who are left out are the overnight crew. A night supervisor is responsible for communication/training with them. This is based upon nine years of operations center experience. (hello to those BBN NOC alumni...) Steve Miller WebTV Service Operations Center
We have a few ideas, all of which have really bad downsides:
- 4day, 10hour rotation. 2-hour overlap. - Downside is overlap, and the fact that nobody will have a regular schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras".
- Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap.
Anyhow,, we're looking for information on how other NOCs around the world do it. We're trying to give good coverage without killing/burning out the NOC team members.
TIA.
Jamie Rishaw
On Oct 2, Steve Miller <smiller@corp.webtv.net> wrote:
We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three days off in a row, including one weekend day. We have our staff meetings/ training on Wednesday afternoons. We overlap the split weeks with Senior/Supervisory staff who work M-F and tie the two ends of the week together. The only ones who are left out are the overnight crew. A night supervisor is responsible for communication/training with them.
Keeping the night shift informed and included in what's going on is very important. I spent a few months as night NOC supervisor at a large ISP on the East Coast and was never sure what was going on -- not because of any failing on the part of my supervisor (hi Jason), but more because there wasn't any mechanism in place for communicating most projects & such with the night staff besides word of mouth from whoever worked late that day. I actually know almost as much about what's going on at night there now, after moving all the way across the country, as I did when I showed up for work every night. That's one of the reasons that I and my cow orkers at that previous job I mentioned actually liked having rotating shifts; it meant we'd never be stuck always working at night and never knowing what's going on, or always working during the busiest times. As long as the schedule is sent out a couple weeks or even months in advance, people can plan their lives around it. ********************************************************* J.D. Falk voice: +1-650-482-2840 Supervisor, Network Operations fax: +1-650-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." *********************************************************
Steve Miller writes:
We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three days off in a row, including one weekend day.
One of the more successful 24x7 operations I've seen did 12 hour days, four days on, four days off, two shifts. People had to get used to it, but for many of the staffers who were used to putting in 12 hour days anyway the prospect of a four day "weekend" every week was pretty good, though it appealed the most to the single operations staffers and to the ones who had non-working spouses. It had a lot of advantages in terms of operational continuity and it actually kept morale pretty high because people got enough days away not to fry from the stress of dealing with users constantly every day. Perry
Having tried the 12hr model in a stressful environment like a NOC, it did not work well as people were burned out by the end of the day and those that did the 12hr shifts were not content with their positions and looked to leave the group sooner than those that had more normal shift hours. The 4-day work week was found to be the best model by giving people 3 days off. At 08:02 PM 10/2/97 -0400, Perry E. Metzger wrote:
Steve Miller writes:
We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three days off in a row, including one weekend day.
One of the more successful 24x7 operations I've seen did 12 hour days, four days on, four days off, two shifts. People had to get used to it, but for many of the staffers who were used to putting in 12 hour days anyway the prospect of a four day "weekend" every week was pretty good, though it appealed the most to the single operations staffers and to the ones who had non-working spouses. It had a lot of advantages in terms of operational continuity and it actually kept morale pretty high because people got enough days away not to fry from the stress of dealing with users constantly every day.
Perry
At 03:56 PM 10/2/97 -0700, Steve Miller wrote:
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
Ahhh, a subject near and dear to all NOC managers.
Steve, I'm glad you piped up and gave the benefit of your experience at BBN and what you've learned since we worked together back then.
To do 7x24 comfortably, you'll need 12 people and a supervisor.
We work 4/10hr days, Sun/Mon/Tue/Wed or Wed/Thur/Fri/Sat. People get three days off in a row, including one weekend day. We have our staff meetings/ training on Wednesday afternoons. We overlap the split weeks with Senior/Supervisory staff who work M-F and tie the two ends of the week together. The only ones who are left out are the overnight crew. A night supervisor is responsible for communication/training with them.
Do you run three 10 hour shifts per day or two? Seems like a lot of overlap and makes me wonder whether the minimum 12 head count rule still holds. Three shifts per day, two crews per week, two people per shift works out to 12 people, but that leaves you without a job or else down a backup supervisor. :-) Do you run some shifts with one person on site and others with three?
This is based upon nine years of operations center experience. (hello to those BBN NOC alumni...)
Steve Miller WebTV Service Operations Center
You might as well do the whole nine yards. Is there a decent ISP network management system or is it still OpenView and a lot of utilities? --Kent Kent W. England Wireline-> fon:650.596.6321 VP of Technology Receptionist-> fon:650.596.1700 GeoNet Communications, Inc. Facsimile-> fax:650.596.1701 555 Twin Dolphin Drive Email-> mailto:kwe@geo.net Redwood City, CA 94065 Web Site-> http://www.geo.net Wireless Text-> mailto:pagekwe@geo.net PGP Key-> http://keys.pgp.com:11371/pks/lookup?op=get&search=0x6C0CDE69
On Oct 2, "Kent W. England" <kwe@geo.net> wrote:
You might as well do the whole nine yards. Is there a decent ISP network management system or is it still OpenView and a lot of utilities?
I've been playing with lots of 'em recently. Of the free ones, only MRTG and Tkined are worth a damn -- even though lots of people swear by NOCOL, I've been swearing at it 'cause the configuration is hard to visually parse and not consistent across the various monitoring programs. I'd have to say that I prefer HP Openview to Tkined, but for what I'm doing the difference is not so great that Openview would be worth the money. Also, Tnm (the Tcl add-on that is used by Tkined) has lots of useful hooks for SNMP, ping, etcetera, and can be used to write custom stuff pretty easily. I'll be distributing some of what I write here once I'm happy with how it functions. ********************************************************* J.D. Falk voice: +1-650-482-2840 Supervisor, Network Operations fax: +1-650-482-2844 PRIORI NETWORKS, INC. http://www.priori.net "The People You Know. The People You Trust." *********************************************************
J.D. Falk boldly claimed:
On Oct 2, "Kent W. England" <kwe@geo.net> wrote:
You might as well do the whole nine yards. Is there a decent ISP network management system or is it still OpenView and a lot of utilities?
I've been playing with lots of 'em recently. Of the free ones, only MRTG and Tkined are worth a damn -- even though lots of people swear by NOCOL, I've been swearing at it 'cause the configuration is hard to visually parse and not consistent across the various monitoring programs.
I'd have to say that I prefer HP Openview to Tkined, but for what I'm doing the difference is not so great that Openview would be worth the money.
Also, Tnm (the Tcl add-on that is used by Tkined) has lots of useful hooks for SNMP, ping, etcetera, and can be used to write custom stuff pretty easily. I'll be distributing some of what I write here once I'm happy with how it functions.
You can get the program that I've written, you can teach it network dependencies, and various other good things.. I'm trying to get a few more features and bugfixes into it before I make my next "official" release, but you can get it from ftp://puck.nether.net/pub/jared/sysmon-0.77-pre.tar.gz It takes sighups, and lots of other neato things... with curses, java, and even a python client to talk to the server. It's unix based, and should build out of the box on any system with gcc. I'm happy to accept bug reports and feature requests. - Jared -- ----------------- jared@puck.nether.net - Nether Network ------------------ CICNet/IAGNet/NetherNet - finger jared@puck.nether.net for pgp key
I've no experience in scheduling for a NOC, but have managed support organizations that operate 24/7. My general conclussions are that not everything works for everyone AND you will get burnout regardless of the schedule. However some things I've learned about 24/7 schedules -- 1. People tend to be better able to deal with 10 hour shifts. Some people can deal with 12 hours shifts but people get more surly and the burn out is higher. 2. Some overlap is good. You need a time to relay implicit knowledge (we all wish everything was written down, but reality dictates otherwise). In fact a Wednesday overlap is probably really not that bad at all - It provides a great time for any kind of triage or department meetings that might need to occur. Jamie Rishaw wrote:
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
We have a few ideas, all of which have really bad downsides:
- 4day, 10hour rotation. 2-hour overlap. - Downside is overlap, and the fact that nobody will have a regular schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras".
- Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap.
Anyhow,, we're looking for information on how other NOCs around the world do it. We're trying to give good coverage without killing/burning out the NOC team members.
TIA.
Jamie Rishaw
It should be noted that larger NOC operations understand that their staff members will either: o Burn out and die. o Learn lots and get a real job with better pay. Thus, many NOC supervisors generally rate the expected lifetime for a NOC staff member at 13 months. Really. Having said that, it would behoove a good NOC supervisor to understand how to best use the resources at their disposal. That means not burning people our prematurely. Or better yet, burn them out. That means that once they get some real experience under their belts, the rest of us can hire them. :-) At least, that seems to be the case at the telco's. - paul At 06:46 PM 10/2/97 -0700, Aaron wrote:
I've no experience in scheduling for a NOC, but have managed support organizations that operate 24/7. My general conclussions are that not everything works for everyone AND you will get burnout regardless of the schedule.
However some things I've learned about 24/7 schedules --
1. People tend to be better able to deal with 10 hour shifts. Some people can deal with 12 hours shifts but people get more surly and the burn out is higher.
2. Some overlap is good. You need a time to relay implicit knowledge (we all wish everything was written down, but reality dictates otherwise). In fact a Wednesday overlap is probably really not that bad at all - It provides a great time for any kind of triage or department meetings that might need to occur.
Jamie Rishaw wrote:
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
We have a few ideas, all of which have really bad downsides:
- 4day, 10hour rotation. 2-hour overlap. - Downside is overlap, and the fact that nobody will have a regular schedule.. they'll be M-Th, then T-Fr, W-Sa, Th-Su, Fr-Mo.. bleh.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras".
- Su-We, We-Sa 10-hour shifts. Alleviates "skewing" schedules one day every week, but creates a 14.5% labor increase on Wednesday when you have a complete overlap.
Anyhow,, we're looking for information on how other NOCs around the world do it. We're trying to give good coverage without killing/burning out the NOC team members.
TIA.
Jamie Rishaw
Hi,
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling. I asked this question to the list a couple of months ago. I've been manager of a new NOC for about 6 weeks now.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras". This is basically what I went with. To integrate in the weekend people I have them do some half shifts during the week. I also have two of the people that willing to be a bit flexible on short notice. I have a two level on-call setup for problems that need extra help. Also, the more expert people can dial or ssh in as needed. Burnout can be a very real problem and I haven't come to that yet. I have hired a mix of experienced and inexperienced (college age) folks. I fully expect that people may either burnout or gain great experience and get better jobs, but that's always a risk. It could happen to me too!
Barry Barry Caplin People's Choice TV Network Operations Center Manager work: bcaplin@pchoicetv.com home: bc@mtiweb.com PGP fingerprint = 9F E9 5C B6 A2 AD 85 AE 1E 4D 11 0B C2 85 60 93 finger bc@mtiweb.com for PGP public key
How about what I had to live with for 8 years in Navy Satellite Communications: 2-48-2-72. It consisted of 4 watch sections, with a rotation of 2-12 hour days, then 48 hours off, 2-12 hour mids, then 72 hours off. I enjoyed this and had plenty of time to be involved in the projects that the day staff was involved with. You always knew your schedule and could have a family life as well. It'll screw with your sleeping schedule but overall it worked for the military and I'm sure it'd work in a NOC environment. Doug At 08:46 AM 10/3/97 -0500, Barry Caplin wrote:
Hi,
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling. I asked this question to the list a couple of months ago. I've been manager of a new NOC for about 6 weeks now.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras". This is basically what I went with. To integrate in the weekend people I have them do some half shifts during the week. I also have two of the people that willing to be a bit flexible on short notice. I have a two level on-call setup for problems that need extra help. Also, the more expert people can dial or ssh in as needed. Burnout can be a very real problem and I haven't come to that yet. I have hired a mix of experienced and inexperienced (college age) folks. I fully expect that people may either burnout or gain great experience and get better jobs, but that's always a risk. It could happen to me too!
Barry
Barry Caplin People's Choice TV Network Operations Center Manager work: bcaplin@pchoicetv.com home: bc@mtiweb.com PGP fingerprint = 9F E9 5C B6 A2 AD 85 AE 1E 4D 11 0B C2 85 60 93 finger bc@mtiweb.com for PGP public key
I worked in the Secretary's Situation Room/Operations Center (open 24 x 7) at the US Department of State from 1983-1984. Teams rotated first two days 8am - 4pm (briefed by outgoing team 7:15am - 8am), next two days 4pm - midnight (brief 3:15pm - 4pm), and final two days midnight - 8am (brief 11:15pm - midnight), and had off the next 56 hours until their rotation started anew. The State Department Bureau of Intelligence and Research Operations Center had to have an alternate schedule for security reasons, i.e. to avoid unauthorized collaboration between the Secretary's Ops staff and INR's Ops staff. They worked two-week shifts of the same hours noted above, i.e., first two weeks 8am - 4pm, next two weeks 4pm - midnight, and final two weeks midnight - 8am. I can't recall how much time the INR teams had off after their sixth week. Both schedules were brutal on families/friends, sleeping patterns and digestive systems, but great for commuting into and out of the District. Debbie Kelly ============================================ Digital Counsel Associates 632 Broadway 10th Floor New York New York 10012 p 212-271-0251 f 212-271-0363 Providing business-driven legal solutions to technology and digital media companies. ============================================ Fight Spam! Join CAUCE (Coalition Against Unsolicited Commercial Email) at http://www.cauce.org and tell your Congresspeople how Spam mailings hurt you. Help protect genuine Internet commerce: Outlaw UCE Spamming.
We're in the process of developing a 24x7 NOC, and can agree upon about everything except scheduling.
- M-F 9-hour shifts, Sa-Su 2x12-hour shifts - Downside is the Sa-Su people are just "extras". This is basically what I went with. To integrate in the weekend people I have them do some half shifts during the week.
yes it's true.. if we get the "right" schedule people will actually be more productive...
participants (14)
-
Aaron -
Barry Caplin -
Debra Kelly -
Doug McClure -
J.D. Falk -
Jamie Rishaw -
Jared Mauch -
Kent W. England -
Paul Ferguson -
Perry E. Metzger -
Robert Perfetti -
Rostand Reymund Lim Abear -
Scott Portmaster List -
Steve Miller