If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation? thoughts, experience? Mike
On 15 Apr 2011, at 14:14, harbor235 wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
Although more geared up for on-call, http://blog.hinterlands.org/2010/07/running-an-oncall-rota/ is a very useful resource. Peter
I've had it done in places where I work where you'll have 3 rotations working 12 hour shifts. In a 2 week pay period they get their 80 hours in a blend 36 one week and 44 the next. It gives some nice consecutive days off time which also doubles as a retention tool for some employees. You might have to get creative to have all the days work out but it can be done. On Fri, Apr 15, 2011 at 9:14 AM, harbor235 <harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
Mike
-- ===================================== Charles L. Mills Email: w3yni1@gmail.com ===================================== Need server hosting, DR or colocation services? See me!
On 04/15/2011 09:25 AM, Charles Mills wrote:
I've had it done in places where I work where you'll have 3 rotations working 12 hour shifts.
In a 2 week pay period they get their 80 hours in a blend 36 one week and 44 the next. It gives some nice consecutive days off time which also doubles as a retention tool for some employees. You might have to get creative to have all the days work out but it can be done.
On Fri, Apr 15, 2011 at 9:14 AM, harbor235<harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
Mike
I worked once in a production plant. They had 4 crews a,b,c,d and worked 6 days on 2 off. It was rotating shifts. As an example crew A worked 6 days 8-4 was off 2 days then started the next shift on 4-midnight, then midnight to 8. So at any one day 3 of the 4 crews were working and the other crew was off. -- Stephen Clark *NetWolves* Sr. Software Engineer III Phone: 813-579-3200 Fax: 813-882-0209 Email: steve.clark@netwolves.com http://www.netwolves.com
On Fri, Apr 15, 2011 at 2:14 PM, harbor235 <harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
Well, if you feel like being "nice" to your employees, or want to stop them from making mistakes a few months/years in to shift work, (or if you're having to set something up abroad), the Working Time Directive can be a useful guideline. (Full details at http://en.wikipedia.org/wiki/Working_Time_Directive and http://www.businesslink.gov.uk/bdotg/action/layer?topicId=1073858926 if you're board, ignore the silly bits) But basically, in general, workers aged 18 and over are entitled to: - work no more than six days out of every seven, or 12 out of every 14 - take a 20-minute break if their shift lasts for more than six hours - work a maximum 48-hour average week And in general, night workers: - should not work more than an average of eight hours in a 24-hour period, averaged over a reference period of 17 weeks If you're an employer, be glad you're in North America :-) HTH, Alex
Have you taken into account number of alarms per hour, inbound call volume for repairs, and how much repair is done at the first tier level? Bare minimum staffing in a busy environment? Randy Sanders -----Original Message----- From: Alex Brooks [mailto:askoorb+nanog@gmail.com] Sent: Friday, April 15, 2011 9:28 AM To: nanog Subject: Re: 365x24x7 On Fri, Apr 15, 2011 at 2:14 PM, harbor235 <harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
Well, if you feel like being "nice" to your employees, or want to stop them from making mistakes a few months/years in to shift work, (or if you're having to set something up abroad), the Working Time Directive can be a useful guideline. (Full details at http://en.wikipedia.org/wiki/Working_Time_Directive and http://www.businesslink.gov.uk/bdotg/action/layer?topicId=1073858926 if you're board, ignore the silly bits) But basically, in general, workers aged 18 and over are entitled to: - work no more than six days out of every seven, or 12 out of every 14 - take a 20-minute break if their shift lasts for more than six hours - work a maximum 48-hour average week And in general, night workers: - should not work more than an average of eight hours in a 24-hour period, averaged over a reference period of 17 weeks If you're an employer, be glad you're in North America :-) HTH, Alex
harbor235 <harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations?
Hours in the working year = 8 * 5 * 48 = 1920 Hours in the calendar year = 24 * 7 * 52 = 8736 Ratio = 4.55 Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Rockall, Malin, Hebrides: South 5 to 7, occasionally gale 8 at first in Rockall and Malin, veering west or northwest 4 or 5, then backing southwest 5 or 6 later. Rough or very rough. Occasional rain. Moderate or good, occasionally poor.
Guys, Thanx alot, there is some great stuff here, also some stuff I had not thought of. thanx again, Mike On Fri, Apr 15, 2011 at 10:36 AM, Tony Finch <dot@dotat.at> wrote:
harbor235 <harbor235@gmail.com> wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations?
Hours in the working year = 8 * 5 * 48 = 1920 Hours in the calendar year = 24 * 7 * 52 = 8736 Ratio = 4.55
Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Rockall, Malin, Hebrides: South 5 to 7, occasionally gale 8 at first in Rockall and Malin, veering west or northwest 4 or 5, then backing southwest 5 or 6 later. Rough or very rough. Occasional rain. Moderate or good, occasionally poor.
On 4/15/2011 3:14 AM, harbor235 wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
Mike
For what it's worth, was part of a datacenter operations department that had a 24x7 team. 4 shifts, 4 staff on each shift (1 was supervisor who did same work as the rest, 1 'point of contact' who stayed in the office). 4 days on, 4 days off, 12 hour shifts, 8-8. Shift teams would alternate between day and night (so 4 day, 4 off, 4 night, 4 off, repeat ad infinitum). During the day that was bolstered by 6 day-staff, Monday to Friday, who would have a staggered start through the day (IIRC 2 start at 8, 2 at 9, 2 at 11)
For what it's worth, was part of a datacenter operations department that had a 24x7 team. 4 shifts, 4 staff on each shift (1 was supervisor who did same work as the rest, 1 'point of contact' who stayed in the office). 4 days on, 4 days off, 12 hour shifts, 8-8. Shift teams would alternate between day and night (so 4 day, 4 off, 4 night, 4 off, repeat ad infinitum). During the day that was bolstered by 6 day-staff, Monday to Friday, who would have a staggered start through the day (IIRC 2 start at 8, 2 at 9, 2 at 11)
And also for what it's worth, my understanding of body circadian rhythm research suggests that "every 8 days" is *WAY* too short a period to be flipping people's shifts from day to night; every 6 months is more reasonable. Cheers, -- jra
Date: Fri, 15 Apr 2011 09:14:05 -0400 Subject: 365x24x7 From: harbor235 <harbor235@gmail.com> To: NANOG list <nanog@nanog.org>
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
You haven't done the math right. You're not even *close* to 'real world' requirements. 3 people only provide 24-hour coverage for *five* days/week. *before* considering any the "stuff" mentioned below. 7x24 requires _4-1/5_ persons per position, assuming 40-hr work week, _before_ allowing for vacations, etc.. Scheduled vacation time adds another 5-8% to the manpower requirement. Don't forget "holidays", 'sick days', and 'personal time'. that's another 5-10%, or so. Do you intend to allow any participation in 'professional' activities, conferences, etc. and/or any on-going training? Another 5% easily. Now, don't foreget about 'in-house' "overhead" actitivites. e.g. 'staff meetings'. 3-5% (i.e., 1-2 hours/week) is a not-unreasonable estimate. These things push the total requirment to just over 5 people per position, but 'shift scheduling' is a *bitch*. Lastly, you have to consider meal-breaks and such _during_ the shift. This means 'n+1' people per shift, to ensure 'n' on duty at all times. Take your minimum required NOC positions, add one person (for the n+1 completeness), then multiply by 5 and you've got a realistic estimate of the *minimum* manpower requirement. Note: You'll probably want 'above minimum' staffing on 1st shift (and possibly some on 2nd) to handle 'routine'/'non-essential' activities.
On 4/15/2011 6:14 AM, harbor235 wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
What is the work distribution? Do you have stable patterns of more-vs-less activity? If things are busy daytime and quiet nighttime, obviously you need different-sized teams. Variable scheduling of staff is often deemed more fair, but I think it makes things less stable. People are constantly having to change their life. A simple model has 3 teams over weekdays/nights, leaving you with weekends, holidays and vacations. I was a part-time operator during college. Full-timer shifts were 3 people; maybe 2 for graveyard. There were 3-5 of us covering things for that added time. But, then, the major operations were purely daytime, during the week. Graveyard shift was quiet enough that we surreptitiously bought a cot... d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net
Variable scheduling of staff is often deemed more fair, but I think it makes things less stable. People are constantly having to change their life.
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span. Full-time night work isn't great, but rotating work is even worse. Apes are generally diurnal, not nocturnal or crepuscular. Shuffling who has to work which days is annoying enough. ---- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span. Full-time night work isn't great, but rotating work is even worse. Apes are generally diurnal, not nocturnal or crepuscular. Shuffling who has to work which days is annoying enough.
I spent several months working in a place with rotating shifts. One week 10pm to 7am, then the next week 2pm to 10pm then the week after 7am to 2pm and then repeat. I never understood why they were different lengths either. It was pretty grim and I'd much prefer to have had shifts change ever few months. Some people claimed they'd have preferred it if we'd changed to the _following_ shift rater than the preceding shift each week but never having tried that I don't know how it would be.
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
One of the places I worked had the following pattern. It was horrible 2 days/shifts of 6am till 6pm 2 days/shifts of 6pm till 6am 4 days off Wayne
I was offered a similar role… but more painful (Imho) 4 days 8am till 8pm 4 days off 4 days 8pm till 8am 4 days off Rinse and repeat. ...Skeeve -- Skeeve Stevens, CEO - eintellego Pty Ltd - The Networking Specialists skeeve@eintellego.net ; www.eintellego.net Phone: 1300 753 383 ; Fax: (+612) 8572 9954 Cell +61 (0)414 753 383 ; skype://skeeve facebook.com/eintellego or eintellego@facebook.com twitter.com/networkceoau ; www.linkedin.com/in/skeeve PO Box 7726, Baulkham Hills, NSW 1755 Australia -- eintellego - The Experts that the Experts call - Juniper - HP Networking - Cisco - Brocade - Arista - Allied Telesis On 17/04/11 9:34 PM, "Wayne Lee" <linkconnect@googlemail.com<mailto:linkconnect@googlemail.com>> wrote: Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span. One of the places I worked had the following pattern. It was horrible 2 days/shifts of 6am till 6pm 2 days/shifts of 6pm till 6am 4 days off Wayne
Having done this for quite a few years my advice is that once you get past the basic arithmetic of people-hour-equivalents etc what you need is a middle manager who is a good "horse trader" because it quickly becomes a market of "I can do grave shift Tuesday if you'll take my Saturday AM, I've got a wedding I have to be at, XXX says he'll take your Friday night if you take the Saturday but if you still want Sunday off you're going to have to find someone to cover that and Monday morning unless..." And that manager has to be responsible so such shuffling really ends up with necessary coverage because people make mistakes, often in their favor (OOPS!), and that can get complicated. Regular-hours life goes on despite peoples' shifts and they're forever having to be at jury duty, doctor's appointments, social engagements, govt offices, religion, dog kennels, kids' school meetings, etc etc etc which are fixed in the assumption that people work roughly 9AM-5PM M-F. Yeah some of that is actually easier for people who work odd hours and even the attraction of odd hour work but a lot of it isn't and comes up only once in a while per employee which means, for the manager, once or twice a week. Another practicality is policies about people "hanging out" during non-work hours because you'll find overnight people are night people even on their days off and often have nothing better to do than come in and use your higher bandwidth etc, or just get out of the house, they'll become friendly with overnight co-workers, which might be fine by itself until someone brings in a couple of six-packs because hey they're not working and there won't be a boss around after 1AM and...again, maybe not a problem until they decide to share their fun a little too much with people on duty who are nearby... The other problem is food, overnight people get hungry and depending on location and facilities for bringing and preparing food opportunities for a 3AM lunch can be limited in many locations and if you don't solve it somewhat employees may come up with suboptimal solutions like deciding it's only fair that they get an extra 20 minutes travel each way for an hour lunch, or cooking mess, etc. And of course who to call when something goes very wrong at 4AM or they'll call you every time, for some value of "you", or worse they'll call the police (for example) when you prefer they hadn't (or the police or other LEO will call them)...you need some procedures and training to anticipate such things, power problems, building management calling at 5AM that they need whoever is on to let some people in to knock down all the walls, they're waiting outside... Finally, a weird thing? If there's overnight telephone support people (some of whom are customers!) will find it, sometimes lonely people who want to hear a human voice at 4AM and sometimes completely crazy people who, well, want to hear a human voice at 4AM, some of them are very good at manipulating whoever will answer the phone, "our little secret please don't hang up please don't hang up really please!", you need policies and training if that's a possibility. Gak, I could tell stories... -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Dial-Up: US, PR, Canada Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
Some people claimed they'd have preferred it if we'd changed to the _following_ shift rater than the preceding shift each week but never having tried that I don't know how it would be.
I've read stuff that confirms that changing to a later shift is much easier than changing to an earlier one. It certainly matches my experience that the jet lag flying to Europe, where I have to get up six hours earlier, is much worse than flying back. It also makes the obvious point that fewer shift changes are easier on the employees than more. R's, John
John Levine wrote:
I've read stuff that confirms that changing to a later shift is much easier than changing to an earlier one. It certainly matches my experience that the jet lag flying to Europe, where I have to get up six hours earlier, is much worse than flying back.
Last time I went to the US I stayed on something pretty close to my usual timetable while there. So I went to bed at about 6pm US time and got up in the very early morning US time. Fortunately, this was fine for the 4-day event I was attending. I think I will do this in future if I can.
It also makes the obvious point that fewer shift changes are easier on the employees than more.
One of my sisters-in-law is a nurse and her shifts seem to be all over the place from day to day. I don't understand how she copes.
Having run 24/7 NOC, customer care and tier 3 engineering/dev support, for 20 years, my two cents are: 1) You need to rotate shifts and have overlap between shifts for training and communication purposes 2) Always rotate forward, due sleep cycles 3) If you want to retain staff and not burn them out, do not rotate more than one a month, I've tried from 2 week to 12 week rotations. Longer rotations mean more staff, when you try various shift schedules you will see why, but they work best. You make your cost on the extra staff by lowering turnover which is expensive to deal with. 4) Take your staff opinions in on schedule but you make the call, someone will always dislike the schedule no matter how hard you try,....... 5) make sure you support shift swapping within reason, so people can deal with personal schedule conflicts Good luck! Michael Young M:647-289-1220 On 2011-04-17, at 12:18, "John Levine" <johnl@iecc.com> wrote:
Some people claimed they'd have preferred it if we'd changed to the _following_ shift rater than the preceding shift each week but never having tried that I don't know how it would be.
I've read stuff that confirms that changing to a later shift is much easier than changing to an earlier one. It certainly matches my experience that the jet lag flying to Europe, where I have to get up six hours earlier, is much worse than flying back.
It also makes the obvious point that fewer shift changes are easier on the employees than more.
R's, John
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
I Fully agree. I think it may pay off to search for people who suffer "Delayed sleep phase syndrome" to do night shift. They'll be happy and you'll actually have someone who is more awake and alert than the average person at that time of day. http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome I think the IT world has a more than average incidence of people with this particular syndrome, at least in my experience. Of course in practice you would want to word your vacancy in such a way it doesn't sound silly. But I think it could be worth it to put an emphasis on it. Greetings, Jeroen -- http://goldmark.org/jeff/stupid-disclaimers/ http://linuxmafia.com/~rick/faq/plural-of-virus.html
On Thu, Apr 21, 2011 at 9:02 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
I Fully agree.
I think it may pay off to search for people who suffer "Delayed sleep phase syndrome" to do night shift. They'll be happy and you'll actually have someone who is more awake and alert than the average person at that time of day.
http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome
I think the IT world has a more than average incidence of people with this particular syndrome, at least in my experience.
Of course in practice you would want to word your vacancy in such a way it doesn't sound silly. But I think it could be worth it to put an emphasis on it.
Greetings, Jeroen
-- http://goldmark.org/jeff/stupid-disclaimers/ http://linuxmafia.com/~rick/faq/plural-of-virus.html
I'd just go with "people who really enjoy energy drinks." -- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
On Fri, April 22, 2011 1:38 pm, Jeffrey Lyon wrote:
On Thu, Apr 21, 2011 at 9:02 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
I Fully agree.
I think it may pay off to search for people who suffer "Delayed sleep phase syndrome" to do night shift. They'll be happy and you'll actually have someone who is more awake and alert than the average person at that time of day.
http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome
I think the IT world has a more than average incidence of people with this particular syndrome, at least in my experience.
Of course in practice you would want to word your vacancy in such a way it doesn't sound silly. But I think it could be worth it to put an emphasis on it.
I'd just go with "people who really enjoy energy drinks."
Many of you folks actually worked Nightshift for any duration? Most folks I know working in shifts are either IT folks or Emergency Services folks. Both groups recognise the value of actually having conventional working hours, at least for part of the time. Folks on permanent night-shift risk becoming isolated from a good chunk of society and I would expect to see some churn over time. One watch centre I worked with used to run a 3 week rotation of days, 'lates' and 'overnights' which averaged out to 40hrs/week during the course of the year. Another used something similar to the 2days-2nights-4off model I mentioned previously. The remainder split the overnight work into weeknights and weekends, and tended to attract students for the weekend shifts. Mark.
On Fri, Apr 22, 2011 at 02:48:16PM +1200, Mark Foster wrote:
On Fri, April 22, 2011 1:38 pm, Jeffrey Lyon wrote:
On Thu, Apr 21, 2011 at 9:02 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
I Fully agree.
I think it may pay off to search for people who suffer "Delayed sleep phase syndrome" to do night shift. They'll be happy and you'll actually have someone who is more awake and alert than the average person at that time of day.
http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome
I think the IT world has a more than average incidence of people with this particular syndrome, at least in my experience.
Of course in practice you would want to word your vacancy in such a way it doesn't sound silly. But I think it could be worth it to put an emphasis on it.
I'd just go with "people who really enjoy energy drinks."
Many of you folks actually worked Nightshift for any duration? Most folks I know working in shifts are either IT folks or Emergency Services folks. Both groups recognise the value of actually having conventional working hours, at least for part of the time. Folks on permanent night-shift risk becoming isolated from a good chunk of society and I would expect to see some churn over time.
One watch centre I worked with used to run a 3 week rotation of days, 'lates' and 'overnights' which averaged out to 40hrs/week during the course of the year.
Another used something similar to the 2days-2nights-4off model I mentioned previously.
The remainder split the overnight work into weeknights and weekends, and tended to attract students for the weekend shifts.
Mark.
night/early morning by preference for nearly 20 years.... YMMV. /bill
On Thu, Apr 21, 2011 at 10:51 PM, <bmanning@vacation.karoshi.com> wrote:
On Fri, Apr 22, 2011 at 02:48:16PM +1200, Mark Foster wrote:
On Fri, April 22, 2011 1:38 pm, Jeffrey Lyon wrote:
On Thu, Apr 21, 2011 at 9:02 PM, Jeroen van Aart <jeroen@mompl.net> wrote:
Bill Stewart wrote:
Rotating shifts between daytime and nighttime is a horrible thing to do to your workers, both for their health and their attention span.
I Fully agree.
I think it may pay off to search for people who suffer "Delayed sleep phase syndrome" to do night shift. They'll be happy and you'll actually have someone who is more awake and alert than the average person at that time of day.
http://en.wikipedia.org/wiki/Delayed_sleep_phase_syndrome
I think the IT world has a more than average incidence of people with this particular syndrome, at least in my experience.
Of course in practice you would want to word your vacancy in such a way it doesn't sound silly. But I think it could be worth it to put an emphasis on it.
I'd just go with "people who really enjoy energy drinks."
Many of you folks actually worked Nightshift for any duration? Most folks I know working in shifts are either IT folks or Emergency Services folks. Both groups recognise the value of actually having conventional working hours, at least for part of the time. Folks on permanent night-shift risk becoming isolated from a good chunk of society and I would expect to see some churn over time.
One watch centre I worked with used to run a 3 week rotation of days, 'lates' and 'overnights' which averaged out to 40hrs/week during the course of the year.
Another used something similar to the 2days-2nights-4off model I mentioned previously.
The remainder split the overnight work into weeknights and weekends, and tended to attract students for the weekend shifts.
Mark.
night/early morning by preference for nearly 20 years.... YMMV.
/bill
I've worked night shift as a police officer, 8 x 5. Some of those guys were there for 20+ years and no one seemed to have a serious issue with it. -- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications - AS32421 First and Leading in DDoS Protection Solutions
----- Original Message -----
From: "Dave CROCKER" <dhc2@dcrocker.net>
There were 3-5 of us covering things for that added time. But, then, the major operations were purely daytime, during the week. Graveyard shift was quiet enough that we surreptitiously bought a cot...
You didn't work for the FAA, Dave, did you? Cheers, -- jra
On 4/17/2011 8:19 AM, Jay Ashworth wrote:
----- Original Message -----
From: "Dave CROCKER"<dhc2@dcrocker.net>
There were 3-5 of us covering things for that added time. But, then, the major operations were purely daytime, during the week. Graveyard shift was quiet enough that we surreptitiously bought a cot...
You didn't work for the FAA, Dave, did you?
No, or we would have gotten more sleep. d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net
Timely article on the FAA's involvement with sleep schedules: http://www.ajc.com/news/air-traffic-controller-scheduling-913244.html "Union spokesman Doug Church said up to now, 25 percent of the nation's air traffic controllers work what he called a "2-2-1″ schedule, working afternoon to night the first two days, followed by a mandatory minimum of eight hours for rest before starting two morning-to-afternoon shifts, another eight or more hours for sleep, then a final shift starting between 10 p.m. to midnight. "Maybe we need to work in more time for rest," Church said. "You’re forcing yourself to work at a time when the body is used to sleeping." Frank -----Original Message----- From: Dave CROCKER [mailto:dhc2@dcrocker.net] Sent: Sunday, April 17, 2011 1:15 PM To: Jay Ashworth Cc: NANOG Subject: Re: 365x24x7 On 4/17/2011 8:19 AM, Jay Ashworth wrote:
----- Original Message -----
From: "Dave CROCKER"<dhc2@dcrocker.net>
There were 3-5 of us covering things for that added time. But, then, the major operations were purely daytime, during the week. Graveyard shift was quiet enough that we surreptitiously bought a cot...
You didn't work for the FAA, Dave, did you?
No, or we would have gotten more sleep. d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net
On Apr 17, 2011, at 11:47 20PM, Frank Bulk wrote:
Timely article on the FAA's involvement with sleep schedules: http://www.ajc.com/news/air-traffic-controller-scheduling-913244.html "Union spokesman Doug Church said up to now, 25 percent of the nation's air traffic controllers work what he called a "2-2-1″ schedule, working afternoon to night the first two days, followed by a mandatory minimum of eight hours for rest before starting two morning-to-afternoon shifts, another eight or more hours for sleep, then a final shift starting between 10 p.m. to midnight.
"Maybe we need to work in more time for rest," Church said. "You’re forcing yourself to work at a time when the body is used to sleeping."
Also see http://www.google.com/hostednews/ap/article/ALeqM5hstTegGafIYTakRavF4WEEPblz... "People who change shifts every few days are going to have all kinds of problems related to memory and learning, Fishbein said. This kind of schedule especially affects what he called relational memories, which involve the ability to understand how one thing is related to another. ... "Controllers are often scheduled for a week of midnight shifts followed by a week of morning shifts and then a week on swing shifts. This pattern, sleep scientists say, interrupts the body's natural sleep cycles." --Steve Bellovin, https://www.cs.columbia.edu/~smb
My guys work 12 hour shifts. 2 days on, 2 days off, 3 days on, 2 days off, 2 on 3 off. The three days on is always friday-sunday so every other weekend they either have a 3 day weekend or 3 days of work. In a pay period, with 30 minute lunch per shift it comes to 80.5 hours. I keep my guys on the same shifts for consistancy. Aaron Sent via DROID on Verizon Wireless -----Original message----- From: Steven Bellovin <smb@cs.columbia.edu> To: frnkblk@iname.com Cc: NANOG <nanog@nanog.org>, dcrocker@bbiw.net Sent: Mon, Apr 18, 2011 04:12:04 GMT+00:00 Subject: Re: 365x24x7 On Apr 17, 2011, at 11:47 20PM, Frank Bulk wrote:
Timely article on the FAA's involvement with sleep schedules: http://www.ajc.com/news/air-traffic-controller-scheduling-913244.html "Union spokesman Doug Church said up to now, 25 percent of the nation's air traffic controllers work what he called a "2-2-1″ schedule, working afternoon to night the first two days, followed by a mandatory minimum of eight hours for rest before starting two morning-to-afternoon shifts, another eight or more hours for sleep, then a final shift starting between 10 p.m. to midnight.
"Maybe we need to work in more time for rest," Church said. "You’re forcing yourself to work at a time when the body is used to sleeping."
Also see http://www.google.com/hostednews/ap/article/ALeqM5hstTegGafIYTakRavF4WEEPblz... "People who change shifts every few days are going to have all kinds of problems related to memory and learning, Fishbein said. This kind of schedule especially affects what he called relational memories, which involve the ability to understand how one thing is related to another. ... "Controllers are often scheduled for a week of midnight shifts followed by a week of morning shifts and then a week on swing shifts. This pattern, sleep scientists say, interrupts the body's natural sleep cycles." --Steve Bellovin, https://www.cs.columbia.edu/~smb
\On Apr 18, 2011, at 7:59 AM, Aaron Wendel wrote:
My guys work 12 hour shifts. 2 days on, 2 days off, 3 days on, 2 days off, 2 on 3 off. The three days on is always friday-sunday so every other weekend they either have a 3 day weekend or 3 days of work.
In a pay period, with 30 minute lunch per shift it comes to 80.5 hours. I keep my guys on the same shifts for consistancy.
Aaron
My wife is a nurse working second shift, 12-hour shifts, 7p-7a (actually 6:45p to 7:15a to allow for a little overlap). Her hospital has it worked out on a 6-week schedule, with Wednesdays being the new pay week. Nurses there work 3 days a week, for 36 hours. Here's how they do it (these are calendar weeks) Week 1 - Su M T F Sa Su Week 2 - W Week 3 - M T W Week 4 - M T F Sa Su Week 5 - Th Week 6 - M T I know this is also a decades-long struggle in the railroad industry too (My business does a lot of contract work with that industry). In particular locomotive engineers and conductors. 100 percent on-call, maximum 12-hours on duty with 10 off (recently changed from 8). Fatigue is quite critical there too, you don't want an engineer falling asleep pulling a train full of HazMat. For datacenter work, I'd think a schedule like the above would be doable. You end up working every third weekend, and yes, weeks 1 and 4 aren't pleasant, it's followed by a 1-day week so you've got plenty of time to recover. -Andy
The best schedule I ever worked for this was divided into essentially 2 teams: SMTW and WTFS There were 3 shifts on each team, though, that could be load adjusted by creating additional time slots. The nice thing about the SMTW/WTFS structure was that we always had overlap on Wednesday which we would take advantage of for training, maintenance, or other "crew-heavy" things that occasionally needed to get done. Training would take 2 weeks with the A-shfit one week and the B shift the following week (or vice versa). We ran 10-hour shifts (there's a provision for 4 10 hour shifts not involving overtime pay if agreed ahead of time). Shifts ran 9p-8a, 6a-5p, and 11a-10p with the crews staggering their lunches. Owen
In my previous life at a large backbone provider's managed security services SOC/NOC we had the following: Shifts where divided into front and back half with 10 hour days. Front and back half segments of the shift where also split with half working Sun-Wed and half working Mon-Thur alternating. That had them alternating weekend coverage but also alternating 4 day and 2 day 'weekends'. One shift lead did M-F 8 hour days and was primary escalation for their reports during off hours/weekends, with the lead having to fill any emergency schedule holes. This provided shift overlap to cover issue hand offs and had everyone in the office Tue-Thur to cover meetings, training, any large projects, etc. At min the shifts where 9 staff (4 front half, 4 back half, and the team lead). So 27 min staffing. We normally had some additional folks per shift and they would do M-F 8 hours like the lead or fill a gap on weekends and flex the overages during the week during the mid week overlap. We did do the 30 day rotation of changing to the next shift (ie Morning -> Days -> Midnight -> Morning) but once we got beyond staffing with 20 year old single guys it was basically impossible to keep someone very long who was willing to do that as it kills any outside activities like taking classes, kids schedules, etc.) -- --- James M Keller
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
It depends a lot on how you structure your shifts; the problem is getting everyone 40 hours without unnecessary overlap. The TV master control facility in which I'm working presently does it by doing overlapping 10 hour shifts; it takes 10 people to have 2 on-shift at all times. You work 6 hours with one person, and 4 with the other. Your 3 teams estimate is, I suspect, derived from dividing 24 hrs/day by 8/hrs shift... but that doesn't take weekends into account, and you don't necessarily want to have 3 more teams who only work 2 days a week. I don't think there's *any* way to do it that guarantees you'll always have the same people working together; fortunately, I also don't think that's all that necessary in that environment. Cheers, -- jra
On Sun, Apr 17, 2011 at 8:00 AM, Jay Ashworth <jra@baylink.com> wrote:
The TV master control facility in which I'm working presently does it by doing overlapping 10 hour shifts; it takes 10 people to have 2 on-shift at all times. You work 6 hours with one person, and 4 with the other.
My brother-in-law once had a job tending a TV relay station; the shift was drive up the mountain, work 48 hours, drive back, but unless something was broken he only had to read meters every three hours and could nap in between. ---- Thanks; Bill Note that this isn't my regular email account - It's still experimental so far. And Google probably logs and indexes everything you send it.
On 15/04/11 6:14 AM, harbor235 wrote:
If I were going to provide a 365x24x7 NOC, how many teams of personnel do I need to fully cover operations? I assume minimally you need 3 teams to cover the required 24 hr coverage, but there is off time and schedule rotation?
thoughts, experience?
There have been a lot of comments that talk about the operational issues, and about how well it works or doesn't work to rotate schedules. In my experience, the most important thing to do is to decide how you are going to setup the schedules BEFORE you hire. Then advertise for the exact work schedule(s) you need to fill, and pay an appropriate shift differential so that you get people who are happy to work the worst shifts for slightly better pay. People don't like being treated like slaves, expected to just shut up and work a crappy schedule. You will have a much more reliable team if you treat the staff like valued members of your company, don't try to trick them or over work them, or expect them to disrupt their lives over and over as your shifts change. Some people are naturally night-owls - find them and hire THEM for the swing and night shifts. Also, don't forget that your 24x7x365 staff will have to work holidays, and be sure to work in an appropriate holiday bonus pay schedule as well. Otherwise expect to eventually have a bad case of the blue flu someday and end up having to call managers (directors/ VPs, YOU) away from their holiday with their family to keep your NOC staffed. jc
participants (29)
-
Aaron Wendel -
Adam Atkinson -
Alex Brooks -
Andy Ringsmuth -
Barry Shein -
Bill Stewart -
bmanning＠vacation.karoshi.com -
Charles Mills -
Dave CROCKER -
Frank Bulk -
harbor235 -
James M Keller -
Jay Ashworth -
JC Dill -
Jeffrey Lyon -
Jeroen van Aart -
John Levine -
Mark Foster -
Michael Young -
Owen DeLong -
Paul Graydon -
Peter Hicks -
Robert Bonomi -
Sanders, Randall K -
Skeeve Stevens -
Steve Clark -
Steven Bellovin -
Tony Finch -
Wayne Lee