On Mon, Feb 01, 2010 at 09:21:52PM -0500, Chadwick Sorrell wrote:
Hello NANOG,
Long time listener, first time caller. [snip] What measures have you taken to mitigate human mistakes?
Have they been successful?
Any other comments on the subject would be appreciated, we would like to come to our next meeting armed and dangerous.
Define your processes well, have management sign off so no blame game and people realize they are all on the same side Use peer review. Don't start automating until you have a working system, and then get the humans out of the repetitive bits. Don't build monolithic systems. Test your automation well. Be sure have the symmetric *de-provisioning* to any provisioning else you will be relying on humans to clean out the cruft instead of addressing the problem. Extend accountability throughout the organization - replace commission- minded sales folks with relationship-minded account management. Always have OoB. Require vendors to be *useful* under OoB conditions, at least to your more advanced employees. Expect errors in the system and in execution; develop ways to check for them and be prepared to modify methods, procedures and tools without multiple years and inter-departmental bureaucracy. Change and errors happen, so capitalize on to those events to improve you service and systems rather than emphasizing punishment. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE