On 7/27/16, David Hubbard <dhubbard@dino.hostasaurus.com> wrote:
Full automation is planned but does not eliminate the need for the software. Zero human auditing of fully automated processes and data collection are not acceptable to various certifying entities, the relevant auditors, the inevitably involved lawyers, and won’t pick up on bad data, like a bad thermometer or snmp counter that says a CRAC is 65 degrees when it’s really 90. So I’m still going to need a management solution to the issue whether it’s to tell someone to do the work or to tell someone to check the automated work.
You have a ticketing system - right? Create a cron job that creates a ticket to check whatever. Regards, Lee
David
On 7/27/16, 7:19 PM, "Lee" <ler762@gmail.com> wrote:
On 7/27/16, David Hubbard <dhubbard@dino.hostasaurus.com> wrote: > Hi all, curious if anyone has recommendations on software that helps manage > routine duties assigned to operations staff?
Have computers do the routine scut work - not people.
> For example, let’s say we have a P&P that says someone from the netops group > must check that Rancid is successfully backing up all router configs > bi-weekly.
You've got the source code for rancid, so change rancid-run to do something like LOGFILE=$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S`; export LOGFILE change the ) >$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S` 2>&1 to ) >$LOGFILE 2>&1
and then in control_rancid do something like grep "clogin error:" $LOGFILE | sort | uniq -c >$TMP.fail if [ -s $TMP.fail ]; then # got some output, mail the report ...
Do the same type thing for checking on > backup failures, backup internet circuit status, out of band interfaces, etc.
Automate the checks, put the scripts in crontab & mail out an "OhNoes!" or "all clear" msg at the end. At which point you're left with the problem of making sure the managers are looking at the emails & making sure whatever problems are found actually get fixed :)
Regards, Lee