On 12/31/18 3:31 PM, Keith Medcalf wrote:
It could have been worse: https://www.cio.com.au/article/65115/all_systems_down/
"Make network changes only between 2am and 5am on weekends."
Wow. Just wow. I suppose the IT types are considerably different than Process Operations. Our rule is to only make changes scheduled at 09:00 (or no later than will permit a complete backout and restore by 15:00) Local Time on "Full Staff" day that is not immediately preceded or followed by a reduced staff day, holiday, or weekend-day.
We had fairly extensive discussion on this list decades ago. Deploy non-emergency changes early Tuesday morning local time, a few hours before regular working hours. Agreed about "not immediately preceded or followed by a reduced staff day, holiday, or weekend-day." Because you won't know it's really working until the actual users have no reported problems. Tuesdays give a couple of working days to ensure that there were no hidden ill affects. Weekends are terrible for that reason. As are Mondays and Fridays, because actual users aren't around in overseas locations. Even those of us who have operated regional ISPs still can effect the world. And those of us with multiple datacenters world-wide have to ensure that changes in one place don't affect the others.... As I remember, at the time this NANOG wisdom propagated to legal blogs. Because so much of what network operators do has legal implications.