A follow-up thought on redundancy issues. - paul [snip]
Date: Mon, 21 Oct 1996 12:54:05 -0700 (PDT) From: risks@csl.sri.com Subject: RISKS DIGEST 18.54
[snip]
Date: Fri, 18 Oct 96 11:03 EST From: William Hugh Murray <0003158580@mcimail.com> Subject: Re: Rats take down Stanford ... (RISKS-18.53)
PGN's request for redundancy brings to mind the story of the infrastructure computer center in Trumbull, Connecticut. It is an old story but bears repeating.
Seems that a squirrel got into a transformer and brought down the external power supply. The UPS kicked in, engine generators came on line, and the center operated in this mode for about an hour and a half. At the end of that time the external power was restored. The external power, the UPS, and the engine generators went inot a deadly embrace. The whole thing came down and would not come back up.
I take two lessons from this. First, redundancy adds some complexity and a lot of redundancy adds a lot of complexity. At some point the redundancy begins to introduce failure modes and failure events that would not have exited in its absence. There is an upper bound to such redundancy.
Second, test redundant systems through to resumption of normal operations. In this case, the operators had tested to ensure that the redundant systems would come online in the event of a failure of the primary system. They had not tested to see what would happen when the primary system was restored to normal operation.
Who would have even thought about it? I confess that I would not have.
William Hugh Murray, New Canaan, Connecticut
[snip]