With all due respect and acknowledgment of the tremendous contributions of ISC and you yourself Mr. Hankins, I have to comment that failover in isc-dhcp is broken by design because it requires the amount of handholding and operator thinking in the event of a failure that you explained to us at length is required. Failure needs to be handled automatically and without any intervention at all, otherwise you might as well not have it and I think most network operators would agree.
Note that this method of handling failover is inherent in the original DHCP failover design. See http://tools.ietf.org/id/draft-ietf-dhc-failover-12.txt Specifically, quoting from the above draft, "While this technique works in some domains, having the only server to which a DHCP client can communicate voluntarily shut itself down seems like something worth avoiding. The failover protocol will operate correctly while both servers are unable to communicate, whether they are both running or not. At some point there may be resource contention, and if one of the servers is actually down, then the operator can inform the operational server and the operational server will be able to use all of the failed server's resources." I certainly cannot speak for "most network operators". However, I will note that I have been aware of this behavior of the IDC DHCP server as long as I have been running it in failover mode.
I am certainly not prepared to develop proof of concept code or go the full route of developing such a server myself, however, I belive firmly that a failover implementation in dhcp could be designed as a counterpoint to the current implementation that is reliable, simple, scalable and requiring no special procedures once a 'break' occurs.
And which implements failover protocol in the IETF draft? Steinar Haug, Nethelp consulting, sthaug@nethelp.no