[ On , January 24, 2001 at 14:31:20 ( -0800), Sean Donelan wrote: ]

Subject: Monitoring highly redundant operations

But he does raise an interesting problem. How do you know if your highly redudant, diverse, etc system has a problem. With an ordinary system its easy. It stops working. In a highly redudant system you can start losing critical components, but not be able to tell if your operation is in fact seriously compromised, because it continues to "work."

The real problem is that the most critical part of the puzzle has _not_ been made "highly redundant" in this case. If at least one of your registered authoritative DNS servers are not responding from the point of view of any _and_ every user on the Internet, your hosts (MX records, etc.) don't exist for those people and their e-mail to you may well bounce and they will not view your web pages. The only way to ensure that your DNS is highly redundant and working is to ensure that you've got maximum possible dispersion of _registered_ authoritative servers throughout the network geography, just like the root and TLD servers are widely distributed. Note this is just as important (if not more so!) for any delegated sub-domains in your zone too, and equally important for any related zones (eg. passport.com in this case). The only really effective way to measure the effectiveness of your nameserver dispersion is to make it terribly easy for anyone anywhere to report any problems they percieve to you via as many optional channels as possible -- you can't be everywhere at once, but if you make it easy for people to send you information out-of-band then you'll get lots of early warning when various chunks of the Internet can't see your nameservers and/or your other hosts. Now if the majority of DNS cache server operators don't get too paranoid you could try to set up a mesh of equally widely dispersed monitoring systems that cross-check the availability of test records from your zone by querying any number of regional and remote cache servers. You'd make the TTL of these test records the minimum recommended by major nameserver software vendors (300 seconds?) and then query the whole group every TTL+N seconds. Obviously you're probably going to have to report your results out-of-band, and/or have independent people at each monitoring site who are responsible for investigating problems immediately and doing what they can locally to resolve as them. -- Greg A. Woods +1 416 218-0098 VE3TCP <gwoods@acm.org> <robohack!woods> Planix, Inc. <woods@planix.com>; Secrets of the Weird <woods@weird.com>

Sean Donelan <sean@donelan.com> observed,

But he does raise an interesting problem. How do you know if your highly redudant, diverse, etc system has a problem. With an ordinary system its easy. It stops working. In a highly redudant system you can start losing critical components, but not be able to tell if your operation is in fact seriously compromised, because it continues to "work."

I suspect answers here aren't going to be found in traditional engineering, but more in a discipline that deals with extremely complex systems where a full failure may be irretrievable. I'm thinking of clinical medicine. The initial problem there indeed may be subtle. I have a substantial amount of medical experience, but it easily was 2-3 hours before I recognized, in myself, early symptoms of a cardiac problem. It seemed so much like indigestion, and then a pulled muscle. I remember relaxing, and then recognizing a chain of minor events...sweating...mild but persistent left arm pain radiating into the chest...shortness of breath...and then a big OH SH*T. My first point is having what physicians call a "high index of suspicion" when seeing a combination of minor symptoms. I suspect that we need to be looking for patterns of network symptoms that are sensitive (i.e., high chance of being positive when there is a problem) but not necessarily selective (i.e., low probability of false positives). Once the index of suspicion is triggered, the next thing to look for is not necessarily direct indication of a problem, but a more selective surrogate marker: objective criteria, especially when analyzed as trends, point in the direction of an impending failure. In emergency medicine, the EKG often isn't as informative as TV drama would suggest. A constantly improving area, however, has been measurement, especially successive measurements, of blood chemicals that indicate cardiac tissue is being damaged or destroyed. Early in the use of cardiac-related enzymes, it was a matter of considering several nonspecific factors in combination. SGOT, CPK and LDH are all enzymes that will elevate with tissue damage. The problem is that any one can be elevated by problems in different areas: liver and heart, heart and skeletal muscle, etc. You need to look for elevations in a couple of areas that are associated with the heart, AND look for normal values for other tests that rule out liver disease, etc. The biochemical techniques have constantly improved, but you still need to look at several factors. The second-phase analogy for networking could be more frequent polling and trending, or relatively benign tests such as traceroutes, etc. Only after there is a clear clinical problem, or several pieces of laboratory evidence, does a physician jump to more invasive tests, or begin aggressive treatment on suspicion. In like manner, you wouldn't do a processor-intensive trace on a router, or do a possibly disruptive switch to backup links, unless you had reasonable confidence that there was a problem.