EXECUTIVE SUMMARY ----------------- At 1830 EDT on 8/23/2000, the Network Solutions Registry was notified of a problem with four root name servers--b.root-servers.net, g.root-servers.net, j.root-servers.net and m.root-servers.net. The root zone on each of these servers was missing delegation information (NS records) for the com zone. The NSI Registry took immediate action, working with the operators for each of these root name servers to correct the problem. B.root, g.root and j.root were corrected by 1900 EDT; m.root by 1950 EDT. At no point did NSI make any changes to the root zone. The absence of the com zone delegation information on these name servers resulted from BIND name server behavior that is described in greater detail below. NSI acted immediately to resolve the problem, implemented immediate procedures to avoid recurrence in the short term, and has an action plan to implement a more robust longer-term solution. ROOT CAUSE ---------- If a BIND name server is authoritative for both a child zone and its parent zone, and the name server is restarted or reloaded with a missing zone database file for the child zone, the name server removes the child zone's delegation information from the in-memory copy of the parent zone. As part of NSI's zone generation process, there is a small time interval when the com zone database is not present on the NSI name server in question (during the backup of the old database and copying of the new database). On Wednesday, the name server was manually restarted during this interval; therefore, the name server removed the com zone delegation information from the root zone held in memory. NSI had never encountered this BIND behavior before, nor were several BIND experts we contacted aware of its existence. Subsequent investigation by NSI has revealed that BIND versions 8.1.2 and 8.2.2-P5 exhibit this behavior, but BIND 9.0.0rc4 does not. Nominum has indicated it will pursue a patch to BIND 8. CORRECTIVE ACTIONS ------------------ To restore service, NSI immediately reinstated the com zone database file on our server, increased the root zone's serial number and restarted the name server. B.root, g.root, and j.root immediately loaded the new root zone and were correctly issuing referrals to the com zone within minutes. M.root (in Japan) took nearly an hour to load the new root zone. NSI is modifying the current zone generation process to eliminate the existing small interval during which the com zone database file is not present on this nameserver. Until then, NSI is manually querying the root zone to ensure no delegations have been automatically dropped. -------------------------------------- Brad Verd gTLD Operations Manager Network Solutions Registry Email - bverd@netsol.com ---------------------------------------