What are the hours of the config window on Tues and Fri mornings? Do you have a rough idea of the typical outage duration (i.e. do you upgrade the boxes serially so a 5 minute restart occurs on each node one after the other or do you hit the big red button and they all go at once or what?). Do you usually start right at the beginning of the window or if nothing tricky is going on do you wait till near the end?
Thanks, Dan
Dan, I'm glad you asked. The current routing config install windows are from 05:00 - 08:00 EST on Tue and Fri mornings. Starting this week, we will be adding an additional window on Thur morning from 05:00 - 08:00 EST. We need the additional window for the next couple of months in order to complete the remaining T3 AS cutovers. We need to send one of two signals to the routing daemon in order to install the new configuration file: kill -USR1 or kill -USR2. A kill -USR1 just re-reads the new configuration file into the rcp_routed. A kill -USR2 is a little more severe and actually briefly stops and restarts the rcp_routed. This results in a brief period of down EGP/BGP sessions and loss of connectivity, and it usually takes about 5 minutes for all backbone routing to re-converge. These kills get done on the RCPs on the T1 backbone, and on the CNSS, DNSS, or ENSS boxes on the T3 backbone. The severity of the kill and whether a kill needs to be done at all depends on the type of change/addition made in the new configuration file for that node. Most sites running EGP will only need a kill -USR1, while most sites using BGP need a kill -USR2, or an equivalent (stopping and restarting the BGP session), to install the new changes. Adding or deleting a peer always takes a kill -USR2 to the node. Adding a new ENSS to the T3 backbone requires that a kill -USR2 be done on each backbone node to enable the new IBGP sessions. A few exceptions: On the T3 backbone, ENSS 134 (Boston) and ENSS 135 (San Diego) have explicitly configured outbound announcements in order to give them the T1 backbone route (129.140) without also sending them all the other routes coming in from the T1 backbone via the interconnects. Installing any new T3 routes always requires a kill -USR2 on E134 and E135 (usually each config day). The active T1/T3 interconnects in Ann Arbor and Houston also almost always need a kill -USR2 or the equivalent with each configuration update. The kills usually take place in a serial fashion but can begin at varying times during the window, depending on the workload, special problems, and when we or the NOC can schedule the time to do the kills. Each window is usually quite unique, except that the interconnects are usually done last. Sometimes we piggy-back the routing configuration installs on top of some other backbone-wide software update that gets scheduled on the same night, to minimize down time. Sometimes all the kills can be accomplished in 15-20 minutes, while other times we literally need the entire window, because we wait for routing to readjust on one node before disrupting routing on the next node. If we are doing kill -USR2s on the entire T3 backbone, we'll usually do all boxes at each POP at the same time. --Steve Widmayer Merit/NSFNET