Hi. Following is an excerpt from the May report which will be included in this month's Internet Monthly Report. The missing text was already sent to this list as RS/960 deployment notes each week during the backbone hardware upgrades. Mark +----------------------------------------------------------------+ ANSNET/NSFNET Backbone Engineering Report May 1992 Jordan Becker, ANS Mark Knopper, Merit becker@ans.net mak@merit.edu T3 Network Status ================= The major event for May was the successful completion of the deployment of new "RS/960" DS3 interface cards on the entire T3 backbone. The five week deployment went very smoothly, and the new cards have increased the reliability and performance of the network. Cutovers of traffic from the T1 to the T3 backbones were suspended and delayed until June because of the deployment activities. Other activities of interest include new build and routing daemon software on the T3 network nodes. Two of the three links to CA*Net (Toronto to Ithaca, and Montreal to Princeton) were upgraded from fractional-T1 rate to full T1 rate this month. The T1 backbone is stable now that it is carrying less than 50% of the total NSFNET traffic, though the RCP nodes continue to be closely monitored since they are still carrying the full routing advertisements. Full traffic statistics for the T3 net are not available because the initial software release for the RS/960 support on the T3 backbone did not support returning the interface counters from the SNMP MIB. This will be corrected for the month of June. Also, the net to net matrix statistics are not available. A sampling technique is being developed which will allow this data to be collected once again. Routing Daemon Changes ====================== The rcp_routed program for the T3 network was upgraded to a new version containing several improvements. These included a change to allow Cisco peer routers to accept a full routing table (~4000 networks) from the ENSS. There is now an option for rcp_routed to add a time delay when the peer is a slower processor. All Ciscos in the backbone now have a full routing table rather than using default. Another change was made to fix a problem that caused an internal-BGP connection to be reset. Several bug fixes that causes crashes or daemon restarts were fixed. SNMP Daemon Changes =================== The RS/960 software build does not include the ability to obtain ethernet interface statistics. This will be corrected the week of June 8. As of June 3, traffic statistics were available for the ENSS T3 interfaces. These figures can be used to closely approximate traffic counts for the ENSS nodes. It should be noted that both the if[In|Out]McastPkts and if[In|Out]UcastPkts variables should be queried and summed for both T3 and ethernet interfaces, for those who are polling the backbone nodes. RS/960 Deployment Chronology ============================ Following is a summary of the 5 week deployment activities for May which completed installation of the RS/960 DS3 hardware adaptors in all T3 network nodes. ... Step 5: May 22-23 The fifth, and final, stage of the Phase 3 deployment was commenced at 23:00 hours Friday, May 22 and completed by turning on all traffic through a re-built, RS960 based backbone at 09:38 Saturday morning. The problems encountered were relatively minor and were dealt with rather quickly, enabling return to full operation ahead of schedule. The resulting stability of the T3 network has been excellent since this final cutover. We are very grateful to all of the regional-techs that assisted and cooperated with us during the 5 week deployment. Week 5 Problems =============== At Washington POP (CNSS59) after the rebuild, the machine failed to reboot properly in "SECURE" mode. The service switch was found to be defective and was replaced. CNSS57, targeted to be a spare, was cannibalized for the spare part. At Washington POP (CNSS58) the RS960 T3 card in slot 4 failed to load its microcode and the card was replaced. At Washington POP (CNSS56) the node did not recognize an RS960 T3 card (no POSID). This indicates that card was not identifying itself to the system and the card was replaced. At College Park (ENSS136) the RS960 T3 card came up in NONAP mode (would not talk to the system) and was replaced. At Greensboro POP (CNSS73) a corrupted ODM database was suspected and the node system software was reloaded from a prepared tape. Deborah Matties-Sharp and Kraig Owen, in anticipation, had all sites prepared with bootable back-up tapes and their foresight paid off. Also at Greensboro, the communication cable from the DSU servicing Georgia Tech. (ENSS138) was found to be defective (this cable previously was operational) and was replaced by removing an identical cable from CNSS75 which no longer needed a DSU. The problem was difficult to isolate because the DSU would respond to query of the serial number, but would respond with garbaled messages to other queries. It was noted that at Greensboro LBO's had been set at "LONG" (or HI) but after the DSU change the T3 circuit to Georgia Tech had to be set "SHORT" (or LOW). We are investigating this further. Lastly, at New York POP the cable from the RS6000 serial port#S2 for the CNSS35 modem was replaced as planned. Results ======= It took 27 minutes to have all routes come up from the first site (Hartford POP) to the last site Georgia Tech. (ENSS138). We have installed new SNMP agent (and sub-agent) software across the T3 backbone to support the new variables collected on the RS960 T3 cards SNMP access for some variables on the T960 ethernet/T1 cards are temporarily in-accessible. We expect to have this supported shortly. SNMP client programs may have to be re-tested to insure they are collecting valid data. Because there are no more hybrid links (e.g. Hawthorne T3<->RS960 T3), the T3 network link metrics have been reset to normal values in order to split traffic equally across equal hop paths. A party was held at Mark Knopper's house on May 23 to commemorate the complete deployment of the RS/960 cards. As part of the festivities, an authentic Hawthorne card was buried amid a not-so-solemn ceremony.