Hi. This will appear in the Internet Monthly Report from ISI. Happy reading. Mark ANSNET/NSFNET Backbone Engineering Report September 1992 Jordan Becker, ANS Mark Knopper, Merit becker@ans.net mak@merit.edu T3 Backbone Status ================== During August, new RS/960 FDDI adapters were installed in 4 production ENSS nodes. Activities are underway in preparation for dismantling of the T1 backbone which is targeted for November. Several steps to be completed prior to dismantling the T1 backbone include support for OSI CLNP transport over the T3 backbone, and the deployment of the redundant backup circuits for the T3 ENSS gateways at each regional network. Further activities in support of the Phase IV upgrade to the T3 backbone are in progress. Backbone Traffic and Routing Statistics ======================================= The total inbound packet count for the T1 network was 3,512,296,181, down 10.0% from August. 334,806,439 of these packets entered from the T3 network. The total inbound packet count for the T3 network was 15,680,453,501 up 20.1% from August. 118,261,088 of these packets entered from the T1 network. The combined total inbound packet count for the T1 and T3 networks (less cross network traffic) was 18,739,682,155 up 13.4% from August. As of September 30, the number of networks configured in the NSFNET Policy Routing Database was 6640 for the T1 backbone, and 5920 for the T3 backbone. Of these, 1292 networks were never announced to the T1 backbone and 1207 were never announced to the T3 backbone. For the T1, the maximum number of networks announced to the backbone during the month (from samples collected every 15 minutes) was 5068; on the T3 the maximum number of announced networks was 4458. Average announced networks on 9/30 were 5034 to T1, and 4428 to T3. New FDDI Interface Adapter for ENSS Nodes ========================================= We have installed new RS960 FDDI adapters for the RS/6000 router on ENSS128 (Palo Alto), ENSS135 (San Diego), ENSS129 (Champaign), and ENSS132 (Pittsburgh). These adapters provide improved reliability and performance. We are making adjustments to the Maximum Transmission Unit (MTU) on most T3 network interfaces to support this deployment of FDDI adapters on selected ENSS nodes. The goal is to maximize performance for applications designed to exploit T3/FDDI bandwidth, while maintaining satisfactory performance for sites that still interconnect to the T3 routers via an ethernet interface. The MTU on all internal T3 interfaces will initially be set to 2900, and will remain at 1500 on all ethernet interfaces. The reason for this is to minimize packet fragmentation for traffic that is transiting the T3 backbone via FDDI interfaces on each end, as well as for traffic that will transit an FDDI on one end and an ethernet interface on the other. We are actively developing plans for subsequent FDDI deployments on additional ENSS nodes. Phase-IV T3 Network Upgrade Status ================================== The upgrades to the T3 backbone discussed in the August engineering report have started according to schedule and will complete prior to the dismantling of the T1 backbone. Activities completed so far include: 1) T3 ENSS FDDI interface upgrades to the new RS/960 card. This has been completed at ENSS128, ENSS135, ENSS132, and ENSS129. 2) T3 DSU PROM upgrades. We are in the process of installing new T3 DSU firmware in all network DSUs. This supports additional SNMP function and fixes a few non-severe bugs. The testing of the PROMs were completed in September, and the installation of these PROMs will be completed by late October. 3) The CNSS routers in the Washington D.C. area (CNSS56, CNSS57, CNSS58, CNSS59, CNSS60) were successfully re-located to an MCI POP in downtown Washington D.C. on 9/12 for closer proximity to several ENSS locations. 4) The installation of three new CNSS nodes (CNSS104, CNSS105, CNSS107, CNSS108) in the Atlanta MCI POP was completed on 9/26. The T3 circuit between Greensboro (CNSS73) and Georgia Tech (ENSS138) was rehomed to the to Atlanta POP (CNSS105). T3 Link Metrics Changed ======================= The link metrics on the interior T3 backbone links have been adjusted to approximate the latency on the physical T3 links, as was discussed in the August '92 engineering report. The new metrics result in more east<->west traffic traversing the northern T3 route (New York-Cleveland-Chicago-San Francisco) than the southern route (New York-Washington-Greensboro-Atlanta-Houston-Los Angeles-San Francisco). Also, there are no longer any equal cost redundant routes inside the T3 network. This avoids assymetric internal T3 backbone routing. Dismantling the T1 Backbone =========================== The current target for dismantling the T1 backbone is November '92. This will occur after (1) the remaining networks using the T1 backbone have been cut over to the T3 backbone (these are: ESnet, EASInet and CA*net); (2) the OSI CLNP transport capability over the T3 backbone is in place; (3) the T3 ENSS nodes are backed up by additional T1 circuits terminating at alternate backbone POPs. These activities are described below. Remaining Network Cutovers -------------------------- The ESnet cutover is waiting for a new version of software to be configured for the ESnet router peers at FIX-West and FIX-East. A routing plan for this cutover has been developed with ESnet and Argonne National Lab staff. A new T1 ENSS will be installed in CERN, Switzerland to provide connectivity to the T3 backbone for EASInet. Cutover of EASInet traffic will occur when this installation is complete. The Seattle RT E-PSP for CA*net is being converted to run the CA*net software and operate as part of CA*net's domain. It will run gated and speak BGP with the T3 ENSS. Once this has been debugged and tested the Princeton and Ithaca connections will similarly be upgraded. OSI Support Plan ---------------- We have begun to test the RT/PC OSI encapsulator software that was described in the August '92 engineering report. We plan to continue to run this over the T3 test network during early October, and begin deployment of the encapsulator software on production RT-EPSP nodes in late October. Native CLNP switching services will be available in a future release of the RS/6000 system software which is scheduled for deployment on the T3 network in early 1993. T1 ENSS Backup Circuits ----------------------- The installation of new T1 circuits between the existing T3 ENSS nodes, and a backup CNSS node is being scheduled to begin in late October. The initial topology for T3 ENSS backup is illustrated in a postscript map that is available via anonymous FTP on ftp.ans.net in the file /pub/info/t3enss-backup.ps . We are in contact with several regionals to schedule the installation of these backup links. We will discuss specific backup routing plans and other details with each regional. Increase Routing Table Sizes on T3 Network ========================================== We continue to experience an increase in ANSNET/NSFNET advertised networks, (see Backbone Traffic and Routing Statistics, above) The current on-card routing table size on the T3 router RS960 card (T3/FDDI) and T960 card (T1/ethernet) supports 6,000 destination networks with up to 4 alternate routes per destination. The current on-card routing tables are managing on the order of 12K routes (including alternate routes to the same destination). We are now testing new adapter microcode for the RS960 and T960 interfaces that will be deployed shortly on the T3 backbone that supports up to 10,000 destination networks with up to 4 alternate routes per destination network. We also continue to work on support for on-card route caching which will significantly increase the upper limit on the number of routes to unique destination networks that may be supported on the T3 network. This software will be available with the AIX 3.2 operating system release of the RS/6000 router software in early 1Q93. Notable Outages in September ============================ MCI Maryland Fiber Outage - 9/13 ------------------------- On Sunday evening 9/13 there was a major MCI fiber outage on the east coast that affected T3 circuits between New York<->Washington, and New York<->Cleveland. The same outage also caused several T1 safetynet circuits to go down. The fiber outage was cleared early Monday morning 9/14. During the outage, we had 4 circuits on MCI restoration facilities that were moved over to temporary restoration facilities and then moved back after the outage: New York <-> Cleveland T3, and T1 Safetynet St. Louis <-> Lafayette T1 Purdue Austell <-> Pittsburugh T1 Backbone During the fiber outage, some users may have experienced performance problems due to alternate routing over safetynet circuits, but there were did not appear to be any T3/T1 ENSS/NSS nodes that were isolated. CNSS32 Outage - 9/14 ------------- Following the fiber outage described above, there was an unrelated equipment problem that occured on the New York T3 backbone router (CNSS32). This problem started on Monday morning and was ultimately resolved by replacing the RS/6000 I/O planar board hardware on Tuesday evening. Traffic was routed around CNSS32 during this outage with the exception of the period 12:30-13:30 EST on 9/15 during which the hardware failure induced a network segmentation causing several ENSS nodes to be isolated. AIX 3.1 TCP Bug --------------- We have identified a bug in the AIX 3.1 system software that can leave RS/6000 router TCP sessions (e.g. external BGP) in a deadlock state. We have observed some isolated problems caused by this bug on ENSS134, and ENSS136 in September. We are working to isolate and correct this problem.