Hi. This was included in this month's Internet Monthly Report. Mark ANSNET/NSFNET Backbone Engineering Report July 1992 Jordan Becker, ANS Mark Knopper, Merit becker@ans.net mak@merit.edu T3 Backbone Status ================== Traffic migration from the T1 to the T3 backbone is in its final stages, with only a few networks on T1 left to be cut over to T3. New software is being deployed on the T3 backbone nodes to further improve performance and stability. Load-related problems continue to occur on the T1 network's Routing Control Processors and are being addressed with software changes to improve RCP efficiency. Activities for the "Phase IV" upgrade of the T3 backbone are being planned including FDDI adapter upgrades and other changes that will support dismantling of the T1 backbone later this year. Traffic Migration and Statistics ================================ In July, 12,884,937,091 packets entered the T3 network, up 20% from June. 4,754,950,521 packets entered the T1 network, down 17.5% from June. The total for both networks (minus cross-network traffic) was 17,063,064,412, up 8.4% from June. 6130 networks are configured for the T1 backbone, and 5269 for the T3 backbone. Actual network announcements by mid-level peer routers to the T1 backbone are 4426 networks, and 3508 to the T3. It is interesting to note that 1246 of the configured T1 networks were never actually announced to the backbone during the month; and 882 of the configured T3 networks were never announced to the backbone. New software has been installed to track the number of announced networks over time. During July, NorthWestNet, Terrestrial Wideband (TWB) Net, MilNet and Minnesota Supercomputer Center (MSC) Net were cut over to the T3 backbone. The remainder of the networks for Sprint and International Connections Manager (ICM) were cut over in early August. Networks remaining to be cut over are: ESNet, Mexican Networks (at Boulder), EASINet, and CA*Net. The first two are expected in mid-August. The EASINet and CA*Net peers are currently only physically connected to the T1 backbone and will be moved to allow peering with the T3 backbone in early September. Improvements to the T3 Backbone =============================== New Software Build ================== Version 2.81.31 of the AIX software and microcode has recently been deployed on the T3 backbone nodes. This build includes several significant fixes to problems and improvements to performance. These include: a buffering change to increase performance for T3 to ethernet interface transfers, a fix to a condition that causes an ethernet interface to freeze, on-card support for additional ethernet MIB-II variables, changes to the T3 card microcode, and a fix to a bug in ICMP time-stamp processing (now resolves to the millisecond). T3 Routing Software Changes =========================== Numerous improvements have been made to the T3 routing daemon, rcp_routed. These are described in release notes available for anonymous ftp: ftp.ans.net:/pub/info/t3-rcp_routed/Release-Notes The most significant recent change provides a workaround for regionals that generate a default pointing at 140.222 and find their ENSS isolated from the rest of 140.222 T3 system due to loss of the ENSS link to the CNSS. Other changes include better handling of BGP external metrics, and various reliability and operational improvements. Routing stability measurements have been made over the last few months. This information, in addition to the detailed logs have helped identify routing disturbances, isolate problems involving connectivity loss (IS-IS adjacency loss or BGP disconnect) which should not have occurred, and correct the problems. This has resulted in a measurable improvement in BGP stability. FDDI Interface Upgrade Planned for August ========================================= We are planning to upgrade the existing FDDI interfaces deployed on several ENSS nodes to new RS960 technology based interface adapters. Due to the complexity of the May '92 T3 upgrade, and the need for operational stability, it was decided not to upgrade the older FDDI interfaces at the same time as the T3 network. Instead, it was decided to decouple the process of testing and deployment of the new T3 technology from the new FDDI technology. The FDDI interface that is installed at each T3 ENSS location will be upgraded from an earlier technology to the newer RS960 adapter technology in August '92. This will raise the peak throughput observed by end users well beyond the current 10-14Mbps that may be observed today. Our objective is to continue to incrementally improve performance while maintaining very high reliability. Once the new FDDI adapters are installed and operational at the existing nodes, we will explore installations at other ENSS nodes that do not yet support FDDI. New SNMP Daemon Installed ========================= A new release of the SNMP software on the T3 backbone was installed. With the deployment of build 2.81.31 and SNMP version 3.213, in and out octet counts are available for the T960 ethernet and T1 cards. The current MIB variable map for the ethernet, T1 and T3 interfaces supported by the RS6000 nodes is as follows. inU outU inNU outNU inOct outOct InErr OutErr ==================================================================== T960 Ethernet | U+B U+B 0 0 OK OK IN+OUT 0 | RS960 T3 | 0 0 IN OUT OK OK OK OK | ==================================================================== The pending operating system upgrade of the RS/6000 routers to AIX 3.2 will allow for more complete support of the appropriate MIBS for each adapter type. RS/960 Memory Parity Problem ============================ A manufacturing problem with memory chips on a small number of the RS960 T3 interface adapters has been identified and corrected. However to protect against reccurrence or new introduction of memory problems, we are scheduling diagnostics to be run on some nodes along with other scheduled maintenance over the next few weeks. DSU Synchronization Problem =========================== A problem was observed in the first few weeks after the April-May deployment of RS/960 T3 cards and DSUs, where a T3 DSU would lose synchronization with its peer on nodes that connect to the Houston POP. This problem has not been reproducible in our test labs or in the T3 Research Network after extensive experimentation, nor has it occurred on production nodes since then. DSU PROM Upgrade ================ The firmware in the T3 Technologies DSUs will be upgraded to a new level, allowing extended remote monitoring capability. This upgrade will be performed in the late August timeframe with plenty of advanced notice. T1 ENSS Memory Upgrades ======================= Each of the T1 ENSS nodes on the T3 backbone were recently upgraded from 16MB -> 32MB of main memory to provide capacity for increased paging. T1 Backbone Status ================== A new rcp_routed release was deployed early in July which increased the BGP packet size to 4096. This was necessary to allow a full complement of networks to be announced from the backbone to regional peers (the same change was made to the T3 rcp_routed). Later in the month another new rcp_routed version was deployed which contained changes to support compression of the IS-IS protocol LSP. This is necessary to allow the T1 network to sustain the current limit of 2200 networks being announced in from external mid-level peers to a single NSS. NSS 11 at Houston exhibited some problems after the Milnet networks were cut over to the T3 backbone. Some mid-level peers announce all of the networks learned from the T3 backbone into the T1 backbone, and these are stored by the T1 node but are marked as unusable. This caused the Houston RCP to run out of memory. A change was made to the peer routers to prevent these networks from being announced to the T1 network, and a future release of the rcp_routed will include a change so that these routes are not stored. T3 Backbone Phase IV Planning ============================= Upgrades to the T3 backbone are being planned to support future performance and functional enhancements, and to allow the dismantling of the T1 backbone. The major features of this plan include: 1) T3 ENSS FDDI interface upgrades to new RS/960 card. 2) A T1 circuit will be installed at each T3 ENSS to allow a backup connection to a different CNSS. This will provide some redundancy in the case of T3 circuit or primary CNSS failure. 3) Network source/destination pair traffic statistics collection for the T3 nodes. This feature is provided on the T1 backbone currently. 4) Open Systems Interconnection/Connectionless Network Protocol (OSI/CLNP) support. This feature is provided on the T1 backbone currently. 5) Upgrade selected T3 ENSS ethernet interfaces to new RS/960 ethernet card. 6) Topology Changes: o Moving the Washington D.C. area CNSS to a new MCI POP for closer proximity to several ENSS locations. The tail circuits of the existing network attachments to this POP will be reduced to local access circuits only. o Deploy a new CNSS in Atlanta to reduce the GA Tech T3 tail to local access only, and provide expansion capability in the southeast.