We will include a more detailed report in the February Internet Monthly Report, but this summary gives a general report of events on the backbones over the last month. Mark ANSNET/NSFNET Backbone Engineering Report Preview of February, 1992 End-Of-Month Report ============================================= Mark Knopper, Merit Jordan Becker, ANS Summary ======= The T3 network continues to perform extremely well as it has since last November. During February, we have cut over a significant load of traffic from the T1 backbone to the T3 backbone. Midlevel networks that were cutover to use T3 in February include SURAnet (at both College Park and Atlanta), NyserNet/PSI, San Diego (SDSC and CERFNet), and SesquiNet. These are in addition to the other midlevel & regional networks that have previously been cut over to use the T3 system. We are coordinating with several other midlevel networks that we plan to cutover to T3 during the month of March. The midlevel networks continue to peer with both the T1 and T3 networks, and continue to use the T1 backbone to communicate with sites that have not yet cut over to the T3 backbone. This minimizes the load on the T1/T3 interconnect gateways in Ann Arbor, Houston, and San Diego. The interconnect gateway load has decreased as we have added load to the T3 system. We are also in the process of installing a 4th interconnect gateway at Princeton. T3 Network Status ================= Performance on the T3 network has been very good. There have been reports of intermittent packet loss, but this has been isolated to sources outside of the T3/T1 systems. As we migrate additional traffic onto the T3 network we are collecting daily reports on peak traffic load, and any packet discards on all CNSS/ENSS nodes in order to detect any problems caused by the added load. We are now measuring daily average sustained loads of about 5Mbps across a typical CNSS node (all interfaces) with an average packet size of about 200 bytes. The T3 backbone continues to be very reliable. There was a hardware problem on the backbone CNSS machine at Cleveland that resulted in two outages. We also continue to experience an occasional black link on CNSS-CNSS links. With the redundancy in the T3 network, none of these problems resulted in extended outages for any end users. There was also a routing software problem with the flooding of external BGP updates between the IBM and Cisco routers which we have worked around, and an SNMP monitoring problem which was identified and corrected. T1 Network Status Summary ========================= Performance and reliability on the T1 backbone has not been as good as T3 during February, but this is improving as we have (1) cut significant traffic over to the T3 system (2) are fixing several chronic problems that have been isolated and focused on during the last month. Some of these chronic problems and fixes include: - Congestion at NSS10 has been minimized by cutover of PSI/NyserNet traffic to the T3 system. We also replaced the PSP- 10-16 node to ensure that we were not having any intermittent hardware problems. - We recorded an above average number of T1 circuit outages reported during February. The increase in outages was largely due to two unrelated fiber cuts in the MCI network during the week of 2/17. - The "DCD Waffle" problem reported previously has been further diagnosed as two separate problems, and software corrections to these problems have already been partially deployed with the remaining software updates being tested on the research network for deployment during the coming week. A detailed description of these problems will be documented in the February Internet Monthly report. - An intermittent PSP crash problem is still being investigated. We suspect that this involves an RT kernel bug in the virtual memory system. - The fix to the "internally generated" ICMP net unreachable message problem has been successfully tested on the research network and will be deployed on the production network during the coming week. This should serve to minimize host problems resulting from any routing instabilities in the T1 network. - A new interior routing problem was identified where IS-IS link state PDUs were being truncated at sites where the regional was announcing over 2000 networks to the T1 backbone. A patch to the routing software to allow up to 2200 networks announced to a single NSS was already deployed as a near term fix. A new routing daemon that compresses the encoding of networks in the IS-IS link state packet is now being tested for deployment as a longer term solution.