Hi. This report will appear in this month's Internet Monthly Report. Mark ANSNET/NSFNET Backbone Engineering Report December 1992 Jordan Becker, ANS Mark Knopper, Merit becker@ans.net mak@merit.edu Network Status Summary ====================== Following the T1 backbone disconnection on Dec. 2nd, the T3 is now supporting all NSFNET services. A 2nd ethernet interface was installed in ENSS136 to support interim connectivity to the MAE-East facility. The former T1 backbone class B address (129.140) was returned to the NIC. CA*Net peer router hardware reconfigurations will be completed in January to allow direct T3 network peering. A plan has been developed to dismantle and remove the RT routers at regional sites. Following the software changes resulting from instabilities observed during the MBONE multicasts in November, the T3 backbone routing stability has been very good in December. Daily reports are now generated on internal and external peer network routing stability. Seven new RS960 FDDI interfaces were installed on ENSS nodes in December. Reliability and performance of the FDDI hardware has been very good. Backbone Traffic and Routing Statistics ======================================= The total inbound packet count for the T3 Backbone (measured using SNMP interface counters) was 22,009,352,089 up 5.0% from November. As of December 31, the number of networks configured in the NSFNET Policy Routing Database was 8561 for the T3 backbone. Of these, 1760 were never announced to the T3 backbone. The maximum number of networks announced to the T3 backbone during the month (from samples collected every 15 minutes) was 6187. Average announced networks on 12/31 were 6154. Previously when the T1 Backbone was operational the overall average number of networks announced via the primary configured AS path was around 88%. Since the T1 Backbone was dismantled this average has gone to around 95%. Graphs of this information are available for anonymous ftp on merit.edu, in pub/nsfnet/offnet, as postscript files. Routing Software and Stability on the T3 Network ================================================ Higher than usual MBONE activity in November resulted in exterior routing instabilities and routing instability internal to ANSnet. Following the immediate corrective actions that were taken in November, a number of software changes were made in December. Changes to the RS960 microcode and efficiency improvements in the routing software have resulted in improved ANSnet internal routing stability (regardless of the presence of external route flapping). The MBONE participants have made changes to route MBONE traffic over the T3 backbone whenever possible, resulting in improved routing stability within certain US regional networks. Detailed reports of internal ANSnet and external peer network routing instabilities are now generated on a daily basis. The ANSnet internal report attempts to summarize outages based on number and duration of the IBGP disconnects. The program that generates this report looks at IBGP disconnects in a short time window. It tries to assign the outage to the node with the most disconnects, and then eliminate all disconnects on other machines associated with that router. If after eliminating disconnects related to the router assigned to the outage, more disconnects remain in the same time window, the process is repeated, assigning secondary and tertiary responsibility for the outage and so on until all disconnects are accounted for. The time window is extended to cover all disconnects that are not separated by more than a five minute period in which there is no change to internal routing in the network. Using this method, data sufficient to generate reports on internal ANSnet routing stability has been collected continuously since October '92. Over the course of December '92, all but two ANSnet nodes reported 99.0% internal routing stability (no changes to internal routing). These two nodes reported 98.9% stability, mostly due to a problem early in December. Most nodes reported 99.0% to 99.5% stability. Two reported better than 99.5% stability. The vast majority of this instability is due to the regular configuration updates and scheduled maintenance. With the dismantling of the T1 backbone, activities are underway to reduce the duration of time required to process the bi-weekly router configuration runs, and scheduled maintenance. Similar data on external routing stability has been collected continuously since December '92. The external routing stability reports will record an external route which is withdrawn from the ANSnet due to: (1) an EGP route timing out (not being advertised by an external peer), (2) an explicit BGP unreachable received from an external peer, (3) the loss of an external EGP or BGP session that is not accompanied by loss of internal IBGP sessions. Excluded are the following cases: (1) we do not observe changes in the next hop at the ENSS, (2) the route is replaced by another route on the same ENSS, (3) a route is lost and backed up by another route on the same ENSS, (4) connectivity is lost due to ANSnet internal problems (e.g. configuration runs or other). Some of this data has already been shared with specific regional networks to get feedback. In January, these reports will be fully automated and distributed to peer networks which are advertising networks that are withdrawing reachability excessively (e.g. exhibiting great instability). After the recent FIX-East reconfiguration, a bug in rcp_routed was observed that would result in an ENSS installing a route locally that had previously been advertsized by a peer, where the route was rejected on the ENSS due to policy constraints (no NACR request). Such a route would incorrectly be installed on the ENSS (but not advertised) when the destination becomes completely unreachable. On ENSS136, the existence of multiple peers coupled with the use of default routing by some peers could form a routing loop when this occurs. This problem in the rcp_routed software has been corrected and is deployed on ENSS136. Deployment of the same changes across the rest of the ANSnet is scheduled for early January. Another planned change to rcp_routed in January involves fixing the metric_in function for EGP. This will allow regional peer routers to be able to have an EGP session with the same ENSS on two different interfaces (FDDI and ethernet) and prefer the FDDI via metrics. RS960 FDDI Deployment Status ============================ During the month of December, we installed RS960 FDDI adapters at the following 7 locations: Boston (E134), Argonne (E130), FIX-E (E145), Houston (E139), Ithaca (E133), FIX-W (E144), Seattle (E143). Domain Name Service for ANSnet ============================== In December, ANS made two adjustments to the DNS administration of the ANS backbone. First, the "t3.nsf.net" domain was renamed to "t3.ans.net", and second, we added geographical designators to all CNSS nodes. ENSS names were not changed other than in being moved to the "t3.ans.net" domain. The following excerpt of a traceroute from ANS Elmsford to Merit in Ann Arbor illustrates the ANSnet DNS conventions. 3 t1-2.New-York-cnss35.t3.ans.net (140.222.35.4) 7 ms 7 ms 7 ms ANS Elmsford's ENSS is attached to T1 Port #2 on CNSS35 (a T1 concentrator in the New York POP). 5 t3-3.New-York-cnss32.t3.ans.net (140.222.32.4) 8 ms 8 ms 8 ms 6 t3-1.Cleveland-cnss40.t3.ans.net (140.222.40.2) 23 ms 22 ms 24 ms The T3 point-to-point line between the New York POP and the Cleveland POP is on these two T3 ports on CNSS32 and CNSS40. 7 t3-0.Cleveland-cnss41.t3.ans.net (140.222.41.1) 22 ms 23 ms 23 ms The Ann Arbor ENSS is attached to T3 Port #0 on CNSS41 (the T3 concentrator in the Cleveland POP). 8 t3-0.enss131.t3.ans.net (140.222.131.1) 30 ms 27 ms 27 ms 9 merit.edu (35.1.1.42) 29 ms 39 ms 31 ms The merit.edu machine is one hop from ENSS131 which is physically located in a University of Michigan building. While every effort was taken to ensure the accuracy of the DNS data, typos and glitches may have crept in. If you notice any problems, please report them by sending mail to hostmaster@ans.net. OSI Support on T3 Backbone ========================== In early December, OSI CLNP forwarding over the T3 backbone was configured via encapsulation of CLNP over IP packets using the EON method (RFC1070) until native CLNP switching services are available on the T3 routers. One RT from the T1 NSS node at each midlevel/peer site is being used for the EON encapsulation function. It is our intention to keep this RT and one other for statistics collection purposes running at each site. CA*Net Reconfiguration Status ============================= CA*Net is working with Merit and ANS to complete their plan to change their connections to the US (at Seattle, Ithaca and Princeton) so that the US nodes will run the CA*Net RT kernel and become logically part of the Canadian routing domain. This project has not yet been completed, and so currently the three NSS nodes from the T1 Backbone (but without the T1 circuits) are being used to route traffic between CA*Net and NSFNET. Once the software installation is complete the circuits will be moved over to these RTs and the full NSS noes will be disabled. AIX 3.2 Migration Plan ====================== System testing continued in December for the new AIX 3.2 operating system for the RS/6000 routers. This software will be installed on the T3 test network in January for final system testing prior to deployment in early February. GATED Routing Software Development ================================== Work continued on the development of GATED as replacement for rcp_routed software on the T3 backbone. The same Interior Gateway Protocol (IGP) used by rcp_routed will be supported initially on GATED as a transitional measure to simplify the deployment of GATED. Full IS-IS integrated routing will follow in GATED, along with BGP4 with CIDR routing capability. New ANSnet Nodes Installed ========================== ENSS Customer Access Date Active ---- -------- ------ ----------- 206 CERN T1 12-01-92 187 CIX T1 12-04-92 209 H.W. Wilson T1 12-04-92 207 Siemens 56k 12-07-92 177 N.Y. Bank T1 12-09-92 199 Dept. of Trans. 56k 12-11-92 179 MCNC Backup 56k 12-15-92 Cisco PROM Upgrade on ANSNET ============================ We are finalizing tests that will allow us to upgrade the software level on all ANSnet Cisco routers to 9.0(3). This will fix several bugs that currently exist on the current 8.3.X that is installed across the network.