- NANOG - lists.nanog.org

RE: Followup British Telecom outage reason
by Kevin Gannon 26 Nov '01

26 Nov '01

> I'm referring to the _vendor's_ support costs - as in, you don't need as > many people in the TAC if people don't keep running into IOS bugs; you > don't need as large of a RMA pool if the hardware is more reliable, etc. What percentage of TAC personnel's time is spent dealing with calls that ultimately result in a BugID? NANOG isn't representative; mostly, TAC exists to take calls from idiots who bought a box that they don't know how to configure. Large network operators have a staff of people to handle that, so when they call TAC, the box is probably broken. I don't think that's the case with the majority of TAC cases, though. I would imagine a lot once I got queued to the team that deals with the "I just bought an 803 from pc world" and good did it sound busy in the background. Yet when you ring for network down situations its like a dead calm in the background hmmm. Or how much time is spent telling you its a bug you can't see ? ;-). Regards, Kevin

1 0

RE: Followup British Telecom outage reason
by Kevin Gannon 24 Nov '01

24 Nov '01

Anyone have the Bug ID's ? Regards, Kevin -----Original Message----- From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Sean Donelan Sent: 24 November 2001 19:17 To: Neil J. McRae Cc: nanog(a)merit.edu Subject: Re: Followup British Telecom outage reason On Sat, 24 Nov 2001, Neil J. McRae wrote: > I'd be surprised if it was the GSR, and in anycase that doesn't > absolve anyone. If it was a software issue- why wasn't the software > properly tested? Why was such a critical upgrade rolled out across > the entire network at the same time? It doesn't add up. It appears to be yet another CEF bug. If you want to use a GSR you are stuck using some version of IOS with a CEF bug. The question is which bug do you want. Each version of IOS has a slightly different set. Several US network providers have also been bitten by CEF bugs too. While trying to fix one set of bugs, BT upgraded of their network. I'm not sure if they were upgrading at 9am in the morning, or had upgraded earlier and the bug finally came out under load at 9am. When the BT network melted down, Cisco suggested installing a different version of IOS, which had previously been tested. At noon, BT found the new version had an even worse bug, sending packets out the wrong interface. It was until 2200 (13 hours later), BT and Cisco found a version of IOS which stablized the network. "Stablized" not fixed. The running version of IOS still has a bug, but it isn't as severe.

1 0

The Cidr Report
by CIDR Report 24 Nov '01

24 Nov '01

This is an auto-generated mail on Fri Nov 23 23:00:00 PST 2001 It is not checked before it leaves my workstation. However, hopefully you will find this report interesting and will take the time to look through this to see if you can improve the amount of aggregation you perform. Check http://www.employees.org/~tbates/cidr-report.html for a daily update of this report. NEW: Check http://www.employees.org/~tbates/cidr-report-region.html for the regional version of this report. NEW: Check http://www.employees.org/~tbates/autnums.html for a complete list of autonomous system number to name mappings as used by the CIDR-Report. The report is split into sections: 0) General Status List the route table history for the last week, list any possibly bogus routes seen and give some status on ASes. 1) Gains by aggregating at the origin AS level This lists the "Top 30" players who if they decided to aggregate their announced classful prefixes at the origin AS level could make a significant difference in the reduction of the current size of the Internet routing table. This calculation does not take into account the inclusion of holes when forming an aggregate so it is possible even larger reduction should be possible. 2) Weekly Delta A summary of the last weeks changes in terms of withdrawn and added routes. Please note that this is only a snapshot but does give some indication of ASes participating in CIDR. Clearly, it is generally a good thing to see a large amont of withdrawls. 3) Interesting aggregates Interesting here means not an aggregate made as a set of classful routes. Thanks to GX Networks for giving me access to their routing tables once a day. Please send any comments about this report directly to CIDR Report <cidr-report(a)cisco.com>. ------------------------------------------------------------------------------ CIDR REPORT for 23Nov01 0) General Status Table History ------------- Date Prefixes 161101 105107 171101 104669 181101 104452 191101 104732 201101 104515 211101 104367 221101 104601 231101 104388 Check http://www.employees.org/~tbates/cidr.plot.html for a plot of the table history. Possible Bogus Routes --------------------- AS Summary ---------- Number of ASes in routing system: 7341 Number of ASes announcing only one prefix: 3130 (0 cidr, 3130 classful) Largest number of cidr routes: 0 announced by Largest number of classful routes: 1416 announced by AS701 1) Gains by aggregating at the origin AS level --- 23Nov01 --- ASnum NetsNow NetsCIDR NetGain % Gain Description AS1221 1202 928 274 22.8% Telstra Pty Ltd AS701 1416 1175 241 17.0% UUNET Technologies, Inc. AS4151 316 179 137 43.4% USDA AS4755 328 197 131 39.9% Videsh Sanchar Nigam Ltd. Autonom AS16473 203 77 126 62.1% Bell South AS577 319 196 123 38.6% Bell Advanced Communications Inc. AS7018 814 695 119 14.6% AT&T AS6595 162 57 105 64.8% DoD Education Activity Network As AS705 277 175 102 36.8% UUNET Technologies, Inc. AS852 390 296 94 24.1% Telus Advanced Communications AS2048 206 116 90 43.7% State of Louisiana AS8010 124 36 88 71.0% PublicNet Communications, Inc. AS4323 396 311 85 21.5% Time Warner Communications, Inc. AS1239 475 399 76 16.0% Sprint AS19632 85 10 75 88.2% Metropolis Intercom S.A. AS1273 327 257 70 21.4% Cable & Wireless ECRC GmbH AS12302 106 38 68 64.2% MobiFon S.A. AS2149 369 302 67 18.2% Performance Systems, Inc. AS6471 133 67 66 49.6% ENTEL CHILE S.A. AS17557 142 76 66 46.5% Pakistan Telecom AS4200 126 61 65 51.6% Apex Global Information Services, AS3464 161 96 65 40.4% Alabama SuperComputer Network AS724 209 150 59 28.2% DLA Systems Automation Center AS7046 294 235 59 20.1% UUNET Technologies, Inc. AS16758 63 6 57 90.5% IKON Office Solutions AS13999 70 14 56 80.0% Mega Cable S.A. de C.V. AS703 273 219 54 19.8% UUNET Technologies, Inc. AS3908 300 248 52 17.3% Supernet, Inc. AS17561 113 62 51 45.1% Internet service provision to Wes AS855 143 93 50 35.0% Canadian Research Network For the rest of the previous weeks gain information please see http://www.employees.org:80/~tbates/cidr-report.html 2) Weekly Delta Please see http://www.employees.org:80/~tbates/cidr-report.html for this part of the report 3) Interesting aggregates Please see http://www.employees.org:80/~tbates/cidr-report.html for this part of the report

1 0

Qwest reporting widespread DSL outages
by Joe St Sauver 24 Nov '01

24 Nov '01

Happy Thanksgiving... According to a support person at Qwest's DSL Technical Support number (800-247-7285), Qwest is experiencing widespread intermittent loss of connectivity for DSL customers (symptomology is a customer's Cisco 675 or 678 will report loss of WAN physical layer, link will then come back up after a brief period of time, problem repeats). This is occurring even for customers running CBOS 2.4.3 (Qwest's currently recommended CBOS revision, per http://www.qwest.com/dsl/customerservice/csco675ups.html ). This has been going on (according to the Qwest technical support person) for multiple hours, and Qwest has no further information. Qwest suggests DSL customers contact their ISP for further information (although it is clear that this is not an ISP-level issue at time time). No Qwest web page or other information about the outage is available according to the Qwest support person. Regards, Joe St Sauver (joe(a)oregon.uoregon.edu) University of Oregon Computing Center Disclaimer: the preceding is not, of course, an official Qwest announcement and your milage may vary.

4 3

BT monopoly causes widespread network failure
by smd＠clock.org 24 Nov '01

24 Nov '01

Sean Donelan writes: | The joy of single provider service. No, you mean: the joy of a rapacious monopoly with a spineless regulator. BT has successfully stalled opening itself up to competition as required by EU and UK law, by sitting on the local loop. Meanwhile, they themselves are deliberately slow on rolling out modern broadband services (e.g., xDSL) compared to the EU average. In short, the UK is *the* backwater of Europe when it comes to high-speed Internet connectivity -- it is rare to find at all, and when you find it, it's not cheap. That BT's outage affected so many people and organizations is a horrific indictment of the disastrous policy line the regulator has taken, and the incompetence of the government when it comes to promoting its stated aim of increased connectivity for UK residents. The official line is at: http://www.oftel.gov.uk/publications/local_loop/llufacts/llufacts0501.htm Even the OFFICIAL statistics are depressing: Number of fixed lines in the UK: 35 million Number of BT ADSL connections: 50 thousand There are FOUR sites where local loop unbundling can happen, and they are not remotely in the biggest population centres. By comparison: http://news.zdnet.co.uk/story/0,,t269-s2092106,00.html BT admits that it has only handed over 163 residential lines to other operators By comparison, during last night's election coverage in Denmark, there were 29000 DSL-speed connections to streaming servers operated by one of the main TV operators. Nearly everyone I know in the IT field has DSL at home. Operators are offering low-price SDSL. Unbundling is working. Not only is Denmark a much smaller economy than the UK, it also has some very different geographical challenges: it is made up of 400 islands. Yet, even in areas of relatively low population concentration (esp compared to, for example, London), it is possible to get DSL connectivity provided from at least one operator, and often several (4+). Most EU countries can claim the same sucess as DK. Shame on the UK's government, which is directly responsible for the regulatory environment which deprives a huge percentage of the population from getting better, cheaper service than they get from their expensive, unreliable monopoly called British Telecom! | [BT lost] | of its national IP backbone on Tuesday affecting DSL and Dialup | service across multiple ISPs in the UK. These ISPs had NO CHOICE wrt multihoming. They can thank their government. | According to news | reports, this affected almost all DSL service in the UK. Right, because of the 50000 or so DSL customers in the country, a maximum of *163* are not supplied by BT. | What happened to BT? Is this a unique "feature" of the UK marketplace | or can the same thing happen in the USA? You have evil incumbents in the USA too. Beware. Sean.

7 7

Somalia
by Gwendolynn ferch Elydyr 23 Nov '01

23 Nov '01

Does anyone have more details on what exactly happened here? http://news.bbc.co.uk/hi/english/world/africa/newsid_1672000/1672220.stm (brief summary: BT & AT&T apparently shut down Somalia's external Internet and phone connections) cheers! ========================================================================== "A cat spends her life conflicted between a deep, passionate and profound desire for fish and an equally deep, passionate and profound desire to avoid getting wet. This is the defining metaphor of my life right now."

1 0

Re: Somalia
by Matthew R. Potter 23 Nov '01

23 Nov '01

A news link from Xinhua News Agency dated Nov 14... http://library.northernlight.com/FC20011114230000093.html?cb=0&dx=1006&sc=0# doc Matt

1 0

Teleglobe issues?
by Tim Devries 22 Nov '01

22 Nov '01

Has anyone seen any issues w/ regards to teleglobe? It seems they lost some power in Toronto, and caused a wack of lost connectivity. Anyone have any more information on this? Tim

1 0

Qwest multi-state outage
by Dan Hollis 22 Nov '01

22 Nov '01

Anyone have details? Looks like most of their customer IP network is down. -Dan -- [-] Omae no subete no kichi wa ore no mono da. [-]

1 0

Code Red -> Router Memory depletion?
by Mike Lewinski 22 Nov '01

22 Nov '01

We've seen two routers experiencing problems this AM that appear to be related to client swervers infected with the IIS Code Red virus. I say appear because of the timing with cpu profiles on downstream routers where infections broke out, but I don't have any direct evidence. The first one was a border router: Jul 19 08:00:47 5093: 2w5d: %SYS-2-MALLOCFAIL: Memory allocation of 65540 bytes failed from 0x603BF35C, pool Processor, alignment 0 Jul 19 08:00:47 5094: -Process= "BGP Router", ipl= 0, pid= 86 # sh ver uptime is 4 hours, 46 minutes System returned to ROM by bus error at PC 0x603BFCFC, address 0xFFFFFFF0 at 05:57:21 UTC Thu Jul 19 2001 The other one is a client aggregation router Jul 19 12:02:49 192: %SYS-2-MALLOCFAIL: Memory allocation of 1964 bytes failed from 0x314DA4A, pool Processor, alignment 0 Jul 19 12:02:49 193: -Process= "OSPF Router", ipl= 0, pid= 32 (This router is still functioning, but not allowing any incoming connections on telnet). -Mike

4 5