On 10/5/2002 at 12:30:36 +0000, Tim Thorne said:

After reading all the stories about what supposedly happened does anyone know what really happened? Did UUNet US really do an IOS upgrade on a sizable proportion of their border routers in one go? This seems like suicide to me. What possible reason could there be for a network-wide roll out of an untested IOS apart from being in the mire already?

The assumption that it was untested is probably an unfair one. Once a network reaches a certain size, it is very difficult to simulate it in a lab. Number of routes/updates, variety of packet destinations, different card revisions and layouts... heck, even statistically, you have problems. An issue that appears 5% of the time will only show up in a a 10-router test lab half the time, but in a 400-router network it'll pop up on about 20 routers and wreck your whole day. And when you're out of cash, you can't really afford to devote lots of hardware to a lab. I'm not saying that their testing procedures were correct, or that they tested the image as well as they could have... but the assumption that, if it blew up in the field it must not have been tested at all probably isn't accurate. -Dave