It seems like a lot of folks have run into software upgrade problems all together. @Home, Telocity, Abovenet, Exodus and probably more recently had some type of snafu which their customer service people said was related to upgrades in progress. Are we at that point in the upgrade cycle where everyone needs to make changes at the same time? It doesn't seem to be a common vendor problem, as far as I can tell it was a different issue in each case. But after a relatively slow period in December (holidays) and January; I noticed an uptick in people sending me mail about issues during the first couple of weeks of February. I'm theorizing there is some natural calendar function which results in people doing upgrades about the same time at the beginning of February. The same thing seemed to happen with California power plants. After being down only 3,000MW; in February California is down 10,000MW of plant capacity due to scheduled and unscheduled maintenance.
New Year, new budget, new money, new projects?
It seems like a lot of folks have run into software upgrade problems all together. @Home, Telocity, Abovenet, Exodus and probably more recently had some type of snafu which their customer service people said was related to upgrades in progress.
On Thu, Feb 15, 2001 at 06:17:17PM -0800, Sean Donelan wrote:
Are we at that point in the upgrade cycle where everyone needs to make changes at the same time? It doesn't seem to be a common vendor problem, as far as I can tell it was a different issue in each case. But after a relatively slow period in December (holidays) and January; I noticed an uptick in people sending me mail about issues during the first couple of weeks of February.
There are a few interesting things I would point out regarding the upgrade cycle. Note, I have no data to put these together, they are only my theories. Juniper releases software along a fairly predictable schedule. Also, as their releases are less frequent, and they are still a bit in the "adding new things people really need" phase, I think there is a greater tendancy for a number of providers to go to new Juniper software in the same timeframe (eg, a couple of weeks after a release). Cisco had a period of time (which I think has come to an end, but you can never be sure) where software releases seemed to have a terrible tendancy to come out, pass all the lab tests you could throw at them, only to crash and burn in a matter of days in the real network. In general terms I'd lump any software in the last six to nine months into that catagory. Router instability is multiplied by crashes. I think crashes are similar to atomic chain-reactions. When one router crashes, particularly in the core it causes a huge amount of churn for other routers, causing all of the classic problems and making them more likely to reboot. It only takes a couple of relatively minor problems occuring within a short timeframe to greately stress the system, and increase the likelyhood of more crashes. As such, a medium level problem introduced by a vendor can trigger many more minor level problems. There is also a semi-related topic. "Upgrade" does not always mean "new software". Sometimes upgrade means new hardware and software. Sometimes upgrade just means new hardware. It depends both on the provider and the exact work being performed. This makes it much harder to align event reports from different vendors to isolate causes. I think a good part of this is the nothing done in december, peoeple come back and plan in january for a new year, and by feburary the first of those plans are underway. Much like rush hour or other human self synchronizations providers probably are most likely to "need to upgrade" at the same time their clients "need to upgrade" the tools and systems they are working on and most depend on the network. -- Leo Bicknell - bicknell@ufp.org Systems Engineer - Internetworking Engineer - CCIE 3440 Read TMBG List - tmbg-list-request@tmbg.org, www.tmbg.org
participants (3)
-
Leo Bicknell -
Neil J. McRae -
Sean Donelan