From: Per Gregers Bilse <bilse@EU.net>

...

Because loss of connectivity for a day costs 10's of thousands of dollars and, evidently, even the "big" companies are not immune from "acts of god" like what happened today.

Please distinguish between singly-connected and singly-homed, they really are two different things. If Sun has just a plain, regular connection, a garden variety leased line outage will take them out. If OTOH they had had a connection to one of BBN's other CA POPs, they would have been just fine. It is intuitively obvious that the advantage of multihoming does not outweigh the additional complexity, cost and failure modes, whereas multiconnecting can be done at reasonable cost while actually providing more reliable service overall. All information clearly indicates that the PA outage was one of these near-enough disasters that can and do happen, just like hurricanes and floods and whatever. You can't blame BBN for that, but you could conceivably blame Sun, InfoWorld and/or whoever for having just a single connection, if they're supposed to take their Internet business seriously -- do these people really want to be taken out by a blown CSU/DSU?

I did not say multi-homing by itself would be a cure-all. I picked Sun as an example among the affected customers because they do have several interconnected physical locations in the bay area, US-wide and abroad and also in-house expertise on routing. It is just that a whole bunch of smart people running Sun's NOC must have felt helpless.. If only they had a pipe going out to somewhere else, another provider in the bay area in this case, they could have reconfigured routers and minimized the hit. In hindsight, solutions to tolerate a given failure are very easy to come by but shouldn't be talked down. I did not blame BBN--shit happens--but I wouldn't assume that their NOC is necessarily designed to be more fault tolerant than what a smaller ISP can or what the people at Sun could if they didn't somehow assume IP transit out of BBN was fault tolerant enough. For all you know, Sun might have multiple pipes connecting to BBN, multipe routers and CSU/DSU spares. I know for sure a client of BBN's cupertino POP was affected because all BBN traffic in the bay area, as far as I can tell, went/goes through the stanford location. Sanjay.