Jo? wrote:
I'm confussed, but please pardon the ignorance. All the data centers we have are at minimum keys to access data areas. Not that every area of fiber should have such, but at least should they? Manhole covers "can" be keyed. For those of you arguing that this is not enough, I would say at least it?s a start.
That is an option, but it doesn't address the real problem. The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! "The net" was created to enable communications bttn point A and point B in this exact scenario. No one should be surprised that ATT would cut-corners on critical infrastructure. The good news is that this incident will likely result in increased Federal scrutiny if not regulation. We know how spectacularly energy and banking deregulation failed. Is that mistake being repeated with telecommunications? The bad news is that some of the $16M/yr ATT spends lobbying Congress (for things like fighting number portability and getting a free pass on illegal domestic surveillance) will likely be redirected to ask for money to "fix" the problem they created. This assumes ATT is as badly managed, and the US FCC and DHS are better managed, than has been the case for the last 8 years. Time will tell. For a good "man in the street" perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning: http://www.jivamedia.com/askdrdawn/askdrdawn.php http://geekspeak.org/ Roger Marquis
The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about!
s/DARPA/ARPA/; s/BBM/BBN/; s/Internet/ARPAnet/. BBN won the contract to build the first four IMPs. Theory and research about it is older, look at: http://www.lk.cs.ucla.edu/LK/Bib/REPORT/PhD/proposal-01.html But you are right, redundancy is the issue, cost is the factor. Jorge.
On Sat, 11 Apr 2009, Roger Marquis wrote:
The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! "The net" was created to enable communications bttn point A and point B in this exact scenario.
Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. Read the original DARPA papers, they were not about making sure grandma could still make a phone call.
For a good "man in the street" perspective of how the outage effected things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning:
Why didn't the "man in the street" pharmacy have its own backup plans? Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides AT&T or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions? Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy. Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc. Readiness works better if everyone does their part, including grandma. Next time it won't be AT&T, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or .... or .... or .... . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, .... Everything fails sometimes. What's your plan? http://www.ready.gov/ personal opinion only
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful. I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate. The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer. Sorry for the random thoughts... -Mike On Sat, Apr 11, 2009 at 4:11 PM, Sean Donelan <sean@donelan.com> wrote:
On Sat, 11 Apr 2009, Roger Marquis wrote:
The real problem is route redundancy. This is what the original contract from DARPA to BBM, to create the Internet, was about! "The net" was created to enable communications bttn point A and point B in this exact scenario.
Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed.
Read the original DARPA papers, they were not about making sure grandma could still make a phone call.
For a good "man in the street" perspective of how the outage effected
things like a pharmacy's ability to fill subscriptions and a university computer's ability to boot check out a couple of shows broadcast on KUSP (Santa Cruz Public Radio) this morning:
Why didn't the "man in the street" pharmacy have its own backup plans?
Why didn't the pharmacy also have a COMCAST or RCN broadband connection for alternative Internet access besides AT&T or Verizon, a Citizens Band radio channel 9 for alternative emergency communications besides 9-1-1, a satellite phone for alternative communications besides local cell phones, and a Hughes VSAT dish for yet even more diversity? Why was the pharmacy relying on a single provider? Or do it the old-fashion way before computers and telecommunications; keep a backup paper file of their records so they could continue to fill prescriptions?
Why didn't the pharmacy have more self-diversity? Probably the usual reason, more diversity costs more. That may be the reason why hospitals have more diversity than neighborhood pharmacies; and emergency rooms have other ways to get medicine. Maintaining diversity and backups is probably also part of the reason why filling a prescription at a hospital is much more expensive than filling a prescription at your neighborhood pharmacy.
Likewise, why didn't grandma have her own pharmacy backup plan. Don't wait until the last minute to refill a critical presciption, have backup copies of prescriptions with her doctor, have an account with an alternative pharmacist in case her primary pharmacist isn't reachable, etc.
Readiness works better if everyone does their part, including grandma.
Next time it won't be AT&T, it will be Cox or Comcast or Qwest or Level 3 or Global Crossing or .... or .... or .... . It won't be vandalism, it will be an earthquake, backhoe, gas main explosion, operator error, ....
Everything fails sometimes. What's your plan?
personal opinion only
While OT the news reports indicated ATMs were offline and many credit card processing machines were down. This is no big shock because many ATM networks are on frame relay and POS credit card machines use POTS lines. The outage also impacted mobile service too if it hadn't been said. I hope we can put this thread to rest soon. -r On Sat, Apr 11, 2009 at 04:25:26PM -0700, Mike Lyon wrote:
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful.
I also wonder if the bigger pharmacies such as Longs, Walgreens, Rite-Aid, Etc had thought about these kinds of issues? I personally doubt it. I bet you they went dark along with everyone else. Unfortunate.
The funny thing is that the California lottery would be somewhat immuned to this kind of disaster as they actually use Hughes VSAT at every single retailer.
Sorry for the random thoughts...
-Mike
Mike Lyon wrote:
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful.
...
Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Don't really care so much about the bank's security, especially if it was one that received some the bailout money :) I was more worried about if people could make withdraws from their bank accounts. Deposits they could do as they could enter them in later but withdraws I think would be different. On Sat, Apr 11, 2009 at 5:19 PM, Roy <r.engehausen@gmail.com> wrote:
Mike Lyon wrote:
Anyone know how banks in the Bay Area did through this? I wonder how many banks went dark and whether they had any backup plans/connectivity. Me thinks its doubtful.
...
Because of the loss of the alarm systems, many banks went to a method where only one or two people were let in at a time. Extra security was also posted because of the inability to call 911.
Sean Donelan wrote:
,,,, Uh, not exactly. There was diversity in this case, but there was also N+1 breaks. Outside of a few counties in the Bay Area, the rest of the country's telecommunication system was unaffected. So in that sense the system worked as designed. ....
About eight or ten years ago I went to PacBell (or whatever it was called at the time) and requested that two large facilities get a sonet ring between them. I was told I couldn't have it because they were both fed through a single set of conduits and one backhoe could cut both sides of the ring. It wouldn't be diverse so they wouldn't provison it unless I paid for the digging of new paths. So much for their theory of diverse. Sounds like the rules are different for them. There are one thing to also point out. That train track next to the manholes in South San Jose is the major line between the Bay Area and Southern CA. There are at least three or four fiber paths for different companies buried along those tracks. There are also connections from Gilroy to the Hollister/San Juan Bautista area and thence to Salinas. It would have been very simple for the telcos to provision a backup path southward.
participants (6)
-
Jorge Amodio -
Mike Lyon -
Ravi Pina -
Roger Marquis -
Roy -
Sean Donelan