2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts, Solutions, and Opportunities, Paula Rhea, Verizon A more interactive presentation from her, in the aisles. Verizon Business group--combined MCI/Verizon team. Agenda Hurricane Katrina Recap Telecom infrastructure impacts telecom provider successes business continuity planning conclusions references appendices: case examples. Many of the people in this room would be considered part of the critical infrastructure for the nation by the department of homeland security After world trade center 9/11 issue, there was a lessons learned; hopefully there will be a similar report post Katrina. New Orleans is still very much like a war zone right now; it's definitely a disaster recovery training session for many industries. Neighborhoods are wiped out; no capital investments, infrastructure in holding pattern. Many with no power, 20% of houses condemned. Neighborhoods that are entirely silent--eerie. Aim is not to diss anyone specifically, certainly not in this room; aimed to be an assessment in a neutral fashion. Critical infrastructure: food and water supply energy transportation healthcare banking/finance telecommunications/infrastructure Oddly enough, much of critical infrastructure is privately owned, rather than government owned. The domino model says that any one piece will cause the rest to start to fall. 35th largest city in US port of new orleans is #1 in US by tonnage 50% of total US grain exports shipped via gulf 10.8% of total US refining from new orleans 5th largest port Key space shuttle facility in Michoud supported fuel tanks for international space station Storm recap hirricane hit aug 29 2005 135MPH winds, 20foot storm surge sent inland 55foot surges logged in gul pior to landfall levee failures create secondary crisis 2.3M homes without power spawned 33 reported tornadoes in NA 1090 fatalities in LA recorded to date people dancing about cat 5 dropping to cat 4, thought they were spared, then levees broke; had been predicted the year before. :( Still 2500 people missing/unaccounted for. Map of eastern LA parishes st bernards/plaque mines parishes between the lake and the gulf, hardest hit when levees broke as water headed back towards gulf. Lack of interoperability between parish govt systems. New orleans telecom impact (multi-carrier) 1.75M lines down immediately following kat. 38 911 centers out (1/3) initially 1,000 cellular towers out two class 4 toll switches initially out of service no power/unable to secure extended diesel fuel Traffic out of lata logjammed with toll switches out. LECs had backup power systems, but no fuel. Took 4 days to inspect causeway to allow emergency crews into the city with main bridge out. Most nurses and doctors were in suburbs, not in city. Central offices post katrina new orlenas lake co CLLI NWORLALK Venice LA CA CLLI: VENCLAMA Buras CO CLLI: BURSLAMA 19 COs are totally destroyed, and will have to be rebuilt. These slides are public domain info, no inside info. I2/Abilene link from Houston to Atlanta initially out, restored on sept 8 2005 fiber optic path on lake pchatrain bridge offline following hurricane katrina wifi, wimax and voip play key role in area communications public internet was actually very resilient Telecom provider successes: alphabetic 1,000 amateur radio operators helped bellsouth cingular cisco cox, iridium added 10,000+ new phones to first responders MCI Nortel Sprint/Nextel donated up to 10M Verizon donated 8M and 200 workers Carriers have mutual aid agreements; Verizon sent 200 people who volunteered to spend 8 weeks living in a tent to help rebuild--had to work with armed guards. The CO rebuilds wasn't any type of upgrade, it was bulldozing damaged/destroyed facilities, digging new vaults, and starting over to restore just what was in place before hand. Bill Norton--COs underwater, can you imagine some type of preventative design that could have been put in place to help avoid impacts like this? Most of the area is reclaimed land, 2 miles below sea level (some dispute about that number). Bill wonders if they could be built above sea level somehow. Even if they were, Paula notes that they wouldn't have power, wouldn't have 2 weeks of diesel fuel to run them, etc. Really, it comes back to the levees. Randy Bush noted that early on, community based wifi was one of the early-on means of communication to daisy-chain packets along. Roland, from Cisco; did some logistical work with relief; Verizon donated eVDO boxes to make eVDO to wifi bridges, did VoIP over wifi to eVDO boxes to juryrig connectivity. But doesn't work so well with towers down, and no power. With the cell phone infrastructure down, that really hurt too. Thanks to Todd Underwood/Renesys for their graphs; did a pre-and-post analysis routingwise. Top red is LA; about 170 networks totally out during the bulk period. teal/TX not impacted, MS also hit, in tierms of percentage more so than LA AL somewhat hit. There's likely to be some significant number of businesses that never recover from this. 6months later, port of NO at 50% capacity 130,000 lines remain out in LO; awaiting local govt repopulation lans cellular service fully operational in occupied areas outside of those with restritcted access US congtressional budget office estimates damage at $60B in insured and uninsured losses; insurance is tagging it at $75B. It would have been far better to have spent money helping fix the levees up front. BCP will your organization be able to survive a disaster? 1 what staf, materials, proceedures, and equipment is vital to the firm 2 what suppliers and resources do I need daily 3 what will you do if your building is not accessible 4 have you planned for payroll continuity 5 who should yo invo BCP compontents NIST 800-34 document develop contingency plan intro/activation stage/recovery phase/reconstitution stage/ plan backup. People are still in emergency mode six months later disaster recovery journal (DRJ) survey IAEM, international association of emergency managers GAP in business continuity % stating bc plan is important 62% 60% have a plan 58% tested in last 12 months 18% never tested plan 55% implmented security measures only 49% have bckup servers Telecom conclusinos interdepenedt infrsatrurctures incrase risk business continuity management should include plans for "worst case" scenrios operate--interopearte factor in data recovery and storage into plans be flexible June 1st, hurricane season starts again DNA records lost from courts, many records that might have been mandated also lost. Vision of public internet realized ARPANet launched predecessor in 1969 Goals: use disparate computers to communicate in order to provide high reliability and survivablility Katrina proved it is possible for Internet to meet that mandate.