2006.02.15 Katrina Recover Panel moderator: Sean Donelan, Cisco Members: Paula Rhea, Verizon Josh Snowhorn, Terremark Bobby Cates, NASA Sean Donelan was with SBC when Katrina hit, now with Cisco. Dave couldn't be here, but Sean will do his Bellsouth slides. Lessons Learned Industry has to be able to function as a first responder to provide critical infrastructure in support of state/local response. certain sectors may need heightened support, including power and voice/data communications Providing security in times of crisis may fall back to the private sector Need to understand how the Government works in a crisis National Response Plan, FEMA system, etc. Bellsouth lost COs for first time in 100+ years of business. When you get a direct hit, you will be impacted, period. More important is how your recover! Most national disasters are pretty quick; we know how to deal with short term, but as the issue drags on, security for personnel becomes more and more vital, and is turned over to private sector, public security is engaged on more important issues. We need to help shape up the government to avoid issues like Katrina from happening again. Bellsouth, lsessons learned partnerships with other carriers, state and local government, the power companies, and the federal government made the difference experience and trust are key in a crisis Get involved--know how to reach teh communications ISAC and national coordinationg center in a crisis 703-607-4950 NCS at NCS.gov or NCC: telecom-isac at ncs.gov operational 24/7/365 Konw what programs are available to you and your customers GETS/TSP/WPS Bobby Cates from NASA up next. Supported first responders right after Katrina; they were providing video coverage, supported voice over IP, sat phones, etc. in the first days. The commercial facilities were better than gov't lines, actually. When president came in, military took over all satellite frequencies, so VoIP over commercial internet was what was left. Phones from Bill Woodcock from PCH, servers from some bay area folks; got gear loaded onto a C5 that was warming up and flew it out, the costs was less than one set of satellite phones. TSP was interesting for Katrina; for higher bandwidth, higher pricing, not much diff for TSP and non-TSP restorral. Wimax and voip pretty much saved the day, easy to implement. Josh Snowhorn, NOTA, didn't take Katrina too bad, but Norma? hit him hard. Only 3 cat 5 hurricanes (andrew, 92, camille 89, and 1935). cat 4 and cat 3 hit more often, hundreds a year of the smaller ones. Saffir-simpson hurricane scale. cat 1; winds 74-95, wind, water cat 2: 96-110, storm surge 6-8ft cat 3: 111-130, surge 9-12ft much structural damage cat 4: 131-155mph, surge 13-18ft at landfall (katrina at coast) cat 5: 155mph, surge 18ft Wilma was cat 5 before landfall, as was katrina; wilma was lowest barometric pressure every recovered. 27 named storms last year, lots of warm water heading into the gulf. formed off the bahamas, very little warning before it hit into south florida. did a bunch of power line damage. NOTA faced many issues during the storm. 2005 most storms in recorded history 2005 hurricane season went 27 named storms representing first time in history that the naming scheme went into the greek alphabet lowest NOTA--pre wilma, 3 happy balls 100mph winds on the curtain should be able to withstand it. Lost one of their roof balls during wilma NOTA lost commercial power, went on gensets for 31 hours during katrina in July NOTA lost commercial power for 10 hours with wilma, but had to stay off for 30 hours it was so dirty majority of enterprises and business in s florida without power for 10 days the day after wilma, had no less than 20 truckloads of servers and infrastructure arrive at NAP loading docks with sales people and contracts within 2 days of the passing of wilma, we began to recieve phone calls askign for fuel truck help from undersea cable operators and large enterprises; everyone pitched in to help all of the other operators in the area 12 undersea cables coming in, you cut them off, s.america largely goes away. only 1 carrier fully lost a CO in north miami, bringing down their circuits that came out of the NAP; water came in, shorted things out. Many companies did not plan properly for power failures and staff recovery and access to systems after the storms have passed. large portion didn't have DR plans or backups staff who loose their homes need food/water, won't go to work thow who want to work cannot go to devestated offices so they need to work from phone getting employees acces to systems is the singular issue that IT diretors face post katrina and wilma KEEP a dialup access poitn; it's often the only thing left in a disaster like that. Sean: to NASA; for packet traffic, what traffic did you see--lots of traffic you didn't plan for, or business as usual. The emergency response was for NCS, FEMA, DoD, as well as their 2 centers that were hit. They had a Ku band sattellite, could go to 45mb max. but no QoS, so everyone has to be good neighbors on it. NASA was using less traffic on the band than everyone else. voip, other traffic was not too big, biggest use was the football video stream. Bill Norton asks if there was any impact due to the loss C-band dishes on the 2 big ones; Ken was up there in the small Ku-dome, 11meter dome, thinner pannels so Ku band can get through it. It's a protective surround for the fully mobile dish, one panel got pentrated, then the whole ball went to pieces. What are the lessons learned that will change how we behave in the future. Paula notes that 100 mile used to be a sufficient rule for disaster site; now, with a 250mph swath, expand distance, and make it inland! NASA found they actually accomplished more than they expected to; Josh notes they host 911 systems, DoD, etc, worked well with them; GC lost their offices, they hopped into their building. Biggest challenge is staffing, making sure everyone has food, water, making sure they're safe and their families are safe so they can focus on the work that needs to be done. Bill meant more about the sensitive web of interconnectedness that could affect multiple pieces. Power, physical bridge outages, water, etc. What physical lessons did we learn from it? Internet design is a semi-mesh, not a full mesh; there were some sites that stayed up the whole time, not all paths out of NO were impacted. IP can re-route, but you have to have a physical path, and a policy that lets it do so. Chris Morrow, how do organizations that have BCP for simple things like server backups, etc. deal with the bigger cases, like "I lost my whole office/central office"--how do you make use of them. MPLS/private IP service; product sold for backup using vstat access. five large firms called, were able to give them satellite access to stay on line. MCI blue truck, an empty truck with PCs in it, FEMA rented 3 of them, turned into a mobile office. Josh notes that Mobile worker recovery units, mobile switches, etc had already been deployed to NO, when wilma hit, everything was already deployed to NO so nothing left for south florida. Again, Josh emphasises KEEP A POTS LINE!!! someone comes to mike, notes that with such a huge disaster, did it allow for shedding old legacy gear infrastructure, practices, etc? Paula notes there's a huge capital plant infrastructure; even though parts are destroyed, you want to get dial tone back as quickly as possible; string cables across trees, etc to get dial tone as soon as possible. You don't have time to do a structured upgrade to latest gear, you go back to lowest common denominator recovery gear to get them back up. Again, many thanks to all the panelists, the moderators, and everyone else for participating. Fill out your surveys and be back from break a little before 11am for a short movie from Bill Fenner from PARC.