- Test - lists.nanog.org

keeping the routing table in check: step 1
by Edward B. DREGER 16 Feb '06

16 Feb '06

Hopefully this thread will be quick and less convoluted. Rather than simply alluding to "one prefix per ASN", I'd like to detail an allocation scheme that works toward that. Find the largest contiguous block. Split in half. Round to appropriate boundary. Assign. Space at the end of the block is reserved for expansion. Ignoring special subnets for simplicity: 0/x, 128/x, 64/x, 192/x, 32/x, 96/x, 160/x, 224/x, 16/x, 48/x, 80/x, 112/x, 144/x, 176/x, 208/x assuming all grow at equal rates. 96/x ends up growing quickly? No problem. Skip 112/x for the time being. In short, allocate IP space logarithmically. Start with /1 alignment, proceed to /2, then /3, and so on. Keep the array as sparse as possible so an assignment can be extended without hitting, say, a stride 4 boundary. Perhaps RIRs should look at filesystems for some hints. Imagine a filesystem that's 30% full yet has as much fragmentation as IPv4 space. Something is wrong. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc(a)brics.com -*- jfconmaapaq(a)intc.net -*- sam(a)everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.

4 4

(OT) VERIZON NOC CONTACT PLEASE!!!
by Rich Sena 16 Feb '06

16 Feb '06

All other flame away... -- Rich Sena - ras(a)thick.net ThickNET Consulting "On the way to understanding; you understand, and forget."

2 2

a not so radical proposal for PI in ip6 world: Hierarchial routing tables
by Joe Maimon 16 Feb '06

16 Feb '06

Since the list seems to be accepting topics relating to ipv6 and PI multihoming AGAIN, I thought I would chime in again with my pet idea. Hierarchical routing. It worked for name resolution. It would work for todays routing table, which is the routing equivalent to the host file of old. Let there be a single large prefix that will be used for all PI assignments. Let the single large prefix be further divided into "geographic regions" (nod to Michael D.) Let policies for assignment from this prefix take geographic location into account, and be SMALL, so that another prefix will never be neccessary. Filtering policies include the recommendation to filter out all prefixes longer than either the single large one and/or the local geographic ones. All ISP's hosting customers with prefixes inside the PI prefix would need to advertise at a minimum both the global PI prefix and the appropriate geographic PI prefix. Fast Forward 5 years. Now there are 8 million PI prefixes advertised. Most ISP's routers cannot hold them and all the other 2 million non PI prefixes. Instead ALL ISP's operate a 2 tiered hierarchy of routers. Those that have non-PI prefixes and those that do. The routers with only non-PI prefixes may hold the ISP's local PI prefixes but will filter out any others learned. The routers with only PI prefixes peer using multi-hop BGP to other providers routers who have PI prefixes. The ISP must announc the global and applicable geographic PI prefixes so that PI destined traffic is attracted to their AS. The ISP tries to keep a full table of PI prefixes so that they can MPLS TE/L2Tpv3/source route/ incoming traffic attracted by their non PI routers to their peer's handoff. This is the key. Using a mechanism OTHER than a full table on all routers who handle the packet transiting through the AS. If a full table is impossible, the ISP may choose to operate an array of the routers, each ibgp advertising one of the geo PI prefixes and holding only a full table for that prefix. Other filtering schemes are possible. Some ISP's may opt to host NO PI customers. Then they do not need such an expensive router and they do not need to announce any PI prefixes. Some ISP's may opt to NOT announce the PI prefixes. They will have to rely on their PI-Peers to attract the desired traffic and route it to their AS. Some ISP's may opt to purchase routers capable of a 12 million prefix RIB and they will not operate such a hierarchy at all, except that their peers will undoubtedtly include ISP's who do and will need mutli-hop bgp, and those that dont and will filter all those PI prefixes out. Perhaps one day multicast prefixes will need the same kind of hierarchy. Or some other class of prefixes. But obviously a global table is going to need severe limits in expansion. As for the problems of attracting unwanted traffic and handing packets back and forth between an AS's forwarding plane routers and routing plane routers, I am betting that if the problem of large table growth is so severe, the economics would favor using twice/thrice the bandwidth and fewer expensive routers than saving bandwidth and purchasing many large expensive routers. In other words, this is a solution that can be tested and driven by the marketplace. It needs no new technology standards. All that is neccessary is some policies for PI assignment. Customers who want it will spur its adoption. ISP's wishing to host those customers will spur its implementation. ISP's filtering their table to reduce size will ensure thatg hierarchy is needed. Last time proposed, I had about two directly related comments. Feel free to ignore me again this time. Joe

1 0

a truly radical proposal
by Edward B. DREGER 16 Feb '06

16 Feb '06

Consider: RIRs refuse to grant ASNs to dual-homed leaves. Transit providers _must_ cooperate with each other. Hopefully they coalesce joint ASNs and space responsibly before sending such merrily along to the global table. Voici, non-portable ASNs and "minimum height to ride" requirements for a portable, personal ASN. This really isn't much more restrictive than today's policies, and changing an ASN is easy for small leaves. The biggest pitfall I see: Imagine having to renumber when _any_ upstream changes. I propose that this is reasonable for > /24, where such is the status quo. For <= /24 non-PI space, one could use non-coop space from a specific upstream -- again, the status quo. On a related note: Someone pointed out off-list that the "coop ASN" approach requires creating a new ASN when one changes providers. Yes, but this still uses no more ASNs (unused ones are returned, right?) than if each leaf registered its own portable ASN. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita ________________________________________________________________________ DO NOT send mail to the following addresses: davidc(a)brics.com -*- jfconmaapaq(a)intc.net -*- sam(a)everquick.net Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.

2 1

reg-ops now becoming fully operational
by Gadi Evron 16 Feb '06

16 Feb '06

As originally sent to the registrars list by Rick Wesson... Through 2005, the reg-ops (Registrar Operations) mailing list which was established after the first Panix incident, was working by trial and error, learning from past mistakes, formalizing reporting guidelines and operating procedures. The mailing list now holds representatives from most of the big registrars. reg-ops also holds liaisons to NSP-SEC, DA/MWP, ICANN, SSAC, NANOG and the root servers. Other than the registrars, there are also a limited number of vetted individual from the anti-spam, anti-phishing and security industry. The purpose of the list is to relay reliable information in real-time from the industry to registrars on issues such as scams, phishing, etc. Real-time issues are handled with care while other reports can be made digested, periodically. The list is mostly operational and therefore there is not much chatter. However, security and operational issues which concern registrars or cooperation/information sharing happens when it is needed. The list is not open to the public, and subscription requires vetting. Please contact me or Gadi Evron <ge(a)linuxbox.org> directly to be added to the group. Thanks, Gadi.

4 4

IRS goes IPv6!
by Jeroen Massar 15 Feb '06

15 Feb '06

I Ar Es, At least they have received the 2610:30::/32 allocation from ARIN. Lets see if they how taxing they find IPv6 ;) Greets, Jeroen -- OrgName: Internal Revenue Service OrgID: IRS Address: 1111 Constitution Ave. NW City: Washington StateProv: DC PostalCode: 20224 Country: US NetRange: 2610:0030:0000:0000:0000:0000:0000:0000 - 2610:0030:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF CIDR: 2610:0030:0000:0000:0000:0000:0000:0000/32 NetName: IRSNET6 NetHandle: NET6-2610-30-1 Parent: NET6-2610-1 NetType: Direct Allocation NameServer: NS1.TREAS.GOV NameServer: NS2.TREAS.GOV NameServer: NS21.TREAS.GOV NameServer: NS1.CIS.FED.GOV Comment: RegDate: 2006-02-13 Updated: 2006-02-13

14 17

Re: IRS goes IPv6!
by Andrew Dul 15 Feb '06

15 Feb '06

> -------Original Message------- > From: Christopher L. Morrow <christopher.morrow(a)verizonbusiness.com> > Subject: Re: IRS goes IPv6! > Sent: 14 Feb '06 08:31 > > On Tue, 14 Feb 2006, Jeroen Massar wrote: > > > I Ar Es, > > > > At least they have received the 2610:30::/32 allocation from ARIN. > > Lets see if they how taxing they find IPv6 ;) > > so.. this is surprising why? the us-gov mandate for ipv6 uptake will mean > lots of us-gov folks will be spinning up justifications that they are a > 'service provider' and need a /32... cause they won't accept PA space (or > I don't think they will accept PA space as a long term solution) ... So isn't this yet another reason why we need a rational PI policy, so organizations don't have make up reasons why they are LIRs.

2 1

NANOG36-NOTES 2006.02.15 talks 5-end Lightning talks, closing notes
by Matthew Petach 15 Feb '06

15 Feb '06

(they weren't kidding about lightning!! ^_^;; ) 2006.02.15 Lightning Talks: Infrastructure (DNS and Routing) Security - Status and Update by Sandra Murphy Need for Speed: What's next after 10GE? by Mike Hughes A Brief Look at Some DNS Query Data by John Kristoff The impact of fiber access to ISP backbones in .jp by Kenjiro Cho New Network Monitoring Interest Group by Mike Caudill Understanding the Network-Level Behavior of Spammers by Nick Feamster (presented by Randy Bush) 12:20-12:30 Closing Remarks Steve Feldman, CNET, Susan Harris, Merit Reload your agenda for the slides!! Fun with gnuplot, DNS query data, John Kristoff X asis, source port of client query to DNS server; Y axis, how many times that port was used. Looking at recursive server for an institution open to inside and outside on 2005.11.22 starting at 1024, lots of clients use that port, then declining to the right; 1025 is most popular port; wraps at 5000, windows starts over. to the right, UNIX boxes start with high ports. Port 137, windows stuff, all bogus windows lookups Port 5353, is multicast DNS, MACs use it, also bogus Some very interesting outliers, either misconfigured or poorly thought out OS/stacks. Graphs are similar at different institutions, and at large ISPs. If you take out the external queries, points below 1024 (except 53) seem to be machines behind PAT boxes. Port 1900 is plug and play port, so windows can't use it, so it's a low outlier. external queries show more outliers in low range. looking at PTR queries internally; no elbow at 1025. 5353 standout is still there, multicast PTR queries, all bogus. MX queries, same thing. AAAA stuff, not many outliers, very clean; possibly bogus, though. A windows box trying to contact IRC server (neutered bot box); keep using same source port over again until firewall/virus software moved it. UNIX box used port range constantly across the range, more normal (trojaned box) Normal UNIX box shows more normal rows of different ports. looking at source ports, what other useful info and patterns can you start to discern? Look at TTL, dest ports, all sorts of fun you can start to discover. Sandra Murphy sandy at spart.com sandy at tislabs.com DNS and routing security DNSsec is live, sweden has signed top level zone, RIPE signing reverse zones, some reverse delegations. http://www.dnssec-deployment.org/ open working grope, dnssec deployment initiative focused on deployment issues, active mailing list, regular telecons. organizes workshops at conferences, etc. screenshot of the site; has roadmaps, working group signups, mailing lists, operator guidelines, links to NIST, etc., events, and actions. DNSSEC-tools project create tools/patches for web browsers and such. http://www.dnssec-tools.org/ current release is v0.9 from 2/10/2006 Firefox 1.5RPM to check DNS sec records back Shot of tools being released.. zonesigner tool is how you sign and maintain a signed zone. Some very detailed documents on how you sign and maintain a signed zone, as well as mailing lists. sourceforge link for dnssec-tools bundle Securing the routing infrastructure: big problem, no traction on deployable solutions 3 workshops with a wide net of interested parties. operators, iSP, access, content providers, vendors, security DHS hosted, anxious to find a solution http:///www.hsrpacyber.com/public/ Operators' emphasis a strong call from the operators for an authenticated list of authorized prefix originations (accurate, complete secure) respond to customr requests to route prefixes useful in debugging routing difficulties NEW ARIN policy suggestion recommendation new field in address templates (direct and subdelegations) for list of permitted ASes Benefits inhereits self-discipline of completign form (IRR entries aren't always done) inherits scrutiny of ARIN process on creation ARIN is authority for who is allocated prefixes Any IRR would have to check prefix with RIR Authentication and currency in IRRs authentication IRR objects RIR run IRRS have internal access to authentication for prefix holders non-RIR run IRRS would have to find a way to get that authentication from the RIRs samee is true for RIR IRR objec referring to nonmember resoureces Currency for IRR objects reclaimed resources have to result in IRR purges why not a TTL in IRR objects? Handles non-RIR IRRs This solicits requests and feedback. Try the DNSSec tools, try signing a zone, see how it works. Try the client system that does the DNSsec validation. Participate in ARIN ppml list on routing security, etc. Mike Hughes, what's next after 10GE mike at linx.net Channels geoff huston for scary graph. curve of traffic growth. By end of 2006, he'll be at 150Gb; if he takes last 3 months, he'll be at 300Gb in one metro. where is it coming from? ADSL2, Wimax, FTTx, skype, voip, p2p, etc. consolidation fewer people with bigger pipes. think back to seattle chap from force10 came and asked what do you want, 40g or 100g? we can do 40g now expensive at oc768 cheap at 4x10GE can't we just do 8x10GE rate limit/transfer cap users implment QoS thorttle p2p apps either doesn't scale, isn't an option, is costly and complex We either build and scale, or spend money to not scale. It's easier to overprovide, actually. Gary Bachula, VP Internet2 Research came to conclusion that it was far more cost effective to simply provide more bandwidth. We already need something faster than 10GE and 40GE we're already building 8x10GE link agg bundles on a single spans anyhow. common engineering sense says that your backbone has to be some multiple larger than your largest customer connection. Selling 10G transit means backbone needs to be multiples of that! Your vendor needs you! Probably--even if they don't realize it yet! Stand up Ted Seely! Some vendors are saying the next ethernet standard is 5 years out. Too late! Apparently, the IEEE 802.3 HSSG isn't convinced that it needs to start working on the next ethernet standard is it only going to happen if we drive it? answer seems to be yes! So let's start beating up our vendors!! Kenjiro Cho Impact of fiber access to ISP backbones in .jp IIJ/WIDE yes, we DO need 100G! residential broadband 21 million broadband subcriber 15 mllion for DSL 3 million for CATv 4 million fo rFTTH 100mb bidir fiber is 40USD/month 4% of heavy hitters account for 75% of inbound volume fiber users account for 86% of inbound volume DSL is only 14% no clear boundry between heavy hitters and normal users data set sampled netflow data from japanese ISP ratio of ifber and DSL unique users in dataset heavy-hitters denote users who send more than 2.5GB/day. graphs. heavy hitters statistically follows power law up to 200GB/day, 19Mbps sustained! no clear boundry between heavy-hitters and normal users lines at 2.5GB/day and the top 4% heavy hitters 4% in total uses, 10% in fiber, 2% in DSL CDF of traffic volume of heavy-hitters top 4% use 75% of inbound, and 60% of outbound correlation of inbound and outbound volumes per user fiber and DSL graphs 2 clusters one below unity line, another in high volume more heavy hitters in fiber, more lightweight users in DSL no differences between DSL and fiber except heavy hitters. fairly constant in heavy hitter usage fiber peak is 80% of combined peak. inbound much larger for heavy hitters, reversed for others? protocols/ports 83% is TCP dynamic ports RBB home users, DOM, other domestic, INTL both ends are clssified by commercial geo-IP dbs 62% of residential traffic is user-to-user 90% is inside Japan among RBB and DOM possible language, cultural barriers p2p super-nodes among bandwidth rich domestic fiber users count peer numbers for 50th percentile traffic expected 2 types; downloads, video (few streams), other with MANY peers (p2p). but couldn't find such a split. implications: we tend to attribute the skews to divide between heavy hitters and rst of users buthere are diverese and widespread heavy hitters heavy-hitters are no longer exceptional extremes which came first--start on DSL, become heavy hitter, then move to fiber? or start with fiber, and then find uses for the fiber? is this specific to japan? need to find faster links, re-evaluate prices!! Mike Caudill mcaudill at cisco.com FIRST forum of incident response and security team some special interest groups Vendor SIG CVSS SIG common vunlernabilty scoring system. Abuse SIG Network Monitoring SIG it's a members-only grope, so SIGs are just focus groups within SIG Abuse SIG, formed out of ECOAT (european) aims to further the cooperation of internet abuse fighting teams of network/information service providers, and jointly produce tangible results that will benefit its constituency Network Monitoring SIG to discuss and collaborate on various issues, such as sensor detection methodology, common rule-sets for detection, data exchange formats 2006 Conference June 25-30 2006 in Baltimore, Maryland http://www.first.org/ first-chair at first.org Still getting some chairs for the SIGs, just getting rolling. Randy Bush, IIJ Spamming with BGP spectrum agility Airudh Ramachandran, Nick Feamster Collection: two domains instrumented with mailavenger on same network sinkhole domain 1 continuous spam collection since aug 2004 no real email addresses--sink everything 10 million+ pieces of spam sinkhole domain 2 re monitors BGP as path and traceroute back to source upon receipt of every source. spamming techniques mostly botnets, of course DNS hijack of CanC to get botnet topology and geophgraphy Correlation with Bobax victims from georgia tech botnet sinkhole distance in IP space of client IP from mx record coordinate, low-banwidht sending BGP spectrum agility LOG IP address of SMTP relays join with BGP route advertisments seen at network where pam trap is col-located /8's are being announced. 61.0.0.0/8 4678 66.0.0.0/8 21562 82.0.0.0/8 8717 they bring up the aggregate, send spam from inside the empty holes in the space! 82.00/8; hit you with spam, bring it back down. Why such big prefixes? "Agility" Flexibility: client IPs can be scattered throughout dark space within a large /8 same sender usually returns with different IP address Visibility: route typically won't be filtered (nice and short) Low dampening on the /8s, so they make ideal spam sources. They're using REAL /8s, not the bogons, so they escape those filters. IP addresses are widely distributed across the /8 space IP addresses typically used once only 60-80% use... evidence that it's working: only about half of the IPs spamming from short-lived BGP are listed in any blacklist. mail to feamster within domain cc.gatech.edu for more info Length of short-lived BGP epochs 10% of spam received is coming from short-lived announcements, then plateus and hits the sharp curve of...something? CLOSING: Steve Feldman wraps up with his closing words: many thanks to Brokaw and Yahoo! for hosting, and the party Monday night, thanks to Merit and the program committees, and steering committee. No Venue for next meeting yet, so keep your eyes on the website! Late may, early June. Susan Harris from Merit for her closing words. Thanks from Merit to Steve Feldman (PC chair) And Thanks to the Dallas Yahoo! Team Brokaw Price Brian, Vicki, Mike, Todd, Raj, Brad, Sharon, Meeting stats Attendance: 340 (515 in LA, 458 in Seattle) Only 6% women NAPS: 11 NON-NA 41 Colleges, Universites, 14 Thanks to all the people from Merit who helped behind the scenes, thanks to all the presenters, and we'll see you in Spring (somewhere!). Meeting adjourns at 1216 hours Central Standard Time to the sounds of Jerry Lee Lewis on the piano.

1 0

NANOG36-NOTES 2006.02.15 talk 4 Interdomain Routing Consistency
by Matthew Petach 15 Feb '06

15 Feb '06

Access point movie goes whizzing past very quickly as Bill Fenner narrates. Lets you see where people are congregating, and which talks are more interesting, and when people migrate out of talks; could feed into the survey to tell the program comittee which talks are of more interesting. netdisco, collects data from network elements, plots them, put a front end on it; If you opted in, by emailing him you MAC address, it would render a map with your location on it. has RSS feeds of your location as well. fenner at research.att.net 2006.02.15 An Inter-domain Consistency Management Layer Nate Kushman, MIT Steve Feldman, welcome back, Nate Kushman is up first to talk about routing consistency. Transient BGP loops was with akamai, now at MIT srikanth kandula, dina katabi, john wroclawski Do loops matter? can we do something about them? what is a transient BGP loop? slide showing loop forming. How common are "transient BGP loops" Sprint study, IMC 2003, IMW 2002 looked at packet traces from the sprint backbone up to 90% of the observed packet loss was caused by routing loops 60-100% could be attributed to BGP Is it true on internet? Routing loop damage 20 fvantage points with BGP feeds did pings, traceroutes, watch for loops. correlated on BGP updates, and ttl exceeded on ping, traceroute. In fact, all loops were within 100seconds of BGP updates. 10-15% of all BGP updates caused routing loops!! Collateral damage. they cause impacts on congestions that are part of the loop, causing loss to non-rerouted networks from non-rerouted-to source networks. traceroute to see which links were part of the loop, see which other traces shared a link in common with the loop. there is a marked increase in packet loss in the 100second window around the BGP loop. Prefixes sharing a loopy link see 19% packet loss in general. What should be done? We need to prevent forwarding loops. A loop occurs because: one AS pushes a route update to the data plane, but other ASes are not yet aware of that route change. What about telling everyone about the change before the change actually happens? Suspension: continue to route traffic tell control system not to propagate the route FIB stays same for now, RIB doesn't send route. downstream networks only update forwarding tables once upstreams have acknowledged the path change. More generally: we have proven: loops are prevented in general case convergence properties similar to normal BGP http://nsm.lcs.mit.edu/~nkushman/ incrementally deployable. feedback Clearly: works well for planned maintenace. We can delay move to backup path during those events, at least. 20% of update events caused by planned maintenance Link up events also cause loops, no way to plan for them smoothly now. What about: unplanned link down events trade-off between loss on current path and collateral damage Are we willing to do this in general, to avoid impacting stable prefixes from unstable prefixes. In short: routing loops are a significant performance concern. Bill Norton--hidden question: what is the time domain during which these traffic impacts are seen? Will the propagation path take 10, 20, 30 seconds? A. one event causes many, many loops rippling out, so one update may cause packet loss for many seconds, up to tens of seconds total. Q. you're talking about adding MORE state information into the network. Also adding latency to update acknowledgements. Jared notes that router software bugs tend to exacerbate routing loop issues. You can tune configs to try to minimize the number of loops seen, as well as upgrading to "fixed" code to get better results without more state. Patrick Gilmore asks jared, does tuning help internal sessions or external sessions? Both, it really controls *when* the updates are sent out (immediately vs batched, etc.). Jared notes the internet is being used Someone (Bill?) asks if convergence times are similar to current model, as the slide claims; is that within a few seconds? convergence in the lab is similar, yes. Matt Petach asks about details of convergence; it basically puts you at mercy of the slowest, farthest away router on the network, since it has to get the message, realize it has nobody to send to, and then acknowledge back before anyone else can update FIB; yes, true, so you'd want to put timers in to limit how long you wait; basically, like "wait 5 seconds, and either hear an ACK, or go ahead and update FIB" type timeout, so you don't wait forever for a non-conformant device on the other side of the world. Riverdomain question--with suspension, you're basically in passive mode, listening but not updating, is that correct? Yes, with respect to the links/prefixes in question.

1 0

NANOG36-NOTES 2006.02.15 talk 3 Katrina Panel
by Matthew Petach 15 Feb '06

15 Feb '06

2006.02.15 Katrina Recover Panel moderator: Sean Donelan, Cisco Members: Paula Rhea, Verizon Josh Snowhorn, Terremark Bobby Cates, NASA Sean Donelan was with SBC when Katrina hit, now with Cisco. Dave couldn't be here, but Sean will do his Bellsouth slides. Lessons Learned Industry has to be able to function as a first responder to provide critical infrastructure in support of state/local response. certain sectors may need heightened support, including power and voice/data communications Providing security in times of crisis may fall back to the private sector Need to understand how the Government works in a crisis National Response Plan, FEMA system, etc. Bellsouth lost COs for first time in 100+ years of business. When you get a direct hit, you will be impacted, period. More important is how your recover! Most national disasters are pretty quick; we know how to deal with short term, but as the issue drags on, security for personnel becomes more and more vital, and is turned over to private sector, public security is engaged on more important issues. We need to help shape up the government to avoid issues like Katrina from happening again. Bellsouth, lsessons learned partnerships with other carriers, state and local government, the power companies, and the federal government made the difference experience and trust are key in a crisis Get involved--know how to reach teh communications ISAC and national coordinationg center in a crisis 703-607-4950 NCS at NCS.gov or NCC: telecom-isac at ncs.gov operational 24/7/365 Konw what programs are available to you and your customers GETS/TSP/WPS Bobby Cates from NASA up next. Supported first responders right after Katrina; they were providing video coverage, supported voice over IP, sat phones, etc. in the first days. The commercial facilities were better than gov't lines, actually. When president came in, military took over all satellite frequencies, so VoIP over commercial internet was what was left. Phones from Bill Woodcock from PCH, servers from some bay area folks; got gear loaded onto a C5 that was warming up and flew it out, the costs was less than one set of satellite phones. TSP was interesting for Katrina; for higher bandwidth, higher pricing, not much diff for TSP and non-TSP restorral. Wimax and voip pretty much saved the day, easy to implement. Josh Snowhorn, NOTA, didn't take Katrina too bad, but Norma? hit him hard. Only 3 cat 5 hurricanes (andrew, 92, camille 89, and 1935). cat 4 and cat 3 hit more often, hundreds a year of the smaller ones. Saffir-simpson hurricane scale. cat 1; winds 74-95, wind, water cat 2: 96-110, storm surge 6-8ft cat 3: 111-130, surge 9-12ft much structural damage cat 4: 131-155mph, surge 13-18ft at landfall (katrina at coast) cat 5: 155mph, surge 18ft Wilma was cat 5 before landfall, as was katrina; wilma was lowest barometric pressure every recovered. 27 named storms last year, lots of warm water heading into the gulf. formed off the bahamas, very little warning before it hit into south florida. did a bunch of power line damage. NOTA faced many issues during the storm. 2005 most storms in recorded history 2005 hurricane season went 27 named storms representing first time in history that the naming scheme went into the greek alphabet lowest NOTA--pre wilma, 3 happy balls 100mph winds on the curtain should be able to withstand it. Lost one of their roof balls during wilma NOTA lost commercial power, went on gensets for 31 hours during katrina in July NOTA lost commercial power for 10 hours with wilma, but had to stay off for 30 hours it was so dirty majority of enterprises and business in s florida without power for 10 days the day after wilma, had no less than 20 truckloads of servers and infrastructure arrive at NAP loading docks with sales people and contracts within 2 days of the passing of wilma, we began to recieve phone calls askign for fuel truck help from undersea cable operators and large enterprises; everyone pitched in to help all of the other operators in the area 12 undersea cables coming in, you cut them off, s.america largely goes away. only 1 carrier fully lost a CO in north miami, bringing down their circuits that came out of the NAP; water came in, shorted things out. Many companies did not plan properly for power failures and staff recovery and access to systems after the storms have passed. large portion didn't have DR plans or backups staff who loose their homes need food/water, won't go to work thow who want to work cannot go to devestated offices so they need to work from phone getting employees acces to systems is the singular issue that IT diretors face post katrina and wilma KEEP a dialup access poitn; it's often the only thing left in a disaster like that. Sean: to NASA; for packet traffic, what traffic did you see--lots of traffic you didn't plan for, or business as usual. The emergency response was for NCS, FEMA, DoD, as well as their 2 centers that were hit. They had a Ku band sattellite, could go to 45mb max. but no QoS, so everyone has to be good neighbors on it. NASA was using less traffic on the band than everyone else. voip, other traffic was not too big, biggest use was the football video stream. Bill Norton asks if there was any impact due to the loss C-band dishes on the 2 big ones; Ken was up there in the small Ku-dome, 11meter dome, thinner pannels so Ku band can get through it. It's a protective surround for the fully mobile dish, one panel got pentrated, then the whole ball went to pieces. What are the lessons learned that will change how we behave in the future. Paula notes that 100 mile used to be a sufficient rule for disaster site; now, with a 250mph swath, expand distance, and make it inland! NASA found they actually accomplished more than they expected to; Josh notes they host 911 systems, DoD, etc, worked well with them; GC lost their offices, they hopped into their building. Biggest challenge is staffing, making sure everyone has food, water, making sure they're safe and their families are safe so they can focus on the work that needs to be done. Bill meant more about the sensitive web of interconnectedness that could affect multiple pieces. Power, physical bridge outages, water, etc. What physical lessons did we learn from it? Internet design is a semi-mesh, not a full mesh; there were some sites that stayed up the whole time, not all paths out of NO were impacted. IP can re-route, but you have to have a physical path, and a policy that lets it do so. Chris Morrow, how do organizations that have BCP for simple things like server backups, etc. deal with the bigger cases, like "I lost my whole office/central office"--how do you make use of them. MPLS/private IP service; product sold for backup using vstat access. five large firms called, were able to give them satellite access to stay on line. MCI blue truck, an empty truck with PCs in it, FEMA rented 3 of them, turned into a mobile office. Josh notes that Mobile worker recovery units, mobile switches, etc had already been deployed to NO, when wilma hit, everything was already deployed to NO so nothing left for south florida. Again, Josh emphasises KEEP A POTS LINE!!! someone comes to mike, notes that with such a huge disaster, did it allow for shedding old legacy gear infrastructure, practices, etc? Paula notes there's a huge capital plant infrastructure; even though parts are destroyed, you want to get dial tone back as quickly as possible; string cables across trees, etc to get dial tone as soon as possible. You don't have time to do a structured upgrade to latest gear, you go back to lowest common denominator recovery gear to get them back up. Again, many thanks to all the panelists, the moderators, and everyone else for participating. Fill out your surveys and be back from break a little before 11am for a short movie from Bill Fenner from PARC.

1 0