February 2006 - Test - lists.nanog.org

NANOG36-NOTES 2006.02.15 talks 5-end Lightning talks, closing notes
by Matthew Petach 15 Feb '06

15 Feb '06

(they weren't kidding about lightning!! ^_^;; ) 2006.02.15 Lightning Talks: Infrastructure (DNS and Routing) Security - Status and Update by Sandra Murphy Need for Speed: What's next after 10GE? by Mike Hughes A Brief Look at Some DNS Query Data by John Kristoff The impact of fiber access to ISP backbones in .jp by Kenjiro Cho New Network Monitoring Interest Group by Mike Caudill Understanding the Network-Level Behavior of Spammers by Nick Feamster (presented by Randy Bush) 12:20-12:30 Closing Remarks Steve Feldman, CNET, Susan Harris, Merit Reload your agenda for the slides!! Fun with gnuplot, DNS query data, John Kristoff X asis, source port of client query to DNS server; Y axis, how many times that port was used. Looking at recursive server for an institution open to inside and outside on 2005.11.22 starting at 1024, lots of clients use that port, then declining to the right; 1025 is most popular port; wraps at 5000, windows starts over. to the right, UNIX boxes start with high ports. Port 137, windows stuff, all bogus windows lookups Port 5353, is multicast DNS, MACs use it, also bogus Some very interesting outliers, either misconfigured or poorly thought out OS/stacks. Graphs are similar at different institutions, and at large ISPs. If you take out the external queries, points below 1024 (except 53) seem to be machines behind PAT boxes. Port 1900 is plug and play port, so windows can't use it, so it's a low outlier. external queries show more outliers in low range. looking at PTR queries internally; no elbow at 1025. 5353 standout is still there, multicast PTR queries, all bogus. MX queries, same thing. AAAA stuff, not many outliers, very clean; possibly bogus, though. A windows box trying to contact IRC server (neutered bot box); keep using same source port over again until firewall/virus software moved it. UNIX box used port range constantly across the range, more normal (trojaned box) Normal UNIX box shows more normal rows of different ports. looking at source ports, what other useful info and patterns can you start to discern? Look at TTL, dest ports, all sorts of fun you can start to discover. Sandra Murphy sandy at spart.com sandy at tislabs.com DNS and routing security DNSsec is live, sweden has signed top level zone, RIPE signing reverse zones, some reverse delegations. http://www.dnssec-deployment.org/ open working grope, dnssec deployment initiative focused on deployment issues, active mailing list, regular telecons. organizes workshops at conferences, etc. screenshot of the site; has roadmaps, working group signups, mailing lists, operator guidelines, links to NIST, etc., events, and actions. DNSSEC-tools project create tools/patches for web browsers and such. http://www.dnssec-tools.org/ current release is v0.9 from 2/10/2006 Firefox 1.5RPM to check DNS sec records back Shot of tools being released.. zonesigner tool is how you sign and maintain a signed zone. Some very detailed documents on how you sign and maintain a signed zone, as well as mailing lists. sourceforge link for dnssec-tools bundle Securing the routing infrastructure: big problem, no traction on deployable solutions 3 workshops with a wide net of interested parties. operators, iSP, access, content providers, vendors, security DHS hosted, anxious to find a solution http:///www.hsrpacyber.com/public/ Operators' emphasis a strong call from the operators for an authenticated list of authorized prefix originations (accurate, complete secure) respond to customr requests to route prefixes useful in debugging routing difficulties NEW ARIN policy suggestion recommendation new field in address templates (direct and subdelegations) for list of permitted ASes Benefits inhereits self-discipline of completign form (IRR entries aren't always done) inherits scrutiny of ARIN process on creation ARIN is authority for who is allocated prefixes Any IRR would have to check prefix with RIR Authentication and currency in IRRs authentication IRR objects RIR run IRRS have internal access to authentication for prefix holders non-RIR run IRRS would have to find a way to get that authentication from the RIRs samee is true for RIR IRR objec referring to nonmember resoureces Currency for IRR objects reclaimed resources have to result in IRR purges why not a TTL in IRR objects? Handles non-RIR IRRs This solicits requests and feedback. Try the DNSSec tools, try signing a zone, see how it works. Try the client system that does the DNSsec validation. Participate in ARIN ppml list on routing security, etc. Mike Hughes, what's next after 10GE mike at linx.net Channels geoff huston for scary graph. curve of traffic growth. By end of 2006, he'll be at 150Gb; if he takes last 3 months, he'll be at 300Gb in one metro. where is it coming from? ADSL2, Wimax, FTTx, skype, voip, p2p, etc. consolidation fewer people with bigger pipes. think back to seattle chap from force10 came and asked what do you want, 40g or 100g? we can do 40g now expensive at oc768 cheap at 4x10GE can't we just do 8x10GE rate limit/transfer cap users implment QoS thorttle p2p apps either doesn't scale, isn't an option, is costly and complex We either build and scale, or spend money to not scale. It's easier to overprovide, actually. Gary Bachula, VP Internet2 Research came to conclusion that it was far more cost effective to simply provide more bandwidth. We already need something faster than 10GE and 40GE we're already building 8x10GE link agg bundles on a single spans anyhow. common engineering sense says that your backbone has to be some multiple larger than your largest customer connection. Selling 10G transit means backbone needs to be multiples of that! Your vendor needs you! Probably--even if they don't realize it yet! Stand up Ted Seely! Some vendors are saying the next ethernet standard is 5 years out. Too late! Apparently, the IEEE 802.3 HSSG isn't convinced that it needs to start working on the next ethernet standard is it only going to happen if we drive it? answer seems to be yes! So let's start beating up our vendors!! Kenjiro Cho Impact of fiber access to ISP backbones in .jp IIJ/WIDE yes, we DO need 100G! residential broadband 21 million broadband subcriber 15 mllion for DSL 3 million for CATv 4 million fo rFTTH 100mb bidir fiber is 40USD/month 4% of heavy hitters account for 75% of inbound volume fiber users account for 86% of inbound volume DSL is only 14% no clear boundry between heavy hitters and normal users data set sampled netflow data from japanese ISP ratio of ifber and DSL unique users in dataset heavy-hitters denote users who send more than 2.5GB/day. graphs. heavy hitters statistically follows power law up to 200GB/day, 19Mbps sustained! no clear boundry between heavy-hitters and normal users lines at 2.5GB/day and the top 4% heavy hitters 4% in total uses, 10% in fiber, 2% in DSL CDF of traffic volume of heavy-hitters top 4% use 75% of inbound, and 60% of outbound correlation of inbound and outbound volumes per user fiber and DSL graphs 2 clusters one below unity line, another in high volume more heavy hitters in fiber, more lightweight users in DSL no differences between DSL and fiber except heavy hitters. fairly constant in heavy hitter usage fiber peak is 80% of combined peak. inbound much larger for heavy hitters, reversed for others? protocols/ports 83% is TCP dynamic ports RBB home users, DOM, other domestic, INTL both ends are clssified by commercial geo-IP dbs 62% of residential traffic is user-to-user 90% is inside Japan among RBB and DOM possible language, cultural barriers p2p super-nodes among bandwidth rich domestic fiber users count peer numbers for 50th percentile traffic expected 2 types; downloads, video (few streams), other with MANY peers (p2p). but couldn't find such a split. implications: we tend to attribute the skews to divide between heavy hitters and rst of users buthere are diverese and widespread heavy hitters heavy-hitters are no longer exceptional extremes which came first--start on DSL, become heavy hitter, then move to fiber? or start with fiber, and then find uses for the fiber? is this specific to japan? need to find faster links, re-evaluate prices!! Mike Caudill mcaudill at cisco.com FIRST forum of incident response and security team some special interest groups Vendor SIG CVSS SIG common vunlernabilty scoring system. Abuse SIG Network Monitoring SIG it's a members-only grope, so SIGs are just focus groups within SIG Abuse SIG, formed out of ECOAT (european) aims to further the cooperation of internet abuse fighting teams of network/information service providers, and jointly produce tangible results that will benefit its constituency Network Monitoring SIG to discuss and collaborate on various issues, such as sensor detection methodology, common rule-sets for detection, data exchange formats 2006 Conference June 25-30 2006 in Baltimore, Maryland http://www.first.org/ first-chair at first.org Still getting some chairs for the SIGs, just getting rolling. Randy Bush, IIJ Spamming with BGP spectrum agility Airudh Ramachandran, Nick Feamster Collection: two domains instrumented with mailavenger on same network sinkhole domain 1 continuous spam collection since aug 2004 no real email addresses--sink everything 10 million+ pieces of spam sinkhole domain 2 re monitors BGP as path and traceroute back to source upon receipt of every source. spamming techniques mostly botnets, of course DNS hijack of CanC to get botnet topology and geophgraphy Correlation with Bobax victims from georgia tech botnet sinkhole distance in IP space of client IP from mx record coordinate, low-banwidht sending BGP spectrum agility LOG IP address of SMTP relays join with BGP route advertisments seen at network where pam trap is col-located /8's are being announced. 61.0.0.0/8 4678 66.0.0.0/8 21562 82.0.0.0/8 8717 they bring up the aggregate, send spam from inside the empty holes in the space! 82.00/8; hit you with spam, bring it back down. Why such big prefixes? "Agility" Flexibility: client IPs can be scattered throughout dark space within a large /8 same sender usually returns with different IP address Visibility: route typically won't be filtered (nice and short) Low dampening on the /8s, so they make ideal spam sources. They're using REAL /8s, not the bogons, so they escape those filters. IP addresses are widely distributed across the /8 space IP addresses typically used once only 60-80% use... evidence that it's working: only about half of the IPs spamming from short-lived BGP are listed in any blacklist. mail to feamster within domain cc.gatech.edu for more info Length of short-lived BGP epochs 10% of spam received is coming from short-lived announcements, then plateus and hits the sharp curve of...something? CLOSING: Steve Feldman wraps up with his closing words: many thanks to Brokaw and Yahoo! for hosting, and the party Monday night, thanks to Merit and the program committees, and steering committee. No Venue for next meeting yet, so keep your eyes on the website! Late may, early June. Susan Harris from Merit for her closing words. Thanks from Merit to Steve Feldman (PC chair) And Thanks to the Dallas Yahoo! Team Brokaw Price Brian, Vicki, Mike, Todd, Raj, Brad, Sharon, Meeting stats Attendance: 340 (515 in LA, 458 in Seattle) Only 6% women NAPS: 11 NON-NA 41 Colleges, Universites, 14 Thanks to all the people from Merit who helped behind the scenes, thanks to all the presenters, and we'll see you in Spring (somewhere!). Meeting adjourns at 1216 hours Central Standard Time to the sounds of Jerry Lee Lewis on the piano.

1 0

NANOG36-NOTES 2006.02.15 talk 4 Interdomain Routing Consistency
by Matthew Petach 15 Feb '06

15 Feb '06

Access point movie goes whizzing past very quickly as Bill Fenner narrates. Lets you see where people are congregating, and which talks are more interesting, and when people migrate out of talks; could feed into the survey to tell the program comittee which talks are of more interesting. netdisco, collects data from network elements, plots them, put a front end on it; If you opted in, by emailing him you MAC address, it would render a map with your location on it. has RSS feeds of your location as well. fenner at research.att.net 2006.02.15 An Inter-domain Consistency Management Layer Nate Kushman, MIT Steve Feldman, welcome back, Nate Kushman is up first to talk about routing consistency. Transient BGP loops was with akamai, now at MIT srikanth kandula, dina katabi, john wroclawski Do loops matter? can we do something about them? what is a transient BGP loop? slide showing loop forming. How common are "transient BGP loops" Sprint study, IMC 2003, IMW 2002 looked at packet traces from the sprint backbone up to 90% of the observed packet loss was caused by routing loops 60-100% could be attributed to BGP Is it true on internet? Routing loop damage 20 fvantage points with BGP feeds did pings, traceroutes, watch for loops. correlated on BGP updates, and ttl exceeded on ping, traceroute. In fact, all loops were within 100seconds of BGP updates. 10-15% of all BGP updates caused routing loops!! Collateral damage. they cause impacts on congestions that are part of the loop, causing loss to non-rerouted networks from non-rerouted-to source networks. traceroute to see which links were part of the loop, see which other traces shared a link in common with the loop. there is a marked increase in packet loss in the 100second window around the BGP loop. Prefixes sharing a loopy link see 19% packet loss in general. What should be done? We need to prevent forwarding loops. A loop occurs because: one AS pushes a route update to the data plane, but other ASes are not yet aware of that route change. What about telling everyone about the change before the change actually happens? Suspension: continue to route traffic tell control system not to propagate the route FIB stays same for now, RIB doesn't send route. downstream networks only update forwarding tables once upstreams have acknowledged the path change. More generally: we have proven: loops are prevented in general case convergence properties similar to normal BGP http://nsm.lcs.mit.edu/~nkushman/ incrementally deployable. feedback Clearly: works well for planned maintenace. We can delay move to backup path during those events, at least. 20% of update events caused by planned maintenance Link up events also cause loops, no way to plan for them smoothly now. What about: unplanned link down events trade-off between loss on current path and collateral damage Are we willing to do this in general, to avoid impacting stable prefixes from unstable prefixes. In short: routing loops are a significant performance concern. Bill Norton--hidden question: what is the time domain during which these traffic impacts are seen? Will the propagation path take 10, 20, 30 seconds? A. one event causes many, many loops rippling out, so one update may cause packet loss for many seconds, up to tens of seconds total. Q. you're talking about adding MORE state information into the network. Also adding latency to update acknowledgements. Jared notes that router software bugs tend to exacerbate routing loop issues. You can tune configs to try to minimize the number of loops seen, as well as upgrading to "fixed" code to get better results without more state. Patrick Gilmore asks jared, does tuning help internal sessions or external sessions? Both, it really controls *when* the updates are sent out (immediately vs batched, etc.). Jared notes the internet is being used Someone (Bill?) asks if convergence times are similar to current model, as the slide claims; is that within a few seconds? convergence in the lab is similar, yes. Matt Petach asks about details of convergence; it basically puts you at mercy of the slowest, farthest away router on the network, since it has to get the message, realize it has nobody to send to, and then acknowledge back before anyone else can update FIB; yes, true, so you'd want to put timers in to limit how long you wait; basically, like "wait 5 seconds, and either hear an ACK, or go ahead and update FIB" type timeout, so you don't wait forever for a non-conformant device on the other side of the world. Riverdomain question--with suspension, you're basically in passive mode, listening but not updating, is that correct? Yes, with respect to the links/prefixes in question.

1 0

NANOG36-NOTES 2006.02.15 talk 3 Katrina Panel
by Matthew Petach 15 Feb '06

15 Feb '06

2006.02.15 Katrina Recover Panel moderator: Sean Donelan, Cisco Members: Paula Rhea, Verizon Josh Snowhorn, Terremark Bobby Cates, NASA Sean Donelan was with SBC when Katrina hit, now with Cisco. Dave couldn't be here, but Sean will do his Bellsouth slides. Lessons Learned Industry has to be able to function as a first responder to provide critical infrastructure in support of state/local response. certain sectors may need heightened support, including power and voice/data communications Providing security in times of crisis may fall back to the private sector Need to understand how the Government works in a crisis National Response Plan, FEMA system, etc. Bellsouth lost COs for first time in 100+ years of business. When you get a direct hit, you will be impacted, period. More important is how your recover! Most national disasters are pretty quick; we know how to deal with short term, but as the issue drags on, security for personnel becomes more and more vital, and is turned over to private sector, public security is engaged on more important issues. We need to help shape up the government to avoid issues like Katrina from happening again. Bellsouth, lsessons learned partnerships with other carriers, state and local government, the power companies, and the federal government made the difference experience and trust are key in a crisis Get involved--know how to reach teh communications ISAC and national coordinationg center in a crisis 703-607-4950 NCS at NCS.gov or NCC: telecom-isac at ncs.gov operational 24/7/365 Konw what programs are available to you and your customers GETS/TSP/WPS Bobby Cates from NASA up next. Supported first responders right after Katrina; they were providing video coverage, supported voice over IP, sat phones, etc. in the first days. The commercial facilities were better than gov't lines, actually. When president came in, military took over all satellite frequencies, so VoIP over commercial internet was what was left. Phones from Bill Woodcock from PCH, servers from some bay area folks; got gear loaded onto a C5 that was warming up and flew it out, the costs was less than one set of satellite phones. TSP was interesting for Katrina; for higher bandwidth, higher pricing, not much diff for TSP and non-TSP restorral. Wimax and voip pretty much saved the day, easy to implement. Josh Snowhorn, NOTA, didn't take Katrina too bad, but Norma? hit him hard. Only 3 cat 5 hurricanes (andrew, 92, camille 89, and 1935). cat 4 and cat 3 hit more often, hundreds a year of the smaller ones. Saffir-simpson hurricane scale. cat 1; winds 74-95, wind, water cat 2: 96-110, storm surge 6-8ft cat 3: 111-130, surge 9-12ft much structural damage cat 4: 131-155mph, surge 13-18ft at landfall (katrina at coast) cat 5: 155mph, surge 18ft Wilma was cat 5 before landfall, as was katrina; wilma was lowest barometric pressure every recovered. 27 named storms last year, lots of warm water heading into the gulf. formed off the bahamas, very little warning before it hit into south florida. did a bunch of power line damage. NOTA faced many issues during the storm. 2005 most storms in recorded history 2005 hurricane season went 27 named storms representing first time in history that the naming scheme went into the greek alphabet lowest NOTA--pre wilma, 3 happy balls 100mph winds on the curtain should be able to withstand it. Lost one of their roof balls during wilma NOTA lost commercial power, went on gensets for 31 hours during katrina in July NOTA lost commercial power for 10 hours with wilma, but had to stay off for 30 hours it was so dirty majority of enterprises and business in s florida without power for 10 days the day after wilma, had no less than 20 truckloads of servers and infrastructure arrive at NAP loading docks with sales people and contracts within 2 days of the passing of wilma, we began to recieve phone calls askign for fuel truck help from undersea cable operators and large enterprises; everyone pitched in to help all of the other operators in the area 12 undersea cables coming in, you cut them off, s.america largely goes away. only 1 carrier fully lost a CO in north miami, bringing down their circuits that came out of the NAP; water came in, shorted things out. Many companies did not plan properly for power failures and staff recovery and access to systems after the storms have passed. large portion didn't have DR plans or backups staff who loose their homes need food/water, won't go to work thow who want to work cannot go to devestated offices so they need to work from phone getting employees acces to systems is the singular issue that IT diretors face post katrina and wilma KEEP a dialup access poitn; it's often the only thing left in a disaster like that. Sean: to NASA; for packet traffic, what traffic did you see--lots of traffic you didn't plan for, or business as usual. The emergency response was for NCS, FEMA, DoD, as well as their 2 centers that were hit. They had a Ku band sattellite, could go to 45mb max. but no QoS, so everyone has to be good neighbors on it. NASA was using less traffic on the band than everyone else. voip, other traffic was not too big, biggest use was the football video stream. Bill Norton asks if there was any impact due to the loss C-band dishes on the 2 big ones; Ken was up there in the small Ku-dome, 11meter dome, thinner pannels so Ku band can get through it. It's a protective surround for the fully mobile dish, one panel got pentrated, then the whole ball went to pieces. What are the lessons learned that will change how we behave in the future. Paula notes that 100 mile used to be a sufficient rule for disaster site; now, with a 250mph swath, expand distance, and make it inland! NASA found they actually accomplished more than they expected to; Josh notes they host 911 systems, DoD, etc, worked well with them; GC lost their offices, they hopped into their building. Biggest challenge is staffing, making sure everyone has food, water, making sure they're safe and their families are safe so they can focus on the work that needs to be done. Bill meant more about the sensitive web of interconnectedness that could affect multiple pieces. Power, physical bridge outages, water, etc. What physical lessons did we learn from it? Internet design is a semi-mesh, not a full mesh; there were some sites that stayed up the whole time, not all paths out of NO were impacted. IP can re-route, but you have to have a physical path, and a policy that lets it do so. Chris Morrow, how do organizations that have BCP for simple things like server backups, etc. deal with the bigger cases, like "I lost my whole office/central office"--how do you make use of them. MPLS/private IP service; product sold for backup using vstat access. five large firms called, were able to give them satellite access to stay on line. MCI blue truck, an empty truck with PCs in it, FEMA rented 3 of them, turned into a mobile office. Josh notes that Mobile worker recovery units, mobile switches, etc had already been deployed to NO, when wilma hit, everything was already deployed to NO so nothing left for south florida. Again, Josh emphasises KEEP A POTS LINE!!! someone comes to mike, notes that with such a huge disaster, did it allow for shedding old legacy gear infrastructure, practices, etc? Paula notes there's a huge capital plant infrastructure; even though parts are destroyed, you want to get dial tone back as quickly as possible; string cables across trees, etc to get dial tone as soon as possible. You don't have time to do a structured upgrade to latest gear, you go back to lowest common denominator recovery gear to get them back up. Again, many thanks to all the panelists, the moderators, and everyone else for participating. Fill out your surveys and be back from break a little before 11am for a short movie from Bill Fenner from PARC.

1 0

NANOG36-NOTES 2006.02.15 talk 2 Katrina--telecom infrastructure impacts
by Matthew Petach 15 Feb '06

15 Feb '06

2006.02.15 Hurricane Katrina: Telecom Infrastructure Impacts, Solutions, and Opportunities, Paula Rhea, Verizon A more interactive presentation from her, in the aisles. Verizon Business group--combined MCI/Verizon team. Agenda Hurricane Katrina Recap Telecom infrastructure impacts telecom provider successes business continuity planning conclusions references appendices: case examples. Many of the people in this room would be considered part of the critical infrastructure for the nation by the department of homeland security After world trade center 9/11 issue, there was a lessons learned; hopefully there will be a similar report post Katrina. New Orleans is still very much like a war zone right now; it's definitely a disaster recovery training session for many industries. Neighborhoods are wiped out; no capital investments, infrastructure in holding pattern. Many with no power, 20% of houses condemned. Neighborhoods that are entirely silent--eerie. Aim is not to diss anyone specifically, certainly not in this room; aimed to be an assessment in a neutral fashion. Critical infrastructure: food and water supply energy transportation healthcare banking/finance telecommunications/infrastructure Oddly enough, much of critical infrastructure is privately owned, rather than government owned. The domino model says that any one piece will cause the rest to start to fall. 35th largest city in US port of new orleans is #1 in US by tonnage 50% of total US grain exports shipped via gulf 10.8% of total US refining from new orleans 5th largest port Key space shuttle facility in Michoud supported fuel tanks for international space station Storm recap hirricane hit aug 29 2005 135MPH winds, 20foot storm surge sent inland 55foot surges logged in gul pior to landfall levee failures create secondary crisis 2.3M homes without power spawned 33 reported tornadoes in NA 1090 fatalities in LA recorded to date people dancing about cat 5 dropping to cat 4, thought they were spared, then levees broke; had been predicted the year before. :( Still 2500 people missing/unaccounted for. Map of eastern LA parishes st bernards/plaque mines parishes between the lake and the gulf, hardest hit when levees broke as water headed back towards gulf. Lack of interoperability between parish govt systems. New orleans telecom impact (multi-carrier) 1.75M lines down immediately following kat. 38 911 centers out (1/3) initially 1,000 cellular towers out two class 4 toll switches initially out of service no power/unable to secure extended diesel fuel Traffic out of lata logjammed with toll switches out. LECs had backup power systems, but no fuel. Took 4 days to inspect causeway to allow emergency crews into the city with main bridge out. Most nurses and doctors were in suburbs, not in city. Central offices post katrina new orlenas lake co CLLI NWORLALK Venice LA CA CLLI: VENCLAMA Buras CO CLLI: BURSLAMA 19 COs are totally destroyed, and will have to be rebuilt. These slides are public domain info, no inside info. I2/Abilene link from Houston to Atlanta initially out, restored on sept 8 2005 fiber optic path on lake pchatrain bridge offline following hurricane katrina wifi, wimax and voip play key role in area communications public internet was actually very resilient Telecom provider successes: alphabetic 1,000 amateur radio operators helped bellsouth cingular cisco cox, iridium added 10,000+ new phones to first responders MCI Nortel Sprint/Nextel donated up to 10M Verizon donated 8M and 200 workers Carriers have mutual aid agreements; Verizon sent 200 people who volunteered to spend 8 weeks living in a tent to help rebuild--had to work with armed guards. The CO rebuilds wasn't any type of upgrade, it was bulldozing damaged/destroyed facilities, digging new vaults, and starting over to restore just what was in place before hand. Bill Norton--COs underwater, can you imagine some type of preventative design that could have been put in place to help avoid impacts like this? Most of the area is reclaimed land, 2 miles below sea level (some dispute about that number). Bill wonders if they could be built above sea level somehow. Even if they were, Paula notes that they wouldn't have power, wouldn't have 2 weeks of diesel fuel to run them, etc. Really, it comes back to the levees. Randy Bush noted that early on, community based wifi was one of the early-on means of communication to daisy-chain packets along. Roland, from Cisco; did some logistical work with relief; Verizon donated eVDO boxes to make eVDO to wifi bridges, did VoIP over wifi to eVDO boxes to juryrig connectivity. But doesn't work so well with towers down, and no power. With the cell phone infrastructure down, that really hurt too. Thanks to Todd Underwood/Renesys for their graphs; did a pre-and-post analysis routingwise. Top red is LA; about 170 networks totally out during the bulk period. teal/TX not impacted, MS also hit, in tierms of percentage more so than LA AL somewhat hit. There's likely to be some significant number of businesses that never recover from this. 6months later, port of NO at 50% capacity 130,000 lines remain out in LO; awaiting local govt repopulation lans cellular service fully operational in occupied areas outside of those with restritcted access US congtressional budget office estimates damage at $60B in insured and uninsured losses; insurance is tagging it at $75B. It would have been far better to have spent money helping fix the levees up front. BCP will your organization be able to survive a disaster? 1 what staf, materials, proceedures, and equipment is vital to the firm 2 what suppliers and resources do I need daily 3 what will you do if your building is not accessible 4 have you planned for payroll continuity 5 who should yo invo BCP compontents NIST 800-34 document develop contingency plan intro/activation stage/recovery phase/reconstitution stage/ plan backup. People are still in emergency mode six months later disaster recovery journal (DRJ) survey IAEM, international association of emergency managers GAP in business continuity % stating bc plan is important 62% 60% have a plan 58% tested in last 12 months 18% never tested plan 55% implmented security measures only 49% have bckup servers Telecom conclusinos interdepenedt infrsatrurctures incrase risk business continuity management should include plans for "worst case" scenrios operate--interopearte factor in data recovery and storage into plans be flexible June 1st, hurricane season starts again DNA records lost from courts, many records that might have been mandated also lost. Vision of public internet realized ARPANet launched predecessor in 1969 Goals: use disparate computers to communicate in order to provide high reliability and survivablility Katrina proved it is possible for Internet to meet that mandate.

1 0

NANOG36-NOTES 2006.02.15 talk 1 ipv6fix (and boy, does it need it)
by Matthew Petach 15 Feb '06

15 Feb '06

Morning intro notes--don't forget to fill out your SURVEYS!!!! six lightening talks signed up, should be very cool. If you have slides, get them to Steve Feldman start with! Wireless movie after break should be cool to watch. Ren? Steve mistakenly introduces her, she corrects them. Don't forget to give feedback via the Survey forms!! 2006.02.15 v6fix: Wiping the Slate Clean for IPv6 Kenjiro Cho, WIDE/IIJ, Ruri Hiromi, WIDE/Intec NetCore Will be talking about their efforts to deploy IPv6, called v6fix. v6fix is an effort to solve problems in the current v6 deployment. focuses on v4/v6 dual stack environments. it's a technical analysis of real world problem Kenjiro will talk about tools and measurements. deployment status majority of equipment out there is v6 available from major vendors still many applications and appliances just work with v4 v6 is starting to get into various business fields Many people lack knowledge/experience with v6. when non-experts hit problems, they're clueless. Example: illiteracy. Hotel internet systems have instructions for guest. troubleshooting: if you have IPv6 enabled, please disable IPv6--brochure in guest room. Cause of problem: combination. DNS redirection returns specific A record for AAAA clients stub-resolver accepts the A for AAAA, can't get out. Wiping the slate clean for the v6 faulty behaviours only 1% and combinatorial often, but could be fatal to deployment. slow fallback to v4 after v6 errors misbehaving DNS resolvers filtering of ICMPv6 DNS misconfigurations poorly configured tunnels lack of peering or v6 paths v6fix activities (research group) identify/analysze/solve real-world tech problems in v6 deployment. Enemy: "after disabling v6, my problems went away" Cooperation needed between researches, implementers, ops. v6fix topics harmful effects of the on-link assumption. misbehaving DNS servers and resolvers slow fallback to v4 after v6 failures Examples: case 1: DNS loop at hotel real story of hotel internet system--went to same room, investigated. DNS is intercepted, redirected to signup page ipv6 users can't get beyond first page hotel instructions say to disable v6 erroneus DNS redirection system and stub-resolver redirection system always returns specific A record when getting non-A queries client's stub resolver queries AAAA for any address, blindly accepts A return response. case 2: DNS server slowdown Japanse ISP ISP upgraded a DNS cache to BIND9, recieved complaints about slowdown. recompiling BIND9 with --disable-ipv6, fixed problem, reported to JANOG Caused by older BIND9 w/o IPv6 connectivity server w/o v6 connectivity always tries to talk over v6, ends up failing back to v4 after timeouts fixed in BIND9.2.5 and 9.3.1 Common factors 1 problems appear only with specific combinatorial conditions 2 implementors and operators didn't notice until reported 3 even for professionals, not easy to track down problems. Kenjiro Cho, Tools: v6 tools and measurement results Goal: to understand the macro-level v6 healthiness current methodologies wide area meaasuremetn of behaviours of 2nd/3rd level DNS servers dual stack issues DNS server measurements of .jp domain AAAA responses: 0.13% DNS servers can't deal with AAAA requests Most are lame delegation type errors. ignore AAAA queries respond with RCODE 3 ("name error") NXDOMAIN dual-stack path analysis measurement techniques specifically designed for dual-stack take measurements for v4 and v6 at same time compare v6 results with v4 results extract problems that exist in v6 only methodology dual-stack node discovery create dual-stack node list by monitoring DNS AAAA replies. dual stack ping run ping/ping6 to target dual-stack nodes select a few representative nodes per site (/48) by RTT dual-stack traceroute trace/trac6 to selected nodes visula v6 MTU to look at issues visualize path issues distribution of v6/v4 RTTs 4000 ping targets v4 on x-axis, v6 on y axis individual nodes far above unity line--leaf issues paths and PMTU visualization from NYSERNET to ARIN sites Many of ARIN paths via jp! (lack of peering) >From ISC to ARIN sites--paths look much better, but lots of blue == lots of tunnels Abilene case: well known problem. Abilene trying to encourage v6 adoption no AUP, tunnel services for v6 but ended up with horrible v6 paths, mostly with tunnels ISPs are reluctant to move to paid v6 connectivity Abilene thinking about suspending its relaxed AUP for v6 tool tries to illustrate such issues, convince users to move to native v6 dual stack traceroute to ABILENE from WIDE (v4 upper, v6 lower) similar RTTs/hops for v4/v6; native dual-stack paths dual-stack trace to ABILENE from IIJ similar RTTs, but different paths: currently more common dualstack traceroue to ABILINE fro ES v6 RTTs much larger than v4: roundabout tunnels Conclusion: faulty behaviours are only 1% and often combinatorial, but can be fatal to acceptance of v6 slow fallback to v4 on v6 errors knowledge sharing need to realize the dangers of harmful of adoption of v6 cooperation among researchers, implementers, and ops need to act now, or will bring negative impact to v6 deployment Acknowlegements v6fix members, etc. http://v6fix.net/ overview, documents, and fact database. contact at v6fix.net reports of issues are welcome

1 0

dnsauth3.sys.gtei.net DNS record is poisoned???
by Joe Shen 15 Feb '06

15 Feb '06

Hi, Today, some of our customers could not resolve state.gov by our cache server. I found state.gov is served by dnsauth1.sys.gtei.net, dnsauth2.sys.gtei.net, dnsauth3.sys.gtei.net. Using some others' DNS servers I found their IP addresses should be 4.2.49.2, 4.2.49.3, 4.2.49.4. But, our cache server(BIND9.3.1) got some othere IPs( I've tried restart bind9.3.1). So, it always failed to resolve state.gov. After restarting BIND9.3.1 again, I did "rndc flush" for several times, then it comes back. Why? is there something poisoned ? Joe =========== BIND9 got wrong server IP ==== > set debug > dnsauth1.sys.gtei.net Server: dnsv2.zjhzptt.net.cn Address: 202.101.172.133 ;; res_nmkquery(QUERY, dnsauth1.sys.gtei.net, IN, A) ------------ Got answer: HEADER: opcode = QUERY, id = 58203, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 1, authority records = 3, additional = 2 QUESTIONS: dnsauth1.sys.gtei.net, type = A, class = IN ANSWERS: -> dnsauth1.sys.gtei.net internet address = 128.121.126.139 ttl = 86084 (86084) AUTHORITY RECORDS: -> gtei.net nameserver = dnsauth2.sys.gtei.net ttl = 172565 (172565) -> gtei.net nameserver = dnsauth3.sys.gtei.net ttl = 172565 (172565) -> gtei.net nameserver = dnsauth1.sys.gtei.net ttl = 172565 (172565) ADDITIONAL RECORDS: -> dnsauth2.sys.gtei.net internet address = 169.132.13.103 ttl = 86084 (86084) -> dnsauth3.sys.gtei.net internet address = 192.67.198.6 ttl = 86084 (86084) ------------ Non-authoritative answer: Name: dnsauth1.sys.gtei.net Address: 128.121.126.139 > ============================== Restart bind and do "rndc flush" 6 times, I got: ====================== > set debug > state.gov Server: hzdnsv2.zjhzptt.net.cn Address: 202.101.172.133 ;; res_nmkquery(QUERY, state.gov, IN, A) ------------ Got answer: HEADER: opcode = QUERY, id = 20953, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 1, authority records = 3, additional = 3 QUESTIONS: state.gov, type = A, class = IN ANSWERS: -> state.gov internet address = 164.109.48.80 ttl = 1778 (1778) AUTHORITY RECORDS: -> state.gov nameserver = dnsauth3.sys.gtei.net ttl = 1778 (1778) -> state.gov nameserver = dnsauth1.sys.gtei.net ttl = 1778 (1778) -> state.gov nameserver = dnsauth2.sys.gtei.net ttl = 1778 (1778) ADDITIONAL RECORDS: -> dnsauth1.sys.gtei.net internet address = 4.2.49.2 ttl = 172767 (172767) -> dnsauth2.sys.gtei.net internet address = 4.2.49.3 ttl = 172767 (172767) -> dnsauth3.sys.gtei.net internet address = 4.2.49.4 ttl = 172767 (172767) ------------ Non-authoritative answer: Name: state.gov Address: 164.109.48.80 > ================================== __________________________________ Meet your soulmate! Yahoo! Asia presents Meetic - where millions of singles gather http://asia.yahoo.com/meetic

1 0

Interesting paper by Steve Bellovin - Worm propagation in a v6 internet
by Suresh Ramasubramanian 15 Feb '06

15 Feb '06

http://www.cs.columbia.edu/~smb/papers/v6worms.pdf - courtesy Schneier on Security and then the ITU newslog. > Internet Worms and IPv6 > > Bruce Schneier's Schneier on Security points to a paper dismissing the > myth that worms won't be able to propagate under IPv6.

5 8

FYI: reg-ops now becoming fully operational
by Rick Wesson 15 Feb '06

15 Feb '06

Through 2005, the reg-ops (Registrar Operations) mailing list which was established after the first Panix incident, was working by trial and error, learning from past mistakes, formalizing reporting guidelines and operating procedures. The mailing list now holds representatives from most of the big registrars. reg-ops also holds liaisons to NSP-SEC, DA/MWP, ICANN, SSAC, NANOG and the root servers. Other than the registrars, there are also a limited number of vetted individual from the anti-spam, anti-phishing and security industry. The purpose of the list is to relay reliable information in real-time from the industry to registrars on issues such as scams, phishing, etc. Real-time issues are handled with care while other reports can be made digested, periodically. The list is mostly operational and therefore there is not much chatter. However, security and operational issues which concern registrars or cooperation/information sharing happens when it is needed. The list is not open to the public, and subscription requires vetting. Please contact me or Gadi Evron <ge(a)linuxbox.org> directly to be added to the group. Thanks, -rick

1 0

NANOG36-NOTES 2006.02.14 talk 2 Netflow Visualization Tools
by Matthew Petach 15 Feb '06

15 Feb '06

2006.02.14 talk 2 Netflow tools Bill Yurcik byurcik at ncsa.uiuc.edu NVisionIP and VisFlowConnect-IP probably a dozen tools out there, this is just two of them. Concenses is there's something to this. They're an edge network, comes into ISP domain, their tools are used by entities with many subnet blocks. Overview Project Motifivation Netflows for Security Two visualization tools NVisionIP VisFlowConnect-IP Summary Internet Security: N-Dimensional Work Space large--already lots of data to process complex--combinatorics explode quickly time dynamics--things can change quickly! Visualizations can help! in near-realtime overview-browse-details on demand People are wired to do near-realtime processing of visual information, so that's a good way to present information for humans. HCI says use overview-browse-details paradigm. Netflows for security can identify connection-oriented stats to see things like attacks, DoS, DDoS, etc. Most people don't use the data portion of the flow field, the first 64 bytes, they just look at header info or aggregated flow records. Can spot how many users are on your system at a given time, to schedule upgrades. Who are your top talkers? How long do my users surf? What are people using the network for? Where do users go? Where did they come from? Are users following the security policy? What are the top N destination ports? Is there traffic to vulnerable hosts? Can you identify and block scanners/bad guys? This doesn't replace other systems like syslog, etc.; it integrates and works alongside them. architecture slide for NCSA. Can't really do sampled view for security, so probably need distributed flow collector farm to get all the raw data safely. Two visualization tools: NVisionIP, VisFlowConnect-IP focus on quick overview of tools security.ncsa.uiuc.edu/ 3 level hierarchical tool; galaxy view (small multiple view) ((machine view)) Galaxy is overview of the whole network. color and shape of dots is each host in a network. settable parameters for each dot. Animated toolbar and clock show changes over time in the galaxy. Lets you get high-level content quickly and easily. Domain view lets you drill in a bit more; small multiple view looks at the traffic within the block. upper histogram is lower, well known ports; lower histogram is ports over 1024 You can click on a given multiple view entry to delve into one machine. Many graphs for each machine in the most detailed view. well known ports first, then rest of ports (sorted) then source and destination traffic broken out. Designed for class Bs. http://security.ncsa.uiuc.edu/distribution/VisFlowConnectDownload.html 3 vertical lines, comes from edge network perspective; middle line is edge network to manage. You set range of networks you care about. Outside lines are people sourcing or sinking traffic to you, from outside domains. There's a time axis, traffic only shown for the slice of time currently under consideration. Uses VCR-like controls to move time forward/backward Lets you see traffic/interactivity, drill into that domain, see host level connectivity flows. Shows MS Blaster virus traffic as an example. Example 2, a scan example. Just because it looks like one IP hitting many others doesn't mean it's really a security incident, though; could be a cluster getting traffic. web crawlers hitting NCSA web servers make for a very charateristic pattern over time. Summary Netflows analysis is non-trivial, NVisionIP VisFlowConnect-IP lots of references listed in very fine blue font. http://security.ncsa.uiuc.edu/distribution/NVisionIPDownload Avi Freedman, Akamai, Argus was mentioned a lot; it lets you grab symmetric netflows, but also does TCP analysis, shows some performance data as well. not sure if people are studying the impact of correlating argus data with flow data. Roland Douta? of Cisco; many people are using netflow to track security issues. They now have ingress and egress flow data on many of their platforms. In reading paper describing it, there's data conversion that needs to happen into an internal format that nVision can understand. It reads log files at the moment, takes about 5 minutes to process files. Lets them take different file data sources, make the tool for visualization independent of the input format. They can read large files, but there is a performance hit when doing it. Are they planning on doing further work on the tool to collect TCP flags, for frags, drop traffic, etc? They've looked at it, but they leave it to IDS tools for flag activity. Might be of interest to consider for future versions of the tools. Last question came up, echoed about argus. Question about interactivity, they are working on feedback through tools. Question about alarming on patterns; but once you start alarming or putting up visual indicators, it distracts from rest of the overall pattern, you tend to miss other information.

3 2

NANOG36 Wednesday schedule, lightning talks
by Steve Feldman 14 Feb '06

14 Feb '06

Here is the revised NANOG36 agenda for Wednesday, Feb. 15: 9:00-9:30 v6fix: Wiping the Slate Clean for IPv6 Kenjiro Cho, WIDE/IIJ, Ruri Hiromi, WIDE/Intec NetCore 9:30-10:00 Hurricane Katrina: Telecom Infrastructure Impacts, Solutions, and Opportunities Paula Rhea, Verizon 10:00-10:30 Katrina Recover Panel moderator: Sean Donelan, Cisco 10:30-11:00 BREAK 11:00-11:20 An Inter-domain Consistency Management Layer Nate Kushman, MIT 11:20-12:20 Lightning Talks: Infrastructure (DNS and Routing) Security - Status and Update by Sandra Murphy Need for Speed: What's next after 10GE? by Mike Hughes A Brief Look at Some DNS Query Data by John Kristoff The impact of fiber access to ISP backbones in .jp by Kenjiro Cho New Network Monitoring Interest Group by Mike Caudill Understanding the Network-Level Behavior of Spammers by Nick Feamster (presented by Randy Bush) 12:20-12:30 Closing Remarks Steve Feldman, CNET, Susan Harris, Merit

1 0