(I think these were the toughest to take notes on, since they went by so fast; took the most cleaning up afterwards. But they were also the best talks of the 3 days. I wish we could have flipped, and taken more time on Tuesday for them so we really could have dug in and asked the questions we were itching to ask. ^_^; --Matt) 2006.06.07 Lightning talks Marty Hannigan, Renesys: [slides are at: http://www.nanog.org/mtg-0606/pdf/lightning-talks/1-hannigan.pdf Critical infrastructure, root server location analysis Where to stick your servers. :) he took some public info out there on root-servers.org talked to some people, extrapolated from larger set of data. operator demographics. in US: 3 corp a, c, j 2 edu b and d 1 mil g 2 research e/h 3 nonprofit f, i, l autonomica is responsible for l, but hosts "some" instances on a CDN; CDN is a US formed entity in EU: 1 non profit k asia/japan: 1 nonprofit m 92% of system operated in US, 8% non-us; 5% margin of error +-. US entity type non-us 8% us corp 39% us mil 23% us edu 15% us nonprofit 15% where? in 54 countries all religions all methods of governance politically: 79% are democratic governments 21% in other forms of government global diversification for security and performance instances spread across continents different networks different proceedures different software different hardwware different weaknesses weaknesses become strength, since they are diverse; no one weakness knocks out all servers. little less open to insider malfeasance Global distribution NA 38% EU 35% Asia 12% AUs 8% east EU 3% LA 2% Africa 2% ANT 0% getting reasonable coverage in the world situating a root server relationship 101 who you know ICANN, operator, IX, and RIR relationships regulators how you spin it national pride performance and security betterment of user experience Threats no different from anyone else direct attacks proxy attacks botnets easy money miscreants masking other activities Not sure what motivations to attack root servers; can't extort money from nonprofits let's attack a root server target $-root location; eu hosting facility multi-post cabinet config with cabling and power under floor unlocked cabinet, single factor facility entry physical attack open cabinet door access to power hijack attempt advertise a route return bad answers network attack spoof source random host queries packet floods summary: root system is less likely to be subject to insider attack or weakness but can be attacked by layer 3 there is likely good resarch data coming across those interfaces trend towards a collapsed root system, where root and TLD share same hardware or networks should be more closely examined. slides will be up soon, talk to him in the hallway NEXT, Anton Kapela Network RTTs [slides are at: http://www.nanog.org/mtg-0606/pdf/lightning-talks/2-kapela.pdf I'm pinging 10: high rate active probes we're pinging stuff really quickly adjusted host kern.hz to 1000 select() gets pretty accurate +-1ms emmission accuracy stuff is responding Interesting 0.001% of data relates to end-to-end queuing what has been sampled? some cisco 7513s IOS 12.3 mainline linux 2.4.20 freebsd 4.8 NT4 sp6 various end-to-end paths on u-wisc network raw data isn't terrible interesting. in adaptive link layer protocols, see rate shifting manifested in RTT wireless, HPNA/HCNA, powerline ethernet 10,30,60,90 second peaks fourier transforms, wavelet transforms, frequency domain 1000 seconds at 10ms intervals break into composite, aggregate graph at top, 0-50hz span on x axis, y axis is contribution summary of entire graph. bottom right graph is rough 200 samples of a range from 0-5hz, 100pps, deduce delay at half that sampling rate. delay is not a simple boring thing; has scheduler delays, path dynamics not visible before to see queue depths. shark fins showed up; congestion events do occur, are quite measurable. when links are hot, queues are obvious, esp. on highly multiplexed links. bottom left, cubic resonance, several tens of thousands of multiplexed flows hitting odd resonance. pinging windows machine, composite spectral fingerprint; 10,20,25,30 spikes Linux fewer spikes freebsd low and flat IOS is 10, 20, 30 and grass of 1hz spacing below 10hz. win32 delay spectrum also has 1hz fuzz below 10hz. Sampled RTT and performed signal analysis of it; now what? is network time continuous? is round trip time discreet or continous? no changes in revealed as you go down lower is delay a "signal' anyway what's with the 0 hz DC component in the FT output? could this be used for fingerprinting? yes, could be like next nmap. packet-level fingerprinting is trivial to fake; but IP stack scheduler behaviour doesn't change so easily. NEXT: Mikael Abrahamsson Affect on traffic from the TPB bust with Kurtis Lindqvist [slides are at: http://www.nanog.org/mtg-0606/pdf/lightning-talks/3-abrahamsson.pdf Bittorrent background p2p protocol for filesharing. text string, upload to tracker, get IPs of other clients that have done the same thing, clients connect to each other, develop a swarm. clients communicate even when tracker vanishes. just can't get new clients joining Thepiratebay.org run by a handful of individuals aged 22 to 28 used ~100mb at peak peaked at 2M concurrent users stats code in tracker indicated that total p2p traffic was close to 100gig/sec thus far, largest bittorrent site/tracker in world photo slide showing the physical gear 10 high-end small servers in half a rack in stockholm, sweden web frontends, db servers, trackers on the stats not an exact science at least a german ISP had an outage at the same time bust was around 12.00 CET may 31st (euro time) data collected from Euro-IX members some saw no difference. Netnod aggregated, biggest drop, about 10+Gb drop very quickly in Netnod stockholm *very* visible. stats server was slashdotted, lost an hour of stats. LINX London, saw about 5Gb drop out of 80Gb AMSIX dropped about 5Gb out of 160Gb DECIX frankfurt, germany, drop before noon, FCIX, helsinki, Finland drop fairly visibile NIX, in norway, drop also visible. doesn't show private exhanges/private peerings Brussels (BNIX) also saw drop. netflow export from big US ISP, large chunk of bittorrent traffic packets faded off. Thepiratesbay.org was back online 72 hours later in Amsterdam, Netherlands and traffic started coming back June 6th is a holiday, watch the stats this coming week. :) Aftermath Police took ALL hosted equipment at the same site by the same hosting company (small one, only a few racks), caused quite a few community web sites to go down plus commercial customers Has spawned a lot of discussion in Sweden regarding all issues involved. Front page material every day, even video surveillance of the raid from surveillance cameras has been posted on youtube.com Accusations of police/politicians being influenced by White House and MPAA and others Q: Bill Norton: what about other tracker sites, why didn't traffic just shift to them? A: some did, but torrent files have the tracker hard coded in them, so they can't just flip over to other tracker sites on their own. Q: Roland Dobbins, back up in several countries now including Russia, is traffic back? A: Keep watching the graphs. And if you want to see the bust, search for "pirate bay" and "police", there's one link on youtube. NEXT: Alex Pilosov/Pilosoft Adam Rothschild/Voxel Nathan Patrick/Sonic.net [slides are at: http://www.nanog.org/mtg-0606/pdf/lightning-talks/4-pilosov.pdf Passive Metro WDM how it works single mode fiber: mutiple wavelengths also called "colours" or "lambdas" coexisting separately pluggable optics as enabler low cost for passive optical equipment, particularly grey market Dark fiber IRUs are very cheap. low opex/capex how does it work? O band Original 1260 - 1360 E band Extended 1360 - 1460 S band Short 1460 - 1530 C band Conventional 1530 - 1565 L band Long 1565 - 1625 implementation options active WDM cisco 15xxx, cienna, movaz, others passive WDM using optical filters self-assembled patch panels complete systems (CUBO) pictures of components GWDM/WWDM wideband multiplexing (1350/1550) 2GE fdx per pair, 1 GE fdx per strand single strand networking the receiver is *always* wideband low cost for transcievers (LX/ZX, <$500) 10GE possible (ER/LR) Active xWDM beyond this scope everyone knows how to do it, it just costs more. Passive CWDM wavelength, wide channels, 8 channels 1470-1610 conventional 1270-1470 low range cost is cheap ~$1000 per strand per end for CUBOs, $300-$1000 per GBIC depending on quality (CUBO, Taiwanese hw manufacturers) no Xenpaks, GBICs only 20nm channel spacing low availability on 'low range' GBICs/SFPs Passive DWDM each channel is narrow 0.8nm == 24 dense channel per single coarse channel 160 channels easily 25Ghz spacing research at 12.5Ghz Xenpaks available $9k+ few GBICs at $1500+ Filters: build/add as you grow by mixing and matching available in various ranges (center wavelength, bandpass width) Going from GWDM to GWDM/CWDM to GWDM/CWDM/DWDM Testing and management optical power meter communication is key OOB access: HOOTS, cell phone you need to talk site-to-site to coordinate make sure cell phones don't depend on fiber optical power monitoring/APD receivers in GBICs (show interface blah trans) spectrum analyzer Caveats few complete commercial systems available systems require clue and duct tape to put together need to tune with attenuators if signal is too strong, attenuators differ with wavelength flaky GBIC/SFP vendors small-time passive optical vendors expensive equipment for testing (spectrum analyzer, light sources, etc) lack of operational expertise (get hit by a bus) Exotic options Circulators (same wave both ways) Interleavers (half the light, double the waves) CWDM light into DWDM channel (similar to above) 10GE LX4/LR multiplexing Simple Economics 2GE GWDM ~$1k 8GE CWDM ~$5k-10k 2*10GE ~$5k-10k N*10GE DWDM ~N*$10k prices include passive and active components, per end, fdx over one pair Prices an order of magnitude lower than commercial systems from Cienna, Cisco. List of vendors Cloudy YAYA, Orient DONG, [lots of names on slide, go read it yourself] Questions? mail them! alex@pilosoft.com asr@voxel.net np@sonic.net Q: Martin, what do you about timing? A: No need for timing, each channel is separate, no timing needed to run this. Q: mike hughes, linx; one thing to look at if you're looking at GWDM/WWDM, or going bidir on one strand, watch out for back reflections--running several channels bidir would see itself reflected back, would declare linkup A: don't run two waves bidir on it--just don't do it, it's not worth it, it's too ghetto. NEXT: Mohit Lad Alerting prefix owners of hijacks in near-realtime UCLA, joint work with a bunch of other names [slides are at: http://www.nanog.org/mtg-0606/pdf/lightning-talks/5-lad.pdf PHAS project? Three properties of a security solution ability to see "bad" information ability to distinguish between "good" and "bad" info incentive to fix the problem The PHAS (prefix hijack alert system) approach use updates from existing BGP monitors (route views and RIPE RIS) if false origination, send notification. push complexity of detection to user look at email registration to decide who is allowed to announce prefixes. don't filter out false vs real changes. PHAS origin monitor 131.179.0.0/16, UCLA block recommend multiple email addresses, including some that are *not* on your blocks! Message Delivery apply local rules before generating alarms you shouldn't recieve duplicates of notifications due to topological mesh-ness, it's difficult for a hijacker to get all notifications for a block. Evaluation: messages per AS Dec 2005 map prefixes to origin AS using routing table most AS receive less than 100 messages per month most less than 10 local filters can limit legitimate origin changes. readily deployable routeviews and RIPE RIS already collect data alarm generation not dependent on cooperation from other networks monitoring or knowing correct origins alarm authentication: single source low overhead. summary comprehensive study using archived data developing near-realtime system interested in receiving notifications send email to: mohit@cs.ucla.edu massey@cs.colostate.edu ongoing efforts covered prefix hijack false last hop reference: PHAS: usenix security 2006. Q: Danny McPherson--that's associated with origin AS, and origin AS could be spoofed, does it look at combination of prefix, origin, and next hop up? A: they are doing it on origin AS and next hop, they'll do some more thinking about that case. NEXT: Rick Wesson, Support Intelligence [hehe] Understanding abuse, aggregate it, push it back to operators, let them know what they're doing to other people. [no slides, he does a live presentation of his tool] How do I believe you? realtime data visualization, Feb 8th, 2006 visualization. 130 different data sources, 90% passive; 10,000 domain aggregated spam trap, very evil SMTP that filters and bans IP for some time. 1.2million events per day aggregated, about 700,000 unique IPs for the global internet. BGP peers, aggregate based on announcements made. Put into tool so network operators can visualize their prefixes, drill in, and see abuse each prefix generates. hover over point, it shows the operator, IP address, and what the problem was (spam, insecure web server, etc) This shows problem areas that need to be addressed! disseminate this information, help ISPs clean up their networks. Can also pass along information of abuse that has happened to you. If you have an AS, he can tell you what your AS has been used for, abused for, owned, etc. email him for more info...except he didn't list his email info. ^_^; Break! short!