- Test - lists.nanog.org

Re: a fun hijack: 1/8, 2/8, 3/8, 4/8, 5/8, 7/8, 8/8, 12/8 briefly announced by AS 23520 (today)
by Josh Karlin 07 Jun '06

07 Jun '06

> Wonder if it was intentional or a 'classful' issue. This is why we (Level > 3) and ATT announce the /9s of 4/8, 8/8, and 12/8 :) > > -Kevin The /9s were stolen too, as well as a host of other prefixes. I just listed the biggies that I was pretty sure didn't belong to 23520. No clue if it was intentional or not, but I would also like to know. Josh

1 0

2006.06.06 NANOG-NOTES DDoS attack information collection
by Matthew Petach 07 Jun '06

07 Jun '06

Information collection on DDoS attacks, Anna Claiborne, Prolexic Technologies. [slides are at: http://www.nanog.org/mtg-0606/pdf/anna-claiborne.pdf DDoS mitigation service. personal experience mitigating over 150 DDoS attacks. Popular topic, but nobody talks about how you can defend yourself or take legal action; only thing you can do is collect information. 0.1% of DDoS attacks end in an arrest, that's out of the reported number to the US Secret Service, and that's out of the ones that fall into their jurisdiction. These are real losses: A major US corp lost over $2mil in a 20 hour outage An offshore gambling comp. lost estimated $4m in 3 days Online payment processor lost $400,000 in 72 hours online retailer lost $20K/day over 3 weeks. These are directly reported losses; doesn't include lost PR, etc. Canadian retailer spend 50K on hardware mitigation, they got kicked out of 3 datacenters due to the DDoS attacks, spent 20K on IT and security consultants, and another $6K on a different mitigation that also failed. Basic Information Collection Get packet captures--either from machine being attacked, or a span port, or from upstream device, tcpdump -n -s0 -C (get full length of raw packet, limit pcap file to 5MB or smaller) take 3 or 4 over 15 minutes, to start, and then repeat every hour Determine the type of attack and duration (ex SYN flood lasting 6 hours) Obtain as complete a list as possible of source IP addresses Save bandwidth graphs, flow data, pps graphs, any and all visual material relating to the attack Save any contact with the attacker, email, chat conversation, phone calls, etc. Get loss figures from management--downtime, per hour losses, per day losses, section 18 of some law, have to substantiate losses over $5k before you can take legal action against someone. Recommendations have a plan! DDoS is stressful Put all attack information in a central location God monitoring doesn't have to be expensive, a simple fiber card in a 1u box can be a mirror port for a large volume of traffic Don't have to have expensive hardware like arbor boxes. Limit to 100mb to prevent killing your capture box. Graphs and flow data can be retrieved from upstream Find the source Use list of source addresses, find a reputable hosting company, you may even see a friend's IP Approach the network with the infected machine, give them as much information as possible, it can take time finding someone willing to help Obtaining information is dependent on who you are dealing with, be as helpful as possible. Get information from the infected machine netstat, tcpdumps, who is logged in, web logs, access logs Get and save the source code responsible process can take hours to weeks--prolexic has huge contact list, and even for them can be really difficult And SAVE all your information to a central location! and back it up! Examine the source code scripts are best, you know exactly what's going on compiled code, run strings on it best case, you can get a name or identification for who wrote it, passwords, domain names, port usage worst case you can obtain information that doesn't make sense...yet (it may fit into a bigger context later) Locate controlling server Examine TCP connection table or source code to find the controlling server verify your information, scan or connect to the suspect machine contact abuse where the server is hosted, explain the situation have as much information possible to verify your conclusion and validate your identity Good luck, most abuse contacts are less than helpful Raises a good question: how to improve awareness and legitimate requests answered. (may be able to get FBI to provide warrants to seize machines that are being used to control attacks against you, but takes time and documentation) Hunting the attacker (not for the faint of heart!) Review all information gathered so far on the attack contact the attacker, establish a report save all information and/or conversations (important note, if conversations aren't on a public server, they can't be used) Piecing the information together to form a high level view of the exploit, attack, and attacker A long process, most attackers are highly motivated and skilled, you usuallly have to wait for them to slip up! Resources: local FBI field office department of cybercrime department of homeland security CERT Cymru--great guys, if they have to help you NHTCU--EU, cyber crime divisions in local offices Local US secret service--division of electronic crimes DDoSDB.org -- under development at the moment. how to identify/recognize different types of attacks may be able to put their attack database open to the public up there. A success story The tracking of x3m1st/eXe responsible for hundreds of extortion based DDoS attacks tracked for months eventually lead to his arrest. hid behind four levels of compromised servers. eXe and his group only talked on private IRC servers; made the mistake of connecting from his home domain, from a machine registered to his real name; that was his slip up, Ivan arrested in Russia. Tracking Pkeglhema/aaabaa targetted redhat linux boxes for his zombies they generally sat on higher bandwidth links. PHP/cross scripting vulnerability; insert the script without validity checking. Used cpanel holes, mySQL holes, he browsed zeroday, modified code in a few hours to use new holes, The result: synflood over 10G, knocked upstreams off, and got them null routed, bunch of outbound networks also null routed. some conversations recorded, he was paid by an employer, he'd done this before for other employers. He eventually got away. English as a second language, always from hacked university, attacking six other sites that also sold similar items as the client under protection. They'd had phone calls from competitors trying to push them out of business, and was during the busiest time of year for them. He was most professional attacker she's dealt with, he never slipped up, he'd been doing this for years. Logged in from China or Japan. She turned over info to FBI, let them pursue things further. Matters to address in community Better abuse contacts, specific to DDoS Centralized repository specifically for DDoS profiling Information gathering is extremely resource intensive, but worth it. Null routing IP space is not a good idea from either side DDDoS is everyone's problem. fix your open recursive DNS servers!! NHTCU--Mike Hughes, rolled into SOCA, serious organized crimes something--DDoS is way down on the list, they're more into big crimes. Watch for more developments in that space though. NHTCU was more approachable, Q: Bill Woodcock--could she talk more about public vs private IRC servers---what is the legal issue? A: private IRC server is any run that is not publically accessible, is only open to the group. Any machine that is hacked is a private IRC server, since it is not intended for public access. public--a machine run so that anyone can connect to it, and intended as such. You can assert the conversation, but it is hearsay; it can help in court, but it is itself not admissable as evidence. Q: Tony Kapella, 5nines--what does Prolexic suggest customers do to make sure their host hasn't been compromised to the point where netstat and other utilities are affected? A: Well, you have to trust the people you work with to be able to verify that the information they're seeing is accurate. But for boxes that neither side has access to, like colocation boxes, you could just be out of luck. Q: Gene Kim? what if the server is located outside the US? A: fine as long as it's publically accessible. Q: What about private messages? A: fine as long as it's a public IRC server Q: Louis Lee, equinix--suggest mirror switch port to address Tony's issue; capture unaffected traffic to a virgin machine when possible. Q: Rob seastrom, bluetrust--what is the incidence of encrypted communication, and multiple C&C hosts? A: This only works for easiest case scenario of non-spoofed attack with centralized C&C attack. Peer to peer, proxy servers, etc. you need to go to an expert. Q: Stuart Phillips, New Metra...he's cut off...raise it at the security BOF. Announcements--if you've not picked up your shirt, pick it up, JD Frazer, userfriendly did the image. A few short of goal of six for lightning talks; sign up, or we'll have Randy sing at you. PGP signing during this break too. Be back at 10 after.

1 0

Subject: Found power supply at NANOG37
by Duane Wessels 07 Jun '06

07 Jun '06

Found: HP laptop power supply left on a large round table late tuesday night in the main hallway. Here's hoping you have enough juice left to read this email...

1 0

ipv6 @ sprint, somebody home?
by Jeroen Massar 07 Jun '06

07 Jun '06

<ipv6(a)sprint.net>: host kay.sprintlink.net[199.0.233.8] said: 553 5.3.0 <ipv6(a)sprint.net>... User Unknown (in reply to RCPT TO command) It's must be 6/6/6 that it ain't working. I guess they are scared that IPv6 might scare their fisherprice routers ;) Anybody a *working* contact so that they can also be nicely reminded of the fact that the 6bone has come to an end and that they should nicely ask their "paying customers" to stop announcing 6bone space? http://www.sixxs.net/tools/grh/lg/?=prefix&find=3ffe::/16 And http://www.sprintv6.net/ doesn't contain any contact info before you say "google" it. Then again the following url clearly shows their 'interrest' http://www.sprintv6.net/aspath/bgp-page-complete.html Last change on the tree detected on Sun DEC 11 2005, h.22:50 Fisherprice, fisherprice.... </sarcastic mode> Greets, Jeroen -- Just in case, yes ipv6(a)sprint.net is whois 3ffe:2900::/24 : ipv6-site: SPRINT origin: AS6175 descr: SprintLink 6bone Reston, VA, US country: US prefix: 3FFE:2900::/24 ... contact: RJR2-6BONE remarks: This object is automatically converted from the RIPE181 registry notify: ipv6(a)sprint.net changed: auto-dbm(a)whois.6bone.net 20010117 changed: rrockell(a)sprint.net 20010423 changed: hgoes(a)eu.uu.net 20020925 source: 6BONE And no I am not mailing their ipv6-support addy which doesn't respond ;) PS: Happy Slayer Day: http://www.nationaldayofslayer.org !!!

5 4

2006.06.06 NANOG-NOTES network-level spam behaviour
by Matthew Petach 07 Jun '06

07 Jun '06

2006.06.06 Nick Feamster, Network-level spam behaviour [slides are at: http://www.nanog.org/mtg-0606/pdf/nick-feamster.pdf Spam unsolicited commercial email feb 2005, 90% of all email is spam common filtering techniques are content based DNS balcklist queries are significant fraction of DNS traffic today. (DNSbls) Using IP address based spam black lists isn't so useful. How spammers evade blacklists will be discussed as well. Problems with content-based filters ...uh oh, some technical glitches... Content-based properties are malleable low cost to evasion altering content based on scripts is too easy customized emails are easy to generate content based filters need fuzzy hashes over content, etc. high cost to filter maintainers as content changes, filters need to be updated. constantly tweaking spamassasain rules is a pain. false positives are always an issue. Content-based filters are applied at the destination too little, too late -- wasted network bandwidth, storage, etc. ; many users recieve and store the same spam content. Network level spam filtering is robust (hypothesis) network-level propeerties are more fixed hosting or upstream ISP (as number) botnet membership location in the network IP address block country? are there common ISPs that host the spammers, for example? Avoid receiving mail from machines that are part of botnets. Challenge--which properties are most useful for distinguishing spam traffic from legitimate email? very little if anything is known about these characteristics yet! Randy gave a lightning talk last NANOG about some of this. Some properties listed. Spamming techniques mostly botnets, of course other techniques too we're trying to quantify this coordination characteristics how we're doing this correlations with Bobax victims from georgia tech botnet sinkhole other possilities: heuristics distance of client IP from the MX record coordinated, low-bandwidth sending looked at pcaps coming in from hijacked command and control station from bots trying to talk to it; spamming bots, Bobax drone botnet, exclusively used to send spam. Collection two domains instrumented with MailAvenger (both on the same network) sinkhole domain 1 continuous spam collection since aug 2004 no real email addresses--sink everything 10 million + pieces of spam sinkhole domain #2 recently registered Nov 2005 "clean control" domain posted at a few places not much spam yet--perhaps being too conservative contact page with random email contact, look at who crawls, and then who spams the unique email addresses Monitoring BGP route advertisments from same network Also capturing traceroutes, DNSBL results, passive TCP host fingerprinting, simultaneous with spam arrival (results in this talk focus on BGP+ spam only) Mail Avenger, not an MTA, it forks to sendmail or postfix, it sits in front of MTA, does things like do DNSBL lookups, add headers, passive OS fingerprinting, as the spam is arriving. Also logged BGP routes from same network that got the spam; see connectivity to the spamming machine at the time. Picture of collection up at MIT network. Mail Collection: MailAvenger X-Avenger header. best guess at operating system, POF, DNSBL lookups, traceroutes back to mail relay at the time the mail was sent (used for debugging BGP) distribution across IP space plot /24 prefix vs how much spam coming from it. steeper lines mean more spam from that part of the IP space; you can see where spam is coming from. bunch comes from apnic, cable modem space, etc. few interesting things to note; still redoing legitimate mail characteristics. from georgia tech mail machines, it's legit plus spam, need to split out better. between 90.* and 180.*, legitimate mail mainly. Is IP-based blacklisting enough? Probably not: more than half of spamming client IPs appear less than twice. Roughly 50% of the IPs showed up less than twice; but that's a single sinkhole domain, would help more across multiple domains. emphasizes need to collaborate across multiple domains to build blacklists; any one domain won't see repeated patterns of IPs. Distribution across ASes 40% of spam coming from the US BGP spectrum agility Log IP addresses of SMTP relays Join with BGP route advertisements seen at network where spam trap is co-located. A small club of persistent players appears to be using this technique 61.0.0.0/8 AS4678 66.0.0.0/8 AS21562 82.0.0.0/8 AS8717 somewhere between 1-10% of all spam (some clearly intentional, others might be flapping) about 10 minute announcement time of the /8 while spam is flooded out. Might be interesting to couple this with route hijacking alerting to filter out if this is really a hijacking vs a flapping legitimate route. A slightly different pattern; announce-spam-withdraw on a minute-by-minute basis. really really egregious! Why such big prefixes? flexibility: client IPs can be scattered throughout dark space within a large /8 same sender usually returns with different IP addresses visibility: route typically won't be filtered (nice and short prefix length) Characteristics of IP-agile senders IP addresses are widely distributed across the /8 spce IP addresses typically appear only once at the sinkhole Depending on which /8, 60-80% of these IP addresses were not reachable by traceroute when we spot-checked some IP addresses were in allocated, albeit unannounced space Some AS paths associated with the routes contained reserved AS numbers Odd AS numbers injected, usually well-known to make it look more legitimate. Length of short-lived BGP epochs 10% of spam coming from short-lived BGP events Spam from Botnets Example: Bobax approximate size: 100k bots one sinkhole domain--this is ONLY stuff that is verifiable as coming from bots via command and control hijacked IPs, intersect the single sinkhole domain, so much smaller data subset, but well correlated and verified. Proportionally less spam from bots in 61-90 range; that tends to be where BGP route hijacks happen instead. Most Bot IP addresses do not return 65% of bots only send mail to a domain once over 18 months. Some hang around for a *long* time. About 20% stick around for several months. collaborative spam filtering seems to be helping track bot IP addresses. Most bots send low volumes of spam most bot IP addresses send very little spam regardless of how long they have been spamming Effectiveness of blacklisting: only about half of the IPs spamming from short-lived BGP are listed in any blacklist spam from IP-agile senders tend to be listed in fewer blacklists Looking at 8 different spam blacklists, checking when the spam arrives at the sinkhole. Known Bobax drones listed in more DNSbls than the BGP agile senders. About 90-95% of the Bobax bot drones are listed in one or more DNSBLs. Suggests some of the spamming bots are listed more than other techniques--that is, bots are easier to identify than BGP-agile spammers or spammers using other techniques. Harvesting tracking web-based harvesting register domain, set up MX record post, link to page with randomly generated email addresses Example Phish: a flood of email for a phishing attack for paypal.com all to: addresses harvested in a single crawl on January 16th 2006 emails received from IPs different from those who crawl. X-mailer headers totally diffrent. Lessons for better spam filters: effective spam filtering requires a btter notion of end-host identity distribution of spamming IP addresses is highly skewed detection based on network-wide, aggregate behavioru may be more fruitful than focusing on individual IPs large, emergent properties. two critical pieces of the puzzle botnet detection securing the internet's routing infrastructure compare distributions of spam to legitimate mail, see if certain spaces are more likely to send spam than legitimate mail. Questions: Q: Steve Bellovin, columbia university bots from strange ASes, is tunnelling taking place from bots to BGP speakers? A: Not sure if there's evidence or not; some data from TORS?? but TORS latency may be too high. Q: Fingerprinting to try to identify who is doing things, see how many hosts are actually doing this? Many addresses being used, how many hosts does it actually represent? A: Not sure, haven't checked that. Haven't checked on aliasing, since not much was seen from a single IP. NAT'ing? What about hosts hopping? (same host using multiple IPs?) Not sure, they didn't do that correlation. Q: Randy Bush, IIJ, they did do OS fingerprinting, so some of that are in the paper. didn't do anything with the traceroutes, though. Q: Matt asks what the difference between the two domains was; was one of them a recognizable word or name, or were they both random character strings? A: they were both random character strings, but one of them had been used to host a real website for a while, which might explain why it gets such a huge volume of spam compared to the other. Q: Matt points out that for some networks, receiving spam is actually a good thing, as it helps balance out traffic ratios, which helps during peering negotiations. Q: Randy Bush, IIJ, responding to Matt about traffic ratios: only those backbones who are on ADSL should they care which way traffic goes. :P Curious to work with large networks, see if filters could be installed to detect it, and possibly take action.

1 0

RE: Zebra/linux device production networking?
by Mark D. Kaye 07 Jun '06

07 Jun '06

Hi, I am also newbie poster so likewise plz be kind. I tend to agree with the comments made so far, however depending upon the business, budgets are not always available that might match the requirements and hence I can to some degree understand the use of such boxes for small organisations. I would be interested to know how many "software" (for want of a better description) routers are in live production in this kind of environment i.e. the 99.9999% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. Also does anyone have any peering policies which would exclude peers with "software" routers specifically, most have a requirement for the ability to support stable BGP peering but I have not seen any specific exclusions for such "devices"? Mark ________________________________________ From: owner-nanog(a)merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Tiffany Snyder Sent: 06 June 2006 23:29 To: Nick Burke Cc: nanog(a)merit.edu Subject: Re: Zebra/linux device production networking? IMHO, it's a bad idea. A less intrusive alternative might be a FreeBSD based platform running Xorp/Quagga. Tiffany. On 6/6/06, Nick Burke <mrmud(a)mrmud.org> wrote: Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found "many people" who do this for their "core devices". I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_t c024.htm and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)

3 2

2006.06.06 NANOG-NOTES DNS reflector attacks
by Matthew Petach 07 Jun '06

07 Jun '06

(I was going to try to get all the notes from today's panels out before going to bed, but I fell asleep on my keyboard finishing up these notes, so I think I'm going to wait and send the batch of Tuesday and Wednesday notes out after things wrap up on Wednesday. Sorry about the delay, but I need a bit more sleep I think. ^_^;; --Matt) 2006.06.06 Morning welcome, and introduction of Chris Morrow, panelist Please fill out survey today if you're going to be leaving! Frank Scalzo, Verisign Recent DNS reflector attacks. Attacker breaks into innocent authoritative DNS server, publishes large text record; then does queries from zombie army against that record, with sources spoofed with victim IP. 5 gig attack, 2.2G made it, 3gig didn't. E.TN.CO.ZA DNS attack, 64 byte query, 63:1 amplification, 4028 byte answer 34,668 reflectors. Victim sees 5G of traffic, 144,142bps per reflector, 13.5packets per second 4.5DNS answers per second. reflectors won't see this as anomolous for the most part; top talker only sent 8.5 answers per second. No visibility into the attacker at all, but best guess was 79Mb of source generated 5GB of responses. Record was maliciously installed; 2 auth servers, 1 compromised; 65% response, 35% name error. Answer comes in 3 fragments, larger than normal MTU. Attack came in 3 phases. first port 666, then port 53 and 666, then all 53. Port shifts are nearly instant, so fast command and control system in place for it. Filter out open recursive DNS servers; you can't put ACL in for 500,000 DNS servers. What about limiting DNS packets to 512 bytes? will break things. What about blocking 53 outside of your network hierarchy, force people to use your resolvers? What about discarding fragments? Challenge is getting your upstream to implement it, unless you have hardware and pipes to handle the flood coming at you to start with. Some ISPs won't do it unless they see live attack traffic, and a 24 minute attack is too short lived for ISPs to see and react to. data from Jan 11 - Feb 27 this year. Attack queries/second consistent with avg reflector qps. one reflector sent 1.9M DNS answers to 1593 victims, 605 different queries to generate answers. 180TB of attack traffic on Feb 1st. after feb 15th, ramped down. Assume 4KB response packet, see attacks between 3G and 7G, the scary part is that it only took 130Mb to generate the 7G attack, and the 3 gig attacks are all from less than a fastE connected compromised web server. 500,000 reflectors with 2G source could generate a 120GB DoS attack. Top victim got over 130Tb of attack traffic, top bunch are all over 100Tb 65,461 ports used, Top port is less than 10% of traffic though top 20 domains used, mostly innocent bystanders. Internet root . was second highest domain used; certainly can't filter *that* out. Fundamental challenge; UDP lacks 3 way handshake, easy to spoof DNS is easy target, so many unsecured DNS servers Other UDP servers need to be evaluated as well DNS closing 500,000 open recursive DNS servers will be very, very painful. poor separation between authoritative and recursive DNS servers. BIND allow-query ACLs, recursive DNS servers should not accept queries from outside. What if it's an embedded system like a wireless gateway? We depend on large records for DNSSec, etc. Beyond open recursive DNS servers root domain "." was used most authoritative name servers will answer with an upward referral doesn't include actual IPs, but it's still 438 bytes, and pretty much every DNS server responds to it. Source validation IETF BCP 38 How do you manage 70,000 ACLs on 500 routers? what about people who are multi-homed with static routes? what about legacy stuff that works but shouldn't? strict RPF breaks with traffic asymmetry; loose RPF doesn't help with this. ISPs see the problem as long, hard, expensive to overcome, and they're right. If we never start trying, we'll never fix it! Close open recursive DNS servers DNS servers should include filtering SOHO router vendors should fix their DNS proxy code, don't listen on outside interface BCP 38 otherwise we'll be jumping from protocol to protocol. Questions? Q: What does verisign do to protect their DNS servers? A: Anycast, massive peering and transit capacity Q: Jared Mauch, NTT/America; he turned on unicast rpf on the NANOG upstream link. 372,000 packets that people here have sent failed the RPF check. BCP 38 is hard Paul Quinn asked what percentage of the traffic that is. Bora Akyul, Broadcomm--any data on source ranges on the packets being seen? He could look at the 1 in 10,000 netflow sampling to see, but the individual link is a /30, looks like a normal customer link. The Merit router isn't RPF'ing either. Q: Ren Provo asks when they will peer; A: not yet, next few months, Miami Terremark, and other sites domestically and internationally in next year and a half.

1 0

Fwd: At Dig BIG, Nothing Runs Like a Deere (Really Backhoes and Fiber together)
by Robert Boyle 07 Jun '06

07 Jun '06

All, Just in case any of you want to see how the other half lives... and destroy some infrastructure after learning more about how to build it this week. :) > Work with >JOHN DEERE Equipment > to build REAL utilities in a REAL work setting. > >Dig BIG takes place at the best place on the planet for underground >working and learning - The Planet Underground. > >For two BIG days, The Planet Underground becomes a huge work site >with acres of REAL underground utilities. > >You won't want to miss this REAL BIG opportunity to use the John >Deere equipment at Dig BIG. Work side by side with colleagues with >the best equipment available! > >Free registration available for a limited >time.<http://underspace.com/mailer/redir.php?id=239&st_id=164> I subscribe to Underground Focus magazine and I just received this in my inbox. Register at http://www.underspace.com/TPU/register_digbig.php in case you really do want to attend. Now back to the regularly scheduled noise with some content due to NANOG 37. -Robert Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin

1 0

Lightning talk selections
by Steve Feldman 07 Jun '06

07 Jun '06

The Program Committee has selected these lightning talks for presentation during the general session tomorrow morning: Analysis of DNS Root Server Location, Martin Hannigan Metro WDM in provider networks, Alex Pilosov Fashonably Late - What Your Networks RTT Says About Itself, Anton Kapela Thepiratebay busted - network impact, Mikael Abrahamsson Alerting prefix owners of hijacks in near real time, Mohit Lad Reigning in the botnets operating on your network, Rick Wesson Thanks to everyone who submitted a proposal, unfortunately we couldn't fit them all into the available time. Steve Feldman PC chair

1 0

2006.06.05 NANOG-NOTES Peering BOF notes
by Matthew Petach 06 Jun '06

06 Jun '06

(This time around I opted to go to the peering BOF and take some notes. It's the one downside to parallel tracks--wish I could be in two places at once. ^_^;; --MNP) 2006.06.05 Peering BOF Bill Norton introduces the Agenda; unfortunately, my laptop took so long to boot, I missed the Agenda slide. Doug Toy?, Transit Cost Survey, data collected at NANOG 36; he's just here to present the collected info, not really representing anyone. Recap: At NANOG 36, people indicated their cost per Mb and commit level. length of contract was usually 1-2 years. 42 data samples collected avg $25/Mb $95/$10 were the extremes. Avg commit level 1440 Mbps Other observations as expected, cost per Mbs tends to decrease as the commit level increases. Tier1's are more expensive Cost tends to vary more with Tier 1 providers than with others. between 0-500Mb commit level, prices are all over; at higher commits, prices level out at the bottom. Question: Mbps, is that the cap, the usage, inbound plus outbound? A: That's the general 95th percentile higher of the two inbound and outbound. Committed amount. Graphs tend to approach a hard bottom; as commit increases, doesn't change all that much. Bottom is around $10/Mb, even though commit levels increase. Of samples collected, 2/3 were from Tier1 providers. 90% of contracts are 1-2 year in length, so didn't cause much variance. Tier 1 definition is based on Wikipedia definition. Questions from audience? Q: Data looked pretty clean; were there samples pulled out to make it look cleaner? A: No, other than people who left fields blank on the survey. Q: was there a timestamp of when contract started? A: much of it wasn't complete. Mostly within the 1-2 year range for length as well as start date, so nothing really ancient in there. BillNorton; people had some concerns about violating NDA or contract details when filling out the survey. Where do we draw the line in doing these types of surveys? SteveGibbard; NDA is agreement between transit provider and customer, and this was anonymized and voluntary. Data is interesting, both for purchasers and sellers of transit. Q: 42 samples graphed, there were 80-100 people in the room at the time; so the real comments from the rest of people weren't counted? A: No, there were less than 50 submissions total, of which 42 were complete. Q: Patrick; many people put more than one transit provider on their form; how were those other transit providers handled? A: no clue, he just got a spreadsheet with data. Back to Bill Norton Peering Lists Issue -- make available to customer prospects? 15 mins. Peering disclosure dilemma: customers often asked for peering list, sometimes peerings restricted under NDAs Metric for determining connectedness, capacity, resiliancy. Is there a better metric for customers? IX capacity in/out Peering pipe size? ISPs are getting commonly asked about this, based on hands raised in the room. How many people lose business because the customer doesn't get an answer? Sylvie, VSNL notes they provide the info when they're under an RFP; they won't give capacity, they give an aggregate, they won't go peer by peer, that would be a violation of NDA. BillN: are the NDAs written to allow total numbers like that? Sylvie: you should not disclose capacity per location or per circuit, but they don't forbid aggregate total numbers. BillN: is there something else that could be given to the customer that would satisfy their question without revealing what Chuck: A lot of ISPs lie about their peerings; he runs AMSIX, people claim to have multiple gig to the peering exchange, he knows they don't really have that much. Patrick: but he can look at the peering stats on AMSIX--Chuck notes only members can. Patrick: customers ask how many gigs they can send to a provider; it's available headroom, so they ask their upstreams how much available headroom is left. Most providers are having a lot of trouble getting the right capacities to the right networks. The reason many don't answer is they don't like the answer they have to give. Ted Seely, Sprint, how do you solve the problem? There's lots of traffic that needs to be exchanged, how do you fix it? Patrick: how about everyone upgrade to 10gigE in many places? If you can't afford it, stop selling bandwidth. But most people can't go to all the different providers, they have to buy from a small subset of providers. RAS: No technology problem with doing it; it's the money. Not charging enough to cover the costs of the technology you need to install to cover the bandwidth. Ted notes you can't just link at one spot, you have to connect at six places, and you need to have links in and out of the site to support the volume, etc. Patrick: can you tell us how much they exchange? 40Gb times 6 providers in six locations is probably more traffic that Sprint has in total. Ted Seeley; it's a time scale issue--yes, it can be solved, but in what timeframe? BillN feels it's reasonable information for a customer to request. Is there a proxy that customers could use that would give similar information? Tristan Horn, Q: has anyone use skitter to try to figure out the answer on their own? Ted: last skitter pass was March, 2005--data isn't current enough to be useful. Michael Abrams, tcp test, consumers can run to see how much traffic they can pass along; it becomes very quickly apparent who has problems, and consumers tend to reach quickly to that, go elsewhere. Shows packet loss and latency at different throughputs. Sweden, size of CA, they have 4 exchange points, and want more. Still gamers are most vocal about any problematic interconnects. Video Peering Video Stats and distribution models -- all YouTube Peering Personal -- Colin They're getting into the peering fray, and only a year old. This is gigs and gigs, has potential to dwarf current peering traffic. Current issues could be tiny compared to the flood of potential issues when hundreds of Gbs comes flooding towards customers. Voxley? on the web; 320x240 pixel image, want to do a one hour episode, will be about 210MB file. currently, show is watched by 10,000,000 households. So, that's 2.1PB to transfer all those episodes. If you try to stream it, via say iTunes, over 3 days, would be about 64Gbps solid for 3 days. That's a pretty big number. That's one episode, from one show. American Idol is even bigger. He claims a better way is using peer to peer. 2.5% of the agents could be propagation nodes, or 256,000 PCs as transfer agents; that would fit on existing broadband connections. What about multicast? problem at core, end distribution; pushing to 256,000 is much more doable. Ted Seeley notes you still have to pass the unicast streams eventually, so you still haven't solved the issue. If you don't do multicast or caching at the edge, you've just shuffled the problem. Swedish police, 100Gb of peer-to-peer traffic at peak, AMSIX lost 10gig, LINX lost 5gigs, probably lost about 40Gb weds/thurs last week when the swedish police shut it down. Which site? Pirate Bay torrent tracker. Peer to peer has similar replication pattern to multicast? But it's not localized properly, unlike multicast. RickW, the pirate bay site, that site was only distributing meta information. peer to peer traffic can be 12-20% higher because of the duplicated traffic. Peer to peer networks aren't as efficient as unicast streams. Matt notes 64Gb, 256,000 subscribers isn't that big a stretch anymore. Peer to peer is unlike multicast, because it doesn't have any localization to the best of his understanding. Do peer to peer systems use latency to control who they share with? Somewhat; faster peers are selected over slower peers. Comment from audience is that live events are still going to be the challenge; HDTV is getting gb/sec from cameras, needs to feed it out, no chance to cache, so multicast ends up being the only viable option for it. Question about is the assumption that getting it over the internet is the right model, vs using settop boxes? How do you get a whole country to change their viewing habits? Korea Telecom uses peer to peer to pass content around; it is limited to a certain DSLAM or POP, which answers Matt's localization/topology issue. Ted Seeley; yes, people are watching material online vs set top boxes, the paradigm is shifting already. RAS, works through numbers; a DVD of MPEG2 will be around 4GB, if you drop out the special bits, audio tracks, etc; heavily compressed, 600MB, to around 1GB mpeg4 for DVD quality, 2-4GB compressed for HD quality. if we take 1000MB filesize, cost per movie to deliver this. Assume $10/Mb, you get 1Mbit/sec/month for $1, Assume normal traffic curve, 150GB/month for your per Mb $10 point. Assume it's mpeg4 compressed to 1GB, that's 150 movies per $10/month, that's less than the cost of postage. How does it best make sense to distribute these files? Compression reduces file sizes about 15% per year. interstreamtv.com, has article about it. 1Mb/sec will do 320GB/month. So number of movies would vary. Peer to peer want immediately, and want quality, so the model might be problematic. Caching at the pop edge would help. And live events, multicast would help, use unicast to fix lost packets. How do you compensate for lossy networks? Forward error correction, which means you lose the benefit of compression you were hoping to get. Larger buffer won't help for retransmits unless it's TCP. Curtis points out live streaming vs Richard's downloaded movies. RAS was replicating Netflix model, you order in advance, get it later that day. If you want to stream live stuff, it needs to be multicast, provider needs to do the replication. How much of it is unavailable in the country where the provider is located, that's why you can't cache or work with the provider to distribute it. Brokaw notes more streaming is coming; more video stuff will be coming in the months and years to come. For the access networks, how many are multicast enabled, and are ready for helping this scale? 3 hands. Does the community agree we have a bit of a challenge here? Matt Peterson, encryption to make sure the file is different for each user, to make sure it's legal. Bittorrent with RSS to get metadata, weather channel model, ultraDNS, Akamai model multicast model What about skycache? One-to-many distribution? Colin from YouTube will start off the peering personals. They've recieved an AS, they push about 20Gb sustained, their first datacenter went live in the middle of March. It is definitely growing. Not multicast enabled, no demand for it. The average show they have is 4 minutes, hundreds of millions of different videos, not a good multicast target. 20% growth month to month, 50 million videos viewed per day. 20gbit/sec outbound? "Ask a ninja net neutrality" video. Product launched in December, that's when they saw real traffic; been moving more and more in house in March and April. They're pushing 20gig in house, not from CDN network. This is is all short form video content, hopefully no licensed content sneaking in. San Jose Lundy on equinix fabric PAIX on paix fabric Los Angeles on equinix fabric Linking all 3 locations together. Metro, long-haul providers take a long time. Lundy is poorly connected. Getting connectivity at 10 gig (Fiber swaps, metro links, cross datacenter crossconnects) Other items Starting to see traffic from v6-v4 gateways most hardware providers do v6 support in software Seeing v6 traffic from v6-to-v4 gateways from Asia. Are people doing much successful v6 on modern devices? Force10 s50, 4500 from cisco, all seem to be software based v6. Should they have a native v6 presence, go dual stack, what hardware would be good to use? sup720 will be the cheapest hardware to do hardware v6 at line rate 10 gig. Can't do it on the lower end. Person from puerto rico notes that hardware switches can do 200Gb/sec of IPv6 switching, why can't routers do it? Frustrated audience reactions from all over about differences between switching and routing. Robert Seastrom--should you do v6 at all? Should you be a pioneer, and make the v6 people happy, sure, do it; if you want to make money, no. Q: how do they make money, and will that model support their bandwidth? it's advertising supported, hopefully it will support their needs. Q: how do you compete with Google? No idea, he's not a business person. Q: Content providers often don't get peers at public sites, much harder to get peers than access providers; are they running into that? A: Right now, peers are still coming to him; he won't know about resistance until he goes out asking for it. Summary: Open Peering Policy peering(a)youtube.com colin(a)youtube.com AS# 36561 How much of the 20gig is going across peering links vs transit links? It's all transit right now--he's just now asking about peering. Jeffrey notes that if there are people who pay transit to get your content, they will peer regardless of ratios. DanGolding notes that the tier1's are stuck on a treadmill; they can't peer with you even if they desire it, for fear of messing with their own ratios. Josh asks if they will go to other locations than just the west coast of US. Colin notes that they're doing a datacenter month. Chicago is next. Dan Golding, network neutrality on the peering front. One year old company, video content, already doing 20gig/sec. This is a buttload of traffic, and they're already getting into the peering mode; will this traffic ultimately dwarf the rest of our traffic? Network Neutrality, an open discussion. Tier1research. What it means to peer. network neutrality: an introduction what is network neutrality simply put, it's the idea that internet service providers should treat all legitimate traffic the same way regardless of origin and destination. Why should I care? a significant portion of the internet is based off mutual... The Bell perspective An unfair load the bell companies-att and verizon-aren't big fans of neutrality. they feels they're carrying traffic for others and are uncompensated or not compensated sufficiently Google, Yahoo, making big profits from the sweat of their brows. The Bell Answer $$$ needs to flow from content carriers to the Bells. How would Bells make this work again? everything old is new again. in the PSTN world, the bells charge reciprocal compensation for voice calls terminating on their networks. Talk of prioritization... [he's really whizzing through his slides!] Content perspective The internet works because of neutrality non-neutrality would stifle innovation The bells have a bad business model, they're asking content to help prop it up. Prioritization is a protection model; The bells are in a power position their control of the internet is at an all time high. the conglomerate controls a huge amount of access, they are tier 1 internet peers, SFI peering with each others. Others are tempted to jump on. MSOs see non-neutrality as a way to keep other's VoIP off their networks On the other hand, the Bells will squeeze them; the cable companies lack peering. His take; leans towards content. Would be fair, except their networks were built under monopoly auspices. Consumers have little choice in many locations some sympathy for bells; it's their network, after all. How will this end? possibility #1; regulation of internet to enforce neutrality. regulation makes the internet less flexible probably vague law with lots of FCC regulations Regulatory Affairs officer has to vet ACLs Difficult to do filtering quickly ability to provide tiered services will be more difficult. Possibility #2 bells get their way VZ and ATT become tier 0 internet carriers depeer everyone else as soon as current restrictions expire charging reciprocal compensation to all others where are the MSOs? other access providers? why buy transit from anyone other than a bell? potentially disastrous scenario internet growth is stifled forces you to always buy transit from the bell companies. Possiblity #3 threat of regulation causes a continuation of status quo. best solution loss of neutrality could destroy the internet internet regulation could stifle it; and the bells are VERY good at regulation. perhaps the evolution of tiered consumer services could come about by volume, not quality consumers warming to this model content providers sponsor some access models. Patrick Gilmore: Tier 0, how does that work when VZ and ATT don't make up the bulk of the internet anymore. What about the rest of the world? The bells get regulations changed when the world doesn't move in the direction they like; reciprocal compensation got removed when the ISPs became CLECs and got lots of compensation from the Bells. Now, regulations modified. Sean Donelan; it's net neutrality, not internet neutrality, so what about other players? broadcasters, radio, etc. For voice calls, not such a big issue. broadcast networks don't have much of an idea of what this is about; not sure they see the applicability of this yet. When they do, they have huge lobbying power as well. Bells have very sophisticated lobbying efforts, make them very smart. The newbies like Google and Yahoo aren't that savvy in washington yet. 60% liklihood that the end result is regulation; is *not* a happy outcome. Is there some pivot point that may make people change their models? Patrick Gilmore asks for non-US peering coordinators; who would care if ATT/VZ depeered you? Not many respond Curtis--not just a US problem; similar discussions in European countries. At what layer does the competition come into play? In European, there's no text on the table yet, but there's discussions taking place at least. Somewhat of a reverse problem, it's the new players looking at tax monies. AMSIX, much better competition on the last mile and the infrastructure, the big players can't play the game the same way. Dan Golding notes that if we had many different ways of getting local loop to your house, it would be less of an issue. Incent development of alternate methods; wifi, 3G/4G/5G networks, etc. Mikhail Abramson, with high speed cellular, mobile is making network neutrality less of an issue, since you do have more options. The GSM providers are happy with the internet bandwidth usage on mobile data, it's making them really good money. Matt Peterson? Last comment about cell phones; but people are used to packet munging; most people are used to a clampdown at this point; they may not even notice anymore. Until Grandma using YouTube understands network neutrality, it'll be this community only fighting/caring. If we all went to a common $10 provider, we could create a new tier0 and bypass the bellheads. A few peering personals. Jeffrey Papen, MySpace, AS 33739, well over 20gigs, in LA and by august, SJ, CHI, ASH, all Equinix locations. Needs to be in peeringdb.org!! People are now talking about putting peeringdb.org in their peering contracts. Stafford Rau, CLEC in Portland, ops, in WA, northwest, currently acquiring ELI from frontier, will be about twice as big. SIX westin, PAIX, AS12003, 7845, will roll all of it including ELI into 7385, they don't get 5650 unfortunately. 2-3gig transit now. Philip, MSO in Canada 5769, 13571, about 15gigs, videotron.ca Matt Peterson, SIX seattle, copy it in SF, neutral, non-profit, 365 Main, mostly content providers, CNET, craigslist, others. load up your blogs, SFMIX, 365 Main, he or Tim Pozar from UnitedLayer. MetaInterfaces, video, 5gigs, makes money, tells you what kind of video it is. ^_^; Vish from Netflix, looking for potential peers in SJC, CHI, ASH, vish(a)netflix.com really aiming for eyeballs. Ann claiborne, prolexic technologies, also in peeringDB, NOTA, LINX, peering(a)prolexic.com hoping to come to west coast. Dash to restrooms and then to the next BOF.

6 5