(I was going to try to get all the notes from today's panels out before going to bed, but I fell asleep on my keyboard finishing up these notes, so I think I'm going to wait and send the batch of Tuesday and Wednesday notes out after things wrap up on Wednesday. Sorry about the delay, but I need a bit more sleep I think. ^_^;; --Matt) 2006.06.06 Morning welcome, and introduction of Chris Morrow, panelist Please fill out survey today if you're going to be leaving! Frank Scalzo, Verisign Recent DNS reflector attacks. Attacker breaks into innocent authoritative DNS server, publishes large text record; then does queries from zombie army against that record, with sources spoofed with victim IP. 5 gig attack, 2.2G made it, 3gig didn't. E.TN.CO.ZA DNS attack, 64 byte query, 63:1 amplification, 4028 byte answer 34,668 reflectors. Victim sees 5G of traffic, 144,142bps per reflector, 13.5packets per second 4.5DNS answers per second. reflectors won't see this as anomolous for the most part; top talker only sent 8.5 answers per second. No visibility into the attacker at all, but best guess was 79Mb of source generated 5GB of responses. Record was maliciously installed; 2 auth servers, 1 compromised; 65% response, 35% name error. Answer comes in 3 fragments, larger than normal MTU. Attack came in 3 phases. first port 666, then port 53 and 666, then all 53. Port shifts are nearly instant, so fast command and control system in place for it. Filter out open recursive DNS servers; you can't put ACL in for 500,000 DNS servers. What about limiting DNS packets to 512 bytes? will break things. What about blocking 53 outside of your network hierarchy, force people to use your resolvers? What about discarding fragments? Challenge is getting your upstream to implement it, unless you have hardware and pipes to handle the flood coming at you to start with. Some ISPs won't do it unless they see live attack traffic, and a 24 minute attack is too short lived for ISPs to see and react to. data from Jan 11 - Feb 27 this year. Attack queries/second consistent with avg reflector qps. one reflector sent 1.9M DNS answers to 1593 victims, 605 different queries to generate answers. 180TB of attack traffic on Feb 1st. after feb 15th, ramped down. Assume 4KB response packet, see attacks between 3G and 7G, the scary part is that it only took 130Mb to generate the 7G attack, and the 3 gig attacks are all from less than a fastE connected compromised web server. 500,000 reflectors with 2G source could generate a 120GB DoS attack. Top victim got over 130Tb of attack traffic, top bunch are all over 100Tb 65,461 ports used, Top port is less than 10% of traffic though top 20 domains used, mostly innocent bystanders. Internet root . was second highest domain used; certainly can't filter *that* out. Fundamental challenge; UDP lacks 3 way handshake, easy to spoof DNS is easy target, so many unsecured DNS servers Other UDP servers need to be evaluated as well DNS closing 500,000 open recursive DNS servers will be very, very painful. poor separation between authoritative and recursive DNS servers. BIND allow-query ACLs, recursive DNS servers should not accept queries from outside. What if it's an embedded system like a wireless gateway? We depend on large records for DNSSec, etc. Beyond open recursive DNS servers root domain "." was used most authoritative name servers will answer with an upward referral doesn't include actual IPs, but it's still 438 bytes, and pretty much every DNS server responds to it. Source validation IETF BCP 38 How do you manage 70,000 ACLs on 500 routers? what about people who are multi-homed with static routes? what about legacy stuff that works but shouldn't? strict RPF breaks with traffic asymmetry; loose RPF doesn't help with this. ISPs see the problem as long, hard, expensive to overcome, and they're right. If we never start trying, we'll never fix it! Close open recursive DNS servers DNS servers should include filtering SOHO router vendors should fix their DNS proxy code, don't listen on outside interface BCP 38 otherwise we'll be jumping from protocol to protocol. Questions? Q: What does verisign do to protect their DNS servers? A: Anycast, massive peering and transit capacity Q: Jared Mauch, NTT/America; he turned on unicast rpf on the NANOG upstream link. 372,000 packets that people here have sent failed the RPF check. BCP 38 is hard Paul Quinn asked what percentage of the traffic that is. Bora Akyul, Broadcomm--any data on source ranges on the packets being seen? He could look at the 1 in 10,000 netflow sampling to see, but the individual link is a /30, looks like a normal customer link. The Merit router isn't RPF'ing either. Q: Ren Provo asks when they will peer; A: not yet, next few months, Miami Terremark, and other sites domestically and internationally in next year and a half.