On Wed, 25 Oct 2006, Simon Waters wrote:

On Wednesday 25 Oct 2006 15:59, you wrote:

...

just guessing but: 1) it's 'hard'

<rant>

How I reach the zillions of IT guys out there to say "don't do DNS inhouse, you'll only mess up" is the remaining question; slashdot? </rant>

wanna present all this rant and the proper solution to <rant> at the next nanog? :)

On Wed, 25 Oct 2006 18:59:05 -0000, "Chris L. Morrow" said:

On Wed, 25 Oct 2006, Simon Waters wrote:

...

How I reach the zillions of IT guys out there to say "don't do DNS inhouse, you'll only mess up" is the remaining question; slashdot? </rant> wanna present all this rant and the proper solution to <rant> at the next nanog? :)

Preaching to the choir, I suspect. Or are the people who are known offenders likely to actually be reached by the presentation? This is a common thread I keep encountering - the sites with enough clue to send somebody to NANOG aren't usually the sites I need to send a cluegram to. If anybody has a viable suggestion for that...

On Wed, 25 Oct 2006, Matt Ghali wrote:

On Wed, 25 Oct 2006, alex@pilosoft.com wrote:

...

I'm seeing *.register.com down (including ns*) from everywhere. Just a heads-up.

I'll take your word on exhaustively checking every possible address. BTW, do you mean nameservers down, webservers down, or something else? Did the Internet break? *.register.com means nameservers, webservers, whois servers, etc. Of course, Internet does not break, but we've received quite a number of calls about "internet is down" - given that register.com serves a large number of domains, yes, this is operationally affecting.

...

Would be interesting to see the RFO for that one, including the "why we didn't have any DNS servers offsite

They colo in more than a half-dozen facilities around the world.

...

or used anycast to at least limit amount of damage".

I also have information from a pretty good source that they actually do quite a bit of anycast. Not that I can see - possibly that depends on a specific domain's webservers.

The glue servers for register.com themselves: Name: ns1.register.com Address: 216.21.234.96 Name: ns2.register.com Address: 216.21.226.96 Name: ns3.register.com Address: 216.21.234.97 Name: ns4.register.com Address: 216.21.226.97 (note just two different /24s) Both of those /24s were down/down about 30 minutes ago, and are flapping/flapping now. route-views.oregon-ix.net>show ip bgp 216.21.234.73 ... BGP routing table entry for 216.21.234.0/24, version 5214460 701 7018 4264 13910, (suppressed due to dampening) 157.130.10.233 from 157.130.10.233 (137.39.3.60) Origin IGP, localpref 100, valid, external Dampinfo: penalty 898, flapped 5 times in 00:35:15, reuse in 00:03:50 route-views.oregon-ix.net>show ip bgp 216.21.226.97 BGP routing table entry for 216.21.226.0/24, version 5214460 ... 701 7018 4264 13910, (suppressed due to dampening) 157.130.10.233 (inaccessible) from 157.130.10.233 (137.39.3.60) Origin IGP, localpref 100, valid, external Dampinfo: penalty 861, flapped 5 times in 00:36:13, reuse in 00:03:00

From various vantage points, both /24s are routed exactly the same (7018 in NYC).

-alex

On Wed, 25 Oct 2006, Matt Ghali wrote:

On Wed, 25 Oct 2006, alex@pilosoft.com wrote:

...

I'm seeing *.register.com down (including ns*) from everywhere. They are apparently under a multi-gbps ddos of "biblical proportions".

As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;) Compliance of icann-accredited gtld-registrars with rfc2182 might be a good subject for research (again, thanks to rs for idea).... -alex

On 26 Oct 2006, Paul Vixie wrote:

...

...

...

I'm seeing *.register.com down (including ns*) from everywhere.

...

...

They are apparently under a multi-gbps ddos of "biblical proportions".

i wonder if that's due to the spam they've been sending out?

...

As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions -

no. really, not.

...

such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify.

there is no zone anywhere, including COM, the root zone, or any other, that is immune from worst-case DDoS. anycast all you want. diversify. build a name service infrastructure larger than the earth's moon. none of that will matter as long as OPNs (the scourge of internet robustness) still exist. This isn't 2001, and, I will argue that it *is*, in fact, possible to be

Paul, this isn't nanae. Let's not sling accusations like that wildly. protected from a "worst case" ddos, and not at obscene price. However, even if you argue that point, there's no excuse for not being prepared at all, and not following the BCP. While we all may be guilty of not having topologically/geographically diverse DNS - for someone whose core business is DNS, that's unexcusable.

...

Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;)

that's an easy but catty criticism, and baseless. i'm sure that some way could be found to improve register.com's infrastructure, and i don't just mean by stopping the spamming they've been doing. but it's not trivial and in the face of well-tuned worst-case DDoS, nothing will help. Well, let's talk about "worst-case ddos". Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company.

I know that the above was just rough back-of-the-envelope, and things are far more complicated than that, but this discussion does not really belong to nanog-l.

...

Compliance of icann-accredited gtld-registrars with rfc2182 might be a good subject for research (again, thanks to rs for idea).... i've been wondering if ICANN's accredidation could be revoked for spammers, and register.com has indeed been spamming. and it may also be that they are out of compliance with RFC 2182. but that would be like catching al capone for income tax evasion just because you couldn't pin murder on him. Things like that, and accusations like that, I don't think really belong to nanog-l.

(speaking for myself only)

On Oct 26, 2006, at 12:14 AM, alex@pilosoft.com wrote:

On 26 Oct 2006, Paul Vixie wrote:

...

i wonder if that's due to the spam they've been sending out? Paul, this isn't nanae. Let's not sling accusations like that wildly.

Accusations and objective facts are two separate things.

...

there is no zone anywhere, including COM, the root zone, or any other, that is immune from worst-case DDoS. anycast all you want. diversify. build a name service infrastructure larger than the earth's moon. none of that will matter as long as OPNs (the scourge of internet robustness) still exist. This isn't 2001, and, I will argue that it *is*, in fact, possible to be protected from a "worst case" ddos, and not at obscene price.

You are mistaken.

However, even if you argue that point, there's no excuse for not being prepared at all, and not following the BCP. While we all may be guilty of not having topologically/geographically diverse DNS - for someone whose core business is DNS, that's unexcusable.

We agree.

...

...

Given that register.com is/was public (I think?) - I wonder what are their sarbox auditors saying about it now ;)

that's an easy but catty criticism, and baseless. i'm sure that some way could be found to improve register.com's infrastructure, and i don't just mean by stopping the spamming they've been doing. but it's not trivial and in the face of well-tuned worst-case DDoS, nothing will help. Well, let's talk about "worst-case ddos". Let's say, 50mpps (I have not heard of ddos larger that that number). Let's say, you can sink/filter 100kpps on each box (not unreasonable on higher-end box with nsd). That means, you should be able to filter this attack with ~500 servers, appropriately place. Say, because you don't know where the attack will come in, you need 4 times more the estimated number of servers, that's 2000 servers. That's not entirely unreasonable number for a large enough company.

Even assuming your numbers, which I do not grant, you are still mistaken. There is no single "appropriately[sic] place" which can absorb 50Mpps. If you meant "appropriately placed" (as in topologically dispersed locations), a well crafted attack could still guarantee _at least_ a partial DoS from an end user PoV. It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. This is not speculation, this is fact. All a good provider can do, even with 1000s of server, is minimize the impact of any DoS. Oh, and putting 2K servers into the "right" places is not a trivial expense, even for a large company. Last time I checked, 10GE pipes were not handed out for free. And you can't just rack these things in mom-and-pop colo saying "well, it has a GigE on the motherboard" when the colo has an OC3 to the 'Net. The Cap- and Op-Ex involved in doing what you suggest properly is large enough to probably be prohibitively expensive for a company like register.com.

I know that the above was just rough back-of-the-envelope, and things are far more complicated than that, but this discussion does not really belong to nanog-l.

We disagree. Keeping large name servers running is _absolutely_ a network operations topic. Not only is the defense mostly network based (since the network is the most likely thing to break), network operators are the people who get the phone calls when DNS does break. -- TTFN, patrick

On Oct 26, 2006, at 1:31 AM, alex@pilosoft.com wrote:

...

It is essentially impossible to distinguish end-user requests from (im)properly created DoS packets (especially until BCP38 is widely adopted - i.e. probably never). Since there is no single place - no 13 places - which can withstand a well crafted DoS, you are guaranteed that some users will not be able to reach any of your listed authorities. Yeah - I know it hard-to-impossible to do that, and it is a tug-of-war between worm writers (to generate queries indistinguishable from real client-resolver-generated queries) and trying-to-detect-malformed- queries (such as duplicated qid, or from IP space that shouldn't be hitting this specific node). You probably dealt with more ddos than rest of us combined, so I bow to your superior knowledge.

First, thanx for the nod, but there are some here who have dealt with more than I have. But I think I've seen enough to know something about it. You can try things like "filter IP addresses which should not be going to node X", but what happens if the DDoS changes the network topology enough that you can't be certain users are going where you did not? If the DDoS is large, this is pretty much guaranteed. Worse, suppose the topology changes for reasons unrelated to a DDoS. You could end up DoS'ing end users without an attack! (You could theoretically only put the filters in place when an attack is happening, but that has other problems - which may or may not be worse.) Filtering on things like duplicated query IDs is not possible on router hardware doing 10s of Gbps or millions of PPS. And doing it on the server is not useful if there are more bits / pps than the router can process. Remember, servers can't answer packets that are dropped before they get to the servers. Etc., etc., etc. Overall, we are losing the war. What good providers, like the roots, Ultra, etc., do is to minimize the effect of any attack. If a "miscreant" fires the "DDoS of biblical proportions" and only 5% of users are affected, I consider that a success. Unfortunately, those 5% don't think so, but one can only do what one can do. Besides, if it truly is an attack of biblical proportion, those 5% are probably having much larger problems than name resolution. Couple other comments: From all indications I've seen (and most are not authoritative, but it's all the info I have), this was not a DDoS of "biblical proportions". There were no whole networks to go offline, there were no massive swaths of address space flapping, there were no entire peering points being congested, etc. A few Gbps does not count as "biblical" any more. Whether this attack used spoof-source or not, BCP38 is _VITAL_, IMHO, to helping curb these things. It guarantees, at the very least, that you know where the attack is sourced. Filtering become much easier. Reaching the right operators to help with the problem becomes orders of magnitude easier. And if the miscreants just start using BotNets with real IP address, GOOD. It's not the End All Be All answer, but it is a _huge_ step in the right direction. Unfortunately, as Jared has pointed out, the equipment vendors have to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the "ip bcp38" config option. :) -- TTFN, patrick

The network hardware vendors do need to include the feature to support BCP-38. It'll help us out on a number of fronts especially with some of the recent cyber attacks. We're in process of reaching out to many of the companies and many providers to encourage the implementation of BCP-38. We've gotten a lot of great feedback from many of you and its greatly appreciated. You know who you are :) Especially some of the feedback related to the hardware OS issues. -Jerry Jerry@jdixon.com or jerry.dixon@us-cert.gov Sent via BlackBerry from Cingular Wireless -----Original Message----- From: Robert Boyle <robert@tellurian.com> Date: Thu, 26 Oct 2006 12:04:03 To:"Patrick W. Gilmore" <patrick@ianai.net>, nanog@merit.edu Subject: Re: DNS DDoS [was: register.com down sev0?] At 11:21 AM 10/26/2006, you wrote: Unfortunately, as Jared has pointed out, the equipment vendors have

to help the operators support this. So let's all call your favorite router vendor and ask them when they will have the "ip bcp38" config option. :)

Even better would be the option: "no ip bcp38" Make it so a conscious action is needed to disable it, but PLEASE put that in the release notes so when the config doesn't "change" we know that something really did change... :) R Tellurian Networks - Global Hosting Solutions Since 1995 http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 "Well done is better than well said." - Benjamin Franklin

On Thu, Oct 26, 2006 at 12:14:43AM -0400, alex@pilosoft.com wrote:

On 26 Oct 2006, Paul Vixie wrote:

...

i wonder if that's due to the spam they've been sending out? Paul, this isn't nanae. Let's not sling accusations like that wildly.

There's nothing "wild" about it -- Paul is one of the most sober, reasoned observers of the spam problem, and if he told me that my servers were sending spam, then I'd darn well go investigate. Right now. Besides -- it's not like this isn't common knowledge in the anti-spam world. I'm sure I'm not the only one who's had unsatisfying correspondence with register.com wherein they refuse to lift a finger to stop the abuse from/facilitated by their operation. ---Rsk

Once upon a time, Don <don@calis.blacksun.org> said:

Some facts: 3. The attack was large enough to affect many other customers in the same data center- one with a lot of bandwidth off AT&T's backbone.

Is this what got Red Hat over the last couple of days as well? I think they have a lot of their stuff on AT&T's network. -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.

5. AT&T (at least when I've dealt with them in their datacenters) does not support BGP community strings for null routing (or any strings for that matter :) Think about that for a second. To stop an attack Register.com would need to call AT&T and request a filter/null route. Since AT&T operations is based in Singapore (again this was last time I dealt with them) I'm sure getting those filters/routes in probably doesn't happen nearly fast enough. I have heard that AT&T is currently in the process of setting up communities- maybe someone who knows more could comment.

Well, this is not exactly true. AT&T does support BGP communities, although their communities aren't all that powerful, IMO. To my knowledge, you are correct when you say that they do not support any null-routing capabilities. I would love to find out the procedure and string required to request/implement null routing via a community. For those who would like to see AT&T's official guide, it can be found at: http://www.onesc.net/communities/as7018 charles

On Fri, 27 Oct 2006, Joseph S D Yao wrote:

On Wed, Oct 25, 2006 at 10:10:05PM -0400, alex@pilosoft.com wrote: ...

...

As pointed out by Rob Seastrom in private email, RFC2182 addresses things of biblical proportions - such as dispersion of nameservers geographically and topologically. Having 3 secondaries, only one of them on separate /24, and none of them on topologically different network does not qualify. ...

ns1.register.com. 600 IN A 216.21.234.96 ns2.register.com. 600 IN A 216.21.226.96 ns3.register.com. 600 IN A 216.21.234.97 ns4.register.com. 600 IN A 216.21.226.97

I am not saying that register.com IS doing this, just that you can't say that they're NOT just from this evidence.

I think Alex could have included a few lines of traceroute to these hosts showing that they all end behind: 7 tbr1-p014001.wswdc.ip.att.net (12.123.8.98) 9.754 ms 9.685 ms 9.608 ms 8 tbr1-cl4.sl9mo.ip.att.net (12.122.10.30) 29.708 ms 29.593 ms 33.498 ms 9 12.122.85.178 (12.122.85.178) 36.300 ms 28.558 ms 28.521 ms So... it sorta looks like both /24's are behind something in StLouis, Missouri ( to me atleast ).

On Sat, 2006-10-28 at 17:36 +0000, Chris L. Morrow wrote:

So... it sorta looks like both /24's are behind something in StLouis, Missouri ( to me atleast ).

My tests from 2 years ago showed the same thing, both /24s were behind the same system in Exodus' NYC DC in Manhattan (IIRC). That is what prompted me to move everything to the rcom partner side which uses eNom. -Jim P.