Any ATT DNS admins out there?
If so, would you mind hitting me up offlist? I have a few questions that i am unable to get answered through normal channels. Cheers, Mike
You guys might want to be aware that isprime.com (I am not affiliated or representing them, just passing on info since friends and I noticed this) is actively under a DOS where lots of people's dns servers around the world are being queried with bogus sourced dns requests not from port 53 for 'NS? .'. This then bounces back to their authoritative nameservers which are getting traffic overload. They've asked that those of us that can should block all but port 53 from the following two IP's (their dns servers as seen on whois) so as not to block legitimate dns info: 66.230.128.15 66.230.160.1 Here is the response from their abuse department: To: todd@fries.net Subject: Re: dos info? From: ISPrime Support <support@isprime.com> Date: Tue, 20 Jan 2009 15:16:02 -0500 (EST) Hello, These are the result of a spoofed dns recursion attack against our servers. The actual packets in question (the ones reaching your servers) do NOT originate from our network as such there is no way for us to filter things from our end. If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these machines make legitimate outbound dns requests so an inbound filter of packets to udp/53 from either of these two sources is perfect. If you are receiving queries from 66.230.128.15/66.230.160.1 these servers are authoritative nameservers. Please do not blackhole either of these IPs as they host many domains. However, these IPs do not make outbound DNS requests so filtering requests to your IPs from these ips with a destination port of 53 should block any illegitimate requests. An ACL similar to: access-list 110 deny udp host 66.230.160.1 neq 53 any eq 53 access-list 110 deny udp host 66.230.128.15 neq 53 any eq 53 Is what you want. I would also suggest taking a look at the excellent CYMRU secure bind template (assuming you are running bind), to help you configure your nameservers so that you do not participate in this attack: http://www.cymru.com/Documents/secure-bind-template.html. Thanks for your help in mitigating this attack against us. Please let me know if I can be of further assistance. ISPrime Support support@isprime.com ICQ: 136633378 On 2009-01-20, at 15:14:33, "Todd T. Fries" <todd@fries.net> wrote:
I was told to write here for your writeup on what to block and such to help you guys out given the DOS that is ongoing.
Thanks, -- Todd Fries .. todd@fries.net _____________________________________________ | \ 1.636.410.0632 (voice) | Free Daemon Consulting, LLC \ 1.405.227.9094 (voice) | http://FreeDaemonConsulting.com \ 1.866.792.3418 (FAX) | "..in support of free software solutions." \ 250797 (FWD) | \ \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 37E7 D3EB 74D0 8D66 A68D B866 0326 204E 3F42 004A http://todd.fries.net/pgp.txt Penned by Mike Lyon on 20090109 16:41.04, we have: | If so, would you mind hitting me up offlist? I have a few questions that i | am unable to get answered through normal channels. | | Cheers, | Mike
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded:
From: ISPrime Support <support@isprime.com> These are the result of a spoofed dns recursion attack against our servers. The actual packets in question (the ones reaching your servers) do NOT originate from our network as such there is no way for us to filter things from our end. If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these machines make legitimate outbound dns requests so an inbound filter of packets to udp/53 from either of these two sources is perfect. If you are receiving queries from 66.230.128.15/66.230.160.1 these servers are authoritative nameservers. Please do not blackhole either of these IPs as they host many domains. However, these IPs do not make outbound DNS requests so filtering requests to your IPs from these ips with a destination port of 53 should block any illegitimate requests.
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question. Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away. Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic? Even if I used a REJECT policy, I'd expect the ICMP messages to go back to the appropriate - as in real - hosts, rather than the spoofing sources. Something here is very odd, very odd indeed... or I'm being dumb. It's happened before. Graeme
Hello, Representing ISPrime here. This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts. It's somewhat absurd to suggest that we are attacking our own nameservers, I assure you, we didn't spend many hours looking for your specific nameserver to start sending 10 requests per second for the root zone, and our nameservers serve many popular domains. Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation. -Phil AS23393 On Jan 21, 2009, at 12:08 PM, Graeme Fowler wrote:
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded:
From: ISPrime Support <support@isprime.com> These are the result of a spoofed dns recursion attack against our servers. The actual packets in question (the ones reaching your servers) do NOT originate from our network as such there is no way for us to filter things from our end. If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these machines make legitimate outbound dns requests so an inbound filter of packets to udp/53 from either of these two sources is perfect. If you are receiving queries from 66.230.128.15/66.230.160.1 these servers are authoritative nameservers. Please do not blackhole either of these IPs as they host many domains. However, these IPs do not make outbound DNS requests so filtering requests to your IPs from these ips with a destination port of 53 should block any illegitimate requests.
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Even if I used a REJECT policy, I'd expect the ICMP messages to go back to the appropriate - as in real - hosts, rather than the spoofing sources.
Something here is very odd, very odd indeed... or I'm being dumb. It's happened before.
Graeme
On Wed, 21 Jan 2009, Phil Rosenthal wrote:
This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts.
I'm only receiving NS queries for "." from spoofed 66.230.128.15 and 66.230.160.1 via above.net (of my three transit providers) and none from peering. This usually indicates a single source, such as one rooted machine on non-BCP38 net spewing most of a gigabit.
Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation.
Have you had any luck tracking back the source of the spoofed packets? If me talking to above.net sounds useful, let me know. -- Aaron
On Wed, 2009-01-21 at 12:27 -0500, Phil Rosenthal wrote:
Representing ISPrime here.
Well... representing myself and nobody else, so if that stretches my credibility thin so be it.
It's somewhat absurd to suggest that we are attacking our own nameservers, I assure you, we didn't spend many hours looking for your specific nameserver to start sending 10 requests per second for the root zone, and our nameservers serve many popular domains.
I just checked to make sure I did not make that assertion. I did not. I observed something odd, and stated as much to see if anyone else did. I apologise if you read my message as insinuating what you stated, but I assure you that wasn't the intention. I did say "maybe I'm being dumb", and that is indeed the answer - I applied a temporary netfilter ruleset, then made it permanent - and it switched the DROP and LOG statements round so that... the packet got dropped first and the log statements never got hit. Schoolboy error (and interesting that someone else has observed this behaviour before!)... Normal service has been resumed. I should write a haiku here (sorry, MLC, poor joke).
Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation.
In a previous job I've been on the receiving end of similar attacks so I have a large degree of understanding of the pressure you're under at the moment. I wish you the best of luck sorting it out. Graeme
-----Original Message----- From: Graeme Fowler [mailto:graeme@graemef.net] Sent: Wednesday, January 21, 2009 11:08 AM To: Nanog Mailing list Subject: Re: isprime DOS in progress
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Even if I used a REJECT policy, I'd expect the ICMP messages to go back to the appropriate - as in real - hosts, rather than the spoofing sources.
Something here is very odd, very odd indeed... or I'm being dumb. It's happened before.
Graeme
In looking at my query logs I am seeing only requests from 66.230.160.1 and 66.230.128.15 so I've done the same thing with iptables and the rules are resulting in an ever growing number of packets being dropped. # iptables -nvL | grep -F -B 1 -A 1 66.230.160.1 | awk '{ print $1,$2,$3,$8,$10,$11,$12 }' pkts bytes target source 49517 2228K DROP 66.230.160.1 udp spt:!53 dpt:53 35905 1616K DROP 66.230.128.15 udp spt:!53 dpt:53
Can't some upstream provider find a source of the DNS NS . questions that is other than isprime?
Graeme Fowler wrote:
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
I've seen that behaviour in the past, but not this time? I've seen a few of these attacks bouncing off my nameservers recently, and when I add "DROP" rules to my firewall, the incoming traffic disappears soon after. But the most recent set (66.230.160.1 and 66.230.128.15) are still hammering away... -- Harald
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters? These clients may look similar to the DOS attack, but there are subtle differences: Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses. These 3 clients started probing my DNS server almost simultaneously on January 8th: Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory. However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing. Bjørn
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now. -Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
Hi, I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1 Regards, Steve -----Original Message----- From: Phil Rosenthal [mailto:pr@isprime.com] Sent: Saturday, 24 January 2009 4:12 AM To: nanog@nanog.org Subject: Re: isprime DOS in progress Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now. -Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
Looks to me like the target has moved, anyone else seeing similar? Jan 23 20:19:08 LND02 named[9611]: client 63.217.28.226#39489: view external: query (cache) './NS/IN' denied Jan 23 20:19:09 LND02 named[9611]: client 63.217.28.226#20558: view external: query (cache) './NS/IN' denied Jan 23 20:19:11 LND02 named[9611]: client 63.217.28.226#38525: view external: query (cache) './NS/IN' denied Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#41535: view external: query (cache) './NS/IN' denied Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#51220: view external: query (cache) './NS/IN' denied Jan 23 20:19:13 LND02 named[9611]: client 63.217.28.226#28869: view external: query (cache) './NS/IN' denied Jan 23 20:19:14 LND02 named[9611]: client 63.217.28.226#12337: view external: query (cache) './NS/IN' denied Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#41346: view external: query (cache) './NS/IN' denied Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#56831: view external: query (cache) './NS/IN' denied Jan 23 20:19:17 LND02 named[9611]: client 63.217.28.226#13352: view external: query (cache) './NS/IN' denied Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#55466: view external: query (cache) './NS/IN' denied Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#24586: view external: query (cache) './NS/IN' denied Jan 23 20:19:19 LND02 named[9611]: client 63.217.28.226#43105: view external: query (cache) './NS/IN' denied On Fri, 2009-01-23 at 19:46 +0000, Steven Lisson wrote:
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards, Steve
-----Original Message----- From: Phil Rosenthal [mailto:pr@isprime.com] Sent: Saturday, 24 January 2009 4:12 AM To: nanog@nanog.org Subject: Re: isprime DOS in progress
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now.
-Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/ On Fri, Jan 23, 2009 at 3:20 PM, Luke Sheldrick <luke@sheldrick.co.uk> wrote:
Looks to me like the target has moved, anyone else seeing similar?
Jan 23 20:19:08 LND02 named[9611]: client 63.217.28.226#39489: view external: query (cache) './NS/IN' denied Jan 23 20:19:09 LND02 named[9611]: client 63.217.28.226#20558: view external: query (cache) './NS/IN' denied Jan 23 20:19:11 LND02 named[9611]: client 63.217.28.226#38525: view external: query (cache) './NS/IN' denied Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#41535: view external: query (cache) './NS/IN' denied Jan 23 20:19:12 LND02 named[9611]: client 63.217.28.226#51220: view external: query (cache) './NS/IN' denied Jan 23 20:19:13 LND02 named[9611]: client 63.217.28.226#28869: view external: query (cache) './NS/IN' denied Jan 23 20:19:14 LND02 named[9611]: client 63.217.28.226#12337: view external: query (cache) './NS/IN' denied Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#41346: view external: query (cache) './NS/IN' denied Jan 23 20:19:15 LND02 named[9611]: client 63.217.28.226#56831: view external: query (cache) './NS/IN' denied Jan 23 20:19:17 LND02 named[9611]: client 63.217.28.226#13352: view external: query (cache) './NS/IN' denied Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#55466: view external: query (cache) './NS/IN' denied Jan 23 20:19:18 LND02 named[9611]: client 63.217.28.226#24586: view external: query (cache) './NS/IN' denied Jan 23 20:19:19 LND02 named[9611]: client 63.217.28.226#43105: view external: query (cache) './NS/IN' denied
On Fri, 2009-01-23 at 19:46 +0000, Steven Lisson wrote:
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards, Steve
-----Original Message----- From: Phil Rosenthal [mailto:pr@isprime.com] Sent: Saturday, 24 January 2009 4:12 AM To: nanog@nanog.org Subject: Re: isprime DOS in progress
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now.
-Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
On Sat, 2009-01-24 at 07:21, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Wrong approach, they are *innocent* in this as are the new targets. insert into your favourite acl: deny udp host 66.230.160.1 neq 53 any eq 53 deny udp host 66.230.128.15 neq 53 any eq 53 But it's much less work to add a filter on the name server as others have mentioned.
Noel Butler wrote:
On Sat, 2009-01-24 at 07:21, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Wrong approach, they are *innocent* in this as are the new targets.
insert into your favourite acl: deny udp host 66.230.160.1 neq 53 any eq 53 deny udp host 66.230.128.15 neq 53 any eq 53
But it's much less work to add a filter on the name server as others have mentioned.
Having the world trying to keep up with ACL entries seems futile. Is there really nothing to be done about this? (Yes, I know, BCP38, but obviously the accomplice providers don't care.) ~Seth
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security. My two cents. -- Jeffrey Lyon, Leadership Team jeffrey.lyon@blacklotus.net | http://www.blacklotus.net Black Lotus Communications of The IRC Company, Inc. Look for us at HostingCon 2009 in Washington, DC on August 10th - 12th at Booth #401. On Fri, Jan 23, 2009 at 9:05 PM, Seth Mattinen <sethm@rollernet.us> wrote:
Noel Butler wrote:
On Sat, 2009-01-24 at 07:21, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Wrong approach, they are *innocent* in this as are the new targets.
insert into your favourite acl: deny udp host 66.230.160.1 neq 53 any eq 53 deny udp host 66.230.128.15 neq 53 any eq 53
But it's much less work to add a filter on the name server as others have mentioned.
Having the world trying to keep up with ACL entries seems futile. Is there really nothing to be done about this? (Yes, I know, BCP38, but obviously the accomplice providers don't care.)
~Seth
On Fri, 23 Jan 2009, Jeffrey Lyon wrote:
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security.
Because non of us wantsto spend the next two days working on fixing mail for our organizations. That said, when speaking of DDoS, I believe they are mostly the same bots. And yes, while there is a lot we can do to mitigate DDoS, we are that helpless. All our strategies show as much. Gadi.
Jeffrey Lyon wrote:
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security.
Back to my original question: is there really not a better solution? ~Seth
On Fri, 23 Jan 2009 18:33:14 PST, Seth Mattinen said:
Back to my original question: is there really not a better solution?
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
On Jan 23, 2009, at 10:31 PM, Valdis.Kletnieks@vt.edu wrote:
On Fri, 23 Jan 2009 18:33:14 PST, Seth Mattinen said:
Back to my original question: is there really not a better solution?
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part..
The Cameros or SUVs and baseball bats are normally not in sync with local legal traditions, depending on your local legal statutes. A certain ISP in California, however, seems to have learned a more subtle lesson, perhaps with left more satisfying results. And while I don't think that is any serious fix, I also think that there might well be a deterrence effect, to whatever extent that works - it might not kill a system, but it can provide deterrence. Hell, I expect the perpetrator for this attack to be on this list. So it becomes more more step on the arms race.
On Fri, Jan 23, 2009 at 10:31 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Fri, 23 Jan 2009 18:33:14 PST, Seth Mattinen said:
Back to my original question: is there really not a better solution?
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
Step one, find a device on your netowrk seeing the traffic step two, follow the stream(s) of traffic back to its ingress (hopefully a customer link on your network) step three, watch for associated traffic to the source of the dns queries, correlate this with other sources on your network to find/identify the control point for this effort. -chris
On Jan 23, 2009, at 9:10 PM, Christopher Morrow wrote:
On Fri, Jan 23, 2009 at 10:31 PM, <Valdis.Kletnieks@vt.edu> wrote:
On Fri, 23 Jan 2009 18:33:14 PST, Seth Mattinen said:
Back to my original question: is there really not a better solution?
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
Step one, find a device on your netowrk seeing the traffic step two, follow the stream(s) of traffic back to its ingress (hopefully a customer link on your network) step three, watch for associated traffic to the source of the dns queries, correlate this with other sources on your network to find/identify the control point for this effort.
You missed one.. Step 4: enable BCP 38 or similar ingress source address spoofing mitigation mechanism on all customer ingress interfaces (note: uRPF *loose* mode no-fixie these attacks) - as you should have had in the first place such that you didn't have to trace those spoof packets step-by-step back through your network. No more excuses, people.. -danny
On Jan 23, 2009, at 8:53 PM, Danny McPherson wrote:
You missed one.. Step 4: enable BCP 38 or similar ingress source address spoofing mitigation mechanism on all customer ingress interfaces ... No more excuses, people..
Sad fact is that there are zillions of excuses. Unfortunately I suspect the only way we're going to make any progress on this will be for laws to be passed (or lawsuits to be filed) that impose a financial penalty on ISPs through which these attacks propagate. Regards, -drc
On Jan 23, 2009, at 10:06 PM, David Conrad wrote:
Sad fact is that there are zillions of excuses. Unfortunately I suspect the only way we're going to make any progress on this will be for laws to be passed (or lawsuits to be filed) that impose a financial penalty on ISPs through which these attacks propagate.
Yep, some external force is apparently necessary, unfortunately. We've been encouraging, and asking, and measuring intensely for over a dozen years now, and the application of anti- spoofing is still dismal < ~60%). I used to be sympathetic to the arguments about infrastructure support, resources, tools, etc.. I consider those argument no longer valid and operators who don't implement ingress BCP 38 style filtering remiss. -danny
David Conrad wrote:
Sad fact is that there are zillions of excuses. Unfortunately I suspect the only way we're going to make any progress on this will be for laws to be passed (or lawsuits to be filed) that impose a financial penalty on ISPs through which these attacks propagate.
Careful what you ask for. You might get it, and I'm sure the outcome wouldn't be liked by any. Forgery is bad, but I've seen plenty of DDoS without forgery that can do serious damage. Forgery just makes analysis and back tracking harder. Getting sued because you had some stealth botnet that suddenly fires up is not a good deal; and probably why ISP's still manage to hold onto some immunities. OT, though, I'm sure. The last DoS with forgery that I asked a provider to backtrack, in the small hopes that it was a concentrated attack with forgery and not a forging botnet, was met with "flows? tracking? We can't see anything. We'll happily remove the block so you can see if it's still going on if you want." Now I have fun trying to explain towards upstream management why a good security team and policy is important in anyone we purchase transit from. I think they understand it about as much as the transit providers did. Even when tracked, it is rare that you can get enough interest, time or technical ability to backtrack to a controller. Gaining access to the infected machine and grabbing the bot code is even more rare. That being said, a lot of botnets are already monitored and watched. Unfortunately, there are legal issues when they cross international boundaries; just as there are with child exploitation sites which are hosted in places that are more accepting/tolerant of such things. Jack
On Jan 24, 2009, at 1:34 PM, Jack Bates wrote:
Now I have fun trying to explain towards upstream management why a good security team and policy is important in anyone we purchase transit from.
Apart from commercial DDoS mitigation services, how many folks have SLAs which specify DoS-related response-times, ETRs, specify levels of service degradation, et. al. as part of their transit contracts, peering agreements, hosting/co-location agreements, etc.? How many end-customers have these terms written into their contracts with their SPs? Has anyone ever de-peered or terminated a transit or hosting/co- location relationship specifically due to DoS issues? If so, was it based upon specific contractual clauses related to DoS, or was some other metric used to justify ending the arrangement (i.e., non- adherence to traffic ratios due to DoS traffic, or other effects)? Did you have to pay a termination fee to get out of the arrangement? ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // +852.9133.2844 mobile All behavior is economic in motivation and/or consequence.
Jack, On Jan 23, 2009, at 9:34 PM, Jack Bates wrote:
David Conrad wrote:
Sad fact is that there are zillions of excuses. Unfortunately I suspect the only way we're going to make any progress on this will be for laws to be passed (or lawsuits to be filed) that impose a financial penalty on ISPs through which these attacks propagate. Careful what you ask for. You might get it, and I'm sure the outcome wouldn't be liked by any.
You misunderstand. I am certainly not asking for it, in fact I consider it something of a tragedy that it appears the only way action can be taken is to take it away from technologists and give it to lawyers and politicians. And no, the outcome most likely won't be liked, just like CAN-SPAM was a sad joke.
Getting sued because you had some stealth botnet that suddenly fires up is not a good deal; and probably why ISP's still manage to hold onto some immunities. OT, though, I'm sure.
It would seem that as ISPs implement DPI and protocol-specific traffic shaping, they damage the arguments that they can make claiming they have "common carrier" status with the inherent immunities that status provides. I can hear the argument now: if an ISP can throttle BitTorrent (or whatever) for specific nodes, why can't they also limit the source addresses of packets coming from those nodes? Regards, -drc
In message <8C5F1FEC-FF51-4BA2-A762-C13BC275E806@virtualized.org>, David Conrad writes:
It would seem that as ISPs implement DPI and protocol-specific traffic shaping, they damage the arguments that they can make claiming they have "common carrier" status with the inherent immunities that status provides. I can hear the argument now: if an ISP can throttle BitTorrent (or whatever) for specific nodes, why can't they also limit the source addresses of packets coming from those nodes?
They can and should. I suspect many of them do as they usually apply these filters to home networks. BCP 38 is ~10 years old now. It should have been factored into the purchasing decision of all the current equipement. If it wasn't then the operator was negligent. Mark
Regards, -drc
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
On Sat, Jan 24, 2009 at 8:01 PM, Mark Andrews <Mark_Andrews@isc.org> wrote:
In message <8C5F1FEC-FF51-4BA2-A762-C13BC275E806@virtualized.org>, David Conrad writes:
It would seem that as ISPs implement DPI and protocol-specific traffic shaping, they damage the arguments that they can make claiming they have "common carrier" status with the inherent immunities that status provides. I can hear the argument now: if an ISP can throttle BitTorrent (or whatever) for specific nodes, why can't they also limit the source addresses of packets coming from those nodes?
They can and should. I suspect many of them do as they usually apply these filters to home networks.
BCP 38 is ~10 years old now. It should have been factored into the purchasing decision of all the current equipement. If it wasn't then the operator was negligent.
Mark
BCP 38 isn't a license, it's a technique. -- Martin Hannigan martin@theicelandguy.com p: +16178216079
In message <d99aaed40901241734g691cd581q20e9c88eb76093b7@mail.gmail.com>, Marti n Hannigan writes:
On Sat, Jan 24, 2009 at 8:01 PM, Mark Andrews <Mark_Andrews@isc.org> wrote:
In message <8C5F1FEC-FF51-4BA2-A762-C13BC275E806@virtualized.org>, David Conrad writes:
It would seem that as ISPs implement DPI and protocol-specific traffic shaping, they damage the arguments that they can make claiming they have "common carrier" status with the inherent immunities that status provides. I can hear the argument now: if an ISP can throttle BitTorrent (or whatever) for specific nodes, why can't they also limit the source addresses of packets coming from those nodes?
They can and should. I suspect many of them do as they usually apply these filters to home networks.
BCP 38 is ~10 years old now. It should have been factored into the purchasing decision of all the current equipement. If it wasn't then the operator was negligent.
Mark
BCP 38 isn't a license, it's a technique.
There are plenty of cases in common law where as a owner of something and you havn't taken reasonable steps to protect or prevent injury that, were well known, you will be proved to be negligent. BCP 38 is falling into that sort of category. Every operator here should be worried about what will happen when someone decides to sue them to recover damaged caused by spoofed traffic. It's just a matter of time before this happens. Remember every router inspects packets to the level required to implement BCP 38. This is not deep packet inspection. This is address inspection which every router performs. Did you know about "BCP 38"? What steps did you take to implement "BCP 38"? I suspect that a lawyer will be able to demonstrate to a judge that even as a common carrier that a operator should have been deploying BCP 38. Mark
-- Martin Hannigan martin@theicelandguy.com p: +16178216079
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sat, Jan 24, 2009 at 6:05 PM, Mark Andrews <Mark_Andrews@isc.org> wrote:
BCP 38 isn't a license, it's a technique.
There are plenty of cases in common law where as a owner of something and you havn't taken reasonable steps to protect or prevent injury that, were well known, you will be proved to be negligent.
BCP 38 is falling into that sort of category.
Every operator here should be worried about what will happen when someone decides to sue them to recover damaged caused by spoofed traffic. It's just a matter of time before this happens. Remember every router inspects packets to the level required to implement BCP 38. This is not deep packet inspection. This is address inspection which every router performs.
Did you know about "BCP 38"? What steps did you take to implement "BCP 38"?
I suspect that a lawyer will be able to demonstrate to a judge that even as a common carrier that a operator should have been deploying BCP 38.
I think each point above is true -- BCP38 is indeed a technique, but failure to universally implement it defaults to (almost) a tragedy of the commons. After ~10 years, it is surreal to me that we, as a community, are still grappling with issues where it could be beneficial for the Internet community at-large. I mean, it _is_ a BCP. - - ferg p.s. Even when Dan Senie and I drafted RFC2827/BCP38, we were doing nothing more than documenting what everyone (well, maybe not everyone) already knew anyway -- that we all need to bite the bullet and just do it. -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.3 (Build 3017) wj8DBQFJe8qeq1pz9mNUZTMRAmXvAJ4h2V/p6Ak+woMbT9BTCOYrEKMlXACdFaFe icfmMA4432St/zl5j3yfQiA= =iWAr -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/
I think each point above is true -- BCP38 is indeed a technique, but failure to universally implement it defaults to (almost) a tragedy of the commons.
After ~10 years, it is surreal to me that we, as a community, are still grappling with issues where it could be beneficial for the Internet community at-large. I mean, it _is_ a BCP.
The community isn't grappling with the issue. For the most part the NANOG community has implemented BCP 38. The problem is that there are lots of ISPs that are not part of the community and I get the sense that the this number continues to grow. In a sense NANOG has a problem with dwindling market-share. A shrinking percentage of ISPs are part of the NANOG community, and NANOG participants have less and less influence on decisions in the ISPs that they work for, probably because most of them do not work for ISPs but work for telephone companies which have expanded into the ISP business. And yes, I too work for a telco that is now also a major ISP in its telco market area. p.s. Even when Dan Senie and I drafted RFC2827/BCP38, we were doing nothing more than documenting what everyone (well, maybe not everyone) already knew
anyway -- that we all need to bite the bullet and just do it.
Personally, I think that the network operations community represented in NANOG needs to do more outreach to forums where the telecoms community gather rather than ghettoizing Internet ops and engineering. It's all well and good to have NANOG lists and meetings, but once things like BCP-38 reach consensus, how many NANOG members would consider going to something like FutureNet Expo and presenting on the topic? --Michael Dillon
Every time I see a post like the one below on this list, I can't help but feel like big brother has infiltrated the list. There's no mess like the ones government will create for you. Lorell -----Original Message----- From: David Conrad [mailto:drc@virtualized.org] Sent: Friday, January 23, 2009 11:06 PM To: Danny McPherson Cc: NANOG list Subject: Re: Are we really this helpless? (Re: isprime DOS in progress) On Jan 23, 2009, at 8:53 PM, Danny McPherson wrote:
You missed one.. Step 4: enable BCP 38 or similar ingress source address spoofing mitigation mechanism on all customer ingress interfaces ... No more excuses, people..
Sad fact is that there are zillions of excuses. Unfortunately I suspect the only way we're going to make any progress on this will be for laws to be passed (or lawsuits to be filed) that impose a financial penalty on ISPs through which these attacks propagate. Regards, -drc No virus found in this incoming message. Checked by AVG - http://www.avg.com Version: 8.0.176 / Virus Database: 270.10.12/1911 - Release Date: 1/23/2009 7:28 AM
Lorell, On Jan 25, 2009, at 5:27 PM, Lorell Hathcock wrote:
Every time I see a post like the one below on this list, I can't help but feel like big brother has infiltrated the list.
Someone stating the obvious implications of the lack of the Internet operations community to address a known problem that threatens pretty much anyone on the Internet feels like Big Brother has infiltrated the list to you? Odd.
There's no mess like the ones government will create for you.
Agreed. Too bad the technologists haven't been able to solve the problem. To be clear, I'm not suggesting laws or lawsuits are a good thing. Rather, it's generally what happens when an industry isn't able to figure out how to wipe its own butt. Regards, -drc
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Valdis.Kletnieks@vt.edu wrote: <SNIP>
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
But if they were in eastern Europe or Russia, wouldn't that solution be considered standard business practice and thus be legal? Jon - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-224-2494 s: 843-564-4224 http://www.linkedin.com/in/jonrkibler My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkl7LUMACgkQUVxQRc85QlNCjQCdF1UAaO7sCU5rTN6xoN7UrDX9 EL4AoKKTLj0U5sAWsMinCRot10BFB0Go =WHAs -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.
But if they were in eastern Europe or Russia, wouldn't that solution be considered standard business practice and thus be legal?
Assuming that you really believe such an outrageous statement, I went to <http://www.themoscowtimes.com/index.htm> to search for stories about people being arrested for the behavior that you think is "legal". Interestingly, the first story on the front page is about such an arrest, and you can type "arrest" into the search box to find several other stories. Note that the Moscow Times is an English language newspaper in Russia that tends to be critical of the government. But at least it is written by people who live there and who have a much better idea of what is going on than the folklore that is reported on in the West. Yes, this is network operational, because the Internet is a global network, and you really do need to understand that the majority of network operators in Eastern Europe or Russia, are not out to get you. If you have operational issues with them you should contact them directly and sort it out. The criminal elements in cyberspace are also criminals in their own country and very few network operators are actually in league with them. Treat other network operators as innocent unless they are proven guilty. For instance go to <http://www.cert.ru/>, click on the Union Jack flag for English language version and then on the Conference link for the 7th Nov 2008 to see presentations from Finland, Estonia, Bulgaria, and Russia. Some of the slides are even in English, or English and Russian. And if you attend one of the RIPE meetings in Europe you can actually meet network operators from these countries and learn that they are pretty much like you, running a business and sorting out problems. -- Michael Dillon
Jon Kibler wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Valdis.Kletnieks@vt.edu wrote: <SNIP>
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
But if they were in eastern Europe or Russia, wouldn't that solution be considered standard business practice and thus be legal?
As an eastern european, I can tell you that you're watching way too much TV :) We also have law enforcement that can take down rogue cracker networks. The only problem police has in this matters is somewhat lack of experience and lack of resources. Otherwise they don't have a problem knocking down doors to catch bad guys.
Valdis.Kletnieks@vt.edu wrote:
On Fri, 23 Jan 2009 18:33:14 PST, Seth Mattinen said:
Back to my original question: is there really not a better solution?
Well, we *could* hunt down the perpetrators, pool some $$, and hire 3 or 4 baseball-bat wielding professional explainers to go explain our position to them. Figuring out how to do so without breaking any laws is the tough part...
Or better yet, how Marsellus Wallace said it in Pulp Fiction: " I'ma call a coupla hard, pipe-hittin' niggers, who'll go to work on the homes here with a pair of pliers and a blow torch." Now that would be fun and actually send a strong message across the board :))
On 1/23/09, Seth Mattinen <sethm@rollernet.us> wrote:
Jeffrey Lyon wrote:
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security.
Back to my original question: is there really not a better solution?
~Seth
When in doubt, throw more capacity at it. I'm only half-kidding. Sometimes that's the cheaper solution depending on the problem. -brandon -- Brandon Galbraith Voice: 630.400.6992 Email: brandon.galbraith@gmail.com
Seth Mattinen wrote:
Jeffrey Lyon wrote:
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security.
Back to my original question: is there really not a better solution?
This sounds a lot like the conversations which led to the creation of the original Realtime Blackhole List of spam sources. When was that, 1996? -- J.D. Falk Return Path Inc http://www.returnpath.net/
J.D. Falk wrote:
Seth Mattinen wrote:
Jeffrey Lyon wrote:
I respectfully disagree. Network engineers have to keep up with many tasks and preventing DoS/DDoS should be the responsibility of everyone. I see more folks worried about spam than they are actual security.
Back to my original question: is there really not a better solution?
This sounds a lot like the conversations which led to the creation of the original Realtime Blackhole List of spam sources. When was that, 1996?
Perhaps the time has come for a BGP blackhole list like the bogon routes one. Who wants to become a target? ;) ~Seth
What's interesting in all of this is that ISPrime has been experiencing this for most of this week, yet not them or any of us has shared a network that is sourcing this traffic. I know I haven't bothered asking my upstream provider which backbone provider is sending them the "ISPrime" traffic, so I'm just as guilty as anyone. Frank -----Original Message----- From: Seth Mattinen [mailto:sethm@rollernet.us] Sent: Friday, January 23, 2009 8:06 PM To: nanog@nanog.org Subject: Are we really this helpless? (Re: isprime DOS in progress) Noel Butler wrote:
On Sat, 2009-01-24 at 07:21, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Wrong approach, they are *innocent* in this as are the new targets.
insert into your favourite acl: deny udp host 66.230.160.1 neq 53 any eq 53 deny udp host 66.230.128.15 neq 53 any eq 53
But it's much less work to add a filter on the name server as others have mentioned.
Having the world trying to keep up with ACL entries seems futile. Is there really nothing to be done about this? (Yes, I know, BCP38, but obviously the accomplice providers don't care.) ~Seth
On Jan 23, 2009, at 12:20 PM, Luke Sheldrick wrote:
Looks to me like the target has moved, anyone else seeing similar?
Jan 23 20:19:08 LND02 named[9611]: client 63.217.28.226#39489: view external: query (cache) './NS/IN' denied Jan 23 20:19:09 LND02 named[9611]: client 63.217.28.226#20558: view external: query (cache) './NS/IN' denied
Seeing the same here, it's 1 query per second per nameserver--time to work some magic with PF. -- bk
On Jan 23, 2009, at 12:20 PM, Luke Sheldrick wrote:
Looks to me like the target has moved, anyone else seeing similar?
It's switched again. The new target is 206.71.158.30 . Over night it cycled through several different IPs (testing the waters?), and finally started on this one around 10:26 Pacific time this morning. Timeline below. -- bk Jan 23 23:24:47 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep named[32762]: client 208.78.169.236#33027: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep last message repeated 2 times Jan 24 00:51:11 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep last message repeated 2 times Jan 24 00:51:30 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 00:51:30 imhotep last message repeated 2 times Jan 24 01:54:44 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 01:54:44 imhotep last message repeated 2 times Jan 24 01:55:44 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 01:55:44 imhotep last message repeated 2 times Jan 24 01:57:46 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 01:57:46 imhotep last message repeated 2 times Jan 24 02:58:29 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 02:58:30 imhotep last message repeated 2 times Jan 24 03:00:34 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 03:00:35 imhotep last message repeated 2 times Jan 24 03:05:05 imhotep named[32762]: client 208.78.169.236#33027: view ext: query (cache) './NS/IN' denied Jan 24 03:05:05 imhotep last message repeated 2 times Jan 24 03:07:49 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 04:02:38 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 04:02:38 imhotep last message repeated 2 times Jan 24 04:05:43 imhotep named[32762]: client 204.11.51.59#32802: view ext: query (cache) './NS/IN' denied Jan 24 04:05:43 imhotep last message repeated 2 times Jan 24 04:12:52 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 04:12:52 imhotep last message repeated 2 times Jan 24 05:07:37 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 05:07:37 imhotep last message repeated 2 times Jan 24 05:11:35 imhotep named[32762]: client 204.11.51.59#32802: view ext: query (cache) './NS/IN' denied Jan 24 05:11:35 imhotep last message repeated 2 times Jan 24 05:21:36 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 05:21:37 imhotep last message repeated 2 times Jan 24 06:16:06 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 06:16:06 imhotep last message repeated 2 times Jan 24 06:20:19 imhotep named[32762]: client 204.11.51.61#43329: view ext: query (cache) './NS/IN' denied Jan 24 06:20:19 imhotep last message repeated 2 times Jan 24 06:29:37 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 06:29:37 imhotep last message repeated 2 times Jan 24 06:35:11 imhotep named[32762]: client 149.20.52.161#61452: view ext: notify question section contains no SOA Jan 24 07:23:06 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 07:23:06 imhotep last message repeated 2 times Jan 24 07:28:27 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 07:28:27 imhotep last message repeated 2 times Jan 24 07:40:25 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 07:40:25 imhotep last message repeated 2 times Jan 24 08:29:57 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 08:29:57 imhotep last message repeated 2 times Jan 24 08:36:10 imhotep named[32762]: client 204.11.51.61#43330: view ext: query (cache) './NS/IN' denied Jan 24 08:36:11 imhotep last message repeated 2 times Jan 24 08:52:45 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 08:52:45 imhotep last message repeated 2 times Jan 24 08:55:54 imhotep named[32762]: client 149.20.58.131#59151: view ext: query (cache) 'localhost/A/IN' denied Jan 24 09:36:38 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 09:36:38 imhotep last message repeated 2 times Jan 24 09:43:53 imhotep named[32762]: client 204.11.51.61#43330: view ext: query (cache) './NS/IN' denied Jan 24 09:43:54 imhotep last message repeated 2 times Jan 24 09:53:56 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 10:05:28 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 10:05:28 imhotep last message repeated 2 times Jan 24 10:26:09 imhotep named[32762]: client 206.71.158.30#18971: view ext: query (cache) './NS/IN' denied Jan 24 10:26:11 imhotep named[32762]: client 206.71.158.30#47622: view ext: query (cache) './NS/IN' denied Jan 24 10:26:13 imhotep named[32762]: client 206.71.158.30#16077: view ext: query (cache) './NS/IN' denied
Caveat: my PERL is _terrible_. http://www.smtps.net/pub/dns-amp-watch.pl This assumes you're using BIND. My logs roll on the hour, so I run it from cron at 1 minute before the hour. Depending on how long it takes to process your logs, you might need to tweak. -- bk CA cert: http://www.smtps.net/pub/smtps-dot-net-ca-2.pem
I would not recommend sucking in your dns log into array, rather, read line by line and iterate over the file, line by line. Frank -----Original Message----- From: Brian Keefer [mailto:chort@smtps.net] Sent: Saturday, January 24, 2009 6:50 PM To: nanog@nanog.org Subject: Tracking the DNS amplification attacks (was: isprime DOS in progress) Caveat: my PERL is _terrible_. http://www.smtps.net/pub/dns-amp-watch.pl This assumes you're using BIND. My logs roll on the hour, so I run it from cron at 1 minute before the hour. Depending on how long it takes to process your logs, you might need to tweak. -- bk CA cert: http://www.smtps.net/pub/smtps-dot-net-ca-2.pem
On Jan 24, 2009, at 7:00 PM, Frank Bulk wrote:
-----Original Message----- From: Brian Keefer [mailto:chort@smtps.net]
Caveat: my PERL is _terrible_.
http://www.smtps.net/pub/dns-amp-watch.pl
I would not recommend sucking in your dns log into array, rather, read line by line and iterate over the file, line by line.
Frank
Yep, you're absolutely right. I copied that bit from another script without thinking. It's fixed now, along with the logic error on an empty array. Thanks for the feedback. -- bk
On Sat, Jan 24, 2009 at 9:00 PM, Frank Bulk <frnkblk@iname.com> wrote:
I would not recommend sucking in your dns log into array, rather, read line by line and iterate over the file, line by line.
Frank
True.. reading into an array can get a bit nasty, if your server logs are a few gigabytes in size. Could use C, also... http://pastebin.com/f4c2ff010 Scanning your logs after the fact is definitely not as good as separating DNS servers that are authoritative for zones and picking nameserver software such as TinyDNS or similar options for authoritative DNS usage that won't respond to queries for the root or other zones the DNS server is not directed to be used for, and using acls/firewalls to prevent outside queries against other DNS servers that aren't delegated zones. It's a bit difficult to apply a BIND patch that doesn't exist yet in vendor-supplied implementations of BIND, at least.. -- -J
There's another new IP: 67.192.144.0 . Initially (around 2AM Pacific) the query rate was 1 per second, but is now down significantly. -- bk
and just now it changed to 64.57.246.146. Interestingly, the IP changed within minutes of me posting to NANOG. -- bk On Jan 27, 2009, at 6:34 AM, Brian Keefer wrote:
There's another new IP: 67.192.144.0 .
Initially (around 2AM Pacific) the query rate was 1 per second, but is now down significantly.
-- bk
Today we have from 208.69.36.12, DNS 67.192.144.x divers of this networkes attakes ----- Original Message ----- From: "Brian Keefer" <chort@smtps.net> To: <nanog@nanog.org> Sent: Tuesday, January 27, 2009 3:42 PM Subject: Re: Tracking the DNS amplification attacks (was: isprime DOS inprogress)
and just now it changed to 64.57.246.146. Interestingly, the IP changed within minutes of me posting to NANOG.
-- bk
On Jan 27, 2009, at 6:34 AM, Brian Keefer wrote:
There's another new IP: 67.192.144.0 .
Initially (around 2AM Pacific) the query rate was 1 per second, but is now down significantly.
-- bk
On 1/24/2009 at 4:50 PM, Brian Keefer <chort@smtps.net> wrote: Caveat: my PERL is _terrible_.
http://www.smtps.net/pub/dns-amp-watch.pl
This assumes you're using BIND. My logs roll on the hour, so I run it from cron at 1 minute before the hour. Depending on how long it takes to process your logs, you might need to tweak.
FWIW, I find it easier to track this using tcpdump. I don't like running BIND with query logging. Here's a filter that catches these, port 53 && (udp[10:4] == 0x01000001) && (udp[20:2] == 0x0000) How it works is left as an exercise for the reader. When I sniff the link to a server authorative for several domains, 17:29:55.792127 IP 72.249.127.168.3966 > 206.220.220.100.53: 18501+ NS? . (17) 17:29:57.116367 IP 69.64.87.156.58419 > 206.220.220.100.53: 62419+ NS? . (17) 17:29:57.804987 IP 72.249.127.168.33108 > 206.220.220.100.53: 4637+ NS? . (17) 17:29:58.959680 IP 72.20.3.82.23084 > 206.220.220.100.53: 14310+ NS? . (17) 17:29:59.818994 IP 72.249.127.168.60876 > 206.220.220.100.53: 22791+ NS? . (17) 17:30:01.622728 IP 69.64.87.156.30151 > 206.220.220.100.53: 13557+ NS? . (17) 17:30:01.628899 IP 72.20.3.82.49015 > 206.220.220.100.53: 14250+ NS? . (17) 17:30:01.821214 IP 72.249.127.168.13831 > 206.220.220.100.53: 51065+ NS? . (17) 17:30:03.342856 IP 69.64.87.156.1926 > 206.220.220.100.53: 38768+ NS? . (17) 17:30:03.818706 IP 72.249.127.168.33663 > 206.220.220.100.53: 12720+ NS? . (17) 17:30:05.186647 IP 72.20.3.82.7649 > 206.220.220.100.53: 52079+ NS? . (17) 17:30:05.815718 IP 72.249.127.168.37241 > 206.220.220.100.53: 345+ NS? . (17) 17:30:07.816144 IP 72.249.127.168.23784 > 206.220.220.100.53: 56874+ NS? . (17) 17:30:07.849503 IP 69.64.87.156.33190 > 206.220.220.100.53: 20113+ NS? . (17)
I extracted all logs from one of my dns servers that reflected an "'./NS/IN' denied" message, pumped them into a database and ran a few queries. The first query shows the number of "denied" messages on my dns server, sorted by date. The amount of traffic definitely picked up on January 21st: +-------------+-------------+ | date | count(date) | +-------------+-------------+ | 03-Jan-2009 | 20 | | 04-Jan-2009 | 173 | | 05-Jan-2009 | 407 | | 06-Jan-2009 | 6429 | | 07-Jan-2009 | 6391 | | 08-Jan-2009 | 1421 | | 09-Jan-2009 | 398 | | 10-Jan-2009 | 402 | | 11-Jan-2009 | 257 | | 12-Jan-2009 | 174 | | 13-Jan-2009 | 168 | | 14-Jan-2009 | 451 | | 15-Jan-2009 | 959 | | 16-Jan-2009 | 31410 | | 17-Jan-2009 | 79418 | | 18-Jan-2009 | 64788 | | 19-Jan-2009 | 90391 | | 20-Jan-2009 | 71683 | | 21-Jan-2009 | 104413 | | 22-Jan-2009 | 104344 | | 23-Jan-2009 | 105686 | | 24-Jan-2009 | 105853 | | 25-Jan-2009 | 1757 | +-------------+-------------+ This report shows the number of queries grouped by host IP: +-----------------+-------------+ | host | count(host) | +-----------------+-------------+ | 10.168.69.6 | 1059 | | 123.127.121.245 | 528 | | 202.106.83.125 | 530 | | 203.121.29.11 | 426 | | 203.121.29.12 | 402 | | 206.71.158.30 | 45047 | | 209.123.8.64 | 361 | | 209.123.8.99 | 617 | | 211.72.249.201 | 786 | | 211.95.81.245 | 530 | | 213.61.92.192 | 863 | | 216.201.82.19 | 4548 | | 216.201.83.2 | 3411 | | 216.240.131.173 | 1081 | | 219.142.91.125 | 530 | | 220.181.168.251 | 451 | | 58.26.5.43 | 426 | | 58.26.5.44 | 367 | | 60.247.99.245 | 530 | | 61.129.61.245 | 5 | | 63.217.28.226 | 130907 | | 66.230.128.15 | 123551 | | 66.230.160.1 | 176558 | | 66.238.93.161 | 789 | | 69.31.52.214 | 15 | | 69.50.137.175 | 22068 | | 69.50.142.11 | 114048 | | 69.50.142.110 | 15483 | | 74.86.34.144 | 1188 | | 76.9.16.171 | 57275 | | 76.9.31.42 | 72669 | | 91.199.112.18 | 344 | +-----------------+-------------+ And finally, I looked at all log entries reflecting the host ip '206.71.158.30'. The first time my dns server logged that IP address was on January 24th: +-------------+-------------+ | date | count(date) | +-------------+-------------+ | 24-Jan-2009 | 43441 | | 25-Jan-2009 | 1606 | +-------------+-------------+ Finally, when I focused strictly on logs from January 24th, 5 hosts came up: +---------------+-------------+ | host | count(host) | +---------------+-------------+ | 10.168.69.6 | 51 | | 206.71.158.30 | 43441 | | 63.217.28.226 | 57955 | | 66.230.160.1 | 4014 | | 76.9.16.171 | 392 | +---------------+-------------+ A tail end of the logs related to 206.71.158.30 indicate queries originating, on average, about one second apart: | 25-Jan-2009 | 00:22:58.644 | 206.71.158.30 | | 25-Jan-2009 | 00:22:59.056 | 206.71.158.30 | | 25-Jan-2009 | 00:23:00.565 | 206.71.158.30 | | 25-Jan-2009 | 00:23:00.643 | 206.71.158.30 | | 25-Jan-2009 | 00:23:00.949 | 206.71.158.30 | | 25-Jan-2009 | 00:23:02.640 | 206.71.158.30 | | 25-Jan-2009 | 00:23:04.330 | 206.71.158.30 | | 25-Jan-2009 | 00:23:04.639 | 206.71.158.30 | | 25-Jan-2009 | 00:23:05.283 | 206.71.158.30 | | 25-Jan-2009 | 00:23:06.646 | 206.71.158.30 | | 25-Jan-2009 | 00:23:06.792 | 206.71.158.30 | | 25-Jan-2009 | 00:23:07.176 | 206.71.158.30 | | 25-Jan-2009 | 00:23:08.653 | 206.71.158.30 | | 25-Jan-2009 | 00:23:10.556 | 206.71.158.30 | | 25-Jan-2009 | 00:23:10.653 | 206.71.158.30 | | 25-Jan-2009 | 00:23:11.509 | 206.71.158.30 | | 25-Jan-2009 | 00:23:12.652 | 206.71.158.30 | | 25-Jan-2009 | 00:23:13.018 | 206.71.158.30 | | 25-Jan-2009 | 00:23:13.402 | 206.71.158.30 | | 25-Jan-2009 | 00:23:14.656 | 206.71.158.30 | | 25-Jan-2009 | 00:23:16.665 | 206.71.158.30 | | 25-Jan-2009 | 00:23:16.783 | 206.71.158.30 | | 25-Jan-2009 | 00:23:17.736 | 206.71.158.30 | | 25-Jan-2009 | 00:23:18.666 | 206.71.158.30 | | 25-Jan-2009 | 00:23:19.245 | 206.71.158.30 | | 25-Jan-2009 | 00:23:19.629 | 206.71.158.30 | | 25-Jan-2009 | 00:23:20.662 | 206.71.158.30 | | 25-Jan-2009 | 00:23:22.658 | 206.71.158.30 | | 25-Jan-2009 | 00:23:23.010 | 206.71.158.30 | | 25-Jan-2009 | 00:23:23.963 | 206.71.158.30 | | 25-Jan-2009 | 00:23:24.665 | 206.71.158.30 | | 25-Jan-2009 | 00:23:25.472 | 206.71.158.30 | | 25-Jan-2009 | 00:23:25.856 | 206.71.158.30 | +-------------+--------------+---------------+ Andrew Brian Keefer wrote:
On Jan 23, 2009, at 12:20 PM, Luke Sheldrick wrote:
Looks to me like the target has moved, anyone else seeing similar?
It's switched again. The new target is 206.71.158.30 .
Over night it cycled through several different IPs (testing the waters?), and finally started on this one around 10:26 Pacific time this morning.
Timeline below.
-- bk
Jan 23 23:24:47 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep named[32762]: client 208.78.169.236#33027: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep last message repeated 2 times Jan 24 00:51:11 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 00:51:11 imhotep last message repeated 2 times Jan 24 00:51:30 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 00:51:30 imhotep last message repeated 2 times Jan 24 01:54:44 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 01:54:44 imhotep last message repeated 2 times Jan 24 01:55:44 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 01:55:44 imhotep last message repeated 2 times Jan 24 01:57:46 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 01:57:46 imhotep last message repeated 2 times Jan 24 02:58:29 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 02:58:30 imhotep last message repeated 2 times Jan 24 03:00:34 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 03:00:35 imhotep last message repeated 2 times Jan 24 03:05:05 imhotep named[32762]: client 208.78.169.236#33027: view ext: query (cache) './NS/IN' denied Jan 24 03:05:05 imhotep last message repeated 2 times Jan 24 03:07:49 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 04:02:38 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 04:02:38 imhotep last message repeated 2 times Jan 24 04:05:43 imhotep named[32762]: client 204.11.51.59#32802: view ext: query (cache) './NS/IN' denied Jan 24 04:05:43 imhotep last message repeated 2 times Jan 24 04:12:52 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 04:12:52 imhotep last message repeated 2 times Jan 24 05:07:37 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 05:07:37 imhotep last message repeated 2 times Jan 24 05:11:35 imhotep named[32762]: client 204.11.51.59#32802: view ext: query (cache) './NS/IN' denied Jan 24 05:11:35 imhotep last message repeated 2 times Jan 24 05:21:36 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 05:21:37 imhotep last message repeated 2 times Jan 24 06:16:06 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 06:16:06 imhotep last message repeated 2 times Jan 24 06:20:19 imhotep named[32762]: client 204.11.51.61#43329: view ext: query (cache) './NS/IN' denied Jan 24 06:20:19 imhotep last message repeated 2 times Jan 24 06:29:37 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 06:29:37 imhotep last message repeated 2 times Jan 24 06:35:11 imhotep named[32762]: client 149.20.52.161#61452: view ext: notify question section contains no SOA Jan 24 07:23:06 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 07:23:06 imhotep last message repeated 2 times Jan 24 07:28:27 imhotep named[32762]: client 204.11.51.60#32831: view ext: query (cache) './NS/IN' denied Jan 24 07:28:27 imhotep last message repeated 2 times Jan 24 07:40:25 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 07:40:25 imhotep last message repeated 2 times Jan 24 08:29:57 imhotep named[32762]: client 208.37.177.61#42517: view ext: query (cache) './NS/IN' denied Jan 24 08:29:57 imhotep last message repeated 2 times Jan 24 08:36:10 imhotep named[32762]: client 204.11.51.61#43330: view ext: query (cache) './NS/IN' denied Jan 24 08:36:11 imhotep last message repeated 2 times Jan 24 08:52:45 imhotep named[32762]: client 208.78.169.235#46265: view ext: query (cache) './NS/IN' denied Jan 24 08:52:45 imhotep last message repeated 2 times Jan 24 08:55:54 imhotep named[32762]: client 149.20.58.131#59151: view ext: query (cache) 'localhost/A/IN' denied Jan 24 09:36:38 imhotep named[32762]: client 208.37.177.62#46265: view ext: query (cache) './NS/IN' denied Jan 24 09:36:38 imhotep last message repeated 2 times Jan 24 09:43:53 imhotep named[32762]: client 204.11.51.61#43330: view ext: query (cache) './NS/IN' denied Jan 24 09:43:54 imhotep last message repeated 2 times Jan 24 09:53:56 imhotep named[32762]: client 63.217.28.226#53: view ext: query (cache) './NS/IN' denied Jan 24 10:05:28 imhotep named[32762]: client 208.78.169.234#42517: view ext: query (cache) './NS/IN' denied Jan 24 10:05:28 imhotep last message repeated 2 times Jan 24 10:26:09 imhotep named[32762]: client 206.71.158.30#18971: view ext: query (cache) './NS/IN' denied Jan 24 10:26:11 imhotep named[32762]: client 206.71.158.30#47622: view ext: query (cache) './NS/IN' denied Jan 24 10:26:13 imhotep named[32762]: client 206.71.158.30#16077: view ext: query (cache) './NS/IN' denied
-- Andrew Fried andrew.fried@gmail.com
On 2009-01-23, at 14:46, Steven Lisson wrote:
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Are they responses to queries? Or are they queries directed at servers in your network? The latter are to be expected, I think. Joe
On 24/01/2009, at 6:46 AM, Steven Lisson wrote:
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards, Steve
Hi Steve, There is at least an iptables rule you can use to drop this specific query, assuming your nameservers run linux. http://www.stupendous.net/archives/2009/01/24/dropping-spurious-nsin-recursi... The bind-users mailing list suggested having the ISPs trace back the flows and find the networks emitting the spoofed packets, and have those networks implement BCP 38. While that's the 'right' solution (everyone should be doing ingress filtering, sure, impossible to argue against it), not every network out there is operated by people who give a damn. This will work at least until the kiddies improve their scripts to query for names that actually exist. On 24/01/2009, at 8:21 AM, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Good luck with that. Right now they're targetting ISPrime, and you've just made the DoS even more effective for them. With any luck, the rest of the world will follow suit and the bad guys win! yay! :) Short of getting the rest of the world to properly implement ingress filtering (ha, ha), I think dropping the specific packets that generate the reflected traffic is good enough for now. The load on the reflectors is minimal. Nathan.
In message <9A251497-E94C-4693-8E89-3FD3ACF6D138@stupendous.net>, Nathan Ollere nshaw writes:
On 24/01/2009, at 6:46 AM, Steven Lisson wrote:
Hi,
I agree with seeing no traffic to/from 66.230.128.15 but am still seeing flows 'from' 66.230.160.1
Regards, Steve
Hi Steve,
There is at least an iptables rule you can use to drop this specific query, assuming your nameservers run linux.
http://www.stupendous.net/archives/2009/01/24/dropping-spurious-nsin-recursi... e-queries/
The bind-users mailing list suggested having the ISPs trace back the flows and find the networks emitting the spoofed packets, and have those networks implement BCP 38.
It was also said here.
While that's the 'right' solution (everyone should be doing ingress filtering, sure, impossible to argue against it), not every network out there is operated by people who give a damn.
I would suggest that you don't want to peer with such networks. I would suggest that deploying BCP 38 be a requirement for peering.
This will work at least until the kiddies improve their scripts to query for names that actually exist.
On 24/01/2009, at 8:21 AM, Chris McDonald wrote:
We [AS3491] null0'd the IP earlier. Rest-of-world encouraged to do the same :/
Good luck with that. Right now they're targetting ISPrime, and you've just made the DoS even more effective for them. With any luck, the rest of the world will follow suit and the bad guys win! yay! :)
Short of getting the rest of the world to properly implement ingress filtering (ha, ha), I think dropping the specific packets that generate the reflected traffic is good enough for now. The load on the reflectors is minimal.
Nathan.
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: Mark_Andrews@isc.org
I'm not sure you're entirely out of the water yet: 17:13:45.680944 76.9.16.171.53868 > XXXXXXXX.53: 58451+ NS? . (17) 17:13:45.681251 XXXXXXXX.53 > 76.9.16.171.53868: 58451 Refused- 0/0/0 (17) CIDR: 76.9.0.0/19 NetName: ISPRIME-ARIN-3 In addition to the one that Brian Keefer mentioned a few days ago (206.71.158.30). But on that subject, I figured I'd toss in a (sad) anecdote about security and upgrades. I'd upgraded this nameserver to bind-9 some time ago, during a bit of a security panic. And in the process, I screwed it up - I'd updated the machine itself, but had failed to propagate the changes to the master that sends updates to all of the servers. The obvious thing happened: after a while, this nameserver pulled its updates from the master, and downgraded to bind-8 again, which we didn't notice until I saw it spitting full cached NS responses to isprime hosts. Human error strikes again. Apologies for letting my host be an amplifier. -Dave On Jan 23, 2009, at 1:11 PM, Phil Rosenthal wrote:
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now.
-Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
I just took a snapshot of my bind logs from the past two hours (on 01/25/209 at 14:40 EST). Based on what I'm seeing, four DNS servers are still under attack at varying levels. 206.71.158.30 is bearing the brunt of the attacks. And as you indicated, 76.9.16.171 is still being targeted, although to a lesser degree than before. +---------------+-------------+ | host | count(host) | +---------------+-------------+ | 10.168.69.6 | 3 | | 206.71.158.30 | 6513 | | 63.217.28.226 | 182 | | 66.230.160.1 | 266 | | 76.9.16.171 | 92 | +---------------+-------------+ -- Andrew Fried andrew.fried@gmail.com David Andersen wrote:
I'm not sure you're entirely out of the water yet:
17:13:45.680944 76.9.16.171.53868 > XXXXXXXX.53: 58451+ NS? . (17) 17:13:45.681251 XXXXXXXX.53 > 76.9.16.171.53868: 58451 Refused- 0/0/0 (17)
CIDR: 76.9.0.0/19 NetName: ISPRIME-ARIN-3
In addition to the one that Brian Keefer mentioned a few days ago (206.71.158.30).
But on that subject, I figured I'd toss in a (sad) anecdote about security and upgrades. I'd upgraded this nameserver to bind-9 some time ago, during a bit of a security panic. And in the process, I screwed it up - I'd updated the machine itself, but had failed to propagate the changes to the master that sends updates to all of the servers. The obvious thing happened: after a while, this nameserver pulled its updates from the master, and downgraded to bind-8 again, which we didn't notice until I saw it spitting full cached NS responses to isprime hosts. Human error strikes again. Apologies for letting my host be an amplifier.
-Dave
On Jan 23, 2009, at 1:11 PM, Phil Rosenthal wrote:
Just a friendly notice, the attack against 66.230.128.15/66.230.160.1 seems to have stopped for now.
-Phil On Jan 22, 2009, at 6:01 AM, Bjørn Mork wrote:
Graeme Fowler <graeme@graemef.net> writes:
I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question.
Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away.
Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic?
Did you filter *only* 66.230.128.15/66.230.160.1, or are you dropping traffic from other sources too? Looks like some of the other source addresses are controlled by the DOSers. Possibly used to detect filters?
These clients may look similar to the DOS attack, but there are subtle differences:
Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:33 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:08:34 canardo named[32046]: client 211.72.249.201#29656: view external: query (cache) './NS/IN' denied Jan 18 05:47:00 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 05:47:01 canardo named[32046]: client 211.72.249.201#29662: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:22 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 06:25:23 canardo named[32046]: client 211.72.249.201#29664: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:41 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:03:42 canardo named[32046]: client 211.72.249.201#29667: view external: query (cache) './NS/IN' denied Jan 18 07:42:08 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 07:42:09 canardo named[32046]: client 211.72.249.201#29670: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:29 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:20:30 canardo named[32046]: client 211.72.249.201#29673: view external: query (cache) './NS/IN' denied Jan 18 08:58:50 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 08:58:51 canardo named[32046]: client 211.72.249.201#29678: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:12 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied Jan 18 09:37:13 canardo named[32046]: client 211.72.249.201#29679: view external: query (cache) './NS/IN' denied
Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:02:51 canardo named[32046]: client 213.61.92.192#46716: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 07:41:21 canardo named[32046]: client 213.61.92.192#46752: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:19:46 canardo named[32046]: client 213.61.92.192#46785: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 08:58:12 canardo named[32046]: client 213.61.92.192#46808: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 09:36:34 canardo named[32046]: client 213.61.92.192#46833: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied Jan 20 10:14:58 canardo named[32046]: client 213.61.92.192#46858: view external: query (cache) './NS/IN' denied
Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 06:27:28 canardo named[32046]: client 66.238.93.161#34373: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:05:55 canardo named[32046]: client 66.238.93.161#34420: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:20 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 07:44:21 canardo named[32046]: client 66.238.93.161#34473: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 08:22:38 canardo named[32046]: client 66.238.93.161#34503: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:00:56 canardo named[32046]: client 66.238.93.161#34540: view external: query (cache) './NS/IN' denied Jan 22 09:39:20 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied Jan 22 09:39:21 canardo named[32046]: client 66.238.93.161#34574: view external: query (cache) './NS/IN' denied
Notice the pattern: 3 probes every 38 minutes Each probe from the same source port Source port increases slowly and steadily
This looks like some application actually waiting for a response. The slow source port change is probably an indication that this client only tests a small number of DNS servers. I guess that this client is either one of the many bots used to send the spoofed requests, or maybe a bot not allowed to spoof its source and therefore used for other purposes. In any case, I assume that other DNS servers may see such control sessions coming from other addresses.
These 3 clients started probing my DNS server almost simultaneously on January 8th:
Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:33:52 canardo named[26496]: client 213.61.92.192#31195: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:29 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:36:30 canardo named[26496]: client 66.238.93.161#11299: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied Jan 8 19:37:47 canardo named[26496]: client 211.72.249.201#29112: view external: query (cache) './NS/IN' denied
Maybe preparing for the attack on ISPrime? I didn't start receiving spoofed requests from 66.230.128.15/66.230.160.1 before January 20th
I just tried filtering the probing addresses. This made the probing stop immediately after dropping a set of 3 probes. But the spoofed requests continuted at the same rate as before, so this does not support my theory.
However, I believe it would be too much of a coincidence if there isn't some connection between the probing and the DOS attack. It would be interesting to hear if others see similar probing.
Bjørn
participants (41)
-
Aaron Hopkins
-
Andrew Fried
-
Bjørn Mork
-
Brandon Galbraith
-
Brian Keefer
-
Chris McDonald
-
Christopher Morrow
-
Crist Clark
-
Dale Carstensen
-
Danny McPherson
-
David Andersen
-
David Conrad
-
Eugeniu Patrascu
-
Frank Bulk
-
Gadi Evron
-
Graeme Fowler
-
Harald Koch
-
J.D. Falk
-
Jack Bates
-
James Hess
-
Jamie A Lawrence
-
Jeffrey Lyon
-
Joe Abley
-
Jon Kibler
-
Justin Krejci
-
Lorell Hathcock
-
Luke Sheldrick
-
Mark Andrews
-
Martin Hannigan
-
Michael Dillon
-
Mike Lyon
-
Nathan Ollerenshaw
-
Noel Butler
-
Paul Ferguson
-
Phil Rosenthal
-
Roland Dobbins
-
Seth Mattinen
-
Steven Lisson
-
Todd T. Fries
-
Valdis.Kletnieks@vt.edu
-
Xaver Aerni