DNS - connection limit (without any extra hardware)
Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us? George Roettger -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Luke Sent: Friday, December 08, 2006 9:41 AM To: nanog@nanog.org Subject: DNS - connection limit (without any extra hardware) Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much. For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper? Thanks Best Regards Luke
On Fri, 8 Dec 2006, Geo. wrote:
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us?
George Roettger
Atually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us. Luke: It is possible the DNS queries made are for non existent domains, fake replies, perhaps even making them something in 1918 space, and they MAY stop being not nice netizens. Gadi.
From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of Luke Sent: Friday, December 08, 2006 9:41 AM To: nanog@nanog.org Subject: DNS - connection limit (without any extra hardware)
Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much.
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper?
Thanks Best Regards
Luke
Actually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us.
Infected machines (bots) will cause a lot more than just DNS issues. Issues like this have a way of getting worse all by themselves if not addressed. Anyway, to play nice.. how about using a router to dampen traffic much like icmp dampening? Would it be possible to do DNS dampening? Geo.
On 8-Dec-2006, at 11:52, Geo. wrote:
Actually, reading your reply (which is the same as my own, pretty much), I figure the guy asked a question and he has a real problem. Assuming he doesn't want to clean them up is not nice of us.
Infected machines (bots) will cause a lot more than just DNS issues. Issues like this have a way of getting worse all by themselves if not addressed.
Anyway, to play nice.. how about using a router to dampen traffic much like icmp dampening? Would it be possible to do DNS dampening?
I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router. There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle). You may find that you can install ipfw (or similar) rules on your nameservers themselves to do this kind of thing. Take careful note of what happens when the client population becomes large, though -- the garbage collection ought to be smooth and painless, or you'll just wind up swapping one worm proliferation failure mode for another. Host-based per-client rate limits scale better if there are many hosts providing service, e.g. behind a load balancer or using something like <http://www.isc.org/pubs/tn/isc-tn-2004-1.html>. As to the wider question, cleaning up the infected hosts is an excellent goal, but it'd certainly be nice if your DNS servers continued to function while you were doing so. Having every non- infected customer phone up screaming at once can be an unwelcome distraction when you already have more man hours of work to do per day than you have (staff * 24). Joe
On Dec 8, 2006, at 12:57 PM, Joe Abley wrote:
I think the trouble comes when you want to limit the request rate *per client source address*, rather than limiting the request rate across the board. That implies the retention of state, and since DNS transactions are brief (and since the client population is often large) that can add up to a lot of state to keep at an aggregation point like a router.
There some appliances which are designed to hold large amounts of state (e.g. f5's big-ip) but you're talking non-trivial dollars for that. Beware enterprise-scale stateful firewall devices which might seem like sensible solutions to this problem. They are often not suitable for use in front of busy DNS servers (even a few hundred new flows per second is a lot for some vendors, despite the apparent marketing headroom based on the number of kbps you need to handle).
Folks should also look at some of the DNS appliances (I know, this is "extra hardware"). Although the usually run BIND, they tend to be fairly optimized and have extra management functionality that may help with the rate limiting (if not, its probably a feature request that the vendors would entertain rapidly, as there's some pretty intense competition). Some folks to talk to - Infoblox and Bluecat. If you have really large DNS rate requirements, I'd consider talking to Nominum. I'm curious as to just how bursty things are - how large of a departure from normality are we talking about? An order of magnitude? Two? - Dan
You may find that you can install ipfw (or similar) rules on your nameservers themselves to do this kind of thing. Take careful note of what happens when the client population becomes large, though -- the garbage collection ought to be smooth and painless, or you'll just wind up swapping one worm proliferation failure mode for another.
Host-based per-client rate limits scale better if there are many hosts providing service, e.g. behind a load balancer or using something like <http://www.isc.org/pubs/tn/isc-tn-2004-1.html>.
As to the wider question, cleaning up the infected hosts is an excellent goal, but it'd certainly be nice if your DNS servers continued to function while you were doing so. Having every non- infected customer phone up screaming at once can be an unwelcome distraction when you already have more man hours of work to do per day than you have (staff * 24).
Joe
On Sun, 10 Dec 2006, Daniel Golding wrote:
Folks should also look at some of the DNS appliances (I know, this is "extra hardware"). Although the usually run BIND, they tend to be fairly optimized and have extra management functionality that may help with the rate limiting (if not, its probably a feature request that the vendors would entertain rapidly, as there's some pretty intense competition). Some folks to talk to - Infoblox and Bluecat.
I'm not sure what you mean by "optimized" here, but I suspect that the only part optimized is the user interface for configuring per-client policies that still do not scale, but I would be glad to be proven wrong.
If you have really large DNS rate requirements, I'd consider talking to Nominum.
I agree with you there; but that's sort of a given :) matto --matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
On Fri, 8 Dec 2006, Gadi Evron wrote:
Luke: It is possible the DNS queries made are for non existent domains, fake replies, perhaps even making them something in 1918 space, and they MAY stop being not nice netizens.
Configuring your nameservers to randomly give bad answers isn't considered being a "nice netizen" either, the last time I checked. --matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
Geo. wrote:
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us?
Has anyone figured out a remote but lawful way to repair zombie machines? Pete
George Roettger
-----Original Message----- *From:* owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]*On Behalf Of *Luke *Sent:* Friday, December 08, 2006 9:41 AM *To:* nanog@nanog.org *Subject:* DNS - connection limit (without any extra hardware)
Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much.
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper?
Thanks Best Regards
Luke
On Fri, 8 Dec 2006, Petri Helenius wrote:
Geo. wrote:
I know this is kind of a crazy idea but how about making cleaning up all these infected machines the priority as a solution instead of defending your dns from your infected clients. They not only affect you, they affect the rest of us so why should we give you a solution to your problem when you don't appear to care about causing problems for the rest of us?
Has anyone figured out a remote but lawful way to repair zombie machines?
Microsoft auto-update, the telephone line, going to a different country with a different set of rules. Gadi.
Aaron Glenn wrote:
On 12/8/06, Petri Helenius <pete@he.iki.fi> wrote:
Has anyone figured out a remote but lawful way to repair zombie machines?
sure, null route the customer until they clean their hosts up
My question was specifically directed towards zombies that are not local to the ISP. Pete
On Fri, 2006-12-08 at 19:56 +0200, Petri Helenius wrote:
Has anyone figured out a remote but lawful way to repair zombie machines?
Very interesting question. I personally believe that OS EULAs and ISP ToS guidelines provide for an ISP or an OS mfg (i.e. Microsoft) to force updates and fixes via any means. That is: if I am your customer and my PC/router/USB-Camera/whatever is throwing crap your way, crap that violates your ToS or indicates that I am out of compliance with an EULA, then I believe others have the right (and IMHO the obligation) to step in and correct things (it's what parents do for their kids everyday). So, according to me, any corrective action is lawful when dealing with customers and equipment that have violated an EULA or ToS guidelines. Just my $.02. ;-) -Jim P.
On Fri, 8 Dec 2006, Jim Popovitch wrote:
On Fri, 2006-12-08 at 19:56 +0200, Petri Helenius wrote:
Has anyone figured out a remote but lawful way to repair zombie machines?
Very interesting question. I personally believe that OS EULAs and ISP ToS guidelines provide for an ISP or an OS mfg (i.e. Microsoft) to force updates and fixes via any means. That is: if I am your customer and my PC/router/USB-Camera/whatever is throwing crap your way, crap that violates your ToS or indicates that I am out of compliance with an EULA, then I believe others have the right (and IMHO the obligation) to step in and correct things (it's what parents do for their kids everyday). So, according to me, any corrective action is lawful when dealing with customers and equipment that have violated an EULA or ToS guidelines.
Sending updates in automated way or forcing updates is only ok if person previously authorized such action, i.e. enabled automated updates. This is in fact dangerous in itself since it also presents single point of potential failure if system providing updates is itself compromised - that is why many choose not to do it and enterprises setup their own updates distribution systems. As far as your question, in my opinion it would be legal for you to check if somebody did or did not do an update but only using tools that check publicly available data reported from the system (i.e. what you can gather by sending it packets to open ports). As an ISP it would be legal for you to warn customer that if they fail to install an update you reserve the right to disconnect their system or limit access to certain ports or only to certain sites (i.e. your own for them to check email but nothing else). And obviously once issue is reported to you (i.e. their machine is spewing and compromised), that is exactly what you should do.
Just my $.02. ;-)
Due to inflation with US currency I'll make it a nickel $.05 :) -- William Leibzon Elan Networks william@elan.net
On Sun, 10 Dec 2006, Petri Helenius wrote:
Virtual patching.
How do I virtual patch the machine in ireland which attacked my mail server just a few minutes ago?
You don't patch the machine in Ireland, but once your "virtual patching box" identifies a hostile system and identifies what it is infected with, it can then do the virtual patching on your end so that all subsequent pkts entering from that machine in Ireland are cleaned and no longer hostile. -Hank
On Sun, 10 Dec 2006, Hank Nussbacher wrote:
On Sun, 10 Dec 2006, Petri Helenius wrote:
Virtual patching.
How do I virtual patch the machine in ireland which attacked my mail server just a few minutes ago?
You don't patch the machine in Ireland, but once your "virtual patching box" identifies a hostile system and identifies what it is infected with, it can then do the virtual patching on your end so that all subsequent pkts entering from that machine in Ireland are cleaned and no longer hostile.
I don't follow. Three monkies? Hitchhiker's Guide towel? Gadi.
-Hank
Hank Nussbacher wrote:
On Sun, 10 Dec 2006, Petri Helenius wrote:
Virtual patching.
How do I virtual patch the machine in ireland which attacked my mail server just a few minutes ago?
You don't patch the machine in Ireland, but once your "virtual patching box" identifies a hostile system and identifies what it is infected with, it can then do the virtual patching on your end so that all subsequent pkts entering from that machine in Ireland are cleaned and no longer hostile. Does it reset the evil bit too?
Pete
On Dec 8, 2006, at 9:56 AM, Petri Helenius wrote:
Has anyone figured out a remote but lawful way to repair zombie machines?
Having remote power control over all of our customer's equipment. Though the customer might not consider that a "repair", I do :-) -- Jo Rhett senior geek Silicon Valley Colocation
On Friday 08 December 2006 14:40, you wrote:
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address.
That may trap an email server or two. Did you consider checking what they are looking up, and lying to them about the TTL/answer "127.0.0.1 for a week" maybe better than NXDOMAIN. I use to slave "." which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross). I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
On Fri, 8 Dec 2006, Simon Waters wrote:
I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
Indeed. It is generally accepted that it is easier to simply scale your service to provide adequate headroom than implement per-client traffic policies. of course, you could also work on cleaning up the mess, but I will charitably assume you are working the problem from both directions simultaneously. matto --matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
of course, my company is working on two main tasks: the first team is focused on discovering what is the virus, and what is the best anti-virus. instead, my team has already scaled our DNS service, by doubling the number of DNSs. I'm not completely satisfied by the "scaling solution": I wish to find a solution that could grant a good quality of the service without placing a lot of DNS in my web-farms Thanks Best Regards Luke On 12/8/06, Matt Ghali <matt@snark.net> wrote:
On Fri, 8 Dec 2006, Simon Waters wrote:
I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
Indeed. It is generally accepted that it is easier to simply scale your service to provide adequate headroom than implement per-client traffic policies.
of course, you could also work on cleaning up the mess, but I will charitably assume you are working the problem from both directions simultaneously.
matto
--matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
I use to slave "." which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross).
I'm not sure to understand your solution. You configure your name-server as a slave-root-server? On 12/8/06, Simon Waters <simonw@zynet.net> wrote:
On Friday 08 December 2006 14:40, you wrote:
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address.
That may trap an email server or two.
Did you consider checking what they are looking up, and lying to them about the TTL/answer "127.0.0.1 for a week" maybe better than NXDOMAIN.
I use to slave "." which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross).
I suspect complex rate limiting may be nearly as expensive as providing DNS answers with Bind9.
On Monday 11 December 2006 16:15, you wrote:
I use to slave "." which can save time on recursive DNS servers when they have a lot of dross to answer (assuming it is totally random dross).
I'm not sure to understand your solution. You configure your name-server as a slave-root-server?
Yes. Most of the root server traffic is answering queries with "NXDOMAIN" for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain. The drawback is you provide the answer with the authority bit set, which isn't what the world's DNS clients should expect, but DNS clients don't care about that one bit (sorry). If the root zone file changed quickly it might also cause other problems! Paul V was very cautious about it as a method of running a DNS server, but if the recursive servers are being barraged with queries for (different) non-existent top level domains I think it is probably preferable to the servers being flattened (and/or passing that load onto the root name servers). If the queries are for existing, or the same, domains each time, it won't provide significant improvement. I suppose any server issuing more than 2000 or so queries a day to the root servers would potentially save bandwidth, and provide a more responsive experience for the end user. But one also has to handle the case of the root zone potentially expiring, not something I ever allowed to happen, but then I'm not the average DNS administrator. I've used this technique extensively myself in the past with no issues, but I'm not using it operationally at the moment. Since the load average on our DNS server is 0.00 to two decimal places I doubt it would make a lot of difference, and we host websites, and email, not randomly misconfigured, home, or business user PCs. So mostly we do lookups in in-addr.arpa, a depressingly large proportion of which fail, or look-ups for a small set of servers we forward email to (most of which exist, or I delete the forward).
On Mon, 11 Dec 2006, Simon Waters wrote:
Yes. Most of the root server traffic is answering queries with "NXDOMAIN" for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain.
That would require configuring my caching server with authoritative zones, and it seems prevailing wisdom (at least with BIND configurations?) is to keep the peanut butter seperate from the chocolate, no matter how great they taste together, to the best of my knowledge. matto --matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
In article <Pine.LNX.4.64.0612111613480.26126@pants.snark.net> you write:
On Mon, 11 Dec 2006, Simon Waters wrote:
Yes. Most of the root server traffic is answering queries with "NXDOMAIN" for non-existant top level domains, if you slave root on your recursive servers, your recursive servers can answer those queries directly (from the 120KB root zone file), rather than relying on negative caching, and a round trip to the root servers, for every new non-existant domain.
That would require configuring my caching server with authoritative zones, and it seems prevailing wisdom (at least with BIND configurations?) is to keep the peanut butter seperate from the chocolate, no matter how great they taste together, to the best of my knowledge.
matto
No. The wisdom is to not make your authoritative servers caches. This is not the same as not making your caches authoritative for certain zones. Just don't have the caches listed in the NS RRsets. Note: You will need to configure your master server(s) to notify the caches for the zone that slave as the automatic mechanisms won't discover them. Mark
--matt@snark.net------------------------------------------<darwin>< Moral indignation is a technique to endow the idiot with dignity. - Marshall McLuhan
On Fri, 8 Dec 2006, Luke wrote:
Hi, as a comsequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much.
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper?
"I have a bots infested network, they really task my services! How can I make my services ignore them so that the clients start calling me and spending my tech support budget?"
Thanks Best Regards
Luke
Gadi.
"I have a bots infested network, they really task my services! How can I make my services ignore them so that the clients start calling me and spending my tech support budget?"
Or: "I have bots on my network and as part of a multi-pronged approach to cleaning my network while keeping the services available to those who aren't infected, I'd like to research ways that I can minimize the effect these bots have on the rest of my customers" Cheers, .pm
On Dec 8, 2006, at 6:40 AM, Luke wrote:
Hi, as a consequence of a virus diffused in my customer-base, I often receive big bursts of traffic on my DNS servers. Unluckly, a lot of clients start to bomb my DNSs at a certain hour, so I have a distributed tentative of denial of service. I can't blacklist them on my DNSs, because the infected clients are too much.
For this reason, I would like that a DNS could response maximum to 10 queries per second given by every single Ip address. Anybody knows a solution, just using iptables/netfilter/kernel tuning/BIND tuning, without using any hardware traffic shaper?
One effective strategy is to make 0wning your customer's system less profitable. Here is a good article by Suresh Ramasubramanian: http://www.circleid.com/posts/ port_25_blocking_or_fix_smtp_and_leave_port_25_alone_for_the_sake_of_spa m/ Some have been successful with notification tools such as those offered by: http://www.perftech.com/ Customers are directed to a free scrub that does not depend upon OS validation status, such as Housecall. -Doug
participants (17)
-
Aaron Glenn -
Daniel Golding -
Douglas Otis -
Gadi Evron -
Geo. -
Hank Nussbacher -
Jim Popovitch -
Jo Rhett -
Joe Abley -
Luke -
Luke C
-
Mark Andrews -
Matt Ghali -
Petri Helenius -
Simon Waters -
william(at)elan.net -
♓