Just a quick note to let folks know about a new vulnerability we have found in some low-rent DNS forwarders---which we have been calling the 'preplay attack'. The finding is that when the vulnerable open resolvers receive a DNS response they just look at the query string in the response to see if they have a request for the given string outstanding. If they do, they accept the result. I.e., there is no validating of the source IP, port numbers or DNS transaction ID in the response. Dumb. This makes poisoning the caches of these boxes trivial (i.e., send a request for www.facebook.com and then immediately send an answer). A few notes ... - We have found 7--9% of the open resolver population---or 2-3 million boxes---to be vulnerable to this cache poisoning attack. (The variance is from different runs of our experiments.) - We have not been able to nail this vulnerability down to a single box or manufacturer. To the contrary our efforts at identifying the boxes indicates it crosses such boundaries. (However, these boxes do seem to be largely situated in residential settings.) - We presented these results at PAM earlier this week. Our paper, slides, etc. with details of the attack (and results about previously known DNS attacks) are available here: http://www.icir.org/mallman/pubs/SCRA14/ - We did give CERT a heads up about this before the paper appeared and they kibitzed the information around to various manufacturers of this sort of gear. My mental model is that this sort of gear is upgraded when it goes kaput. So, vigilance I guess. FWIW. allman -- http://www.icir.org/mallman/
On 14/03/2014 13:45, Mark Allman wrote:
- We have found 7--9% of the open resolver population---or 2-3 million boxes---to be vulnerable to this cache poisoning attack. (The variance is from different runs of our experiments.)
did you characterise what dns servers / embedded kit were vulnerable? If so, can you share the results? Nick
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable?
He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable?
He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues. Has someone / is someone doing this? - merike
Well, at least all this CPE checks in for security updates every night so this should be fixable. Oh wait, no, nevermind, they don't. :-( This is getting to be the vulnerability of the week club for home gateway devices - quite concerning. JL On 3/14/14, 12:05 PM, "Merike Kaeo" <merike@doubleshotsecurity.com> wrote:
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable?
He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues.
Has someone / is someone doing this?
- merike
Why would a CPE have an open DNS resolver from the WAN side? Gary Baribault On 03/14/2014 12:45 PM, Livingood, Jason wrote:
Well, at least all this CPE checks in for security updates every night so this should be fixable. Oh wait, no, nevermind, they don't. :-(
This is getting to be the vulnerability of the week club for home gateway devices - quite concerning.
JL
On 3/14/14, 12:05 PM, "Merike Kaeo" <merike@doubleshotsecurity.com> wrote:
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable? He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues.
Has someone / is someone doing this?
- merike
Why would a CPE have an open DNS resolver from the WAN side?
Honest to god, are you new to computers or something? People have been writing "just good enough" code since the beginning. A resolver package binds to *:53 by default. Some poor firmware guys with no security experience, deadlines, and too few bytes for code storage don't notice or don't know or don't care and install the resolver feature on the firmware that they're designing, then promptly never think about it again "because that feature works and is therefore done." ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Good question, but the reality is that a lot of them are this way. They just forward everything from any source. Maybe it was designed that way to support DDoS as a use case. Imagine a simple iptables rule like -p udp --dport 53 -j DNAT --to 4.2.2.4 I think some forwarders work this way - the LAN addresses can be reconfigured and so it's probably easier if the rule doesn't check the source address.. or maybe it was designed to work this way on purpose, because it's easy to explain as a 'bug' or oversight, rather than deliberate action. Of course, it's crazy to think that some person or organization deliberately did this so they would have a practically unlimited amount of DoS sources. -Laszlo On Mar 15, 2014, at 4:26 PM, Gary Baribault <gary@baribault.net> wrote:
Why would a CPE have an open DNS resolver from the WAN side?
Gary Baribault
On 03/14/2014 12:45 PM, Livingood, Jason wrote:
Well, at least all this CPE checks in for security updates every night so this should be fixable. Oh wait, no, nevermind, they don't. :-(
This is getting to be the vulnerability of the week club for home gateway devices - quite concerning.
JL
On 3/14/14, 12:05 PM, "Merike Kaeo" <merike@doubleshotsecurity.com> wrote:
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable? He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues.
Has someone / is someone doing this?
- merike
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 That's a good question, but I know that during the ongoing survey within the Open Resolver Project [http://openresolverproject.org/], Jared found thousands of CPE devices which responded as resolvers. Further work needs to go into fingerprinting these devices to determine the vendor, version, etc., but it is disturbing to see such brokenness. :-/ - - ferg On 3/15/2014 9:26 AM, Gary Baribault wrote:
Why would a CPE have an open DNS resolver from the WAN side?
Gary Baribault
On 03/14/2014 12:45 PM, Livingood, Jason wrote:
Well, at least all this CPE checks in for security updates every night so this should be fixable. Oh wait, no, nevermind, they don't. :-(
This is getting to be the vulnerability of the week club for home gateway devices - quite concerning.
JL
On 3/14/14, 12:05 PM, "Merike Kaeo" <merike@doubleshotsecurity.com> wrote:
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable? He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues.
Has someone / is someone doing this?
- merike
- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlMkgYQACgkQKJasdVTchbLR1AD9Ey+ISQtaVoJKReLZ6ZzHI7/4 91h+HIQgvazMAne+NMsA/3CCQVw9KG1U6oZdouKexi8ycVw1Y4d4poH+7Yfh4zEh =bFpE -----END PGP SIGNATURE-----
[catching up]
That's a good question, but I know that during the ongoing survey within the Open Resolver Project [http://openresolverproject.org/], Jared found thousands of CPE devices which responded as resolvers.
Not thousands, *tens of millions*. Our estimate from mid-2013 was 32M such devices (detailed in an IMC paper last year; http://www.icir.org/mallman/pubs/SCRA13/). And, that roughly agrees with both the openresolverproject.org numbers and another (not public) study I know of. And, as if that isn't bad enough ... there is a 2010 IMC paper that puts the number at 15M. I.e., the instances of brokenness are getting worse---doubling in 3 years! UGH. allman
On Apr 2, 2014, at 8:38 AM, Mark Allman <mallman@icir.org> wrote:
[catching up]
That's a good question, but I know that during the ongoing survey within the Open Resolver Project [http://openresolverproject.org/], Jared found thousands of CPE devices which responded as resolvers.
Not thousands, *tens of millions*.
Our estimate from mid-2013 was 32M such devices (detailed in an IMC paper last year; http://www.icir.org/mallman/pubs/SCRA13/). And, that roughly agrees with both the openresolverproject.org numbers and another (not public) study I know of. And, as if that isn't bad enough ... there is a 2010 IMC paper that puts the number at 15M. I.e., the instances of brokenness are getting worse---doubling in 3 years! UGH.
One observation: The OpenResolverProject collects responses that come from ports that the query was not sent to (ie: device responds from UDP/12345 not from UDP/53, which obviously is broken and doesn't "work", but they actually return DNS payload which can be used for abuse). Some good news though: http://openresolverproject.org/breakdown-graph1.cgi Since the start of 2014 there seem to be new CPE devices out there that are resolving this issue. The linear nature of the line in the decrease doesn't seem to be something like "ISPs" started blocking udp/53 to customers, which would appear more like a step function. I'm aware of some other studies ongoing to fingerprint CPE and their behaviors/aggregated resolver dependencies. I expect to see some of that data presented at the upcoming DNS-OARC meeting in Warsaw. Getting everyone to update their firmware on devices would go a long way as well. Some vendors have no software QA on this front so add/remove the response on the WAN interface as their releases march forward. - Jared
In message <C7E435C6-344F-49CD-9152-7A9EF2FA6662@puck.nether.net>, Jared Mauch writes:
On Apr 2, 2014, at 8:38 AM, Mark Allman <mallman@icir.org> wrote:
[catching up]
That's a good question, but I know that during the ongoing survey within the Open Resolver Project [http://openresolverproject.org/], Jared found thousands of CPE devices which responded as resolvers.
Not thousands, *tens of millions*.
Our estimate from mid-2013 was 32M such devices (detailed in an IMC paper last year; http://www.icir.org/mallman/pubs/SCRA13/). And, that roughly agrees with both the openresolverproject.org numbers and another (not public) study I know of. And, as if that isn't bad enough ... there is a 2010 IMC paper that puts the number at 15M. I.e., the instances of brokenness are getting worse---doubling in 3 years! UGH.
One observation: The OpenResolverProject collects responses that come from ports that the query was not sent to (ie: device responds from UDP/12345 not from UDP/53, which obviously is broken and doesn't "work", but they actually return DNS payload which can be used for abuse).
Some good news though:
I see axes, legend but no data points. If I hover over various spots on the graph I see data values pop up.
Since the start of 2014 there seem to be new CPE devices out there that are resolving this issue. The linear nature of the line in the decrease doesn't seem to be something like "ISPs" started blocking udp/53 to customers, which would appear more like a step function.
I'm aware of some other studies ongoing to fingerprint CPE and their behaviors/aggregated resolver dependencies. I expect to see some of that data presented at the upcoming DNS-OARC meeting in Warsaw.
Getting everyone to update their firmware on devices would go a long way as well. Some vendors have no software QA on this front so add/remove the response on the WAN interface as their releases march forward.
- Jared
-- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
Have we ascertained if there is a typical configuration adjustment that can be made to reduce or eliminate the likelihood of impact? (From the description it sounds as though this is not possible but it doesn't hurt to ask.) On Fri, Mar 14, 2014 at 09:05:00AM -0700, Merike Kaeo wrote:
On Mar 14, 2014, at 7:06 AM, Stephane Bortzmeyer <bortzmeyer@nic.fr> wrote:
On Fri, Mar 14, 2014 at 01:59:27PM +0000, Nick Hilliard <nick@foobar.org> wrote a message of 10 lines which said:
did you characterise what dns servers / embedded kit were vulnerable?
He said "We have not been able to nail this vulnerability down to a single box or manufacturer" so it seems the answer is No.
It is my understanding that many CPEs work off of same reference implementation(s). I haven't had any cycles for this but with all the CPE issues out there it would be interesting to have a matrix of which CPEs utilize which reference implementation. That may start giving some clues.
Has someone / is someone doing this?
- merike
--- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
On Fri, Mar 14, 2014 at 5:06 PM, Wayne E Bouchard <web@typo.org> wrote:
Have we ascertained if there is a typical configuration adjustment that can be made to reduce or eliminate the likelihood of impact?
I think your best tactic is: Provide specified DNS resolver cache servers. Don't use CPEs for DNS forwarders. The trouble is.... a CPE's management/locally-bound IP address is in many cases... often the same IP address that is a NAT address shared with user traffic; instead of a dedicated separate IP address that traffic can be managed and security controlled. Providing you ensure that the CPE's IP bound address is not overloaded or shared with user traffic ---- you might try firewalling destination port 53 to the CPE, except from the proper upstream DNS resolvers, since nothing else should be "replying" to a DNS request made by the CPE. Look into whether the CPE can use a different, lesser-used UDP port than 53 to forward DNS requests to; use device firewall rules or upstream ACLs to limit which source IP addresses can talk to the service on the CPE's IP. To ascertain effectiveness for a specific CPE, you would need to run a sample exploit with a before and after test.
(From the description it sounds as though this is not possible but it doesn't hurt to ask.)
-- -JH
participants (13)
-
Gary Baribault -
Jared Mauch -
Jimmy Hess -
Joe Greco -
Laszlo Hanyecz -
Livingood, Jason -
Mark Allman -
Mark Andrews -
Merike Kaeo -
Nick Hilliard -
Paul Ferguson -
Stephane Bortzmeyer -
Wayne E Bouchard