Most of our DSL customers have modem/routers that resolve DNS externally. And most of those have no configuration option to stop it. So, we took the unfortunate step of ACL blocking DNS requests to & from the DSL network unless the requests are to our DNS servers. Suboptimal, but it stopped the DNS amplification attacks. -----Original Message----- From: Mikael Abrahamsson [mailto:swmike@swm.pp.se] Sent: Monday, April 01, 2013 11:51 AM To: Chris Boyd Cc: nanog@nanog.org Subject: Re: Open Resolver Problems On Mon, 1 Apr 2013, Chris Boyd wrote:
Just back to the office, and started checking my networks. Found one of the resolvers is a Netgear SOHO NAT box. EoL'd, no new firmware available. Anyone have any feeling for what percentage are these types of boxes?
If you buy "type of box" mean "small SOHO NAT router which does DNS resolving on the WAN interface" then I'd say "a lot". Someone does a rollout of new software and configuration and happens to mess up the config file (or the vendor just happens to enable global dns resolving in the new software) and this slips through testing, then you're there. I believe this happens all the time. That's why the publication of these lists are important, in a lot of cases there are a lot of people who are simply not aware of these devices doing this, and they need to be poked to notice. -- Mikael Abrahamsson email: swmike@swm.pp.se
On Apr 01, 2013, at 11:55 , "Milt Aitken" <milt@net2atlanta.com> wrote:
Most of our DSL customers have modem/routers that resolve DNS externally. And most of those have no configuration option to stop it. So, we took the unfortunate step of ACL blocking DNS requests to & from the DSL network unless the requests are to our DNS servers.
Suboptimal, but it stopped the DNS amplification attacks.
I was going to suggest exactly this. Don't most broadband networks have a line in their AUP about running servers? Wouldn't a DNS server count as 'a server'? Then wouldn't running one violate the AUP? This gives the provider a hammer to hit the user over the head. Although that is quite unlikely, so the better point is that it also gives the provider cover in case some user complains about the provider filtering. You can always make an exception if the user is extremely loud. -- TTFN, patrick
-----Original Message----- From: Mikael Abrahamsson [mailto:swmike@swm.pp.se] Sent: Monday, April 01, 2013 11:51 AM To: Chris Boyd Cc: nanog@nanog.org Subject: Re: Open Resolver Problems
On Mon, 1 Apr 2013, Chris Boyd wrote:
Just back to the office, and started checking my networks. Found one of the resolvers is a Netgear SOHO NAT box. EoL'd, no new firmware available. Anyone have any feeling for what percentage are these types of boxes?
If you buy "type of box" mean "small SOHO NAT router which does DNS resolving on the WAN interface" then I'd say "a lot". Someone does a rollout of new software and configuration and happens to mess up the config file (or the vendor just happens to enable global dns resolving in the new software) and this slips through testing, then you're there. I believe this happens all the time.
That's why the publication of these lists are important, in a lot of cases there are a lot of people who are simply not aware of these devices doing this, and they need to be poked to notice.
-- Mikael Abrahamsson email: swmike@swm.pp.se
On Apr 1, 2013, at 11:03 PM, Patrick W. Gilmore wrote:
You can always make an exception if the user is extremely loud.
It might be a good idea to make pinholes for the Google and OpenDNS recursors, as they're fairly popular. I agree that this is a good idea, similar to the same sort of network access policy as relates to SMTP. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Apr 01, 2013, at 12:09 , "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Apr 1, 2013, at 11:03 PM, Patrick W. Gilmore wrote:
You can always make an exception if the user is extremely loud.
It might be a good idea to make pinholes for the Google and OpenDNS recursors, as they're fairly popular.
I agree that this is a good idea, similar to the same sort of network access policy as relates to SMTP.
Ahhh, silly of me, I read the post form Milt too quickly. I was going to suggest queries _into_ the broadband user space, not out of. If you only block into, OpenDNS, GoogleDNS, etc. are not an issue. Blocking could be done with DPI. It can also be done by blocking UDP port 53. (Don't need to block TCP53 since that removes the amplification problem.) However, there are some (idiotic) name servers that do 53<>53. Not sure how to handle those, or more importantly, how many broadband customers legitimately use an off-net _and_ brain-dead name server? And even if they do, will they fall back to TCP? Of course, since users shouldn't be using off-net name servers anyway, this isn't really a problem! :) -- TTFN, patrick
On Apr 1, 2013, at 11:18 PM, Patrick W. Gilmore wrote:
Of course, since users shouldn't be using off-net name servers anyway, this isn't really a problem! :)
;> It's easy enough to construct ACLs to restrict the broadband consumer access networks from doing so. Additional egress filtering would catch any reflected attacks, per your previous comments. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
* patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Apr 2013, 18:18 CEST]:
Of course, since users shouldn't be using off-net name servers anyway, this isn't really a problem! :)
You're joking, right? Should they also use only the telco-approved search engine, via the telco-hosted portal? -- Niels.
And only the telco approved web sites are accessible, and the only protocol supported is the telco approved http and then only to port 80 ... --- () ascii ribbon campaign against html e-mail /\ www.asciiribbon.org
-----Original Message----- From: Niels Bakker [mailto:niels=nanog@bakker.net] Sent: Monday, 01 April, 2013 14:22 To: nanog@nanog.org Subject: Re: Open Resolver Problems
* patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Apr 2013, 18:18 CEST]:
Of course, since users shouldn't be using off-net name servers anyway, this isn't really a problem! :)
You're joking, right? Should they also use only the telco-approved search engine, via the telco-hosted portal?
-- Niels.
Subject: Re: Open Resolver Problems Date: Mon, Apr 01, 2013 at 10:21:42PM +0200 Quoting Niels Bakker (niels=nanog@bakker.net):
* patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Apr 2013, 18:18 CEST]:
Of course, since users shouldn't be using off-net name servers anyway, this isn't really a problem! :)
You're joking, right? Should they also use only the telco-approved search engine, via the telco-hosted portal?
Far too many (perhaps not Patrick) in this thread are not joking. Laughter gets stuck in my throat, as we say in Sweden. Having proper Internet access is more and more a privilege for the Internet gentry that are clued and able to pay for a box in a colo or similar. The unwashed masses are left with "broadband" We can't call it "Internet" because there are a few raving graybeards that claim they invented it and intended it to be two-way instead of stuffing .flv down peoples facebook-viewing devices while also supplanting cable TV with demand streaming. </rant> What percentage of the SOHO NAT boxes actually are full-service resolvers? I was under the impression that most were mere forwarders; just pushing queries on toward the DHCP'd full service resolvers of the ISP. -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 Everywhere I look I see NEGATIVITY and ASPHALT ...
On Tue, 2 Apr 2013, Måns Nilsson wrote:
What percentage of the SOHO NAT boxes actually are full-service resolvers? I was under the impression that most were mere forwarders; just pushing queries on toward the DHCP'd full service resolvers of the ISP.
What does that help? They can still be amplifiers, it's just that now the ISP resolver will see the resolving load as well. -- Mikael Abrahamsson email: swmike@swm.pp.se
Subject: Re: Open Resolver Problems Date: Tue, Apr 02, 2013 at 05:25:53AM +0200 Quoting Mikael Abrahamsson (swmike@swm.pp.se):
On Tue, 2 Apr 2013, Måns Nilsson wrote:
What percentage of the SOHO NAT boxes actually are full-service resolvers? I was under the impression that most were mere forwarders; just pushing queries on toward the DHCP'd full service resolvers of the ISP.
What does that help? They can still be amplifiers, it's just that now the ISP resolver will see the resolving load as well.
But, yes, of course. Nobody would be so stupid so ast o accept queries on the WAN side and answer them? Would they? </innocent> -- Måns Nilsson primary/secondary/besserwisser/machina MN-1334-RIPE +46 705 989668 My vaseline is RUNNING...
On Apr 01, 2013, at 11:55 , "Milt Aitken" <milt@net2atlanta.com> wrote:
Most of our DSL customers have modem/routers that resolve DNS externally. And most of those have no configuration option to stop it. So, we took the unfortunate step of ACL blocking DNS requests to & from the DSL network unless the requests are to our DNS servers.
Suboptimal, but it stopped the DNS amplification attacks.
Wow. Glad I'm not a customer of yours. * patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Apr 2013, 18:04 CEST]:
I was going to suggest exactly this.
Don't most broadband networks have a line in their AUP about running servers?
Huh? No. Thankfully. Not all of us are mindless consumers. -- Niels.
On Apr 1, 2013, at 4:19 PM, Niels Bakker <niels=nanog@bakker.net> wrote:
On Apr 01, 2013, at 11:55 , "Milt Aitken" <milt@net2atlanta.com> wrote:
Most of our DSL customers have modem/routers that resolve DNS externally. And most of those have no configuration option to stop it. So, we took the unfortunate step of ACL blocking DNS requests to & from the DSL network unless the requests are to our DNS servers.
Suboptimal, but it stopped the DNS amplification attacks.
Wow. Glad I'm not a customer of yours.
I would say this is the wrong solution. Prevent your customers from spoofing is the first step, then ask them to fix their broken CPE. If NETGEAR is listening on the WAN side vs the LAN/INSIDE they need to step up and issue fixed firmware, even if the device is older. Should be a simple fix.
* patrick@ianai.net (Patrick W. Gilmore) [Mon 01 Apr 2013, 18:04 CEST]:
I was going to suggest exactly this.
Don't most broadband networks have a line in their AUP about running servers?
Huh? No. Thankfully. Not all of us are mindless consumers.
I think it's easier to just classify an open-resolver similar to an open-relay without having to invoke the consumer mindset. - Jared
* jared@puck.nether.net (Jared Mauch) [Mon 01 Apr 2013, 22:24 CEST]:
I would say this is the wrong solution. Prevent your customers from spoofing is the first step, then ask them to fix their broken CPE.
I daresay that after ten years of discussion NANOG has reached consensus that implementing BCP38 is a good thing and that all networks should be encouraged to do so. Net neutrality has not been discussed completely to death yet but I'm pretty confident in stating that squeezing consumer connections further down each time some blog hypes up yet another "The Internet is melting!" threat won't scale.
If NETGEAR is listening on the WAN side vs the LAN/INSIDE they need to step up and issue fixed firmware, even if the device is older. Should be a simple fix.
I don't think anybody would disagree with this statement. Netgear did get into action when they DDoS'ed a university's NTP servers; perhaps similar sticks can be shaken in this case. (Is Netgear one of the brands involved? Usually they're better. Pardon me for not reading the whole thread and the other five)
I think it's easier to just classify an open-resolver similar to an open-relay without having to invoke the consumer mindset.
Two posts up in this thread we were talking about net-wide blocks without individual proof of open relay or equivalent status. -- Niels.
In message <44ECD7B5-D9A4-408B-A132-29241DE3A867@ianai.net>, "Patrick W. Gilmore" writes:
On Apr 01, 2013, at 11:55 , "Milt Aitken" <milt@net2atlanta.com> wrote:
Most of our DSL customers have modem/routers that resolve DNS externally. And most of those have no configuration option to stop it. So, we took the unfortunate step of ACL blocking DNS requests to & from the DSL network unless the requests are to our DNS servers.
Suboptimal, but it stopped the DNS amplification attacks.
I was going to suggest exactly this.
Don't most broadband networks have a line in their AUP about running servers? Wouldn't a DNS server count as 'a server'? Then wouldn't running one violate the AUP?
This gives the provider a hammer to hit the user over the head. Although that is quite unlikely, so the better point is that it also gives the provider cover in case some user complains about the provider filtering.
You can always make an exception if the user is extremely loud.
-- TTFN, patrick
Actually a lot don't have such a line. Such lines are tantamount to extortion especially if the ISP supplies commercial grade lines. That said blocking by default with the option to open it up on request, the same as smtp is opened on request, might be viable. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Apr 2, 2013, at 7:53 AM, Mark Andrews wrote:
Such lines are tantamount to extortion especially if the ISP supplies commercial grade lines.
Patrick's talking about consumer broadband access. Such AUP stipulations are quite common. This is in no way 'tantamount to extortion'. Folks can either accept the AUP, or choose not to enter into a contract for the service in question under those conditions; there is no compulsion or coercion to do so. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
In message <CF4E9F59-4A9E-4E03-8EB4-469C3DB15FF4@arbor.net>, "Dobbins, Roland" writes:
On Apr 2, 2013, at 7:53 AM, Mark Andrews wrote:
Such lines are tantamount to extortion especially if the ISP supplies commercial grade lines.
Patrick's talking about consumer broadband access. Such AUP stipulations are quite common.
I know and I would still argue that they are tantamount to extortion.
This is in no way 'tantamount to extortion'. Folks can either accept the AUP, or choose not to enter into a contract for the service in question under those conditions; there is no compulsion or coercion to do so.
So the home user that want to run a server now has to pay for COLO or pay the ISP for it commercial line that is delivered over the same physical circuit for extra dollars which gets what? Maybe a upgraded SLA and maybe some static addresses.
----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com>
Luck is the residue of opportunity and design.
-- John Milton -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Apr 2, 2013, at 8:21 AM, Mark Andrews wrote:
I know and I would still argue that they are tantamount to extortion.
There is no coercion involved, so, by definition, it can't be called 'extortion'. If you don't like the AUP, don't sign up for the service - simple as that. Hyperbole isn't generally helpful. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Apr 1, 2013, at 6:38 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Apr 2, 2013, at 8:21 AM, Mark Andrews wrote:
I know and I would still argue that they are tantamount to extortion.
There is no coercion involved, so, by definition, it can't be called 'extortion'. If you don't like the AUP, don't sign up for the service - simple as that.
Hyperbole isn't generally helpful.
In an oligopoly situation, that's hardly a valid set of choices and is tantamount to extortion. Owen
On Mon, Apr 1, 2013 at 6:45 PM, Owen DeLong <owen@delong.com> wrote:
On Apr 1, 2013, at 6:38 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Apr 2, 2013, at 8:21 AM, Mark Andrews wrote:
I know and I would still argue that they are tantamount to extortion.
There is no coercion involved, so, by definition, it can't be called 'extortion'. If you don't like the AUP, don't sign up for the service - simple as that.
Hyperbole isn't generally helpful.
In an oligopoly situation, that's hardly a valid set of choices and is tantamount to extortion.
Yeah, I thought so, too, but apparently the FCC and the SEC hasn't seen it that way for the past 20 years. Go figure. :-) - ferg -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
On Apr 2, 2013, at 8:52 AM, Paul Ferguson wrote:
Yeah, I thought so, too, but apparently the FCC and the SEC hasn't seen it that way for the past 20 years. Go figure. :-)
The situation is gradually getting better, not worse - and that's progress, even if it isn't as fast as we'd all like. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
Yeah, I thought so, too, but apparently the FCC and the SEC hasn't seen it that way for the past 20 years. Go figure. :-)
The FCC doesn't understand that 4Mbps customer-facing speed on the tail circuit alone does NOT define broadband in a meaningful way. The SEC does not understand that IPv4 risk and the lack of an IPv6 strategy should be a required risk consideration in a Sarbanes Oxley filing. I have little hope that these particular federal agencies will ever agree with me about such nuanced issues. Owen
On Apr 2, 2013, at 8:45 AM, Owen DeLong wrote:
In an oligopoly situation, that's hardly a valid set of choices
There's enough choice in most US markets (not all) to provide for a variety of services offered, AUPs, and price points. Wireless has brought an additional option to many previously underserved areas.
and is tantamount to extortion.
Again, hyperbole doesn't help. Another solution is to move to an area with more/better connectivity options, as some folks move in order to be zoned within a particular school district. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Apr 1, 2013, at 6:54 PM, "Dobbins, Roland" <rdobbins@arbor.net> wrote:
On Apr 2, 2013, at 8:45 AM, Owen DeLong wrote:
In an oligopoly situation, that's hardly a valid set of choices
There's enough choice in most US markets (not all) to provide for a variety of services offered, AUPs, and price points. Wireless has brought an additional option to many previously underserved areas.
With all due respect, sir, you are mistaken. Even in such populous areas as San Jose, there is a limited selection to a majority of the customers, especially if they want more than 1.5Mbps. In the majority of the US where it is rural, there is even less choice. Even where there are multiple providers, they often all provide the same limitations in their AUP unless you go to higher priced services.
and is tantamount to extortion.
Again, hyperbole doesn't help.
If all of the choices to eliminate unreasonable restrictions on how you use the bandwidth you pay for involve paying more money for roughly the same service, then that is not hyperbole. Such is the case for a very large fraction of subscribers in the US.
Another solution is to move to an area with more/better connectivity options, as some folks move in order to be zoned within a particular school district.
It is an option when you live in a neighborhood with a protection racket operating to move out of the neighborhood as well. This does not change the fact that a protection racket is extortion. Owen
On Apr 2, 2013, at 9:09 AM, Owen DeLong wrote:
With all due respect, sir, you are mistaken.
Even in such populous areas as San Jose, there is a limited selection to a majority of the customers, especially if they want more than 1.5Mbps.
I lived in San Jose for several years, and had several choices for broadband - the one I chose was much faster than 1mb/sec, had an AUP which specifically allowed me to run a server, and didn't try to cap my bandwidth, or disable the use of p2p apps, or whatever. I moved away from San Jose in at the tail-end of 2007. It seems likely that at least the same level of choice prevails there today . . . ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Mon, Apr 1, 2013 at 7:38 PM, Dobbins, Roland <rdobbins@arbor.net> wrote:
On Apr 2, 2013, at 9:09 AM, Owen DeLong wrote:
Even in such populous areas as San Jose, there is a limited selection to a majority of the customers, especially if they want more than 1.5Mbps.
I lived in San Jose for several years, and had several choices for broadband - the one I chose was much faster than 1mb/sec, had an AUP which specifically allowed me to run a server, and didn't try to cap my bandwidth, or disable the use of p2p apps, or whatever.
I moved away from San Jose in at the tail-end of 2007. It seems likely that at least the same level of choice prevails there today . . .
So, I lived in San Jose, too, for many years, and I had fewer choices there than I do here now in the Pacific Northwest. In any event, depending on where you are in the U.S., many consumers have a choice between bad and worse. :-) - ferg -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
On Apr 2, 2013, at 9:48 AM, Paul Ferguson wrote:
In any event, depending on where you are in the U.S., many consumers have a choice between bad and worse. :-)
I certainly do agree with that general sentiment. Living abroad, I have more choices in terms of both wired broadband and wireless. And 'unlimited' really means unlimited. If I ever moved back to the US, one of the things I would miss the most is complete freedom in terms of wireless network choice, service level, and traffic tiering. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Apr 2, 2013, at 9:09 AM, Owen DeLong wrote:
In the majority of the US where it is rural, there is even less choice.
Largest in geography <> largest in population.
Even where there are multiple providers, they often all provide the same limitations in their AUP unless you go to higher priced services.
If you don't like the pricing, that's quite different from claiming extortion. Look, I'm no fan of semi-monopolies, 'unlimited' capacity which isn't, and so forth. But there *are* choices in most US broadband markets; maybe not the choices which we'd find ideal, maybe at a price-point higher than we think is fair, but the point is that there are choices, and nobody is forcing anyone to spend money for services he doesn't wish to purchase. I'd like to see UK-style structural separation in the US, as that would greatly increase opportunities to compete. I doubt it will ever happen, though. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
From: Dobbins, Roland [mailto:rdobbins@arbor.net] On Apr 2, 2013, at 7:53 AM, Mark Andrews wrote:
Such lines are tantamount to extortion especially if the ISP supplies commercial grade lines.
Patrick's talking about consumer broadband access. Such AUP stipulations are quite common.
This is in no way 'tantamount to extortion'. Folks can either accept the AUP, or choose not to enter into a contract for the service in question under those conditions; there is no compulsion or coercion to do so.
And that would be a valid response if we actually lived in a place where I, or anyone else, had more than two choices, both offering roughly the same terms and pricing. In my little corner of Fairfax Co, we have Cox or FiOS. Across the Potomac in Montgomery, they can pick between Comcast and FiOS. I hear that in other bits of the US, your cable and telco might be different, but other than the label, nothing else is. Jamie
participants (12)
-
Dobbins, Roland -
Jamie Bowden -
Jared Mauch -
Keith Medcalf -
Mark Andrews -
Mikael Abrahamsson -
Milt Aitken -
Måns Nilsson -
Niels Bakker -
Owen DeLong -
Patrick W. Gilmore -
Paul Ferguson