Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplificat... Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help. For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there... Regards, CB
On Feb 13, 2014, at 12:06 PM, Cb B <cb.list6@gmail.com> wrote:
Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others
http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplificat...
Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help.
For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html
In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there...
While I'm behind some of the inventory projects (so you can go ahead and fix.. let me know if you need/want the URLs to see data for your networks)... I must provide credit to those behind the "Amplification Hell" talk at NDSS. If you are at all interested in what is going on, you should attend or review the content. http://www.internetsociety.org/ndss2014/programme BCP-38 on your customers is going to be critical to prevent the abuse reaching your network. Please ask your vendors for it, and ask for your providers to filter your network to prevent you originating this abuse. If you operate hosted VMs, servers, etc.. please make sure those netblocks are secured as well. You can easily check your network (As can the bad guys!) here: http://spoofer.cmand.org/ - Jared
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/13/2014 9:06 AM, Cb B wrote:
Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others
http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplificat...
Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help.
For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html
In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there...
The alternative is get people to understand that anti-spoofing is good, and efforts to combat spoofing should be encouraged. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL9AR4ACgkQKJasdVTchbJZYwEAivI00Yq7RSMze74GFQKEyCeH pS2s8TH0ba08NWKC22AA/jyN35xonJBzldJA8/xlzhnuLnyOFB0Y7GKZ8NiqRiRl =ItxR -----END PGP SIGNATURE-----
On 02/13/2014 10:06 AM, Cb B wrote:
Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others
http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplificat...
Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help.
For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html
In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there...
UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices. Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol. --John
Regards,
CB
On Feb 13, 2014, at 1:47 PM, John <jschiel@flowtools.net> wrote:
On 02/13/2014 10:06 AM, Cb B wrote:
Good write up, includes name and shame for AT&T Wireless, IIJ, OVH, DTAG and others
http://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplificat...
Standard plug for http://openntpproject.org/ and http://openresolverproject.org/ and bcp38 , please fix/help.
For those of you paying attention to the outage list, this is a pretty big deal that has had daily ramification for some very big networks https://puck.nether.net/pipermail/outages/2014-February/date.html
In general, i think UDP is doomed to be blocked and rate limited -- tragedy of the commons. But, it would be nice if folks would just fix the root of the issue so the rest of us don't have go there...
UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices.
Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol.
Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment. I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks. I've seen maybe 100-200 per-ASN reports handed out to network operators. If you want yours, please e-mail ntp-scan@puck.nether.net to obtain it. Put your ASN in the subject line and/or body. - Jared (and others like Patrick that presented on the projects behalf).
On Friday, February 14, 2014 03:01:27 AM Jared Mauch wrote:
I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks.
Depending on your OS, the fixes can be quite simple or interesting. On my FreeBSD servers, simply updating with "freebsd-update" was enough to fix the issue (in addition to limiting who/what can access the service). On Cisco devices, the ACL's you can attach to the NTP process are quite effective. On Juniper devices, it is less intuitive, and even though NTP is enabled only as a client, it, sadly, runs the server as well. A firewall filter helps here when applied correctly. Can't speak to other OS's. Mark.
On Thu, Feb 13, 2014 at 08:01:27PM -0500, Jared Mauch wrote:
I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks.
A slight exception to that statement, if I may... The right thing to do is for people to not permit services to operate on hosts they do not intend to operate on and not to be visible to those they do not intend to use them. In other words, to properly manage their networks. If that means blocking all access to potentially faulty implementations, then that's the right thing to do. In short, companies should do what is right for their companies and nevermind anyone else. Never forget that researches are just part of the "public" and should never consider that their usage of the internet is any more or less valid to the average third party than the next guy. -Wayne --- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/14/2014 10:22 AM, Wayne E Bouchard wrote:
On Thu, Feb 13, 2014 at 08:01:27PM -0500, Jared Mauch wrote:
I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks.
A slight exception to that statement, if I may...
The right thing to do is for people to not permit services to operate on hosts they do not intend to operate on and not to be visible to those they do not intend to use them. In other words, to properly manage their networks. If that means blocking all access to potentially faulty implementations, then that's the right thing to do. In short, companies should do what is right for their companies and nevermind anyone else.
Never forget that researches are just part of the "public" and should never consider that their usage of the internet is any more or less valid to the average third party than the next guy.
Taken to the logical extreme, the "right thing" to do is to deny any spoofed traffic from abusing these services altogether. NTP is not the only one; there is also SNMP, DNS, etc. - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL+Y68ACgkQKJasdVTchbJ/dgEAqgERvP6HMl2v5fbhZDwI9QKT YEe/c3mN5gZlxsIKFo0A/3BH9KMV6ln7XMrlnk4c/GuwZ9X4LAgqO6l2p8u3aA49 =yWZU -----END PGP SIGNATURE-----
On Fri, Feb 14, 2014 at 10:42:55AM -0800, Paul Ferguson wrote: [snip]
Taken to the logical extreme, the "right thing" to do is to deny any spoofed traffic from abusing these services altogether. NTP is not the only one; there is also SNMP, DNS, etc.
...and then we're back to "implement BCP38 already!" (like one of the authors of the document didn't think of that, ferg? ;-) NB: Some Entities believe all filtering is 'bcp 38' and thus have given this stone-dead logical and sane practice a bad rap. If someone is sloppy with their IRR-based filters or can't drive loose RPF correctly, that isn't the fault of BCP38. The document specifically speaks to aggregation points, most clearly in the introduction: "In other words, if an ISP is aggregating routing announcements for multiple downstream networks, strict traffic filtering should be used to prohibit traffic which claims to have originated from outside of these aggregated announcements." This goes for access, hosting, and most recently virtual hosting in teh cloude. Stop forgery at your edges and your life will be easier. Cheers, Joe -- RSUC / GweepNet / Spunk / FnB / CotSG / Usenix / NANOG
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2/14/2014 4:09 PM, Joe Provo wrote:
On Fri, Feb 14, 2014 at 10:42:55AM -0800, Paul Ferguson wrote: [snip]
Taken to the logical extreme, the "right thing" to do is to deny any spoofed traffic from abusing these services altogether. NTP is not the only one; there is also SNMP, DNS, etc.
...and then we're back to "implement BCP38 already!" (like one of the authors of the document didn't think of that, ferg? ;-)
NB: Some Entities believe all filtering is 'bcp 38' and thus have given this stone-dead logical and sane practice a bad rap. If someone is sloppy with their IRR-based filters or can't drive loose RPF correctly, that isn't the fault of BCP38.
The document specifically speaks to aggregation points, most clearly in the introduction: "In other words, if an ISP is aggregating routing announcements for multiple downstream networks, strict traffic filtering should be used to prohibit traffic which claims to have originated from outside of these aggregated announcements."
This goes for access, hosting, and most recently virtual hosting in teh cloude. Stop forgery at your edges and your life will be easier.
Indeed -- I'm not in the business of bit-shipping these days, so I can't endorse or advocate any particular method of blocking spoofed IP packets in your gear. I can, however, say with confidence that it is still a good idea. Great idea, even. :-) - - ferg - -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlL+y8sACgkQKJasdVTchbKTXAEA0/czP0ECsFX4CyUr6yt4Dkap D0NZT/UIo6h5E/dl0KEA/3hpxN2NLxZRix6JUTVHyv+LZ4RzgpG2myoXbgAq1+WS =QQjA -----END PGP SIGNATURE-----
On 2/14/2014 9:07 PM, Paul Ferguson wrote:
Indeed -- I'm not in the business of bit-shipping these days, so I can't endorse or advocate any particular method of blocking spoofed IP packets in your gear.
If you're dead-end, a basic ACL that permits ONLY your prefixes on egress, and blocks your prefixes on ingress, is perhaps the safest bet. Strict uRPF has it's complications, and loose uRPF is almost too forgiving. If you're providing transit, it gets much more complicated much more quickly, but the same principles apply (they just get to be a less-than-100% solution) :)
I can, however, say with confidence that it is still a good idea. Great idea, even. :-)
Oh yeah :) Jeff
On 02/13/2014 06:01 PM, Jared Mauch wrote:
On Feb 13, 2014, at 1:47 PM, John <jschiel@flowtools.net> wrote: <snip> UDP won't be blocked. There are some vendors that have their own hidden protocol inside UDP packets to control and communicate with their devices.
Thinking on it again, maybe blocking UDP isn't all that bad. Would force the vendors to not 'hide' their protocol.
Be careful what you wish for. I know some people have just blocked all NTP to keep their servers from participating in attacks. This is common in places where they hand off a VM/host to a customer and no longer have access despite it being in their environment. I was being a bit extreme, I don't expect UDP to be blocked and there are valid uses for NTP and it needs to pass. Can you imagine the trading servers not having access to NTP?
The knee jerk reaction to just block NTP is a temporary measure that can be used while other mitigation steps are implemented. I kinda hijacked the NTP issue a bit and expanded it to cover the undocumented uses of device control in UDP. I'll leave that issue for another day, just wanted to raise awareness if it was not already known. --John
I would actually like to ask for those folks to un-block NTP so there is proper data on the number of hosts for those researching this. The right thing to do is reconfigure them. I've seen a good trend line in NTP servers being fixed, and hope we will see more of that in the next few weeks.
I've seen maybe 100-200 per-ASN reports handed out to network operators. If you want yours, please e-mail ntp-scan@puck.nether.net to obtain it. Put your ASN in the subject line and/or body.
- Jared (and others like Patrick that presented on the projects behalf).
participants (8)
-
Cb B
-
Jared Mauch
-
Jeff Kell
-
Joe Provo
-
John
-
Mark Tinka
-
Paul Ferguson
-
Wayne E Bouchard