nmap -sU -pU:123 -Pn -n --script=ntp-monlist serverIP On Mon, Jan 13, 2014 at 3:07 PM, Jared Mauch <jared@puck.nether.net> wrote:
Greetings,
With the recent increase in NTP attacks, I wanted to advise the community of a few things:
There are about 1.2-1.5 million of these servers out there.
1) You can search your IP space to find NTP servers that respond to the ‘MONLIST’ queries.
2) I’ve found some vendors have old embedded versions of NTP including ILO/Service Processors and other parts of the “internet of things”.
3) You want to upgrade NTP, or adjust your ntp.conf to include ‘limited’ or ‘restrict’ lines or both. (I defer to someone else to be an expert in this area, but am willing to learn :) )
4) Please prevent packet spoofing where possible on your network. This will limit the impact of spoofed NTP or DNS (amongst others) packets from impacting the broader community.
5) Some vendors don’t have an easy way to alter the ntp configuration, or have not or won’t be updating NTP, you may need to use ACLs, firewall filters, or other methods to block this traffic. I’ve heard of many routers being used in attacks impacting the CPU usage.
Take a moment and see if your devices respond to the following query/queries:
ntpdc -n -c monlist 10.0.0.1 ntpdc -n -c loopinfo 10.0.0.1 ntpdc -n -c iostats 10.0.0.1
6) If you do VMs/Servers and have a template, please make sure that they do not respond to NTP requests.
Thanks!
- Jared
-- Copyright 2014 Derek Andrew (excluding quotations) +1 306 966 4808 Information and Communications Technology University of Saskatchewan Peterson 120; 54 Innovation Boulevard Saskatoon,Saskatchewan,Canada. S7N 2V3 Timezone GMT-6 Typed but not read.
On 13 Jan 2014, at 21:13 , Derek Andrew <Derek.Andrew@usask.ca> wrote:
nmap -sU -pU:123 -Pn -n --script=ntp-monlist serverIP
Make that “all server IPs” if on different subnets, address families, ...
On Mon, Jan 13, 2014 at 3:07 PM, Jared Mauch <jared@puck.nether.net> wrote:
4) Please prevent packet spoofing where possible on your network. This will limit the impact of spoofed NTP or DNS (amongst others) packets from impacting the broader community.
BCP38! I am always surprised when people need crypto if they fail the simple things.
5) Some vendors don’t have an easy way to alter the ntp configuration, or have not or won’t be updating NTP, you may need to use ACLs, firewall filters, or other methods to block this traffic. I’ve heard of many routers being used in attacks impacting the CPU usage.
Take a moment and see if your devices respond to the following query/queries:
ntpdc -n -c monlist 10.0.0.1 ntpdc -n -c loopinfo 10.0.0.1 ntpdc -n -c iostats 10.0.0.1
And no matter if you use the above nmap or these instructions to check, also check your IPv6 addresses! You need 'restrict -6 default ignore' lines or similar as well, not just a restrict default ignore. — Bjoern A. Zeeb ????????? ??? ??????? ??????: '??? ??? ???? ?????? ??????? ?? ?? ??????? ??????? ??? ????? ????? ???? ?????? ?? ????? ????', ????????? ?????????, "??? ????? ?? ?????", ?.???
On (2014-01-13 21:33 +0000), Bjoern A. Zeeb wrote:
BCP38! I am always surprised when people need crypto if they fail the simple things.
Saying that BCP38 is solution to the reflection attacks is not unlike 5 year old wishing nothing but world peace for christmas, endearing, but it's not going to change anything. BCP38 is completely unrealistic, many access networks are on autopilot, many don't have HW support for BCP38, one port configured has low-benefit, only that machine can stop attacking (but whole world). near term, reducing attack surface is practical to reduce impact (not a solution, just damage control) near term, transit providers who do BGP prefix-list, could use same prefix-list for ACL, segmenting spoofing domains. It's very high pay-off, couple ports configured, whole downstream branch isolated into its own spoofing domain, able to just attack targets inside same domain. mid term, transport area in IETF. DNS, NTP, SNMP, chargen et.al. could trivially change to QUIC/MinimaLT or compared, getting same 0 RTT penalty as UDP without reflection potential. -- ++ytti
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 1/13/2014 11:18 PM, Saku Ytti wrote:
On (2014-01-13 21:33 +0000), Bjoern A. Zeeb wrote:
BCP38! I am always surprised when people need crypto if they fail the simple things. Saying that BCP38 is solution to the reflection attacks is not unlike 5 year old wishing nothing but world peace for christmas, endearing, but it's not going to change anything. BCP38 is completely unrealistic, many access networks are on autopilot, many don't have HW support for BCP38, one port configured has low-benefit, only that machine can stop attacking (but whole world).
That does *not* make it an unworthy goal, nor should it stop people from encouraging it's implementation. - - ferg (co-author of BCP38) - -- Paul Ferguson PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iF4EAREIAAYFAlLVWXoACgkQKJasdVTchbIrtAD/T2bNNAgZOnnlniBPd6sEquxJ v01mmrhJxFTIDFq7EIkA/3vQbiwtEwBeVyCtc3coEkz50zSDh3j9QQjT+TQWCNVs =Al3Y -----END PGP SIGNATURE-----
BCP38 will only ever get implemented if governments and ruling 'net bodies force deployment. There's otherwise very little benefit seen by the access network providers, since the targets are other orgs and the attacks are happening in a different backyard. On 14/01/2014 10:36 AM, Paul Ferguson wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 1/13/2014 11:18 PM, Saku Ytti wrote:
On (2014-01-13 21:33 +0000), Bjoern A. Zeeb wrote:
BCP38! I am always surprised when people need crypto if they fail the simple things. Saying that BCP38 is solution to the reflection attacks is not unlike 5 year old wishing nothing but world peace for christmas, endearing, but it's not going to change anything. BCP38 is completely unrealistic, many access networks are on autopilot, many don't have HW support for BCP38, one port configured has low-benefit, only that machine can stop attacking (but whole world). That does *not* make it an unworthy goal, nor should it stop people from encouraging it's implementation.
- - ferg (co-author of BCP38)
- -- Paul Ferguson PGP Public Key ID: 0x54DC85B2
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iF4EAREIAAYFAlLVWXoACgkQKJasdVTchbIrtAD/T2bNNAgZOnnlniBPd6sEquxJ v01mmrhJxFTIDFq7EIkA/3vQbiwtEwBeVyCtc3coEkz50zSDh3j9QQjT+TQWCNVs =Al3Y -----END PGP SIGNATURE-----
In message <52D7E98B.4040801@userid.org>, Pierre Lamy writes:
BCP38 will only ever get implemented if governments and ruling 'net bodies force deployment. There's otherwise very little benefit seen by the access network providers, since the targets are other orgs and the attacks are happening in a different backyard.
Except the targets are *everybody* including the access networks. This really is "if you are not part of the solution, you are part of the problem" and applies 100% to access networks. And it doesn't require governments, it just requires CEO's with the gumption to say we are not going to accept routes from you, via transit or direct, until you publically state that you are implementing BCP38 within your network and then follow through. If you implement BCP the publish the fact on you sites. This lets others know they are not alone in the fight to make the net a better place. e.g. "We implement BCP 38 on XX% of customer links" "We implement BCP 38 on all of our single homed customers" "We implement BCP 38 on all our data centers traffic" "We implement BCP 38 on all our NOC traffic" "We implement BCP 38 on all our outgoing traffic" "We implement BCP 38 on all our traffic" Machines get owned everywhere including data centers and NOCs. BCP 38 filters should be everywhere. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Mon, Jan 13, 2014 at 11:18 PM, Saku Ytti <saku@ytti.fi> wrote:
On (2014-01-13 21:33 +0000), Bjoern A. Zeeb wrote:
BCP38! I am always surprised when people need crypto if they fail the simple things.
Saying that BCP38 is solution to the reflection attacks is not unlike 5 year old wishing nothing but world peace for christmas, endearing, but it's not going to change anything. BCP38 is completely unrealistic, many access networks are on autopilot, many don't have HW support for BCP38, one port configured has low-benefit, only that machine can stop attacking (but whole world).
near term, reducing attack surface is practical to reduce impact (not a solution, just damage control)
BCP38 (even if not fully deployed) is the only viable form of reducing the attack surface. Other ideas can never reach enough adoption to have any impact (they need to be ~100% deployed before any improvement is seen). As an example, let's imagine you successfully close 99% of the open DNS recursive resolvers, dropping the number of available reflectors from 28M down to 280k. Has that achieved anything? No, the attacks will be just as large. Or even if you do get to 100%, you haven't done anything about the authoritative servers. Or the other protocols, like NTP, Chargen, etc. near term, transit providers who do BGP prefix-list, could use same
prefix-list for ACL, segmenting spoofing domains. It's very high pay-off, couple ports configured, whole downstream branch isolated into its own spoofing domain, able to just attack targets inside same domain.
I see this as a form of BCP38, but imposed on networks by their transit providers, rather than done voluntarily. It would be great if it could work, but I have doubts due to asymmetric routing announcements intended for traffic shaping. mid term, transport area in IETF. DNS, NTP, SNMP, chargen et.al. could
trivially change to QUIC/MinimaLT or compared, getting same 0 RTT penalty as UDP without reflection potential.
I'd expect that to take 20 years or more. Even if new standards are defined, the old servers will only be removed when they physically fail. My crazy proposal: get international agreement that sending spoofed packets is illegal, then trace their sources. Tracing the sources just requires transit providers (or other large networks) to collect and analyze netflow, but that may end up being as infeasible as changing the global legal system. ;) Damian
On (2014-01-14 08:35 -0800), Damian Menscher wrote:
I see this as a form of BCP38, but imposed on networks by their transit providers, rather than done voluntarily. It would be great if it could work, but I have doubts due to asymmetric routing announcements intended for traffic shaping.
Yes, I should have specified 'BCP38 in access networks' as being completely unrealistic. (We do BCP38 on all ports and verify programmatically, but I know it's not at all practical solution globally for access). ACL in transit port is completely harmless, no announcements are needed for traffic to be accepted. There are very modest amount of transit ports globally and each port will create segmentation to the spoofing domains having immediate, significant effect on benefits of spoofed attacks. RPF obviously is non-starter for reasons you stated.
I'd expect that to take 20 years or more. Even if new standards are defined, the old servers will only be removed when they physically fail.
It would have to be carried over UDP initially and that support probably would have to live for 20 years. But new-l4-over-udp version could be deployable rapidly. I'm very optimistic that if we'd have useful L4 for DNS, significant portion of relevant DNS servers could be upgraded rapidly to support it. We may be able to use existing data for this, how many servers went from DNS source port to random source port to add entropy to reduce poisoning attack chance? Good portion of end users are running w7, w8, osx updating itself automatically, so end-user support could come automatically and not require action from users. phones, tablets etc have short upgrade cycles anyhow. Native-udp port could then be policed heavily, making reflected attacks pay-off poor and motivates rest of the users to take actions needed for new l4.
My crazy proposal: get international agreement that sending spoofed packets
Agreed, crazy. -- ++ytti
On Jan 15, 2014, at 12:05 AM, Saku Ytti <saku@ytti.fi> wrote:
(We do BCP38 on all ports and verify programmatically, but I know it's not at all practical solution globally for access).
Anti-spoofing is eminently practical for most types of access network topologies using even slightly modern equipment; uRPF, ACLs, cable IP source verify, DHCP Snooping (which works just fine with fixed-address hosts), PACLs/VACLs, et. al. are some of the more prevalent mechanisms available. In point of fact, anti-spoofing is most useful and most practical at the access-network edge, or as close to it as possible. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On (2014-01-16 14:30 +0000), Dobbins, Roland wrote:
In point of fact, anti-spoofing is most useful and most practical at the access-network edge, or as close to it as possible.
We must disagree on definition of practical. Maybe if I'd reword it realistic we might be closer. It is not going to happen, the most suspect places are places where it's going to be most difficult to get, either fully on autopilot with no technical personnel capable or having the power to make the change or ghetto gear with no capability for it. The longer we endorse fantasy the longer it'll take to promote practical solutions. There is nothing near consensus that IP transit should or even can be ACLd, but it's really simple and I'm happy to volunteer my time with any network wishing to implement it. Very modest amount of ports will produce significant reduction in spoofing pay-off. -- ++ytti
On Jan 16, 2014, at 9:56 PM, Saku Ytti <saku@ytti.fi> wrote:
It is not going to happen, the most suspect places are places where it's going to be most difficult to get, either fully on autopilot with no technical personnel capable or having the power to make the change or ghetto gear with no capability for it.
That may well be the case, unfortunately; my point was that at or near the access edge is the most *technically* and *topologically* feasible place to implement antispoofing mechanisms, as you indicated in your reply. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Tue, Jan 14, 2014 at 09:18:30AM +0200, Saku Ytti wrote:
DNS, NTP, SNMP, chargen et.al. could trivially change to QUIC/MinimaLT or compared, getting same 0 RTT penalty as UDP without reflection potential.
I wouldn't say trivial, but QUIC and MinimaLT are hopefully the future. The near future, I hope! For now I'd just like to mention that OpenNTPD, from the OpenBSD project, is immune to the kind of large NTP amplification attacks now being discussed. It's certainly a good fit for some organizations/setups. http://www.openntpd.org Nicolai
On Tue, Jan 14, 2014 at 09:18:30AM +0200, Saku Ytti wrote:
mid term, transport area in IETF. DNS, NTP, SNMP, chargen et.al. could trivially change to QUIC/MinimaLT
Oh, yes, it'd obviously be trivial to change DNS to use a different transport. This is shown by the massive success of getting EDNS0 universally deployed in under 10 years. Right? A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On Thu, Jan 16, 2014 at 11:27 AM, Andrew Sullivan <asullivan@dyn.com> wrote:
On Tue, Jan 14, 2014 at 09:18:30AM +0200, Saku Ytti wrote:
mid term, transport area in IETF. DNS, NTP, SNMP, chargen et.al. could trivially change to QUIC/MinimaLT
Oh, yes, it'd obviously be trivial to change DNS to use a different transport. This is shown by the massive success of getting EDNS0 universally deployed in under 10 years. Right?
pretty easy to believe that quic would be helpful right? especially if you were interested in: 1) keeping resource utilization down/same on dns servers 2) rtt and latency impacts of extra rtt on upper layer protocols 3) the Xmillions of embedded devices that end users rely upon for dns in their homes seems totally feasible. -chris
On Thu, Jan 16, 2014 at 11:32:05AM -0500, Christopher Morrow wrote:
pretty easy to believe that quic would be helpful right?
Yes. It's also pretty easy to believe that ditching DNS completely in favour of something without 8 billion warts would be helpful.
seems totally feasible.
Certainly, it would be possible to standardize it. Whether it would be "trivial" to get it deployed is quite a different matter. The evidence to date is that there is a very, very long tail in any change having to do with the DNS. We are still, to this day, fighting with sysadmins who are convinced that firewall rules on TCP/53 are perfectly reasonable, even though DNS _always_ used TCP. People who believe there are going to be easy fixes to the issues coming from DNS are deluding themselves. A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On Thu, Jan 16, 2014 at 11:39 AM, Andrew Sullivan <asullivan@dyn.com> wrote:
On Thu, Jan 16, 2014 at 11:32:05AM -0500, Christopher Morrow wrote:
pretty easy to believe that quic would be helpful right?
Yes. It's also pretty easy to believe that ditching DNS completely in favour of something without 8 billion warts would be helpful.
seems totally feasible.
Certainly, it would be possible to standardize it. Whether it would be "trivial" to get it deployed is quite a different matter. The evidence to date is that there is a very, very long tail in any change having to do with the DNS. We are still, to this day, fighting with sysadmins who are convinced that firewall rules on TCP/53 are perfectly reasonable, even though DNS _always_ used TCP.
People who believe there are going to be easy fixes to the issues coming from DNS are deluding themselves.
I totally agree... I was actually joking in my last note :( sorry for not adding the ":)" as requisite in email. So... what other options are there to solve the larger problem of: "Some service is running on a public host, and it can be used to attack a third party" where in all of these cases the third party is someone who's address has been spoofed in the src-address of a packet. -chris
On Thu, Jan 16, 2014 at 11:48:56AM -0500, Christopher Morrow wrote:
I totally agree... I was actually joking in my last note :( sorry for not adding the ":)" as requisite in email.
I'm sorry my humour is now so impaired from reading 1net and other such things that I didn't figure it out!
So... what other options are there to solve the larger problem […]
If I knew, I'd run out an implement it rather than talk about it! A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On Jan 16, 2014 9:08 AM, "Andrew Sullivan" <asullivan@dyn.com> wrote:
On Thu, Jan 16, 2014 at 11:48:56AM -0500, Christopher Morrow wrote:
I totally agree... I was actually joking in my last note :( sorry for not adding the ":)" as requisite in email.
I'm sorry my humour is now so impaired from reading 1net and other such things that I didn't figure it out!
So... what other options are there to solve the larger problem […]
If I knew, I'd run out an implement it rather than talk about it!
A
Well. These reflection attacks have something in common. The big ones (chargen, dns, ntp) are all IPv4 UDP. And these are all *very* big. I hate to throw the baby out with the bathwater, but in my network, IPv4 UDP is overstaying it's welcome. Just like IPv4 ICMP in 2001 - 2003, its fate is nearly certain. I hope QUIC does not stay on UDP, as it may find itself cut off at the legs. CB
-- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On Thu, Jan 16, 2014 at 09:19:44AM -0800, Cb B wrote:
I hate to throw the baby out with the bathwater, but in my network, IPv4 UDP is overstaying it's welcome. Just like IPv4 ICMP in 2001 - 2003, its fate is nearly certain.
I won't speak about the other protocols, but I encourage you to turn off DNS using UDP over IPv4 in your network and report back to us all on how that works out. You may not be able to do it by email, however. Best regards, A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On Jan 16, 2014 9:31 AM, "Andrew Sullivan" <asullivan@dyn.com> wrote:
On Thu, Jan 16, 2014 at 09:19:44AM -0800, Cb B wrote:
I hate to throw the baby out with the bathwater, but in my network, IPv4 UDP is overstaying it's welcome. Just like IPv4 ICMP in 2001 - 2003,
its
fate is nearly certain.
I won't speak about the other protocols, but I encourage you to turn off DNS using UDP over IPv4 in your network and report back to us all on how that works out. You may not be able to do it by email, however.
I white listed google and facebooks auth servers, so its cool. CB
Best regards,
A
-- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
On 16 Jan 2014, at 17:30 , Andrew Sullivan <asullivan@dyn.com> wrote:
On Thu, Jan 16, 2014 at 09:19:44AM -0800, Cb B wrote:
I hate to throw the baby out with the bathwater, but in my network, IPv4 UDP is overstaying it's welcome. Just like IPv4 ICMP in 2001 - 2003, its fate is nearly certain.
I won't speak about the other protocols, but I encourage you to turn off DNS using UDP over IPv4 in your network and report back to us all on how that works out. You may not be able to do it by email, however.
Why not? .org and nanog.org have IPv6-enabled DNS, mailman.nanog.org has an IPv6 address. What else do you need to reply to this list? — Bjoern A. Zeeb ????????? ??? ??????? ??????: '??? ??? ???? ?????? ??????? ?? ?? ??????? ??????? ??? ????? ????? ???? ?????? ?? ????? ????', ????????? ?????????, "??? ????? ?? ?????", ?.???
On (2014-01-16 09:19 -0800), Cb B wrote:
I hope QUIC does not stay on UDP, as it may find itself cut off at the legs.
Any new L4 would need to support both flavours, over UDP and native. Over UDP is needed to be deployable right now and be working to vast majority of the end users. Native-only would present chicken and egg problem, goog/fb/amzn/msft etc won't add support to it, because failure rate is too high, and stateful box vendors won't add support, because no customer demand. And what becomes to deployment pace, good technologies which give benefits to end users can and have been deployed very fast. IPv6 does not give benefit to end users, EDNS does not give benefit to end users. QUIC/MinimaLT/IETF-transport-standardized-version would give benefit to end users, all persistent connections like ssh would keep running when you jump between networks. You could in your homeserver specifically allow /you/ to connect to any service, regardless of your IP address, because key is your identity, not your IP address. (So sort of LISPy thing going on here, we'd make IP more low-level information which it should be, it wouldn't be identity anymore) Parity packets have potential to give much better performance in packet loss conditions. Packet pacing seems much better on fast to slow file transfers. -- ++ytti
On Jan 16, 2014 10:16 AM, "Saku Ytti" <saku@ytti.fi> wrote:
On (2014-01-16 09:19 -0800), Cb B wrote:
I hope QUIC does not stay on UDP, as it may find itself cut off at the legs.
Any new L4 would need to support both flavours, over UDP and native. Over
is needed to be deployable right now and be working to vast majority of
end users. Native-only would present chicken and egg problem, goog/fb/amzn/msft etc won't add support to it, because failure rate is too high, and stateful box vendors won't add support, because no customer demand.
And what becomes to deployment pace, good technologies which give benefits to end users can and have been deployed very fast. IPv6 does not give benefit to end users, EDNS does not give benefit to end users.
QUIC/MinimaLT/IETF-transport-standardized-version would give benefit to end users, all persistent connections like ssh would keep running when you jump between networks. You could in your homeserver specifically allow /you/ to connect to any service, regardless of your IP address, because key is your identity, not your IP address. (So sort of LISPy thing going on here, we'd make IP more low-level information which it should be, it wouldn't be identity anymore) Parity packets have potential to give much better performance in packet loss conditions. Packet pacing seems much better on fast to slow file
UDP the transfers.
-- ++ytti
Then let's go all the way with ILNP. I like that. CB
On Thu, Jan 16, 2014 at 10:48 AM, Christopher Morrow < morrowc.lists@gmail.com> wrote:
On Thu, Jan 16, 2014 at 11:39 AM, Andrew Sullivan <asullivan@dyn.com> wrote:
On Thu, Jan 16, 2014 at 11:32:05AM -0500, Christopher Morrow wrote:
So... what other options are there to solve the larger problem of:
"Some service is running on a public host, and it can be used to attack a third party"
where in all of these cases the third party is someone who's address has been spoofed in the src-address of a packet.
Standardizing some UDP service-neutral PRE-Authentication system comes to mind; operating at the host level, independent of the various services, requiring the client to perform at least as much work as the response packet. E.g. Fwknop Single-Packet Authentication/Port-Knocking Style "The application doesn't see any UDP packets from a source:port number pair, until a source address authenticity event occurs, for that source ip:port number pair." Say instead of "port knocking" though, the client is required to estimate the size of the response to its first round trip of UDP request messages. Then the client's UDP stack must construct and send a Hashcash proof of work, of sufficient difficulty based on the estimated query plus response size, up to the first full round trip; containing a message digest of the first UDP packet the client will send, before sending the packet, or it will be silently discarded. An out-of-band reply will come back to the claimed source, that the client souce IP:Port has to acknowledge within 5 packets. Once the out-of-band reply is acknowledged, the source is confirmed not to be spoofed. UDP response packets and new queries above the estimated initial reply size or after the 5th packet are delayed until definitive confirmation or silently dropped, instead of returned to the claimed source. -chris
--
-JH
On Thu, 16 Jan 2014 13:35:00 -0600, Jimmy Hess said:
Then the client's UDP stack must construct and send a Hashcash proof of work, of sufficient difficulty based on the estimated query plus response size, up to the first full round trip; containing a message digest of the first UDP packet the client will send, before sending the packet, or it will be silently discarded.
An out-of-band reply will come back to the claimed source, that the client souce IP:Port has to acknowledge within 5 packets. Once the out-of-band reply is acknowledged, the source is confirmed not to be spoofed.
How is this any better than a TCP 3-packet handshake with syncookies?
We don't need to change transport, we don't need to port knock. We just need to implementent a slightly modified dns cookies which reminds me that I need to review Donald Eastlake's new draft to be. -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, Jan 16, 2014 at 3:05 PM, Mark Andrews <marka@isc.org> wrote:
We don't need to change transport, we don't need to port knock. We just need to implementent a slightly modified dns cookies which reminds me that I need to review Donald Eastlake's new draft to be.
But a change to DNS doesn't solve the problem for the other thousand or so UDP-based protocols. What would your fix be for the Chargen and SNMP protocols? --
Mark Andrews, ISC
-- -JH
In message <CAAAwwbVJKEok-ydwEQd4cowJ9qAAtbC8mKqwNXrsud55+H9ZEw@mail.gmail.com> , Jimmy Hess writes:
On Thu, Jan 16, 2014 at 3:05 PM, Mark Andrews <marka@isc.org> wrote:
We don't need to change transport, we don't need to port knock. We just need to implementent a slightly modified dns cookies which reminds me that I need to review Donald Eastlake's new draft to be.
But a change to DNS doesn't solve the problem for the other thousand or so UDP-based protocols.
What thousand protocols? There really are very few protocols widely deployed on top of UDP.
What would your fix be for the Chargen and SNMP protocols?
Chargen is turned off on many platforms by default. Turn it off on more. Chargen loops are detectable. SNMP doesn't need to be open to the entire world. It's not like authoritative DNS servers which are offering a service to everyone. New UDP based protocols need to think about how to handle spoof traffic. You look at providing extending routing protocols to provide information about the legitimate source addresses that may be emitted over a link. SIDR should help here with authentication of the data. This will enable better automatic filtering to be deployed. You continue to deploy BCP38. Every site that deploys BCD is one less site where owened machines can be used to launch attacks from. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Jan 16, 2014 5:10 PM, "Mark Andrews" <marka@isc.org> wrote:
In message <
CAAAwwbVJKEok-ydwEQd4cowJ9qAAtbC8mKqwNXrsud55+H9ZEw@mail.gmail.com>
, Jimmy Hess writes:
On Thu, Jan 16, 2014 at 3:05 PM, Mark Andrews <marka@isc.org> wrote:
We don't need to change transport, we don't need to port knock. We just need to implementent a slightly modified dns cookies which reminds me that I need to review Donald Eastlake's new draft to be.
But a change to DNS doesn't solve the problem for the other thousand or so UDP-based protocols.
What thousand protocols? There really are very few protocols widely deployed on top of UDP.
What would your fix be for the Chargen and SNMP protocols?
Chargen is turned off on many platforms by default. Turn it off on more. Chargen loops are detectable.
Somebody has it on. I can confirm multi gb/s size chargen attacks going on regularly. I agree. More chargen off, more bcp 38, but ...yeh.. chargen is a big problem here and now CB
SNMP doesn't need to be open to the entire world. It's not like authoritative DNS servers which are offering a service to everyone.
New UDP based protocols need to think about how to handle spoof traffic.
You look at providing extending routing protocols to provide information about the legitimate source addresses that may be emitted over a link. SIDR should help here with authentication of the data. This will enable better automatic filtering to be deployed.
You continue to deploy BCP38. Every site that deploys BCD is one less site where owened machines can be used to launch attacks from.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
In message <CAD6AjGTE-raK1AnFha+tz+WQGAuUrB7Pr0vfc3J=QnHFu638vw@mail.gmail.com> , Cb B writes:
On Jan 16, 2014 5:10 PM, "Mark Andrews" <marka@isc.org> wrote:
In message <
CAAAwwbVJKEok-ydwEQd4cowJ9qAAtbC8mKqwNXrsud55+H9ZEw@mail.gmail.com>
, Jimmy Hess writes:
On Thu, Jan 16, 2014 at 3:05 PM, Mark Andrews <marka@isc.org> wrote:
We don't need to change transport, we don't need to port knock. We just need to implementent a slightly modified dns cookies which reminds me that I need to review Donald Eastlake's new draft to be.
But a change to DNS doesn't solve the problem for the other thousand or so UDP-based protocols.
What thousand protocols? There really are very few protocols widely deployed on top of UDP.
What would your fix be for the Chargen and SNMP protocols?
Chargen is turned off on many platforms by default. Turn it off on more. Chargen loops are detectable.
Somebody has it on.
I can confirm multi gb/s size chargen attacks going on regularly.
I agree. More chargen off, more bcp 38, but ...yeh.. chargen is a big problem here and now
So go and *report* the traffic streams so that chargen service can be turned off or if the box doesn't support that, the box is replaced / filter. I don't know anyone that *needs* chargen turned on all the time. Most *never* need it to be turned on. India was just declared polio free. Fixing chargen is easier than that. Step 1. make sure you do not have chargen sources. Step 2. report traffic. Step 3. stop accepting all traffic to/from the if step 2 does not help. Mark
CB
SNMP doesn't need to be open to the entire world. It's not like authoritative DNS servers which are offering a service to everyone.
New UDP based protocols need to think about how to handle spoof traffic.
You look at providing extending routing protocols to provide information about the legitimate source addresses that may be emitted over a link. SIDR should help here with authentication of the data. This will enable better automatic filtering to be deployed.
You continue to deploy BCP38. Every site that deploys BCD is one less site where owened machines can be used to launch attacks from.
Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
--047d7bfd030c59198804f02057ae Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
<p dir=3D"ltr"><br> On Jan 16, 2014 5:10 PM, "Mark Andrews" <<a href=3D"mailto:mar= ka@isc.org">marka@isc.org</a>> wrote:<br> ><br> ><br> > In message <<a href=3D"mailto:CAAAwwbVJKEok-ydwEQd4cowJ9qAAtbC8mKqw= NXrsud55%2BH9ZEw@mail.gmail.com">CAAAwwbVJKEok-ydwEQd4cowJ9qAAtbC8mKqwNXrsu= d55+H9ZEw@mail.gmail.com</a>><br> > , Jimmy Hess writes:<br> > > On Thu, Jan 16, 2014 at 3:05 PM, Mark Andrews <<a href=3D"mail= to:marka@isc.org">marka@isc.org</a>> wrote:<br> > ><br> > > > We don't need to change transport, we don't need to = port knock. =A0We<br> > > > just need to implementent a slightly modified dns cookies wh= ich<br> > > > reminds me that I need to review Donald Eastlake's new d= raft to be.<br> > > ><br> > ><br> > > But a change to DNS doesn't solve the problem for the other t= housand or so<br> > > UDP-based protocols.<br> ><br> > What thousand protocols? =A0There really are very few protocols widely= <br> > deployed on top of UDP.<br> ><br> > > What would your fix be for the Chargen and SNMP protocols?<br> ><br> > Chargen is turned off on many platforms by default. =A0Turn it off<br> > on more. =A0Chargen loops are detectable.<br> ></p> <p dir=3D"ltr">Somebody has it on. </p> <p dir=3D"ltr">I can confirm multi gb/s size chargen attacks going on regul= arly. </p> <p dir=3D"ltr">I agree. More chargen off, more bcp 38, but ...yeh.. chargen= is a big problem here and now</p> <p dir=3D"ltr">CB</p> <p dir=3D"ltr">> SNMP doesn't need to be open to the entire world. = =A0It's not like<br> > authoritative DNS servers which are offering a service to everyone.<br=
><br> > New UDP based protocols need to think about how to handle spoof<br> > traffic.<br> ><br> > You look at providing extending routing protocols to provide<br> > information about the legitimate source addresses that may be emitted<= br> > over a link. =A0SIDR should help here with authentication of the data.= <br> > This will enable better automatic filtering to be deployed.<br> ><br> > You continue to deploy BCP38. =A0Every site that deploys BCD is one<br=
> less site where owened machines can be used to launch attacks from.<br=
><br> > Mark<br> > --<br> > Mark Andrews, ISC<br> > 1 Seymour St., Dundas Valley, NSW 2117, Australia<br> > PHONE: +61 2 9871 4742 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 INTERNET: <a hr= ef=3D"mailto:marka@isc.org">marka@isc.org</a><br> ><br> </p>
--047d7bfd030c59198804f02057ae-- -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org
On Thu, Jan 16, 2014 at 11:39:46AM -0500, Andrew Sullivan wrote:
On Thu, Jan 16, 2014 at 11:32:05AM -0500, Christopher Morrow wrote:
pretty easy to believe that quic would be helpful right?
Yes. It's also pretty easy to believe that ditching DNS completely in favour of something without 8 billion warts would be helpful.
seems totally feasible.
Certainly, it would be possible to standardize it. Whether it would be "trivial" to get it deployed is quite a different matter. The evidence to date is that there is a very, very long tail in any change having to do with the DNS. We are still, to this day, fighting with sysadmins who are convinced that firewall rules on TCP/53 are perfectly reasonable, even though DNS _always_ used TCP.
I can point anyone interested to the place in the bind source to force it to reply to all UDP queries with TC=1 to force TCP. should be safe on any authority servers, as a recursive server should be able to do outbound TCP. - Jared -- Jared Mauch | pgp key available via finger from jared@puck.nether.net clue++; | http://puck.nether.net/~jared/ My statements are only mine.
On Thu, Jan 16, 2014 at 12:55:18PM -0500, Jared Mauch wrote:
I can point anyone interested to the place in the bind source to force it to reply to all UDP queries with TC=1 to force TCP. should be safe on any authority servers, as a recursive server should be able to do outbound TCP.
You could also (and for most cases, I recommend you do) enable the Response Rate Limiting patches available on most of the open-source authoritative servers. Sorry I didn't think to mention it earlier. I thought everyone already knew that. But it does appear to help. A -- Andrew Sullivan Dyn, Inc. asullivan@dyn.com v: +1 603 663 0448
Jared Mauch <jared@puck.Nether.net> wrote:
I can point anyone interested to the place in the bind source to force it to reply to all UDP queries with TC=1 to force TCP. should be safe on any authority servers, as a recursive server should be able to do outbound TCP.
However see http://www.potaroo.net/ispcol/2013-09/dnstcp.html Tony. -- f.anthony.n.finch <dot@dotat.at> http://dotat.at/ Forties, Cromarty: East, veering southeast, 4 or 5, occasionally 6 at first. Rough, becoming slight or moderate. Showers, rain at first. Moderate or good, occasionally poor at first.
On Jan 17, 2014, at 6:44 AM, Tony Finch <dot@dotat.at> wrote:
Jared Mauch <jared@puck.Nether.net> wrote:
I can point anyone interested to the place in the bind source to force it to reply to all UDP queries with TC=1 to force TCP. should be safe on any authority servers, as a recursive server should be able to do outbound TCP.
However see http://www.potaroo.net/ispcol/2013-09/dnstcp.html
Yes, I’m aware of the excellent work by Geoff on this topic. There are many things that could be done, including the nonce (or similar) approach NTP took with MONLIST vs MRULIST. Perhaps it’s something like this: http://tools.ietf.org/html/draft-eastlake-dnsext-cookies-03 - Jared
On Thu, Jan 16, 2014 at 2:27 PM, Andrew Sullivan <asullivan@dyn.com> wrote:
On Tue, Jan 14, 2014 at 09:18:30AM +0200, Saku Ytti wrote:
mid term, transport area in IETF. DNS, NTP, SNMP, chargen et.al. could trivially change to QUIC/MinimaLT
Oh, yes, it'd obviously be trivial to change DNS to use a different transport. This is shown by the massive success of getting EDNS0 universally deployed in under 10 years. Right?
Perhaps the problem with EDNS0 is exactly its backward compatibility. A parallel protocol adopted by the usual suspects of authoritative and recursive names servers that at some point becomes required for query volumes larger than x qps could account for most of name resolution on the planet in much less than 10 years. Rubens
Just in case you run a legitimate open NTP server, this iptable stanza helps immensely: ## rate limit ntp $IPTABLES -N NTP $IPTABLES -N BLACKHOLE $IPTABLES -A BLACKHOLE -m recent --set --name ntpv4blackhole --rsource $IPTABLES -A BLACKHOLE -j DROP $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 20 --name ntpv4 --rsource -j BLACKHOLE $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 2 --name ntpv4blackhole --rsource -j DROP $IPTABLES -A NTP -m recent --set --name ntpv4 --rsource -j ACCEPT $IPTABLES -A INPUT -p udp -m udp --dport 123 -j NTP I've found that blocking TCP destination NTP to client servers/networks blocks legitimate NTP synchronization for their clients. Although I wish they'd all just use my on-network NTP server, I can't assume they will. Does anyone have a list or source of pool and vendor (Apple/Microsoft/etc) servers so I can permit based on source before blocking based on destination port?
Seriously, just fix your configuration. The part of NTP being abused is completely unrelated to actually synchronizing time. It's a management query, that has no real reason to be enabled remotely. You don't even need to resort to iptables for this, because NTPD has built in rate limiting (which isn't enabled for management queries, but those are trivial to disable). $ ntpdc -c monlist -n clock.xmission.com remote address port local address count m ver code avgint lstint =============================================================================== 173.209.207.233 42422 198.60.22.240 4727 3 3 0 0 0 24.155.184.100 45285 198.60.22.240 11 3 4 0 6 0 107.0.41.2 48625 198.60.22.240 264 3 4 0 5 0 67.108.239.31 40642 198.60.22.240 77084 3 3 0 0 0 177.65.149.237 62212 198.60.22.240 1085 3 1 0 0 0 209.64.161.162 44786 198.60.22.240 19 3 4 0 7 0 103.7.36.38 51618 198.60.22.240 4 3 3 0 8 0 173.209.207.218 50616 198.60.22.240 4731 3 3 0 0 0 69.61.203.25 20766 198.60.22.240 16379 3 4 0 1 0 68.188.251.223 478 198.60.22.240 2 1 3 0 0 0 75.82.183.104 123 198.60.22.240 1 3 4 0 0 0 63.64.124.129 52839 198.60.22.240 150867 3 4 0 0 0 65.201.33.150 151 198.60.22.240 393 3 2 0 3 0 124.228.119.105 24687 198.60.22.240 31 3 3 0 4 0 64.191.150.130 319 198.60.22.240 4494361 3 2 0 0 0 76.102.124.27 123 198.60.22.240 2 3 4 0 0 0 72.235.200.183 123 198.60.22.240 1 3 4 0 0 0 50.73.42.121 10398 198.60.22.240 11 3 3 0 14 0 63.64.124.144 26984 198.60.22.240 5823740 3 4 0 0 0 71.5.8.194 44699 198.60.22.240 3 3 4 0 0 0 143.112.64.2 1320 198.60.22.240 182 1 3 0 6 0 72.235.19.125 123 198.60.22.240 1 3 4 0 0 0 198.237.66.2 10471 198.60.22.240 499 3 3 0 3 0 12.108.21.226 357 198.60.22.240 10 1 3 0 14 0 174.47.116.250 463 198.60.22.240 24 3 4 0 5 0 72.1.71.73 738 198.60.22.240 19 3 3 0 8 0 67.136.57.10 1026 198.60.22.240 243 3 3 0 5 0 64.199.163.5 306 198.60.22.240 231 3 4 0 4 0 70.77.76.153 32188 198.60.22.240 1 3 4 0 0 0 There is no excuse to still be running a NTP server with monlist enabled. Fix your configuration, and you don't need IPTables rules. On 2/16/2014 1:29 PM, Pete Ashdown wrote:
Just in case you run a legitimate open NTP server, this iptable stanza helps immensely:
## rate limit ntp $IPTABLES -N NTP $IPTABLES -N BLACKHOLE $IPTABLES -A BLACKHOLE -m recent --set --name ntpv4blackhole --rsource $IPTABLES -A BLACKHOLE -j DROP $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 20 --name ntpv4 --rsource -j BLACKHOLE $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 2 --name ntpv4blackhole --rsource -j DROP $IPTABLES -A NTP -m recent --set --name ntpv4 --rsource -j ACCEPT $IPTABLES -A INPUT -p udp -m udp --dport 123 -j NTP
I've found that blocking TCP destination NTP to client servers/networks blocks legitimate NTP synchronization for their clients. Although I wish they'd all just use my on-network NTP server, I can't assume they will. Does anyone have a list or source of pool and vendor (Apple/Microsoft/etc) servers so I can permit based on source before blocking based on destination port?
Just add these to your ntp.conf configuration then restart the service: (Works with all default installations that I've found) restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery -- Kate Gerry Network Manager kate@quadranet.com 1-888-5-QUADRA Ext 206 | www.QuadraNet.com Dedicated Servers, Colocation, Cloud Services and more. Datacenters in Los Angeles, Dallas and Miami. Follow us on: -----Original Message----- From: Brian Rak [mailto:brak@gameservers.com] Sent: Sunday, February 16, 2014 6:38 PM To: Pete Ashdown; NANOG list Subject: Re: OpenNTPProject.org Seriously, just fix your configuration. The part of NTP being abused is completely unrelated to actually synchronizing time. It's a management query, that has no real reason to be enabled remotely. You don't even need to resort to iptables for this, because NTPD has built in rate limiting (which isn't enabled for management queries, but those are trivial to disable). $ ntpdc -c monlist -n clock.xmission.com remote address port local address count m ver code avgint lstint =============================================================================== 173.209.207.233 42422 198.60.22.240 4727 3 3 0 0 0 24.155.184.100 45285 198.60.22.240 11 3 4 0 6 0 107.0.41.2 48625 198.60.22.240 264 3 4 0 5 0 67.108.239.31 40642 198.60.22.240 77084 3 3 0 0 0 177.65.149.237 62212 198.60.22.240 1085 3 1 0 0 0 209.64.161.162 44786 198.60.22.240 19 3 4 0 7 0 103.7.36.38 51618 198.60.22.240 4 3 3 0 8 0 173.209.207.218 50616 198.60.22.240 4731 3 3 0 0 0 69.61.203.25 20766 198.60.22.240 16379 3 4 0 1 0 68.188.251.223 478 198.60.22.240 2 1 3 0 0 0 75.82.183.104 123 198.60.22.240 1 3 4 0 0 0 63.64.124.129 52839 198.60.22.240 150867 3 4 0 0 0 65.201.33.150 151 198.60.22.240 393 3 2 0 3 0 124.228.119.105 24687 198.60.22.240 31 3 3 0 4 0 64.191.150.130 319 198.60.22.240 4494361 3 2 0 0 0 76.102.124.27 123 198.60.22.240 2 3 4 0 0 0 72.235.200.183 123 198.60.22.240 1 3 4 0 0 0 50.73.42.121 10398 198.60.22.240 11 3 3 0 14 0 63.64.124.144 26984 198.60.22.240 5823740 3 4 0 0 0 71.5.8.194 44699 198.60.22.240 3 3 4 0 0 0 143.112.64.2 1320 198.60.22.240 182 1 3 0 6 0 72.235.19.125 123 198.60.22.240 1 3 4 0 0 0 198.237.66.2 10471 198.60.22.240 499 3 3 0 3 0 12.108.21.226 357 198.60.22.240 10 1 3 0 14 0 174.47.116.250 463 198.60.22.240 24 3 4 0 5 0 72.1.71.73 738 198.60.22.240 19 3 3 0 8 0 67.136.57.10 1026 198.60.22.240 243 3 3 0 5 0 64.199.163.5 306 198.60.22.240 231 3 4 0 4 0 70.77.76.153 32188 198.60.22.240 1 3 4 0 0 0 There is no excuse to still be running a NTP server with monlist enabled. Fix your configuration, and you don't need IPTables rules. On 2/16/2014 1:29 PM, Pete Ashdown wrote:
Just in case you run a legitimate open NTP server, this iptable stanza helps immensely:
## rate limit ntp $IPTABLES -N NTP $IPTABLES -N BLACKHOLE $IPTABLES -A BLACKHOLE -m recent --set --name ntpv4blackhole --rsource $IPTABLES -A BLACKHOLE -j DROP $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 20 --name ntpv4 --rsource -j BLACKHOLE $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 2 --name ntpv4blackhole --rsource -j DROP $IPTABLES -A NTP -m recent --set --name ntpv4 --rsource -j ACCEPT $IPTABLES -A INPUT -p udp -m udp --dport 123 -j NTP
I've found that blocking TCP destination NTP to client servers/networks blocks legitimate NTP synchronization for their clients. Although I wish they'd all just use my on-network NTP server, I can't assume they will. Does anyone have a list or source of pool and vendor (Apple/Microsoft/etc) servers so I can permit based on source before blocking based on destination port?
Better yet, why is your ntp server even reachable off net? Providing a public clock service needs a lot more configuration effort than a simple, default one -- as just demonstrated. (However, this is not to say that private servers should have management queries enabled.) On 2/17/2014 9:03 AM, Kate Gerry wrote:
Just add these to your ntp.conf configuration then restart the service: (Works with all default installations that I've found)
restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
-- Kate Gerry Network Manager kate@quadranet.com
1-888-5-QUADRA Ext 206 | www.QuadraNet.com Dedicated Servers, Colocation, Cloud Services and more. Datacenters in Los Angeles, Dallas and Miami.
Follow us on:
-----Original Message----- From: Brian Rak [mailto:brak@gameservers.com] Sent: Sunday, February 16, 2014 6:38 PM To: Pete Ashdown; NANOG list Subject: Re: OpenNTPProject.org
Seriously, just fix your configuration. The part of NTP being abused is completely unrelated to actually synchronizing time. It's a management query, that has no real reason to be enabled remotely. You don't even need to resort to iptables for this, because NTPD has built in rate limiting (which isn't enabled for management queries, but those are trivial to disable).
$ ntpdc -c monlist -n clock.xmission.com remote address port local address count m ver code avgint lstint =============================================================================== 173.209.207.233 42422 198.60.22.240 4727 3 3 0 0 0 24.155.184.100 45285 198.60.22.240 11 3 4 0 6 0 107.0.41.2 48625 198.60.22.240 264 3 4 0 5 0 67.108.239.31 40642 198.60.22.240 77084 3 3 0 0 0 177.65.149.237 62212 198.60.22.240 1085 3 1 0 0 0 209.64.161.162 44786 198.60.22.240 19 3 4 0 7 0 103.7.36.38 51618 198.60.22.240 4 3 3 0 8 0 173.209.207.218 50616 198.60.22.240 4731 3 3 0 0 0 69.61.203.25 20766 198.60.22.240 16379 3 4 0 1 0 68.188.251.223 478 198.60.22.240 2 1 3 0 0 0 75.82.183.104 123 198.60.22.240 1 3 4 0 0 0 63.64.124.129 52839 198.60.22.240 150867 3 4 0 0 0 65.201.33.150 151 198.60.22.240 393 3 2 0 3 0 124.228.119.105 24687 198.60.22.240 31 3 3 0 4 0 64.191.150.130 319 198.60.22.240 4494361 3 2 0 0 0 76.102.124.27 123 198.60.22.240 2 3 4 0 0 0 72.235.200.183 123 198.60.22.240 1 3 4 0 0 0 50.73.42.121 10398 198.60.22.240 11 3 3 0 14 0 63.64.124.144 26984 198.60.22.240 5823740 3 4 0 0 0 71.5.8.194 44699 198.60.22.240 3 3 4 0 0 0 143.112.64.2 1320 198.60.22.240 182 1 3 0 6 0 72.235.19.125 123 198.60.22.240 1 3 4 0 0 0 198.237.66.2 10471 198.60.22.240 499 3 3 0 3 0 12.108.21.226 357 198.60.22.240 10 1 3 0 14 0 174.47.116.250 463 198.60.22.240 24 3 4 0 5 0 72.1.71.73 738 198.60.22.240 19 3 3 0 8 0 67.136.57.10 1026 198.60.22.240 243 3 3 0 5 0 64.199.163.5 306 198.60.22.240 231 3 4 0 4 0 70.77.76.153 32188 198.60.22.240 1 3 4 0 0 0
There is no excuse to still be running a NTP server with monlist enabled. Fix your configuration, and you don't need IPTables rules.
On 2/16/2014 1:29 PM, Pete Ashdown wrote:
Just in case you run a legitimate open NTP server, this iptable stanza helps immensely:
## rate limit ntp $IPTABLES -N NTP $IPTABLES -N BLACKHOLE $IPTABLES -A BLACKHOLE -m recent --set --name ntpv4blackhole --rsource $IPTABLES -A BLACKHOLE -j DROP $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 20 --name ntpv4 --rsource -j BLACKHOLE $IPTABLES -A NTP -m recent --update --seconds 5 --hitcount 2 --name ntpv4blackhole --rsource -j DROP $IPTABLES -A NTP -m recent --set --name ntpv4 --rsource -j ACCEPT $IPTABLES -A INPUT -p udp -m udp --dport 123 -j NTP
I've found that blocking TCP destination NTP to client servers/networks blocks legitimate NTP synchronization for their clients. Although I wish they'd all just use my on-network NTP server, I can't assume they will. Does anyone have a list or source of pool and vendor (Apple/Microsoft/etc) servers so I can permit based on source before blocking based on destination port?
On Feb 16, 2014, at 10:03 PM, Kate Gerry <kate@quadranet.com> wrote:
Just add these to your ntp.conf configuration then restart the service: (Works with all default installations that I've found)
restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
It might be useful to note that OS X has long since included these lines in the default NTP daemon configuration (/etc/ntp-restrict.conf) thus, “No worrys, mate." James R. Cutler - james.cutler@consultant.com PGP keys at http://pgp.mit.edu
Kate Gerry writes:
Just add these to your ntp.conf configuration then restart the service: (Wo= rks with all default installations that I've found)
restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
KOD only works with "limited" in the release of NTP you are using. H
I’ll note that this is less than 140 chars, and therefore fits nicely in a tweet. If you’re on twitter, Signal boost the PSA, please. My edited example: https://twitter.com/wesgeorge/status/435404354242478080 Wes George On 2/16/14, 10:03 PM, "Kate Gerry" <kate@quadranet.com> wrote:
add these to your ntp.conf restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
Anything below this line has been added by my company’s mail server, I have no control over it. ----------- This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
On 2/17/14, 7:26 AM, George, Wes wrote:
I’ll note that this is less than 140 chars, and therefore fits nicely in a tweet.
If you’re on twitter, Signal boost the PSA, please.
My edited example: https://twitter.com/wesgeorge/status/435404354242478080
Wes George
On 2/16/14, 10:03 PM, "Kate Gerry" <kate@quadranet.com> wrote:
add these to your ntp.conf restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
I seem to recall some issue with older Windows clients using peer for synchronization. Does not having "nopeer" contribute to DDoS amplification?
Peer means it considers the other side an equal and they will mutually skew time together. If you have peer on for devices you don't consider your time servers, you're opening yourself up to problems. -Blake On Mon, Feb 17, 2014 at 9:14 AM, Pete Ashdown <pashdown@xmission.com> wrote:
On 2/17/14, 7:26 AM, George, Wes wrote:
I'll note that this is less than 140 chars, and therefore fits nicely in a tweet.
If you're on twitter, Signal boost the PSA, please.
My edited example: https://twitter.com/wesgeorge/status/435404354242478080
Wes George
On 2/16/14, 10:03 PM, "Kate Gerry" <kate@quadranet.com> wrote:
add these to your ntp.conf restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery
I seem to recall some issue with older Windows clients using peer for synchronization. Does not having "nopeer" contribute to DDoS amplification?
Blake: Just to make sure I've got this down, listing a device as a "peer" in the ntp.conf file will create a situation where both devices are saying, "I know what time it is" and splitting the difference? Whereas when you list a device as a "server", it's using that as the authority on the correct time? Example: -- # peer 192.168.1.1 iburst peer 192.168.1.2 iburst # server ntp.colby.edu minpoll 6 maxpoll 10 iburst server bonehed.lcs.mit.edu minpoll 6 maxpoll 10 iburst On 2/17/2014 10:28 AM, Blake Dunlap wrote:
Peer means it considers the other side an equal and they will mutually skew time together. If you have peer on for devices you don't consider your time servers, you're opening yourself up to problems.
-Blake
On Feb 17, 2014, at 10:38 AM, Anthony Williams <alby.williams@verizon.com> wrote:
Blake:
Just to make sure I've got this down, listing a device as a "peer" in the ntp.conf file will create a situation where both devices are saying, "I know what time it is" and splitting the difference? Whereas when you list a device as a "server", it's using that as the authority on the correct time?
That is not exactly correct. Listing a system as peer or server means that the time from that system will be used as input to the synchronization algorithm. The selection process may discard data depending on various criteria regardless of peer/server designation. The operations implications are the requirement for your own robust group of peers > 3 and lots of servers. See • RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification • RFC 5906: Network Time Protocol Version 4: Autokey Specification • RFC 5907: Definitions of Managed Objects for Network Time Protocol Version 4 (NTPv4) • RFC 5908: Network Time Protocol (NTP) Server Option for DHCPv6
If you're trying to actually run a ntp server setup as opposed to just trusting the world, I strongly suggest reading the documentation for the service, as most people don't deploy it correctly while they think they have. At minimum, you want a cluster of 3 - 4 servers internally, configured as peers of each other, and listening to some source of time, preferably multiple like a few on the internet from the big public pool, and if you really care about time, set up a GPS receiver or two. You can definitely go farther than the above, but that's the start to doing it right. Anything short of the above is just trusting the world at large, and you'll likely happily follow along with any time skew like that thing a few months/year ago with either tick or tock. Without the above, you don't have enough sane sources to discredit bad advisers (you need 3 for a time lock). -Blake On Mon, Feb 17, 2014 at 9:38 AM, Anthony Williams <alby.williams@verizon.com
wrote:
Blake:
Just to make sure I've got this down, listing a device as a "peer" in the ntp.conf file will create a situation where both devices are saying, "I know what time it is" and splitting the difference? Whereas when you list a device as a "server", it's using that as the authority on the correct time?
Example: --
# peer 192.168.1.1 iburst peer 192.168.1.2 iburst
# server ntp.colby.edu minpoll 6 maxpoll 10 iburst server bonehed.lcs.mit.edu minpoll 6 maxpoll 10 iburst
On 2/17/2014 10:28 AM, Blake Dunlap wrote:
Peer means it considers the other side an equal and they will mutually skew time together. If you have peer on for devices you don't consider your time servers, you're opening yourself up to problems.
-Blake
For knowledge on the list. We found that our Cisco Nexus 7000s had NTP enabled on our public facing VDCs, even when the command "feature ntp" was not present. I had to explicitly enter "no feature ntp" to prevent the NTP server service from existing on our public facing 7K interfaces. Thanks, Mike -----Original Message----- From: Blake Dunlap [mailto:ikiris@gmail.com] Sent: Monday, February 17, 2014 11:03 AM To: nanog@nanog.org Subject: Re: OpenNTPProject.org If you're trying to actually run a ntp server setup as opposed to just trusting the world, I strongly suggest reading the documentation for the service, as most people don't deploy it correctly while they think they have. At minimum, you want a cluster of 3 - 4 servers internally, configured as peers of each other, and listening to some source of time, preferably multiple like a few on the internet from the big public pool, and if you really care about time, set up a GPS receiver or two. You can definitely go farther than the above, but that's the start to doing it right. Anything short of the above is just trusting the world at large, and you'll likely happily follow along with any time skew like that thing a few months/year ago with either tick or tock. Without the above, you don't have enough sane sources to discredit bad advisers (you need 3 for a time lock). -Blake On Mon, Feb 17, 2014 at 9:38 AM, Anthony Williams <alby.williams@verizon.com
wrote:
Blake:
Just to make sure I've got this down, listing a device as a "peer" in the ntp.conf file will create a situation where both devices are saying, "I know what time it is" and splitting the difference? Whereas when you list a device as a "server", it's using that as the authority on the correct time?
Example: --
# peer 192.168.1.1 iburst peer 192.168.1.2 iburst
# server ntp.colby.edu minpoll 6 maxpoll 10 iburst server bonehed.lcs.mit.edu minpoll 6 maxpoll 10 iburst
On 2/17/2014 10:28 AM, Blake Dunlap wrote:
Peer means it considers the other side an equal and they will mutually skew time together. If you have peer on for devices you don't consider your time servers, you're opening yourself up to problems.
-Blake
On Feb 17, 2014, at 10:14 PM, Pete Ashdown <pashdown@xmission.com> wrote:
Does not having "nopeer" contribute to DDoS amplification?
No: <http://www.kb.cert.org/vuls/id/348126> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Luck is the residue of opportunity and design. -- John Milton
On Monday, February 17, 2014 04:38:06 AM Brian Rak wrote:
There is no excuse to still be running a NTP server with monlist enabled. Fix your configuration, and you don't need IPTables rules.
Juniper's Junos implementation (which is based on FreeBSD) hasn't been patched Using firewall filters is the only way to mitigate the vulnerability. For those with Juniper access: http://kb.juniper.net/InfoCenter/index?page=content&id=JSA10613&actp=SUBSCRIPTION It's not clear when the software patch will be made available. As it were, ScreenOS and JUNOSe are not affected, as they don't support the MONLIST feature. Mark.
On Feb 16, 2014, at 7:59 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
Juniper's Junos implementation (which is based on FreeBSD) hasn't been patched
Using firewall filters is the only way to mitigate the vulnerability.
But doesn't the JunOS ntpd read/parse ntpd.conf? It's worth getting to the admin mode shell prompt and taking a poke around /etc.
On Sun, Feb 16, 2014 at 11:09 PM, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
On Feb 16, 2014, at 7:59 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
Juniper's Junos implementation (which is based on FreeBSD) hasn't been patched
Using firewall filters is the only way to mitigate the vulnerability.
But doesn't the JunOS ntpd read/parse ntpd.conf? It's worth getting to the admin mode shell prompt and taking a poke around /etc.
and good luck with figuring out: 1) when you need to re-do that magic move 2) making sure that the move is automatable over time it's sort of weird that the service on a routing platform supports more than 'the current time is XX:YY::ZZ' anyway, eh?
On Feb 16, 2014, at 8:30 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:
and good luck with figuring out: 1) when you need to re-do that magic move 2) making sure that the move is automatable over time
I was suggesting it as an alternative to just chopping off NTP at your border. Presumably it would be a one-off thing until Juniper issues a patch. As for automating it, 'expect' can be a very useful tool in situations like this. --lyndon
On Monday, February 17, 2014 06:35:46 AM Lyndon Nerenberg wrote:
I was suggesting it as an alternative to just chopping off NTP at your border. Presumably it would be a one-off thing until Juniper issues a patch.
In Junos, applying the right filters to your router's control plane will fix the issue. You don't need to block NTP in the data plane. Mark.
Just for the reference, here is a more complete solution for Junos (took me a while searching the web to figure it out), hope it helps someone. policy-options { prefix-list lo0.0-inet-address { apply-path "interfaces lo0 unit 0 family inet address <*>"; } prefix-list ntp-servers { apply-path "system ntp server <*>"; } } firewall { family inet { filter lo-filter { term ntp-allow { from { source-prefix-list { ntp-servers; lo0.0-inet-address; } protocol udp; destination-port ntp; } then accept; } term ntp-other-discard { from { protocol udp; destination-port ntp; } then { discard; } } term zz-accept { then accept; } } } } On Sun, Feb 16, 2014 at 8:42 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On Monday, February 17, 2014 06:35:46 AM Lyndon Nerenberg wrote:
I was suggesting it as an alternative to just chopping off NTP at your border. Presumably it would be a one-off thing until Juniper issues a patch.
In Junos, applying the right filters to your router's control plane will fix the issue. You don't need to block NTP in the data plane.
Mark.
So, be careful as the Juniper solution varies depending on the platform involved. Make sure you check your devices. It took a few iterations for us to get the right filters on everything. - Jared On Feb 17, 2014, at 12:26 AM, Yucong Sun <sunyucong@gmail.com> wrote:
Just for the reference, here is a more complete solution for Junos (took me a while searching the web to figure it out), hope it helps someone.
policy-options { prefix-list lo0.0-inet-address { apply-path "interfaces lo0 unit 0 family inet address <*>"; } prefix-list ntp-servers { apply-path "system ntp server <*>"; } }
firewall { family inet { filter lo-filter { term ntp-allow { from { source-prefix-list { ntp-servers; lo0.0-inet-address; } protocol udp; destination-port ntp; } then accept; } term ntp-other-discard { from { protocol udp; destination-port ntp; } then { discard; } } term zz-accept { then accept; } } } }
On Sun, Feb 16, 2014 at 8:42 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On Monday, February 17, 2014 06:35:46 AM Lyndon Nerenberg wrote:
I was suggesting it as an alternative to just chopping off NTP at your border. Presumably it would be a one-off thing until Juniper issues a patch.
In Junos, applying the right filters to your router's control plane will fix the issue. You don't need to block NTP in the data plane.
Mark.
On Tue, 18 Feb 2014 09:14:59 -0500 Jared Mauch <jared@puck.nether.net> wrote:
prefix-list ntp-servers { apply-path "system ntp server <*>";
Some people also have a 'boot-server [server]' statement. In the off chance that address is different than those listed in the server statements, you may need to account for it as well. If you can, just make sure it is also listed as one of the configured servers. John
On Tuesday, February 18, 2014 04:14:59 PM Jared Mauch wrote:
So, be careful as the Juniper solution varies depending on the platform involved.
Make sure you check your devices. It took a few iterations for us to get the right filters on everything.
Indeed. In particular, different hardware and software combinations for the EX line have different match conditions for ports compared to the routers. Mark.
On Sun, Feb 16, 2014 at 11:42 PM, Mark Tinka <mark.tinka@seacom.mu> wrote:
On Monday, February 17, 2014 06:35:46 AM Lyndon Nerenberg wrote:
I was suggesting it as an alternative to just chopping off NTP at your border. Presumably it would be a one-off thing until Juniper issues a patch.
In Junos, applying the right filters to your router's control plane will fix the issue. You don't need to block NTP in the data plane.
yes, this.
On Monday, February 17, 2014 06:09:01 AM Lyndon Nerenberg wrote:
But doesn't the JunOS ntpd read/parse ntpd.conf? It's worth getting to the admin mode shell prompt and taking a poke around /etc.
You can get access to the shell and edit the daemon configuration files yourself, but based on similar tactics for other areas of Junos (including things like Cron) that some operators have done, it is with reasonable reliability that any custom changes will not persist through software upgrades. So either you add this to the to-do list each time you upgrade code, or get Juniper to fix it natively (since it is now fixed in FreeBSD). In fairness, I haven't tried to muck about with the Junos FreeBSD shell to do this, partly for the reasons above, mostly because my life is already interesting enough :-). If someone else has and can provide a different perspective, please do. Mark.
If somebody has contacts at Juniper who is involved in this, I'd like to get their contact information. -- Harlan Stenn <stenn@ntp.org> http://networktimefoundation.org - be a member!
On 2/16/14, 7:38 PM, Brian Rak wrote:
Seriously, just fix your configuration. The part of NTP being abused is completely unrelated to actually synchronizing time. It's a management query, that has no real reason to be enabled remotely. You don't even need to resort to iptables for this, because NTPD has built in rate limiting (which isn't enabled for management queries, but those are trivial to disable). Thanks for the tip, monitoring is off. I was under the impression that rate-limiting hadn't made it into a stable version of ntpd yet. Is that incorrect?
Rate limitings been in place for quite some time, but I believe it's only for actual time queries. This DDOS uses monlist, which isn't subject to the same rate limits. You've disabled monlist now, so I bet you'll no longer need all the rate limiting IPTables rules. (Though, you'll still see the incoming garbage for awhile, but NTPD will just discard it so it shouldn't cause problems). On 2/17/2014 2:23 AM, Pete Ashdown wrote:
On 2/16/14, 7:38 PM, Brian Rak wrote:
Seriously, just fix your configuration. The part of NTP being abused is completely unrelated to actually synchronizing time. It's a management query, that has no real reason to be enabled remotely. You don't even need to resort to iptables for this, because NTPD has built in rate limiting (which isn't enabled for management queries, but those are trivial to disable). Thanks for the tip, monitoring is off. I was under the impression that rate-limiting hadn't made it into a stable version of ntpd yet. Is that incorrect?
participants (32)
-
Andrew Sullivan -
Anthony Williams -
Bjoern A. Zeeb -
Blake Dunlap -
Brian Rak -
Cb B -
Christopher Morrow -
Damian Menscher -
Derek Andrew -
Dobbins, Roland -
George, Wes -
Harlan Stenn -
James R Cutler -
Jared Mauch -
Jared Mauch
-
Jimmy Hess -
John Kristoff -
Kate Gerry -
Lyndon Nerenberg -
Mark Andrews -
Mark Tinka -
Mike Walter -
Nicolai -
Paul Ferguson -
Paul S. -
Pete Ashdown -
Pierre Lamy -
Rubens Kuhl -
Saku Ytti -
Tony Finch -
Valdis.Kletnieks＠vt.edu -
Yucong Sun