I’m curious what people are seeing these days on the UDP/123 policers in their networks. I know while I was at NTT we rolled some out, and there are a number of variants that have occurred over the past 6-7 years. I’ve heard from people at the NTP Pool as well as having observed some issues with NTP at Akamai and time sync from time to time. Are you still seeing a lot of NTP attacks in your flows these days? Should we be looking to remove these, similar to how we did for SQL/Slammer after a time? - Jared
Yes, we still see lots of UDP amplification attacks using NTP monlist. We use a filter to block UDP src 123 packets of 468 bytes in length (monlist reply with the max 6 IPs). -Rich On 3/17/20, 8:55 AM, "NANOG on behalf of Jared Mauch" <nanog-bounces@nanog.org on behalf of jared@puck.nether.net> wrote: I’m curious what people are seeing these days on the UDP/123 policers in their networks. I know while I was at NTT we rolled some out, and there are a number of variants that have occurred over the past 6-7 years. I’ve heard from people at the NTP Pool as well as having observed some issues with NTP at Akamai and time sync from time to time. Are you still seeing a lot of NTP attacks in your flows these days? Should we be looking to remove these, similar to how we did for SQL/Slammer after a time? - Jared E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Tue, Mar 17, 2020 at 9:03 AM Compton, Rich A <Rich.Compton@charter.com> wrote:
Yes, we still see lots of UDP amplification attacks using NTP monlist. We use a filter to block UDP src 123 packets of 468 bytes in length (monlist reply with the max 6 IPs).
-Rich
+1 , still see, still have policers Fyi, ipv6 ntp / udp tends to have a much higher success rate getting through cgn / policers / ...
On 3/17/20, 8:55 AM, "NANOG on behalf of Jared Mauch" < nanog-bounces@nanog.org on behalf of jared@puck.nether.net> wrote:
I’m curious what people are seeing these days on the UDP/123 policers in their networks.
I know while I was at NTT we rolled some out, and there are a number of variants that have occurred over the past 6-7 years. I’ve heard from people at the NTP Pool as well as having observed some issues with NTP at Akamai and time sync from time to time.
Are you still seeing a lot of NTP attacks in your flows these days?
Should we be looking to remove these, similar to how we did for SQL/Slammer after a time?
- Jared
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On 17/Mar/20 18:05, Ca By wrote:
+1 , still see, still have policers
Fyi, ipv6 ntp / udp tends to have a much higher success rate getting through cgn / policers / ...
For those that have come in as attacks toward customers, we've "scrubbed" them where there has been interest. Mark.
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability. http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf On Tue, Mar 17, 2020 at 11:17 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 17/Mar/20 18:05, Ca By wrote:
+1 , still see, still have policers
Fyi, ipv6 ntp / udp tends to have a much higher success rate getting through cgn / policers / ...
For those that have come in as attacks toward customers, we've "scrubbed" them where there has been interest.
Mark.
On Wed, Mar 18, 2020 at 8:46 AM Steven Sommars <stevesommarsntp@gmail.com> wrote:
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability.
http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
Yeh, not changing ipv4 filters, Sorry pool. Burned once, twice shy. There is no simple way to do router filters based on ntp app modes. I suggest people be aware of time.google.com And time.cloudflare.com CB
On Tue, Mar 17, 2020 at 11:17 AM Mark Tinka <mark.tinka@seacom.mu> wrote:
On 17/Mar/20 18:05, Ca By wrote:
+1 , still see, still have policers
Fyi, ipv6 ntp / udp tends to have a much higher success rate getting through cgn / policers / ...
For those that have come in as attacks toward customers, we've "scrubbed" them where there has been interest.
Mark.
On Wed, 18 Mar 2020 at 18:05, Ca By <cb.list6@gmail.com> wrote:
Yeh, not changing ipv4 filters, Sorry pool. Burned once, twice shy.
On many edge routers from Juniper, Nokia and Cisco you can create offset based bit-matches. I'm NTP illiterate, but isn't NTP mode in fixed offset after UDP header? So it should be possible to do this in hardware at linerate just the same as matching UDP port on typical edge router. -- ++ytti
On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars <stevesommarsntp@gmail.com> wrote:
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability.
http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
I've advocated a throttle (not a hard block) on udp/123 packets with 468 Bytes/packet (the size of a full monlist response). In your paper you mention NTS extensions can be 200+ bytes. How large do those packets typically get, in practice? And how significant is packet loss for them (if there's high packet loss during the occasional attack, does that pose a problem)? Damian
On 3/18/2020 4:46 PM, Damian Menscher via NANOG wrote:
On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars <stevesommarsntp@gmail.com <mailto:stevesommarsntp@gmail.com>> wrote:
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability.
http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
I've advocated a throttle (not a hard block) on udp/123 packets with 468 Bytes/packet (the size of a full monlist response). In your paper you mention NTS extensions can be 200+ bytes. How large do those packets typically get, in practice? And how significant is packet loss for them (if there's high packet loss during the occasional attack, does that pose a problem)?
I expect to see NTP UDP packets that would approach the MTU limit, in some cases. If a packet is "too big" for some pathway, then are we talking about a fractional packet loss or are we talking about 100% packet loss (dropped mid-way due to size)?
Damian
-- Harlan Stenn <stenn@nwtime.org> http://networktimefoundation.org - be a member!
On Wed, Mar 18, 2020 at 7:05 PM Harlan Stenn <stenn@nwtime.org> wrote:
On 3/18/2020 4:46 PM, Damian Menscher via NANOG wrote:
On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars <stevesommarsntp@gmail.com <mailto:stevesommarsntp@gmail.com>> wrote:
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability.
http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
I've advocated a throttle (not a hard block) on udp/123 packets with 468 Bytes/packet (the size of a full monlist response). In your paper you mention NTS extensions can be 200+ bytes. How large do those packets typically get, in practice? And how significant is packet loss for them (if there's high packet loss during the occasional attack, does that pose a problem)?
I expect to see NTP UDP packets that would approach the MTU limit, in some cases.
If a packet is "too big" for some pathway, then are we talking about a fractional packet loss or are we talking about 100% packet loss (dropped mid-way due to size)?
I implement it as a throttle... but that effectively means "0% packet loss most of the time, and significant packet loss when there's a large attack". Doing it as a complete block (which Sommars showed some networks do) is unnecessary. So my question is whether occasional bursts of hard-block are a problem. If you only need large packets during the initialization phase, but the ongoing communication is done with smaller packets, then my approach may be acceptable, and we just need to convince other network operators to throttle rather than hard-block the large packets. But if you need large packets all the time, and occasional breakage of large packets causes problems, then I'll need to re-think my approach. Damian
NTS is initialized using a relatively expensive, but short lived, TCP TLS session. NTP loss due to rate limiting will require more frequent TCP initializations. The NTP size-blocks I've observed have been hard, not rate limits. Martin Langer provided a table showing sizes between 228 and 1468 bytes depending on the encryption algorithm and number of cookies. I've asked the NTS authors about potential workarounds, but suspect it would be difficult. The authors have also confirmed that size blocks have been detected during NTS tests. The secure time transfer of NTS was designed to avoid amplification attacks. Impairment of UDP port 123 could require use of alternate UDP port(s), which might be unpopular. On Wed, Mar 18, 2020 at 6:46 PM Damian Menscher <damian@google.com> wrote:
On Wed, Mar 18, 2020 at 8:45 AM Steven Sommars <stevesommarsntp@gmail.com> wrote:
The various NTP filters (rate limits, packet size limits) are negatively affecting the NTP Pool, the new secure NTP protocol (Network Time Security) and other clients. NTP filters were deployed several years ago to solve serious DDoS issues, I'm not second guessing those decisions. Changing the filters to instead block NTP mode 7, which cover monlist and other diagnostics, would improve NTP usability.
http://www.leapsecond.com/ntp/NTP_Suitability_PTTI2020_Revised_Sommars.pdf
I've advocated a throttle (not a hard block) on udp/123 packets with 468 Bytes/packet (the size of a full monlist response). In your paper you mention NTS extensions can be 200+ bytes. How large do those packets typically get, in practice? And how significant is packet loss for them (if there's high packet loss during the occasional attack, does that pose a problem)?
Damian
participants (8)
-
Ca By -
Compton, Rich A -
Damian Menscher -
Harlan Stenn -
Jared Mauch -
Mark Tinka -
Saku Ytti -
Steven Sommars