In my opinion you can, just that the participants are not willing to do so due, in part, to the lack of acceptance of BCP38. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 2020-03-14 07:35, nanog@jack.fr.eu.org wrote:

This is the issue (and the leitmotiv) of forged packets: you cannot trace their origin in real world situation

On 3/14/20 12:03 PM, Jean | ddostest.me via NANOG wrote:

...

Hi Bill,

can you post some forged packets please? You can send them offlist if you prefer.

It seems to be similar to what Octopus experience few weeks ago on this list.

Thanks

Jean St-Laurent | CISSP #634103

Hi Bill, thanks for sharing the data. Indeed, I can't offer you a way to backtrack the spoofed packets. Anyway, I'm not sure what could you do legally as there is a very high chance that these people are not in the USA and the CFAA won't apply to them. Here is what I would do if I was in your situation. Since these packets are spoof and malformed, I would block all SYN/ACK based on the length. Depending on your hardware, it's very easy to inspect _only the SYN/ACK by length_ if you have modern hardware. On linux/unix, it's also very straightforward. I'm not sure for windows though. Here is the details of the analysis: Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, but let's use the basic one. In your case, there are none. There is only MSS and the SYN length is 44 bytes. These are spoof packets maybe generated by either TCP-AMP like reported earlier. I would try to block all SYN/ACK coming toward your network with a length of 44 bytes or lower. But, this is weird because it should be 54 bytes. Maybe there is some offloading of some sort in your gear. Now depending on your hardware, it could work or it could kill your router as it will punt the cpu. I guess you have some modern gear. What I do when I am not sure about the length, I start to accept and log at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins. Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than X bytes. It won't kill or block anything legitimate if you do it properly. :) What will happen is that you will not reply to these spoof SYN/ACK with a RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your service providers will be happy and should not penalize you. I'm pretty sure that it will help you as it did for me in the past. Let me know if it's not clear and/or which part is foggy and I'll try to give more details and better explanation. Regards, Jean St-Laurent On 2020-03-14 11:46, William Herrin wrote:

On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG <nanog@nanog.org> wrote:

...

can you post some forged packets please? You can send them offlist if you prefer. Hi Jean,

Here are a couple examples (PDT this morning):

08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], proto ICMP (1), length 56) 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - admin prohibited filter, length 36 IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP (6), length 40) 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a 0x0030: 267b 01bb a057 e903

08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 44) 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a (correct), seq 1216155085, ack 11765035, win 29200, options [mss 1156], length 0 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b 0x0020: 6012 7210 c97a 0000 0204 0484

I have observed no consistency in the remote IP addresses. I receive no more than a few of each and they don't line up with particular networks. Remote ports are heavily 80, 443, 22, 25, etc. but a smattering of less common ports too. I'm not seeing any RSTs at all nor any port-unreachables. Lots of syn/acks and a few time exceeded and host unreachables. I don't know what to make of that.

On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith <andrew.william.smith@gmail.com> wrote:

...

Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message. Clever! I wouldn't have thought of that. Unfortunately as in the example above, the TTLs in the packets encapsulated in ICMP are not especially close to one of the common boundaries.

Regards, Bill Herrin

-- William Herrin bill@herrin.us https://bill.herrin.us/

Bill: I agree with Damian that you should try to ensure responding with RST to SYN/ACK; in fact, attackers are sometimes (often?) _looking_ for networks that do not send RST in response to unsolicited SYN/ACK, to spoof their addresses in syn-flooding and other attacks (eg., syn-ack) against victim servers. Not sending RST could even result in you receiving ICMP unreachable - esp. indicating filtering as you received - since server admins may have installed a filter against your prefix (to deal with such abuse). So, I wonder, it is possible that your network/FW/provider already filter the RST responses so they don't reach the (victim) servers? BTW, I'm covering these issues in my DoS lecture as part of the Net-Sec course (see URL below). The foils are available (although not yet latest version, will upload `soon' :) ), the text of the net-sec (2nd) part - not so much, it may take me quite a while to make this (2nd) part useable. -- Amir Herzberg Comcast professor of Security Innovations, University of Connecticut Homepage: https://sites.google.com/site/amirherzberg/home Foundations of Cyber-Security (part I: applied crypto, part II: network-security): https://www.researchgate.net/project/Foundations-of-Cyber-Security On Sat, Mar 14, 2020 at 7:51 PM Damian Menscher via NANOG <nanog@nanog.org> wrote:

I don't recommend filtering the SYN-ACK packets. That's what Octolus did, and the result was leaving half-open SYN_RECV connections on all the nodes used for reflection. That has two downsides:

- the reflectors will retry the SYN-ACK (several times), which increases your PPS load (amplifying the attack) - the providers may notice the large number of SYN_RECV connections from your network and put you on a blacklist

I don't want to leave you with the impression that it's hopeless... these attacks aren't impossible to stop --- it just requires convincing the transit providers to care.

Damian

On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG < nanog@nanog.org> wrote:

...

Hi Bill,

thanks for sharing the data. Indeed, I can't offer you a way to backtrack the spoofed packets.

Anyway, I'm not sure what could you do legally as there is a very high chance that these people are not in the USA and the CFAA won't apply to them.

Here is what I would do if I was in your situation.

Since these packets are spoof and malformed, I would block all SYN/ACK based on the length.

Depending on your hardware, it's very easy to inspect *only the SYN/ACK by length* if you have modern hardware. On linux/unix, it's also very straightforward. I'm not sure for windows though.

Here is the details of the analysis:

Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, but let's use the basic one.

In your case, there are none. There is only MSS and the SYN length is 44 bytes. These are spoof packets maybe generated by either TCP-AMP like reported earlier.

I would try to block all SYN/ACK coming toward your network with a length of 44 bytes or lower. But, this is weird because it should be 54 bytes. Maybe there is some offloading of some sort in your gear.

Now depending on your hardware, it could work or it could kill your router as it will punt the cpu. I guess you have some modern gear.

What I do when I am not sure about the length, I start to accept and log at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins.

Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than X bytes. It won't kill or block anything legitimate if you do it properly. :)

What will happen is that you will not reply to these spoof SYN/ACK with a RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your service providers will be happy and should not penalize you.

I'm pretty sure that it will help you as it did for me in the past.

Let me know if it's not clear and/or which part is foggy and I'll try to give more details and better explanation.

Regards,

Jean St-Laurent On 2020-03-14 11:46, William Herrin wrote:

On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog@nanog.org> <nanog@nanog.org> wrote:

can you post some forged packets please? You can send them offlist if you prefer.

Hi Jean,

Here are a couple examples (PDT this morning):

08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], proto ICMP (1), length 56) 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - admin prohibited filter, length 36 IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP (6), length 40) 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a 0x0030: 267b 01bb a057 e903

08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 44) 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a (correct), seq 1216155085, ack 11765035, win 29200, options [mss 1156], length 0 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b 0x0020: 6012 7210 c97a 0000 0204 0484

I have observed no consistency in the remote IP addresses. I receive no more than a few of each and they don't line up with particular networks. Remote ports are heavily 80, 443, 22, 25, etc. but a smattering of less common ports too. I'm not seeing any RSTs at all nor any port-unreachables. Lots of syn/acks and a few time exceeded and host unreachables. I don't know what to make of that.

On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith@gmail.com> <andrew.william.smith@gmail.com> wrote:

Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message.

Clever! I wouldn't have thought of that. Unfortunately as in the example above, the TTLs in the packets encapsulated in ICMP are not especially close to one of the common boundaries.

Regards, Bill Herrin

-- William Herrinbill@herrin.ushttps://bill.herrin.us/

On Sun, Mar 15, 2020 at 9:07 AM Amir Herzberg <amir.lists@gmail.com> wrote:

Not sending RST could even result in you receiving ICMP unreachable - esp. indicating filtering as you received - since server admins may have installed a filter against your prefix (to deal with such abuse). So, I wonder, it is possible that your network/FW/provider already filter the RST responses so they don't reach the (victim) servers?

Hi Amir, To be clear: the majority of the addresses at my end are not associated with live hosts. There's nothing there to respond. My surprise about the lack of RSTs is the lack of RSTs from the remote servers back to the addresses which have been spoofed. If the attacker was hitting random ports on those hosts, I'd expect to see some RSTs. If you happen to have decent netflow, try looking for packets sourced from 199.33.224.0/24. You'll find a legitimate route in your tables ending at AS11875 but today, at least, there are no legitimate packets sourced from that address block. Regards, Bill -- William Herrin bill@herrin.us https://bill.herrin.us/

Regardless if you leave half-open SYN_RECV, you will still get abuse reports and blacklist it. These providers who blacklist and report you, are reporting you for bruteforce.. and they only check for SYN_RECV.

From my experience there is nothing you can do about that, as you should not monitor TCP Services for SYN_RECV only.. but rather the full TCP-handshake to be completed.. if not, there are large chances it's spoofed.

Only solution we found was using SYNPROXY / CONNTRACK to validate. On 15.03.2020 00:51:11, Damian Menscher via NANOG <nanog@nanog.org> wrote: I don't recommend filtering the SYN-ACK packets. That's what Octolus did, and the result was leaving half-open SYN_RECV connections on all the nodes used for reflection. That has two downsides: - the reflectors will retry the SYN-ACK (several times), which increases your PPS load (amplifying the attack) - the providers may notice the large number of SYN_RECV connections from your network and put you on a blacklist I don't want to leave you with the impression that it's hopeless... these attacks aren't impossible to stop --- it just requires convincing the transit providers to care. Damian On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me [http://ddostest.me] via NANOG <nanog@nanog.org [mailto:nanog@nanog.org]> wrote: Hi Bill, thanks for sharing the data. Indeed, I can't offer you a way to backtrack the spoofed packets. Anyway, I'm not sure what could you do legally as there is a very high chance that these people are not in the USA and the CFAA won't apply to them. Here is what I would do if I was in your situation. Since these packets are spoof and malformed, I would block all SYN/ACK based on the length. Depending on your hardware, it's very easy to inspect only the SYN/ACK by length if you have modern hardware. On linux/unix, it's also very straightforward. I'm not sure for windows though. Here is the details of the analysis: Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, but let's use the basic one. In your case, there are none. There is only MSS and the SYN length is 44 bytes. These are spoof packets maybe generated by either TCP-AMP like reported earlier. I would try to block all SYN/ACK coming toward your network with a length of 44 bytes or lower. But, this is weird because it should be 54 bytes. Maybe there is some offloading of some sort in your gear. Now depending on your hardware, it could work or it could kill your router as it will punt the cpu. I guess you have some modern gear. What I do when I am not sure about the length, I start to accept and log at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins. Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than X bytes. It won't kill or block anything legitimate if you do it properly. :) What will happen is that you will not reply to these spoof SYN/ACK with a RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your service providers will be happy and should not penalize you. I'm pretty sure that it will help you as it did for me in the past. Let me know if it's not clear and/or which part is foggy and I'll try to give more details and better explanation. Regards, Jean St-Laurent On 2020-03-14 11:46, William Herrin wrote: On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me [http://ddostest.me] via NANOG <nanog@nanog.org> [mailto:nanog@nanog.org] wrote: can you post some forged packets please? You can send them offlist if you prefer. Hi Jean, Here are a couple examples (PDT this morning): 08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], proto ICMP (1), length 56) 45.89.93.26 > 199.33.225.218 [http://199.33.225.218]: ICMP host 45.89.93.26 unreachable - admin prohibited filter, length 36 IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP (6), length 40) 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a 0x0030: 267b 01bb a057 e903 08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 44) 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a (correct), seq 1216155085, ack 11765035, win 29200, options [mss 1156], length 0 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b 0x0020: 6012 7210 c97a 0000 0204 0484 I have observed no consistency in the remote IP addresses. I receive no more than a few of each and they don't line up with particular networks. Remote ports are heavily 80, 443, 22, 25, etc. but a smattering of less common ports too. I'm not seeing any RSTs at all nor any port-unreachables. Lots of syn/acks and a few time exceeded and host unreachables. I don't know what to make of that. On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith <andrew.william.smith@gmail.com> [mailto:andrew.william.smith@gmail.com] wrote: Look inside the ICMP Unreachable backscatter at the truncated original packet that caused the unreachable message. Clever! I wouldn't have thought of that. Unfortunately as in the example above, the TTLs in the packets encapsulated in ICMP are not especially close to one of the common boundaries. Regards, Bill Herrin -- William Herrin bill@herrin.us [mailto:bill@herrin.us] https://bill.herrin.us/ [https://bill.herrin.us/]