On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote:

I don't recommend filtering the SYN-ACK packets. That's what Octolus did, and the result was leaving half-open SYN_RECV connections on all the nodes used for reflection. That has two downsides:

- the reflectors will retry the SYN-ACK (several times), which increases your PPS load (amplifying the attack) - the providers may notice the large number of SYN_RECV connections from your network and put you on a blacklist

I work at Path Network; we're providing DDoS mitigation services for Octolus. I wanted to follow up your message with a couple of points from our POV. The filter we have running for Octolus is a more generic stateful TCP filter, designed primarily to curb spoofed TCP floods. The bigger attacks we see are on the magnitude of 100s of Gbps/Mpps, and a simple fact is that outright dropping an out-of-state packet is multiple times less expensive for us than creating and responding with a RST. We see spoofed TCP attacks much more frequently than a SYN-ACK reflection, and in fact from our automation's point of view this looks exactly like a SYN-ACK flood from randomized source addresses. Aside from the resources cost, sending back RSTs for non-reflected attacks of respectable volume might also not be appreciated by some networks. Knowing the specifics of this long-running reflection attacks, we're considering how to reply with RSTs so to not leave lingering SYN_RECVs on reflectors' side. On 2020-03-15 16:36, Jean | ddostest.me <jean at ddostest.me> wrote:

I believe that Oculus blocked the RST and not the SYN/ACK.

I don't want to leave you with the impression that it's hopeless... these attacks aren't impossible to stop --- it just requires convincing the transit providers to care.

Damian

On Sat, Mar 14, 2020 at 1:31 PM Jean | ddostest.me via NANOG < nanog at nanog.org> wrote:

Hi Bill,

thanks for sharing the data. Indeed, I can't offer you a way to backtrack the spoofed packets.

Anyway, I'm not sure what could you do legally as there is a very high chance that these people are not in the USA and the CFAA won't apply to them.

Here is what I would do if I was in your situation.

Since these packets are spoof and malformed, I would block all SYN/ACK based on the length.

Depending on your hardware, it's very easy to inspect *only the SYN/ACK by length* if you have modern hardware. On linux/unix, it's also very straightforward. I'm not sure for windows though.

Here is the details of the analysis:

Today, all the SYN and SYN/ACK includes a minimum of options like MSS, WS, SACK, NOP (Only a spacer, sometimes 2) and extended TS. There might be others, but let's use the basic one.

In your case, there are none. There is only MSS and the SYN length is 44 bytes. These are spoof packets maybe generated by either TCP-AMP like reported earlier.

I would try to block all SYN/ACK coming toward your network with a length of 44 bytes or lower. But, this is weird because it should be 54 bytes. Maybe there is some offloading of some sort in your gear.

Now depending on your hardware, it could work or it could kill your router as it will punt the cpu. I guess you have some modern gear.

What I do when I am not sure about the length, I start to accept and log at 60 bytes, then 58, 56, 54... 44 until I catch the gremlins.

Once you found the sweet spot, you drop all SYN/ACK toward your /23 lower than X bytes. It won't kill or block anything legitimate if you do it properly. :)

What will happen is that you will not reply to these spoof SYN/ACK with a RST and still allowing RST for legit SYN and SYN/ACK. Akamai and your service providers will be happy and should not penalize you.

I'm pretty sure that it will help you as it did for me in the past.

Let me know if it's not clear and/or which part is foggy and I'll try to give more details and better explanation.

Regards,

Jean St-Laurent On 2020-03-14 11:46, William Herrin wrote:

On Sat, Mar 14, 2020 at 4:02 AM Jean | ddostest.me via NANOG<nanog at nanog.org> <nanog at nanog.org> wrote:

can you post some forged packets please? You can send them offlist if you prefer.

Hi Jean,

Here are a couple examples (PDT this morning):

08:22:43.413250 IP (tos 0x0, ttl 55, id 10108, offset 0, flags [none], proto ICMP (1), length 56) 45.89.93.26 > 199.33.225.218: ICMP host 45.89.93.26 unreachable - admin prohibited filter, length 36 IP (tos 0x0, ttl 69, id 10108, offset 0, flags [DF], proto TCP (6), length 40) 199.33.225.218.9851 > 45.89.93.26.443: [|tcp] 0x0000: 4500 0038 277c 0000 3701 28da 2d59 5d1a 0x0010: c721 e1da 030d 4b61 0000 0000 4500 0028 0x0020: 277c 4000 4506 dae4 c721 e1da 2d59 5d1a 0x0030: 267b 01bb a057 e903

08:25:47.787326 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 44) 104.87.78.95.80 > 199.33.225.143.8667: Flags [S.], cksum 0xc97a (correct), seq 1216155085, ack 11765035, win 29200, options [mss 1156], length 0 0x0000: 4500 002c 0000 4000 3606 e564 6857 4e5f 0x0010: c721 e18f 0050 21db 487d 0dcd 00b3 852b 0x0020: 6012 7210 c97a 0000 0204 0484

I have observed no consistency in the remote IP addresses. I receive no more than a few of each and they don't line up with particular networks. Remote ports are heavily 80, 443, 22, 25, etc. but a smattering of less common ports too. I'm not seeing any RSTs at all nor any port-unreachables. Lots of syn/acks and a few time exceeded and host unreachables. I don't know what to make of that.

On Sat, Mar 14, 2020 at 1:46 AM Andrew Smith<andrew.william.smith at gmail.com> <andrew.william.smith at gmail.com> wrote:

Look inside the ICMP Unreachable backscatter at the truncated original

The SYN-ACKs are dropped; letting them reach the end servers and dropping outgoing RSTs instead would make for poor mitigation. :) On 2020-03-14 23:50, Damian Menscher <damian at google.com> wrote (cont.): packet that caused the unreachable message.

Clever! I wouldn't have thought of that. Unfortunately as in the example above, the TTLs in the packets encapsulated in ICMP are not especially close to one of the common boundaries.

Regards, Bill Herrin

-- William Herrinbill at herrin.ushttps://bill.herrin.us/