The traffic "from" 88.208.0.0/18, 5.11.80.0/21, and 78.140.128.0/18 doesn't match the packet signatures for Masscan, ZMap, or any other well-known scanner. The traffic is likely spoofed. __ *Troy Mursch* @bad_packets On Fri, Aug 16, 2019 at 3:28 PM Jared Smith <jms@vols.utk.edu> wrote:

I would think Shodan/Zmap/pick your multi-IP-block-scanning-tool would portray similar behavior.

Echoing Matt’s “probably shouldn’t worry” sentiment, this could just be someone running an incantation of such tools for research or recreational purposes.

Best, Jared On Aug 16, 2019, 18:21 -0400, Matt Harris , wrote:

On Fri, Aug 16, 2019 at 5:05 PM Jim Shankland <nanog@shankland.org> wrote:

1. Rate seems too slow to do any actual damage (is anybody really bothered by a few bad SYN packets per second per service, at this point?); but

Common technique used by port scanners to evade detection as a DoS attack by fw/ids/etc.

2. IPs/port combinations with actual open services are being targeted (I'm seeing ports 22, 443, and 53, just at a glance, to specific IPs with those services running), implying somebody checked for open services first;

Or they're just checking if certain common ports are open with the intention of later trying known exploits against those which are reachable in order to attempt to compromise the hosts. Build the DB of reachable hosts/ports now, come back with exploits later.

3. I'm seeing this in at least 2 locations, to addresses in different, completely unrelated ASes, implying it may be pretty widespread.

Sounds like a relatively common pattern though.

Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend?

I wouldn't worry too much about it unless you have reason to believe some of the likely-forthcoming exploits may actually work. Of course, if that's the case, you should fix them anyhow.

Have a good weekend!

On 8/16/19 3:50 PM, Emille Blanc wrote:

Have been seeing these at $DAYJOB off and on for the past week. First logged events began for on 2019-08-04, at approx 1500hrs PST.

Impact for us has been negligible, but some older ASA's were having trouble with the scan volume and their configured log levels which has since been remedied.

Thanks for the various responses. The pattern I (and apparently quite a few others) are seeing differs from an ordinary probe in that it is repeated a few times per second (if somebody wants to know who has a visible ssh server on port 22, and what version of sshd is running, they don't have to hit it multiple times per second). It differs from a SYN flood DoS attack in that its rate is too low to be effective. And it differs from both a port probe and a SYN flood attack (or somebody "learning how to use nmap") in that it is targeting a broad set of destinations in parallel; if source addresses are forged, they are from a fairly narrow set of source IPs. The atypical pattern seems noteworthy in itself. Not a crisis, but not quite routine, either. Jim

On Sat, Aug 17, 2019 at 3:36 PM Amir Herzberg <amir.lists@gmail.com> wrote:

Hmm, I doubt this is the output of TCP amplification since Jim reported it as SYN spoofing, i.e., SYN packets, not SYN-ACK packets (as for typical TCP amplification). Unless the given _hosts_ respond with multiple SYN-ACKs in which case these may be experiments by an attacker to measure if these IP:ports could be abused as TCP amplifiers.

Clarifying for those unfamiliar with this attack: - Attacker is sending SYN packets spoofed "from" NL to Jim (and others) - Jim (and others) have applications listening on those ports and respond with SYN-ACK packets to the victim in NL - When the victim (NL) fails to complete the handshake (which they didn't initiate!) Jim (and others) sends another SYN-ACK So they're not probing to see if Jim (and others) are abusable as TCP amplifiers... they've already determined they can be abused and are using those machines to conduct an actual attack against victims in NL. Damian On Sat, Aug 17, 2019 at 6:18 PM Damian Menscher via NANOG <nanog@nanog.org>

wrote:

...

On Fri, Aug 16, 2019 at 3:05 PM Jim Shankland <nanog@shankland.org> wrote:

...

I'm seeing slow-motion (a few per second, per IP/port pair) syn flood attacks ostensibly originating from 3 NL-based IP blocks: 88.208.0.0/18 , 5.11.80.0/21, and 78.140.128.0/18 ("ostensibly" because ... syn flood, and BCP 38 not yet fully adopted).

Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend?

This appears to be a TCP amplification attack. Similar to UDP amplification (DNS, NTP, etc) you can get some amplification by sending a SYN packet with a spoofed source, and watching your victims receive multiple SYN-ACK retries. It's a fairly weak form of attack (as the amplification factor is small), but if the victim's gear is vulnerable to high packet rates it may be effective.

The victim (or law enforcement) could identify the true source of the attack by asking transit providers to check their netflow to see where it enters their networks.

Damian

On 8/17/19 3:16 PM, Damian Menscher wrote:

On Fri, Aug 16, 2019 at 3:05 PM Jim Shankland <nanog@shankland.org <mailto:nanog@shankland.org>> wrote:

I'm seeing slow-motion (a few per second, per IP/port pair) syn flood attacks ostensibly originating from 3 NL-based IP blocks: 88.208.0.0/18 <http://88.208.0.0/18> , 5.11.80.0/21 <http://5.11.80.0/21>, and 78.140.128.0/18 <http://78.140.128.0/18> ("ostensibly" because ... syn flood, and BCP 38 not yet fully adopted).

Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend?

This appears to be a TCP amplification attack.  Similar to UDP amplification (DNS, NTP, etc) you can get some amplification by sending a SYN packet with a spoofed source, and watching your victims receive multiple SYN-ACK retries. It's a fairly weak form of attack (as the amplification factor is small), but if the victim's gear is vulnerable to high packet rates it may be effective.

That thought crossed my mind, but it seems to me that the weak amplification factor, plus the broadly distributed set of forged source addresses (within the blocks cited above), would make the attack ineffective -- the whole point of DDoS being to focus a broadly distributed set of (illegitimately obtained) source resources on a narrow set of destination targets. Attacking 2 /18 blocks plus a /21 block in parallel with a weak-amplification attack doesn't look like a successful DDoS strategy to me. Jim

The victim (or law enforcement) could identify the true source of the attack by asking transit providers to check their netflow to see where it enters their networks.

Damian

The number of TCP syn-ack amplifiers is large. It may suffice to allow clogging a provider or IX, using low load per amplifier, as described. Such low load is likely to be undetected by most operators, and even when detected (e.g. by Jim), only few (e.g. Mike) will have sufficient motivation to block it - esp. considering that there blocking it would often be non-trivial, in Mike's case, the amplifiers were DNS servers and sounds like he simply blocked packets to unallowed networks (good practice for DNS anyway - although I wonder why not block the incoming requests instead). Notice that one annoying aspect of these attacks is that tcp congestion control isn't relevant. The current packets could be part of a research experiment about this threat, or the instrumentation part of preparing such attack. I would not rule out research, since it isn't trivial to know if the attack can be really viable to clog a provider or IX; in fact finding this out in an ethical way appears a non-trivial challenge, I'll give it some thought (ideas welcome). Also I wonder what would be good _defenses_ against such attack. Of course one way would be to prevent spoofed-IP packets, but that goal has proved quite difficult... -- Amir Herzberg Comcast professor for security innovation Dept. of Computer Science and Engineering, University of Connecticut On Sat, Aug 17, 2019 at 11:03 PM Mike <mike-nanog@tiedyenetworks.com> wrote:

On 8/16/19 3:04 PM, Jim Shankland wrote:

...

Greetings,

I'm seeing slow-motion (a few per second, per IP/port pair) syn flood attacks ostensibly originating from 3 NL-based IP blocks: 88.208.0.0/18 , 5.11.80.0/21, and 78.140.128.0/18 ("ostensibly" because ... syn flood, and BCP 38 not yet fully adopted).

Why is this syn flood different from all other syn floods? Well ...

1. Rate seems too slow to do any actual damage (is anybody really bothered by a few bad SYN packets per second per service, at this point?); but

2. IPs/port combinations with actual open services are being targeted (I'm seeing ports 22, 443, and 53, just at a glance, to specific IPs with those services running), implying somebody checked for open services first;

3. I'm seeing this in at least 2 locations, to addresses in different, completely unrelated ASes, implying it may be pretty widespread.

Is anybody else seeing the same thing? Any thoughts on what's going on? Or should I just be ignoring this and getting on with the weekend?

Jim

I am seeing this against my recursive revolvers - one syn in, and many syn-ack's back with no connection obviously. Low rate to be sure, but what was surprising to me was that my revolvers (PowerDNS) definitely have an allowed-networks ACL which specifies my permitted client addresses, and I thought this would effectively filter any inbound queries. But it looks like this ACL is applied only AFTER connection setup. Maybe I have been blind this entire time thinking I wouldn't respond to any packets sent to my resolver from non-allowed-networks addresses. But anyways just a good data-point for anyone else to check your revolvers too and consider the TCP case may behave as mine do. I fixed it by implementing a revised iptables firewall which definitely corrects the issue and drops outright all packets to non-allowed-networks addresses, thank you ipset...

Mike-

Peace, On Sun, Aug 18, 2019 at 6:48 PM Mike <mike-nanog@tiedyenetworks.com> wrote:

[..] I do have an idea that may be potentially a good mitigation strategy and for the exact reason stated above; low load to individual end points may still, in aggregate, overwhelm an IX or provider, so cutting off the SYN-ACK traffic to those hosts which have not requested connections is good internet hygiene...

In theory, yes, but it's incredibly complicated to do that properly at scale.

My idea is to maintain a penaltybox for any client IP that initiated a connection but did not complete, while also maintaining a whitelist of 'frequent fliers' who have previously completed their connections successful.

Unless a connection is completed, you do not know if the source IP address of your client is spoofed or not. (Under certain circumstances you don't know it even then, but it is unlikely that you would have to take such a possibiity into account). Therefore, you should not populate anything in your RAM from such a source. See also my short talk from RIPE 77 for more information: - https://ripe77.ripe.net/presentations/154-ddoswww_ripe77_004.pdf - https://ripe77.ripe.net/archives/video/2336/ Also, odds are a whitelist won't help either.

While looking around, I came across the SYNPROXY netfilter module.

Not sure if it's still supported. I think I've read in LKML that it was dropped since Linux 4.4. Anyhow, it's impossible to scale without a complete rewrite. -- Töma

Peace, On Mon, Aug 19, 2019 at 7:39 AM Damian Menscher via NANOG <nanog@nanog.org> wrote:

Most kernels will return 3-5 SYN-ACK packets for an incoming SYN, so it's not particularly interesting for attackers or defenders.

Well, producing 1000 Gbps as opposed to 200 Gbps is still pretty impressive, isn't it? More on that later, b/c the point here aren't even jiggabits,

it's somewhat pointless to worry about a small amplification factor -- an attacker could [..] use UDP to get a massive bandwidth (or even significant packet) amplification.

Most of the resources hosted by a typical hosting company are essentially Web sites.[citation needed] Unless you are really really dependant on QUIC (and, unless we're all really unlucky and recent initiatives to get rid of TCP/TLS fallback in HTTP/3 would gain support), as a Web hosting company, you can use whatever you want to get rid of UDP completely very quickly, and that won't harm your business a lot. Dealing with TCP flags is a different story: - Your ability to handle them with the likes of RFC 5575 depend on what particular sort of equipment is deployed in your network; - To make matters worse, for a huge portion of customers the ability to connect to an external service/API gateway/Web site via TCP is crucial. A simple example is Google which cannot survive for long if Googlebot keeps being unable to operate. Think also OAuth, Skyscanner, credit scoring systems, insurance companies, etc.; - To ensure proper handling of spoofed SYN/ACKs while still maintaining a possibility to connect to an external service you, as a hosting company under an attack, would have to track all of the outgoing SYNs to match them against received SYN/ACKs later. This is where the "1kGbps-vs-200Gbps" argument becomes important, b/c every existing free connection state tracking solution doesn't scale beyond 200 Gbps at best given the best hardware money can buy given a single machine, and no existing solution is able to share its state across multiple machines. [there are proprietary products doing that though, we have one, but proprietary solutions are always a different kind of story] -- Töma

On Mon, Aug 19, 2019 at 8:12 PM Damian Menscher <damian@google.com> wrote:

A factor of 2 is "rounding error" and we probably shouldn't waste our time on it (eg, by designing solutions to reduce amplification factors) when we could instead be targeting the sources of spoofed traffic.

Ah, fine. Spoofing is obviously the root cause here. I was mostly addressing the statement that factors of 2 to 5 aren't "particularly interesting for attackers or defenders". In my experience they certainly are.

this particular "carpet-bombing" attack isn't likely to be mitigated at the network layer anyway... the load is distributed across thousands of machines which can each trivially handle the state.

Not in a typical DC/ISP environment! With the solution you propose, a perfect routing symmetry is a hard requirement, b/c you need to make sure a returning SYN/ACK hits the very same machine as the initial SYN. As long as you expect a DDoS to be handled somewhere close to the border of your network, this is hardly achievable for a network growing in size. -- Töma

On Mon, Aug 19, 2019, 8:57 PM Valdis Klētnieks <valdis.kletnieks@vt.edu> wrote:

On Mon, 19 Aug 2019 20:44:47 +0300, Töma Gavrichenkov said:

...

Not in a typical DC/ISP environment! With the solution you propose, a perfect routing symmetry is a hard requirement, b/c you need to make sure a returning SYN/ACK hits the very same machine as the initial SYN.

If your load balancer isn't doing something to make that situation work properly, you need to talk to your vendor.

If you're doing load balancing for *outgoing* traffic — and in exactly the same manner as you do with incoming — then maybe. This also assumes that instead of mitigating an attack near the border you set up and keep an internal cluster of filtering machines somewhere and route, in the worst case scenario, *all* of your traffic through that cluster. Depending on the size of your network, it might or might not be an effective solution. -- Töma

On Mon, 19 Aug 2019 21:18:49 +0300, T�ma Gavrichenkov said:

If you're doing load balancing for *outgoing* traffic — and in exactly the same manner as you do with incoming — then maybe.

On the other hand, your servers should probably be doing non-loadbalanced outbound on a different IP address than the inbound load balancer, and thus the syn-ack should have zero trouble getting back to the box it thought the syn came from.