SYN flood messages flooding my mailbox
Wrt- SYN Flooding I'm going to try to consolidate some of these threads. There have been three areas in which defense of SYN flooding has been suggested (maybe more, three that will help): 1. Make the host less susseptible 2. Filter based on source address on inbound packets from singly homed sites. 3. Improve the ability to trace such attacks back to source I'll argue that all three of these are needed since none of them alone can completely solve the problem. Here are some details recapping discussions on the list (sorry for not giving credit, lots of people contributed ideas). 1. Make the host less susseptible [teach engineers to spell?] Specific code fragments were given for BSD and Solaris. BSDI is already in very good shape with recent kernel patch (that preceeded the attacks and was intended to improve HTTP server performance). The comment was made "what about NT". My answer is let Microsoft field that question. At issue is whether a stream of TCP SYNs can swamp a host TCP implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs (until TCP port numbers are exhausted). At a 75 second hold time, this is well under 1kpps (rapid reuse of port numbers has other problems). Some advantage is gained by the fact that SYN ACKs will be sent and legitimate hosts will respond with RST, clearing the port number for reuse. There was also a suggestion of a server to prescreen TCP SYNs. I don't think that is needed. With the host changes, hosts are not invincible but it requires nearly 1 kpps sustained to do damage. Reducing the hold time below 75 seconds (which is on the long side), makes the traffic rates required for a denial of service even higher. What this does is forces the attacker to use relatively long, high pps bursts, which are much more easily traced if providers are prepared to do the tracing (see 3.). 2. Filter based on source address on inbound packets from singly homed sites. A singly homed site cannot have assymetric routing since there is no ohter path. All that is needed is for the router to look up the source address in the forwarding table and if the route does not point to the interface the packet came in on, dispose of the packet. This would mean that even a singly homed university could not become the source of SYN attacks unless the hacker was willing to use source addresses in the range of the addresses of the university. This would make tracing back to the source very easy. 3. Improve the ability to trace such attacks back to source Here is where either router vendors have to take action or providers need to put Unix hosts on their FDDI rings and ethernets to do gather data. The NSS routers did statistical sampling based on a sampling of one in 50 packet headers. If a hacker must send upwards of 60,000 packets for an attack to be effective, even a sampling of 1 in 1,000 would be effective. The NSS was unique as routers go in that a general purpose computer was part of the package. The specialized code in the forwarding path sampled 1 in 50 headers and packed them into a large packet, which when full was sent to the Unix kernel. This reduced the rate of sampling packets to the kernel to a rate that could be easily handled (we could do 1 to 1 sampling in bursts with heavy load on the CPU, 1 in 50 continuous with very little load). What we have suggested to our router vendors is the following: in the forwarding path allow an option to take 1 in N packets (where N could be anything but is expected to be in the range of 50-1000) and pack the first H bytes of the packet plus timestamp into a buffer. Send the buffer out a designated interface when the buffer reaches a configured size (usually the MTU of the interface) to some configured addess. The normal use of this would be to send packets out a dedicated ethernet interface to a network management machine within the POP, though the same information could also be sent remotely if it was being used to monitor attacks. The advantage of a network management machine (which we happen to already have for other purposes like SNMP proxy agents for things that don't do SNMP and secure access for things that don't have a means of secure access), is that it has a disk and can quite easily store hours of samples. If an incident is reported and the end site has a traffic archive with address and port number pairs, a snapshot of the prior few hours is taken and the source is identified. If all major providers did this, a SYN attack reported within a few hours would be traceable. ---> Michael Dillon writes:
Now here is something that could be used by sites to protect against SYN flood attacke assuming that they can build a special custom box with enough RAM to buffer the sockets for 30 seconds or more. How high a rate can SYN floods come in at? I've heard of 1,000 per sec which implies that this box needs to hold open 30,000 to 75,000 potential sockets. Is there any problem within IPv4 (seq #'s?) that would make this inherently impossible?
Yep. Increasing the number of packets required for an effective attack makes it more difficult to launch an attack that is not easily traced. ---> Roderick Murchison, Jr. writes:
Ok. say you have a firewall between your network and you Internet connection. If that firewall could detect and *detain* a segment with the SYN option set, then see if the set source IP answers an ICMP echo request, we could effectively determine whether or not the SYN could be dropped at the firewall and not sent through to spam our hosts. If the source responds, release the SYN and let it pass through to the intended host. If it does not, trash the SYN and log the failure.
ICMP is a bad choice (already pointed out on the list). I don't think this is needed if the host implementations are fixed. It would be quite easy to send 60,000 different source addresses, which would make your solution worsen the problem. Better to just have the host send the SYN ACK, let it either blacxkhole (no such destination) or return a RST (a real host was found). ---> Dima Volodin writes:
Apart from the obvious sexism :-) of the message let me point out that there are some other tricks one can play with DNS, e.g. throwing in any addresses for some other's servers in additional records in your DNS reponces. We are hit by this DNS/named deficiency _every freaking day_.
You need to update to a newer version of BIND. This has been fixed. ---> Jim Forster writes:
Again, the rule is "dont accept packets from an interface if there's no route for their source addresses pointing back to the same interface". Note that that route does not have to be the best one -- just that the router gets it from somewhere.
Without discussing it with the right folks here ahead of time, I suspect we could do this at good speed in some, but not all routers, in our product line. The solution I have in mind would not be suitable for some places in the net. We'd put the extra checks in the slow path which Curtis hates so much, and then use our 'flow-switching' cache, which is keyed by src/dest adresses & ports. So packets which fail the source address scrutiny in the slow path aren't put in the flow-switching cache. I can't recall if we cache negatives there, but in any event apparently the attacks involve SYN flows on the order of 100's of PPS, which might go through the slow path OK. BTW, I believe the criterion Vadim suggest is similar to that used in RPF Multicast flooding.
Now the big question: is this useful in routers carrying a default route?
I have to highly commend Jim and Cisco for their quick response and willingness to help address the problem. I must say that this is a typical very helpful response from Cisco. What I am asking for ("we" if there is agreement) is the following. 1. Create an option to look up the source address in the forwarding table. If the forwarding entry for the source address does not point to the interface from which the packet came, drop the packet. Enable this by default. Allow it to be shut off on a per interface basis. This is most important on end system routers so if it cannot be made to run on some routers with hardware assist without overly impacting the forwarding path speed, then that's OK. 2. Create an option to pack 1 in N packet into ... (what I have written above as #3). It would be real handy if this could be done on all routers, including those with hardware assist. This would mean that any Cisco based ISP could very easily trace the source of SYN attack to their own border and there would be no excuse not to. [We've also asked for this feature for a number of other reasons having to do with network capacity planning and statistical studies.] Best Regards to all that are contributing to solving this problem, Curtis
Curtis Villamizar writes:
There have been three areas in which defense of SYN flooding has been suggested (maybe more, three that will help):
1. Make the host less susseptible 2. Filter based on source address on inbound packets from singly homed sites. 3. Improve the ability to trace such attacks back to source
I'll argue that all three of these are needed since none of them alone can completely solve the problem.
Highly agreed.
At issue is whether a stream of TCP SYNs can swamp a host TCP implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs (until TCP port numbers are exhausted).
No hash table was being used for the infant connection queue. This needs to be fixed. An adaptive mechanism to lower the hold time also is needed.
2. Filter based on source address on inbound packets from singly homed sites.
A singly homed site cannot have assymetric routing since there is no ohter path. All that is needed is for the router to look up the source address in the forwarding table and if the route does not point to the interface the packet came in on, dispose of the packet. This would mean that even a singly homed university could not become the source of SYN attacks unless the hacker was willing to use source addresses in the range of the addresses of the university. This would make tracing back to the source very easy.
I want to double-emphasize this -- responsible netizens have to put filtering in at their routers to prevent this sort of thing.
3. Improve the ability to trace such attacks back to source
And here we have a problem -- the providers are not, currently, well equipped to deal with this sort of tracing. I've had a lot of trouble getting action on this sort of thing. Given good characteristics for the packet filters it is pretty doable to find the rogue packets even in extremely heavy traffic flows, but without cooperation to put equipment to do this up it is not possible to take advantage of the technology. Perry
1. Make the host less susseptible [teach engineers to spell?]
Specific code fragments were given for BSD and Solaris. BSDI is already in very good shape with recent kernel patch (that preceeded the attacks and was intended to improve HTTP server performance).
I just put up a page (pretty raw and for techies for now) that gives objects for sun4c and sun4m architectures under SunOS 4.1.x, and gives pointers to what to fix for those with NetBSD/FreeBSD/BSDi and kernel source. It's at http://www.netaxs.com/~freedman/syn/ I'm going to send a blurb about it off to inet-access, though it's possible that corporate customers (non-ISPs) would be attacked and might come to providers for assistance. (If they could even figure out what was going on...) You can point customers at this if they're being SYN-stormed. With these mods, even a Sparc 1+ can resist any realistic net-based SYN attack.
implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs
Hmm. Interesting. I was told that NetBSD did not... Which version of BSD should I look at? A hash table on a static array of PCBs is a much better solution than letting a linked list get to 2000 entries...
(until TCP port numbers are exhausted). At a 75 second hold time,
For production I set this to 15 seconds, though we didn't hear any complaints when I ran our web server at 7 seconds for a few days.
this is well under 1kpps (rapid reuse of port numbers has other problems). Some advantage is gained by the fact that SYN ACKs will be sent and legitimate hosts will respond with RST, clearing the port number for reuse.
There was also a suggestion of a server to prescreen TCP SYNs. I don't think that is needed.
If your customer has Macs or NT boxes, it might be. A box that sits there and sends RSTs for all packets it believes to be part of a SYN storm would help Macs and other platforms for which we don't have kernel source :)
2. Filter based on source address on inbound packets from singly homed sites.
A singly homed site cannot have assymetric routing since there is no ohter path. All that is needed is for the router to look up the source address in the forwarding table and if the route does not point to the interface the packet came in on, dispose of the packet. This would mean that even a singly homed university could not become the source of SYN attacks unless the hacker was willing to use source addresses in the range of the addresses of the university. This would make tracing back to the source very easy.
This would be a very cool router feature but can be done fairly well with outbound garbage filters.
What I am asking for ("we" if there is agreement) is the following.
1. Create an option to look up the source address in the forwarding table. If the forwarding entry for the source address does not point to the interface from which the packet came, drop the packet. Enable this by default. Allow it to be shut off on a per interface basis. This is most important on end system routers so if it cannot be made to run on some routers with hardware assist without overly impacting the forwarding path speed, then that's OK.
Yes.
2. Create an option to pack 1 in N packet into ... (what I have written above as #3). It would be real handy if this could be done on all routers, including those with hardware assist. This would mean that any Cisco based ISP could very easily trace the source of SYN attack to their own border and there would be no excuse not to. [We've also asked for this feature for a number of other reasons having to do with network capacity planning and statistical studies.]
This sounds good as well.
Best Regards to all that are contributing to solving this problem,
Curtis
Avi
In message <199609161637.MAA20184@netaxs.com>, Avi Freedman writes:
implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs
Hmm. Interesting. I was told that NetBSD did not... Which version of BSD should I look at? A hash table on a static array of PCBs is a much better solution than letting a linked list get to 2000 entries...
Oops. That's in a BSDI patch (PATCH K210-019) but I'm not sure about FreeBSD or NetBSD distributions since I don't have one handy. Curtis ps- (My 6 year old has a FreeBSD system, but its 2.0.5. Got to get him to upgrade. :)
In message <199609161637.MAA20184@netaxs.com>, Avi Freedman writes:
implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs
Hmm. Interesting. I was told that NetBSD did not... Which version of BSD should I look at? A hash table on a static array of PCBs is a much better solution than letting a linked list get to 2000 entries...
Oops. That's in a BSDI patch (PATCH K210-019) but I'm not sure about FreeBSD or NetBSD distributions since I don't have one handy.
I'm not sure about prior releases, but FreeBSD 2.1.0 and above do hash PCB's, with a default hash size of 128. The constant name is TCBHASHSIZE in netinet/tcp_subr.c. To raise it, just add options "TCBHASHSIZE=2048" # More capable TCB hash ... and if you feel the need, add options "UDBHASHSIZE=1024" # More capable UDB hash to your kern config file.
Curtis
ps- (My 6 year old has a FreeBSD system, but its 2.0.5. Got to get him to upgrade. :)
Version 2.1.5-Release is now available. ;-) Cheers, -Matt
Curtis Villamizar wrote:
In message <199609161637.MAA20184@netaxs.com>, Avi Freedman writes:
implementation. This is a denial of service exposure that has gone unaddressed in host implementations until recently. BSD now uses a hash table on the TCP PCBs (protocol control blocks in the kernel) and with change of removal of the check can support close to 64K-2000 PCBs
Hmm. Interesting. I was told that NetBSD did not... Which version of BSD should I look at? A hash table on a static array of PCBs is a much better solution than letting a linked list get to 2000 entries...
Oops. That's in a BSDI patch (PATCH K210-019) but I'm not sure about FreeBSD or NetBSD distributions since I don't have one handy.
The SYN_RCVD bug has been fixed in FreeBSD source. i should know, i wrote the patch. as a result, the attacker has to sink the machine in less than 75 seconds, else it begins to free resources. before the patch the attacker had ~11 minutes to do the deed. (would have been 2 hours but for retransmission of the SYN-ACK packet by the target) the bug is dicsussed in detail on page 191 of tcp/ip illustrated by rick stevens. we have not yet moved to a hask table. soon. our SO_MAXCONN is 128, rather than the common 5. jmb -- Jonathan M. Bresler FreeBSD Postmaster jmb@FreeBSD.ORG FreeBSD--4.4BSD Unix for PC clones, source included. http://www.freebsd.org/ PGP 2.6.2 Fingerprint: 31 57 41 56 06 C1 40 13 C5 1C E3 E5 DC 62 0E FB
Curtis
ps- (My 6 year old has a FreeBSD system, but its 2.0.5. Got to get him to upgrade. :)
darn tooting! ;)
Jonathan M. Bresler wrote:
we have not yet moved to a hask table. soon.
i stand corrected. <blush> Matt Bush <xomox@boris.eden.com> is correct. we are hashing control blocks. (hmm....when did that slip in there ;) jmb -- Jonathan M. Bresler FreeBSD Postmaster jmb@FreeBSD.ORG FreeBSD--4.4BSD Unix for PC clones, source included. http://www.freebsd.org/ PGP 2.6.2 Fingerprint: 31 57 41 56 06 C1 40 13 C5 1C E3 E5 DC 62 0E FB
participants (5)
-
Avi Freedman
-
Curtis Villamizar
-
Jonathan M. Bresler
-
Matt Bush
-
Perry E. Metzger