net.ipv4.tcp_keepalive_intvl = 15 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_time = 90 net.ipv4.tcp_fin_timeout = 30
As discussed, those do not affect TCP_TIMEWAIT_LEN. There is a lot of misinformation out there on this subject so please don't just Google for 5 min. and chime in with a "solution" that you haven't verified yourself. We can expand the ephemeral port range to be a full 60K (and we have as a band-aid), but that only delays the issue as use grows. I can verify that changing it via: echo 1025 65535 > /proc/sys/net/ipv4/ip_local_port_range Does work for the full range, as a spot check shows ports as low as 2000 and as high as 64000 being used. While this works fine for the majority of our sites as they average well below that, for a handful peak hours can spike above 1000 connections per second; so we would really like to see something closer to an ability to provide closer to 2000 or 2500 connections a second for the amount of bandwidth being delivered through the unit (full gigabit). But ideally we would find a way to significantly reduce the number of ports being chewed up for outgoing connections. On the incoming side everything just makes use of the server port locally so it's not an issue. Trying to avoid using multiple source addresses for this as it would involve a fairly large configuration change to about 100+ units; each requiring coordination with the end-user, but it is a last resort option. The other issue is that this is all essentially squid, so a drastic re-design of how it handles networking is not ideal either. On Thu, Dec 6, 2012 at 8:25 AM, Kyrian <kyrian@ore.org> wrote:
On 5 Dec 2012, rps@maine.edu wrote:
Where there is no way to change this though /proc
...
Those netfilter connection tracking tunables have nothing to do with the kernel's TCP socket handling.
No, but these do...
net.ipv4.tcp_keepalive_intvl = 15 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_time = 90 net.ipv4.tcp_fin_timeout = 30
I think the OP was wrong, and missed something.
I'm no TCP/IP expert, but IME connections go into TIME_WAIT for a period pertaining to the above tuneables (X number of probes at Y interval until the remote end is declared likely dead and gone), and then go into FIN_WAIT and then IIRC FIN_WAIT2 or some other state like that before they are finally killed off. Those tunables certainly seem to have actually worked in the real world for me, whether they are right "in theory" or not is possibly another matter.
Broadly speaking I agree with the other posters who've suggested adding other IP addresses and opening up the local port range available.
I'm assuming the talk of 30k connections is because the OP's proxy has a 'one in one out' situation going on with connections, and that's why your ~65k pool for connections is halved.
K.
-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net