Outbound mail filtering on large mail / web server farms - just an idea or two that I have
On 11/20/05, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
When you aggregate all .forward email out through a single box, stuff that's slipping through your filters starts to stick out like a sore thumb when you analyze the mail queues on that box, so you can tune your inbound filters better. Quite a useful thing to do, really.
I originally wrote this lot below as boilerplate for large webhosting providers that find themselves with several racks full of pizzabox colos running a web control panel like ensim or cpanel so that the people actually operating the colos may not have too much clue .. and these places are typically riddled with lots and lots of exploitable cgi / php scripts that are broken into and used to send spam using injection / xss etc holes .. Some of the ideas here might well apply to what I was talking about in this thread as well - the two kind of tie in together Comments welcome regards srs ------------------------------------------ First and most important thing to do - Please set your webserver up so that it uses suexec / phpsuexec to make sure perl and php scripts run as the uid/gid of the the user rather than the webserver's uid/gid like nobody or www-data. Never mind if your webhosting software like ensim or plesk inserts headers showing the spammer's uid/gid, path of the calling script etc - serves as a useful backup, and is especially useful when you are searching your logs for cgi scripts that that spammers upload onto your server and then use to proxy their spam from another server through your IP [so your IP shows up in our logs as sending spam, rather than the spammer's actual IP] I'd also strongly suggest something like this if you can manage it 1. Block port 25 outbound on all your webservers. Port 25 inbound as well if you happen to have a mailserver infrastructure for your users thats separate from the webservers they are on, rather than MX'ing your users domains onto the webserver they're hosted on. 2. Set your exim local installs to smarthost through a separate, filtered cluster of servers (spamassassin + bogofilter or something like that - possibly DCC for bulk email detection as well). Smarthosting - see http://www.hserus.net/exim.html For extra points you could do smtp auth on the filtered smarthost as well, to help you jump on issues faster. Set it up so the user's local uid/gid gets used to auth to the remote exim box .. centralized ldap or mysql userdb does the trick for that. That way spammers cant spam out direct through cgis - but peoples normal email and script generated traffic goes out just fine through your filtered gateways. -------------------------------------------
--On November 20, 2005 8:48:08 PM +0530 Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
I originally wrote this lot below as boilerplate for large webhosting providers that find themselves with several racks full of pizzabox colos running a web control panel like ensim or cpanel so that the people actually operating the colos may not have too much clue .. and these places are typically riddled with lots and lots of exploitable cgi / php scripts that are broken into and used to send spam using injection / xss etc holes ..
Some of the ideas here might well apply to what I was talking about in this thread as well - the two kind of tie in together
I've considered a similar setup. Requiring all mgd servers to always use their local mailers, then at the nearby edge, NATing all outbound SMTP port 25 traffic to a set of mail relays setup to do greylisting, rate limiting, and possibly IDENT checks to make (reasonable more) sure that it's the mail server user talking and not some random software. Note that I've done none of it...the idea's a bit insane, but, it would definitely make it easier to spot and treat the problems, the only big black eye here is AOL who would probably rate limit the outbound servers quite often, which they already do to our normal mail systems even when things are going well, again, because of forwards. I'd imagine there's a way I could get just the (AOL) forwarded mail pushed to a separate machine with our current (older version) Postfix setup but I haven't actually looked into it. We use SQL based tables for everything in order to make automation much simpler on our end. I hope this all wasn't too non-operational, it seems relevant to me, so hopefully it's not noise.
On 11/20/05, Michael Loftis <mloftis@wgops.com> wrote:
quite often, which they already do to our normal mail systems even when things are going well, again, because of forwards. I'd imagine there's a way I could get just the (AOL) forwarded mail pushed to a separate machine
The difference is of course that when you separate .forward traffic to a separate IP you tell AOL its a forwarding server. And setup reverse dns + hostname for that box that says something like "dotforward.wgops.com" ... Once you do that you should be reasonably good to go
with our current (older version) Postfix setup but I haven't actually looked into it. We use SQL based tables for everything in order to make automation much simpler on our end.
Implementing these suggestions is much simpler once you do that -- Suresh Ramasubramanian (ops.lists@gmail.com)
--On November 21, 2005 8:55:39 AM +0530 Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
On 11/20/05, Michael Loftis <mloftis@wgops.com> wrote:
quite often, which they already do to our normal mail systems even when things are going well, again, because of forwards. I'd imagine there's a way I could get just the (AOL) forwarded mail pushed to a separate machine
The difference is of course that when you separate .forward traffic to a separate IP you tell AOL its a forwarding server. And setup reverse dns + hostname for that box that says something like "dotforward.wgops.com" ...
Once you do that you should be reasonably good to go
Oh I understand the concept perfectly well. It's just that I can't see through to an implementation easily. The system I'm referring to has no internal way of telling the difference easily between forwarded mail and 'other' mail, it's all passed into the same set of virtual tables and the only difference is local versus remote delivery. I can't classify .forward style traffic out from the regular in/out flows. I'm probably not making a whole lot of sense either right now, lets blame that on low caffeine count. I could pass all aol.com traffic (via transport) to a different box but I can't do that just for forwarded mail because inbound mail and outbound (locally generated/initial submission on port 25/etc) aren't handled seperately at all. I can see how they could be, but I don't see any provisions inside of Postfix 2.0 to handle that without separate instances. Not impossible, just impractical right now. The eventual 'plan' is to do almost exactly that, separate instances to handle/classify mail differently based on where the mail was submitted. I guess I'm really curious as to how others might implement something like this. I'd run three instances (machines if you must view it like that) of the MTA, one inbound the outside, say $world, the other $local, the third (call it $forward if you will) is where $world would send all of it's forwarded/outbound mail to that won't be delivered locally. $local would handle local delivery and external delivery for local machines. I think in Exim though it's cleaner because you can specify special processing for other steps. My system has a bit more complication because of the fact we don't use any filesystem. The mail users are purely virtual to the mail system, only existing as an LDAP entry and as a Cyrus Mailbox.
On 11/21/05, Michael Loftis <mloftis@wgops.com> wrote:
Oh I understand the concept perfectly well. It's just that I can't see through to an implementation easily. The system I'm referring to has no internal way of telling the difference easily between forwarded mail and 'other' mail, it's all passed into the same set of virtual tables and the only difference is local versus remote delivery. I can't classify .forward
Separate your MXs from your outbound servers And have local traffic rewritten to an internal domain (say wgops.com.internal) just for local delivery .. all other outbound traffic on your MXs can be assumed to be either * Bounces [route em out of a separate host as well - tracking those helps block a ton of spam] * Forwarded emails Your outbounds of course go out via a separate set of boxes. Multiple machines. Or multiple postfix instances on the same box. Whatever --srs
On 11/20/05, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
I originally wrote this lot below as boilerplate for large webhosting providers that find themselves with several racks full of pizzabox colos running a web control panel like ensim or cpanel so that the
Damn. The reason I wrote that is coming back to me, VERY PAINFULLY :( There's someone out there sending out spam advertising "hoodia" (diet pills made from a cactus of some kind) in huge quantities through hacked contact forms. My staffers are getting about two or three dozen tickets from various people who are on servers blocked since the last 2..3 days because of this kind of spam. And right now the spam levels for this lot are spiking sky high If any webhosts are listening - please sweep your network for this stuff pronto. And please implement what I suggested. srs
On Sun, 20 Nov 2005, Suresh Ramasubramanian wrote:
For extra points you could do smtp auth on the filtered smarthost as well, to help you jump on issues faster. Set it up so the user's local uid/gid gets used to auth to the remote exim box .. centralized ldap or mysql userdb does the trick for that.
That way spammers cant spam out direct through cgis - but peoples normal email and script generated traffic goes out just fine through your filtered gateways.
Our most common successful spam incidents involve exploited vulnerabilities in web forms. It's difficult for spammers to get email out of our network, because we block port 25, our MXs only accept incoming email, and our outgoing relays have names that spammers can't be bothered to find out. However, web forms come preconfigured, so if the spammer can exploit it they don't have to know anything about our email setup. Secure SMTP between the web server and the outgoing relay won't help. Recent versions of Exim have a rate-limiting feature which I am using to mitigate this vulnerability - though it's hard to deploy without disrupting legitimate users. Tony. -- f.a.n.finch <dot@dotat.at> http://dotat.at/ BISCAY: WEST 5 OR 6 BECOMING VARIABLE 3 OR 4. SHOWERS AT FIRST. MODERATE OR GOOD.
participants (3)
-
Michael Loftis -
Suresh Ramasubramanian -
Tony Finch