Unless, I missed the posts about this,.. I just (and still am experiencing) a distributed spam attack. I have a small machine at a colo. Today I check my inbox and there are 2000+ extra messages to a domain I have 'zbot.net'. The messages are doing 4 letter combinations for the recipient. (abde, abdf, etc.) The from's are all mybestplacetoshop@ainet.us I check my qmail queue -> its at 13405 messages. I shut down mail and remove the email from the queue. Here is the kicker. I check where these are coming from, they are from all over the place. I check for IP address spoofing... not happening. No IP options or TCP options. This came from like about 300 different networks, and yes I don't accept source routing (IP Options). Anyways, it happened to my machine, I stopped accepting mail to that domain from qmail-smtpd, so I'm back to normal. If anyone want's a tcpdump of the connection attempts or the emails. Let me know. Dru Nelson San Carlos, California
dru-nanog@redwoodsoft.com wrote:
Unless, I missed the posts about this,.. I just (and still am experiencing) a distributed spam attack.
We get these almost continually.... it is incredibly depressing to look at the logs. Backup-only MX here see upwards of 10K messages on bad days, mostly attacks of that type. Some of the domains chosen for the attack are ridiculous (are 4 valid addresses really worth that effort?). I have come to the conclusion that distributed dictionary attacks will eventually get the goods. Sure you can reject by pattern match on ainet.us for this case, but that's not going to help when someone with a large network of spambots sets up a job that: 1) uses completely random from addresses, subject lines and message content 2) uses an attack algorithm to distribute the load so you only see any given source IP every other day I suspect that this type of attack is currently ongoing, underneath the obvious noise of the cruder tools. The only solution I see for the service provider is to recommend their subscribers choose long, complicated usernames not likely to be found in a dictionary. If anyone has better thoughts as to defense for the above scenario, I would love to hear it. I used to believe that running a catchall alias was an effective deterrent until the b*st*rds started sending complete spams and not just RCPT TO. The only alternative I see is a blacklist populated by some type of distributed detection system... if enough of us under attack contributed 550 unknown user logs, there should be an easily definable threshold for human error. Mike -- With all the spam I get, maybe mlewinski isn't such a bad idea for username after all.
Unless, I missed the posts about this,.. I just (and still am experiencing) a distributed spam attack.
We get these almost continually....
Yep... same here.
it is incredibly depressing to look at the logs. Backup-only MX here see upwards of 10K messages on bad days, mostly attacks of that type.
yep same here... before we ducked for cover (see below) I could grep 800 megs of just "REJECTED" out of our maillog file (two per day). Very depressing. To make it even more depressing we were only getting harvested on about two dozen of the several thousand domains we run MX for.
Some of the domains chosen for the attack are ridiculous (are 4 valid addresses really worth that effort?).
Well, they don't know that until the dictionary the domain do they? <Sigh.>
I have come to the conclusion that distributed dictionary attacks will eventually get the goods. Sure you can reject by pattern match on ainet.us for this case, but that's not going to help when someone with a large network of spambots sets up a job that:
1) uses completely random from addresses, subject lines and message content
Correct. That is exactly what we have seen.
2) uses an attack algorithm to distribute the load so you only see any given source IP every other day
Yep. My list of "attacking IP's" was several thousand deep before I gave up.
I suspect that this type of attack is currently ongoing, underneath the obvious noise of the cruder tools.
yes. We started seeing it (moderate volume) in July of this year. By August it was equal to "regular" client traffic. By early-October is was kneecapping our mailservers. Managing the "ignore" list started to become a full-time job, so we surrendered and started using an external blocking service. (see below) Before that we tried filtering at the router(s) and maintaining "ignore" lists on the servers, but it broke all sorts of things you *want* to have happen with secondary mail servers, especially the ones we have off-site.
The only solution I see for the service provider is to recommend their subscribers choose long, complicated usernames not likely to be found in a dictionary.
That doesn't do *anything* to stop the attack, it just hides the user from being harvested (easily.) It managed to find a couple of my weird addresses though, so while you can run, you can't hide forever.
If anyone has better thoughts as to defense for the above scenario, I would love to hear it.
We have been offering Postini <http://www.postini.com> "spam & virus filtering" to our clients since May. They offer a service that detects, and blocks/ignores the originating harvest spambots. They call it "ActiveEMS"... we tried it on our own domain (one of the first targeted) and we saw it drop like a rock. So we made it "mandatory" for our clients now... they can opt-out of the filtering, but we still hide our mailservers behind theirs, even if our client opts out. That way, the client's *domain* stays protected, but they can read all the spams their hearts desire. It *still* does some wonky stuff with secondaries, so I might have to buy (grumble) their services as secondary MX spooling.
I used to believe that running a catchall alias was an effective deterrent until the b*st*rds started sending complete spams and not just RCPT TO.
In fact, in this scenario the catch-all is like pouring gasoline on the fire without some giant water tank on the roof to... oh, wait... wrong thread. Sorry. The only clients we haven't moved to Postini are those with "catch-all" addresses. Those break under Postini... well, they don't really "break" accept the bank, as clients get charged per-address. We are spreading clues as much as we can to discourage catch-alls. I hope to have all but the completely entrenched converted by year-end. Then we just have to wait until they get harvested... then they'll change their mind. We have one client, who owns close to 50 domains... all with a catch-all going to his *one* address. He went from getting maybe 30 spams a week to several hundred a day... just because a single domain was harvested by these attacks.
The only alternative I see is a blacklist populated by some type of distributed detection system... if enough of us under attack contributed 550 unknown user logs, there should be an easily definable threshold for human error.
Interesting alternative... the hard part is making it work. How does it face the spambots, but still not refuse actual legit mail traffic coming into your primary MX? What is the threshold where it recognizes an attack from the normal traffic and start feeding the BS to the Bots? I have about 4 gigs of 550 logs to contribute.
Mike -- With all the spam I get, maybe mlewinski isn't such a bad idea for username after all.
heh. Totally OT, but a nice bonus with Postini was re-acquainting myself with somebody I knew from a Network Manager's user group (ANMA) I was in back in the early 90's. The salesdroid at Postini introduced me to my "install engineer" and it was a guy who was the pres of the Bay Area chapter... when I was the pres of the Northwest one. We both had a good laugh. Small world. -- Chuck Goolsbee V.P. Technical Operations _________________________________________________________________ digital.forest Phone: +1-877-720-0483, x2001 where Internet solutions grow Int'l: +1-425-483-0483 19515 North Creek Parkway Fax: +1-425-482-6871 Suite 208 http://www.forest.net Bothell, WA 98011 email: cg@forest.net
We just recently started using GatewayDefender's Business service. So far, I've only received about 1 or 2 spam a day -- down from nearly 40-60. Not bad in my estimation. (http://www.gatewaydefender.com) -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu]On Behalf Of chuck goolsbee Sent: Wednesday, November 20, 2002 4:16 AM To: nanog@merit.edu Subject: Re: Weird distributed spam attack
Unless, I missed the posts about this,.. I just (and still am experiencing) a distributed spam attack.
We get these almost continually....
Yep... same here.
it is incredibly depressing to look at the logs. Backup-only MX here see upwards of 10K messages on bad days, mostly attacks of that type.
yep same here... before we ducked for cover (see below) I could grep 800 megs of just "REJECTED" out of our maillog file (two per day). Very depressing. To make it even more depressing we were only getting harvested on about two dozen of the several thousand domains we run MX for.
Some of the domains chosen for the attack are ridiculous (are 4 valid addresses really worth that effort?).
Well, they don't know that until the dictionary the domain do they? <Sigh.>
I have come to the conclusion that distributed dictionary attacks will eventually get the goods. Sure you can reject by pattern match on ainet.us for this case, but that's not going to help when someone with a large network of spambots sets up a job that:
1) uses completely random from addresses, subject lines and message content
Correct. That is exactly what we have seen.
2) uses an attack algorithm to distribute the load so you only see any given source IP every other day
Yep. My list of "attacking IP's" was several thousand deep before I gave up.
I suspect that this type of attack is currently ongoing, underneath the obvious noise of the cruder tools.
yes. We started seeing it (moderate volume) in July of this year. By August it was equal to "regular" client traffic. By early-October is was kneecapping our mailservers. Managing the "ignore" list started to become a full-time job, so we surrendered and started using an external blocking service. (see below) Before that we tried filtering at the router(s) and maintaining "ignore" lists on the servers, but it broke all sorts of things you *want* to have happen with secondary mail servers, especially the ones we have off-site.
The only solution I see for the service provider is to recommend their subscribers choose long, complicated usernames not likely to be found in a dictionary.
That doesn't do *anything* to stop the attack, it just hides the user from being harvested (easily.) It managed to find a couple of my weird addresses though, so while you can run, you can't hide forever.
If anyone has better thoughts as to defense for the above scenario, I would love to hear it.
We have been offering Postini <http://www.postini.com> "spam & virus filtering" to our clients since May. They offer a service that detects, and blocks/ignores the originating harvest spambots. They call it "ActiveEMS"... we tried it on our own domain (one of the first targeted) and we saw it drop like a rock. So we made it "mandatory" for our clients now... they can opt-out of the filtering, but we still hide our mailservers behind theirs, even if our client opts out. That way, the client's *domain* stays protected, but they can read all the spams their hearts desire. It *still* does some wonky stuff with secondaries, so I might have to buy (grumble) their services as secondary MX spooling.
I used to believe that running a catchall alias was an effective deterrent until the b*st*rds started sending complete spams and not just RCPT TO.
In fact, in this scenario the catch-all is like pouring gasoline on the fire without some giant water tank on the roof to... oh, wait... wrong thread. Sorry. The only clients we haven't moved to Postini are those with "catch-all" addresses. Those break under Postini... well, they don't really "break" accept the bank, as clients get charged per-address. We are spreading clues as much as we can to discourage catch-alls. I hope to have all but the completely entrenched converted by year-end. Then we just have to wait until they get harvested... then they'll change their mind. We have one client, who owns close to 50 domains... all with a catch-all going to his *one* address. He went from getting maybe 30 spams a week to several hundred a day... just because a single domain was harvested by these attacks.
The only alternative I see is a blacklist populated by some type of distributed detection system... if enough of us under attack contributed 550 unknown user logs, there should be an easily definable threshold for human error.
Interesting alternative... the hard part is making it work. How does it face the spambots, but still not refuse actual legit mail traffic coming into your primary MX? What is the threshold where it recognizes an attack from the normal traffic and start feeding the BS to the Bots? I have about 4 gigs of 550 logs to contribute.
Mike -- With all the spam I get, maybe mlewinski isn't such a bad idea for username after all.
heh. Totally OT, but a nice bonus with Postini was re-acquainting myself with somebody I knew from a Network Manager's user group (ANMA) I was in back in the early 90's. The salesdroid at Postini introduced me to my "install engineer" and it was a guy who was the pres of the Bay Area chapter... when I was the pres of the Northwest one. We both had a good laugh. Small world. -- Chuck Goolsbee V.P. Technical Operations _________________________________________________________________ digital.forest Phone: +1-877-720-0483, x2001 where Internet solutions grow Int'l: +1-425-483-0483 19515 North Creek Parkway Fax: +1-425-482-6871 Suite 208 http://www.forest.net Bothell, WA 98011 email: cg@forest.net
It *still* does some wonky stuff with secondaries, so I might have to buy (grumble) their services as secondary MX spooling.
We have started distribiting the list of valid addresses to secondary MX servers to reduce the store and forward load of dictionary attacks on those servers. Using a fast response RBL helps, but whitelisting is a chore. (http://openrbl.org pick one)
I used to believe that running a catchall alias was an effective deterrent until the b*st*rds started sending complete spams and not just RCPT TO.
We have never run catchall, but I am thinking about funneling LUser into pattern matching (spamassassin, or similar) and then used to build a time limited local ipfw or ipfirewall table. We have enough horsepower to filter at the routers, but prefer to let the routers route, and let the MX boxes filter.
In fact, in this scenario the catch-all is like pouring gasoline on the fire without some giant water tank on the roof to... oh, wait... wrong thread. Sorry.
We tried water cooling, but it quit working when they patched the roof. ;-} -bryan bradsby Texas State Government Net NOC: 512-475-2432 877-472-4848 -- "The most likely way for the world to be destroyed, most experts agree, is by accident. That's where we come in. We're computer professionals. We cause accidents." -- Nathaniel Borenstein co-author of MIME.
2) uses an attack algorithm to distribute the load so you only see any given source IP every other day Yep. My list of "attacking IP's" was several thousand deep before I gave up.
Back when I used to analyze dialup spammers (well over a year ago) I felt that a large part of the spam problem could be traced back to just a handful of very prolific abusers. Some were "professionals", with 4 to 8 phone lines at home, others seemed to be mixing their home and work phone access. One(?) person laundered all his calls through 800-number accessible switchboards (hotels and resorts). I still think pursuing just these heavy hitters could pay off big for everyone. For a short time at least. If you want to try some simple analysis on your own: - once you have a spammer's userid and caller ID, pull every record for that userid and caller ID. This will give you several new userids and phone numbers. Pull all of those too, and keep repeating until nothing new pops out. Search all of your logs, for as far back as possible. Watch out for mixed case and trailing spaces. - every few iterations, use a round of reverse number lookups at anywho.com, and the address and name lookups at infospace.com to expand your phone numbers. - if any of the numbers trace back to businesses, knock off (wild card) the last one or two digits of the phone numbers and search again. - Google any distinctive (personal?) userids. (obNanog: I doubt many other groups' members have access to the needed records)
participants (6)
-
Bryan Bradsby -
chuck goolsbee -
dru-nanog＠redwoodsoft.com -
Jacob M Wilkens -
Mike Lewinski -
sjj＠pobox.com