Best practices for abuse@ mailbox and network abuse complaint handling?
Can anybody point me at best practices for monitoring and responding to abuse complaints, and good solutions for accepting complaints about network abuse? Any recommended outsourced services for processing abuse complaints? My interest in this is to more effectively respond to complaints about "bad" network traffic and abuse originating from IP addresses allocated to my employer and to the not-for-profit ISP I help run. (Two similar needs, two very different budgets). My employer has multiple Internet-connected networks, several IP allocations, and several hundred active domains. Currently abuse@* is sporadically monitored by a Messaging team, and any complaints which seem relevant to the Network or Security groups are forwarded to the appropriate internal contact. This is inefficient and untimely. Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99% are email abuse complaints from customers who subscribed to company-run mailing lists and then forgot about it (I've worked hard to educate management on responsible mass mailing). But every once in a while there is a legitimate network-related "incident", and my team does need to see those messages in a timely manner. How do other network operators address the need for timely notification of network abuse? Some people are clueful enough to pull up the ARIN records and contact us by phone, but I don't want to depend on the victim of an attack sourced from our network being bright and persistent. Thanks, Kevin Kadow
My experience is that there's no substitute for a human abuse administrator. You can't manage your abuse queue with a script; not even a really fancy script; not even if it's so fancy that it's called a "Software Suite." You need a human (with clue about things like SMTP and email headers) to be reading the abuse mailbox so that they can recognize and deal with the complaints that represent genuine issues. For a small number of complaints this can be a small part of someone's job; for a larger number you will need one or more people doing abuse full-time. Many aspects of the abuse-handling process can be automated by a savvy abuse admin, but the abuse admin cannot be eliminated if you want to preserve your ability to appropriately respond to network incidents in a reasonable time. To see what happens when you eliminate the humans from your abuse handling, try sending an abuse complaint to yahoo or hotmail. Outsourcing could theoretically work, but the "outside" abuse administrator would need significant access to your network to track down and deal with issues. A powerless abuse admin with no ability to fix the issues he finds would be pretty useless. I haven't seen such a service. There are email management services like Postini but they mostly just filter incoming email for spam and virii. Here's a list of email abuse related best-practices; some of these are great; some are total crap (and some I didn't look at): http://spamcon.org/directories/best-practices.shtml The bestprac.org stuff looks pretty good; this appears to be relevant: http://www.bestprac.org/principles/isp.htm K K wrote:
Can anybody point me at best practices for monitoring and responding to abuse complaints, and good solutions for accepting complaints about network abuse? Any recommended outsourced services for processing abuse complaints?
The issue I see with most of the options (abuse.net, spamcop, etc) is they're focused on the spam problem, while my department is made up of network operations, information security, and CERT, anything to do with web servers, domains, and SMTP is handled by a different business unit in another state entirely. While 99.99% of our abuse@ mail is either spam or complaints about spoofed spam forging our domains as the source and has nothing to do with network operations, about once a month something truly network related will come into that mailbox, and my team won't be alerted to these events in a timely manner. Only fix I can see right now is for us to make it part of our daily workload to troll the abuse@ mailbox on the off chance that something in there is relevant to network operations/security/CERT. Is this what other NANOs do? The clueful victims will look up our ASN/ARIN records and eventually make the right phone call -- or report the problem to law enforcement, who definitely know how to reach us ;) I'm hoping to find either a better and widely accepted way to handle non-spam-related network abuse complaints (hacking, DoS, etc), or at least best practices for triage on the huge volume of mail that comes into abuse@, procedures such that the rare legitimate complaint about non-spam network abuse can be routed to my team in a timely manner. Thanks, Kevin
K K wrote: [..]
I'm hoping to find either a better and widely accepted way to handle non-spam-related network abuse complaints (hacking, DoS, etc), or at least best practices for triage on the huge volume of mail that comes into abuse@, procedures such that the rare legitimate complaint about non-spam network abuse can be routed to my team in a timely manner.
whois is the right one. But IMHO the ARIN whois is a bit limited and also odd, but that might be because I am used to seeing a different kind of data ;) In RIPE db we have a nice IRT (Incident Response Team) object which is meant for this, see amongst others: http://www.ripe.net/info/ncc/presentations/irt-tfcsirt6/sld001.html http://www.ripe.net/db/support/security/irt/irt-h2.html Next to that there is the 'abuse-mailbox' line which can be inserted with most objects, similarly to irt. These will at least allow your users to find you. Some of the tools out there that auto-spam abuse@ when they get a silly portscan use those fields, so at least you will get it at the right address and not at every other single address that is listed in whois. Greets, Jeroen
The issue I see with most of the options (abuse.net, spamcop, etc) is
Hey, leave abuse.net out of this, please. It's just a database of contact addresses. You might want to take a look at Abacus from word-to-the-wise.com. It's a ticketing package specifically designed for abuse desks. You still have to run your own abuse desk, but it does a lot of the grunt work for you. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Information Superhighwayman wanna-be, http://www.johnlevine.com, ex-Mayor "More Wiener schnitzel, please", said Tom, revealingly.
On Fri, 12 May 2007, John Levine wrote:
The issue I see with most of the options (abuse.net, spamcop, etc) is
Hey, leave abuse.net out of this, please. It's just a database of contact addresses.
You might want to take a look at Abacus from word-to-the-wise.com. It's a ticketing package specifically designed for abuse desks. You still have to run your own abuse desk, but it does a lot of the grunt work for you.
I think he was asking about reporting abuse and not handling abuse desk complaints. Personally I generally report non-spam complaints to same abuse designated mailbox (it is abuse after all) but also CC data from abuse contacts from ASN whois. -- William Leibzon Elan Networks william@elan.net
On 5/11/07, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
1. Mitigate [port 25 management, walled gardens and such] Cut down on the number of abuse causing issues
I believe we are doing a good job of mitigating abuse events originating from our network, but when it does happen, I want to know about it. We get perhaps one "real" abuse issue a month, plus a bunch of complaints from customers who signed up for the website and the confirmed-opt-in marketing email, and then decided that they were being spammed when they got exactly what they had signed up for. And then there are the people who receive spam where a company owned domain is primitively spoofed as the sender or as a header. Luckily the only time my group is involved in email abuse and spam issues is when an irate customer decides to make threats against the company or company infrastructure in their complaint (actually happened just last week, which is part of why I started this thread). On 5/11/07, william(at)elan.net <william@elan.net> wrote:
On Fri, 12 May 2007, John Levine wrote:
The issue I see with most of the options (abuse.net, spamcop, etc) is
Hey, leave abuse.net out of this, please. It's just a database of contact addresses.
And it does a fine job at being a database of DOMAIN contact addresses, but what abuse.net doesn't do is provide any information on NETWORK contacts, it will only look up names, not IPs -- for those the victim need to be clueful enough to know what an ASN is and how to look up the ASN contact details... I was hoping that there would be someplace like abuse.net where we could register our IPs and ASN, so non-NANOGers could know to contact network-abuse@ when they think our network is attacking them?
I think he was asking about reporting abuse and not handling abuse desk complaints.
I'm asking how to make sure that my team receives the non-spam-related incoming complaints when a remote network operator feels they are "under attack" by our IPs, how to effectively make a separate POC for network-abuse@ without having a human watch the abuse@ mailbox and forward network related issues to network-abuse@ where I can follow up. Right now there isn't much of an abuse desk to handle complaints, and I cannot depend on the messaging group to manually read through all the abuse@ mail, pick out real network incidents, and forward them to the operations/security/CERT team in a timely manner. And don't get me started on the L1 "help" desk. On 5/11/07, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
Separate POCs as far as possible (postmaster for block related issues, abuse for spam related issues, and a block interface like the one we have around - http://spamblock.outblaze.com/ip.add.re.ss), and quick, automated escalations
Aside from making sure ASN whois and ARIN are pointing to a network-abuse@ mailbox, how do I get victims to use separate POCs? Or can anybody point to an outsourced service which will do effective triage on abuse@ complaints, dealing with the forged headers and other simple problems, and forwarding real events and LE requests to the appropriate company POCs?
Personally I generally report non-spam complaints to same abuse designated mailbox (it is abuse after all) but also CC data from abuse contacts from ASN whois.
That's exactly the problem -- non-spam complaints end up going to the same abuse designated mailbox, but outside of NANOG nobody even knows what ASN stands for. Kevin
On May 12, 2007, at 8:57 PM, K K wrote:
On 5/11/07, william(at)elan.net <william@elan.net> wrote:
On Fri, 12 May 2007, John Levine wrote:
The issue I see with most of the options (abuse.net, spamcop, etc) is
Hey, leave abuse.net out of this, please. It's just a database of contact addresses.
And it does a fine job at being a database of DOMAIN contact addresses, but what abuse.net doesn't do is provide any information on NETWORK contacts, it will only look up names, not IPs -- for those the victim need to be clueful enough to know what an ASN is and how to look up the ASN contact details...
I was hoping that there would be someplace like abuse.net where we could register our IPs and ASN, so non-NANOGers could know to contact network-abuse@ when they think our network is attacking them?
Perhaps abuse.net could include links to: http://www.cymru.com/BGP/asnlookup.html http://www.completewhois.com/ http://www.routeviews.org/ etc. The desire seems to be network operations are to remain unaffected by spam sourced by the ASN. Some view spam as a user problem, and not a network management issue. A resent paper published on ASRG on how to operate a black-hole list excluded mention of ASNs. The nature of the Internet however requires more attention to the ASN.
Personally I generally report non-spam complaints to same abuse designated mailbox (it is abuse after all) but also CC data from abuse contacts from ASN whois.
That's exactly the problem -- non-spam complaints end up going to the same abuse designated mailbox, but outside of NANOG nobody even knows what ASN stands for.
There is a reason for that. -Doug
I was hoping that there would be someplace like abuse.net where we could register our IPs and ASN, so non-NANOGers could know to contact network-abuse@ when they think our network is attacking them?
That would be nice, wouldn't it? There's two reasons I don't do that. One is that unlike domains, there's no sensible default for unknown addresses. (Most ASNs have good contact info in the RIRs, but some don't, and a fair number particularly in Asia are so tiny that they really need the reports sent to the next one up.) The other is that I can barely keep up with the updates for domain names, and there's no way I could do IPs without a major rewrite and a bunch of trustworthy volunteers to share the load.
Perhaps abuse.net could include links to:
http://www.cymru.com/BGP/asnlookup.html http://www.completewhois.com/ http://www.routeviews.org/ etc.
Most abuse.net users barely have figured out that the From: line is not always the right place to find the address to complain to. R's, John
At one place I worked at abuse@, webmaster@, hostmaster@, and all other generic addresses where aliases on the level 1 helpdesk mailbox. Since the L1 desk's address was plastered all over the public site, they already had the pleasure of dealing with tons of spam and clueless users. In the event that someone had a legit network complaint, they would open a ticket and forward to the appropriate group. The only down side was the additional training. The most time consuming component was we needed to train the L1 staff how to read email headers and determine if the originated from our network. Hope that helps, Adam Stasiniewicz ________________________________ From: owner-nanog@merit.edu on behalf of K K Sent: Fri 5/11/2007 5:10 PM To: nanog@merit.edu Subject: Re: Best practices for abuse@ mailbox and network abuse complaint handling? The issue I see with most of the options (abuse.net, spamcop, etc) is they're focused on the spam problem, while my department is made up of network operations, information security, and CERT, anything to do with web servers, domains, and SMTP is handled by a different business unit in another state entirely. While 99.99% of our abuse@ mail is either spam or complaints about spoofed spam forging our domains as the source and has nothing to do with network operations, about once a month something truly network related will come into that mailbox, and my team won't be alerted to these events in a timely manner. Only fix I can see right now is for us to make it part of our daily workload to troll the abuse@ mailbox on the off chance that something in there is relevant to network operations/security/CERT. Is this what other NANOs do? The clueful victims will look up our ASN/ARIN records and eventually make the right phone call -- or report the problem to law enforcement, who definitely know how to reach us ;) I'm hoping to find either a better and widely accepted way to handle non-spam-related network abuse complaints (hacking, DoS, etc), or at least best practices for triage on the huge volume of mail that comes into abuse@, procedures such that the rare legitimate complaint about non-spam network abuse can be routed to my team in a timely manner. Thanks, Kevin
On 5/11/07, K K <kkadow@gmail.com> wrote:
Can anybody point me at best practices for monitoring and responding to abuse complaints, and good solutions for accepting complaints about network abuse? Any recommended outsourced services for processing abuse complaints?
Well, there's a few things 1. Mitigate [port 25 management, walled gardens and such] => Cut down on the number of abuse causing issues 2. Automate => Abacus or other abuse desk optimized ticketing system, as John Levine said => Feedback loops (ARF formatted) from various ISPs => Ditto, automated feeds from Phishtank, Netcraft, your local CERT 3. Spread the load intelligently => Whatever can be handled by tier 1 should be handled by tier 1
Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99%
So? A little filtering should handle a lot of that, procmail even. At least to file the obvious crap into a different folder that can be looked at and blown away
to educate management on responsible mass mailing). But every once in a while there is a legitimate network-related "incident", and my team does need to see those messages in a timely manner.
Separate POCs as far as possible (postmaster for block related issues, abuse for spam related issues, and a block interface like the one we have around - http://spamblock.outblaze.com/ip.add.re.ss), and quick, automated escalations. Ditto tools to automate as much of the "search" stuff as possible. Prioritizing incidents in your queue as well (stuff like LE requests, largescale network incidents etc can usually be spotted from the subject line itself) Takes time to build that kind of setup, but the time spent is well worth it MAAWG's working on an abuse desk best practice doc over the last few meetings, it should be well worth reading when it does come out. --srs -- Suresh Ramasubramanian (ops.lists@gmail.com)
On 5/11/07, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99%
So? A little filtering should handle a lot of that, procmail even. At least to file the obvious crap into a different folder that can be looked at and blown away
I've had good luck using Spamhaus's ZEN blacklist to filter mail inbound to our abuse desk. Tagging and manually reviewing all messages for a month before enabling it as a true filter showed me that the risk of dropping legitimate email on the floor seemed to be nil. My experience: http://tinyurl.com/3a4xdu Regards, Al Iverson -- Al Iverson on Spam and Deliverabilty, see http://www.spamresource.com News, stats, info, and commentary on blacklists: http://www.dnsbl.com My personal website: http://www.aliverson.com -- Chicago, IL, USA
On May 12, 2007, at 10:01 AM, Al Iverson wrote:
On 5/11/07, Suresh Ramasubramanian <ops.lists@gmail.com> wrote:
Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99%
So? A little filtering should handle a lot of that, procmail even. At least to file the obvious crap into a different folder that can be looked at and blown away
I've had good luck using Spamhaus's ZEN blacklist to filter mail inbound to our abuse desk. Tagging and manually reviewing all messages for a month before enabling it as a true filter showed me that the risk of dropping legitimate email on the floor seemed to be nil.
ZEN is great but add greylisting and it gets even better, and throw in FuzzyOCR to take care of the image spam. G
* ops.lists@gmail.com (Suresh Ramasubramanian) [Sat 12 May 2007, 05:25 CEST]:
On 5/11/07, K K <kkadow@gmail.com> wrote:
Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99% So? A little filtering should handle a lot of that, procmail even. At least to file the obvious crap into a different folder that can be looked at and blown away
Difficult, as spam complaints generally include the original spam and thus trigger SpamAssassin (almost) just as hard. Otherwise, looking forward to your 98% effective procmail recipe -- Niels.
At least to file the obvious crap into a different folder that can be looked at and blown away
Difficult, as spam complaints generally include the original spam and thus trigger SpamAssassin (almost) just as hard.
A complaint with copy of the original spam has headers in the body of the message, which most spam doesn't. It's not perfect, but considering what a simple test that is, it works pretty well as part of a first cut. R's, John
On 5/12/07, Niels Bakker <niels=nanog@bakker.net> wrote:
* ops.lists@gmail.com (Suresh Ramasubramanian) [Sat 12 May 2007, 05:25 CEST]:
On 5/11/07, K K <kkadow@gmail.com> wrote:
Probably 98% of the mailbox is from are spammers who've harvested or randomly targeted abuse@ addresses for male enhancement, maybe 1.99% So? A little filtering should handle a lot of that, procmail even. At least to file the obvious crap into a different folder that can be looked at and blown away
Difficult, as spam complaints generally include the original spam and thus trigger SpamAssassin (almost) just as hard.
SpamAssassin isn't the only way to filter spam. It's not bad, but pure content filtering is probably not the way to go for an abuse desk.
Otherwise, looking forward to your 98% effective procmail recipe
Mine didn't even need procmail, just piped inbound mail through a shell script I wrote, with about an 80% success rate, and no false positives. Regards, Al Iverson -- Al Iverson on Spam and Deliverabilty, see http://www.spamresource.com News, stats, info, and commentary on blacklists: http://www.dnsbl.com My personal website: http://www.aliverson.com -- Chicago, IL, USA
On 5/13/07, Niels Bakker <niels=nanog@bakker.net> wrote:
Difficult, as spam complaints generally include the original spam and thus trigger SpamAssassin (almost) just as hard.
Otherwise, looking forward to your 98% effective procmail recipe
Start with something as simple as "to or cc your abuse desk" .. and ask yourself how many times your abuse desk has been bcc'd on email in the past. -- Suresh Ramasubramanian (ops.lists@gmail.com)
participants (11)
-
Al Iverson -
Albert Meyer -
Douglas Otis -
Gerry Boudreaux -
Jeroen Massar -
John Levine -
K K -
Niels Bakker -
Stasiniewicz, Adam -
Suresh Ramasubramanian -
william(at)elan.net