On Fri, 5 Sep 1997, Phil Howard wrote:

One copy of this same spam (but who knows if it is or is not really the same spammer) I got appeared to be from PSI. It came from a PSI connection and used Earthlink as a mail hop. I complained to abuse@psi.net and they sent back a reply claiming the mail came from Earthlink. Well, literally I did get it from Earthlink, but it originated from PSI's IP address, unless Earthlink faked the IP (but then why would they leave their own address on it).

That's why I tend to believe a lot of ISPs ... and more often the BIGGER ones than the smaller ones ... don't know what is going on.

I had two very similar incidents of PSI not knowing what was going on. I've gotten a lot of spam that originated from PSI dialup users but using Earthlink as a mail relay; for example, this one: Return-Path: mail.earthlink.net@italy.it.earthlink.net Return-Path: <mail.earthlink.net@italy.it.earthlink.net> Received: from hops.cs.jhu.edu [this is where I received the mail] by blaze.cs.jhu.edu with SMTP; Wed, 9 Apr 1997 04:31:17 GMT Sender: mail.earthlink.net@italy.it.earthlink.net Received: from italy.it.earthlink.net (italy-c.it.earthlink.net [204.250.46.18]) by hops.cs.jhu.edu (8.6.12/8.6.9) with ESMTP id AAA05428 for <jelson@poincare.cs.jhu.edu>; Wed, 9 Apr 1997 00:31:15 -0400 Received: from LOCALNAME (ip55.rocky-mount.nc.pub-ip.psi.net [38.30.63.55]) by italy.it.earthlink.net (8.8.5/8.8.5) with SMTP id MAA14529; Tue, 8 Apr 1997 12:15:13 -0700 (PDT) Message-Id: <199704081915.MAA14529@italy.it.earthlink.net> Comments: Authenticated sender is <barnhillj@mail.earthlink.net> In the above case, someone dialed into PSI (ip55.rocky-mount...) and relayed mail through Earthlink. I complained to PSInet and they told me "Sorry, nothing we can do, this is coming from Earthlink." More recently, though, something much more insidious started to happen: spammers have started forging Received: lines in the headers to misdirect attempts at tracing the source of the mail! Here's one beautiful example of a spam header I received (my mailhost here was blaze.cs.jhu.edu): From: mailman@domaol.net Received: from fs.IConNet.NET by blaze.cs.jhu.edu with ESMTP; Wed, 9 Apr 1997 07:54:13 GMT Sender: mailman@domaol.net Received: from 199.173.160.250 (ip19.new-haven.ct.pub-ip.psi.net [38.11.102.19]) by fs.IConNet.NET (8.8.5/8.8.5) with SMTP id DAA12207; Wed, 9 Apr 1997 03:54:27 -0400 (EDT) Received: from mailhost.bethere.net(alt2.bethere.net(214.756.86.9)) by bethere.net (8.8.5/8.6.5) with SMTP id GAA04732 for <friend@public.com>; Wed, 09 Apr 1997 02:52:20 -0600 (EST) To: friend@public.com Message-ID: <37474743565665.JDL9087@bethere.net> At first glance, it would appear the above spam originated from bethere.net. When I looked more closely, though, I realized that tracing the Received: lines up from the bottom shows the mail going from alt2.bethere.net to bethere.net, then suddenly jumping from a dialup in PSInet to fs.IConNet.NET. How did it get from bethere.net to PSInet?? The answer, of course, is that the mail really originated from a PSInet dialup, using IConNet.NET as a spam relay; the bottom Received: line is an utter forgery, presuambly added by the spam-mailing software. In fact, it's not even a very good forgery, because the supposed IP address of alt2.bethere.net is invalid (the 2nd octet is 756). When I [again] wrote to PSInet to complain about spam coming from their users, I was told I should complain to bethere.net instead -- a domain that does not even exist! As a final, even more depressing footnote to this already sad story: a few days after I saw this new trend of getting spam with forged Received: lines, I actually got an advertisement for spamming software that prominently listed one of its features as being that it could add forged sendmail-like headers in order to misdirect investigations! (To add insult to injury, I received 8 copies of this ad via the wonders of spam.) -Jeremy -------------------------------------------------------------------------- NOTE: This message expresses my personal views and should not be taken to represent the views or policies of the United States Government or NIH. Jeremy Elson Division of Computer Research and Technology National Institutes of Health Bethesda, MD Email: jeremy.elson@nih.gov Phone: (301) 402-0349

Russ Haynal wrote...

At 04:35 PM 9/5/97 -0400, Jeremy Elson wrote:

...

I had two very similar incidents of PSI not knowing what was going on. I've gotten a lot of spam that originated from PSI dialup users but using Earthlink as a mail relay; for example, this one:

Recognize that Earthlink is a "national" provider by virtue of the fact that its customers are allowed to connect through PSI and UUNEt POPs (and other ISP's POPs?). Just last week I established an Earthlink dial-up acount for one of my relatives. Many of the Earthlink POP phone numbers turned out to be Phone numbers belonging to PSI, UUNET. It was interesting that the PPP Dial-up logon user ID was of the form: "ELN/userid" The "ELN/" in front of the userid stands for Earthlink Network, so that PSI/UUNET knows to which ISP to route the particular dial-up user.

I would suggest that your particular Spammer IS an Earthlink User, (who happens to dial-in through a PSI POP) In this instance, I guess PSI would have to be considered "just an innocent carrier" like the local phone company that also helps the Spammer reach his ISP (Earthlink)

PSI is not exactly an innocent carrier here. There are several reasons. The reverse DNS identifies the port as PSI. If the port IP address is exclusive to a reseller, then it should be delegated to the reseller. If the address is overloaded and could be different resellers at different times, then PSI forces themselves to be in the loop to identify who's customer was using it at the time. They better be providing to the customer (e.g. Earthlink) the list of which users connect when for tracking purposes. PSI can choose who they do business with. If PSI was reselling to Cyberpromo then I'd have no qualms about blocking the entirety of PSI. Earthlink may well be elevated to near Cyberpromo if all of what I hear about continues. In much the same way as any backbone is called on to drop a smaller ISP that is regularly spamming, the presence facility provider can drop a presence reseller if that reseller is causing them problems. So as long as PSI is in the loop (and they are for overloaded ports or incorrectly delegated reverse DNS) then it's a problem for PSI and PSI is the one that needs to deal with at least some aspect of it (many solutions depending on their business preferences).

[One alternative thought, (and it's a messy one)... Most ISP's can restrict their mail gateways to accept their customers only, but I wonder if Earthlink would be able to configure its mail server to prohibit customers of PSI's and UUNEt's Dial-up services from using Earthlink's mail server.]

If the addresses are overloaded, then they can't. If so, PSI (and UUNET) would be introducing more problems. PSI is still in the loop. PSI still needs to take some action somewhere. It's up to them. -- Phil Howard KA9WGN +-------------------------------------------------------+ Linux Consultant | Linux installation, configuration, administration, | Milepost Services | monitoring, maintenance, and diagnostic services. | phil at milepost.com +-------------------------------------------------------+

turned out to be Phone numbers belonging to PSI, UUNET. It was interesting that the PPP Dial-up logon user ID was of the form: "ELN/userid" The "ELN/" in front of the userid stands for Earthlink Network, so that PSI/UUNET knows to which ISP to route the particular dial-up user.

For what it's worth, the prefix (.*)/(.*) $1 is not relevant to the L3 routing of traffic (ie no P2TP tunnel or such) but rather is relevant to how the login is authenticated. The prefix tells the network access server (or the authentication infrastructure) to which authentication servers to send the login authentication request. -alan

You should also take a look at smtpd from Obtuse (ftp://ftp.obtuse.com/pub/smtpd/beta) It allows you to block relaying in many different ways some of which you dont see in sendmail filters. For instance, you can refuse relaying for IP X because ip X's authorative name servers dont include Y. Its also flexible in deploying a single file across all your mail servers which takes care of relaying and spam. On Fri, 5 Sep 1997, Rod Nayfield wrote:

At 04:35 PM 9/5/97 -0400, Jeremy Elson wrote:

...

The answer, of course, is that the mail really originated from a PSInet dialup, using IConNet.NET as a spam relay; the bottom Received: line is an utter forgery, presuambly added by the spam-mailing software. In fact, it's not even a very good forgery, because the supposed IP address of alt2.bethere.net is invalid (the 2nd octet is 756).

Yes, it seems that once a spammer finds your site (fs.iconnet.net, mine) they share it with others. What was a trickle (in April, when you got spammed) became a flood as the "disposable dial-ppp / third-party relay" technique became widespread. At the time we had approximately 15 "open" mail servers - but only one was ever abused - they either share with each other or have common sources/techniques of scanning for "open" servers.

X-Disclaimer: if you're not interested in sendmail techniques to keep spam off your network, delete now.

Anyway, we were able to dig up with a nice simple solution that solves some problems that ISPs have. The reason I'm posting is because it took a long time to find the solution and most sources of information (spam.abuse.net, etc) are aimed at small sites, not ISPs who provide mail-relay and MX backup for their customers. The solution is located at

http://www.informatik.uni-kiel.de/%7Eca/email/check.html http://www.informatik.uni-kiel.de/%7Eca/email/rules/check.tar

what we do now, with most help from Claus A�mann's site:

= We now have four files that control our anti-abuse sendmail (in order):

1. Spammer These user addresses can't send mail 2. SpamDomains These domains can't send mail 3. LocalIP These IP addresses can relay mail 4. RelayTo Mail destined to these domain names can go through

Thus, our customers can use our mail servers to relay (#3), and anyone else must be sending to our customers (#4) or they get rejected. Plus we can block any spammer, customer or non-customer (#1,2). Now we only have to worry about our downstreams spamming, where we actually have leverage.

Things that need work: script to dynamically create localip file (point a program at your cisco and let it "sh ip bgp filter x" to get your list, which you can then edit) . merge spammer and spamdomains into one file with wildcards (*@*.b.com , user@*.c.com , *@port15.dial.d.net) . cidr and substring matching are not the same (you can take 10.1.0.0/17 and make 128 /24 entries, or one /16 entry and allow the other /17 through)

I'm thinking of building on this and sharing my results with Claus and any other interested parties. Suggestions / Comments / Ideas please e-mail me. Thanks for your time.

-Rod

Regards Peter Marelas -- Phase One Interactive - Sun Solaris/Unix/Networking Consultant P.O Box 549, Templestowe 3106 Melbourne, Australia URL: http://www.phase-one.com.au/

I'll just make this one comment, as I think this whole thread is probably off-topic, but this tactic has been used for quite some time by spammers. Even if they aren't using a version with the bogus timestamp, following the headers down, the forged line becomes obvious when you realise that the psi host never received it from bothere.net, plus there *is* no bothere.net. For further information on this topic, I would suggest either the spam-l mailing list, or send mail to spam-request@zorch.sf-bay.org. Many of these issues have long been hashed, and current topics on the spam problem are more properly discussed on one of those lists. Steve Mansfield steve@nwnet.net NorthWestNet Network Engineer 425-649-7467

On Fri, Sep 05, 1997 at 04:35:17PM -0400, Jeremy Elson wrote:

...

More recently, though, something much more insidious started to happen: spammers have started forging Received: lines in the headers to misdirect attempts at tracing the source of the mail! Here's one beautiful example of a spam header I received (my mailhost here was blaze.cs.jhu.edu):

From: mailman@domaol.net Received: from fs.IConNet.NET by blaze.cs.jhu.edu with ESMTP; Wed, 9 Apr 1997 07:54:13 GMT Sender: mailman@domaol.net Received: from 199.173.160.250 (ip19.new-haven.ct.pub-ip.psi.net [38.11.102.19]) by fs.IConNet.NET (8.8.5/8.8.5) with SMTP id DAA12207; Wed, 9 Apr 1997 03:54:27 -0400 (EDT) Received: from mailhost.bethere.net(alt2.bethere.net(214.756.86.9)) by bethere.net (8.8.5/8.6.5) with SMTP id GAA04732 for <friend@public.com>; Wed, 09 Apr 1997 02:52:20 -0600 (EST) ^^^^^^^^^^^ To: friend@public.com Message-ID: <37474743565665.JDL9087@bethere.net> [ "how did it get there?" ] The answer, of course, is that the mail really originated from a PSInet dialup, using IConNet.NET as a spam relay; the bottom Received: line is an utter forgery, presuambly added by the spam-mailing software. In fact, it's not even a very good forgery, because the supposed IP address of alt2.bethere.net is invalid (the 2nd octet is 756).

This is a known spamming program; the highlighted mistake would probably work _exceptionally_ well in your procmail file. :-)

Cheers, -- jra -- Jay R. Ashworth jra@baylink.com Member of the Technical Staff Unsolicited Commercial Emailers Sued The Suncoast Freenet "People propose, science studies, technology Tampa Bay, Florida conforms." -- Dr. Don Norman +1 813 790 7592

I enjoyed reading Randy's comments, as always. Here's the fine print. Blackholing spammers is tricky. For instance, recently the professional spammers got so good at locating third party relay sites that they no longer have to overload other folks' relays in order to get the spam out. So now rather than finding one relay and handing it 50 envelopes each with 10,000 recipients, they find 500 relays and hand each one 1,000 envelopes each with 1 recipient. They add random gibberish to each message body so that tight checksums like MD5 won't be able to detect the duplicates. (Yes, loose checksums are available and they are being employed.) What this means, though, is that third party relays are no longer being given so much mail to deliver (by any given spammer, that is) that they come to us (the anti-spam crowd) screaming for anti-relay solutions such as Eric Allman's excellent http://www.sendmail.org/antispam.html logic. Oh sure, the next day or the next week the relay will be abused again, but now that it no longer brings the relay (and its upstream link) to its knees, the operators of these relays are feeling considerably less natural pressure to turn off third party relaying. Microsoft's Exchange 5.0 adds relay support and the default is ON. So blackholing the spammers led them to relay their spam via third parties, but like all naive parasites they failed to use any kind of quotas and they irritated (in some cases killing) their host bodies. Now they're smarter. So now whenever I am spammed I blackholed the relay's /32 for ten days. This is twice the 5-day queue limit that Host Requirements recommends for mail, and it is the Sendmail-8 default. (Sendmail-5's default was 3 days -- ouch!) I often find that during the ten day blackhole period, a mail relay's operator discovers that their connectivity isn't very good for some reason, and finds out that I am the reason, and threatens to sue me. At the moment there are 92 hosts in this ten day "holddown period" and while three of them have asked how they can prevent third party relay in their mailers, two others have sent official-looking letters with words like "cease" and "desist" in them. The spammers are going to make it as hard as possible to block them. For a while they used to abuse "popular" relays and shell machines and so on, in the mistaken belief that nobody would block a popular and necessary host resource just to get stop spam. I think I've told the story of the firebombing of Dresden to at least a half dozen popular host resource owners in the last two years. Blocking relays stops spams in progress. I've seen this happen often enough that I know it's what I have to do. But I've had two blackhole mirror sites drop off the list because they could not afford to block somebody that I had to block. (There is of course a way to block my blocks, and several mirror sites do that routinely.) But blocking relays doesn't stop the phenomena of spam, in fact it doesn't even slow it down. Consider the fact that I only blackhole when I am myself spammed. Don't you think that if it were in a spammer's power they would try to avoid spamming me? Consider the fact that all Sendmails ever installed (including the one you'd get right now from ftp.sendmail.org) allow full relay between arbitrary sources and destinations, and that changing it is HARD. Spammers do still send a lot of spam directly. When I screwed the pooch in a system upgrade to my anti-spam blackhole route server and had to spend two hours "wide open" I was spammed *once* *a* *minute* by various nets which I normally block. So I know that the blackhole list does some good. But it is not a fix to the underlying problem, and while I have no direct economic incentive to block spam, spammers perceive a very real and direct economic incentive to send it to all of us. So, yes, do sign up for the blackhole. If half the ISP's in the country would just refuse to exchange packets with most of AGIS's customers, maybe the other half would feel so much pain that they would come along for the ride. (Right now AGIS picks up a huge amount of business since disconnected spammers always end up buying connectivity from AGIS when noone else will sell it to them.) Who knows, perhaps we can isolate the spammers so they can only spam eachother. But be aware that blackholing people, especially on my say so, will lead you to get complaints from your users about unreachability, and complaints from other ISP's users about unreachability, and that while these are probably fewer complaints than you're getting right now about spam, the war won't be over until the last spammer's head is stuck onto a spear at the city limits. If you want to blackhole spammers, I can help. But it's NOT a magic bullet. Now as to money. I've hired somebody to do the paperwork of signing up new eBGP4 anti-spam routing feed recipients. I will shortly start charging some kind of quarterly fee to said recipients to cover some of my costs. If you decide to start feeding each other, just make sure that the route origin is always my server since I need to be able to revoke a black hole route in real time whenever (a) I make a mistake or (b) somebody calls me asking for help with their spam problem and they are on my blackhole list. If you cache this data or disconnect it from its source, I'm still liable for the business losses of blackholed network owners even though I won't have any control over continued propagation. Don't put me in that position, please. I am also getting ready to start work on my company's next commercial product, and it looks like a spam filtering SMTP gateway is going to be it even though I've got this drop-dead idea for optimal HTTP redirects that I've been wanting to implement for about the last 14 months. Oh well, "follow the money."

Well I think you hit the nail on the head with this paragraph. Whilst Microsoft and the standard sendmail distribution ship with realying on by default, 95% of sites will probably relay. If this was changed (yup, makes installation harder), 95% of sites wouldn't have relaying on by default.

Last time I asked Eric Allman to make non-relay the default, he claimed that it would do more harm than good since it would be a change from previous behaviour and if not configured perfectly would make a Sendmail upgrade into something that would break previously working config files. I was sympathetic, BIND has the same problem from time to time. Maybe I should ask him again -- depending on how much spam he's received in the year or so since I last asked him, he may be willing to change his mind. Microsoft's situation is inverted and they have NO excuse at all. Could somebody who hasn't been burned to a crisp by IETF politics please write a "Mail Relay Requirements" RFC that we can brandish at these vendors? (Dave Crocker seems like a logical choice for this given his past credits.)

One of the big problems we found is that if you naively blackhole a route, you only stop backtraffic to that destination. Some sites were sending us so many SYN opens, that as our SYNACKs never got there, we ended up turning a mild source of SPAM into a powerful SYN flood attack. The solution is to (a) ensure you are running kernels capable of handling this reasonably well,

As it happens, you need this anyway. There are a lot of SYN-flood generators out there and there are a lot of sociopathic teenagers (of all ages) willing to torpedo your mail servers just for fun, with or without a blackhole list.

and (b) (more important) ensure that your blackholing router returns ICMP unreachable for these nets, not simply swallows the packet. For various reasons this is difficult to do with Cisco's without unpleasant things like telnet <blackholed address> giving you a logon onto the router.

I think the access-list that controls telnet is still in force in this case, so if you're only allowing telnet from your NOC (which seems wise, right?) then only your NOC will get the IOS prompt when telnetting to a black hole. The ICMP thing is hard for another reason, which is that modern TCP's ignore ICMP _even_in_SYN_WAIT_ which seems utterly wrong to me. (I patched it here but not everybody has OS source for their mail relays.) This is another possible topic to be covered in a Mail Relay Requirements RFC, perhaps.

I'll publish the fix when we have it honed. (The unreachable should make the kernel drop the record of the half open connection).

Should, but doesn't, unless you have OS kernel source and know how to use it.

One particular site (something at AT&T worldnet - no compulsion about naming them as this was so ridiculous) was sending us one open every minute *per mail message queued* (i.e. they were running with -q1m). This is seriously clueless. We spent the best part of a half a day's engineering time trying to get through to a clueful person there. Evenetually we got through to the person allegedly running the server who had no idea how or why it had been set up like that, but didn't want to change it, or disable relaying. So now they are in the appropriate access list deny in the relevant border router even for incoming packets. No complaints yet.

The Worldnet people are not stupid. They've spent a huge amount of time and money preventing third party relay even though they have nationwide dialup pools which they lease to other providers, and vice versa. The only spam I get via Worldnet's relays comes from Worldnet customers at this point, which is a heck of a feat, one worthy of succession by, oh, let's say UUNET.

Is it possible to get the list of sites included in this feed. Paul Vixie used to publish these site on his web site, but the "Rogue" sites area has been taken down. Apparently a large number of list "memebers" threaten legal action. They alleged some used this list as a list of targets for retaliation.

http://spam.abuse.net/ is hosted on a vixie machine but not edited by any vixie people. scott hazen mueller, a former employee here, is in charge of that page. i have no editorial input at all other than as an individual contributor. in other details, the above paragraph is factually correct.

We were freuquent reviewers of the list. We used the information there to update our sendmail filters. We were very dissapointed to see it go.

The only trouble I have with Paul's list was it occasionally needed fine tuning by hand. Some sites had to be permitted, even though they may have been guilty of UCE. On one occasion, they had an ISP on their list. This ISP was close enough to us in geographic region that it was reasonable to expect some of our users were emailing users on their system. We could not therefore black hole them.

thank you for that explaination. the black hole list is not for everybody, though i believe that with the "fine tuning" you described, it can be of use to anybody who wants to do more work and get less spam.

It would seem to me that this would be more serious if this were an automatic feed, sent directly to our routers. There would be no way to review the sites on the list before they were blocked.

Perhaps Paul would create a mailing list for network operators interested in receiving his blackhole list. As new updates to the group were made, they could be sent to the list. So as not to appear as though we are looking for something for nothing, we would be willing to donate resources to run the list.

because of my cease-and-desist problem, and my very real worries about "conspiracy in restraint of trade", i will not make the list available other than through a protocol like BGP/TCP which allows me to revoke an entry from the list in real time and without anyone else's participation.