John, its about the same situation you get when people use manually updated bogon filters. A much larger problem, I must admit .. having ISPs follow the maawg best practices might help, that - and attending MAAWG sessions (www.maawg.org -> Published Documents) That said most of the larger players already attend MAAWG - that leaves rural ISPs, small universities, corporate mailservers etc etc that dont have full time postmasters, and where you're more likely to run into this issue. If you see actual large carriers with outdated blocklist entries after they're removed from (say) the spamhaus pbl, then maybe MAAWG needs to come to nanog / arin meetings .. plenty of maawg types attend those that all that needs to be done is to free up a presentation slot or two. --srs On Tue, Sep 8, 2009 at 11:13 PM, John Curran<jcurran@arin.net> wrote:

Folks -

  It appears that we have a real operational problem, in that ARIN   does indeed reissue space that has been reclaimed/returned after   a hold-down period, and but it appears that even once they are   removed from the actual source RBL's, there are still ISP's who   are manually updating these and hence block traffic much longer   than necessary.

  I'm sure there's an excellent reason why these addresses stay   blocked, but am unable to fathom what exactly that is...   Could some folks from the appropriate networks explain why   this is such a problem and/or suggest additional steps that   ARIN or the receipts should be taking to avoid this situation?

Thanks! /John

John Curran President and CEO ARIN

On Sep 8, 2009, at 11:16 AM, Ronald Cotoni wrote:

...

Tom Pipes wrote:

...

Greetings,

We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. This block has been cursed (for lack of a better word) since we obtained it.  It seems like every customer we have added has had repeated issues with being blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc).  I understand there is a process to getting removed, but it seems as if these IPs had been used and abused by the previous owner.  We have done our best to ensure these blocks conform to RFC standards, including the proper use of reverse DNS pointers.

I can resolve the issue very easily by moving these customers over to our other direct assigned 66.254.192.0/19 block.  In the last year I have done this numerous times and have had no further issues with them.

My question:  Is there some way to clear the reputation of these blocks up, or start over to prevent the amount of time we are spending with each customer troubleshooting unnecessary RBL and reputation blacklisting? I have used every opportunity to use the automated removal links from the SMTP rejections, and worked with the RBL operators directly.  Most of what I get are cynical responses and promises that it will be fixed. If there is any question, we perform inbound and outbound scanning of all e-mail, even though we know that this appears to be something more relating to the block itself.

Does anyone have any suggestions as to how we can clear this issue up?  Comments on or off list welcome.

Thanks,

--- Tom Pipes T6 Broadband/ Essex Telcom Inc tom.pipes@t6mail.com

Unfortunately, there is no real good way to get yourself completely delisted.  We are experiencing that with a /18 we got from ARIN recently and it is basically the RBL's not updating or perhaps they are not checking the ownership of the ip's as compared to before. On some RBL's, we have IP addresses that have been listed since before the company I work for even existed.  Amazing right?

-- Suresh Ramasubramanian (ops.lists@gmail.com)

MAAWG's changing that - and most of the large ISPs are maawg members. Things can get better, sure ... --srs On Tue, Sep 8, 2009 at 11:29 PM, Jason Bertoch<jason@i6ix.com> wrote:

Suresh Ramasubramanian wrote:

...

That said most of the larger players already attend MAAWG - that leaves rural ISPs, small universities, corporate mailservers etc etc that dont have full time postmasters, and where you're more likely to run into this issue.

I've found the opposite to hold true more often.  Smaller organizations can use public blacklists for free, due to their low volume, and so have little incentive to run their own local blacklist.  I've typically seen the larger organizations run their own blacklists and are much more difficult to contact for removal.

-- Suresh Ramasubramanian (ops.lists@gmail.com)

Suresh Ramasubramanian wrote:

John, its about the same situation you get when people use manually updated bogon filters.

A much larger problem, I must admit .. having ISPs follow the maawg best practices might help, that - and attending MAAWG sessions (www.maawg.org -> Published Documents)

That said most of the larger players already attend MAAWG - that leaves rural ISPs, small universities, corporate mailservers etc etc that dont have full time postmasters, and where you're more likely to run into this issue.

I was always under the impression that smaller orgs were not allowed to join the MAAWG club. ~Seth

Jay Hennigan wrote:

By the way, among the members...

Experian CheetahMail ExactTarget, Inc Responsys, Inc. Vertical Response, Inc Yesmail

Have you been reading from my blacklist again, Jay? Justin

J.D. Falk wrote:

Seth Mattinen wrote:

...

I was always under the impression that smaller orgs were not allowed to join the MAAWG club.

I've heard that, too, but have no idea where it comes from. It's not true; there's no size requirement or anything like that.

http://www.maawg.org/ has the membership application and other info.

The $4000/year minimum membership fee is a non-starter for small organizations who are already strapped for operating cash as it is. This is probably where the perception comes from. -- William Astle lost@l-w.ca

John Curran wrote:

Folks -

It appears that we have a real operational problem, in that ARIN does indeed reissue space that has been reclaimed/returned after a hold-down period, and but it appears that even once they are removed from the actual source RBL's, there are still ISP's who are manually updating these and hence block traffic much longer than necessary.

I'm sure there's an excellent reason why these addresses stay blocked, but am unable to fathom what exactly that is... Could some folks from the appropriate networks explain why this is such a problem and/or suggest additional steps that ARIN or the receipts should be taking to avoid this situation?

I don't think there is an excellent reason, more likely inertia and no real incentive to put forth the effort to proactively remove addresses. Many ISPs and organizations have their own private blocklists not associated with the widely known DNSBLs. Typically during or immediately after a spam run the mail administrator will manually add offending addresses or netblocks. Spamtrap hits may do this automatically. There isn't any real incentive for people to go back and remove addresses unless they're notified by their own customers that legitimate mail coming from those addresses is being blocked. Because these blocklists are individually maintained, there is no central registry or means to "clean them up" when an IP assignment changes. To make matters worse, some organizations may simply ACL the IP space so that the TCP connection is never made in the first place (bad, looks like a network problem rather than deliberate filtering), some may drop it during SMTP with no clear indication as to the reason (less bad, as there is at least a hint that it could be filtering), and some may actually accept the mail and then silently discard it (worst). In addition there are several DNSBLs with different policies regarding delisting. Some just time out after a period of time since abuse was detected. Some require action in the form of a delisting request. Some require a delisting request and a time period with no abuse. Some (the old SPEWS list) may not be easily reached or have well defined policies. In meatspace, once a neighborhood winds up with a reputation of being rife with drive-by shootings, gang activity and drug dealing it may take a long time after the last of the graffiti is gone before some cab drivers will go there. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

Right on point -- we have a long list of manually entered netblocks in our spam appliance's blacklist that we've accumulated over time. Besides the mistakes we've made, we've had to delist perhaps 5 over the last 2 years, none due to ARIN reallocations. Most times it's our customer calling our helpdesk and saying "I can't get an e-mail from so-and-so". There's a strong (time resource) disincentive for us to review netblocks and then delist them. Ideally our spam appliance vendor would show us a top ten of non-hit netblocks and we would remove them then (i.e. if no one has hit an IP in that range for a month, the spammer has probably moved on), or as another person suggested, just have the spam appliance age them out (change the action applied from "blocked" to "do nothing". One of the potential community-based approaches would be to have a hosted RBL, with a 'view' for each SP or enterprise. That is, each RBL would be unique, but if I trusted organization B, I could request to use their RBL entries, too. Rather than managing a manual list, it would be managed on the web with more management tools: - search by date added, size of netblock, hits, etc. - auto expiration/aging - notification if netblock assigned to a new owner - comparison against other RBLs (no use having it on my company's RBL is Spamhaus has added it) than an admin of a small operation would likely have. Contact info could be made available, mechanism to request delisting, etc. Frank -----Original Message----- From: Jay Hennigan [mailto:jay@west.net] Sent: Tuesday, September 08, 2009 1:14 PM To: John Curran Cc: nanog@nanog.org Subject: Re: Repeated Blacklisting / IP reputation John Curran wrote:

Folks -

It appears that we have a real operational problem, in that ARIN does indeed reissue space that has been reclaimed/returned after a hold-down period, and but it appears that even once they are removed from the actual source RBL's, there are still ISP's who are manually updating these and hence block traffic much longer than necessary.

I'm sure there's an excellent reason why these addresses stay blocked, but am unable to fathom what exactly that is... Could some folks from the appropriate networks explain why this is such a problem and/or suggest additional steps that ARIN or the receipts should be taking to avoid this situation?

I don't think there is an excellent reason, more likely inertia and no real incentive to put forth the effort to proactively remove addresses. Many ISPs and organizations have their own private blocklists not associated with the widely known DNSBLs. Typically during or immediately after a spam run the mail administrator will manually add offending addresses or netblocks. Spamtrap hits may do this automatically. There isn't any real incentive for people to go back and remove addresses unless they're notified by their own customers that legitimate mail coming from those addresses is being blocked. Because these blocklists are individually maintained, there is no central registry or means to "clean them up" when an IP assignment changes. To make matters worse, some organizations may simply ACL the IP space so that the TCP connection is never made in the first place (bad, looks like a network problem rather than deliberate filtering), some may drop it during SMTP with no clear indication as to the reason (less bad, as there is at least a hint that it could be filtering), and some may actually accept the mail and then silently discard it (worst). In addition there are several DNSBLs with different policies regarding delisting. Some just time out after a period of time since abuse was detected. Some require action in the form of a delisting request. Some require a delisting request and a time period with no abuse. Some (the old SPEWS list) may not be easily reached or have well defined policies. In meatspace, once a neighborhood winds up with a reputation of being rife with drive-by shootings, gang activity and drug dealing it may take a long time after the last of the graffiti is gone before some cab drivers will go there. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay@impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

On Tue, 8 Sep 2009, John Curran wrote:

...

I'm sure there's an excellent reason why these addresses stay blocked, but am unable to fathom what exactly that is... Could some folks from the appropriate networks explain why this is such a problem and/or suggest additional steps that ARIN or the receipts should be taking to avoid this situation?

Most small to midsize networks probably have a "block it and forget it" policy. The facts that the spammer moved on, the IPs eventually got returned to the RIR and reallocated to a different network go unnoticed until the new LIR/ISP notifies those blocking the addresses that the addresses have changed hands. Ideally, the network doing the blocking will know when they started blocking an IP, look at whois, and agree that the block no longer makes sense. I'm sure some will have no idea when or why they started blocking an IP, and might be reluctant to unblock it. This assumes you can actually get in touch with someone with the access and understanding of the issues to have a conversation about their blocking. Some networks make that nearly impossible. I ran into such situations early on when trying to contact networks about their outdated bogon filters when Atlantic.net got a slice of 69/8.

This blocking (or variations of it) has been a problem for about a decade.

http://www.michnet.net/mail.archives/nanog/2001-08/msg00448.html

I don't think there is any blanket solution to this issue. Too many of the networks doing the blocking likely don't participate in any forum where the RIRs will be reach people who care and can do something.

It should be pretty clear that reused IP space is going to represent a problem. There is no mechanism for "LIR/ISP notif[cation to] those blocking the addresses that the addresses have changed hands." Even if there were, this would be subject to potential gaming by spammers, such as SWIP of a block to SpammerXCo, followed by an automatic unblock when the "ISP" unSWIP's it and SWIP's it to "EmailBlasterB" - of course, the same company. How do we manage this into the future? IPv6 shows some promise in terms of delegation of larger spaces, which could in turn suggest that reuse policies that discourage rapid reuse would be a best practice. However, that is more or less just acknowledging the status quo; networks are likely to continue blocking for various reasons and for random periods. A remote site being unable to communicate with us is not particularly important except to the extent that it ends up distressing users here; however, for larger sites, the blocked list could end up being significant. It seems like it *could* be useful to have a system to notify of network delegation changes, but it also seems like if this was particularly important to anyone, then someone would have found a trivial way to implement at least a poor man's version of it. For example, record the ASN of a blocked IP address and remove the block when the ASN changes... ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Tue, 8 Sep 2009, Joe Greco wrote:

...

It seems like it *could* be useful to have a system to notify of network delegation changes, but it also seems like if this was particularly important to anyone, then someone would have found a trivial way to implement at least a poor man's version of it. For example, record the ASN of a blocked IP address and remove the block when the ASN changes...

That too, would be easily gamed by spammers. Just get multiple ASN's and bounce your dirty IPs around between them to clean them. The IP space being a direct (RIR->LIR) allocation having been made after the blocking was initiated is a pretty clear sign that the space has actually changed hands, and seems like it would be fairly difficult (if at all possible) to game.

Right, but they'll only do that if they're aware of such a system and it is significant enough to make a dent in them. Further, it would be a mistake to assume that *just* changing ASN's would be sufficient. The act of changing ASN's could act as a trigger to re-whois ARIN for an update of ownership, for example. The fact is that the information to trigger a re-query of ownership upon a redelegation sort-of already exists, though it is clearly imperfect. My point was that if it was actually useful to "notice" when an IP delegation moved, someone would already have made up a system to do this somehow. So my best guess is that there isn't a really strong incentive to pursue some sort of notification system, because you could pretty much do it as it stands. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

there is a fundamental disconnect here. the IP space is neutral. it has no bias toward or against social behaviours. its a tool. the actual/real target here are the people who are using these tools to be antisocial. blacklisting IP space is always reactive and should only beused in emergency and as a -TEMPORARY- expedient. IMHO of course., YMMV. --bill On Tue, Sep 08, 2009 at 01:43:39PM -0400, John Curran wrote:

Folks -

It appears that we have a real operational problem, in that ARIN does indeed reissue space that has been reclaimed/returned after a hold-down period, and but it appears that even once they are removed from the actual source RBL's, there are still ISP's who are manually updating these and hence block traffic much longer than necessary.

I'm sure there's an excellent reason why these addresses stay blocked, but am unable to fathom what exactly that is... Could some folks from the appropriate networks explain why this is such a problem and/or suggest additional steps that ARIN or the receipts should be taking to avoid this situation?

Thanks! /John

John Curran President and CEO ARIN

On Sep 8, 2009, at 11:16 AM, Ronald Cotoni wrote:

...

Tom Pipes wrote:

...

Greetings,

We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. This block has been cursed (for lack of a better word) since we obtained it. It seems like every customer we have added has had repeated issues with being blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc). I understand there is a process to getting removed, but it seems as if these IPs had been used and abused by the previous owner. We have done our best to ensure these blocks conform to RFC standards, including the proper use of reverse DNS pointers.

I can resolve the issue very easily by moving these customers over to our other direct assigned 66.254.192.0/19 block. In the last year I have done this numerous times and have had no further issues with them.

My question: Is there some way to clear the reputation of these blocks up, or start over to prevent the amount of time we are spending with each customer troubleshooting unnecessary RBL and reputation blacklisting? I have used every opportunity to use the automated removal links from the SMTP rejections, and worked with the RBL operators directly. Most of what I get are cynical responses and promises that it will be fixed. If there is any question, we perform inbound and outbound scanning of all e-mail, even though we know that this appears to be something more relating to the block itself.

Does anyone have any suggestions as to how we can clear this issue up? Comments on or off list welcome.

Thanks,

--- Tom Pipes T6 Broadband/ Essex Telcom Inc tom.pipes@t6mail.com

Unfortunately, there is no real good way to get yourself completely delisted. We are experiencing that with a /18 we got from ARIN recently and it is basically the RBL's not updating or perhaps they are not checking the ownership of the ip's as compared to before. On some RBL's, we have IP addresses that have been listed since before the company I work for even existed. Amazing right?

On Tue, Sep 08, 2009 at 02:34:10PM -0500, Joe Greco wrote:

...

there is a fundamental disconnect here. the IP space is neutral. it has no bias toward or against social behaviours. its a tool. the actual/real target here are the people who are using these tools to be antisocial. blacklisting IP space is always reactive and should only beused in emergency and as a -TEMPORARY- expedient.

IMHO of course., YMMV.

Show me ONE major MTA which allows you to configure an expiration for an ACL entry.

call me old skool... VI works a treat and I'm told there is this thing called emacs ... but i remain dubious.

The problem with your opinion, and it's a fine opinion, and it's even a good opinion, is that it has very little relationship to the tools which are given to people in order to accomplish blocking. Kind of the question I was contemplating in my other message of minutes ago.

if all you have is a hammer... folks need better tools.

If people were given an option to "block this IP for 30 minutes, 24 hours, 30 days, 12 months, 5 years, or forever" - I wonder how many people would just shrug and click "forever."

which is their choice. please show me the mandate for accepting routes/packets from any/everywhere? me, i'd want the option to "block 192.0.2.0/24 as long as it is announced by AS 0 and the whois data points to RIAA as the registered contact" e.g. not just a temporal block. or - if traffic from 192.0.2.80 increases more than 65% in a 150 second interval, block the IP for 27 minutes. or - allow any/all traffic from 192.0.2.42 - regardless of the blocking on 192.0.2.0/24 the mind boggles.

This may lead to the discovery of another fundamental disconnect - or two.

such is the course of human nature.

Sigh.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

...

Show me ONE major MTA which allows you to configure an expiration for an ACL entry.

The problem with your opinion, and it's a fine opinion, and it's even a good opinion, is that it has very little relationship to the tools which are given to people in order to accomplish blocking. Kind of the question I was contemplating in my other message of minutes ago.

If people were given an option to "block this IP for 30 minutes, 24 hours, 30 days, 12 months, 5 years, or forever" - I wonder how many people would just shrug and click "forever."

This may lead to the discovery of another fundamental disconnect - or two.

Sigh.

... JG

A cron job/schedule task with a script that removes said line would most likely do wonderous things for you. I could see a comment before each listing with a time/date that you use some regex fu on to figure out how long it was there and how long it should be there for. Simple! You could also automate it with a web frontend for noobs so they don't have to manually edit configuration files.

You /COMPLETELY/ missed the point. If this was something that people felt was truly useful, then there would be support for something like this. I mean, we've only had about 15 years of spam-as-a-real-problem on the Internet. The perception by most admins is that when you block someone, you want to block them for a Really Long Time. If this wasn't true, then there would likely be an automatic feature built in to MTA ACL entries to expire. I didn't say you /couldn't/ do it. The problem is that the average spam spewer is a long-term thing, so when you ACL off a host, you've probably deemed the sender to be of no significant value to you, and you're not expecting that they're suddenly going to become whitehat in two weeks, or even six months. Therefore, there's no default support built into MTA's for this, because it /doesn't/ do anything "wonderous" for you. I would agree that in the best case, we would want a default behaviour of ACL removal when an IP block is reallocated by the RIR, but I don't see an easy way to get there as a default behaviour of an MTA. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

...

...

...

...

"Joe" == Joe Greco <jgreco@ns.sol.net> writes:

Joe> Show me ONE major MTA which allows you to configure an expiration Joe> for an ACL entry. Any MTA which supports using an sql db as its backend. Postfix is a fine example. You just define the table and the query to either have an until column, or have a column with the timestamp of when the entry was added and have the query ignore rows which are older than some given time. And with postfix, using its sql proxy capability, using a sql backend is fully performant. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 1024D/ED7DAEA6

...

...

...

...

"Joe" == Joe Greco <jgreco@ns.sol.net> writes:

Joe> So, you agree, MTA's do not implement this functionality. It's Joe> obviously possible to make it happen through shell scripting, Joe> database tricks, No, I do not agree. The sql backend is part of the MTA; features added by offering a sql backend for tables of this sort (I'd use a cidr access restriction in postfix) are still features of the MTA. And actually using the power of sql when using sql is not a trick; rather it is the /point/. IOW, the MTA is the sum of its parts; when using sql lookups the db is part of the MTA. -JimC -- James Cloos <cloos@jhcloos.com> OpenPGP: 1024D/ED7DAEA6

and then that's PART of the MTA. Otherwise, it's an add-on of some sort. Given that the point I was making was about capabilities *included* in the MTA, and given that I *said* you could add on such functions, it's kind of silly to try to confuse the issue in this manner.

CommuniGate Pro supports time limited blacklisting, at least for Ips it blacklists itself based on protocol violations & c.

John,

ARIN's role as the entity engaged in legal contractual relationship with the previous owners of the space puts it in the position to insert enforceable contract clauses to deter and/or mitigate "graffiti" in allocations.

That's complicated. How do you define "graffiti"? Just for starters. Given that even a whitehat network can generate occasional complaints, and most commercial networks generate various levels of cruft, would you consider it "graffiti" if a block of IP space assigned to a hotel wifi network in Seattle got itself permanently ACL'ed by a college in Miami, when someone inadvertently omitted the port 25 filter, and as a result, the mail admins in Miami judged that the likelihood of ever receiving legitimate mail from there was about 0.0001%? How would you even know?

Policy proposals probably are not required for this.

Space originally from outside ARIN, thats another kettle of fish.

ARIN is also in the position to refuse allocations for entities who dont clean up after themselves. Policy likely required.

How exactly do you do that? Spammers don't mind submitting fraudulent applications. How does ARIN tell that SpamNetA is actually the same operation as FooIspB, even though they might be legally registered as different companies?

And finally, if this problem continues to worsen (as it likely will when greenfield becomes scarce), a viable business opportunity should emerge for reputable organizations to do cleanup on behalf of the new owners, for a reasonable fee/retainer and after suitable financial/contractual guarantees.

Cost of business, efficiency of scale and all that. Perhaps the bill could even be sent to the previous owners.

That's likely to stand up in court. Not.

Operationally, I dont see how the problem can be mitigated solely by those who are already informed.

I agree that it's problematic. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wed, Sep 9, 2009 at 7:18 PM, Alex Lanstein <ALanstein@fireeye.com> wrote:

Along the same lines, I noticed that the worst Actor in recent memory (McColo - AS26780) stopped paying their bills to ARIN and their addresses have been returned to the pool.

It's my opinion that a very select number of CIDR blocks (another example being the ones belonging to Cernel/InternetPath/Atrivo/etc, if it were ever fully extinguished) are, and forever will be, completely toxic and unusable to any legitimate enterprise. Arguments could be made that industry blacklists can and should be more flexible, but from the considerably more innocuous case in this thread, that is apparently not the modus operandi

With regards to Cernel/Internet Path/UkrTelGrp, it needs to be "extinguished" first. :-) - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFKqGZIq1pz9mNUZTMRAnE3AKCL76mNabIzAf5FCWRfqci3YW5QKACgtLNJ AXSIGuT1tIe0R+tm+VL/Flc= =NYQS -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawgster(at)gmail.com ferg's tech blog: http://fergdawg.blogspot.com/

On Wed, Sep 9, 2009 at 11:30 PM, Leo Vegoda <leo.vegoda@icann.org> wrote:

On Sep 9, 2009, at 7:18 PM, Alex Lanstein wrote:

...

Along the same lines, I noticed that the worst Actor in recent memory (McColo - AS26780) stopped paying their bills to ARIN and their addresses have been returned to the pool.

It's my opinion that a very select number of CIDR blocks (another example being the ones belonging to Cernel/InternetPath/Atrivo/etc, if it were ever fully extinguished) are, and forever will be, completely toxic and unusable to any legitimate enterprise. Arguments could be made that industry blacklists can and should be more flexible, but from the considerably more innocuous case in this thread, that is apparently not the modus operandi

Putting these addresses back into use does not mean that they have to be allocated to networks where they'll number mail servers. ARIN staff is doubtless aware of the history of these blocks and will presumably do their best to allocate them to networks that aren't intended to host mail servers.

Regards,

Leo

Not sure when ICANN got into the business of economic bailouts, but the mechanism that ICANN has defined seems patently unfair. Determining who is worthy of allocations based on a class without community input into a policy debate is "bad". ObOps: Chasing down all of this grunge ain't cheap or fair. Best, Martin -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants

On Thu, Sep 10, 2009 at 4:21 PM, David Conrad <drc@virtualized.org> wrote:

On Sep 9, 2009, at 8:41 PM, Martin Hannigan wrote:

...

Not sure when ICANN got into the business of economic bailouts,

??

The blog posting implies it: "AfriNIC and LACNIC have fewest IPv4 /8s and service the regions with the most developing economies. We decided that those RIRs should have four of the easiest to use /8s reserved for them." There is also a possible unintended consequence. If v4 address space markets do end up being legitimized (I do believe that they will FWIW) ICANN is in effect declaring one class of space more valuable than another an arbitrarily assigning that value.

but the mechanism that ICANN has defined seems patently unfair.

...

RFC 2777 is unfair? Or are you unhappy that LACNIC and AfriNIC have 2 /8s from the least tainted pools?

I don't have a comment on the RFC. There is currently a global policy that the RIR's and ICANN agreed to that defines the allocation of /8's from IANA to RIR's. That policy doesnt include a set-aside and I think that arbitrarily adding one is not in the spirit of cooperation. I think that it's "good" that ICANN is being proactive, but I also think that it's "bad" that they chose this to be proactive about. It's possible that not everything is above the table as well. I think that the perception is reality here though. ICANN has arbitrarily created process that impacts RIR's unequally. To me, that's unfair. Question is -- do a few /8's really matter? In the end game, I think that they do all considered. Best, Marty -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants

On Fri, Sep 11, 2009 at 4:23 PM, David Conrad <drc@virtualized.org> wrote:

Marty,

It's possible that not everything is above the table as well.

...

Actually, no. The whole point in publishing the algorithm IANA is using in allocating /8s is to allow anyone to verify for themselves we are following that algorithm.

Sorry, poor wording on my part. See below.

I think that the perception is reality here though. ICANN has arbitrarily

...

created process that impacts RIR's unequally. To me, that's unfair.

As stated, we followed existing RIR practices regarding treatment of LACNIC and AfriNIC. Oddly, the RIR CEOs were happy with the algorithm when we asked them about it.

I honestly don't think that it's up to them to create a set-aside either, hence my comment about behind the scenes activities. I appreciate you detailing that, but I honestly don't think it matters since as you mentioned you get accused of this all of the time. I would expect that ICANN would not only follow the rules, but safeguard them as well. Numbering policy usually goes to the members of each of the RIR communities, just as the IANA to RIR policy did. The algorithm itself is great. The set-aside is the problem. I'd be happy with the algorithm and all of the space. It would be more fair to us all and not appear as a cost shifting or potential windfall. Best, -M< -- Martin Hannigan martin@theicelandguy.com p: +16178216079 Power, Network, and Costs Consulting for Iceland Datacenters and Occupants

On Sun, Sep 13, 2009 at 7:43 AM, John Curran <jcurran@arin.net> wrote:

On Sep 11, 2009, at 6:52 PM, Martin Hannigan wrote:

...

I honestly don't think that it's up to them to create a set-aside either, hence my comment about behind the scenes activities. I appreciate you detailing that, but I honestly don't think it matters since as you mentioned you get accused of this all of the time. I would expect that ICANN would not only follow the rules, but safeguard them as well.

[ clip ]

what would normally have been a behind the scenes implementation issue has now been publicly detailed, and I, for one, thank the IANA for their clear and timely communications on this matter.

I do as well. ICANN does good work in this area and I would not want to appear as though I am saying otherwise.

...

Numbering policy usually goes to the members of each of the RIR communities, just as the IANA to RIR policy did. The algorithm itself is great. The set-aside is the problem.

This is not formation of global Internet numbering policy, it's implementation of the existing policy regarding IANA to RIR /8 block assignments. Regardless, the global nature of the Internet means that we'll all deal with connectivity issues with these blocks once they're allocated. Any and all efforts that the networking community can take now to get these blocks cleaned up now would be most helpful.

Well, ok then :-). I agree to disagree. Anything that affects the flow or quality of IPv4 address space is a policy issue in my mind, especially when a justification for an action is linked to a social issue. I know that it was said that ICANN didn't really mean it when they said that they created this action with "developing economies" in mind, at least not in the way that it is defined[1], but it's hard to say after the fact. Best Regards, Marty 1. http://en.wikipedia.org/wiki/Developing_economies

On Thu, 10 Sep 2009, Mark Andrews wrote:

What a load of rubbish. How is ARIN or any RIR/LIR supposed to know the intent of use?

Why don't we just blacklist everything and only whitelist those we know are good? Because the cost of determining who is good and who is not has a great cost. If you buy an IP block, regardless of your intent, that IP block should not have the ill-will of the previous owner passed on with it. If the previous owner sucked, the new owner should have the chance to use that IP block without restriction until they prove that they suck, at which point it will be blocked again. That system seems to work well enough: blacklist blocks when they start do be evil, according to your own (you being the neteng in charge) definition of evil. ARIN needs to be impartial. If they are going to sell the block, they should do their best to make a coordinated effort to make sure the block is as unencumbered as possible. I get that there is a sense that ARIN needs to do more due dilligence to determine if the receiving party is worthy of that block, but I'm not aware of the process, and from the grumblings it doesn't seem like fun.

Note we all could start using IPv6 and avoid this problem altogether.

Because as we know IPv6 space is inexhaustable. Just like IPv4 was when it began its life. ;-) That won't avoid the problem, it will simply put the problem off until it rears its head again. I'm sure that IPv6 space will be more easily gotten until problems arise, and in a few years (maybe decades, we can put this problem on our children's shoulders), we'll be back where we are now -- getting recycled IP space that is blocked or encumbered due to bad previous owners. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

On Thu, Sep 10, 2009 at 04:42:13PM +0200, Benjamin Billon wrote:

...

Why don't we just blacklist everything and only whitelist those we know are good? <snip>

...

Note we all could start using IPv6 and avoid this problem altogether. <snip> Yeah. When ISP will start receiving SMTP traffic in IPv6, they could start to accept whitelisted senders only.

"IPv6 emails == clean"

Utopian thought?

abt 8 years too late... --bill

You're not Hotmail =)

Benjamin Billon wrote:

...

Why don't we just blacklist everything and only whitelist those we know are good? <snip>

...

Note we all could start using IPv6 and avoid this problem altogether. <snip> Yeah. When ISP will start receiving SMTP traffic in IPv6, they could start to accept whitelisted senders only.

I've been reciveving smtp traffic including spam on ipv6 since 2001.

"IPv6 emails == clean"

Utopian thought?

Peter Beckman wrote:

On Thu, 10 Sep 2009, Mark Andrews wrote:

...

What a load of rubbish. How is ARIN or any RIR/LIR supposed to know the intent of use?

Why don't we just blacklist everything and only whitelist those we know are good?

Because the cost of determining who is good and who is not has a great cost. If you buy an IP block, regardless of your intent, that IP block should not have the ill-will of the previous owner passed on with it.

You don't buy ip blocks or at least not from ARIN. Among other things that ARIN does not guarantee is routability.

If the previous owner sucked, the new owner should have the chance to use that IP block without restriction until they prove that they suck, at which point it will be blocked again. That system seems to work well enough: blacklist blocks when they start do be evil, according to your own (you being the neteng in charge) definition of evil.

ARIN needs to be impartial. If they are going to sell the block, they should do their best to make a coordinated effort to make sure the block is as unencumbered as possible. I get that there is a sense that ARIN needs to do more due dilligence to determine if the receiving party is worthy of that block, but I'm not aware of the process, and from the grumblings it doesn't seem like fun.

...

Note we all could start using IPv6 and avoid this problem altogether.

Because as we know IPv6 space is inexhaustable. Just like IPv4 was when it began its life. ;-)

That won't avoid the problem, it will simply put the problem off until it rears its head again. I'm sure that IPv6 space will be more easily gotten until problems arise, and in a few years (maybe decades, we can put this problem on our children's shoulders), we'll be back where we are now -- getting recycled IP space that is blocked or encumbered due to bad previous owners.

Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

On Wed, Sep 9, 2009 at 11:48 PM, Mark Andrews <marka@isc.org> wrote: <skip a note about isc having quite a few legacy blocks>

Note we all could start using IPv6 and avoid this problem altogether. There is nothing stopping us using IPv6 especially for MTA's.

that'd solve the spam problem... for a while at least. (no ipv6 traffic == no spam) -Chris (yes, I'm yanking mark's chain some)

On Wed, 09 Sep 2009 20:30:02 PDT, Leo Vegoda said:

Putting these addresses back into use does not mean that they have to be allocated to networks where they'll number mail servers. ARIN staff is doubtless aware of the history of these blocks and will presumably do their best to allocate them to networks that aren't intended to host mail servers.

Those streaming video servers in that returned /24 are going to work *real* well talking to a network that implemented the block as a null route rather than a port-25 block.

On Tue, Sep 08, 2009 at 10:16:33AM -0500, Ronald Cotoni wrote:

Tom Pipes wrote:

...

Greetings,

We obtained a direct assigned IP block 69.197.64.0/18 from ARIN in 2008. This block has been cursed (for lack of a better word) since we obtained it. It seems like every customer we have added has had repeated issues with being blacklisted by DUL and the cable carriers. (AOL, AT&T, Charter, etc). I understand there is a process to getting removed, but it seems as if these IPs had been used and abused by the previous owner. We have done our best to ensure these blocks conform to RFC standards, including the proper use of reverse DNS pointers.

I can resolve the issue very easily by moving these customers over to our other direct assigned 66.254.192.0/19 block. In the last year I have done this numerous times and have had no further issues with them.

My question: Is there some way to clear the reputation of these blocks up, or start over to prevent the amount of time we are spending with each customer troubleshooting unnecessary RBL and reputation blacklisting? I have used every opportunity to use the automated removal links from the SMTP rejections, and worked with the RBL operators directly. Most of what I get are cynical responses and promises that it will be fixed. If there is any question, we perform inbound and outbound scanning of all e-mail, even though we know that this appears to be something more relating to the block itself.

Does anyone have any suggestions as to how we can clear this issue up? Comments on or off list welcome.

Thanks,

--- Tom Pipes T6 Broadband/ Essex Telcom Inc tom.pipes@t6mail.com

Unfortunately, there is no real good way to get yourself completely delisted. We are experiencing that with a /18 we got from ARIN recently and it is basically the RBL's not updating or perhaps they are not checking the ownership of the ip's as compared to before. On some RBL's, we have IP addresses that have been listed since before the company I work for even existed. Amazing right?

This is not actually a new problem. ISPs have been fighting this for some time. When a dud customer spams from a given IP range and gets it placed in various RBLs, when that customer is booted or otherwise removed, that block will probably get reissued. The new customer then calls up and says, "my email isn't getting through." All it takes is a little investigation and the cause becomes clear. In my experience, there is absolutely no way to deal with this other than contacting the companies your customer is trying to email one by one. Not all of them will respond to you but when they are slow or do not act at all, quite often if the recipient on the other end calls them up and says, "WTF?" it generates more action. Sadly, I do not foresee this problem getting any easier. Best practices for the public or subscription RBLs should be to place a TTL on the entry of no more than, say, 90 days or thereabouts. Best practices for manual entry should be to either keep a list of what and when or periodically to simply blow the whole list away and start anew to get rid of stale entries. Of course, that is probably an unreal expectation. -Wayne --- Wayne Bouchard web@typo.org Network Dude http://www.typo.org/~web/

Wayne E. Bouchard wrote:

Best practices for the public or subscription RBLs should be to place a TTL on the entry of no more than, say, 90 days or thereabouts. Best practices for manual entry should be to either keep a list of what and when or periodically to simply blow the whole list away and start anew to get rid of stale entries. Of course, that is probably an unreal expectation.

I've had to implement something similar for my RTBH trigger router. After manually-adding nearly 20,000 static routes of hosts that scanned for open proxies or attacked SSH daemons on my network I had to trim the block list considerably because many of my older PEs couldn't handle that many routes without problems. I already named each static with a reason for the block(SSH, Telnet, Proxy-scan, etc) but ended up prepending a date to that string as well: 20090908-SSH-Scan. That way I can parse the config later on and create config to negate everything that's older than 3-4 months. If one of those old IPs is still trying to get to me after 4 months then it will get readded the next time I process my logs entries. If they aren't trying to hit me then they'll no longer be consuming space in my RIB. Justin

On Sep 14, 2009, at 6:49 AM, Rich Kulawiec wrote:

... For example: Ron Guilmette has recently pointed out that notorious spammer Scott Richter has apparently hijacked *another* /16 block -- 150.230.0.0/16. I've dropped that block into various local blacklists, and in some cases, various local firewalls. The entry is essentially permanent, because there's no reason for me to make it otherwise. Perhaps one day ARIN will yank it back, along with all his other blocks, and blacklist him for life; but (a) I doubt it and (b) I'm not willing to wait. The best course of action for me is to just consider it scorched earth and move on.

To the extent that you're aware of a fraudulently transferred address block, please report it to <https://www.arin.net/resources/fraud/>. Thanks! /John John Curran President and CEO ARIN

Christopher Morrow wrote:

The end of the discussion was along the lines of: "Yes, we know this guy is bad news, but he always comes to us with the proper paperwork and numbers, there's nothing in the current policy set to deny him address resources. Happily though he never pays his bill after the first 12 months so we just reclaim whatever resources are allocated then." (yes, comments about more address space ending up on BL's were made, and that he probably doesn't pay because after the first 3 months the address space is 'worthless' to him...)

How should this get fixed? Is it possible to make policy to address this sort of problem?

-chris

If this is the case one could argue that ARIN should be reserving this "worthless" address space to be used when they receive similar requests in the future. There's no reason personX should get fresh, clean address space when they make additional requests. Regards, Chris

I haven't followed this entire string. Are you saying ARIN is repeatedly handing out address space to known abusers? If that's the case then yes, some form of policy should be worked on. If on the administrative level ARIN is not researching returned blocks for abuse complaints and working to clean them up, then...I suppose policy could be proposed. I'm just not sure if that's really where the brunt of assignments to abusers is happening.

From experience I learned the most effective place for abuse stopping is at the network level. Back in 2001 my network had serious problems with this. Making a sale was more important than ensuring abuse didn't occur. However, I worked to install a policy that required customer review before assigning them address space. If public records showed abuse (which was really easy to find) or public records showed a business model that would be really only something leading to abuse complaints then engineering had the veto power to not permit the potential customer onto our network. We managed to go from allot of abuse to essentially zero in 1 year. Then we worked to clean up the damaged blocks.

Granted, if a network or company goes out of business they wont care if the addresses are clean when they return them to ARIN. So maybe this is where some proposal could focus. Also, if this is a case where an entity is able to qualify for direct ARIN allocations and they are habitual at turning over because their business is essentially abusing the network, then policy could focus there as well. Its easy to create a new company name, but from experience the owners name still stays the same for the most part, so a review of the company before allocation would catch that. In reality, we would all benefit if policy to stop it before it happens and policy to clean it up before reissuing existed at the registry and the network level. It would be interesting to see what legal and staff would have to say about taking those types of measures. Controlling this type of abuse and the clean up of it is one of the older arguments for not permitting just anyone direct allocations from ARIN. Abuse and clean up is better managed and cared for at the larger Network levels. Im not looking to open a debate on this last comment. ;o) Its just something that popped into my head as to one of the explanations for why specific levels of qualifications for direct allocations from ARIN existed with IPv4. My 2cents. sorry if it seemed long Cheers, Marla Azinger Frontier Communications Sr Data Engineer -----Original Message----- From: Christopher Morrow [mailto:morrowc.lists@gmail.com] Sent: Monday, September 14, 2009 9:40 AM To: Chris Marlatt Cc: John Curran; nanog@nanog.org Subject: Re: Hijacked Blocks On Mon, Sep 14, 2009 at 11:58 AM, Chris Marlatt <cmarlatt@rxsec.com> wrote:

Christopher Morrow wrote:

...

The end of the discussion was along the lines of: "Yes, we know this guy is bad news, but he always comes to us with the proper paperwork and numbers, there's nothing in the current policy set to deny him address resources. Happily though he never pays his bill after the first 12 months so we just reclaim whatever resources are allocated then." (yes, comments about more address space ending up on BL's were made, and that he probably doesn't pay because after the first 3 months the address space is 'worthless' to him...)

How should this get fixed? Is it possible to make policy to address this sort of problem?

-chris

If this is the case one could argue that ARIN should be reserving this "worthless" address space to be used when they receive similar requests in the future. There's no reason personX should get fresh, clean address space when they make additional requests.

That implies some process changes inside ARIN (I think) and effectively saving 'your old space' for some period of time in escrow for you. This doesn't sound unreasonable, perhaps you put forth some policy verbiage on ppml? -chris

I think ARIN is no party to contact all RBL's and do any cleanup of 'contaminated' address space. The only steps ARIN might do are: - When requesting address space, one should be able to indicate whether receiving previous used address space would be unwanted or not. - When assigning address space, ARIN should notify receivers if it's re-used or virgin address space. - When address space got returned to ARIN and there is evidence of abuse, they have to mark that address space as 'contaminated' and only re-assign that space to new end-users who have indicated to have no problem with that. With kind regards, Michiel Klaver IT Professional

On Mon, Sep 14, 2009 at 4:47 PM, Randy Bush <randy@psg.com> wrote:

...

I haven't followed this entire string.  Are you saying ARIN is repeatedly handing out address space to known abusers?  If that's the case then yes, some form of policy should be worked on.

i might walk more slowly and with a bit less self-righteousness.  this is not a simple area.  are we sure we want the rirs to become the net content police?  how are they to judge?

that was part of my set of points... but if there is a feeling (in the community) that ARIN should be doing something differently, bitching about it on nanog-l or ppml isn't helping. What does help is following the proper process for ARIN policies or suggestions.

e.g., prudent isps act against a customer when there is a court order, not when the net gossip says they're bad actors.  i.e., the decision of who is a bad actor is passed on to the society's normal judicial process.

It's worked out so well so far, yea.... though there are at least 2 things being discussed: 1) not allocating to known offendors (even those who've been through the court system and had judgements against them, which would be following your proposed path) 2) dealing with rbl'd netblocks once they are returned to ARIN and the re-allocated to 'new' people. Both really do, to not just be this same discussion in 12 more moons, need either a policy proposal through ARIN or suggestion to the arin-suggest system. As an aside, what happens to things in APNIC/AFRINIC/RIPE in these circumstances? Say what will happen to: 85.255.112.0/20 in RIPE-land, or 116.50.8.0/20 in APNIC-land? -Chris

On Mon, Sep 14, 2009 at 5:12 PM, Randy Bush <randy@psg.com> wrote:

...

1) not allocating to known offendors (even those who've been through the court system and had judgements against them, which would be following your proposed path)

[ i made no proposal.  i was just a bit scared by the instant "we need to DO SOMETHING" reaction. ]

Oh yea, so I think after reading the 60+ messages on this thread I got to the end saying: o Yes, there is some sort of ongoing problem. o No, this isn't simple to solve. o The 'bad actors' here have the ability to fuzz several variables, more than ARIN Staff could deal with (sanely) I think. o There has to be a reason this hasn't made it past where we are in the discussion today previously.

 but what you say seems somewhat prudent, at least leaving the judgement where it normally is.

I don't have lots of faith in the global judicial system working on this in a sane fashion either, I was just saying ARIN/RIPE/APNIC have (and continue to) allocate blocks to known criminals (ones judged by some court system to have actually broken a law). Part of this problem also is that what's illegal in $COUNTRY1 is legal in $COUNTRY2 (or even inside the US by county or state, weee!) So what 'law breaking' should an RIR follow/adhere to? (See point 4 above)

...

2) dealing with rbl'd netblocks once they are returned to ARIN and the re-allocated to 'new' people.

tough one.  those 'bad' blocks will seem less and less bad over time. but can we not expect the hostfolk at the rirs to kinda order the available queue on this, as well as other appropriate attributes?  i.e. is this not more process than policy?

Sure things are less bad over time, once they are removed from 'bad actor control', but unless the myriad of 'nutcases' that run BL's hear from someone the 'trust' (or will listen to or whatever it takes) things stay on the BL. Additionally, RBL's for 'spam' (smtp) are only the tip of the iceberg here... route-filters, as-filters, firewall rules, null routes, dns-blackholes...

i am not saying all is well here.  i am just trying to move slowly and pretend to think while on my first cuppa.

gotcha -chris

Be careful about what you are asking for.

...

i am not saying all is well here.  i am just trying to move slowly and pretend to think while on my first cuppa.

I agree with Randy that this is a reasonable approach. In the transition from the old IANA to FrICANNstein and the separation of numbers and names, some illuminated minds mixed the Kool-Aid with the concept that in the role of technical coordinator to "guarantee the security and stability of the Internet" everything goes ... And now we have a bunch of IGFers claiming for regulation and control developing policies that may have serious consequences, more when the same entity developing the policy is the one that implements it, and its only accountable to itself and also acts as the tax collector agency. I don't believe anyone is denying that there is an issue, but Randy's call for prudence is well founded. My .02

...

If this is the case one could argue that ARIN should be reserving this "worthless" address space to be used when they receive similar requests in the future. There's no reason personX should get fresh, clean address space when they make additional requests.

That implies some process changes inside ARIN (I think) and effectively saving 'your old space' for some period of time in escrow for you. This doesn't sound unreasonable, perhaps you put forth some policy verbiage on ppml?

Several people have suggested several kinds of things ARIN could or should do. If you have a suggestion that would change ARIN internal operations, but doesn't really affect who gets numbers, or how many integers, you can easily make a suggestion to ARIN: https://www.arin.net/app/suggestion/ If you want to see certain parties get more, fewer, or different numbers, you can propose a policy: https://www.arin.net/participate/how_to_participate.html Seriously, both of these processes are lightweight. There's a form asking for contact info, then just explain your idea, and why it's a good idea. You should subscribe to arin-ppml@arin.net, too, but you don't have to. If you're not sure which way to go, or you want somebody to help shape the right words to do what you want, or any other kind of help understanding or shepherding your proposal through, you can email any of the members of the ARIN Advisory Council: https://www.arin.net/about_us/ac.html If you would rather kick your idea around in person before submitting it in writing, you can bring it up at the ARIN Open Policy Hour, 5:30 Tuesday of NANOG/ARIN, or just catch an Advisory Council member at ARIN. The ARIN community has tried to make it as easy as possible to propose changes and participate, but a message to NANOG may not be quite enough. Lee

On Mon, 14 Sep 2009, Adrian Minta wrote:

Hijacking is not possible since most ISP's use filters based on RIPE Database.

This is not true. Some filter, but most don't. -- Mikael Abrahamsson email: swmike@swm.pp.se