Sorry for bringing this old topic back. But we have made some academic effort investigating the spamming behaviors using assymetric routing (we named it "triangualr spamming"). This work appeared in this year's IEEE Security & Privacy conference. You can take a look at it if you are interested (and feedbacks are welcome): http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf One of the high-level findings is that we developed probing techniques to verify that indeed most ISPs are only blocking 1) "outgoing traffic of destination port 25" instead of 2) "incoming traffic with source port 25", which means that these ISPs are vulnerable to the assymetric routing attack. Finally, many thanks to all your useful inputs :) Regards. -Zhiyun On Oct 22, 2009, at 12:33 PM, Valdis.Kletnieks@vt.edu wrote:

On Thu, 22 Oct 2009 13:22:17 EDT, Zhiyun Qian said:

...

Hi all,

What is the common practice for enforcing port blocking policy (or what is the common practice for you and your ISP)? More specifically, when ISPs try to block certain outgoing port (port 25 for instance), they could do two rules: 1). For any outgoing traffic, if the destination port is 25, then drop the packets. 2). For any incoming traffic, if the source port is 25, then drop the packets.

Note that either of the rule would be able to block outgoing port 25 traffic since each rule essentially represent one direction in a TCP flow. Of course, they could apply both rules. However, based on our measurement study, it looks like most of the ISPs are only using rule 1). Is there any particular reason why rule 1) instead of rule 2)? Or maybe both?

Note that some spammers use assymetric routing with forged packets - they spew out a stream of crafted packets from a compromised machine, showing a different source IP. The claimed source IP is also under the spammer's control, and just basically has to catch the inbound SYN/ACK and forward it to the spam-sender (and, if wanted, sink the other ACKs and forward the SMTP replies from the server to the real sender). So you can have a big sending pipe that doesn't get ingress-filtered(*) sending the spam, and do the control from a throwaway that may have a small pipe.

(*) Of course it's not ingress-filtered - if somebody is selling a spammer a big pipe for this, they can arrange to fail to filter. ;)

The upshot is, of course, that you want to do (1) because you never actually see the (2) packets, they're going someplace else...

You are exactly right. We also talked about stateful firewall that can protect the GoodNet. For NAT box, depends on the type of NAT, it is possible to setup port forwarding on the router (mostly home routers) via uPnP without any authentication (I think many home routers are like this by default). And since the machine in GoodNet is also compromised, it would not be difficult to achieve this. Regards. -Zhiyun On Sep 2, 2010, at 5:45 PM, William Herrin wrote:

On Thu, Sep 2, 2010 at 5:59 PM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf

One of the high-level findings is that we developed probing techniques to verify that indeed most ISPs are only blocking 1) "outgoing traffic of destination port 25" instead of 2) "incoming traffic with source port 25", which means that these ISPs are vulnerable to the assymetric routing attack.

If I understand your idea correctly:

1. GoodNet filters TCP destination port 25 packets from his customer PwndBox, preventing PwndBox from spamming.

2. BadGuy on BadNet sends a forged TCP SYN packet to SpamVictim allegedly from PwndBox on GoodNet.

3. PwndBox receives the response packets from SpamVictim and tunnels them to BadGuy allowing BadGuy to complete the spam.

4. GoodNet didn't stop it because PwndBox never sent any packets to TCP port 25.

5. Since the IP address used was GoodNet's, GoodNet's reputation is damaged..

6. GoodNet could prevent this attack vector by also blocking packets with TCP source port 25 sent -to- PwndBox.

Is that correct?

I observe that if PwndBox is behind a stateful firewall such as a COTS NAT box, that also prevents this attack.

Regards, Bill Herrin

-- William D. Herrin ................ herrin@dirtside.com bill@herrin.us 3005 Crane Dr. ...................... Web: <http://bill.herrin.us/> Falls Church, VA 22042-3004

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem. In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline. Regards. -Zhiyun On Sep 2, 2010, at 8:19 PM, Suresh Ramasubramanian wrote:

Zhiyun, this is by far the most comprehensive paper I've seen on asymmetric routing spam .. a technique that's as old as, for example, Alan Ralsky. So been around for about a decade.

Congratulations, great effort. Do you have more results available (in more detail than were published in this paper)? Should be worth seeing.

thanks --srs

On Fri, Sep 3, 2010 at 3:29 AM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

Sorry for bringing this old topic back. But we have made some academic effort investigating the spamming behaviors using assymetric routing (we named it "triangualr spamming"). This work appeared in this year's IEEE Security & Privacy conference. You can take a look at it if you are interested (and feedbacks are welcome):

http://www.eecs.umich.edu/~zhiyunq/pub/oakland10_triangular-spamming.pdf

-- Suresh Ramasubramanian (ops.lists@gmail.com)

Great. Thanks for the information. -Zhiyun On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

BCP38 / RFC2827 were created specifically to address some quite similar problems. And googling either of those two strings on nanog will get you a lot of griping and/or reasons as to why these aren't being more widely adopted :)

--srs

On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.

In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.

Regards. -Zhiyun

Ingress filtering is the correct tool for the job. The whole point here is that packets are coming from somewhere they should not, and they are thus spoofed. The tools have been in place to deal with this for a very long time now. The drafts that became RFC 2267 (precursor of RFC 2827 / BCP38) date from mid-1996. Paul and I wrote the original drafts to solve something else, but the issue is the same. Solving the vector you're concerned about doesn't need another layer of implementation in the mail servers. The packet routing fabric needs to handle it, and doing so addresses far more than just the email situation. I agree it'd be nice to get the asymmetric attack stopped, but disagree we need yet another mechanism to do it. - Dan On Sep 2, 2010, at 10:55 PM, Zhiyun Qian wrote:

I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).

-Zhiyun On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

...

BCP38 / RFC2827 were created specifically to address some quite similar problems. And googling either of those two strings on nanog will get you a lot of griping and/or reasons as to why these aren't being more widely adopted :)

--srs

On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.

In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.

Regards. -Zhiyun

On Sep 3, 2010, at 10:23 PM, William Herrin wrote:

Frankly, Zhiyun offers the first truly rational case I've personally seen for packet filtering based on the TCP source port.

While the paper is entertaining and novel, and reflects a lot of creativity and hard work on the part of the research team, it's doubtful that any serious spammer has ever sent spam this way. I've certainly never run across it, nor do I know anyone else who has done so. The lack of citations of documented cases in the footnotes, or indeed any projections or discussion of the postulated commonality of this technique tends to support the above view, IMHO. Spammers typically do business with botmasters, and those botmasters have thousands/tens of thousands/hundreds of thousands/millions of bots at their disposal. The supposed economies of scale achieved by 'triangular spamming' (a better name would be something like 'bifurcated false-flag proxying', as spamming is just a use-case of the more generalized, though esoteric technique described in the paper) are far outweighed by its operational complexity and the sheer volume of botnets available to pump out spam 24/7. The supposed performance benefits described in the paper are likely considerably exaggerated, given the RTT and resultant latency of the return traffic via the remote proxy half. The sheer economies of scale offered by conventional botnets greatly outweigh the benefits and caveats of the described technique. The use of routers cracked via credential brute-forcing (no iACLs, no vty ACLs, no AAA, 'cisco/cisco') and configured with GRE tunnels and NAT, sometimes in conjunction with prefix-hijacking, is a more commonly-used spamming technique than that described in the paper. There are a lot of really smart people engaged in all kinds of security-related research, and it's encouraging to see such talented folks thinking outside of the box. In future, vetting of postulated scenarios with the operational community prior to embarking upon lengthy, resource-intensive research projects may be one way to ensure that subsequent efforts are even more tightly focused on more proximate threats, and can also help reduce the continued citation of canards such as attempts to overload such opaque, arbitrary, and unreliable metrics as TTL with more significance than they actually warrant. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness. Owen Sent from my iPad On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).

-Zhiyun On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

...

BCP38 / RFC2827 were created specifically to address some quite similar problems. And googling either of those two strings on nanog will get you a lot of griping and/or reasons as to why these aren't being more widely adopted :)

--srs

On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.

In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.

Regards. -Zhiyun

Patrick W. Gilmore wrote:

...

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

He's right though. tcp/25 blocks are a hack. Easy man's way out. Honestly, it'd be nicer if edge or even core systems could easily handle higher level filtering for things like this. There's plenty of systems that watch traffic patterns and issue blocks based on those patterns. I was working with a hotel today concerning just that. They were only doing a generic 500 connections in x period, block mac. They are now adding a tighter rule for 15 tcp/25 connections in 1 minute, block tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid reasons for mail blasts to be leaving a hotel and 15/minute is plenty of grace for a normal user. At an ISP level, it would work fine, though methods for determining exceptions would have to be planned (though that could easily be handled by customer classifications like everything else).

Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).

Blocking inbound vs outbound is another story, though. Getting people to implement spoof protections is more useful. I'd be interested to see your data for concluding 5-nines of users, or did you just make that up? Jack

On Sep 2, 2010, at 10:41 PM, Franck Martin wrote:

Have you heard of the submission port?

Yes... Many of the idiots that block outbound 25 also block outbound 587 and sometimes 465.

Why Clients of an hotel would run a MTA anyhow?

Huh? I think you misunderstand... The problem is hotels blocking people from submitting outbound messages to their home MTA, not people trying to run an MTA inside their hotel room. NAT pretty much guarantees running an MTA inside the hotel room is impossible in most circumstances. (That might improve when IPv6 starts hitting hotel rooms, but, for now, it's just not there). Yes, some hotels offer you the option of a public IP (and usually when I take that option, I have fewer network problems in general. One of the reasons I tend to prefer Hilton brands when possible.) Owen

----- Original Message ----- From: "Jack Bates" <jbates@brightok.net> To: "NANOG list" <nanog@nanog.org> Sent: Friday, 3 September, 2010 4:08:54 PM Subject: Re: ISP port blocking practice

Patrick W. Gilmore wrote:

...

...

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

He's right though. tcp/25 blocks are a hack. Easy man's way out. Honestly, it'd be nicer if edge or even core systems could easily handle higher level filtering for things like this. There's plenty of systems that watch traffic patterns and issue blocks based on those patterns.

I was working with a hotel today concerning just that. They were only doing a generic 500 connections in x period, block mac. They are now adding a tighter rule for 15 tcp/25 connections in 1 minute, block tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid reasons for mail blasts to be leaving a hotel and 15/minute is plenty of grace for a normal user. At an ISP level, it would work fine, though methods for determining exceptions would have to be planned (though that could easily be handled by customer classifications like everything else).

...

Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).

Blocking inbound vs outbound is another story, though. Getting people to implement spoof protections is more useful. I'd be interested to see your data for concluding 5-nines of users, or did you just make that up?

Jack

Patrick W. Gilmore wrote:

...

Yes... Many of the idiots that block outbound 25 also block outbound 587 and sometimes 465.

Could you point to more than one instance? I've not yet found one. And I think I spend at least as much time in hotels & 3G & airports & etc. as you anyone else here.

I can't remember the ISP, but yes, I've run across this. I had to have my helpdesk inform the customer that they'll have to complain and gripe at the ISP they were using or make other arrangements as I only support 25/587 (customer didn't want to use webmail). Problem is, people hear "block ports", they get in the habit, and the next thing you know, they are blocking ports out of ignorance with no comprehension of what they are breaking. I'd much rather see rate detection setups that let me send however I want, but limit the connections per time interval. It implies that some thought might go into determining the rates. Of course, the only setup I've done like this in testing in my network involved flow analyzers and dynamic acl's. Jack

FWIW, I had it happen at a local library. Used their webform to send a message mentioning that blocking 25 was good, but blocking 587 and 465 was bad. It took several days but they did fix it.

that was the condition at narita red carpet a few years back. had to pull a chain at ugs in chicago to find someone who knew what i meant. randy

I have had it happen in some metro areas on sprint. I have experienced it in at least a dozen hotels over the last 12 months. I have run into it in various airports with free public wifi. I have run into the problem in several coffee shops. By far, the worst offenders are the most expensive hotels where the Internet access, damaged as it is generally goes for $25+ per day. I almost always end up getting free Internet as a result because I report the issue as a problem and their technical support usually can't spell tcp let alone understand what I mean when I say a port is blocked. Even worse is the ones that silently redirect your smtp (regardless of port) session to their MTA. Fortunately, my configuration is good enough that it just breaks in these cases, but I know many people who thought they were connecting to their own server via TLS only to later discover that their mail was relayed in clear text through several third party servers. (most mua's seem to have an unfortunate default to "ssl or tis if available" and keep right on sending even if tis negotiations are rejected.) Owen Sent from my iPad On Sep 4, 2010, at 12:08 AM, JC Dill <jcdill.lists@gmail.com> wrote:

Patrick W. Gilmore wrote:

...

On Sep 3, 2010, at 8:22 AM, Owen DeLong wrote:

...

On Sep 2, 2010, at 10:41 PM, Franck Martin wrote:

...

Have you heard of the submission port?

Yes... Many of the idiots that block outbound 25 also block outbound 587 and sometimes 465.

Could you point to more than one instance? I've not yet found one. And I think I spend at least as much time in hotels & 3G & airports & etc. as you anyone else here.

FWIW, I had it happen at a local library. Used their webform to send a message mentioning that blocking 25 was good, but blocking 587 and 465 was bad. It took several days but they did fix it.

jc

On Sep 3, 2010, at 8:02 PM, Patrick W. Gilmore wrote:

Could you point to more than one instance? I've not yet found one.

I've yet to run across this, either, FWIW, except on extremely restrictive special-purpose endpoint networks. Doesn't mean that it doesn't happen, but it doesn't seem to be nearly as prevalent as TCP/25 blockage on general SP access networks. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@arbor.net> // <http://www.arbornetworks.com> Sell your computer and buy a guitar.

On Sep 2, 2010, at 9:08 PM, Jack Bates wrote:

Patrick W. Gilmore wrote:

...

...

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness. Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

He's right though. tcp/25 blocks are a hack. Easy man's way out. Honestly, it'd be nicer if edge or even core systems could easily handle higher level filtering for things like this. There's plenty of systems that watch traffic patterns and issue blocks based on those patterns.

I was working with a hotel today concerning just that. They were only doing a generic 500 connections in x period, block mac. They are now adding a tighter rule for 15 tcp/25 connections in 1 minute, block tcp/25 (or mac, doesn't matter to me). Of course, we didn't see valid reasons for mail blasts to be leaving a hotel and 15/minute is plenty of grace for a normal user. At an ISP level, it would work fine, though methods for determining exceptions would have to be planned (though that could easily be handled by customer classifications like everything else).

This seems to me like it would be much more effective and less damaging without being significantly harder to implement.

...

Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).

Blocking inbound vs outbound is another story, though. Getting people to implement spoof protections is more useful. I'd be interested to see your data for concluding 5-nines of users, or did you just make that up?

I'm all for anti-spoof (BCP-38) and strict/loose (as appropriate) RPF. I implement those things on networks I run. That's not damage, that's blocking things that shouldn't happen. I'd also like to see his data to support his claim that it is somehow effective at reducing spam. Owen

On Sep 2, 2010, at 8:54 PM, Patrick W. Gilmore wrote:

On Sep 2, 2010, at 11:48 PM, Owen DeLong wrote:

...

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Since at least part of your premise ('ineffective anti-spam measures') has been objectively proven false to fact for many years, I guess we can ignore the rest of your note.

Really? So, since so many ISPs are blocking port 25, there's lots less spam hitting our networks? That's really news to me... I'm still seeing an ever increasing number of attempts to deliver spam on my mailservers. I'd say that it has been pretty ineffective.

Also, just so everyone doesn't think I'm in favor of "damaging" the network, I would much prefer a completely open 'Net. Who wouldn't? Since that is not possible, we have to do what we can to damage the network as little as possible. Port 25 blocking is completely unnoticeable to something on the order of 5-nines worth of users, and the rest should know how to get around it with a minimum of fuss (including things like "ask your provider to unblock" in many cases).

Not really true. First, i dispute your 5-nines figure, second, yes, i can usually get around it, but seems each network requires a different workaround. Since, like many of us, I use a lot of transient networks, having to reconfigure for each unique set of brokenness is actually wasting more of my time than the spam this brokenness was alleged to prevent. I suppose I should just shut up and run an instance of my SMTP daemon on port 80. After all, since IPv4 addresses are so abundant, rather than use port numbers for services, let's use IP addresses and force everything to ports 80 and 443. Owen

Sent from my iPad On Sep 3, 2010, at 10:10 PM, John Levine <johnl@iecc.com> wrote:

...

Really? So, since so many ISPs are blocking port 25, there's lots less spam hitting our networks?

It's been extremely effective in blocking spam sent by spambots on large ISPs. It's not a magic anti-spam bullet. (If you know one, please let us know.)

That simply hasn't been my experience. I still get lots of spam from booted hosts in large provider networks, and yes, that includes many that block 25. As near as I can tell, 25 blocking is not affecting spammers at all, just legitimate users. There was a time when it was effective, but the spammers have long since adapted. Now we are only breaking the Internet. We are no ,onger accomplishing anything ireful. It's pure momentum.

...

workaround. Since, like many of us, I use a lot of transient networks, having to reconfigure for each unique set of brokenness is actually wasting more of my time than the spam this brokenness was alleged to prevent.

Is there some reason you aren't able to configure your computers to use tunnels or SUBMIT? They seem to work pretty well for other people.

Many of the transient networks I deal with block 22, 25, 465, and 587. They also often block protocols 41 and 43 or do not provide a public address, rendering those protocols unusable anyway. Yes, I am now running ssh and s,tp processes on ports 80 and 443 to get around this, but, that consumes an extra address for something that should be handled by a port number. Personally, i'd rather use port numbers for l4 uniqueness rather than IP Addresses. Owen

R's, John

...

It's been extremely effective in blocking spam sent by spambots on large ISPs. It's not a magic anti-spam bullet. (If you know one, please let us know.)

That simply hasn't been my experience. I still get lots of spam from booted hosts in large provider networks, and yes, that includes many

It's extremely effective for us (not a large provider by any means). We block outbound 25 on all dynamic IP customers - to date it's never been a problem for our customers. Customer's who have static assignments are not blocked by default. Paul -----Original Message----- From: John R. Levine [mailto:johnl@iecc.com] Sent: Friday, September 03, 2010 3:20 PM To: Owen DeLong Cc: nanog@nanog.org Subject: Re: ISP port blocking practice that block 25. As near as I can tell, 25 blocking is not affecting spammers at all, just legitimate users. I know people at large ISPs with actual data. Port 25 blocking is quite effective. R's, John

I asked around and got this presentation, but you can search for OP25B too: http://www.anacom.pt/streaming/Honda.pdf?contentId=988141&field=ATTACHED_FILE Some non-anecdotal data about the effectiveness of blocking port 25.

Does the data show that blocking was effective, as in the host didn't detect the block and proceed around it, or, merely that lots of hosts try the direct approach first?

Yes. R's, John

-----Original Message----- From: Owen DeLong [mailto:owen@delong.com] Sent: Friday, September 03, 2010 1:10 PM To: John Levine Cc: nanog@nanog.org Subject: Re: ISP port blocking practice

Sent from my iPad

COOL!

On Sep 3, 2010, at 10:10 PM, John Levine <johnl@iecc.com> wrote:

...

...

Really? So, since so many ISPs are blocking port 25, there's lots

less spam

...

...

hitting our networks?

It's been extremely effective in blocking spam sent by spambots on large ISPs. It's not a magic anti-spam bullet. (If you know one, please let us know.)

That simply hasn't been my experience. I still get lots of spam from booted hosts in large provider networks, and yes, that includes many that block 25. As near as I can tell, 25 blocking is not affecting spammers at all, just legitimate users.

There was a time when it was effective, but the spammers have long since adapted. Now we are only breaking the Internet. We are no ,onger accomplishing anything ireful. It's pure momentum.

So.... How are you getting messages from a user who is sending a message from a network with port 25 blocked? If there is some kind of alternate port usage, or tunneling going on, then there would have been no way to stop it with a filter without doing even more filtering. This additional filtering would likely increase the number of blocked port numbers. This would start breaking other valid protocols. Since you have no suggestions on how to actually handle this issue, I would suggest that you stop criticizing the ones trying to solve the problem for the excessive majority (likely < 99.999%) of users. It is OK to represent the needs of a minority, but the average user doesn't even notice these types of filters and it prevents (largely) ISPs from spending time removing customer IP addresses from RBLs and other filtering mechanisms.

...

...

workaround. Since, like many of us, I use a lot of transient

...

...

having to reconfigure for each unique set of brokenness is actually wasting more of my time than the spam this brokenness was alleged to

networks, prevent.

...

Is there some reason you aren't able to configure your computers to

use

...

tunnels or SUBMIT? They seem to work pretty well for other people.

Many of the transient networks I deal with block 22, 25, 465, and 587. They also often block protocols 41 and 43 or do not provide a public address, rendering those protocols unusable anyway.

Yes, I am now running ssh and s,tp processes on ports 80 and 443 to get around this, but, that consumes an extra address for something that should be handled by a port number.

I'm sorry that you have/had to deal with a provider doing this. I would call it bad form to block ports used for completely valid reasons (not abused services) and would stand by you on those issues.

Personally, i'd rather use port numbers for l4 uniqueness rather than IP Addresses.

With you here brother. :)

Owen

BTW... In a previous post you mentioned "Net Neutrality". Port 25 blocking has NOTHING to do with "Net Neutrality" as long as you block port 25 in a non-partisan manner. If I block port 25 to provider X and not to provider Y for any reason other than abuse/security/network stability specific reasons (meaning to be financially or ethically unreasonable), then it may be considered not being "neutral" in the terms of "Net Neutrality". I would NEVER do such a thing. - Brian CONFIDENTIALITY NOTICE: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copying, use, disclosure, or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. Thank you.

On Fri, 03 Sep 2010 08:12:01 -0400, Owen DeLong <owen@delong.com> wrote:

Really? So, since so many ISPs are blocking port 25, there's lots less spam hitting our networks?

Less than there could be. It appears a lot less effective because there are so many ISPs not doing any blocking. Both of my residential connections are open, and always have been. (even dialup was unblocked. which I always found odd since the UUNET wholesale dialup agreement requires the RADIUS response contain a packet filter limiting port 25 to your mail server(s).) If I block port 25 on my network, no spam will originate from it. (probablly) The spammers will move on to a network that doesn't block their crap. As long as there are such open networks, spam will be rampant. If, overnight, every network filtered port 25, spam would all but disappear. But spam would not completely disappear -- it would just be coming from known mailservers :-) thus enters outbound scanning and the frustrated user complaints from poorly tuned systems... --Ricky

Composed on a virtual keyboard, please forgive typos. On Sep 6, 2010, at 1:36, Claudio Lapidus <clapidus@gmail.com> wrote:

Hello all,

On Fri, Sep 3, 2010 at 11:30 PM, Ricky Beam <jfbeam@gmail.com> wrote:

...

If I block port 25 on my network, no spam will originate from it. (probablly) The spammers will move on to a network that doesn't block their crap. As long as there are such open networks, spam will be rampant. If, overnight, every network filtered port 25, spam would all but disappear. But spam would not completely disappear -- it would just be coming from known mailservers :-) thus enters outbound scanning and the frustrated user complaints from poorly tuned systems...

That won't be probably the case. Here recently we conducted a rather comprehensive analysis on dns activity from subscribers, and we've found that in IP ranges that already have outgoing 25 blocked we were still getting complaints about originating spam. It turned out that the bots also know how to send through webmail, so port 25 blocking renders ineffective there.

I believe you have confused "not 100% effective" with "ineffective". And webmail is but one additional vector. Bots know how to use smarthosts, corporate e-mail, triangulation, etc. If you gave up on each because one step did not solve the problem, you would have no chance at a solution. When you unblocked port 25, did spam complaints go up or down? There are a great many providers who have evidence that port 25 blocking lowers complaints even if there are bots that know their way around it. Second, assume you can wave a magic wand and block all webmail access. Do you honestly believe the bots will not use port 25 to send spam directly? Security requires layers. And it is a bit shocking how many people do not realize this. -- TTFN, patrick

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sun, Sep 5, 2010 at 6:13 PM, Franck Martin <franck@genius.com> wrote:

In many countries, the presence of bots consume a non-trivial amount of bandwidth. In developing countries, this is a non trivial amount of $$$ (http://mobile.slashdot.org/story/10/09/05/1620212/UN-Tech-Group-Finds-Mo st-Expensive-Broadband)

Blocking port 25 allows to help identify which hosts are consuming bandwidth (likely to have a bot). Identifying and removing these hosts from the network is crucial and economically viable, unfortunately these are skills sometimes not available in such countries.

Just saying...

It looks like Germany is now all aboard a program regarding this: http://www.cio.com.au/article/359491/germany_launch_antibotnet_program_cons umers/ FYI, - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.5.3 (Build 5003) wj8DBQFMhEYkq1pz9mNUZTMRAkA0AJ44e0uMzlR4iXwpZAE8RQSUM3NmegCg4kV1 WOyT+X0/daNa92gGYG53Rtg= =yMOX -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson  Engineering Architecture for the Internet  fergdawgster(at)gmail.com  ferg's tech blog: http://fergdawg.blogspot.com/

On Sep 5, 2010, at 6:18 PM, Jon Lewis wrote:

On Sun, 5 Sep 2010, Claudio Lapidus wrote:

...

...

If I block port 25 on my network, no spam will originate from it. (probablly) The spammers will move on to a network that doesn't block their crap. As long as there are such open networks, spam will be rampant. If, overnight, every network filtered port 25, spam would all but disappear. But spam would not completely disappear -- it would just be coming from known mailservers :-) thus enters outbound scanning and the frustrated user complaints from poorly tuned systems...

That won't be probably the case. Here recently we conducted a rather comprehensive analysis on dns activity from subscribers, and we've found that in IP ranges that already have outgoing 25 blocked we were still getting complaints about originating spam. It turned out that the bots also know how to send through webmail, so port 25 blocking renders ineffective there.

Anti-spam is a never ending arms race. Originally, the default config for most SMTP servers was to relay for anyone. 10 years ago, sending spam through open SMTP relays was quite common. Eventually, the default changed, nearly all SMTP relays now restrict access by either client IP or password authentication, and the spammers adapted to open proxies. Today, nobody in their right mind sets up an open HTTP proxy, because if they do, it'll be found and abused by spammers in no time. These too have mostly been eliminated, so the spammers had to adapt again, this time to botted end user systems.

Getting rid of the vast majority of open relays and open proxies didn't solve the spam problem, but there'd be more ways to send spam if those methods were still generally available. The idea that doing away with open relays and proxies was ineffective, so we may as well not have done and should go back to deploying open relays and open proxies it is silly.

Doing away with open relays and open proxies didn't really interfere with legitimate traffic on a meaningful level. Blocking outbound SMTP is causing such problems. If a better job was done of blocking only 25, perhaps this would be less so. Unfortunately, many hotel networks and such are doing one or more of the following: Blocking ALL SMTP ports (25, 465, 587) Blocking SSH in some cases (fortunately rare, rendering the SMTP thing mostly easy to work around) Blocking IMAPs (while leaving IMAP open?!?) Blocking POP3s (while leaving POP3 open?!?) Blocking just about everything except 80 and 443 The absolute worst ones are proxying ALL SMTP traffic to their server whether it is the address you tried to relay through or not. Generally the ones that have done this have cited the complaints they got from outright blocking SMTP as the reason they felt the need to do so. When I pointed out that not blocking SMTP and only blocking 25 could be a viable alternative, they basically laughed at me. The question isn't just what is or isn't effective, or, even how much it reduces spam complaints. There is also the question of how much legitimate traffic suffers collateral damage in your spam mitiigation techniques. Owen

With all the different webmail systems, it seems unlikely to me (though I definitely wouldn't say impossible) that bots are spamming through your webmail (unless you work for gmail, hotmail, etc. and are an attractive enough target that it made sense to code a bot to automate utilizing your webmail interface).  Bots being used as proxies seems far more likely to me for the general case of "bots" spamming through an ISP's webmail.

Many providers and hosts use the same webmail packages so the work to automate is a bit lower than one might think. We have seen bots sending spam using our squirrelmail and roundcube webmail using credentials gleaned from phishing activity.

On Sun, Sep 05, 2010 at 09:18:54PM -0400, Jon Lewis wrote:

Anti-spam is a never ending arms race.

That's really the question at hand here -- whether or not there's any benefit to continuing the "never ending arms race" game. Some people think there is. Others question whether anything is really being accomplished. Certainly we're playing it out like an arms race -- ISPs block something, spammers find a new way to inject spam, and so on. The end result of lots of time spend on blocking thins, less functionality for customers ... but no decrease in spam.

Originally, the default config for most SMTP servers was to relay for anyone. 10 years ago, sending spam through open SMTP relays was quite common. Eventually, the default changed, nearly all SMTP relays now restrict access by either client IP or password authentication, and the spammers adapted to open proxies. Today, nobody in their right mind sets up an open HTTP proxy, because if they do, it'll be found and abused by spammers in no time. These too have mostly been eliminated, so the spammers had to adapt again, this time to botted end user systems.

Getting rid of the vast majority of open relays and open proxies didn't solve the spam problem, but there'd be more ways to send spam if those methods were still generally available. The idea that doing away with open relays and proxies was ineffective, so we may as well not have done and should go back to deploying open relays and open proxies it is silly.

Is it? It's likely true that the amount of span sent through open relays today is smaller than the amount of spam send through open relays 10 years ago. If the objective is "less spam via open relays", closing down open relays was a raging success. But that's not the objective. The objective is less spam, and there's certainly not less spam today than there was 10 years ago. Of course, those who worked to close open relays might argue that there would be even more spam today if there were still open relays. But they don't know that and there's no real evidence to support that. The theory behind closing open relays, blocking port 25, etc., seems to be: (a) That will make it harder on spammers, and that will reduce spam -- some of the spammers will find other other ways to inject spam, but some will just stop, OR (b) Eventually, we'll find technical solutions to *all* the ways spam is injected, and then there will be no more spam. There's little evidence for either. -- Brett

Having worked in past @ 3 large ISPs with residential customer pools I can tell you we saw a very direct drop in spam issues when we blocked port 25. -jim Sent from my BlackBerry device on the Rogers Wireless Network -----Original Message----- From: "Patrick W. Gilmore" <patrick@ianai.net> Date: Mon, 6 Sep 2010 17:54:49 To: NANOG list<nanog@nanog.org> Subject: Re: ISP port blocking practice On Sep 6, 2010, at 9:22 AM, Brett Frankenberger wrote:

On Sun, Sep 05, 2010 at 09:18:54PM -0400, Jon Lewis wrote:

...

...

Getting rid of the vast majority of open relays and open proxies didn't solve the spam problem, but there'd be more ways to send spam if those methods were still generally available. The idea that doing away with open relays and proxies was ineffective, so we may as well not have done and should go back to deploying open relays and open proxies it is silly.

Is it? It's likely true that the amount of span sent through open relays today is smaller than the amount of spam send through open relays 10 years ago. If the objective is "less spam via open relays", closing down open relays was a raging success. But that's not the objective. The objective is less spam, and there's certainly not less spam today than there was 10 years ago.

Of course, those who worked to close open relays might argue that there would be even more spam today if there were still open relays. But they don't know that and there's no real evidence to support that.

You are incorrect. There is vast evidence that closing open relays resulted in less spam. You can do a very simple experiment to satisfy your own curiosity. Open your SMTP host or HTTP proxy, wait a couple days and see what happens.

The theory behind closing open relays, blocking port 25, etc., seems to be: (a) That will make it harder on spammers, and that will reduce spam -- some of the spammers will find other other ways to inject spam, but some will just stop, OR (b) Eventually, we'll find technical solutions to *all* the ways spam is injected, and then there will be no more spam.

To be clear, even if there were not "vast evidence" blocking port 25 helped lower spam loads (and there_is_), it should still be filtered on residential / dynamic pools. There is more DDoS today than ever before. I guess we should all enable directed broadcast again. Miscreants aren't using smurf attacks (or at least I haven't seen it, therefore it doesn't exist, right?), and there are other tons of other ways to DDoS people. So we should just open them back up, right? If that doesn't sound ridiculously stupid to you, then you know nothing of DDoS fighting either. And if it does sound stupid to you, .. well, I think you get the point.

There's little evidence for either.

You are wrong. If you do not actually know something (and "I haven't heard of it" or "my friends don't like it" or "I don't see how ..." does not equal "I -know-"), then please refrain from making factual sounding statements. [Yeah, yeah, this is NANOG. Chances of that happening are nil. But at least the people who are willing to make such statements are self-identifying for easy future reference.] -- TTFN, patrick

The theory behind closing open relays, blocking port 25, etc., seems to be: (a) That will make it harder on spammers, and that will reduce spam -- some of the spammers will find other other ways to inject spam, but some will just stop, OR (b) Eventually, we'll find technical solutions to *all* the ways spam is injected, and then there will be no more spam.

There's little evidence for either.

the empirical evicence seems to support this theory. spam has continually increased, wich occasional dips when the spammers have to find a new path when an old one is covered. i suspect that, if we opened smtp relays again, unblocked 25 for consumer chokeband, etc., total spam received would likely increase a bit. but my guess, and i mean guess, is that the limiting parameter could well be how many bots the perps can get, not how well those bots are blocked. but, let's look at the upside. with the dying economy, the spam war provides employment for many of our friends. randy

No. It'd just increase a LOT, astronomically.

...

i suspect that, if we opened smtp relays again, unblocked 25 for consumer chokeband, etc., total spam received would likely increase a bit.  but my guess, and i mean guess, is that the limiting parameter could well be how many bots the perps can get, not how well those bots are blocked.

i keep hearing that, but am having a hard time finding supporting data. randy

...

i keep hearing that, but am having a hard time finding supporting data.

Might see the stats from http://cbl.abuseat.org - by AS. Then compare the stats on a non port 25 filtered network (they have stats by AS) to stats on a network that is filtered on port 25

The networks that are filtered on port 25 will of course have any bots on that network originating spam by other means (social networks, webmail scripting etc), or other types of nastiness (DDoS etc). But you won't find them mailing out direct on port 25.

The bots are very much there - and if the port 25 filtering were to be taken out, you'd at once see the increase in spam volumes.

thank you. this make sense. i'm more focused on the receiving end and signatures. so more of the same spam does not bother me much. but i can see that pure count would matter to many. randy

On Sep 5, 2010, at 10:36 AM, Claudio Lapidus wrote:

Hello all,

On Fri, Sep 3, 2010 at 11:30 PM, Ricky Beam <jfbeam@gmail.com> wrote:

...

If I block port 25 on my network, no spam will originate from it. (probablly) The spammers will move on to a network that doesn't block their crap. As long as there are such open networks, spam will be rampant. If, overnight, every network filtered port 25, spam would all but disappear. But spam would not completely disappear -- it would just be coming from known mailservers :-) thus enters outbound scanning and the frustrated user complaints from poorly tuned systems...

That won't be probably the case. Here recently we conducted a rather comprehensive analysis on dns activity from subscribers, and we've found that in IP ranges that already have outgoing 25 blocked we were still getting complaints about originating spam. It turned out that the bots also know how to send through webmail, so port 25 blocking renders ineffective there.

--cl.

Perhaps a new BCP is coming from MAAWG suggesting we now block outbound port 80. Owen

Most of us tend to do only default settings,it would better if we dig better into our settings and apply stricter rules to enhance security Sent from my HTC HD2 on Android On 13 Sep 2010 13:55, "Brian Johnson" <bjohnson@drtel.com> wrote:

Brian J.

...

-----Original Message----- From: Ricky Beam [mailto:jfbeam@gmail.com] Sent: Friday, September 03, 2010 9:30 PM To: Owen DeLong; Patrick W. Gilmore Cc: NANOG list Subject: Re: ISP port blocking practice

On Fri, 03 Sep 2010 08:12:01 -0400, Owen DeLong <owen@delong.com> wrote:

...

Really? So, since so many ISPs are blocking port 25, there's lots less spam hitting our networks?

Less than there could be. It appears a lot less effective because there are so many ISPs not doing any blocking. Both of my residential connections are open, and always have been. (even dialup was unblocked. which I always found odd since the UUNET wholesale dialup agreement requires the RADIUS response contain a packet filter limiting port 25 to your mail server(s).)

If I block port 25 on my network, no spam will originate from it. (probablly) The spammers will move on to a network that doesn't block their crap. As long as there are such open networks, spam will be rampant. If, overnight, every network filtered port 25, spam would all but disappear. But spam would not completely disappear -- it would just be coming from known mailservers :-) thus enters outbound scanning and the frustrated user complaints from poorly tuned systems...

--Ricky

This is what we (network admins) get paid to do! If we are running a server that is a security risk to the net, then we can't complain when it gets filtered. It is our job to do our due diligence and ensure our servers are not spam hot-beds or open relays (or other bad stuff, etc...).

The port 25 blocking simply prevents the largest volume of hosts in an ISP network, the users, from being a spam delivery platform.

- Brian

CONFIDENTIALITY NOTICE: This email message, including any attachments, is

intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, copying, use, disclosure, or distribution is prohibited. If you are not

for the sole use of the the intended recipient, please

contact the sender by reply e-mail and destroy all copies of the original message. Thank you.

It may be a recommended practice from MAAWG, but, it's still damage to the network which is often routed around. It's a minor inconvenience to spammers and a slightly bigger problem for legitimate users. I don't see the win here. Just because they recommend it doesn't make it a good recommendation. MAAWG appears to have a single priority... Reducing spam by whatever means possible, regardless of cost or efficacy. Some of their recommendations (most, even) are good and useful. Some are easy to implement, ineffective, and ill-conceived. Outbound blocking of port 25 from people attempting to reach their home MTA/MSA with TLS and SMTP-AUTH just because they don't have a static address is an example of easy to implement, ineffective, and ill-conceived. Owen On Sep 2, 2010, at 8:56 PM, Franck Martin wrote:

Blocking outbound port 25 in certain conditions (mainly anything with a dynamic IPv4), is a recommended practice from MAAWG.org and others, they have a few useful documents for ISPs to deal with their network.

----- Original Message ----- From: "Owen DeLong" <owen@delong.com> To: "Zhiyun Qian" <zhiyunq@umich.edu> Cc: "NANOG list" <nanog@nanog.org> Sent: Friday, 3 September, 2010 3:48:20 PM Subject: Re: ISP port blocking practice

We should be seeking to stop damaging the network for ineffective anti spam measures (blocking outbound 25 for example) rather than to expand this practice to bidirectional brokenness.

Owen

Sent from my iPad

On Sep 3, 2010, at 12:25 PM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

I skimmed through these specs. They are useful but seems only related specific to IP spoofing prevention. I see that IP spoofing is part of the asymmetric routing story. But I was more thinking that given that IP spoofing is not widely adopted, the other defenses that they can more perhaps more easily implement is to block incoming traffic with source port 25 (if they already decided to block outgoing traffic with destination port 25). But according to our study, most of the ISPs didn't do that at the time of study (probably still true today).

-Zhiyun On Sep 2, 2010, at 9:20 PM, Suresh Ramasubramanian wrote:

...

BCP38 / RFC2827 were created specifically to address some quite similar problems. And googling either of those two strings on nanog will get you a lot of griping and/or reasons as to why these aren't being more widely adopted :)

--srs

On Fri, Sep 3, 2010 at 7:47 AM, Zhiyun Qian <zhiyunq@umich.edu> wrote:

...

Suresh, thanks for your interest. I see you've had a lot of experience in fighting spam, so you must have known this. Yes, I know this spamming technique has been around for a while. But it's surprising to see that the majority of the ISPs that we studied are still vulnerable to this attack. That probably indicates that it is not as widely known as we would expect. So I thought it would be beneficial to raise the awareness of the problem.

In terms of more results, the paper is the most detailed document we have. Otherwise, if you interested in the data that we collected (which ISPs or IP ranges are vulnerable to this attack). We can chat offline.

Regards. -Zhiyun

On Thu, Sep 02, 2010 at 04:59:57PM -0500, Zhiyun Qian wrote:

One of the high-level findings is that we developed probing techniques to verify that indeed most ISPs are only blocking 1) "outgoing traffic of destination port 25" instead of 2) "incoming traffic with source port 25", which means that these ISPs are vulnerable to the assymetric routing attack.

Folks interested in port blocking may also find useful another academic work we did a few years ago that sought to broadly characterize the prevalence of port blocking, albeit under the guise of neutrality: http://rbeverly.net/research/papers/truck-pam07.html While we found that email ports (e.g. 25, 110, 143) were more than twice as likely to be blocked than a control port, other ports such as 136 were more widely blocked (136 is an innocuous profile port, but often suffers collateral damage because it lies between the microsoft and netbios 135-139 ports). Also, the asymmetric spam problem is covered in some detail in our 2009 IMC spoofer paper: http://rbeverly.net/research/papers/spoofer-imc09.html rob

On Thu, 22 Oct 2009, Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:

My experience is that port 587 isn't used because ISPs block it out-of-hand. Or in the case of Rogers in (at least) Vancouver, hijack it with a proxy that filters out the AUTH parts of the EHLO response, making the whole point of using the submission service ... pointless.

You mentioned this last June. Can anyone else corroborate it? Rogers says they don't do that, and lots of other people seem to be able to use port 587 on Rogers (and other ISPs) without problems. All the cases I've looked at, where someone claimed an ISP was blocking port 587, it turned out to be some other problem. The most common reason was related to some security software/host based firewall running on the user's own computer.

Sean Donelan wrote:

On Thu, 22 Oct 2009, Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:

...

My experience is that port 587 isn't used because ISPs block it out-of-hand. Or in the case of Rogers in (at least) Vancouver, hijack it with a proxy that filters out the AUTH parts of the EHLO response, making the whole point of using the submission service ... pointless.

You mentioned this last June. Can anyone else corroborate it? Rogers says they don't do that, and lots of other people seem to be able to use port 587 on Rogers (and other ISPs) without problems.

All the cases I've looked at, where someone claimed an ISP was blocking port 587, it turned out to be some other problem. The most common reason was related to some security software/host based firewall running on the user's own computer.

Please pardon my ignorance here, especially if I've mistaken context... Although I'm not an email engineer, it was at *least* three-1/2 years ago that we switched our users from sending on port 25, to auth over 587 ('users' meaning the clients we have as an ISP/hosting company, which can establish on 587 from within OR without our network). Although I can recall a few edge cases that were brought to my attention for clients (users) not able to submit from a different network, the collateral damage has been ~1%. (ironically however, it seems as though recently those who have gone with a 'stick' have been reporting issues, and we've had to have them switch to port 25 on the SMTP server within the relevant network). I never even imagined that someone would do port 587 blocking as such. I'll have to light up on the network of the next client that complains, and if 587 doesn't get through, I'll start advising the helpdesk that they should redirect the former-client to call the 'ISP'. Someone please tell me that I'm misunderstanding something, so that I don't feel that two years of client notices, research and implementation, and another two years of dealing with manual support calls because they didn't 'see' the 400 warnings of email changes wasn't all a waste. fwiw to keep on topic, I block all 25-in with a small exception list for a few colo boxes, and our incoming MTA collection/filtering cluster. This 'in' ACL is placed on all PE hardware. The incoming mail cluster is attached to it's own PE, which ensures that everything from anywhere on the network eventually filters through this ACL, and that the core is always essentially clean. I then pretty much do the same out-25 on all PE hardware. Because my network is small, I manually/strictly manage the ACLs on any PE that is Internet/Peer facing. colo servers that require SMTP services is either connected to a PE designed to allow it, or is put into a VLAN designed for such. afair, I have only two clients remaining that I haven't forced into a /29 corner where I can SWIP their space, thereby using the 'you're responsible' hammer. Either way, even without the swip, I've made them well aware of the repercussions of failing to at least be attentive. Steve

On Oct 23, 2009, at 12:15 PM, Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:

As for outright blockage of port 587, I get this complaint from many of my clients while they are on the road. It seems hotels love to block it.

I travel a bit (used to a lot) and only found one place that proxied it. Never saw an outright block. A call to the support group actually got if fixed in about 45 minutes. Call and complain if it's broken. You are the customer at that point..... --Chris

Justin Shore wrote:

The vast majority, I'm sorry to say, used Microsoft Exchange which to the best of my knowledge doesn't support RFC2476.

You can configure exchange to use additional smtp virtual servers and bind them to specific ports. You can also require authentication to access the ports and you can restrict it to users. You can also enable it for STARTTLS. What I dont know is whether Exchange has or ever will support SMTPS which is strange considering many versions of outlook default to try that for any secured port other than 25.

I wish more people would use it though. My users wouldn't have cause to get so upset when I tell them that they have to pay monthly for a static IP to use tcp/25. It would reduce my hassles a wee bit.

I have many a time recommended consulting customers to follow up with their mail provider to see if they has any plans to support the rfc standard, but I dont share much enthusiasm for complete adoption. I do believe it is getting better.

Justin

On Thu, Oct 22, 2009 at 6:29 PM, Justin Shore <justin@justinshore.com>wrote:

I can also speak from experience from the enterprise customers of the consulting side of my SP that I worked with before returning to the SP. Not a one of them made use of the MSP port. The vast majority, I'm sorry to say, used Microsoft Exchange which to the best of my knowledge doesn't support RFC2476. I did a little Googling just now and couldn't find any hits to say they did either. Some utilized RPC-over-HTTP. Most at the time didn't, requiring direct SMTP access or VPN.

Depends what you mean by "support RFC2476". Exchange most definitely supports message submission on port 587. Whether it supports RFC2476 to the letter in terms of other requirements I don't know, but submission as a "client access" mechanism is fully supported, with all the usual defaults you'd expect (eg, auth required) Of course, that doesn't mean that it'll be enabled in a specific environment, or even if it is enabled, that it will be exposed from the firewall. Most larger corporates will be using Outlook with Exchange, and thus may only support RPC-over-HTTP as you've stated. Scott

On Oct 22, 2009, at 6:14 PM, Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:

My experience is that port 587 isn't used because ISPs block it out-of-hand. Or in the case of Rogers in (at least) Vancouver, hijack it with a proxy that filters out the AUTH parts of the EHLO response, making the whole point of using the submission service ... pointless.

We use 587 quite a lot (with SMTP Auth and SSL/TLS), and have found _very_ few places block or proxy it. We don't have any/many customers in Rogers service areas though. The biggest reason people don't use it is that it requires some thought and tweaking settings in the "advanced" tab areas of many email clients. Newer email clients are actually starting to look for submission port and SSL support and configuring it autmatically if they find it. Once it's set up correctly we've found customers really like it since their email "just works" in most places. --Chris

Chris Boyd wrote:

On Oct 22, 2009, at 6:14 PM, Lyndon Nerenberg (VE6BBM/VE7TFX) wrote:

...

My experience is that port 587 isn't used because ISPs block it out-of-hand. Or in the case of Rogers in (at least) Vancouver, hijack it with a proxy that filters out the AUTH parts of the EHLO response, making the whole point of using the submission service ... pointless.

We use 587 quite a lot (with SMTP Auth and SSL/TLS), and have found _very_ few places block or proxy it. We don't have any/many customers in Rogers service areas though.

The biggest reason people don't use it is that it requires some thought and tweaking settings in the "advanced" tab areas of many email clients. Newer email clients are actually starting to look for submission port and SSL support and configuring it autmatically if they find it.

Once it's set up correctly we've found customers really like it since their email "just works" in most places.

I completely agree, and after all was said and done, well worth the effort. Even today, if users use their age-old setup manual to set up an email application, they can receive, but not send. We know why immediately when they call in and state this, and we tell them to expect an email to fix it, and then send them something like this: http://eagle.ca/update/mail/Outlook_Express/index.html ...yes, believe it or not, even with the pictures, they will sometimes still get it wrong ;) Years in planning and implementation, but a good, large-scale learning exercise and the achievement of no port 25 that I'm very proud of. Steve

Michael Peddemors wrote:

On October 23, 2009, Steve Bertrand wrote:

...

http://eagle.ca/update/mail/Outlook_Express/index.html

...yes, believe it or not, even with the pictures, they will sometimes still get it wrong ;)

Years in planning and implementation, but a good, large-scale learning exercise and the achievement of no port 25 that I'm very proud of.

Steve

Congratulations, it would be nice if everyone got there, and we push all our clients to adopt such a strategy, but it is always surprising how many still fear.. change.. and the phone calls they fear may come from it.

Thanks. The phone calls is what we 'feared' the most. For most things, I'm able to come up with hackery/workarounds to enable change with no client impact, but not in this case. What we did was go on a massive PR campaign via email and web for nearly two years while I ran both 587 and 25 in parallel. Also, (for the most part), we'd have the users make the changes pro-actively during unrelated calls. Getting closer to the 'due date', I set up a in-band, on-the-side network of sensors that monitored for port 25 traffic across the network segments. The sensors had access to RADIUS and other systems that automated the task of retrieving the username (or client ident of some sort) who was using the IP in question during that time period. The results would then be emailed to me. Sometimes the support staff would make a few cold-calls here or there to further knock down the list when things were slow. Most of the domain hosting and non-resi clients have their own 'techs', so they were pretty good. Slowly but surely, I started blocking 25 on segments of the network. At this point, I'd say that we had about 80% coverage. On and after doomsday, the call volume wasn't overly bad (I think we had 6 staff at that time). Because we were very prepared (with the handy-dandy pictorials), calls incoming were exceptionally short: "yep, you can't send. Read this email we're about to send and you'll be good to go". We of course impounded into their minds that "oh, you didn't follow the instructions we've been sending for the last two years" for good measure. Collateral damage was minimalistic, but was quickly spotted via the sensors. Adjustments were made, and here we are. I'd have no fear in doing it again, now that I know what to expect :) Although we have only ~10k access users and on top of that ~400 hosted domains, I do believe that the effort can scale up to any scope, so long as the proper preparations are made in advance. I believe renumbering my network twice prior to that helped with keeping me sensible and realistic in how I needed to prepare though. Cheers, Steve

On Oct 22, 2009, at 3:39 PM, Justin Shore wrote:

Zhiyun Qian wrote:

...

Hi all, What is the common practice for enforcing port blocking policy (or what is the common practice for you and your ISP)? More specifically, when ISPs try to block certain outgoing port (port 25 for instance), they could do two rules: 1). For any outgoing traffic, if the destination port is 25, then drop the packets. 2). For any incoming traffic, if the source port is 25, then drop the packets.

I block on both generally. I block inbound and outbound for residential customers in dynamic pools. I block inbound only for residential with statically-assigned IPs. That way a customer can request (and pay for) a static IP and be able to get around out outbound SMTP block. Few companies use the MSP port (tcp/587). I'm not sure why more don't make the effort but most don't. To make up for that we allow static residential customers to evade that filter with a static IP. We still block inbound though. We also allow them to use our SMTP servers and SmartHosts if they want with no requirements on source domains (like some providers require, essentially requiring the customer to advertise for you). The inbound block isn't really all that useful as you elude to. However I use it more often than not to look for people scanning out ranges for open relays. I use that data for feed my RTBH trigger router and drop the spammer's traffic on the floor (or the poor, unfortunate owner of the compromised PC that's been 0wned.

Blocking ports that the end user has not asked for is bad. Doing it and refusing to unblock is worse. Some ISPs have the even worse practice of blocking 587 and a few even go to the horrible length to block 465. A few hotel gateways I have encountered are dumb enough to think they can block TCP/53 which is always fun.

We block several other things too. Netbios traffic gets dropped both ways. MS-SQL traffic gets dropped both ways (a few users have complained about this but very few stick to their guns when you point out that their traffic is traversing the web completely unencrypted). I block default and common proxy ports such as 3128, 7212, and 6588 in both directions. Squid is too easy to misconfigure (done it myself). GhostSurf and WinGate have both been abusable as open proxies in various releases. I also block 8000, 8080 and 8081 towards the customers. These are some of our most commonly scanned ports (usually all 3 at once plus some or all of the 80xx ones). I've encountered many compromised residential CPEs that the users either enabled themselves or were enabled by default. I don't block those 3 ports on outbound flows though; too many false positives.

And finally we also block several different types of ICMPs. First off we block ICMP fragments. Then we permit echo, echo-reply, packet-too-big, and time-exceeded. The rest get explicitly dropped. IPv6 will change this list dramatically. I haven't had time to research ICMPv6 thoroughly enough to say any more than that.

Basically I just pick out some of the really bad ports and block them. This gives me a wealth of data with which to null-route compromised PCs scanning my networks.

Lovely for you, but, not particularly helpful to your customers who may actually want to use some of those services. Owen

Isn't blocking any port against the idea of Net Neutrality? Justin Shore wrote:

Owen DeLong wrote:

...

Blocking ports that the end user has not asked for is bad.

I was going to ask for a clarification to make sure I read your statement correctly but then again it's short enough I really don't see any room to misinterpret it. Do you seriously think that a typical residential user has the required level of knowledge to call their SP and ask for them to block tcp/25, tcp & udp/1433 and 1434, and a whole list of common open proxy ports? While they're at it they might ask the SP to block the C&C ports for Bobax and Kraken. I'm sure all residential users know that they use ports 447 and 13789. If so then send me some of your users. You must be serving users around the MIT campus.

...

Doing it and refusing to unblock is worse.

How you you propose we pull a customer's dynamically-assigned IP out of a DHCP pool so we can treat it differently? Not all SPs use customer-facing AUTH. I can think of none that do for CATV though I'm sure someone will now point an oddball SP that I've never heard of before.

...

Some ISPs have the even worse practice of blocking 587 and a few even go to the horrible length to block 465.

I would call that a very bad practice. I haven't personally seen a mis-configured MTA listening on the MSP port so I don't think they can make he claim that the MSP port is a common security risk. I would call tcp/587 a very safe port to have traverse my network. I think those ISPs are either demonstrating willful ignorance or marketing malice.

...

A few hotel gateways I have encountered are dumb enough to think they can block TCP/53 which is always fun.

The hotel I stayed in 2 weeks ago that housed a GK class I took had just such a proxy. It screwed up DNS but even worse it completely hosed anything trying to tunnel over HTTP. OCS was dead in the water. My RPC-over-HTTP Outlook client couldn't work either. Fortunately they didn't mess with IPSec VPN or SSH. Either way it didn't matter much since the network was unusable (12 visible APs from room, all on overlapping 802.11b/g channels). The average throughput was .02Mbps.

...

Lovely for you, but, not particularly helpful to your customers who may actually want to use some of those services.

I take a hard line on this. I will not let the technical ignorance of the average residential user harm my other customers. There is absolutely no excuse for using Netbios or MS-SQL over the Internet outside of an encrypted tunnel. Any user smart enough to use a proxy is smart enough to pick a non-default port. Any residential user running a proxy server locally is in violation of our AUP anyway and will get warned and then terminated. My filtering helps 99.99% of my userbase. The .001% that find this basic security filter intolerable can speak with their wallets. They can find themselves another provider if they want to use those ports or pay for a business circuit where we filter very little on the assumption they as a business have the technical competence to handle basic security on their own. (The actual percentage of users that have raised concerns in the past 3 years is .0008%. I spoke with each of them and none decided to leave our service.)

We've been down the road of no customer-facing ingress ACLs. We've fought the battles of getting large swaths of IPs blacklisted because of a few users' technical incompetence. We've had large portions of our network null-routed in large SPs. Then we got our act together and stopped acting like those ISPs who we all love to bitch about, that do not manage their customer traffic, and are poor netizens of this shared resource we call the Internet. Our problems have all but gone away. Our residential and business users no longer call in on a daily basis to report blacklisting problems. We no longer have reachability issues with networks that got fed up with the abuse coming from our compromised users and null-routed us. I stand by our results as proof that what we're doing is right. Our customers seem to agree and that's what matters.

Justin

On 23/10/09 17:58 -0400, James R. Cutler wrote:

Blocking the well known port 25 does not block sending of mail. Or the message content.

It does block incoming SMTP traffic on that well known port.

I think the relevant neutrality principle is that traffic is not blocked by content.

My personal definition doesn't quite gel with that. You're deciding for the customer how they can use their connection, before you have any evidence of nefarious activity. Would you consider restricting a customer's outgoing port 25 traffic to a specific mail server a step over the net neutrality line? -- Dan White

On 23/10/09 17:43 -0500, Justin Shore wrote:

...

It does block incoming SMTP traffic on that well known port.

Then the customer should have bought a class of service that permits servers.

That justification is a slippery slope. At what point do you draw the line on what constitutes business use? Is running a web server business use? A mail server? What about a server which participates in a peer to peer network? VPN? I certainly think you're within your right as a network operator to determine what is business use. I don't happen to feel that running a protocol server in and of itself meets that definition.

...

Would you consider restricting a customer's outgoing port 25 traffic to a specific mail server a step over the net neutrality line?

I do this all the time. For example I don't let my customers send or receive mail (or any traffic for that matter) from prefixes originating from AS32311 (Colorado spammer Scott Richter). Now if I was blocking mail to dnc.org, gop.com, greenpeace.org, etc or restricting Vonage to .05% of my bandwidth then yeah that would violate net neutrality principles. The difference is one stifles speech and is anti-competitive. The other mitigates a network security and stability risk.

I think I worded my question a bit wrong. I meant to hypothetically propose a common scenario: You only allow your customers to connect to your SMTP server, and if they attempt to connect to *any* other SMTP server, they are blocked or redirected to your SMTP server. -- Dan White

On Oct 23, 2009, at 10:54 PM, Owen DeLong wrote:

On Oct 23, 2009, at 3:43 PM, Justin Shore wrote:

...

Dan White wrote:

...

On 23/10/09 17:58 -0400, James R. Cutler wrote:

...

Blocking the well known port 25 does not block sending of mail. Or the message content. It does block incoming SMTP traffic on that well known port.

Then the customer should have bought a class of service that permits servers.

Then you shouldn't be marketing what the customer bought as "Internet Access".

We disagree. But is this really the place to discuss what MARKETING people should be doing? :) Blocking port 25 is not, IMHO, a violation of Network Neutrality. I explained why in a very long, probably boring, post. Your definition of Network neutrality may differ. Which is fine, but doesn't make mine wrong. As for how it is marketed, well, I'm not even going to try to argue that. -- TTFN, patrick

On Fri, Oct 23, 2009 at 5:43 PM, Justin Shore <justin@justinshore.com> wrote:

[...]  Just because someone bought themselves a Camry doesn't mean that Toyota is deciding for them that they can't haul 1000lbs of concrete with it. [...]

Server does not necessarily equal business. A server that handles a few personal mailboxes for a residential user is not 1000lbs of concrete. Offhand, I can think of a lot of uses for various types of servers at a residence that don't require special business features, and are generallly low-traffic. Some people might be a little upset if they brought their brand new leased Camry home, to find their particular dealer had made an ad-hoc decision to weld the trunk shut, and didn't tell them about it directly and immediately, when advertising the vehicle. You want to haul a few groceries home? Shoulda asked for a "business" camry. Nevermind that the manufacturer has no separate product for that, it was a dealer's arbitrary decision to block that particular "port", anticipating customers would otherwise try to do evil things with it (like try to haul concrete). Anyways... like it or not.. blocking of outbound/inbound 25 may be common. But how common was the original question.. not 'is it a good idea?' or not. I would suggest that blocking the destination port 25, outgoing traffic from the end-user's point of view is the more preferred choice, it is more efficient, in that the block is closer to the source, and there are fewer wasted bits of traffic. -- -J

The original intent of Net Neutrality laws had nothing to do with blocking or not on random ports. It had to do with giving an unfair advantage to the provider in question to sell competing services. Much like anti-trust legislation doesn't stop a company from cornering a market, just stops them from using that Market Power to create artificial advantages in other market segments to the detriment of the public. Putting this into a hypothetical example: If your access to iTunes, NetFlix, Amazon, etc. is filtered / rate limited / whatever, but you can order On-Demand from the company who owns the same piece of <fiber| copper> which delivers the Internet, especially if that is the only broadband connection you can buy because of monopoly / duopoly protection, then this is deemed "unfair". Anti-net-neutrality people spin this into "but suppose I want to charge extra for premium access to YouTube?" A != B, their argument does not address the underlying point of the rule. Do not be fooled by such spin. Back to the original question, no provider is blocking port 25 to force you to buy their mail services - they usually give it out for free with a connection! Add in the "reasonable network management / preventing abuse / best common practices" argument, which are backed up with 3rd party documents, and I don't think anyone rational would call this a violation of Network Neutrality. The problem is, not everyone is rational, and no law is written perfectly. Some spammer _will_ file a lawsuit claiming their spam is CAN-SPAM compliant, therefore legal, and the provider cannot block them. This will use legal resource, and may even result in un- blocking while a judge who knows what the box on his secretary's desk where his "e-mail" (huh?) gets printed is found. And that could take a while. Would that life were perfect. But it is not, so we muddle through as best we can. And I think we can do better than allowing a gov't sponsored monopoly use that monopoly to decide what things we can and cannot do based on what will raise their profit margin. One other personal comment: I believe if you paid for a resource, you should be allowed to dictate its use, even to your customers (as long as it is clear when they paid what the rules are). That said, there are no cable or DSL providers in the US who "paid" 100% for their resources, no matter what the executives at these companies think. This is not opinion, this is fact. OTOH: Companies who set up WiMAX or satellite or other technologies which do not depend on gov't grants, tax relief, right-of-way, monopoly protection, etc. should be allowed to do as they please. Yes, this includes filtering iTunes so you buy their movie streaming service. Of course, I have no idea if such a company exists, but it certainly is not impossible to set one up. I just don't know if it can make any money against companies that are funded by, granted exclusive rights by, or are helped in other ways by the gov't. I fear the answer is "no way in hell". All of this is IMHO, of course. -- TTFN, patrick On Oct 23, 2009, at 7:33 PM, James R. Cutler wrote:

No, blocking a port does not restrict a customers use of the network any more than one way streets restrict access to downtown stores. It just forces certain traffic directions in a bicycle/motorcycle/car/ van/truck neutral manner. Carry anything you want. Others laws restrict incendiary content.

On Oct 23, 2009, at 6:15 PM, Dan White wrote:

...

On 23/10/09 17:58 -0400, James R. Cutler wrote:

...

Blocking the well known port 25 does not block sending of mail. Or the message content.

It does block incoming SMTP traffic on that well known port.

...

I think the relevant neutrality principle is that traffic is not blocked by content.

My personal definition doesn't quite gel with that. You're deciding for the customer how they can use their connection, before you have any evidence of nefarious activity.

Would you consider restricting a customer's outgoing port 25 traffic to a specific mail server a step over the net neutrality line?

-- Dan White

James R. Cutler james.cutler@consultant.com

...

Isn't blocking any port against the idea of Net Neutrality?

Yes.

Owen

No. The idea of net neutrality, in this context, is for service providers to avoid making arbitrary decisions about the services that a customer will be allowed. Blocking 25, or 137-139, etc., are common steps taken to promote the security of the network. This is not an arbitrary decision (and I am defining it this way; I will not play semantics about "arbitrary". Read along and figure out what I mean.) For 25, SMTP has proven to be a protocol that has adapted poorly to modern life, and a variety of issues have conspired that make it undesirable to allow random home PC's to use 25. Reasonable alternatives exist, such as using 587, or the ISP's mail server. A customer isn't being disallowed the use of SMTP to send mail (which WOULD be a problem). A customer may use any number of other mail servers to send mail. Not a serious issue, and not arbitrary... it's generally considered a good, or even best current, practice. Blocking VoIP from your network to Vonage, because you want your customers to buy your own VoIP service? That's a very clear problem. There's no justifiable reason that any viable broadband service provider would have for blocking VoIP. Yet there could be a reason to forbid VoIP; I can, for example, imagine some of the rural WISP setups where the loads caused on the infrastructure interfere with providing service. Similarly, it'd be ridiculous to expect an 802.11b based rural WISP to be able to support HD Netflix streaming, or dialup ISP's to be able to support fast downloading of movies. These are not arbitrary restrictions, but rather technological ones. When you buy a 56k dialup, you should expect you won't get infinite speed. When you buy WISP access on a shared 802.11b setup, you should expect that you're sharing that theoretical max 11Mbps with other subs. It gets murkier when you get into situations such as where your cableco has sold you a 15Mbps Internet connection, but proceeds to "traffic engineer" your activities down to a slower speed. There are real questions that should be addressed; for example, if you are paying extra for a "premium" service (as in when the default speed is 7Mbps and you've upgraded), should a customer expect that they will actually get substantially more capacity? How does the reliance on overcommit affect things? The ideal is to sell a high speed connection to someone who uses none of it, of course... but if you're selling lots of capacity, and betting that only a little will be used at a time, and you've guessed wrong, the big question is, is that tolerable, or is net neutrality going to force you to provide what you've sold? So, now, back to blocking... many service providers block 80, on the basis that they don't want customers running servers. This could very well be a net neutrality issue. It's probably not a security issue. It's a decision being made at a business level, in order to promote the purchase of "business class" services. It's an arbitrary decision about what a customer will be allowed to do. There's lots of interesting stuff to think about. Net neutrality isn't going to mean that we kill BCP38 and port 25 filtering. It is about service providers arbitrarily interfering with the service that they're providing. Customers should be given, to the maximum extent reasonably possible, Internet connectivity suitable for general purpose use. Where service providers start infringing on that, that's what should be addressed by network neutrality. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Oct 24, 2009, at 3:17 AM, Joe Greco wrote:

...

...

...

Isn't blocking any port against the idea of Net Neutrality?

Yes.

Owen

No.

The idea of net neutrality, in this context, is for service providers to avoid making arbitrary decisions about the services that a customer will be allowed.

Right.

...

Blocking 25, or 137-139, etc., are common steps taken to promote the security of the network. This is not an arbitrary decision (and I am defining it this way; I will not play semantics about "arbitrary". Read along and figure out what I mean.) For 25, SMTP has proven to be a protocol that has adapted poorly to modern life, and a variety of issues have conspired that make it undesirable to allow random home PC's to use 25. Reasonable alternatives exist, such as using 587, or the ISP's mail server. A customer isn't being disallowed the use of SMTP to send mail (which WOULD be a problem). A customer may use any number of other mail servers to send mail. Not a serious issue, and not arbitrary... it's generally considered a good, or even best current, practice.

A common practice of breaking the network for your customers does not make the network any less broken and does not make the action network neutral

The SMTP protocol has adapted just fine. Certain operators of SMTP servers, on the other hand, are a different issue. I don't take exception if you want to block those SMTP servers. I do take exception if you block the protocol entirely.

587 is the exact same protocol as 25, just with different host configuration policies. As such, I would hold up 587 as an example to prove my point.

Except it doesn't. 587 is "submission done right"; whereas 25 is "transit." 587 and 25 are conceptually completely different, even if they use a common underlying protocol. That's why 587 not only does not prove your point, but it actually allows me to show that it isn't SMTP being interfered with, but rather just the uncontrolled submission of e-mail to remote machines. Does network neutrality mean that dialup operators will have to allow PPP users to connect without a login and password?

...

Blocking VoIP from your network to Vonage, because you want your customers to buy your own VoIP service? That's a very clear problem. There's no justifiable reason that any viable broadband service provider would have for blocking VoIP. Yet there could be a reason to forbid VoIP; I can, for example, imagine some of the rural WISP setups where the loads caused on the infrastructure interfere with providing service.

Some providers block outbound 25 to other email service providers because they want your outgoing email to go only through their own unauthenticated, unsecure mail servers. (I have had at least one former ISP refuse to unblock port 25 or 587 for me to a host that was running TLS and SMTPAUTH while they insisted that I use their port 25 server which did not listen on port 587 and would not accept TLS or SMTPAUTH).

Blocking 25 isn't a problem. Blocking 587 is. Requiring all e-mail to go through their servers is also a problem. That's because there is a good reason for the 25 blocking, one that you can trivially work around on 587. Blocking 587 is overreaching, and is dictating that you must use their servers. That is not neutral.

...

Similarly, it'd be ridiculous to expect an 802.11b based rural WISP to be able to support HD Netflix streaming, or dialup ISP's to be able to support fast downloading of movies. These are not arbitrary restrictions, but rather technological ones. When you buy a 56k dialup, you should expect you won't get infinite speed. When you buy WISP access on a shared 802.11b setup, you should expect that you're sharing that theoretical max 11Mbps with other subs.

Right... Those are not arbitrary, they are valid. Blocking all access to port 25 is, on the other hand, arbitrary.

It's not, because there is an obvious ongoing problem with infected end-user machines sending spam, and no particular reason that an end- user machine needs to be able to send e-mail to random remote sites. A huge amount of good is accomplished for the 'net as a whole when a service provider blocks 25. They're not preventing you from sending e-mail, they're just requiring that it be sent in a manner that complies with current community standards. And there are standards, and you can submit via 587 to alternative e-mail services of your choice. It is not entirely ideal, but it is laughable to construe 25 blocking as making it impossible (or even hard) to send e-mail, given that it most certainly isn't.

...

There's lots of interesting stuff to think about. Net neutrality isn't going to mean that we kill BCP38 and port 25 filtering. It is about service providers arbitrarily interfering with the service that they're providing. Customers should be given, to the maximum extent reasonably possible, Internet connectivity suitable for general purpose use. Where service providers start infringing on that, that's what should be addressed by network neutrality.

BCP-38 is good. SMTP blocking is not in BCP-38.

Not allowing a user to send forged packets is a perfectly legitimate action. Not allowing a user to send or receive valid packets properly formatted, carrying legitimate traffic for purposes which are not a violation of the providers AUP, on the other hand, is not good.

Oh. Really. But the problem is, you can't play both "BCP38 is good" and "25 blocking is bad." They're of the same cloth. If I'm assigned 24.1.2.3 by Comcast, and Comcast filters my ingress to prevent me from emitting other addresses, you claim that's fine because it's BCP38. There's a problem: I can validly emit a variety of other addresses, in particular any address in 206.55.64.0/20 and some other networks. I am not "forging" packets if I emit 206.55.64.0/20-sourced addresses down a Comcast pipe. How many people realistically have this problem? Well, potentially, lots. Anyone who uses a VPN could have a legitimate IP address on their machine; because of BCP38 (and other security policy) it is common for a VPN setup to forward Internet-bound traffic back to the VPN server rather than directly out the Internet. In some cases, one could reasonably argue that this is undesirable. But overall, security is greatly increased by eliminating the ability to inject forged traffic. We do this through BCP38. BCP38 carries with it some amount of inconvenience to users whose legitimate traffic can not be sent due to the simplistic filtering typically employed. This does not mean BCP38 violates net neutrality, any more than it means 25 blocking violates it. On the other hand, if your ISP is intercepting your DNS, forcing /all/ SMTP through their servers, mandating the use of web proxy servers that add banner ads, blocking VoIP, and RST'ing BitTorrent traffic, then you have a serious net neutrality problem. As operators, the readers in this group should be uniquely qualified to understand: common technical steps taken to ensure the security and continued smooth operation of your network are probably not violating net neutrality, but once you move into the realm of steps taken that damage a competitor, degrade or forbid particular services, or other decisions made for "business" reasons, where such things affect the set of potential things a user could reasonably expect to want to be able to do, then you have to look a bit more carefully at it. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Joe Greco wrote:

...

There's a problem: I can validly emit a variety of other addresses, in particular any address in 206.55.64.0/20 and some other networks. I am not "forging" packets if I emit 206.55.64.0/20-sourced addresses down a Comcast pipe.

How many people realistically have this problem? Well, potentially, lots. Anyone who uses a VPN could have a legitimate IP address on their machine; because of BCP38 (and other security policy) it is common for a VPN setup to forward Internet-bound traffic back to the VPN server rather than directly out the Internet. In some cases, one could reasonably argue that this is undesirable.

I would like to take the opportunity to urge vendors of routers and firewalls to take extra special care and attention to make sure that The Right Thing can always happen whenever multiple egress services are employed.

This means that policy routing for network AND ALL locally generated traffic should be available and work as the operator intends it to.

Right now things still suck pretty hard, depending on what you are using.

Who defines what "The Right Thing" is? Allowing (what are to the service provider) random IP's inbound, even if there's some mechanism to limit it, means that the ISP now has some additional responsibilities to be able to transport packets for space that isn't theirs; a transit upstream or peer might filter, especially for smaller service providers. Basically, allowing this dooms BCP38. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Oct 25, 2009, at 4:58 PM, Joe Greco wrote:

...

Joe Greco wrote:

...

There's a problem: I can validly emit a variety of other addresses, in particular any address in 206.55.64.0/20 and some other networks. I am not "forging" packets if I emit 206.55.64.0/20-sourced addresses down a Comcast pipe.

How many people realistically have this problem? Well, potentially, lots. Anyone who uses a VPN could have a legitimate IP address on their machine; because of BCP38 (and other security policy) it is common for a VPN setup to forward Internet-bound traffic back to the VPN server rather than directly out the Internet. In some cases, one could reasonably argue that this is undesirable.

I would like to take the opportunity to urge vendors of routers and firewalls to take extra special care and attention to make sure that The Right Thing can always happen whenever multiple egress services are employed.

This means that policy routing for network AND ALL locally generated traffic should be available and work as the operator intends it to.

Right now things still suck pretty hard, depending on what you are using.

Who defines what "The Right Thing" is?

Allowing (what are to the service provider) random IP's inbound, even if there's some mechanism to limit it, means that the ISP now has some additional responsibilities to be able to transport packets for space that isn't theirs; a transit upstream or peer might filter, especially for smaller service providers.

Basically, allowing this dooms BCP38.

Allowing the operator the configuration OPTION in all cases is good. Rational defaults in favor of BCP-38 are acceptable. The inability to override those defaults is bad. Owen

[tangent of interst for the archives] On Sat, Oct 24, 2009 at 02:07:42PM -0500, Joe Greco wrote: [snip]

If I'm assigned 24.1.2.3 by Comcast, and Comcast filters my ingress to prevent me from emitting other addresses, you claim that's fine because it's BCP38.

There's a problem: I can validly emit a variety of other addresses, in particular any address in 206.55.64.0/20 and some other networks. I am not "forging" packets if I emit 206.55.64.0/20-sourced addresses down a Comcast pipe.

Only in your service agreement allows this. Most folks realized both - the bad guys figured out this 'triangle routing' ages ago (was common to send bulk abuse traffic down broadband and receive the ack stream on dialup Back In The Day) and specificlly disallow it. - such hacks to attempt multihoming without BGP fail in spectacular ways nd can't be reled on for any real traffic. So while you may have an allocation and therefore not be 'forging' by strict definitions, you are injecting martian traffic as far as the resi broadband provider is concerned and it should be dropped. -- RSUC / GweepNet / Spunk / FnB / Usenix / SAGE

Folks, I would love to see the IETF OPSEC WG publish a Best Common Practices document on ISP Port filtering. The document would capture information similar to that offered by Justin. Would anybody on this list be willing to author an Internet Draft? Ron (co-director IETF O&M Area) Justin Shore wrote:

Zhiyun Qian wrote:

...

Hi all,

What is the common practice for enforcing port blocking policy (or what is the common practice for you and your ISP)? More specifically, when ISPs try to block certain outgoing port (port 25 for instance), they could do two rules: 1). For any outgoing traffic, if the destination port is 25, then drop the packets. 2). For any incoming traffic, if the source port is 25, then drop the packets.

I block on both generally. I block inbound and outbound for residential customers in dynamic pools. I block inbound only for residential with statically-assigned IPs. That way a customer can request (and pay for) a static IP and be able to get around out outbound SMTP block. Few companies use the MSP port (tcp/587). I'm not sure why more don't make the effort but most don't. To make up for that we allow static residential customers to evade that filter with a static IP. We still block inbound though. We also allow them to use our SMTP servers and SmartHosts if they want with no requirements on source domains (like some providers require, essentially requiring the customer to advertise for you). The inbound block isn't really all that useful as you elude to. However I use it more often than not to look for people scanning out ranges for open relays. I use that data for feed my RTBH trigger router and drop the spammer's traffic on the floor (or the poor, unfortunate owner of the compromised PC that's been 0wned.

We block several other things too. Netbios traffic gets dropped both ways. MS-SQL traffic gets dropped both ways (a few users have complained about this but very few stick to their guns when you point out that their traffic is traversing the web completely unencrypted). I block default and common proxy ports such as 3128, 7212, and 6588 in both directions. Squid is too easy to misconfigure (done it myself). GhostSurf and WinGate have both been abusable as open proxies in various releases. I also block 8000, 8080 and 8081 towards the customers. These are some of our most commonly scanned ports (usually all 3 at once plus some or all of the 80xx ones). I've encountered many compromised residential CPEs that the users either enabled themselves or were enabled by default. I don't block those 3 ports on outbound flows though; too many false positives.

And finally we also block several different types of ICMPs. First off we block ICMP fragments. Then we permit echo, echo-reply, packet-too-big, and time-exceeded. The rest get explicitly dropped. IPv6 will change this list dramatically. I haven't had time to research ICMPv6 thoroughly enough to say any more than that.

Basically I just pick out some of the really bad ports and block them. This gives me a wealth of data with which to null-route compromised PCs scanning my networks.

...

Also, is it common that the rules are based on tcp flags (e.g. SYN, SYN-ACK)? One would think block SYN packet is good enough.

I don't get that deep into it. Forged packets of types other than SYN can still reek havoc on existing flows. I think it's better to block all and move on.

Justin

.

On Nov 3, 2009, at 8:51 PM, mark [at] edgewire wrote:

Hi all,

Just out of curiosity for those whom may manage Hotel Wifi networks (I know I know, not really ISP level but since we're on the topic of port blocking). Does anyone actually make an effort to be blocking port 443? I've had that experience at a few Hotels in Philippines and I can't think of a valid reason as to why those ports would be dropping traffic. Would like to hear from anyone whom has had this experience.

I've found that some public (eg: Hospital) networks have very draconian security policies on their guest wireless. The University of Michigan hospitals block IMAP over SSL (tcp/993), SMTP-Submit (tcp/ 587) and all the vpn software I had at my disposal. This blocking is becoming more common to force people to HTTP/HTTPS ONLY based systems. They make utilizing these networks from a mobile device hard, and quickly forget your mac authentication quickly and are overall poorly run (no feedback loop to get things unblocked that are legit). I have found that I am having to vpn-out more often from these 'guest' networks to obtain "real" internet access. I recommend running a few gateways (eg: pptp, ipsec, openvpn) to get around these issues. (I have found some well run hotel networks that intercept tcp/3128 and send it to a local squid cache). - Jared

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Bertrand wrote:

Jon Kibler wrote:

...

To answer that question, I would start with ingress and egress filtering by IP address, protocol, etc.: 1) Never allow traffic to egress any subnet unless its source IP address is within that subnet range.

Sorry to nit, but shouldn't your uRPF setup take care of this (and many other of your list items), long before ACL?

It's absolutely great if you have your list implemented, but imho, all ISP's, no matter how small should investigate and implement urpf. It's especially fun to play with RTBH.

To be honest, the smaller you are, the easier it is to implement (ie. urpf strict everywhere! :)

Steve

Agree for the most part. However: 1) The overwhelming majority of routers I have audited do not have uRPF implemented and most admins do not comprehend it, but they do comprehend (usually) ACLs. 2) L3 switching does not always support it, leaving potential for abuse if the network has any donut holes. 3) uRPF works best on egress but does little on outside ingress (e.g., bogons). 4) Defense in depth dictates using more than one way to detect an attack, so use both ACLs and uRPF. Jon - -- Jon R. Kibler Chief Technical Officer Advanced Systems Engineering Technology, Inc. Charleston, SC USA o: 843-849-8214 c: 843-813-2924 s: 843-564-4224 s: JonRKibler e: Jon.Kibler@aset.com e: Jon.R.Kibler@gmail.com http://www.linkedin.com/in/jonrkibler My PGP Fingerprint is: BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkrhc+gACgkQUVxQRc85QlNAgACfZgrSuZ7dC1A38oIXB3lInUOc FnIAniWiQcVpJzp/ooh4LOHwEzPXUWo3 =dKbZ -----END PGP SIGNATURE----- ================================================== Filtered by: TRUSTEM.COM's Email Filtering Service http://www.trustem.com/ No Spam. No Viruses. Just Good Clean Email.

On Thu, 22 Oct 2009 22:36:13 EDT, Jon Kibler said:

4) Never allow traffic to ingress any network if the source address is bogus.

4a) Never flag a source address as bogus unless you can verify it is bogus *today*, not when you installed the filter. Out of date bogon filters are evil.