Brandon Galbraith wrote (on Sun, Jul 22, 2007 at 06:28:55PM -0500):

Agreed. If you're savvy enough to have a problem because of this, you're savvy enough to a) Use another set of DNS servers or b) Use your own local resolver.

-brandon

Oh. And when they implement Plan B (inspecting each DNS packet for IRC.* and substituting their own answer as a reply), then what? -- _________________________________________ Nachman Yaakov Ziskind, FSPA, LLM awacs@ziskind.us Attorney and Counselor-at-Law http://ziskind.us Economic Group Pension Services http://egps.com Actuaries and Employee Benefit Consultants

Brandon Galbraith wrote:

On 7/22/07, *Sean Donelan* wrote: DNS is just another application protocol that runs over IP. You don't have to use those DNS servers to resolve names.

Possibly, you do (based on experience).

Agreed. If you're savvy enough to have a problem because of this, you're savvy enough to a) Use another set of DNS servers or b) Use your own local resolver.

For awhile, Comcast blocked/redirected all DNS queries, sending them to their own servers. Then, their servers didn't work properly.... Comcast still blocks port 25. And last week, a locally well-known person was blocked from sending outgoing port 25 email to their servers from her home Comcast service. It took some days to find out that Comcast had (without any notice) turned off her outgoing email (Monday), due to spam complaints! Needless to say, her MacBook isn't sending spam -- but many thousands of folks have her email address in their (presumably infected M$) address books. The official response: We don't support Thunderbird. You could use web email instead. When you pull stunts like that, you shouldn't complain about legislation.

Quoting Sean Donelan <sean@donelan.com>:

On Sun, 22 Jul 2007, William Allen Simpson wrote:

...

Comcast still blocks port 25. And last week, a locally well-known person was blocked from sending outgoing port 25 email to their servers from her home Comcast service.

MSA port 587 is only 9 years old. I guess it takes some people longer than others to update their practices. Based on what I know how comcast's abuse systems implement their port 25 restrictions, I think it is extremely unlikely it was based on other people having her e-mail address in their Outlook programs.

Indeed. There's just not enough info to make anything but wild guesses about this.

Some people complain ISPs refuse to take action about abuse and compromised computers on their networks. On the other hand, people complain when ISPs take action about abuse and compromised computers on their networks. ISPs are pretty much damned if they do, and damned if they don't.

Gotta love the techie world :)

Several ISPs have been redirecting malware using IRC to "cleaning" servers for a couple of years trying to respond to the massive number of bots. On occasion they pick up C&C server which also contains some "legitimate" uses. Trying to come up with a good cleaning message for each protocol can be a challenge.

I'm still unsure that this is either a good idea or a bad idea... changing the DNS can only help until the bots start connecting directly to IP addresses. Then where do we go? NAT those connections to elsewhere? It's one of those lovely arms races where things just get more and more invasive. In the short term, it's a good thing - the amount of spam I get from their network has halved - which is great - however in the long term, the writers of this crudware will find another way to do business (web? ftp?).

Yes, false positives and false negatives are always an issue. People running sevaral famous block lists for spam and other abuse also made mistakes on occasion.

And these people have been flamed senseless. I like to think of it as a case of the work the blocklists do is excellent and saves many a network from being overrun by spam - however there is always collateral damage from things like this. The good far outweighs the bad however. -- Steven Haigh Email: netwiz@crc.id.au Web: http://www.crc.id.au Phone: (03) 9017 0597 - 0404 087 474

On Sun, 22 Jul 2007, Raymond L. Corbin wrote:

I agree. They are at least trying to clean up their network. If they are having a lot of problems with zombie bots that DDoS / Spam then this is a good way to stop it, for now. The small group of users can either use other nameservers or something like psybnc to connect if they want to get on IRC.

It doesn't seem to be rogue Cox engineers. Several major ISPs have all taken action against these particular IRC servers (not! IRC in general). They either re-direct the traffic to a cleaning server, or are blackholing the traffic completely. Yes, it could have been some type of false positive; but when multiple ISPs all start re-acting to something, I think there might be more to the story. Especially when those ISPs are noted for not responding to incidents. One ISP, it might be the ISP. Multiple ISPs, gotta start looking at what has them disturbed. Its hard to wake those dragons.

...

I'm still unsure that this is either a good idea or a bad idea... changing the DNS can only help until the bots start connecting directly to >IP addresses. Then where do we go? NAT those connections to elsewhere? It's >one of those lovely arms races where things just get more and more >invasive.

I don't foresee the programming of IP addresses instead of IP addresses.

That mainly indicates a lack of vision, including the inability to see what is currently going on.

Because if/when they are found and their exploited server is shut down, their dedicated server turned off for AUP violations etc they will loose access to all of the bots set to that IP address. This happens a lot and when it does they simply change the DNS.

Right. It's certainly convenient. However, it is pretty convenient to have a list of addresses to try (the code isn't even that hard), and so it isn't like wiping out a single IP address is going to solve the problem. In fact, it is pretty convenient to make a "downloadable list," so that it can be updated. We'll even conveniently pretend that this technology doesn't already exist.

...

And these people have been flamed senseless. I like to think of it as a case of the work the blocklists do is excellent and saves many a network from being overrun by spam - however there is always collateral damage from things like this. The good far outweighs the bad however.

I agree. They are at least trying to clean up their network. If they are having a lot of problems with zombie bots that DDoS / Spam then this is a good way to stop it, for now. The small group of users can either use other nameservers or something like psybnc to connect if they want to get on IRC.

So where do you draw the line? Do we start nameserving known phish domains? Suspected phish domains? Your competitor's web site? The instant you start feeling that it is okay to stop providing clear channel Internet access and start providing only a subset is the instant that you need to do some really careful examination of what you're up to and why. Pure blocking is less evil than interception and redirection. However, blocking a known legitimate IRC site is pretty nasty. Redirecting it somewhere else? Wow, that's pure evil, and I'd hope Cox gets it from both sides. We can break a lot of things in the name of "saving the Internet." That does not make it wise to do so. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Hiya, Plenty of boxes can do redirection in the middle such as Redback, Ellacoya etc. We redirect customers who are infected to a web page when the first connect. Then every few hours they get re-directed again, just enough so it's a bit annoying. If they ignore this for a few weeks, they get redirected more frequently :) -- Leigh Sean Donelan wrote:

On Sun, 22 Jul 2007, Joe Greco wrote:

...

We can break a lot of things in the name of "saving the Internet." That does not make it wise to do so.

Since the last time the subject of ISPs taking action and doing something about Bots, a lot of people came up with many ideas involving the ISP answering DNS queries with the addresses of ISP cleaning servers.

Just about every commercial WiFi hotspot and hotel login system uses a fake DNS server to redirect users to its login pages. Many universities use a fake DNS server to redirect student computers to cleaning sites.

What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up?

As far as I know, PPPOE is the only network access protocol that includes the feature of redirecting a host to a network operator's system; but Microsoft has decided not to implement it.

On Mon, 23 Jul 2007, Joe Greco wrote:

I think there's a bit of a difference, in that when you're using every commercial WiFi hotspot and hotel login system, that they redirect everything. Would you truly consider that to be the same thing as one of those services redirecting "www.cnn.com" to their own ad-filled news page?

Let's get "real." That's not what those ISPs are doing in this case. They aren't pretending to be the real IRC server (the redirected IRC server indicates its not the real one). The ISP isn't send ad-fill messages. The irc.foonet.com server clearly sends several cleaning commands used by several well-known, and very old, Bots. I might have given the server a different name, but its obviously not trying to impersonate the real irc server. Do you prefer ISPs to break everything, including the users VOIP service (can't call 9-1-1), e-mail service (can't contact the help desk), web service (can't look for help)? Or should the ISP only disrupt the minimum number of services needed to clean the Bot?

On Mon, 23 Jul 2007, Joe Greco wrote:

So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP.

If those users are so technically unsophisticated, do you really expect the other users with infected computers to figure out how to disinfect their computer and remove the Bots instead? So you have potentially tens of thousands of infected computers with Bots making connections to an IRC server. You know many of those bots are well-known, old bots that have built-in removal commands. But 99% of those users don't have the technical knowledge to clean their machine themselves or know what a Bot is. On the other hand, you have 1% of users are sophisticated enough to use IRC servers. And a few percentage of overlap between the two groups. What do you do? a. nothing b. terminate tens of thousands of user accounts (of users who are mostly "innocent" except their computer was compromised) c. block all IRC d. redirect IRC connections to a few servers known to be used by Bots e. something else

On Mon, 23 Jul 2007, Joe Greco wrote:

...

So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP.

If those users are so technically unsophisticated, do you really expect the other users with infected computers to figure out how to disinfect their computer and remove the Bots instead?

So you have potentially tens of thousands of infected computers with Bots making connections to an IRC server. You know many of those bots are well-known, old bots that have built-in removal commands. But 99% of those users don't have the technical knowledge to clean their machine themselves or know what a Bot is. On the other hand, you have 1% of users are sophisticated enough to use IRC servers. And a few percentage of overlap between the two groups.

What do you do? a. nothing b. terminate tens of thousands of user accounts (of users who are mostly "innocent" except their computer was compromised) c. block all IRC d. redirect IRC connections to a few servers known to be used by Bots e. something else

e. something else. Because: a. is wrong. b. doesn't fix the problem, merely shoots yourself in the foot. c. is impossible, stupid, and pointless to try. d. has been proven to be implemented incompetently by Cox. e. includes solutions known to work, including walled gardens, etc. Again, you do not need to hijack someone else's domain name and redirect portions of their namespace at one of your own servers in order to fix this problem. Well, of course, that assumes that you're technically competent. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

Hint: there is no bot. My traffic is being redirected regardless. Were I a Cox customer (and I'm not), I'd be rather ticked off.

Hint: the bots are on computers connecting to the irc server, not the irc server.

Hint: I know. As I said, for the challenged, THERE IS NO BOT. MY TRAFFIC IS BEING REDIRECTED REGARDLESS.

...

Interfering with services in order to clean a bot would be a much more plausible excuse if there was a bot. There is no bot.

So are you claiming no bots ever try to connect to that server?

I don't care if bots ever try to connect to that server. I can effectively stop the bots from connecting to servers by shutting down the Internet, but that doesn't make that solution reasonable or correct. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

...

So are you claiming no bots ever try to connect to that server?

I don't care if bots ever try to connect to that server. I can effectively stop the bots from connecting to servers by shutting down the Internet, but that doesn't make that solution reasonable or correct.

Do spam block lists only block spam e-mails, or do they on occasion sometimes block both legitimate e-mail and spam?

It depends on the list.

Yes, your IRC was probably listed by mistake. And more than likely someone will correct that mistake.

Yes, it is agravating to you personally because it is your server that was blocked by mistake.

Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots?

The practice of blocking public EFnet servers? Yes, when there are better solutions to the problem at hand. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

...

Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots?

The practice of blocking public EFnet servers?

As I've said multiple times, sometimes mistakes happen and the wrong things end up on a list. I doubt that was the intent.

Many people have suggested blocking C&C servers used by bots over the years.

There's a difference between blocking actual C&C servers and blocking general IRC servers that are incidentally being used as C&C servers.

...

Yes, when there are better solutions to the problem at hand.

Please enlighten me.

Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Wow, I didn't even have to strain myself. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

...

Please enlighten me.

Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set).

Wow, you are recommending ISPs wiretap their subscribers.

I suspect some privacy advocates will be upset with ISPs doing that.

"Some privacy advocates" will be upset with ISP's doing what Cox is doing. Maybe you missed that. If we assume that it is okay for Cox to actually intercept the IRC sessions of their users, we're wayyyy far into that mess anyways. I'm saying "do it right" if you're going to do it at all. Personally, I'd prefer that they didn't do it, but that set of solutions is more complex. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

"Some privacy advocates" will be upset with ISP's doing what Cox is doing. Maybe you missed that. If we assume that it is okay for Cox to actually intercept the IRC sessions of their users, we're wayyyy far into that mess anyways. I'm saying "do it right" if you're going to do it at all.

Would it be better if ISPs just blackholed certain IP addresses associated with Bot C&C servers instead of trying to give the user a message. That doesn't require examining the data content of any messages. The user just gets a connection timeout.

Compared to hijacking DNS and intercepting sessions? Yes. Absolutely. See, it isn't that hard to come up with better ideas.

...

Personally, I'd prefer that they didn't do it, but that set of solutions is more complex.

So it is better for ISPs to do nothing, than attempt something that isn't perfect.

Well, that's not what I said, now, is it. I did say that there's a set of solutions out there to deal with that.

Thanks. I'll remember that the next time someone complains about ISPs not caring about abuse or bots on networks.

Interestingly enough, some of us care. Some of us care enough to run clean networks AND to make sure that what we're selling isn't compromised by deliberate DNS hijackings and site redirections. Hmm. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set).

Wow, you are recommending ISPs wiretap their subscribers.

I suspect some privacy advocates will be upset with ISPs doing that.

Suppose I add a firewall rule to my router to block traffic to a particular port. Does my router thereby "wiretap" every packet passing through it because it needs to find out its destination port in order to determine if the rule applies or not? It is sometimes a tricky issue when you filter through legitimate traffic to stop illegitimate traffic. But a rule that this is always wiretapping of anything subjected to the automated inspection leads to ridiculous results. DS

On Mon, 23 Jul 2007, Joe Greco wrote:

...

...

...

Yes, when there are better solutions to the problem at hand.

Please enlighten me.

Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set).

Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner.

Mmmmm... okay. Would you like solution #1 or solution #2? (You can pay for solutions above and beyond that) Solution #1: you know you need to intercept "irc.vel.net", so you inject an IGP /32 route for the corresponding IP address, and run it through your IDS sensor. Now, you're not going to be able to claim that you actually have even 100Mbps of traffic bound for that destination, so that's well within the capabilities of modern IDS systems. This has the added benefit of not hijacking someone's namespace, AND not breaking normal communications with the remote site. Bonus points for doing it on Linux or FreeBSD and selectively port forwarding actual observed bot clients to a local cleansing IRCD, then mailing off the problem IP to support so they can contact the customer about AV and bot cleansing software, etc. Oh, you were going to claim that your routers can't handle a few extra /32 routes? I guess I have no help for you there. You win. So on to #2. Solution #2: You really can't handle a few extra /32 routes? Then go ahead, and hijack that DNS, but run it to a transparent proxy server that is in compliance with the ideas outlined above. Cost effective? One FreeBSD box, some freely available tools, and some time by some junior engineer with a little IRC experience, and you have a great tool available, which doesn't inconvenience legitimate users but does accomplish *MORE* than what Cox did.

Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'.

So I'm supposed to invent a solution that does WAY MORE than what Cox was trying to accomplish, and then you'll listen? Forget that (or pay me).

Additionally, please do NOT require in-line placement unless you can do complete end-to-end telco-level testing (loops, bit pattern testing, etc),

Ok.

also it'd be a good idea to have a sensible management interface for these devices (serial port 9600 8n1 at least along with a scriptable ssh/telnet/text-ish cli).

Ok.

Looking at DPI (which is required for your solution to work) you are still talking about paying about 500k/edge-device for a carrier-grade DPI solution that can reliably do +2gbps line-rate inspection and actions.

Yeah, I see that. Not. (I do see your blind spot, though.)

This quickly becomes non-cost-effective if your network is more than 1 edge device and less than 500k customers... Adding cost (operational cost you can only recover via increased user fees) is going to make this not deployable in any real network.

Uh huh.

...

Wow, I didn't even have to strain myself.

sarcasim aside, this isn't a simple problem and at scale the solutions trim down quickly away from anything that seems 'great' :( using DNS and/or routing tricks to circumvent known bad behaviours are the only solutions that seem to fall out. Yes they aren't subscriber specific, but you can't get to subscriber specific capabilities without a fairly large cost outlay.

That's not true. The problem is isolating the traffic in question. Since you DO NOT HAVE GIGABITS OF TRAFFIC destined for IRC servers, this becomes a Networking 101-style question. A /32 host route is going to be effective. Manipulating DNS is definitely the less desirable method, because it has the potential for breaking more things. But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On 7/24/07, Joe Greco <jgreco@ns.sol.net> wrote:

...

The problem is isolating the traffic in question. Since you DO NOT HAVE GIGABITS OF TRAFFIC destined for IRC servers, this becomes a Networking 101-style question. A /32 host route is going to be effective. Manipulating DNS is definitely the less desirable method, because it has the potential for breaking more things. But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did.

Yup - though I still dont see much point in specialcasing IRC.

This is probably true. However, in this case, apparently Cox felt there was some benefit to tackling this class of bot. My guess would have been that they were abandoned, and as such, there wouldn't have been much point to doing this. However, maybe that wasn't the case.

It would probably be much more cost effective in the long run to have something rather more comprehensive.

Sure, but that actually *is* more difficult. It isn't just a technical solution. It has to involve actual ongoing analysis of botnets, and how they operate, plus technical countermeasures. Are there ISP's who are willing to devote resources to that?

Yes there are a few bots around still using IRC but a lot of them have moved to other, better things (and there's fun "headless" bots too, hardcoded with instructions and let loose so there's no C&C, no centralized domain or dynamic dns for takedown.. you want to make a change? just release another bot into the wild).

Hardly unexpected. The continuing evolution is likely to be pretty scary. Disposables are nice, but the trouble and slowness in seeding makes them less valuable. I'm expecting that we'll see compartmentalized bots, where each bot has a small number of neighbors, a pseudo-scripting command language, extensible communication ABI to facilitate the latest in detection avoidance, and some basic logic to seed/pick neighbors that aren't local. Build in some strong encryption, have them each repeat the encrypted orders to their neighbors, and you have a structure that would be exceedingly difficult to deal with. Considering how long ago that sort of model was proposed, it is actually remarkable that it doesn't seem to have been perfected by now, and that we're still blocking IRC. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Tue, Jul 24, 2007 at 12:00:40PM -0500, Joe Greco wrote:

...

Yes there are a few bots around still using IRC but a lot of them have moved to other, better things (and there's fun "headless" bots too, hardcoded with instructions and let loose so there's no C&C, no centralized domain or dynamic dns for takedown.. you want to make a change? just release another bot into the wild).

Hardly unexpected. The continuing evolution is likely to be pretty scary. Disposables are nice, but the trouble and slowness in seeding makes them less valuable. I'm expecting that we'll see compartmentalized bots, where each bot has a small number of neighbors, a pseudo-scripting command language, extensible communication ABI to facilitate the latest in detection avoidance, and some basic logic to seed/pick neighbors that aren't local. Build in some strong encryption, have them each repeat the encrypted orders to their neighbors, and you have a structure that would be exceedingly difficult to deal with.

Considering how long ago that sort of model was proposed, it is actually remarkable that it doesn't seem to have been perfected by now, and that we're still blocking IRC.

Thats because there is a huge world out there of badly protected hosts just waiting to become bots and a fairly basic set of tactics being deployed to prevent them. ie until globally it is somewhat more difficult to build a botnet there is no need to develop complicated solutions. the simpler ones are proven, easy to roll out, easy to modify. its just supply and demand... Steve

On Jul 24, 2007, at 8:59 AM, Joe Greco wrote:

But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did.

Actually, it's requires a bit more planning and effort, especially if one gets into sinkholing and then reinjecting, which necessitates breaking out of the /32 routing loop post-analysis/-proxy. It can and is done, but performing DNS poisoning with an irchoneyd setup is quite a bit easier. And in terms of the amount of traffic headed towards the IRC servers in question - the miscreants DDoS one another's C&C servers all the time, so it pays to be careful what one sinkholes, backhauls, and re-injects not only in terms of current traffic, but likely traffic. In large networks, scale is also a barrier to deployment. Leveraging DNS can provide a pretty large footprint over the entire topology for less effort, IMHO. Also, it appears (I've no firsthand knowledge of this, only the same public discussions everyone else has seen) that the goal wasn't just to classify possibly-botted hosts, but to issue self-destruct commands for several bot variations which support this functionality. [Note: This is not intended as commentary as to whether or not the DNS poisoning in question was a Good or Bad Idea, just on the delta of effort and other operational considerations of DNS poisoning vs. sinkholing/re-injection.] Public reports that both Cox and Time-Warner performed this activity nearly simultaneously; was it a coordinated effort? Was this a one- time, short-term measure to try and de-bot some hosts? Does anyone have any insight as to whether this exercise has resulted in less undesirable activity on the networks in question? ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Culture eats strategy for breakfast. -- Ford Motor Company

On Jul 24, 2007, at 11:01 AM, Joe Greco wrote:

Since what I'm talking about is mainly IDS-style inspection of packets, combined with optional redirection of candidate hosts to a local "cleanser" IRCD, the only real problem is dumping outbound packets somewhere where the /32 routing loop would be avoided.

On a large network, particularly a network which hasn't already deployed a sinkhole/re-injection system which covers all the relevant portions of this topology, that's a lot of work. Transit networks are probably more likely than broadband access networks to've done so, IMHO (I've no knowledge of whether any of the relevant SPs have or haven't done so, that's just a general observation).

Presumably it isn't a substantial challenge for a network engineer to implement a policy route for packets from that box to the closest transit (even if it isn't an optimal path). It's only IRC, after all. ;-)

But we're talking about multiple destination points, and the static nature of [Cisco, at least] PBR doesn't always lend itself well to that kind of model. Multipoint GRE potentially does, but again, that's more infrastructure to plan and deploy.

Similar in complexity, just without the networking angle.

In much the same way that flying an airplane is similar to driving a car, just without the 35,000-feet-in-the-air angle. ;>

I don't see how what I suggest could be anything other than a benefit to the Internet community, when considering this situation.

I think sinkholing and re-injection is a very useful technique, and constantly exhort operators who haven't done so to implement it. My point was that if one hasn't implemented it, there's a substantial effort involved, one that's more complex than implementing DNS poisoning.

If your network is generating a gigabit of traffic towards an IRC server, and is forced to run it through an IDS that only has 100Mbps ports, then you've decreased the attack by 90%.

And one may've backhauled a gig of traffic across a portion of one's topology which can ill-afford it, causing collateral damage. Always a concern with any kind of redirection technology, it's important to monitor.

Your local clients break, because they're suddenly seeing 90% packet loss to the IRC server, and you now have a better incentive to fix the attack sources.

There are other incentives which are less traumatic, one hopes. ;>

Am I missing some angle there? I haven't spent a lot of time considering it.

See above. ;>

...

Yes, there is some truth there, especially in networks made up of independent autonomous systems. DNS redirection to a host would favor port redirection, so an undesirable side effect would be that all Cox customers connecting to irc.vel.net would have appeared to be coming from the same host. It is less effort, but more invasive.

Yes - sometimes more invasive methods are necessary, depending upon one's goal.

The point is that there are other ways to conduct such an exercise. In particular, I firmly believe that any time there is a decision to break legitimate services on the net, that we have an obligation to seriously consider the alternatives and the consequences.

Yes, but it's also very easy to second-guess what others are doing when not in full possession of the facts. None of us are, so it's probably a bit premature to speculate about someone else's chain of reasoning and then attack his logic, in the absence of any concrete information regarding same. ;> ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice Culture eats strategy for breakfast. -- Ford Motor Company

On Tue, 24 Jul 2007, Joe Greco wrote:

...

So I'm supposed to invent a solution that does WAY MORE than what Cox was trying to accomplish, and then you'll listen? Forget that (or pay me).

Since it was a false positive,

Fact not in evidence, as much as it'd be good if it were so. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Tue, 24 Jul 2007, Joe Greco wrote:

...

...

On Mon, 23 Jul 2007, Joe Greco wrote:

...

...

...

Yes, when there are better solutions to the problem at hand.

Please enlighten me.

Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set).

Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner.

Mmmmm... okay. Would you like solution #1 or solution #2? (You can pay for solutions above and beyond that)

I tried to be nice and non-sarcastic. I outlined requirements from a real network security professional on a large transit IP network. You completely glossed over most of it and assumed a host of things that weren't in the requirements. I'm sorry that i didn't get my point across to you, please have a nice day.

As far as "Please enlighten me" followed by "Please do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner" goes, I don't consider that to be non-sarcastic. I consider it to be either very rude, or perhaps a challenge. I attempted to reply in an approximately equivalent tone. But, now, what exactly did I gloss over? And what things did I assume that weren't in the requirements? It's already been demonstrated that it doesn't need to handle 1Gbps, 2Gbps, or 20Gbps, so those "requirements" are irrelevant. You then said:

Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'.

But I'm not about to be trapped into building a solution that does WAY MORE than what Cox was trying to do. That it was a requirement from a "real network security professional" is not relevant, as we're discussing ways to accomplish what Cox was trying, without the related breakage. You further said:

Additionally, please do NOT require in-line placement unless you can do complete end-to-end telco-level testing (loops, bit pattern testing, etc),

To which I said: "Ok.", because my solution meets that measure. It does not require in-line placement, condition met. You went on to say:

also it'd be a good idea to have a sensible management interface for these devices (serial port 9600 8n1 at least along with a scriptable ssh/telnet/text-ish cli).

And again I said: "Ok.", because my solution can be built on a FreeBSD or Linux box, and as a result, you gain those features too. Condition met. And finally, you say:

Looking at DPI (which is required for your solution to work) you are still talking about paying about 500k/edge-device for a carrier-grade DPI solution that can reliably do +2gbps line-rate inspection and actions.

And I finally said: "Yeah, I see that. Not." Because I don't fundamentally believe that you need to do deep packet inspection of all packets in order to accomplish what Cox was doing. So what exactly did I assume that wasn't in the requirements (and by that, I mean the requirements to do what Cox was attempting to do, not the requirements of some random "real network security professional")? If you really think I glossed over something that was important, then by all means, point it out and call me on it. Don't just say HAND. Part of network engineering is being a little bit clever. Brute force is great when you really need to move 20Gbps of traffic. Any idiot can buy big iron to move traffic. However, putting your face in front of the firehose is a bad way to take a sip of water. With that in mind, I will once again point out that doing the equivalent what Cox was trying to do did not /require/ the ability to do deep packet inspection at 20 gigabits per second, and as a result, I'm exercising my option of being a clever network engineer, rather than one who simply throws money and big iron at the problem. You asked for enlightenment. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Tue, 24 Jul 2007, Suresh Ramasubramanian wrote:

On 7/24/07, Chris L. Morrow <christopher.morrow@verizonbusiness.com> wrote:

...

Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. Additionally,

Right. However one consolation is that all this is edge filtering, outbound. And some of it can be pushed off onto the CPE (e&oe availability of patched CPE)

I'd love to see CPE dsl/cable-modem providers integrate with a 'service' that lists out 'bad' things. it'd be nice if the user could even tailor that list (just C&C or C&C + child-porn or C&C older not than X days/hours/minutes) ... I think it might even help, and be vendor agnostic (from a provide and hardware) perspective.

Outbound traffic volumes wont be as horrendously high as those inbound, and should be a bit easier to categorize than inbound traffic

DNS and routing tricks are the silliest, to some - but well, they work for a lot of low hanging fruit. And as you say, they are cost effective.

and they are at the control of the provider, which I think is also somewhat important, people don't know or don't want to police themselves (in general). Also, the false positive issue will still exist, regardless of where this 'service' exists. So we'll have to learn to deal with it and provide workarounds that grandma can do on her own without calling her vendor (equipment or service). -Chris

On 7/23/07, Joe Greco <jgreco@ns.sol.net> wrote:

All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box.

%age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable? Like anything else, its a numbers game.

On Mon, Jul 23, 2007 at 02:48:05PM -0500, Joe Greco wrote:

...

On 7/23/07, Joe Greco <jgreco@ns.sol.net> wrote:

...

All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box.

%age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable?

That's fairly irrelevant. The fact is that this isn't targetting infected boxes, it's targetting everyone.

its relevant because you specified freebsd and hence it becomes necessary to consider what % of users have freebsd boxes and how many of those are infected

...

Like anything else, its a numbers game.

All of computing is a numbers game. That doesn't make it right to go around breaking random services just because it might fix some random problem.

"right" .. whats that then? you're buying a product, you have T&Cs, you are protected by consumer law.. what moral of society is being breached for it not to be "right"? and neither the services are random or the problem. they are quite specific and the solution has been calculated to be the path of least resistance for the whole. you sound a lot like a consumer more than a network operator.. i'm not saying i would like what cox do if i were a consumer of theirs but they are dealing with an issue on their subscription service and they dont seem to be doing anything particularly radical do you have a better suggestion for them? incidentally, if you are a consumer and a tech-savvy one, why dont you just circumvent the restriction? Steve

On Mon, 23 Jul 2007 11:39:35 EDT, Sean Donelan said:

messages. The irc.foonet.com server clearly sends several cleaning commands used by several well-known, and very old, Bots.

Old and well-known bots. Remember that for a moment, and think "6 month old antivirus signatures" for a bit....

service (can't look for help)? Or should the ISP only disrupt the minimum number of services needed to clean the Bot?

Is there any indication that the commands actually pushed have a *significant* chance of actually wiping any resident bots, or is it "That's an old worn-out magic word" time? It's one thing if 95% of the time, hijacking the connection and pushing command strings actually cleans a bot up. It's another thing entirely if it only works 5 or 10% of the time because most of the bots currently out there are no longer susceptible to that cleaning method.

On 7/23/07, Sean Donelan <sean@donelan.com> wrote:

What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up?

Most large carriers that are also MAAWG members seem to be pushing walled gardens for this purpose. -- Suresh Ramasubramanian (ops.lists@gmail.com)

On 7/23/07, Sean Donelan <sean@donelan.com> wrote:

But, like other attempts to respond to network abuse (e.g. various block lists), sometimes there are false positives and mistakes. When it happens, you tweak the filters and undue the wrong block. Demanding zero chance of error before ISPs doing anything just means ISPs won't do anything.

Running email abuse desks for about a decade now makes me tend to agree with you .. and completely unfiltered pipes to the internet for customer broadband are a pipe dream, most places. -- Suresh Ramasubramanian (ops.lists@gmail.com)

On Mon, 23 Jul 2007 michael.dillon@bt.com wrote:

...

Running email abuse desks for about a decade now makes me tend to agree with you .. and completely unfiltered pipes to the internet for customer broadband are a pipe dream, most places.

If ISPs were able to standardize consumer Internet access services using a gateway box, then the necessary filtering could be done on the gateway which runs a secure OS. Of course its not too late to do this. Essentially all the consumer edge infrastructure needs to be upgraded to transition to IPv6. Rather than providing raw unfiltered Internet access over IPv6, ISPs could use a standard gateway box.

would you like that in black plastic? with a nice dial on top to spin? :)

When I say "standardize", I mean that ISPs could collectively work out the specs for such an IPv6 Internet gateway in the IETF along with vendors and other interested parties. Once a standard spec is agreed upon, vendors will make such boxes at the price-point that you need.

I think that was discussed in v6ops actually just 5 mins ago.

I would also expect that I can buy such a box and manage it myself if I choose, rather than having the ISP manage it for me as with most users.

but it connects to my network, and if you touch it you could damage my network... we could maybe get some legislation to fix this...

I would also expect the box to have no NAT, use real IPv6 addresses, and provide various firewall features to protect my home network better than an IPv4 NAT box without preventing me from using new peer-to-peer protocols like SIP.

See the v6ops draft on CPE security... maybe that's a step in the right direction? I'm sure the author would like some commentary.

* Sean Donelan:

On Sun, 22 Jul 2007, William Allen Simpson wrote:

...

Comcast still blocks port 25. And last week, a locally well-known person was blocked from sending outgoing port 25 email to their servers from her home Comcast service.

MSA port 587 is only 9 years old. I guess it takes some people longer than others to update their practices.

You missed the "to their servers" part (I don't think it's singular "they" 8-). At the intra-ISP level, submission vs. SMTP does not really matter because it's all local policy. If they block her on 25/TCP on their own servers, they can easily block her on 587/TCP, too.

On Mon, 23 Jul 2007, Patrick W. Gilmore wrote:

They can, but they do not. AFAIK, not a single ISP redirects port 587 to their own servers.

I work for an ISP that has totally blocked TCP/25 for all use. We require all users to use 587 (with authentication when connecting to our own mail system). We have substantially over 1M broadband users in 10-15 european countries (I don't know the exact number). This took planning, lots of information and HOWTOs to users, and some helpdesk backing to get into place, but it's done, and it works. It was less painful that we dreaded. Unfortunately we don't have internet operations in native english speaking country, so this will be in whatever it might autodetect your language to. http://www.tele2mail.com/ As you can see, there are configuration manuals for all major email clients, for instance outlook: http://www.tele2mail.com/manual/outlook/ So my recommendation is for other ISPs to do the same thing. Yes, I know IP providers should only move IP packets and don't care about the contents, but... well... you know. -- Mikael Abrahamsson email: swmike@swm.pp.se

On Mon, 23 Jul 2007, Jeroen Wunnink wrote:

It's a lot more trouble for hosting providers that provide customers with webhosting and E-mail services.

Why? What stops you from migrating them to TCP/587? I'd imagine direct TCP/25 access to your servers would be spotty at best, anyway. Where I'm at, there are more ISPs blocking TCP/25 to anything but their own email servers, that those who do not block. -- Mikael Abrahamsson email: swmike@swm.pp.se

* Patrick W. Gilmore:

IOW: ISPs have no real reason to stop port 587, they do have a reason (whether you agree it is sufficient or not) to filter port 25.

Sorry for being unclear: If I block 25/TCP to *my* *own* servers for a *customer*, I will make sure that I block 587/TCP as well. (Legitimate reasons would be infrastructure protection, for instance.) This subthread started by someone who reported that someone else had been prevent from accessing their ISP's smarthosts over 25/TCP. This is not a 25 vs 587 issue, AFAICT.

* steve.wilcox@packetrade.com (Stephen Wilcox) [Mon 23 Jul 2007, 01:21 CEST]:

...

It looks like cox is hijacking dns for irc servers. <snip> isn't there a law against hijacking dns? What can i do to persue this? no, its their network and they play by their rules.. the law would

On Sun, Jul 22, 2007 at 02:56:13PM -0700, Andrew Matthews wrote: prevent them from inserting data into 3rd party servers or from masquerading as someone they are not or other marketing unfairness (such as serving their site in place of their competitors).

You mean, like, sending you to their own machine instead of the actual irc.vel.net (a legit EFnet server) when you ask for it?

if you are a cox customer you might want to have a reasoned discussion with them and find out more details and whether you can reach a resolution. if they dont play ball tho you ultimately would have to vote with your $$ and switch..

This is a ridiculous argument as in many places there is only one game in town for affordable high speed internet for end users. -- Niels. -- "The Mac doesn't have a one-button mouse, it has a five-button mouse, with four of the buttons on the keyboard." -- Peter da Silva <peter@taronga.com>

On Jul 22, 2007, at 6:28 PM, Niels Bakker wrote:

...

...

if you are a cox customer you might want to have a reasoned discussion with them and find out more details and whether you can reach a resolution. if they dont play ball tho you ultimately would have to vote with your $$ and switch.. This is a ridiculous argument as in many places there is only one game in town for affordable high speed internet for end users.

Yes, but at least the incumbents have their cash cows protected (who me? cynical?)

However, you don't have to switch providers to run your own caching server. Unless Cox is intercepting all DNS queries (instead of just mucking about with the caching servers they operate), running your own caching server will likely solve the problem.

I'll accept that argument once you've explained to all your family members how to do it - and they've actually done it, successfully. Let's be real now. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

I'll accept that argument once you've explained to all your family members how to do it - and they've actually done it, successfully.

Let's be real now.

If we're going to be "real now," consider how rarely ISPs have done this over the last several years.

Its very hard to wake the dragon. Yes, ISPs can do all sorts of awful things, but the reality is most of the big ISPs are extremely conservative at taking any steps that disrupts customers traffic. While they sometimes make a mistake, it takes a lot to get big ISPs to do anything. Since 2005 when ISPs started doing this, how many false positives have come up?

I don't think it is "real" to think big ISPs are going to redirect customer traffic in order to steal customer credit card numbers or destroy a competitor.

I can't help but notice you totally avoided responding to what I wrote; I would have to take this to mean that you know that it is fundamentally unreasonable to expect users to set up their own recursers to work around ISP recurser brokenness (which is essentially what this is). That was my point. And, incidentally, I do consider this a false positive. If any average person might be tripped up by it, and we certainly have a lot of average users on IRC, then it's bad. So, the answer is, "at least one false positive." ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

...

And, incidentally, I do consider this a false positive. If any average person might be tripped up by it, and we certainly have a lot of average users on IRC, then it's bad. So, the answer is, "at least one false positive."

The only way any human activity will NEVER have a single false positive, i.e. mistake, is by never doing anything.

Do people really want ISPs not to do anything?

I'd prefer that ISP's tends towards taking no action when taking action has a strong probability of backfiring. For example, even if you had no clue that it was a legitimate EFNet IRC server, irc.vel.net is trivially Googleable and you can determine that it will therefore be used by various real users. Redirecting this would be a bad thing. On the flip side, redirecting irc.jgreco.net, because you found it in some bot's connection directory, when Googled, indicates that there are no matched documents. While this isn't conclusive proof that it won't break somebody, it is relatively much less likely to be a customer affecting issue. Since the domain is relatively new, it would be a lot more suspicious. You could even try connecting to it (if it existed) to see what the deal was. I would still be irate if someone owned a portion of my namespace in that manner, but as a relative comparison, I could see a much better case for it. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On 7/23/07, Sean Donelan <sean@donelan.com> wrote:

On Mon, 23 Jul 2007, Joe Greco wrote:

...

I'd prefer that ISP's tends towards taking no action when taking action has a strong probability of backfiring.

Everything has a chance of backfiring. So ISPs should take no action.

Please let me know how your next DDOS attack lasts.

We on EFnet take drones very seriously and do a very good job of cleaning them. I'd say we are probably one of the larger cleanest irc networks. 99% of ddos come from hacked drones running on C&C servers that are not large networks. They run their on ircd or use a tiny network where they will be unnoticed. I also run 2 undernet servers that have a much higher drone count. I don't see my servers over there hijacked. Now if i could find the legality of it, i would. Drew

Quoting Joe Greco <jgreco@ns.sol.net>:

...

...

On Mon, 23 Jul 2007, Joe Greco wrote:

...

And, incidentally, I do consider this a false positive. If any average person might be tripped up by it, and we certainly have a lot of average users on IRC, then it's bad. So, the answer is, "at least one false positive."

The only way any human activity will NEVER have a single false positive, i.e. mistake, is by never doing anything.

Do people really want ISPs not to do anything?

I'd prefer that ISP's tends towards taking no action when taking action has a strong probability of backfiring.

I'd have to say that at this point it is VERY obvious that you have never administered a large (100k users+) network.

You would be incorrect, by a large margin.

The procedures and paths of action you wish the largers ISPs to take are just not practical.

No, they're just a little more difficult. I realize that it's more complex to inject a blackhole host route into the IGP of your average large ISP than it is to wreak a little configuration havoc on some recursers. That doesn't make the easier solution correct.

From your web site: "Please Note: Be very certain that your alleged abuse incident actually originated here before submitting a complaint. Do not sumbit a complaint without full headers, logs, and timestamps. We are not a commercial ISP and it is highly unlikely that your abuse incident actually originated here."

Spelling mistakes and "under construction" pages from 2002 aside, it shows that you look to be familiar with dealing with smaller scale operations.

Yes, sol.net is not a commercial ISP. We're a small, very clean network that provides access to a limited number of other businesses. We're not selling $9.95/month DSL, and the businesses that actually live on our net and sell things have somewhat more "modern" web sites. They're highly vetted, and the last legitimate abuse incident fades in my recollection. However, since a lot of the services we run are still under the legacy domain name, I feel it appropriate to maintain some basic information and contact stuff under the sol.net domain name, even though we stopped using that for business purposes /many/ years ago, and it is used pretty much exclusively for network and other Internet infrastructure systems. Since I get to set the policies, we simply don't take dirty clients. However, we do take on a bunch of unusual things, and there is a sufficient supply of misdirected complaints that we've got a warning on the web page. You wouldn't believe the number of complaints we were getting about "hacking" back when we were serving up SpamCop's graphic images (which is approximately the era which caused us to add that little statement in red). And it doesn't really say anything about what I've done in the past, or what else I also do currently, so really, it might be best to tread rather more carefully. Now, if you want to engage in meaningless insults, I'll be happy to congratulate you on that gorgeous Apache 2 Test Page at crc.id.au ... "At least I have the decency to provide some public information on the network I run."

The reality of the matter is that large ISPs can do:

1) Nothing (which makes matters worse in the long run) 2) A disruptive fix (will get some false matches, a handful of IRCers vs 100k+ users is acceptable). 3) Kill accounts.

I see you conveniently left out walled gardens and other prudent and reasonable steps that ISP's and schools are successfully taking. I guess I didn't actually expect an impartial discussion, once you lowered yourself to speeling flamez.

Now lets look at a quick real world result of each of the three above.

1) Your network eventually caves into the ground. You end up being a host for many spam networks and other nasties. Everyone on the internet hates you.

2) A handful of people complain, cry, whimper, and leave. The number of users in this boat won't really have much of an effect on operations or business. Acceptable losses vs doing option 1.

3) You get a reputation of killing 'innocent' peoples accounts due to unknown infections of crud. Business declines, and you end up working for an ISP that would implement option 2.

And, as noted, you conveniently left out solutions that people actually have up and running today. Slick.

In reality, the "purist" ideals of Internet access just does not work.

Well, we're fine with the "purist" ideals over here. It helps to keep problems off the network in the first place. I realize that might not sit too well with ISP's that would rather take money than be a good net neighbour, but that doesn't make it any more right for them. It has more to do with choice than "does not work." ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Mon, 23 Jul 2007, Joe Greco wrote:

I can't help but notice you totally avoided responding to what I wrote; I would have to take this to mean that you know that it is fundamentally unreasonable to expect users to set up their own recursers to work around ISP recurser brokenness (which is essentially what this is).

Its more resonable to expect users to know how to remove bots and fix their compromised computers?

On Mon, Jul 23, 2007 at 12:44:07PM -0400, Sean Donelan wrote:

On Mon, 23 Jul 2007, Joe Greco wrote:

...

I can't help but notice you totally avoided responding to what I wrote; I would have to take this to mean that you know that it is fundamentally unreasonable to expect users to set up their own recursers to work around ISP recurser brokenness (which is essentially what this is).

Its more resonable to expect users to know how to remove bots and fix their compromised computers?

Is it more reasonable to expect bots to implement recursive nameservers? -- Ash bugud-gul durbatuluk agh burzum-ishi krimpatul. -- David W. Hankins "If you don't do it right the first time, Software Engineer you'll just have to do it again." Internet Systems Consortium, Inc. -- Jack T. Hankins

...

On Mon, 23 Jul 2007, Joe Greco wrote:

...

I can't help but notice you totally avoided responding to what I wrote; I would have to take this to mean that you know that it is fundamentally unreasonable to expect users to set up their own recursers to work around ISP recurser brokenness (which is essentially what this is).

Its more resonable to expect users to know how to remove bots and fix their compromised computers?

No amount of IRC redirection is going to remove bots and fix their compromised computers.

... JG

I disagree. A lot of the compromised computers are still using the old versions of like Phatbot, agobot, rxbot, all of which have the remove commands. Placing the .remove in the subject line will effectively remove the bots as they join the channels. The .remove will effectively completely remove the bot from their computer, not everything else, but alteast that bot instance is done. Its one way a lot of IRC networks get rid of the botnets started on their networks, simply glineing them causes them to keep trying to reconnect. Granted it won't stop the more experienced script kiddies, but it will certainly stop the ones who use the preconfigured scripts because they don't know what the soruce code means. As many have said this is more about numbers. The number of infected computers within their network used to DDoS and Spam compared to the number of legitimate IRC users. Unfortunately the number of zombies outweighs the good. Raymond Corbin Support Analyst HostMySite.com

No amount of IRC redirection is going to remove bots and fix their compromised computers.

... JG

Let's not confuse two different forms of IRC redirection, one which I think is perfectly okay and one which is definitely not okay. In the first type, the redirection is an immediate response to a threat that is not completely understood. Clients that connect to the target of the redirect are studied. Legitimate clients are separated from compromised ones as quickly as practically possible. Legitimate clients are given a message, if possible, that the service they are trying to use is temporarily unavailable due to a current threat. If there is some other way they can access the service (for example, direct entry of the IP), they are informed of that. Compromised clients are tracked and, where possible, notified. The current level of the threat is tracked, and when it is sufficiently minimal, the redirect is removed. This model is perfectly reasonable as a response to all kinds of threats. This includes cases where a traffic has to be blocked or redirected due to a new attack. The second form is the one that causes a problem. In this case, no attempt is made to study or understand the threat once a way to block it is found. Collateral damage is ignored, no effort is made to minimize inconvenience to legitimate users. No effort is made to notify compromised users. The filter or redirect is left in place indefinitely, until and unless a large number of complaints is received. The threat is not even monitored. The filter/redirect is considered a permanent solution. While it may officially be regarded as temporary, it is effectively left there and forgotten. Now these are two very different approaches to a threat. However, they both can involve hijacking, filtering, or redirection. I used to work as a consultant, helping companies negotiate contracts with ISPs. One thing I always did was make it clear that the first type of response was perfectly permissible, even if it harmed us, but the second was not, even if it did not harm us. DS

On Sun, Jul 22, 2007 at 02:56:13PM -0700, Andrew Matthews wrote:

...

It looks like cox is hijacking dns for irc servers.

<snip>

...

isn't there a law against hijacking dns? What can i do to persue this?

no, its their network and they play by their rules.. the law would prevent them from inserting data into 3rd party servers or from masquerading as someone they are not or other marketing unfairness (such as serving their site in place of their competitors).

Kinda like masquerading as an IRC server that they are not. The customer may have actually expected to go to that IRC server, and if not, they are in fact masquerading as someone they are not.

if you are a cox customer you might want to have a reasoned discussion with them and find out more details and whether you can reach a resolution. if they dont play ball tho you ultimately would have to vote with your $$ and switch..

That's not too practical. We've pretty much given up on competition in the broadband markets here in the US. It might be interesting to talk to a competent Internet lawyer, however. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On Jul 22, 2007, at 9:29 PM, Steven M. Bellovin wrote:

On Sun, 22 Jul 2007 14:56:13 -0700 "Andrew Matthews" <exstatica@gmail.com> wrote:

...

It looks like cox is hijacking dns for irc servers.

And people wonder why I support DNSsec....

Steve, One of us is confused. It might be me, but right now I think it's you. To be clear, here is the situation as I understand it: Cox has configured their recursive name servers such that when an end user queries the recursive server for a specific host name (names?), the recursive server responds with an IP address the host's owner did not configure. How exactly is DNSSEC going to stop them from doing this? -- TTFN, patrick

DNSSEC provides source authenticity and data integrity. You may get a bogus answer, but with DNSSEC in place at least you have a way of verifying the bogosity (is that a word?) of the reply. I agree with Steve, DNSSEC won't stop these tricks but it makes them detectable. I'm a Cox user at home but I have my Linksys home router configured to use DNS servers of my own choosing rather than Cox' choice. I also tunnel my email through SSH to a mail server I control so that I'm not blocked by their port 25 filters. Marc -----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Steven M. Bellovin Sent: Sunday, July 22, 2007 9:46 PM To: Patrick W. Gilmore Cc: nanog@merit.edu; Patrick W. Gilmore Subject: Re: DNS Hijacking by Cox On Sun, 22 Jul 2007 21:40:05 -0400 "Patrick W. Gilmore" <patrick@ianai.net> wrote:

On Jul 22, 2007, at 9:29 PM, Steven M. Bellovin wrote:

...

On Sun, 22 Jul 2007 14:56:13 -0700 "Andrew Matthews" <exstatica@gmail.com> wrote:

...

It looks like cox is hijacking dns for irc servers.

And people wonder why I support DNSsec....

Steve,

One of us is confused. It might be me, but right now I think it's you.

To be clear, here is the situation as I understand it: Cox has configured their recursive name servers such that when an end user queries the recursive server for a specific host name (names?), the recursive server responds with an IP address the host's owner did not configure.

How exactly is DNSSEC going to stop them from doing this?

If my host expects the response to be signed and it isn't, my host can scream bloody murder. The whole point of DNSSEC is to prevent random changes to DNS replies, whether by hackers or by ISPs. Yes, they can change it, but they can't change it without being caught. --Steve Bellovin, http://www.cs.columbia.edu/~smb

Several people have email me privately to disagree with my statement about DNSSEC, on various grounds. I stand by my statement, but I am making a fair number of assumptions, some perhaps invalid. Let me be less terse. I'm assuming fairly universal deployment. In other words, the root zone is signed, as is most or all of .com/.net/.org are signed and the .ccTLDs of your choice. I don't assume that domains under, say, .com are signed, but of course unsigned zones aren't protected. I assume that user machines have a configuration file with the root keys. (I'm carefully not saying anything here about how root key rollover is done.) I'm assuming that ISPs are *not* changing that configuration file; this may be a dubious assumption. One correspondent felt that it was; I stand by my statement, because if the ISP plays games there and the user falls to a pharming attack that DNSSEC may have prevented, the ISP may be liable. Besides, there's an easier attack for the ISP... DNSSEC can be end-to-end, or it can be terminated by a full-service caching server which has a security association with user resolver (typically via TSIG, which uses shared secrets). In that case, the caching server would verify the digital signatures and set a bit in the response to the user saying "all is cool". This is the most likely way an ISP would spoof a response, since it doesn't strip protection from other zones, doesn't require them to do massive resigning, etc. The way this is signaled by the end-user's resolver and handled by the caching name server is complex (see Section 3.2 of RFC 4035); briefly, though, it's ultimately under the control of the end system. I have no idea what the Windows default will be or if it will let the user's machine do its own validation. My guess, though, is that it will prefer ISP validation but be able to do it itself. Note that 4035 requires a secure channel (i.e., a shared secret) for upstream validation; since that won't exist out of the box, the PC would have to be able to do its own. For obvious reasons, I'm focusing on Windows here. It's 99% certain that Linux and *BSD machines can do their own checking, if they wish; I'll leave to others to speculate what MacOS will do. The net, though, under my assumptions, is that ISP-supplied user configurations will likely have the user's machine trust them, but sophisticated users will be able to override that -- and DNSSEC is very much something for sophisticated users. The ISP can always ignore the RFC and strip out the signature RRs. That's detectable by an end-system that has requested that they be sent along. After all, that's no different than a monkey-in-the-middle attacker doing the same thing -- DNSSEC doesn't distinguish between offensive ISP and other enemies... So -- I think that DNSSEC, if deployed, will protect users who care, even against their ISP. It won't protect the clueless; I'm not sure what will. (See http://didierstevens.wordpress.com/2007/05/07/is-your-pc-virus-free-get-it-i... for an example of people you can't protect with technology...) But it is, as the tube of toothpaste almost says, an effective attack-prevention mechanism if used as part of a program of good Internet hygiene and regular professional care.

James Hess wrote:

On 7/22/07, Steven M. Bellovin <smb@cs.columbia.edu> wrote:

I would suggest not underestimating the ingenuity and persistence of the bad guys to escalate the neverending war, when a new weapon is invented to use against them. If there's a way around it, history has shown, the new weapon quickly becomes worthless, you get to use it maybe for a month or two.

With my Undernet admin hat on, we have regular issues with botnets and the like for years and probably will for the foreseeable future. In my personal experience we see a new "crop" of script kiddies about every 6 months to a year. Generally they start with whatever publically available tools they can get their hands on and thus obvious tactics work well against them at this stage. However they soon learn to customize their bots to evade detection, some more successfully than others. Many of those then are persistent well after the original bot runner has gone back to school and given up on the bots. We have services detecting botnets in realtime and they just scroll past generally faster than you want to think about it (at least one a second). While I fully support people deciding to clean up their corner of the Internet, I'm not sure that this is the most effective way for cox to be doing it[1]. If you're interested in finding people that Undernet detects as being open proxies or such like, put an IDS rule looking for ":[^ ]* 465 [^ ]* :AUTO ". The interesting question is what to do about it. We can ban them, but they just either move them to another network, or disguise them to make them harder to find and ban.[2] Also the constant reconnects themselves can almost overwhelm a server. I almost want to submit patches to the botnet codebases to implement exponential back off, or infact /any/ kind of reasonable delay between connection attempts. We try reporting them to abuse@ contacts, generally good abuse@ contacts don't have many (any?) drones to report, and bad abuse@ contacts don't appear to care that they're causing others issues. So what would people on this list suggest we do? ---- [1]: On the other hand ff you are someone at cox that's knows what's going on with this dronetrap thing, send me an email, I'm interested in discussing how you can improve your dronetrap. I have Ideas. [2]: This is not to say we don't ban them, we do -- it's the only reasonable thing we've found to do. As I also believe in trying to post interesting/useful facts to this list a quick grep shows the current worst offenders (grouped by /24) being: 89.40.17.0/24, 89.40.18.0/24, 89.40.16.0/24, 208.98.39.0/24, 65.188.46.0/24, 195.144.253.0/24, 196.211.173.0/24, 66.178.177.0/24, 205.144.218.0/24. 65.188.43.0/24

On 7/23/07, Perry Lorier <perry@coders.net> wrote:

With my Undernet admin hat on, we have regular issues with botnets and the like for years and probably will for the foreseeable future.

In my personal experience we see a new "crop" of script kiddies about every 6 months to a year. Generally they start with whatever publically available tools they can get their hands on [...]

Agreed

doing it[1]. If you're interested in finding people that Undernet detects as being open proxies or such like, put an IDS rule looking for ":[^ ]* 465 [^ ]* :AUTO ".

I'm not so sure Undernet is the only IRC network to ever begin a banned reason message with the word "AUTO". I suspect it would be most useful if "detected drones" by most major IRC network would be visible to cooperating ISPs for further analysis, not just Undernet. What I would see as ideal is for some form of "detected BOT" incident reporting protocol to be utilized at both ends. Rather than use IDS to snoop for our "user banned" messages, let IRC networks run their automated bot detection methods, and when a bot is found, post that fact to a short-term real-time incident database of some sort that would limit visibility of the information (to the ISP responsible, or to someone asking a Yes/No question about a specific ip address and approximate date & time). The ISP would then be able to use the database as a geiger counter and apply more exhaustive (CPU-intensive) monitoring to the activities of a bot-reported connection for a short time, and ascertain if they can verify the "user is a bot" claim, and maybe take some abuse response if the user is actually infected with a spambot or floodbot. Due to the massive volume, automation would be a must, however. This relies on the IRC network's bot detection heuristics being kept up to date, but for the IRC networks' sake, they already have to be. And unfortunately also relies on someone maintaining a database, that could be costly, and would all be a waste if no ISP was able to utilize it to actually isolate bot-infected computers, or if no IRC network actually reported to the DB.

them harder to find and ban.[2] Also the constant reconnects themselves can almost overwhelm a server. I almost want to submit patches to the

Not just "almost"; the constant reconnects are themselves a DDoS against the server that has banned them, or the entire network, depending on the "target" of the reconnects. Merely perhaps, an unintentional one, rather than deliberate attack. Use of firewall rules in addition to ordinary K-Lines should serve well to mitigate some of the incidental reconnect spam's negative impacts on IRCD. Firewall rules are just impossible to implement across all servers on most IRC networks, as each server is administered independently, and contacts to make the necessary changes are not likely to all be available or on IRC at the same time. (Plus there's no automated tool I'm aware of an IRC network can use that says "mass-firewall all channel members of channel #botnet for 96 hours, on all servers, even if all admins of the IRC network ceded some firewall control to other admins)

botnet codebases to implement exponential back off, or infact /any/ kind of reasonable delay between connection attempts.

I suspect boxes IRC Servers run on should enforce something like "sane delay", such as with "new connection throttling" at the OS level. By that I mean, for instance, more than 3 connects to port 6667 in 60 seconds would be met with all SYN packets dropped from that source for a few minutes thereafter... Or the IRC server could be designed as to impose a delay before transmitting the 465 numeric to notify the bot that they are banned. Maybe you drop them from the IRC network, send the 465 numeric, and leave the connection open for a few minutes. Until the bot gets the disconnect, the probability of a reconnect is essentially 0%.... Malware is generally too dumb to notice the connection is "hung" and all its messages are silently going to /dev/null. I used to have an alternative to /KILL called /HURT or /HUSH, which instead of disconnecting a user, would cause IRCD to blackhole the session and yet pretend to hold the connection open. I never met a bot that automatically noticed this had happened and responded by dropping/attempting to reconnect... -- -J

James Hess wrote:

I suspect it would be most useful if "detected drones" by most major IRC network would be visible to cooperating ISPs for further analysis, not just Undernet.

I'd dare to say that most of us major networks hardly see a small percentage of the big botnets around, the miscreants have since a long time back learned to use own C&Cs where the connected IPs of a connected client is hidden from all but themselves. But it certainly would not hurt if there was a good way to report drones to ISPs and actually get some attention to the problem. A bunch of small streams quickly build up to a larger river in the end, I guess. Perhaps a larger issue for the ISPs is what to actually DO with their infected customers. To what extent is the ISP responsible for what their users do and how their computers are setup? I do not have a clear answer to that. Since almost every user is using the web a nice system could be to redirect reported PCs through a proxy the ISP controls where the user can get information about what to do about problems and at the same time still reach the Internet after chosing to click away the information; or something along those lines. -- /ahnberg.

Peter Dambier wrote:

The problem is, you dont know what is behind that probably NATted ip address. Probably you have 3 unix machines running smtp and uucp and a single infected windows box and maybe some VoIPs and ...

This is why I spoke of merely intercepting web traffic to inform, to not interrupt other services that may use the same link. I am in the same situation myself, sharing lots of stuff via the same fiber to my house. I even have TV through it. So I actually thought of that. And an ISP probably knows a bit more about their customer base than what we do, so this idea would ofcourse have to adapt to that. But as said, its a complicated matter and probably not a good idea either way before we know who is supposed to do what and for whom. -- /ahnberg.

On Tue, 24 Jul 2007, Mattias Ahnberg wrote:

But it certainly would not hurt if there was a good way to report drones to ISPs and actually get some attention to the problem. A bunch of small streams quickly build up to a larger river in the end, I guess.

I'd point at the IETF/INCH-WG work for a standard abuse@ reporting process/format... If you send hundreds or thousands of reports to an ISP abusedesk (or ISPs' abusedesks) a standard machine parsable format is a key ingredient. I think that atleast one ISP would love to see standards formatted reports about it's users (abuse@uu.net), provided that the appropriate information was included.

Since almost every user is using the web a nice system could be to redirect reported PCs through a proxy the ISP controls where

there are a few nice walled garden solutions... there isn't in general one-size-fits-all, and they aren't always easy to convince folks to deploy :) but sure.

Once upon a time, Steven M. Bellovin <smb@cs.columbia.edu> said:

Several people have email me privately to disagree with my statement about DNSSEC, on various grounds. I stand by my statement, but I am making a fair number of assumptions, some perhaps invalid. Let me be less terse.

Okay, so instead of changing the DNS record, they snoop it and redirect the IPs. What have you gained? How many IRC servers (especially those used by the botnets) use SSL, and how many clients validate the cert? -- Chris Adams <cmadams@hiwaay.net> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.

On Sun, 22 Jul 2007, Steven M. Bellovin wrote:

Yes, they can change it, but they can't change it without being caught.

also assuming your application understands a non-signed vs signed response... no apps currently do, aside from the FireFox plugin supported (I think) by Sparta still?