It need be neither momentous nor monumental - Just say it's 0.0.0.0 / 0 with some occasional exceptions. Regards Marshall Eubanks On Sat, 7 Feb 2004 11:56:28 -0500 "Wayne Gustavus (nanog)" <nanog@wgustavus.com> wrote:

This would essentially be impossible and not a good idea. Large volumes of hosts/zombies involved in such attacks originate from residential cable/dsl subscribers. This user base primarily uses dynamically assigned IP space. Hence, the IP of tonight's attacker could be the IP of tomorrow's legitimate user.

This is the same reason that it is imperative that any complaints sent to ISPs providing such services MUST have a time stamp (with timezone) along with other information relative to the attack/abuse. This is the only way the ISPs can relate the IP with the actual enduser in order to contact them for remediation.

___________________________________________________________ Wayne Gustavus, CCIE #7426 Operations Engineering Verizon Internet Services ___________________________________________________________

-----Original Message----- From: owner-nanog@merit.edu [mailto:owner-nanog@merit.edu] On Behalf Of Drew Weaver Sent: Friday, February 06, 2004 4:15 PM To: nanog@merit.edu Subject: Monumentous task of making a list of all DDoS Zombies.

Is there a list maintained anywhere of all hosts that have been identified as a DDoS zombie? Or attack box? We got hit with an attack from more than 60 IPs last night and I'd like to add them to any list that anyone has started.

Thanks,

-Drew

-----Original Message----- From: Suresh Ramasubramanian [mailto:suresh@outblaze.com] Sent: Saturday, February 07, 2004 9:58 PM To: Wayne Gustavus (nanog) Cc: 'Drew Weaver'; nanog@merit.edu Subject: Re: Monumentous task of making a list of all DDoS Zombies.

<snip>

1. It is arguable whether dynamic IPs are to be treated as legitimate mailhosts. Your colleagues in VOL mailops might tell you something similar too.

No argument there. However, the thread was originally addressing a list of DDoS Zombies, not illegitimate SMTP mailhosts. Arguably zombies used to launch DDoS attacks are treated differently than such hosts. We address both types.

2. An expiring list, where entries inserted are quickly expired, and stats used to add to other lists (such as MAPS DUL / SORBS DUHL) is a good idea, and moreover, it's already been done.

http://cbl.abuseat.org Interesting approach. It would be conceivable that if this resource was Widely used, miscreants could use this service to DDoS there victims without an army of zombies :-) I still submit that it is more advisable to address the root of the problem by finding the true host that generated attack traffic. Automating this process of matching dynamic IP to customer acct with a timestamp and remediation is the goal. __________________________________________________________ Wayne Gustavus, CCIE #7426 Operations Engineering Verizon Internet Services ___________________________________________________________

On Sun, 8 Feb 2004, Suresh Ramasubramanian wrote:

Another thing that helps with easier identification is a practice some ISPs have of inserting the MAC address of the host into the reverse DNS record, with a short TTL. When a new host gets that IP, the MAC address changes too. I have seen at least one ISP do this - and it makes it a whole lot easier for the ISP to quickly identify the host, rather than having to trawl through RADIUS logs or whatever else.

I've made proposals like this in the past, and have investigated some of the issues. I don't know if the world is ready to go that far yet. In practice MAC address tracking only works for a few very specific ISP architectures, such as when the ISP supplies the hardware used to connect to the network. Tracking MAC addresses ends up requiring having to trawl through RADIUS logs because users don't like having to tell the ISP everytime they change ethernet cards or computers. Most end-user home routers now include options to "clone" any MAC address to get around the MAC address requirement of a former (bankrupt) cable ISP. So you need to search which subscriber account was signed on with that MAC address during the suspect time period. Vendors aren't always that careful about assigning unique MAC addresses, and complaints aren't always that careful about reporting the correct MAC addresses. You still need the time information to verify the subscriber was actually online. For dialup users, which don't have ethernet MACs, do you put the user's home phone number in the reverse DNS records? How much privacy should users have or expect? One of the most common requests from law enforcement is how can they get a "unlisted" IP address. The same way they get an unlisted credit card number. Look at the hysteria over browser cookie "tracking" on the web. The "anti-Spyware" programs like to list lots of "spy" cookies which anonymously track visitors to web sites. Instead of Doubleclick tracking users with Cookies, they would be able to track the unique computers from the MAC address in the reverse DNS record over time. Or would this backfire, because then the hackers could find the vulnerable computers again and again even if the IP address changes. The hackers could scan the in-addr.arpa ranges looking for known vulnerable MAC addresses. Look, someone with a MAC address assigned to equipment known to be vulnerable. The problems are similar if the ISP assigns some other "unique" identifier to the subscriber and dynamically updates the reverse DNS record.

Then, there's the little matter of ISPs implementing ingress filtering as per BCP38 / RFC 2827. These DDoS zombies seem to also be used as a ready source of spoofed source addresses for attacks.

There are several ISPs which implement ingress filtering per BCP38/RFC2827. None of them have seen a change in the number of DDOS attacks. The people who track this kind of stuff say that most attacks do not use spoofed addresses. Unfortunately, the data is lacking about the effectiveness of any of these solutions. In the USA, the FDA requires drug producers to show new drugs are safe and effective before being sold to the public. There is no such requirement for people selling security solutions. - Block access from IP addresses without rDNS Data? - Insert MAC address into rDNS (negating previous block) Data? - Implement BCP 38 Data?

On 8-feb-04, at 8:27, Suresh Ramasubramanian wrote:

Of course, prevention is better than cure, so another recourse the ISP has is to be proactive - setting up a scanner to sweep the host that comes up on an IP the moment the dhcp server assigns it. If not a full blown portscan or anything, then at least a quick once-over that looks for signs of the current "big problem" trojans / zombies.

Coming up with new types of probes all the time to check for this would be a huge amount of work. I favor an approach where people no longer get to send data at high speed without the recipient's approval. Just sending data in the blind or any type of scanning could then trigger a severe rate limit or raise an alarm.

...

There are several ISPs which implement ingress filtering per BCP38/RFC2827. None of them have seen a change in the number of DDOS attacks. The people who track this kind of stuff say that most attacks do not use spoofed addresses.

I have heard from someone who hosts one of the mirrors for a site that is a DDoS magnet. I recall his saying that a non trivial number of attacks coming at this mirror were from spoofed source addresses.

People need to make sure only packets with legitimate source addresses escape from their network. Period. Unfortunately, this type of action must be performed at the source and some networks just can't be bothered.

On 8-feb-04, at 10:05, Suresh Ramasubramanian wrote:

...

Coming up with new types of probes all the time to check for this would be a huge amount of work.

Would that be any less work than clearing up the mess left by an infestation of DDoS zombies? :)

Apples and oranges. You need to clean up the zombies regardless of whether they succeeded in attacking the victim or they were stopped.

...

I favor an approach where people no longer get to send data at high speed without the recipient's approval. Just sending data in the blind or any type of scanning could then trigger a severe rate limit or raise an alarm.

It is fairly easy to work around rate limits by just scaling laterally, and compromising a few million more boxes. If the next virus grabs 4M, or 20M boxes instead of just a measly 2M boxes, you can rate limit all you like, bit it really won't help.

Help against what? You're right that if a million boxes send one 125 byte packet per second to the same place, that's still a gigabit worth of traffic, that particular place is going to receive a gigabit worth of traffic. But how are you going to infect a million boxes if you can only scan one address per second? And let's not be so blase assume that all DoS attacks are done with a million zombies at a time.

On Sun, 8 Feb 2004 18:12:46 +0100, Iljitsch van Beijnum <iljitsch@muada.com> writes:

But how are you going to infect a million boxes if you can only scan one address per second?

With a random scanning worm, the expected time could be as low as about a day. Assuming the random scanning model from the paper[1], I get: 0 time: 1 infected host. 11 hours to infect 1000 hosts. 25 hours to infect 800k hosts 31 hours to infect 996k hosts. This model assumes one scan per second per infected host. It is because if a million boxes are vulnerable, then one in 4096 IP addresses should be vulnerable. A random scan would find one such every 4096 seconds, implying a doubling time of about 70 minutes. Scott [1] http://www.icir.org/vern/papers/cdc-usenix-sec02/

...

...

...

...

"Mikael" == Mikael Abrahamsson <swmike@swm.pp.se> writes:

Mikael> On Sun, 8 Feb 2004, Suresh Ramasubramanian wrote: Mikael> I have asked about this before. Wouldnt it be very nice if Mikael> there was a standardized way to report IP-number and Mikael> timestamp and type of complaint? There isn't one yet. Some people are trying to put together a simplistic looking BCP - http://www.tmisnet.com/~strads/spam/bcp.html Mikael> I've seen something produced by some workgroup (RIPE?) but Mikael> that was a huge document about XML and it seemed Mikael> non-trivial to implement. I was more into the idea of Mikael> having basically email headers like: There is a RIPE WG on spam (I think chaired by Rodney Tillotson from JANET/CERT). But I don't recall something like this being proposed .. and XML is a rather unruly beast to manage, especially for joe user. Your idea of headers might work - or something on the lines of send-pr on *bsd. All that the NOC staff receiving it would require is that it stays simple, without stuff like : Frenzied abuse Screenshots from fancy IDS / software firewall products Long lectures on why spam / DDoS / other network abuse is bad A short two or three line summary of the issue, accurate timestamps and a set of excerpts from your logs (not a whole lot, just enough to make the situation obvious) should be enough. Another big help is giving the NOC access to a good ticketing system which understands the difference between customer support and net abuse handling (here, your customers are the problems, for starters). RT3 has a lot of code (courtesy Paul Vixie and the other people at MAPS who were hacking on it) - but there's a nice new product called Abacus - http://word-to-the-wise.com/abacus that looks promising. srs

I'm aware of these - but surely there's something about the user which you can stick into rDNS (hashed / encrypted if you like) that'll identify the user?

The problem with trojans etc is that there so damn many of them, so the less time spent actually tracking down the user who was on IP X at time Y, the better it is for the ISP's staffers who handle complaints about these.

It's not that hard, I assume we are talking about dial-up, cable and xDSL users? We already log all major radius events in a database and it's very easy to look up users in that db, we have a web page for CSR's (customer service representative's), additionally the mail server detects which of our ip ranges is sending worms and automatically disables those users... I see no gain from adding anything in DNS, like reverse records.

Of course, prevention is better than cure, so another recourse the ISP has is to be proactive - setting up a scanner to sweep the host that comes up on an IP the moment the dhcp server assigns it. If not a full blown portscan or anything, then at least a quick once-over that looks for signs of the current "big problem" trojans / zombies.

We perform this today, the problem is, what are the signs for "big problem" trojans and zombies? If there was a tool out there that could perform scanning of computers AND knew about what to look for (does this malware operate on fixed ports) AND could be automatically updated for new malware I would purchase such a tool. Other than scanning for the open ports, I think these zombies are regular open proxies... but that may (will?) change in the future.

4. Quick and immediate isolation of infected hosts - nullroute them, or maybe VLAN them into their own corner of the 'net, where the only thing they can access over http is an ISP support page saying "please un-root your computer, or contact us at 1-800-[foo] for help and more details"

We simply modify their passwords and log them the off today. There is also an entry created in the incident tracking system. But, we have it as a future goal to let them access some pages, like HouseCall etc. Rgds, -GSH

On Sun, 8 Feb 2004, Suresh Ramasubramanian wrote:

...

In practice MAC address tracking only works for a few very specific ISP architectures, such as when the ISP supplies the hardware used to connect to the network.

I'm aware of these - but surely there's something about the user which you can stick into rDNS (hashed / encrypted if you like) that'll identify the user?

But I still don't understand why an ISP unwilling to spend the money to trace uses with RADIUS or other existing methods; is going to want to spend money on interfacing their systems with Dynamic DNS servers and new systems to generate DNS cookies. It increases their cost, and doesn't provide any additional information which they have in their existing systems. On the other hand, if we don't care too much for the privacy implications it would benefit 3rd parties wanting to keep track of individual computers. It would help ISPs, because 3rd parties could take more effective action on their own to ignore traffic from particular computers. Digital rightes management, password guessing, IRC bans, mail blocks, etc could work much more effectively if ISPs provided a unique identifier for subscribers. If software and hardware vendors included a hard-coded unique identifier in every computer, it would be even more effective. Intel has proposed this in the past. Microsoft has a GUID concept for its software. But is the world really ready for this level of identification and tracking?

The problem with trojans etc is that there so damn many of them, so the less time spent actually tracking down the user who was on IP X at time Y, the better it is for the ISP's staffers who handle complaints about these.

As you point out, there are a lot of them. But the goal should be to NOT have the ISP's staffers handle individual complaints. Any "solution" which requires staff to assess and respond individually is not an improvement. That's why I proposed the ICMP Go Away message.

Of course, prevention is better than cure, so another recourse the ISP has is to be proactive - setting up a scanner to sweep the host that comes up on an IP the moment the dhcp server assigns it. If not a full blown portscan or anything, then at least a quick once-over that looks for signs of the current "big problem" trojans / zombies.

I assume you are aware that one of the fastest growing trojan segments includes trojans which can not be detected by port scanners. You are correct that prevention is better than the cure. Unfortunately you've misidentified the point of prevention. The software vendor is in the best position to prevent systems being compromised. A change at Microsoft can prevent 60-70 million computers a year from being vulnerable. As an ISP, even AOL can't fix that many computers.

I have heard from someone who hosts one of the mirrors for a site that is a DDoS magnet. I recall his saying that a non trivial number of attacks coming at this mirror were from spoofed source addresses.

The number of spoofed packets received has very little to do with the number of sources of spoofed packets. But again, it points out the lack of hard data. Yesterday, a red car cut me off, so obviously the problem is red cars and we should prohibit all red cars. Is there any difference in the number of attacks between networks which have deployed BCP38 and networks which haven't? Or perhaps the problem is with the computers connected to the networks, not the networks.

No, I don't claim that BCP38 is a magic bullet either. But I do put it to you that the way to at least mitigate this menace include a combination of several steps -

1. Easy identifying of hosts, at least to the ISP (to avoid privacy concerns)

By whom? Should anyone be able to identify any host any time, or is it only necessary for inter-connected providers to identify the next provider in the chain? Should end-users be complaining to their own provider (i.e. the ISP they are paying money) or calling 3rd party ISPs which have no method to identify who is making the complaint?

2. Sensible filtering practices

Filter for what? What is considered sensible?

3. Proactive network sweeps

Sweeps for what?

4. Quick and immediate isolation of infected hosts - nullroute them, or maybe VLAN them into their own corner of the 'net, where the only thing they can access over http is an ISP support page saying "please un-root your computer, or contact us at 1-800-[foo] for help and more details"

Of course you meant to say contact the person who sold you your computer for help fixing your computer. The police write tickets, they don't fix cars.

5. Cooperation with law enforcement if necessary, to track down and punish the DDoSer.

Of course, the original issue was PTR records for spam, not DDOS. But this isn't the first time people have changed in the middle of a thread. Which ISPs are not cooperating with law enforcement? In the US, if you receive harrassing or threatening phone calls, you have to file a police report. The telephone company only provides the information about the source of the calls to the police for followup. How many people file police reports for spam, ddos, etc.

Your staff will still get a ton of complaints. If these can be parsed by a script that looks for virus / trojan strings in the complaint,extracts the IP (or has your NOC dude just click the IP in his ticketing system, like in RT + IRTT) and the account just goes away - then fine.

So you want a major ISP to simply automatically disable accounts of its users based only on automated detection of an IP address and timestamp in something that APPEARS to be a complaint to an automated script? Do you want to start a pool to see how long it will take before the dictionary complaints start rolling in once such a system becomes publicly known? There is a reason why there are humans (overworked, unfortunately) handling abuse complaints. Make it easy, sure...but make it easy for the human to be able to properly inspect the complaint to see if it's legitimate BEFORE doing anything. But to the original issue of accountability. If an ISP can't write a simple tool to take an IP address & timestamp and spit out a username from radius logs, how do you expect them to implement a hash-based rdns tagging system? Steve ---- Steve Birnbaum SkyVision Global Networks Phone: +44 20 83871750 Email: steve.birnbaum@sky-vision.net Note that it is never the fall that kills, it's the landing.

SD> Date: Sun, 8 Feb 2004 02:01:29 -0500 (EST) SD> From: Sean Donelan SD> Instead of Doubleclick tracking users with Cookies, they SD> would be able to track the unique computers from the MAC SD> address in the reverse DNS record over time. A MAC address is six octets. Append time past Epoch when IP was assigned; that's another four octets. Append six random octets. Encrypt. Make hostname-friendly using %x equivalent. One now has 32 characters that contain the MAC address and time the DHCP lease (or whatever) began, yet are meaningless to those who lack the key. Consider periodically changing the six random octets to protect users with long DHCP leases. It's extra hassle, but one can clearly have tracking _and_ protect user privacy. That leaves the issue of users changing MAC address to evade detection. However, the aforementioned DNS issues have no bearing on this problem, which is a separate topic. Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses : blacklist@brics.com -or- alfra@intc.net -or- curbjmp@intc.net Sending mail to spambait addresses is a great way to get blocked.

SD> Date: Sun, 8 Feb 2004 17:43:34 -0500 (EST) SD> From: Sean Donelan SD> Again, why does an ISP need to spend the money and as you SD> point out the extra hassle, to do this? ISPs already have SD> all the information they need to trace a subscriber from the SD> IP address and timestamp. I'm not sure they need to for the MAC address example. I was pointing out that information contained in reverse DNS could be meaningful, but only to those who should know. Perhaps a better example would be to s/MAC address/user id/ and repeat the example. Then, instead of digging through logs, one could quickly decrypt the user ID. SD> We made this mistake once already by having /etc/passwd files SD> world-readable (encryption would protect the passwords). Totally wrong analogy. /etc/passwd was a one-way hash (useless for this situation)... SD> Don't repeat the mistake. If you suspect a particular ...using crypt(). Note that I never suggested use of weak crypto. SD> computer, know the time, how long would it take to SD> brute-force the remaining six characters? I can think of some frequently-encrypted data that begins with strings like "HTTP/1.1 200 OK". So which is a better starting point for key recovery? Eddy -- EverQuick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita _________________________________________________________________ DO NOT send mail to the following addresses : blacklist@brics.com -or- alfra@intc.net -or- curbjmp@intc.net Sending mail to spambait addresses is a great way to get blocked.