RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]
But suppose you put such a firewall in place. You'll need to configure the firewall properly -- paying as much attention to outbound rules as inbound.
Sounds like a good thing to document in a best practices document that can be used to certify firewall implementations. When trying to solve a social problem, techniques like the Good Housekeeping seal of approval are quite effective. As recommended by the editors of...
You'll need to add anti-virus software. And anti-spyware software. Then you need to make sure the "signature" databases for both of those are updated early and often,
What if the guidelines state that subscription and database oriented techniques for virus detection are not adequate and therefore not compliant. Only heuristic, capability-based systems are acceptable.
And you'll need to de-install IE and Outlook,
Thus ensuring that Firefox/Thunderbird will be the main target of the malware people. Is this necessarily any better? Note that Windows provides an extensive series of hooks which can be used by an application which wishes to subvert the normal operation of the OS. That subversive application could be the security monitor which is required by the ISP for Internet access because it is recommended in your guidelines.
Something which requires this much work just to make it through its first day online, while being used by J. Random Person, is hopelessly inadequate. Which is why systems like this are routinely compromised in huge numbers. Which is why we have a large-scale problem on our hands.
We live in a complex world. Computers are more complex than they were. OSes are more complex. Apps are more complex. Networks are more complex. And SOLUTIONS are more complex. But if the designers of computers, OSes, apps and networks can deal with the complexity, why can't security folks do likewise?
This left me with >1.5M observed hosts seen in a month. They're all sending spam. (How do I know? Because 100% of the mail traffic sent to that server is spam.)
What you did sounds dumb except that you said this is an experiment. Unfortunately, real live email servers do exactly the same, i.e. talk to all comers, because the email architecture is flat like a pancake. Some people consider this to be a Windows malware problem. I consider it to be an email architecture problem. We all know that you need hierarchy to scale networks and I submit that any email architecture without hierarchy is broken by design and no amount of ill-thought-out bandaids will fix it.
Pop quiz, bonus round: how much does it cost Comcast to defend its mail servers from Verizon's spam, and vice versa? Heck, how much does it cost Comcast to defend its mail servers from its own spam?
That actually sounds like an answerable question, if a company took it seriously enough. If the senders and receiver are both on your network, your finance department should be able to come up with some cost figures. --Michael Dillon
On Monday 19 February 2007 13:27, you wrote:
people consider this to be a Windows malware problem. I consider it to be an email architecture problem. We all know that you need hierarchy to scale networks and I submit that any email architecture without hierarchy is broken by design and no amount of ill-thought-out bandaids will fix it.
I look forward to your paper on "the end to end concept, and why it doesn't apply to email" ;) I'm not convinced there is an email architecture problem of relevance to the discussion. People mistake a security problem for its most visible symptoms. The SMTP based email system has many faults, but it seems only mildly stressed under the onslaught of millions of hosts attempting to subvert it. Most of the attempts to "fix" the architecture problem so far have moved the problem from blacklisting IP addresses, to blacklisting domains, or senders, or other entities which occupy a larger potential space than the IPv4 addresses, which one can use to effectively deal with most of the symptom. In comparison, people controlling malware botnets, have demonstrated their ability to completely DDoS significant chunks of network, suggesting perhaps that other protocols are potentially more vulnerable than SMTP, or more approrpiate layers to address the problem at. We may need a trust system to deal with identity within the existing email architecture, but I see no reason why that need be hierarchical, indeed attempts to build such hierarchical systems have often failed to gather a critical mass, but peer to peer trust systems have worked fine for decades for highly sensitive types of data. I simply don't believe the higher figures bandied about in the discussion for compromised hosts. Certainly Microsoft's malware team report a high level of trojans around, but they include things like the Jar files downloaded onto many PCs, that attempt to exploit a vulnerability that most people patched several years ago. Simply identifying your computer downloaded (as designed), but didn't run (because it was malformed), malware, isn't an infection, or of especial interest (other than indicating something about the frequency with which webservers attempt to deliver malware).
On Feb 19, 2007, at 6:04 AM, Simon Waters wrote:
I look forward to your paper on "the end to end concept, and why it doesn't apply to email"
The end-to-end principle has no bearing upon this discussion at all, unless you're referring to firewalls/NATs. ----------------------------------------------------------------------- Roland Dobbins <rdobbins@cisco.com> // 408.527.6376 voice The telephone demands complete participation. -- Marshall McLuhan
On Mon, Feb 19, 2007 at 02:04:13PM +0000, Simon Waters wrote:
I simply don't believe the higher figures bandied about in the discussion for compromised hosts. Certainly Microsoft's malware team report a high level of trojans around, but they include things like the Jar files downloaded onto many PCs, that attempt to exploit a vulnerability that most people patched several years ago. Simply identifying your computer downloaded (as designed), but didn't run (because it was malformed), malware, isn't an infection, or of especial interest (other than indicating something about the frequency with which webservers attempt to deliver malware).
I don't understand why you don't believe those numbers. The estimates that people are making are based on externally-observed known-hostile behavior by the systems in question: they're sending spam, performing SSH attacks, participating in botnets, controlling botnets, hosting spamvertised web sites, handling phisher DNS, etc. They're not based on things like mere downloads or similar. As Joe St. Sauver pointed out to me, "a million compromised systems a day is quite reasonable, actually (you can track it by rsync'ing copies of the CBL and cummulating the dotted quads over time)". So I'm genuinely baffled. I'd like someone to explain to me why this seems implausible. BTW #1: I'm not asserting that my little January experiment is the basis for such an estimate. It's not. It wasn't intended to be, otherwise I would have used a very different methodology. BTW #2: All of this leaves open an important and likely-unanswerable question: how many systems are compromised but not as yet manifesting any external sign of it? Certainly any competent adversary would hold a considerable fraction of its forces in reserve. (If it were me, that fraction would be at least "the majority".) ---Rsk
On Tue, 20 Feb 2007, Rich Kulawiec wrote: Hi Rich, <snip good stuff> thanks for your input, Rich. As always, quite interesting.
BTW #2: All of this leaves open an important and likely-unanswerable question: how many systems are compromised but not as yet manifesting any external sign of it? Certainly any competent adversary would hold a considerable fraction of its forces in reserve. (If it were me, that fraction would be at least "the majority".)
I stopped really counting bots a while back. I insisted, along with many friends, that counting botnets was what matters. When we reached thousands we gave that up. We often quoted anti nuclear weapons proliferation sentiments from the cold war, such as: "why be able to destroy the world a thousand times over if once is more than enough?" we often also changed it to say "3 times" as redudancy could be important. :> Today, it is clear the bad guys can get their hands on as many bots as they need, or in a more scary scenario, want. They don't need that many. As a prime example, I believe that VeriSign made it public that only 200 bots were used in the DNS amplification attacks against them last year. Even if they missed one, two or even three zeroes, it speaks quite a bit as to our fragile infrastructure. If brute force alone can acheive this, what of application attacks, perhaps even 0days? :) Still, we keep surviving and we will still be here next year, too, with bigger and bigger trucks and tubes to hold the Internet together, whether for regular or malicious usage. eCommerce and online banking might not survive in a few years if people such as us here don't keep doing what we do, but that part of it is off topic to NANOG. 10 years ago, almost no one knew what botnets were. Counting and measuring seemed to be very important 3 years ago, and to governments and academics, and even a year ago. Today it is just what funding for botnet research is based on ( :) ), still, I don't really see the relevance. Botnets are a serious issue, but they are only a sympthom of the problem called the Internet. Sitting on different networks and testing them for how many malicious scans happen every second/minute/hour/day and then checking that against how many machines with trivially exploited vulnerabilities exist on these networks can fill in some of the puzzlea, but the delta from what we may see if we consider email attachments and malicious web sites... The factor may be quite big. We will never be able to count how many bots exist. We can count limited parts of that pool such as those seen in spam. Those are several millions every day (which should be scary enough) but not quite the right number. And this is before we get into the academic off-topic discussion of what a bot actually is, which after almost 11 years of dealing with these I find difficult to define. Is it an IP address? A computer? Perhaps an instance of a bot (and every machine could have even hundreds). Welcome to the realm of Internet security operations and the different groups and folks involved (and now industry). It is about Internet security rather than this or that network security or this and that sample detection.
---Rsk
Gadi.
If you can't measure a problem, its difficult to tell if you are making things better or worse. On Tue, 20 Feb 2007, Rich Kulawiec wrote:
I don't understand why you don't believe those numbers. The estimates that people are making are based on externally-observed known-hostile behavior by the systems in question: they're sending spam, performing SSH attacks, participating in botnets, controlling botnets, hosting spamvertised web sites, handling phisher DNS, etc. They're not based on things like mere downloads or similar. As Joe St. Sauver pointed out to me, "a million compromised systems a day is quite reasonable, actually (you can track it by rsync'ing copies of the CBL and cummulating the dotted quads over time)".
Counting IP addresses tends to greatly overestimate and underestimate the problem of compromised machines. It tends to overestimate the problem in networks with large dynamic pools of IP addresses as a few compromised machines re-appear across multiple IP addresses. It tends to underestimate the problem in networks with small NAT pools with multiple machines sharing a few IP addresses. Differences between networks may reflect different address pool management algorithms rather than different infection rates. How do you measure if changes are actually making a difference?
On Wed, 21 Feb 2007, Sean Donelan wrote:
If you can't measure a problem, its difficult to tell if you are making things better or worse.
On Tue, 20 Feb 2007, Rich Kulawiec wrote:
I don't understand why you don't believe those numbers. The estimates that people are making are based on externally-observed known-hostile behavior by the systems in question: they're sending spam, performing SSH attacks, participating in botnets, controlling botnets, hosting spamvertised web sites, handling phisher DNS, etc. They're not based on things like mere downloads or similar. As Joe St. Sauver pointed out to me, "a million compromised systems a day is quite reasonable, actually (you can track it by rsync'ing copies of the CBL and cummulating the dotted quads over time)".
Counting IP addresses tends to greatly overestimate and underestimate the problem of compromised machines.
It tends to overestimate the problem in networks with large dynamic pools of IP addresses as a few compromised machines re-appear across multiple IP addresses. It tends to underestimate the problem in networks with small NAT pools with multiple machines sharing a few IP addresses. Differences between networks may reflect different address pool management algorithms rather than different infection rates.
How do you measure if changes are actually making a difference?
NAT on the one end, DHCP on the other. Time-based calculations along with OS/Client fingerprinting often seem to produce interesting results.
On Wed, Feb 21, 2007 at 12:31:30AM -0500, Sean Donelan wrote:
Counting IP addresses tends to greatly overestimate and underestimate the problem of compromised machines.
It tends to overestimate the problem in networks with large dynamic pools of IP addresses as a few compromised machines re-appear across multiple IP addresses. It tends to underestimate the problem in networks with small NAT pools with multiple machines sharing a few IP addresses. Differences between networks may reflect different address pool management algorithms rather than different infection rates.
Yes, but (I think) we already knew that. If the goal is to provide a minimum estimate, then we can ignore everything that might cause an underestimate (such as NAT). In order to avoid an overestimate, multiple techniques can be used. For example, observation from multiple points over a period of time much shorter than the average IP address lease time for dynamic pools, use of rDNS to identify static pools, use of rDNS to identify separate dynamic pools (e.g., a system which appears today inside hsd1.oh.comcast.net is highly unlike to show up tomorrow inside hsd1.nj.comcast.net), classification by OS type (which, BTW, is one way to detect multiple systems behind NAT), and so on. I think Gadi makes a good point: in one sense, the number doesn't really matter, because sufficiently clueful attackers can already lay their hands on enough to mount attacks worth paying attention to. On the other hand, I still think that it might be worth knowing, because I think "the fix" (or probably more accurately "fixes") (and this is optimistically assuming such exist) may well be very different if we have 50M than if we have 300M on our hands. ---Rsk
michael.dillon@bt.com wrote:
And you'll need to de-install IE and Outlook,
This will not happen. Not even remotely.
Thus ensuring that Firefox/Thunderbird will be the main target of the malware people. Is this necessarily any better? Note that Windows provides an extensive series of hooks which can be used by an application which wishes to subvert the normal operation of the OS. That subversive application could be the security monitor which is required by the ISP for Internet access because it is recommended in your guidelines.
I concur with ISP's looking for IE as some form of guideline. Stupid story... So I call Cox because for the 8mb down I am supposed to be getting, I was maxing out at 2mb, not a big deal. TechGirl: Can you go to your start menu... Me: No I don't use Windows TechGirl: Please hold TechGirl: (five minutes later) Are you using OSX? Me: No. Using Solaris, what would you like me to do? TechGirl: Please hold TechGirl: (minutes later) We don't support Solaris Me: What does an operating system have to do with lousy bandwidth... TechGirl: Please hold TechGirl: (minutes later) I have to escalate this to my manager TechGirl: Please hold Manager: Please go to your start menu... Me: No. As stated I'm not on Windows nor OSX. I use Solaris and I AM CONNECTED the service is horrible Manager: Well we only support Windows and OSX Me: (*ponders what this has to do with cruddy connectivity) Forget it... (Plugs in Windows laptop to make things easier). ISP's have come to rely on the bane of their client's issues. Asking someone to remove IE only to have their support group look for it is a nightmare in itself. Too many people have become so overdependent on Windows.
We live in a complex world. Computers are more complex than they were. OSes are more complex. Apps are more complex. Networks are more complex. And SOLUTIONS are more complex. But if the designers of computers, OSes, apps and networks can deal with the complexity, why can't security folks do likewise?
The issue of security folks dealing with complexities is, they shouldn't have to when it comes to 65% of the problems which lead to incidents. Why should an ISP have to deal with issues that have nothing to do with their networks. I get calls day and night from VoIP customers: "My service is down your service sucks...." 2007-02-19 00:23:36 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600 2007-02-19 07:59:43 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600 2007-02-19 10:58:44 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600 2007-02-19 12:58:05 '212XXX6428' at 212XXX6428@71.231.xxx.xxx:5060 for 3600 This client goes up and down like a see-saw at least 8 times a day. Their provider is horrible. Why should I spend resources trying to fix what has nothing to do with my company. Same applies to anyone in the security industry to a degree. A security engineer can only do so much given parameters most work with. "We're a Windows only shop!" touted the MCSE with glee as he wondered why he spent so much time rebooting.
That actually sounds like an answerable question, if a company took it seriously enough. If the senders and receiver are both on your network, your finance department should be able to come up with some cost figures.
They won't because they haven't been pressed to do so, and it is rare that someone will take it upon themselves to do a good deed when it comes to situations like this. Roland Dobbins wrote:
NATting firewalls don't help at all with email-delivered malware, browser exploits, etc.
Antivirus and ad-aware like programs almost often do when used appropriately. It boils down to education which won't happen. If forced however it is a different story so again I will point to customer sandboxing. And yes firewalls do help if configured properly on the business side of things. I use the same brute forcing script to create firewall rules to block IN AND OUT those offensive networks. So even if say a machine were to get infected, its only momentarily before I catch it, but this is my network(s) and those I manage/maintain. I have zero tolerance for junk and don't mind blocking a /8 if needed. People want to complain then I point out logfiles with information on why their entire class is blocked. michael.dillon@bt.com wrote:
None of this is rocket science. The hardware available today can do this. This hardware is not expensive. It does, however, require systems vendors to have a bit of imagination and that seems to be in rather short supply in the modern world.
Why would a vendor put all their eggs in one basket. "Brand New AntiVirus software... Guaranteed to stop hackers! Only $49.99 per year...", "Brand New AntiMalware software... Guaranteed to stop hackers! Only $19.99 a year!", "Brand New Intrusion Detection Prevention Dissemination Articulation software... Guaranteed to stop nuclear weapons of mass destruction... Guaranteed to keep you off of the Internet..." A vendor isn't going to do much, its truly not in their best interest to halt this garbage... So the irony goes out to again, Microsoft for selling security products that should be implemented beforehand.
participants (7)
-
Gadi Evron -
J. Oquendo -
michael.dillon＠bt.com -
Rich Kulawiec -
Roland Dobbins -
Sean Donelan -
Simon Waters