Re: Broadband routers and botnets - being proactive
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -- "Suresh Ramasubramanian" <ops.lists@gmail.com> wrote:
On 5/12/07, Albert Meyer <from_nanog@corenap.com> wrote:
I and numerous others (including some whom any reasonable NANOG-L poster would respect and listen to) have asked you repeatedly to stop trolling NANOG-L with this botnet crap. It is off-topic here. The last time you pulled this (starting
As frequent as Gadi is with his botnet posts, insecure and wide open CPE getting deployed across a large provider is definitely operational.
Suresh is right -- if you don't think CPE compromises are an operational problem, then I'm not sure what is. :-) [changing gears] I'll even go a step further, and say that if ISPs keep punting on the whole botnet issue, and continue to think of themselves as 'common carriers' in some sense -- and continue to disengage on the issue -- then you may eventually forced to address those issues at some point in the not-so-distant future. I understand the financial disincentives, etc., but if the problem continues to grow and fester, and consumer (and financial institutions) losses grow larger, things may take a really ugly turn. $.02, - - ferg -----BEGIN PGP SIGNATURE----- Version: PGP Desktop 9.6.1 (Build 1012) wj8DBQFGRXxaq1pz9mNUZTMRArMKAJ9r5LymJwHl70u7b3XU5XzvB88WugCfWRFO jWmj4+AadZTVBwQ6VGjUmHE= =oZYK -----END PGP SIGNATURE----- -- "Fergie", a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
On Sat, 12 May 2007, Fergie wrote:
[changing gears]
I'll even go a step further, and say that if ISPs keep punting on the whole botnet issue, and continue to think of themselves as 'common carriers' in some sense -- and continue to disengage on the issue -- then you may eventually forced to address those issues at some point in the not-so-distant future.
I understand the financial disincentives, etc., but if the problem continues to grow and fester, and consumer (and financial institutions) losses grow larger, things may take a really ugly turn.
$.02,
I must admit, vulnerabilities are endless and new exploitation vectors will never end, even if it was possible and we were all 100% secure, someone (an attacker rather than a vulnerability) will find a way to make it 99% again for the right investment or with the right moment of brilliance. Enough with cheap philosophy though... as tired (even exhausted) as I am of the endless repeating circle which security is, on all levels (from the people involved through the interests involved all the way to the same-old-FUD) I still haven't burned out, and I am still here. The world isn't going to end tomorrow, and even if the Internet was to die (which I doubt it will), we will survive. However, in the recent couple of years a new community has been forming which we started refering to as "Internet security operations". These folks, for various motives, work to make the Internet stay up and become safer (actually being safe is a long lost battle we should have never fought the way things were built). With such a community being around, treating issues beyond our little corner of the `net is possible to a level, and at least some progress is made. Some anti virus engineers no longer care only about samples, some network engineers no longer care only about their networks, etc. Is any of this a solution? No. The problems themselves will not go away, they aren't in any significant fashion currently being dealt with beyond the tactical level of a fire brigade. Is it the end than? Of course not. But operations vs. research are determined by intelligence. As we have some intelligence, I can point to yet another annoying vulnerability in the endless circle which those of us who will want to, can study, and if they feel it is justified, defend against. That is the broadband routers issue, which personally I'd really rather avoid. Unfortunately, this limited defense is what most of us can do at our own homes, or tops as a volunteer fire brigade or neighborhood watch. The Internet is the most disconnected global village I can imagine, but we all have the funny uncle on another network and a weird one on yet another. I sometimes feel that the old analogy of the Internet to the Wild West is not quite it. Perhaps we are living in the Wild West, only if instead of wastelands and small towns, we have New York city and the laws of a feudal dark ages Kingdom. Things will eventually change, and some of us will stick around to help that change (or try to). For now though, it is about one vulnerability ignored at a time, and working on our communities. Gadi Evron.
On Saturday 12 May 2007 04:35, Fergie wrote:
Suresh is right -- if you don't think CPE compromises are an operational problem, then I'm not sure what is. :-)
[changing gears]
I'll even go a step further, and say that if ISPs keep punting on the whole botnet issue, and continue to think of themselves as 'common carriers' in some sense -- and continue to disengage on the issue -- then you may eventually forced to address those issues at some point in the not-so-distant future.
I understand the financial disincentives, etc., but if the problem continues to grow and fester, and consumer (and financial institutions) losses grow larger, things may take a really ugly turn.
$.02,
- ferg
I totally agree - the issue keeps getting delayed and nobody is adressing it like it should be, People keep talking about the issue but it NEVER gets solved. Here's my own two cents: Most end-users don't know and probably, don't care about what they subject their systems to, therefore, systems get infected constantly. There will be no resolution of these bandwidth-wasting botnets unless something is done about the end-users who don't know/care about what they're doing, Most end users just "want things to work" without knowing and probably without wanting to know what actually is going on "behind the screnes". Furthermore, as I posted on another list, Users depend too heavily on their "security software" and think if they have a firewall and antivirus, that they can do anything and won't be infected, But as we all (I hope) know, that's not true. It's true ISPs should be held in higher responsibility to security issues such as botnets, but the end-users who end up with bots/trojans on their systems should also be held accounable. Perhaps if users get the weight on their sholders of keeping clean, they will instead of how it currently is where the issue seems to get only talked about but really no collective enforcement anything as I stated earlier. And it's not just users and ISPs that should be dealing with this issue, Datacenters should as well, I can't count how many servers I've seen infected and being used in botnets. I say kudos to those who already combat botnets on their networks, However, To those who do nothing at the moment: I say it's time to start. Oh, one more thing to the first reply to this thread calling this a non-operational issue, Gadi's in the right here: It IS an operational issue that's getting reposted because it's NOT getting solved.
Kradorex Xeron wrote:
Oh, one more thing to the first reply to this thread calling this a non-operational issue, Gadi's in the right here: It IS an operational issue that's getting reposted because it's NOT getting solved.
I recieved 4 emails (from Fergie, Suresh, Colin Johnson and "Kradorex Xeron") disagreeing with my assertion that Gadi's emails are off-topic. I also recieved a few emails saying things like "Sure he's off-topic, but he's a well-known botnet researcher, and a very smart guy, and don't you think you're being too hard on him?" and one saying in essence "Who are you to question a highly respected guy like Gadi?" The 4 people who feel that Gadi's botnet posts are on-topic here in NANOG-L have apparently not read the NANOG-L charter and FAQ so I am providing links here: http://www.nanog.org/aup.html http://www.nanog.org/listfaq.html I agree that Gadi is a highly respected botnet researcher, and I'm just a lowly netadmin at a regional ISP. Shouldn't I just shut up and soak up his glory? If this were a botnet list, yes. But this is a network operator's list, and I'm a network operator. There are lists where botnets are discussed, and Gadi is very active on those lists. There is no need for him to repost his botnet emails to NANOG-L. I don't join the botnet lists and spam them with networking issues, and it's not appropriate for Gadi to spam NANOG-L with botnet crap, regardless of how highly respected he is in his field. Addressing the complaint that my response to Gadi was too harsh, I can only say that, to someone who isn't aware of the history, my response may seem harsh, but anyone who has seen the endless trolling of NANOG-L, the numerous requests (public and private) asking Gadi to cut it out, the extensive discussions on IRC, in private email and elsewhere will understand that the forcefulness of my request is appropriate given the fact that all previous attempts to end this needless disruption of NANOG-L have been ineffective.
Addressing the complaint that my response to Gadi was too harsh, I can only say that, to someone who isn't aware of the history, my response may seem harsh,
I *AM* aware of the history and your response seems harsh. Especially so because you complained about a message which was about exploits in CPE access routers, not botnets. Any kind of router vulnerability/exploit is on topic for NANOG. And people who don't take the trouble to read messages and critique the message content, should not post to the list at all. We don't need you using NANOG to fight your personal flamewar with Gadi.
but anyone who has seen the endless trolling of NANOG-L, the numerous requests (public and private) asking Gadi to cut it out, the extensive discussions on IRC, in private email and elsewhere will understand that the forcefulness of my request is appropriate given the fact that all previous attempts to end this needless disruption of NANOG-L have been ineffective.
Well, since I have some knowledge of these communications and the fact that a number of people have thanked Gadi for his work and urged him to continue posting to the NANOG list from time to time, I do *NOT* understand the forcefulness of your request. The fact is that there are two sides to this story, and that the 8000 or so NANOG members are somewhat divided on the issue. But one thing is clear, messages like yours are not useful to any of the list members, but many of Gadi's messages *ARE* useful to some of the list members. In a group of 8000 people, I expect the best anyone can hope for is that most of the messages on the list will be useful to some of the list members. If that isn't good enough for you, there is a mailing list committee and a steering committee that you can complain to, but privately please, not on the list. --Michael Dillon
michael.dillon@bt.com wrote:
Addressing the complaint that my response to Gadi was too harsh, I can only say that, to someone who isn't aware of the history, my response may seem harsh,
I *AM* aware of the history and your response seems harsh. Especially so because you complained about a message which was about exploits in CPE access routers, not botnets. Any kind of router vulnerability/exploit is on topic for NANOG. And people who don't take the trouble to read messages and critique the message content, should not post to the list at all. We don't need you using NANOG to fight your personal flamewar with Gadi.
I don't see cpe as being all that different than hosts, except that they're slower and less flexible. The thing is it would be really nice to have some functional separation between the business of this list which is operating a network, and the security focused lists, and the botnet/phishing/spam lists, addressing policy lists, the internet standards list, and so forth. You and I and lots of other people on this list are on on many or all of those sorts of lists. While cross-pollination is acceptable and in fact desired dragging the business of one group of community interests in to the domain of another is not appropriate. In the particular case of Gadi, I resent the persistent grandstanding and offers of assistance and assurances that's he's on the job. That's essentially all advertising for his consulting business and I don't think it's appropriate on this list. I for one do not flog the products of my employer on this list, nor do you, or most other people who participate. I tolerate this sort of behavior in the security arena (read bugtrac these days) though I resent the fact that it's de rigeur in the space for many disclosures to essentially be advertising for the consultants doing the work, virus updates are advertising for anti-virus companies etc.
but anyone who has seen the endless trolling of NANOG-L, the numerous requests (public and private) asking Gadi to cut it out, the extensive discussions on IRC, in private email and elsewhere will understand that the forcefulness of my request is appropriate given the fact that all previous attempts to end this needless disruption of NANOG-L have been ineffective.
Well, since I have some knowledge of these communications and the fact that a number of people have thanked Gadi for his work and urged him to continue posting to the NANOG list from time to time, I do *NOT* understand the forcefulness of your request.
The fact is that there are two sides to this story, and that the 8000 or so NANOG members are somewhat divided on the issue. But one thing is clear, messages like yours are not useful to any of the list members, but many of Gadi's messages *ARE* useful to some of the list members. In a group of 8000 people, I expect the best anyone can hope for is that most of the messages on the list will be useful to some of the list members.
If that isn't good enough for you, there is a mailing list committee and a steering committee that you can complain to, but privately please, not on the list.
--Michael Dillon
On Tue, May 15, 2007, Joel Jaeggli wrote:
michael.dillon@bt.com wrote:
Addressing the complaint that my response to Gadi was too harsh, I can only say that, to someone who isn't aware of the history, my response may seem harsh,
I *AM* aware of the history and your response seems harsh. Especially so because you complained about a message which was about exploits in CPE access routers, not botnets. Any kind of router vulnerability/exploit is on topic for NANOG. And people who don't take the trouble to read messages and critique the message content, should not post to the list at all. We don't need you using NANOG to fight your personal flamewar with Gadi.
I don't see cpe as being all that different than hosts, except that they're slower and less flexible.
I see them as more flexible - they don't have a CPE in front of them potentially being a firewall, they can listen() on ports for p2p botnet type action, and they can silently redirect your traffic to completely different IPs or return bogus DNS info, they can see inside your home network and be counted as "internal internet zone" to IE.. (perhaps not operational per-se, but pretty freaking scary.) Adrian
Bearing in mind that I'm not especially a fan of Gadi,
The thing is it would be really nice to have some functional separation between the business of this list which is operating a network, and the security focused lists, and the botnet/phishing/spam lists, addressing policy lists, the internet standards list, and so forth.
The thing is that there's always been too much functional separation between the business of this list which is operating a network, and the security focused lists. The business of operating a network has often conveniently ignored anything that doesn't actually cause the network to collapse, but which regardless makes the network a less-than-nice place to be. Is spam directly related to the business of Network A peering via BGP to Network B? Doubtful. However, where does that change? What sort of things are operational? As long as we choose to interpret "operating a network" as being merely things that involve enable on a router, yes, it's way off-topic. Sadly, many (most?) networks view their operation in a way that emphasizes this sort of attitude. As a result, we still don't have basic security things that should /also/ be a fundamental part of netops, such as BCP38 at any point where it is reasonable to do so (like at virtually every edge).
You and I and lots of other people on this list are on on many or all of those sorts of lists.
In most organizations larger than a handful of people, the netops people are not necessarily the same as the security people, and I've often found that the groups do not understand issues happening in the other arena.
While cross-pollination is acceptable and in fact desired dragging the business of one group of community interests in to the domain of another is not appropriate.
Were they all truly separate, this would be true. They're not all truly separate. Pretending that they're separate would be a convenient way to allow your network to continue peeing in the pool, ignoring problems, which (sadly) doesn't seem to be an unusual attitude at certain networks. Those of us who have been implementing BCP38-style filtering since before BCP38 existed, on the other hand, may take a slightly more mature view of what "network operations" involves, and it sure covers a lot more ground than what you can do with enable on a router. I do not consider host security to be directly connected to netops. However, it certainly has an impact, and to a certain extent, a little occasional discussion is warranted. Gadi may tend to bring along a little too much discussion, though. I think a lot of people would agree with that.
In the particular case of Gadi, I resent the persistent grandstanding and offers of assistance and assurances that's he's on the job.
Okay, annoying, granted.
That's essentially all advertising for his consulting business and I don't think it's appropriate on this list. I for one do not flog the products of my employer on this list, nor do you, or most other people who participate.
Yeah, um, uh, that fink is always trying to sell me something, uh, hm, except I can't remember what, or find its web site, or even substantiate that claim. He posts from linuxbox.org, which seems to have no web page, usually posts without a signature, etc. Maybe you could outline where he's doing all this evil advertising. If you want to paint Gadi with this brush, you should be aware that the criteria necessary to bring him down on that basis will almost certainly cover Paul Vixie and a whole bunch of other highly respected members of this community.
I tolerate this sort of behavior in the security arena (read bugtrac these days) though I resent the fact that it's de rigeur in the space for many disclosures to essentially be advertising for the consultants doing the work, virus updates are advertising for anti-virus companies etc.
I find it sadly ironic that the netops community, which largely runs huge commercial for-profit networks, would think that others would handle the security aspects for them - and do it for free. What's pathetic is that these same large networks usually can't be bothered to do much (or anything) to eliminate the environment which provides work opportunities for security consultants. Gadi? Annoying, definitely. But nothing compared to the resistance of this community to the idea that netops has anything to do with the sorts of security issues Gadi brings up. I just had to comment on this. I'll go back to lurking now. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
On Tue, 15 May 2007, Joe Greco wrote:
The thing is that there's always been too much functional separation between the business of this list which is operating a network, and the security focused lists. The business of operating a network has often conveniently ignored anything that doesn't actually cause the network to collapse, but which regardless makes the network a less-than-nice place to be.
Is spam directly related to the business of Network A peering via BGP to Network B? Doubtful. Clearly, it is not. Packets keep flowing. Internet is more than email.
However, where does that change? What sort of things are operational? Things that affect internet at large. Things that affect our routers.
As long as we choose to interpret "operating a network" as being merely things that involve enable on a router, yes, it's way off-topic. Sadly, many (most?) networks view their operation in a way that emphasizes this sort of attitude. As a result, we still don't have basic security things that should /also/ be a fundamental part of netops, such as BCP38 at any point where it is reasonable to do so (like at virtually every edge). That's certainly on-topic (why is BCP38 is not implemented as much as it should be).
You and I and lots of other people on this list are on on many or all of those sorts of lists.
In most organizations larger than a handful of people, the netops people are not necessarily the same as the security people, and I've often found that the groups do not understand issues happening in the other arena. That's an issue for those organizations. :) However, security people have their own mailing lists, and *forcing* operations people to be involved in security issues is counter-productive. (You don't make your security people to read nanog-l, do you?)
While cross-pollination is acceptable and in fact desired dragging the business of one group of community interests in to the domain of another is not appropriate.
Were they all truly separate, this would be true. They're not all truly separate. Pretending that they're separate would be a convenient way to allow your network to continue peeing in the pool, ignoring problems, which (sadly) doesn't seem to be an unusual attitude at certain networks. Sadly, the alternatives (resulting in trying to police the internet) are much worse than status quo.
Those of us who have been implementing BCP38-style filtering since before BCP38 existed, on the other hand, may take a slightly more mature view of what "network operations" involves, and it sure covers a lot more ground than what you can do with enable on a router.
I do not consider host security to be directly connected to netops. However, it certainly has an impact, and to a certain extent, a little occasional discussion is warranted. When it is affecting internet at large (think nachi/nimda/codered), clearly.
Gadi may tend to bring along a little too much discussion, though. I think a lot of people would agree with that. I think this is the case of a 'boy who cried wolf' too many times.
I find it sadly ironic that the netops community, which largely runs huge commercial for-profit networks, would think that others would handle the security aspects for them - and do it for free. Those who run huge commercial for-profit networks usually have people who are dedicated to security aspects...Ones not so lucky usually have same
<snip> people who are both operations and security and peering - and we are subscribed to all the mailing lists we need to know to get all our jobs done. And I think its the way things should be.
What's pathetic is that these same large networks usually can't be bothered to do much (or anything) to eliminate the environment which provides work opportunities for security consultants. Do I smell a "final ultimate solution" somewhere? :)
[note the cc to nanog-futures - please strip cc to nanog-l from the replies to this email. meta-discussions belong on -futures] alex [speaking for myself only]
On 16 May 2007, at 00:53, Joel Jaeggli wrote:
[snip]
The thing is it would be really nice to have some functional separation between the business of this list which is operating a network, and the security focused lists, and the botnet/phishing/spam lists, addressing policy lists, the internet standards list, and so forth.
While there persists an attitude that security isn't a core part of running a network there will continue to be insecure networks flooded with spam, phishing, botnets et al. I've been running wide area networks since 1995 and I've always seen security as an operational network issue and moreover I find incomprehensible an attitude that sees it otherwise.
You and I and lots of other people on this list are on on many or all of those sorts of lists. While cross-pollination is acceptable and in fact desired dragging the business of one group of community interests in to the domain of another is not appropriate.
In the particular case of Gadi, I resent the persistent grandstanding and offers of assistance and assurances that's he's on the job. That's essentially all advertising for his consulting business and I don't think it's appropriate on this list. I for one do not flog the products of my employer on this list, nor do you, or most other people who participate.
While Gabi is voluble I don't concur. I've never formed the impression that he's advertising anything other than the problem or some [possible] solutions. I've certainly never felt he was advertising his paid services - so much so that this is the first time I was explicitly aware that he offers paid consultancy in this area, if that is indeed the case.
I tolerate this sort of behavior in the security arena (read bugtrac these days) though I resent the fact that it's de rigeur in the space for many disclosures to essentially be advertising for the consultants doing the work, virus updates are advertising for anti-virus companies etc.
[snip] Can I please make a [probably futile] request. If someone thinks something is off-topic but the subject matter is even conceivably marginally on-topic - just skip the post. Don't start a long discussion of the relevance. Inevitably the discussion of topicality takes up more time and attention than the original subject would have. Whenever I see this happen I always suspect that the operational issue is really that the complainants don't have enough real operational work to do and I wish I had their cushy job.
On Wed, 16 May 2007, Ian Mason wrote:
- so much so that this is the first time I was explicitly aware that he offers paid consultancy in this area, if that is indeed the case.
I don't. Nor do I work for a colsultancy. Thanks, Gadi.
Gadi Evron wrote:
On Wed, 16 May 2007, Ian Mason wrote:
- so much so that this is the first time I was explicitly aware that he offers paid consultancy in this area, if that is indeed the case.
I don't. Nor do I work for a colsultancy.
Your work for a vulnerability assessment vendor... or maybe you should update your bio: http://lifeboat.com/ex/bios.gadi.evron
Thanks,
Indeed Joelja
Gadi.
On Wed, 16 May 2007, Joel Jaeggli wrote:
Gadi Evron wrote:
On Wed, 16 May 2007, Ian Mason wrote:
- so much so that this is the first time I was explicitly aware that he offers paid consultancy in this area, if that is indeed the case.
I don't. Nor do I work for a colsultancy.
Your work for a vulnerability assessment vendor...
Thank you very much Joel. I much appreciate you clarifying that point.
or maybe you should update your bio:
http://lifeboat.com/ex/bios.gadi.evron
Thanks,
Indeed Joelja
Gadi.
Gadi Evron however is listed as one of the authors on a rather interesting book "Botnets: The Killer Web Application": http://search.barnesandnoble.com/booksearch/isbnInquiry.asp?z=y&EAN=9781597491358&itm=3 (Howsabout we stand on each others' shoulders rather than each others' toes?) -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Login: Nationwide Software Tool & Die | Public Access Internet | SINCE 1989 *oo*
On 16-May-2007, at 01:04, Ian Mason wrote:
Can I please make a [probably futile] request.
If someone thinks something is off-topic but the subject matter is even conceivably marginally on-topic - just skip the post. Don't start a long discussion of the relevance.
... or if you are unable to resist the temptation to start a meta- discussion, do it on this list, not on the nanog list. It's on-topic, here. Joe
from http://www.nanog.org/listfaq.html Appropriate Topics ... ISP security ... I think DTAG.de is a very insecure ISP. The router is still distributed. There is no warning by DTAG.de. There is no fix. There is an ongoing discussion about a troyan developed and distributed by the german gouvernement or their agencies. So this router is a very likely means how they enter your home. There is an ongoing discussion, fed mostly by our governement, about china hacking german computers - industry espionage. So it is very likely that china uses this governement troyan to break into our computers. The scenario is very likely because we do not grow computer science people here in germany, we have to import them from china, that is what our governement and our industry keeps telling us. Oh, there is a fix. DTAG.de is on strike. If they were not, some 11 million germans might be made into spam bots. That would effect routing world wide and probably in north america too. Cheers Peter and Karin Albert Meyer wrote:
... The 4 people who feel that Gadi's botnet posts are on-topic here in NANOG-L have apparently not read the NANOG-L charter and FAQ so I am providing links here:
http://www.nanog.org/aup.html http://www.nanog.org/listfaq.html ...
-- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de mail: peter@echnaton.arl.pirates http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
participants (13)
-
Adrian Chadd -
Albert Meyer -
alex＠pilosoft.com -
Barry Shein -
Fergie -
Gadi Evron -
Ian Mason -
Joe Abley -
Joe Greco -
Joel Jaeggli -
Kradorex Xeron -
michael.dillon＠bt.com -
Peter Dambier