Re: What would you tell the White House?
Sean, A few ideas... The ISPs who are providing services to their customers have a responsibility to implement the measures necessary to prevent or reduce the impact of malicious attacks such as those which have occurred over the past week. In addition, it's the customer's responsibility to make sure that they have the necessary measures to protect their networks and hosts, and that they've verified that their ISPs are protecting them. I don't think that the federal government needs to pass a law defining how one should protect his networks, or how an ISP should implement network protection measures. If the Internet industry developed a set of rules/guidelines (Barry Greene's document, as well as pertinent RFCs such as 2267, are a good starting place), the customers can shop around to find a provider who will protect his network. After all, this isn't a regulated monopoly, like an RBOC. The ISPs need to put a system in place where they can work together to quickly trace and isolate the source of any attack. Perhaps the vendors need to develop some mechanisms to facilitate this. If we had the ability to quickly conduct a cooperative trace of of an attack, and it would result in the apprehension and eventual prosecution of the attackers, it would serve as a good deterrent to future attacks. My $0.02, -rb ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com
The ISPs need to put a system in place where they can work together to quickly trace and isolate the source of any attack. Perhaps the vendors need to develop some mechanisms to facilitate this.
A good deal of this technology is in place already, but Based on my experience, most ISPs just aren't using it or aren't acting on the data. I don't know if it's because of the administrative cost of managing a secure network, the tight market for talented personnel, or what, but it's really annoying when I go to the trouble of reporting security incidents and nothing happens. This week's logs on my very small network show: 10 events of a user on best.net trying to connect to my RPC port: UTC 02/11/2000 02:45:20.784 TCP connection dropped Source:209.24.82.10, 3714, WAN Destination:209.31.7.40, 111, LAN Best.net's security people said "that box was compromised, block access to the IP address while it's fixed." Huh? How come best.net is letting their users send this crap out? If I can filter in-bound, they can filter out-bound while they fix the system. 5 events of a user at a Korean site running nmap or some other scanner against TCP port 1 on each of my public addresses: UTC 02/13/2000 06:22:26.576 TCP connection dropped Source:211.45.145.2, 3272, WAN Destination:209.31.7.41, 1, LAN The Korean ISP didn't respond. Two weeks ago I got: UTC 02/05/2000 07:32:05.944 Sub Seven Attack Dropped Source:209.245.74.63, 1242, WAN Destination:209.31.7.41, 1243, LAN Level3.net still hasn't responded to that. Ad nauseum. Every week I get probed, hacked on, ping-o-death'd and more, while every week I send copies of the log to the source' security@isp. 30% of the time security@ is an invalid mailbox that bounces (which is why I also cc: abuse@isp), 60% of the time the message is ignored or not responded to, and only 10% of the time do I get a response that some form of action might be taken if they can figure out which user had the IP address at that moment. So, based on my experience, the ISP community isn't taking advantage of the tools they have to do their own enforcement. It would seem to me that the first step in saying "we can take care of this ourselves" is to prove that you're credible. If I were asked, I'd say that the quality of self-policing to date has been quite miserable. -- Eric A. Hall ehall@ehsco.com +1-650-685-0557 http://www.ehsco.com
On Sun, 13 Feb 2000, Eric A. Hall wrote:
Ad nauseum. Every week I get probed, hacked on, ping-o-death'd and more, while every week I send copies of the log to the source' security@isp. 30% of the time security@ is an invalid mailbox that bounces (which is why I also cc: abuse@isp), 60% of the time the message is ignored or not responded to, and only 10% of the time do I get a response that some form of action might be taken if they can figure out which user had the IP address at that moment.
Recently called the NOC of a tier1 provider who hadn't responded to my emails about repeated cracking attempts originating from their network. They told me point blank, they bin ALL abuse emails and only act on phone reports. (Whats the point of maintaining an abuse mailbox then? Boggle.) This might seem pretty outrageous to some, yet it is not too different from other tier1 NOCs I have dealt with regarding attacks. Perhaps its time someone did a public audit of how the tier1 NOCs (mis)handle abuse incidents. Since it seems impossible to change company policies until something really negative and really public happens (eg recent DDOS), perhaps this is whats needed.
So, based on my experience, the ISP community isn't taking advantage of the tools they have to do their own enforcement. It would seem to me that the first step in saying "we can take care of this ourselves" is to prove that you're credible. If I were asked, I'd say that the quality of self-policing to date has been quite miserable.
Miserable isn't the word for it. I think there has yet to be a word invented to describe this pathetic state of affairs. -Dan
At 21:05 13/02/00 -0800, Eric A. Hall wrote: Your conclusions are identical to what I have found. The reasons are: a) profit margin: Almost all ISPs lose money. CEOs and CFOs do not see the dedicated personnel that handles 'abuse@' emails as generating income. Most small ISPs with revenue under $5M/yr will not be able to dedicate an FTE to 'abuse@' handling. ISPs would rather hire another salesman or purchase a larger Cisco router than invest in handling 'abuse@'. We may not like it - but that is what happens. b) lawyers: once you get into major size ISPs (over $100M/yr), they don't move without legal counsel. You were attacked by a Sub Seven port scan? You want the ISP to yank the user off the network? First you need to find a lawyer who understands a bit of the technical jargon. 95% do not. Once you do find such a person, legal counsel of the ISP will first demand proof from the *local* staff that such an attack has occurred. Your complaint logs are not admissible, in his view. Then the lawyer has to check that the hacker was made aware of the existing AUP. That gives the hacker a second chance. Now if the hacker is not really a hacker - but perhaps some user who claims to have his account or system hacked and if you revoke access - he will sue the ISP for every penny since he is working on a multi-million dollar deal and without email he will lose everything; the lawyer will fold his tail and run. I have seen this countless times. c) lack of time: a derivative of (a) above. Severely understaffed, the ISP has lines down and routers overloaded and servers with disk problems, and new customers wanting their connection up NOW! Spam reports and nmap scans fall to the wastebasket in these cases. d) incompetence: a derivative of (a) above. Some ISPs have no idea what is nmap or strobe or cheops and have never heard of ISS, Retina, Netrecon, or Netranger. Their main Internet guru, is an NT techie, who thinks NT is a very secure operating system. See below.
The ISPs need to put a system in place where they can work together to quickly trace and isolate the source of any attack. Perhaps the vendors need to develop some mechanisms to facilitate this.
A good deal of this technology is in place already, but Based on my experience, most ISPs just aren't using it or aren't acting on the data. I don't know if it's because of the administrative cost of managing a secure network, the tight market for talented personnel, or what, but it's really annoying when I go to the trouble of reporting security incidents and nothing happens.
This week's logs on my very small network show:
10 events of a user on best.net trying to connect to my RPC port:
UTC 02/11/2000 02:45:20.784 TCP connection dropped Source:209.24.82.10, 3714, WAN Destination:209.31.7.40, 111, LAN
Best.net's security people said "that box was compromised, block access to the IP address while it's fixed." Huh? How come best.net is letting their users send this crap out? If I can filter in-bound, they can filter out-bound while they fix the system.
Because if Best.net filtered at their end - they may be liable to a lawsuit from the user who had his access blocked.
5 events of a user at a Korean site running nmap or some other scanner against TCP port 1 on each of my public addresses:
UTC 02/13/2000 06:22:26.576 TCP connection dropped Source:211.45.145.2, 3272, WAN Destination:209.31.7.41, 1, LAN
The Korean ISP didn't respond.
Lack of time.
Two weeks ago I got:
UTC 02/05/2000 07:32:05.944 Sub Seven Attack Dropped Source:209.245.74.63, 1242, WAN Destination:209.31.7.41, 1243, LAN
Level3.net still hasn't responded to that.
Profit margin.
Ad nauseum. Every week I get probed, hacked on, ping-o-death'd and more, while every week I send copies of the log to the source' security@isp. 30% of the time security@ is an invalid mailbox that bounces (which is why I also cc: abuse@isp), 60% of the time the message is ignored or not responded to, and only 10% of the time do I get a response that some form of action might be taken if they can figure out which user had the IP address at that moment.
So, based on my experience, the ISP community isn't taking advantage of the tools they have to do their own enforcement. It would seem to me that the first step in saying "we can take care of this ourselves" is to prove that you're credible. If I were asked, I'd say that the quality of self-policing to date has been quite miserable.
I suspect we will only see more attacks and not to expect any solutions from ISPs in the near future. -Hank [the above are my own views and do not reflect light nor the opinions of any companies or organizations for which I do consulting.]
-- Eric A. Hall ehall@ehsco.com +1-650-685-0557 http://www.ehsco.com
- he will sue the ISP for every penny since he is working on a multi-million dollar deal and without email he will lose everything;
Because if Best.net filtered at their end - they may be liable to a lawsuit from the user who had his access blocked.
There is another end on that very-sharp stick: the ISPs that allow their users to do this crap -- esp. when they have been notified of the hack attempts -- will most likely be sued for negligence in the near future. The lawyers will get involved one way or another, and ISPs would do well to choose sides beforehand: do you want to get sued by your users or by Yahoo and eBay, who actually did lose millions and are looking for any way to prevent this from happening again? Who will be more pissed, and who will have the better lawyers: Joe Dialup's 16 year old kid who's playing with the latest warez, or Yahoo's board of directors who's trying to increae their multi-billion dollar market valuation? In the long run, zero-tolerance policies are going to be the only thing that solves the liability problems. Coincidentally, those policies will also be what stops most of the attacks.
I suspect we will only see more attacks and not to expect any solutions from ISPs in the near future.
I suspect that a few high-profile lawsuits would change that. -- Eric A. Hall ehall@ehsco.com +1-650-685-0557 http://www.ehsco.com
<IANAL> The blocking issue is BS. Make the customers... all customers, dialup AND dedicated... sign something that says that they will agree to the AUP and Terms of Service, and specify that traffic will be filtered for security reasons. </IANAL> "Eric A. Hall" wrote:
- he will sue the ISP for every penny since he is working on a multi-million dollar deal and without email he will lose everything;
Because if Best.net filtered at their end - they may be liable to a lawsuit from the user who had his access blocked.
There is another end on that very-sharp stick: the ISPs that allow their users to do this crap -- esp. when they have been notified of the hack attempts -- will most likely be sued for negligence in the near future.
The lawyers will get involved one way or another, and ISPs would do well to choose sides beforehand: do you want to get sued by your users or by Yahoo and eBay, who actually did lose millions and are looking for any way to prevent this from happening again? Who will be more pissed, and who will have the better lawyers: Joe Dialup's 16 year old kid who's playing with the latest warez, or Yahoo's board of directors who's trying to increae their multi-billion dollar market valuation?
In the long run, zero-tolerance policies are going to be the only thing that solves the liability problems. Coincidentally, those policies will also be what stops most of the attacks.
I suspect we will only see more attacks and not to expect any solutions from ISPs in the near future.
I suspect that a few high-profile lawsuits would change that.
-- Eric A. Hall ehall@ehsco.com +1-650-685-0557 http://www.ehsco.com
-- North Shore Technologies, Cleveland, OH http://NorthShoreTechnologies.net Steve Sobol, President, Chief Website Architect and Janitor sjsobol@NorthShoreTechnologies.net - 888.480.4NET - 216.619.2NET
On Tue, 15 Feb 2000 16:03:49 EST, Steve Sobol said:
<IANAL> The blocking issue is BS. Make the customers... all customers, dialup AND dedicated... sign something that says that they will agree to the AUP and Terms of Service, and specify that traffic will be filtered for security reasons. </IANAL>
The problem here is that although IANAL either, and YANAL, you WILL need one to craft an AUP and rules that will work, in spite of users. First thing to remember: The traffic we *want* to stop is the payload traffic of the DDOS system, which in general is NOT filterable. Fortunately, at the current time the *control* traffic is identifiable and filterable in most cases. Second thing to remember: The traffic is being generated by machines that are subverted - and the cracker didn't sign your AUP. You can't code "I will not allow my machine to be subverted" in the AUP, because it's unenforcable. Third thing to remember: Users can be incredibly stupid. Our abuse desk now has a form letter to send out to people who report that our NTP servers are portscanning/probing them. Yes, enough people poke our port 13/37/137 and forget to allow inbound packets for those that it's an issue. If we advertise a system/network change, and then cancel at the last minute, we will still get calls about the change breaking things. Warn your help desk, as they WILL get calls about how the (high-visibility) "filtering broke my Netscape". ;) Fourth thing to remember: Even if the user signs a form saying that traffic will be filtered for security reasons, they *will* either sue you or leave for another provider if down the road, you tweak your security filters and break something they were using. You can't specify up front what filters you will use - that keeps you from filtering new attacks. And you have *two* cases to worry about, one where you start filtering incorrectly and break somebody's service (which is *hopefully* already covered in your contract in the same paragraph as "what you can do if we accidentally unplug the router" ;), and the one where you install the right filter, but it still breaks something (it was alledged on the IETF list that ingress/egress filtering of "bad source address" packets breaks Mobile IP that doesn't implement RFC2344). Valdis Kletnieks Operating Systems Analyst Virginia Tech
Valdis.Kletnieks@vt.edu wrote:
On Tue, 15 Feb 2000 16:03:49 EST, Steve Sobol said:
<IANAL> The blocking issue is BS. Make the customers... all customers, dialup AND dedicated... sign something that says that they will agree to the AUP and Terms of Service, and specify that traffic will be filtered for security reasons. </IANAL>
The problem here is that although IANAL either, and YANAL, you WILL need one to craft an AUP and rules that will work, in spite of users.
Yup.
First thing to remember: The traffic we *want* to stop is the payload traffic of the DDOS system, which in general is NOT filterable. Fortunately, at the current time the *control* traffic is identifiable and filterable in most cases.
Second thing to remember: The traffic is being generated by machines that are subverted - and the cracker didn't sign your AUP. You can't code "I will not allow my machine to be subverted" in the AUP, because it's unenforcable.
Someone replied just earlier today, and I don't think the reply has made it to all of the list recipients yet... they said that it is still a good idea to include language to protect yourself from people attempting to use your network to initiate DOS, whether singly or as part of a DDOS attack. I think that that's really a no-brainer. I don't own my own dialups, but I own a server that I use to offer Unix shell services, so this is a big issue for me (and I do offer dialup access, and I need to be sure that my AUP/TOS is strong enough that if someone violates the dialup provider's AUP/TOS they're also violating mine, and I can nuke their account).
Third thing to remember: Users can be incredibly stupid.
I'm fully aware of that fact, having done tech support for the past five years.
those that it's an issue. If we advertise a system/network change, and then cancel at the last minute, we will still get calls about the change breaking things. Warn your help desk, as they WILL get calls about how the (high-visibility) "filtering broke my Netscape". ;)
Right. Well, in general, I operate on the premise that the customer is always right; however, there are only so many warnings I can give them before I actually have to make the change. If people refuse to listen to me, what am I supposed to do? The best thing to do is to archive the mail you send to the customer mailing list announcing the changes, and if someone complains, point them to the archive and say "there, this is when I first told you it was going to happen, please pay attention next time."
Fourth thing to remember: Even if the user signs a form saying that traffic will be filtered for security reasons, they *will* either sue
Let me put forth a suggestion. When crafting my Acceptable Use Policy some time ago, I turned to the people I know on the anti-spam mailing lists and on news.admin.net-abuse.email because I wanted to do as much as I possibly could to make it very painful for spammers to use me to send spam. I want to do the same thing here. Let's come up with a standard AUP that is worded strongly enough that we'll be able to protect ourselves. I think that a discussion of AUPs is only quasi-operational, at best, and therefore, if we decide that it's not really ontopic for NANOG I'll set up a mailing list on my server. Thoughts? Would anyone actually participate in a discussion like this? -- North Shore Technologies, Cleveland, OH http://NorthShoreTechnologies.net Steve Sobol, President, Chief Website Architect and Janitor sjsobol@NorthShoreTechnologies.net - 888.480.4NET - 216.619.2NET
participants (6)
-
Dan Hollis -
Eric A. Hall -
Hank Nussbacher -
Ron Buchalski -
Steve Sobol -
Valdis.Kletnieks＠vt.edu