Another couple of points... Do you guys scan your customers? If so, what for? OS scans? Specific threats? Infections? How do you handle this in your AUP? Have you had problems with it? What sort of problems have you had and how did you handle them? How do you guys deal with NIDS at multi-Gbps rates? -- Christian Kuhtz <ck@arch.bellsouth.net> -wk, <ck@gnu.org> -hm Sr. Architect, Engineering & Architecture, BellSouth.net, Atlanta, GA, U.S. "I speak for myself only."

As per your request, Christian, I've come back to respond to the original. Be prepared, it'll probably be book length. For those of you tired of thread, sorry, but I really feel that courtesy requires me to answer publically. I've also left most of the original in its entirety, so that any responses I make will contain original context (I usually prefer judicious clipping). Christian Kuhtz wrote:

so, you want to be a good citizen and stewart of the inet. DDoS and security after security attack happens, it won't ever stop. You try to do the best you can to effectively respond to it. You try to inform you customers. You try to educate them. Yet, you realize that you're not doing enough...

What do the rest of you SPs actually do to combat this threat?

I understand that this was addressed to Service Providers, of which I am not. I still have the advantage of long perspective (I hail from arpanet days and before, and no, you won't know me).

How do you keep the hype, fear, panic and dispair of your management team (all the way to the CEO) in check? And that of your customers? Some of it is sometimes warranted.. but, got any ideas for crowd control?

Well, yes. Unfortunately, some of the largest broadband providers seem unwilling or unable to assist. What they should do, and you can do, is this: Create a page, much like the virus myths page (my favorite for dispelling FUD), and remind people of its existence each and every time an event like Code Red or SirCam surfaces. Keep it simple. Tell people who might be affected. Tell them what precautions they can take. Tell them what precautions are being taken. Keep it updated. Remember, the more fire you are in, the more starved for information they are. Delegate the responsibility for actual emergency updates to the equivalent of someone like a secretary, or intern, and make them understand how important what they are doing is. Have them SIGN it, with their name, which gives them ownership and pride in assisting in this effort, and lets everyone know who they should call (not you) if they are concerned about something, and they see no updates.

In our case, we have several hundred thousands of DSL customers today, and the million plus subscriber mark is on the engineering horizon. The problem of security threats & resulting incidents is going to get considerably worse before it gets better. And that's for at least two reasons.. the ramp up of broadband and presumably the declining sophistication of the subscriber population as a result of the greater market penetration.

Your biggest enemy is the evening news, followed closely by all the amateur FUD sites and "security" sites out there. Declining sophistication is relative. Two years ago I heard gamers and such talking about cable modems and DSL. Now I get questions about latency on satellite access, and what is a good firewall, and what switch should I buy for my home network. Really. Sure, the people I deal with on a day to day basis are generally more aware than most (if they have to listen to me yell about things like Bonzai Buddy, believe me, they are AWARE).

Sure, you can try to teach your subscribers to protect themselves. But this is really not the answer. How many unsophisticated subscribers are going to be able to do this in an effective and timely manner?

Well, knowing that you seem to be from Bellsouth (one of the better ones, according to the reviews on DSLReports), I see the following: BellSouth 3.59 (out of 5.0, not bad at all) (I use XO, which is beginning to pull out of the miasma caused by marrying Concentric, and NextLink, and trying to absorb so many homeless COVAD customers.) $60 a month, on average. (XO is $123, but most of their customers are business class. They've never supported the Earthlink style customer on DSL.) B- for Sales rating B- for Install Experience B+ for Reliability C+ for Tech Support B- for Services (Email,DNS,News etc) B for Value for Money I'm not trying to pick on you, those are fairly typical results. The part I would like to point out is the rating for Tech Support. The only provider that got better than a B, no matter how good their rating otherwise, was UUNET, and that was with only 16 reviews (in other words, I don't believe it). I've been with a LOT of ISPs, and a lot of different kinds of services (from compuserve days on). I've been on blacknet, siprnet, arpanet, universities, you name it. You have to know where this is going. What is the number one complaint from the user community? They can't get anyone on the other end of the phone who will help them. It's worse if you're knowledgable than if you're not, since many services think that the only possible response to any problem is reboot the computer, cycle the power on the cable/dsl modem, or reinstall the software.

What do you do in response?

I think that you need to answer the issue above first. Some service providers really seem to try and provide information, but most leave their customers in the dark. I'm happy with the support I get when I actually call tech services at XO, but I've yet to see a single thing on any web page on their site that wasn't a complete waste of my time. I can say a lot worse about some of the previous providers I've had (most notably pacbell, the emporer of the world when it comes to screwing up email).

How do you effectively scale the massive support effort need for collaborative marketing of personal firewalls and the potential for false positives and negatives? Any ideas on the legal exposure of security services?

Nothing wrong with pointing people to good security measures. Nothing wrong with reminding them that you are only providing the pipe for packets, that security (especially on broadband) is an important business, and that they need to be proactive. Collaborative marketing? What happens when what you recommend turns out to have problems, or won't work with what they have? I think that offering multiple services here is a path that you don't want to tread. You go from being a packet provider to being RESPONSIBLE. Ask your lawyers. That's a bad thing. Really. Recommending, as long as you make no profit, and are impartial, sure, fine. Suggesting one only? OFfering a discount? Bad idea.

Like, in the current case, several providers have resorted to blocking port 80 to their non-DIA subscriber base. Is this really scalable? Obviously not for every threat. You can't effectively keep this up with the myriad of threats. Or can you?

Blocking port 80 to cable modem subscribers, sure why not? It says in their terms of service that they shouldn't be running web servers. I just wish that they'd blocked it both ways, so that I wouldn't keep seeing hits from them. Blocking port 80 for others? Nope, not unless that was already part of the TOS. Personally, I'd be asking for reparations, and I'd be pretty angry. No, I don't have a web server (although I have machines that sometimes look like one).

Is it realistic to be able to maintain your own NIDS patterns with the help of your own staff and public resources? Are options like security service providers the only workable option? Do they work at all? How effective are they? IDS will obviously only work against known threats.. how do you create an effective early warning system? How do you provide effective vaccination against an unknown threat?

Like I said above, I don't think it's your job to police the net. Not even if it's your customers' net. Having early warnings, sure, so that you can alert customers who are causing problems for nice folk like me. Doing anything about it, other than cutting off their access if they don't seem inclined to fix it? Nope, bad idea. What is this vaccination you are talking about? For your systems, fine. For mine, no thanks.

How do you respond to potentially massive infections of your subscriber base? Potential zombie manifestations in the 100k's are easily possible. They really do make Code Red's impact to date seem more like a case of a mild flu than any serious infection.

Put up a honeypot, on the inside of your network. Watch for unreasonable access, if you like. I think that this problem is not going to go away, but I think you also need to realize that your cure may be worse than the problem.

So, you do have a responsibility to your customers to protect them. To what extent is this realistic, though? Doesn't this also bear the risk of false security or even potential legal liabilities? How do you manage this risk?

You have a responsibility to your customers to provide packets. I never knew that you had any responsibility to protect anyone, other than cutting off access to idiots who can't tell they've got a problem. I don't understand where in your TOS it says you'll protect your customers. I'd want my credit card protected, if I was foolish enough to pay online (I'm not). I'd want any servers that are hosted by you to be protected. I want my network connection to stay up, and stay fast. Protect what? How? Yes, I realize that you are going to say I'm more sophisticated than most of your customers. I understand that. I'm still saying that it's not your job, not to protect.

You do have also a responsibility to "protect" the rest of the world from zombie gatherings among your subscribers. Same questions apply.

Here's where the importance of paying attention to what's going on comes in. I certainly agree with this statement. If you seem to have ongoing problems coming from inside your network, and you can identify where from, you have CONTACT information that you can use. Pick up the phone. Ask Joe Six Pack what the hell is going on. Maybe he doesn't know. Maybe he's doing it on purpose. I'm not talking about a little scanning here, either. Something like Code Red coming from a machine should trip somebody's trigger. It sure did mine.

So, I think it's clear that something needs to be done, but coming up with a definitive plan of attack is everything but trivial.

See above on the anti-FUD page. I think that some of the problems you are trying to solve are commendable, and some are not yours to solve. Pick the things you can fix, and go from there.

This obviously doesn't just apply to DSL, it applies to Cable and whatever other broadband networks are out there or will evolve...

We want to be a good stewart and citizen of the inet, yet, these questions are tough to answer in any satisfactory way it seems.

(Yes, I've taken some of these questions to various security forums from time to time, but none of them seem to represent a significant number of SPs; suggestions are very welcome).

I'm sure this isn't a comprehensive list.. but, perhaps, it'll get a useful conversation going. Hey, I can hope, right?

Well, I promised it'd be a book, and it is. Say, if anyone from XO is listening, how about creating a status page? -- Open source should be about giving away things voluntarily. When you force someone to give you something, it's no longer giving, it's stealing. Persons of leisurely moral growth often confuse giving with taking. -- Larry Wall