----- Original Message ----- From: "Christian Kuhtz" <ck@arch.bellsouth.net>

Hey there,

Hi Chris,

In our case, we have several hundred thousands of DSL customers today, and

the

million plus subscriber mark is on the engineering horizon. The problem of security threats & resulting incidents is going to get considerably worse before it gets better. And that's for at least two reasons.. the ramp up of broadband and presumably the declining sophistication of the subscriber population as a result of the greater market penetration.

Lack of security knowledge is also a huge problem in the collocation market. The percentage of security-conscious colo customers is low, and uneducated customers tend to be connected to Fast Ethernet or Gigabit Ethernet feeds. One compromise in a well-connected colo can have the same effect as hundreds of broadband compromises, although it's easier to track down and stop.

How do you effectively scale the massive support effort need for

collaborative

marketing of personal firewalls and the potential for false positives and negatives? Any ideas on the legal exposure of security services?

My former employer ran several different managed firewall services, both for broadband customers and colos, with some success ... all firewalls were either built into CPE or designed as part of a "managed colo" environment where large numbers of colo customers were front-ended by a few firewalls. I don't see the broadband issue fixing itself without some built-in stateful inspection firewall in the CPE itself -- if the customer has to pay for an additional piece of hardware or software, it will instantly reduce penetration. If you can do what you need from a firewalling standpoint on the CPE, it makes life a lot easier. If you can provide a default firewall installation on your choice of CPE, configuration scaling becomes much easier. As we were never taken to court on the legal issues of managing a firewall, the only shot in the dark I would make there is that our standard contract made no warranty for host security at all, and accepted no liability, even though the 'wall was managed. It's impossible to manage network security, but not host security, and make any guarantees.

Like, in the current case, several providers have resorted to blocking

to their non-DIA subscriber base. Is this really scalable? Obviously not for every threat. You can't effectively keep this up with the myriad of

port 80 threats.

Or can you?

<this should get the crowd rolling> I'd think that a good default stance would be to block all incoming TCP connections that aren't part of an established session, for all broadband customers. Most of them would never notice, as email and http still work. However, at the scale you're talking about, I don't see blocking anything on the aggregation device itself ... it'd have to happen in the CPE, since firewall rules are going to have to be customized for clients who do need to run servers on their LAN.

So, I think it's clear that something needs to be done, but coming up with

a

definitive plan of attack is everything but trivial.

This obviously doesn't just apply to DSL, it applies to Cable and whatever other broadband networks are out there or will evolve...

Instant results could be had, by: - firewalling *all* broadband customers by default, with a strict ruleset that denies incoming connections. Most of these customers don't know that they're running open servers on what they consider desktop machines. NT and Linux are easy to yell at, but anyone seen a default installation of Solaris 8 lately? OS vendors are not going to fix this any time soon, which leaves greater responsibility on the shoulders of whoever connects the machines to the 'net. - requiring a firewall on all collocations, whether administered by you (for money) or by the customer. Exceptions exist, i'm sure, but there aren't that many of them as compared to the vast number of poorly secured installations. - if you're going to go this far, firewall all T1 customers too ... CPE exists that will happily do this. these three should take care of a lot of the "oops i left that port open" security problems, and I'd be interested to know what percentage of major infections could be cured by just putting up a firewall. Of course, it requires a lot more staff to manage, so that cost is going to affect your bottom line. That doesn't, however, address something like code red exploiting a legitimate web server. As I'm already in security nazi mode, how about: - put it in your contract that you will run vulnerability scans against all customer servers, with advance notice if necessary. Run them. - put a required response time in your contract for a customer to fix any vulnerabilities found. Shut them off if they don't patch the holes. (note that if you are firewalling the customer, you don't have to unplug them, just block access to the server until it's fixed) - recognize your limits. try to stop the canned, automated exploits -- after the fact if necessary, but realize that a determined attacker would require much greater effort to stop. That one may be best left to the customer. This is really just a small start, but if you're going to get this serious about security, some amount of education is necessary ... even if it's just so you can say "i told you so." Translate and bounce all CERT, security focus, etc. exploit announcements to your customer base, making it clear what is in danger and what must be done to fix it. Remind customers that they may be shut down if they're compromised. Put links up to patches on your website. Run an abuse department that responds quickly to customers, and to other providers, within limits. 24 x 7 is necessary, responding instantly to black ice freaking out because someone ran nmap past it is not. I've probably ranted enough already, but I don't think that the cost of adequate security has *ever* been addressed at SPs. I may be wrong, but the emphasis just doesn't seem to be there. In order to protect yourself from your customer base, you have to incur significant costs ... how do you compete with someone who doesn't care? How do you blackball those who don't care, so you can compete with them? Is this something that requires general agreement from the SP community, or is it possible for single companies to make an effort without pricing themselves out of the market? I've probably already stepped on enough toes this morning, so i'll drop it ... except to say it's nice to see someone trying to wrap their head around these issues. Cheers. -travis

Cheers, Chris

-- Christian Kuhtz <ck@arch.bellsouth.net> -wk, <ck@gnu.org> -hm Sr. Architect, Engineering & Architecture, BellSouth.net, Atlanta, GA, U.S. "I speak for myself only."