On Sun, Oct 04, 2009 at 04:33:43AM -0700, Owen DeLong wrote:

Uh... Here I differ. The rest of the internet should put up with the abuse flowing out of your network for 3 days to avoid disruption to you? Why? Sorry, if you have a customer who is sourcing malicious activity, whether intentional or by accident, I believe the ISP should take whatever action is necessary to stop the outflow of that malicious behavior as quickly as possible while simultaneously making all reasonable effort to contact the customer in question.

Exactly correct. The number one priority, which trumps all others, is making the abuse stop. Yes, there are many other things that can and should be done, but that's the first one. Let me also point out that there's a problem with offering simple, automated removal (as was suggested in the message that you replied to): resident malware on abuse-sourcing zombies will very quickly be reprogrammed to avail itself of that mechanism (on a per-ISP basis if necessary, if this becomes widespread). So there should be no automated removal process: the intervention of humans should be required, doubly so as in most cases the putative/former owner of the infected system is unaware of any of this. ---Rsk

Barton F Bruce wrote:

Stopping the abuse is fine, but cutting service to the point that a family using VOIP only for their phone service can't call 911 and several children burn to death could bring all sorts of undesirable regulation let alone the bad press and legal expenses.

While a legitimate concern it's also a red herring, as it's a technical problem looking for a technical solution. Gadi.

On Oct 5, 2009, at 11:23 AM, Barry Shein wrote:

Perhaps someone has said this but a potential implementation problem in the US are anti-trust regulations. Sure, they may come around to seeing it your way since the intent is so good but then again "we all decided to get together and blacklist customers who..." is not a great elevator pitch to an attorney-general no matter how good the intent.

That's not what is being discussed from my understanding. From my understanding, the intent is to share names of known abusers and data necessary to help in tracking DDOS. I don't believe that any ISP is expected to necessarily take any particular action determined by the group with respect to the list of names they are given. I do think that it is reasonable to have an agreement among an industry organization or collaboration which states that ISPs which determine that abuse is being sourced from one of their customers (either through their own processes or by notification from another participant) should be expected to take the necessary steps to mitigate that abuse from exiting said ISPs autonomous system.

Obviously there are ways around that (e.g., it's ok to do credit checks) but one has to be up-front and get approval.

I don't think that's true. I think that as long as your privacy policy and terms of service state that you will share certain information with other operators regarding abuse complaints and (possibly) abusive activities, you are free to share that information. Having a coalition rule that says any member must refuse to service any party on the list would be an anti-trust violation. Having a list, alone, without any rules about how the list is used, is not an anti-trust violation. Just like agreeing ahead of time on the price of gas amongst multiple competitors is an anti-trust violation, posting the price of gas at your service stations is not. Modifying your price to match the price across the street also is not.

I'm just sayin':

a) consult with legal counsel before doing anything in "collusion" with competitors.

Definitely.

b) this is probably not for smaller ISPs until the legal way is cleared by those with plenty of money for lawyering and lobbying.

I'm not so sure that is true, but, they should seek good legal counsel about whatever they plan to do regardless of size. Owen

Gadi Evron wrote:

Barton F Bruce wrote:

...

Stopping the abuse is fine, but cutting service to the point that a family using VOIP only for their phone service can't call 911 and several children burn to death could bring all sorts of undesirable regulation let alone the bad press and legal expenses.

While a legitimate concern it's also a red herring, as it's a technical problem looking for a technical solution.

Gadi.

I think the need for someone being able to call 911 from their VoIP outweighs your right to claim that they should be disconnected from the Internet. Besides, if that provider wants to help out, he might setup a captive portal or something with information regarding tools to clean their computer.

...

I think the need for someone being able to call 911 from their VoIP outweighs your right to claim that they should be disconnected from the Internet.

Again, I don't disagree, but I see it as a technical issue which is solvable. I don't see why this is THE issue. It's really just changing the subject of the discussion.

It is a technical issue that is already solved. There are deploy walled garden/quarantine systems in the US with 911 regulations which will shunt customers who are violated to a clean up section of the network. 911 and other services are not included in this shunt. The deployments are improving over time, as each operator who has deployed gains more experience. For a sample, check out: ftp://ftp-eng.cisco.com/SP-Security/Quarantine/QuaWhitePaper8.pdf And http://tools.ietf.org/html/draft-oreirdan-mody-bot-remediation-00

On Oct 6, 2009, at 1:20 AM, Eugeniu Patrascu wrote:

Gadi Evron wrote:

...

Barton F Bruce wrote:

...

Stopping the abuse is fine, but cutting service to the point that a family using VOIP only for their phone service can't call 911 and several children burn to death could bring all sorts of undesirable regulation let alone the bad press and legal expenses.

While a legitimate concern it's also a red herring, as it's a technical problem looking for a technical solution.

Gadi.

I think the need for someone being able to call 911 from their VoIP outweighs your right to claim that they should be disconnected from the Internet.

Besides, if that provider wants to help out, he might setup a captive portal or something with information regarding tools to clean their computer.

I disagree... Distributed Denials of Service have gotten to the point where they can actually endanger lives. Think about this... In order to be able to make your 911 VOIP call, the VOIP provider has to be able to process your call. The system that is getting disconnected because it is an active source of abuse may be one of many participating in a DOS against someone elses 911 VOIP provider. Removing them from the internet could be saving more lives than it risks. Someone else pointed out that if the system in question has been botted/owned/pwn3d/whatever you want to call it, then, you can't guarantee it would make the 911 call correctly anyway. Owen

mark [at] edgewire wrote:

The end problem is still users and really, these users will click on anything that has a bright and shiny button which says, Ok. Really, does setting up a portal help? Perhaps a "sandboxed" area which has some information on securing their machine and keeping it clean may be the way to go but how much more of a resource will it chew up?

And then the nice phisher people come in and they replicate the quarantine website of various providers (just check IP address, you know the ISP and present the appropriate page) after having lured them to some site they control. Then they simply have a nice big "Install this cool tool to update your computer" link et voila. The problem with all of that boils down to what people have to believe... and how to properly inform them of that... Yes, I think the sandbox/quarantine style things is the way to go for the time being, but there are other more important things that need fixing. (afaik) Most people will get infected by clicking on something at one point in time on some weird website, even after having googled it etc. The problem is that it is really hard to show to the user that a site is 'trustworty' or not, especially as everyone can just get an SSL certificate for faceb0ok.com and m1crosoft.com and soon also for all the nice variants in the IDN space, thus SSL doesn't state anything, it just makes the connection secure (aka unsniffable unless there is a 3 letter acronym doing so, or they have access to either end). And that would not help much either as even Facebook and other such sites have been used to distribute worms, thus it becomes really hard to do it on a domain basis, as what is on a domain at point X in time, will be different at point Y, thus distributing lists becomes problematic too. The company that can come up with a proper universal solution to that problem (and patent it so they can actually get the moneyz) will probly end up doing quite well. Most likely though it means restricting user freedom, which is the counter problem as that is something one doesn't want, and when there is an option to disable it, then people will just disable it. Greets, Jeroen

Re: VOIP, 911, bots Shape their bandwidth down to the minimum required to make a 911 call, around 64Kbps, and capture their web accesses. -- -Barry Shein The World | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: 800-THE-WRLD | Login: Nationwide Software Tool & Die | Public Access Internet | SINCE 1989 *oo*

Someone else pointed out that if the system in question has been botted/owned/pwn3d/whatever you want to call it, then, you can't guarantee it would make the 911 call correctly anyway.

I realize that many NANOG'ers don't actually use the technologies that we talk about, so I'm just going to correct this: You seem to be under the mistaken assumption that most people using VoIP do so using their computer. While it kind of started out that way years ago, it simply isn't so anymore. Most VoIP services can be configured to work with an analog telephony adapter, providing a POTS jack. Most VoIP services even provide one as part of the subscription, sometimes for a fee. There's also a growing number of phones that support Skype or generic VoIP, sometimes alongside a regular POTS line, sometimes not. It is perfectly possible to have an infected PC sitting right next to a nice VoIP-capable DECT cordless phone system, both hooked to the same NAT router. This is, of course, problematic, and would be a useful problem to contemplate how to cause one to break while keeping the other operational. Assuming that the existence of an infected PC in the mix translates to some sort of inability to make a 911 call correctly is, however, simply irresponsible, and at some point, is probably asking for trouble. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

Alexander Harrowell wrote:

On Wednesday 07 October 2009 00:27:55 Joe Greco wrote:

...

Assuming that the existence of an infected PC in the mix translates to some sort of inability to make a 911 call correctly is, however, simply irresponsible, and at some point, is probably asking for trouble.

... JG

Also, someone mentioned that the FCC doesn't in fact mandate that PSTN terminals should be able to make emergency calls even if formally disconnected and asked about cellular.

The opposite is true about GSM and its descendants; whether or not you're a valid roamer for the network you're talking to, have a prepaid balance, have paid your bill, you must be able to make emergency calls. Similarly, even if no SIM card is present, the device should register with the network as "limited service" - i.e. emergency only.

The FCC generally doesn't come into play when you're talking about ILEC telephone service except at a very high level. In California, by PUC regulation telephone companies are required to allow access to 911 so long as there is copper in the facility and it was, at any time, active with any sort of phone service. Ref: http://ucan.org/telenforcers/files/SBC%20complaint%20PUC%20version.pdf Ref2: http://law.onecle.com/california/utilities/2883.html I believe this is also the case in numerous other states.

On Oct 6, 2009, at 4:27 PM, Joe Greco wrote:

...

...

Someone else pointed out that if the system in question has been botted/owned/pwn3d/whatever you want to call it, then, you can't guarantee it would make the 911 call correctly anyway.

I realize that many NANOG'ers don't actually use the technologies that we talk about, so I'm just going to correct this:

You seem to be under the mistaken assumption that most people using VoIP do so using their computer. While it kind of started out that way years ago, it simply isn't so anymore. Most VoIP services can be configured to work with an analog telephony adapter, providing a POTS jack. Most VoIP services even provide one as part of the subscription, sometimes for a fee.

I do use VOIP, bot computer and non-computer based. None the less, the fact remains that should any of my systems become compromised, my ability to make a VOIP phone call is in doubt regardless of what the provider does.

Well, /that's/ obviously not true. Cable providers are already using PacketCable NCS (read: "MGCP lightly modified") to provide completely reliable QoS for their own VoIP-to-the-cablemodem products; you are going to find it tough to impact the service level of such a device. For general VoIP, there's no particularly good reason that the VoIP traffic cannot be QoS'd / filtered to allow VoIP to continue to work while gardening the remaining traffic from the customer. That is completely under the provider's control. Since many of the CPE devices actually have a programmable hardware ethernet switch, it is even possible to do a lot of the work in hardware.

Additionally the problems of DDOS sourced from a collection of compromised hosts could be interfering with someone else's ability to make a successful VOIP call.

I think the above addresses that. There are always risks, of course. The guy pruning tree branches down the street can knock down the cable line, for example. Of course, he probably takes out the phone lines as well... :-)

Abuse sources should be blocked from impacting the rest of the network.

Sure.

This blocking should be as narrow as possible.

Yes, that's my point. We should be able to narrowly block compromised hosts so that we don't screw up legitimate VoIP uses. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again." - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.

On 10/9/09, Rich Kulawiec <rsk@gsp.org> wrote:

On Wed, Oct 07, 2009 at 06:25:53AM -0700, Owen DeLong wrote:

...

Additionally the problems of DDOS sourced from a collection of compromised hosts could be interfering with someone else's ability to make a successful VOIP call.

Much more than that: they could be interfering with the underlying infrastructure, or they could be attacking the VOIP destination, or they could be making fake VOIP calls (see below), or they could be doing ANYTHING. A compromised system is enemy territory, which is why:

...

This blocking should be as narrow as possible.

Blocking should be total. A compromised system is as much enemy-controlled as if it were physically located at the RBN. Trying to figure out which of externally-visible behaviors A, B, C, etc. it exhibits might be malicious and which might not be is a loss,

If an ISP is involved with tracking down DDOS participants or something, I can understand how they'd know a system was compromised. But any kind of blocking because the ISP sees 'anomalous' traffic seems .. premature at best. SANS newsbites has this bit: On Thursday, October 8, Comcast began testing a service that alerts its broadband subscribers with pop-ups if their computers appear to be infected with malware. Among the indicative behaviors that trigger alerts are spikes in overnight traffic, suggesting the machine has been compromised and is being used to send spam. When my son comes home from college, there's a huge spike in overnight traffic from my house. With all the people advocating immediate blocking of pwned systems in this thread, I'm wondering what their criteria is for deciding that the system is compromised & should be blocked. Lee

...

Exactly correct. The number one priority, which trumps all others, is making the abuse stop. Yes, there are many other things that can and should be done, but that's the first one.

Stopping the abuse is fine, but cutting service to the point that a family using VOIP only for their phone service can't call 911 and several children burn to death could bring all sorts of undesirable regulation let alone the bad press and legal expenses.

As far as the Ducth situation with one of the largest providers (KPN) goes this is solved by using a seperate VLAN for VOIP traffic. Only the data VLAN is being "blocked" or actually policy routed towards a walled garden in which users are able to clean themselves up. -- Nils Kolstein SSCPlus

On Sun, 4 Oct 2009, Owen DeLong wrote:

...

* Provide a short period of time (3 days) after notification and before disconnect to give an opportunity to fix the issue without service interruption

Uh... Here I differ. The rest of the internet should put up with the abuse flowing out of your network for 3 days to avoid disruption to you? Why? Sorry, if you have a customer who is sourcing malicious activity, whether intentional or by accident, I believe the ISP should take whatever action is necessary to stop the outflow of that malicious behavior as quickly as possible while simultaneously making all reasonable effort to contact the customer in question.

Yeah, after a few people privately emailed me regarding the same, the short period of time should be thrown out, for the good of the rest of the 'net. The "short period" was initially intended for infections that were not active or immediately impacting, but were detected to be infected none-the-less. Assuming active "bad behavior" immediate disconnect is prudent and wise. As our ability to remotely detect virus and trojans improves, I suspect such an ISP-provided service would as well.

...

* Offer a simple, automated way to get the connection re-tested and unblocked immediately (within 15 minutes) using a web service accessible even if the connection is blocked

Either a web interface or even a telephonic process. It doesn't necessarily need to be automated, but, it shouldn't be a 3 day wait for a technician to get back to you. It should definitely be a pretty rapid process once the abuse is resolved.

Agreed. Another emailer mentioned that it's not always simple to determine if the abuse is resolved or not, nor is it easy to explain this to a non-technical customer in a way that makes them happy with their service being cut off. However it is ignorance and lack of maintenance that makes viruses and botnets so prevelant that it may just be time to bite the bullet and force users to learn how to maintain their machines.

...

* Force the customer to call customer service to ask for a retest or reconnect I don't really see a problem with this, so long as customer service is responsive to such a call.

I like self-service. If it is 3am and staff is not available, making the process automated would be ideal. If the staff is 24/7, agreed. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

Christopher Morrow wrote:

I would also point out that Qwest does this walled-garden approach for their customers (have been for at least 5 years now? DonS@qwest could clarify) and they've seen success with it. Aliant in .ca also has some fairly aggressive anti-malware works installed. There are places where this sort of thing works well, planned and engineered properly. I think Qwest, at least, made some of their reasoning and design/goals publicly available for a time as well.

I think Jonathan Curtis did something similar at Bell, but I only spoke with him about it for a couple of second two years ago, as Rio was rather distracting. So am unsure. Apparently, marketing departments like the idea of being able to send customers that need to pay them to a walled garden. It also saves on tech support costs. Security being the main winner isn't the main supporter of the idea at some places. Gadi.

Justin Shore wrote:

Gadi Evron wrote:

...

Apparently, marketing departments like the idea of being able to send customers that need to pay them to a walled garden. It also saves on tech support costs. Security being the main winner isn't the main supporter of the idea at some places.

I would love to do this both for non-pays and security incidents. I'd like to do something similar to let customers update their provisioning information for static IP changes so cable source verify doesn't freak out. Unfortunately I haven't been able to find any open source tools to do this. I can't even think of commercial ones off the top of my head.

It's a relatively simple concept. Some measure of integration into the DHCP provisioning system(s) would be needed to properly route the customer's traffic to the walled garden and only to the walled garden. Once the problem is resolved the walled garden fixes the DHCP so the customer can once again pull a public IP and possibly flushes ARP caches if your access medium makes that a problem to be dealt with.

I would think that the walled garden portion could be handled well-enough with Squid and some custom web programming to perform tasks to reverse the provisioning issues. I'm sure people have written internal solutions for SPs before but I haven't found anyone that has made that into an OSS project and put it on the Web. I'd love to make this a project but there is little financial gain to my small SP so if it costs much money it won't get management support.

Justin

There is all sorts of kit that will do this for you, Ellacoya, Redback etc. They all have APIs and all work well. The customer keeps their public IP address, but you can then make it belong to another virtual router instance, or you can apply certain firewall/ACL/policy rules to it. For example, my Ellacoyas will, for a walled customer, deny traffic to anything but the walled garden hosts and will then route any port 80 traffic to my proxy server that re-directs it all to a walled garden web server. Then soon as they hand over their payment details and we take payment, a request is sent to the Ellacoya to remove the restrictions. Lovaly. -- Leigh

-----Original Message----- From: Christopher Morrow [mailto:morrowc.lists@gmail.com] Sent: Sunday, October 04, 2009 4:04 PM To: Peter Beckman Cc: NANOG Subject: Re: Dutch ISPs to collaborate and take responsibility for botted clients

On Sun, Oct 4, 2009 at 2:55 PM, Peter Beckman <beckman@angryox.com> wrote:

...

 service being cut off.  However it is ignorance and lack of maintenance  that makes viruses and botnets so prevelant that it may just be time to  bite the bullet and force users to learn how to maintain their machines.

because this works so well with:

1) cars 2) home-security 3) personal security wandering around cities/towns

I note that I'm not particularly against any of the proposal, just the 'people need a drivers license to get on the interwebz'... it's been tried many times before, always without success.

I'm trying to understand your analogy, but it's hidden in the sarcasm. Your assertion is that education (and you've decided to include licensing, for some reason) of drivers and the rest is ineffective? You're not opposed to user education, you just believe it's useless because it will only reduce, not eliminate, badness? Lee