Broadband routers and botnets - being proactive
In this post I'd like to discuss the threat widely circulated insecure broadband routers pose today. We have touched on it before. Today, yet another public report of a vulnerable DSL modem type was posted to bugtraq, this time about a potential WIRELESS flaw with broadband routers being insecure at Deutsche Telekom. I haven't verified this one myself but it refers to "Deutsche Telekom Speedport w700v broadband router": http://seclists.org/bugtraq/2007/May/0178.html If you all remember, there was another report a few months ago about a UK ISP named BeThere with their wireless router being accessible from the Internet and exploitable, as another example: http://blogs.securiteam.com/index.php/archives/826 Two issues here: 1. Illegitimate access to broadband routers via wireless communication. 2. Illegitimate access to broadband routers via the WAN. I'd like to discuss #2. Some ISPs which provide such devices (as in the example of #2 above) use them as bridges only, preventing several attack vectors (although not all). Many others don't. Most broadband ISPs have a vulnerable user-base on some level. Many broadband ISPs around the world distribute such devices to their clients. Although the general risk is well known, like with many other security issues many of us remained mostly quiet in the hope of avoiding massive exploitation. As usual, we only delayed the inevitable. I fear that the lack of awareness among some ISPs for this "not yet widely exploited threat" has resulted in us not being PROACTIVE and taking action to secure the Internet in this regard. What else is new, we are all busy with yesterday's fires to worry about tomorrow's. Good people will REACT and solve the problem when it pops up in wide-exploitation, but what we may potentially be facing is yet another vector for massive infections and the creation of eventual bot armies on yet another platform. My opinion is, that with all these public disclosures and a ripe pool of potential victims, us delaying massive exploitation of this threat may not last. I believe there is currently a window of opportunity for service providers to act and secure their user-base without rushing. Nothing in security is ever perfect, but actions such as changing default passwords and preventing connections from the WAN to these devices would be a good step to consider if you haven't already. My suggestion would be to take a look at your infrastructure and what your users use, and if you haven't already, add some security there. You probably have a remote login option for your tech support staff which you may want to explore - and secure. That's if things were not left at their defaults. Then, I'd also suggest scanning your network for what types of broadband routers your users make use of, and how many of your clients have port 23 or 80 open. Whether you provide with the devices or not, many will be using different ones set to default which may pose a similar threat. Being aware of the current map of vulnerable devices of this type in your networks can't hurt. It is not often that we can predict which of the numerous threats out there that we do not address currently, is going to become exploited next. If you can spare the effort, I'd strongly urge you to explore this front and be proactive on your own networks. The previous unaddressed threat which most of us chose to ignore was spoofing. We all knew of it for a very long time, but some of us believed it did not pose a threat to the Internet or their networks for no other reason than "it is not currently being exploited" and "there are enough bots out there for spoofing to not be necessary". I still remember the bitter argument I had with Randy Bush over that one. This is a rare opportunity, let's not waste it. We are all busy, but I hope some of you will have the time to look into this. I am aware of and have assisted several ISPs, who spent some time and effort exploring this threat and in some cases acting on it. If anyone can share their experience on dealing with securing their infrastructure in this regard publicly, it would be much appreciated. Thanks. Gadi Evron.
Gadi, I and numerous others (including some whom any reasonable NANOG-L poster would respect and listen to) have asked you repeatedly to stop trolling NANOG-L with this botnet crap. It is off-topic here. The last time you pulled this (starting a 4-day troll-fest about a nonexistent "INNURNET EMERGENCY") I asked you to stop it, and not one of the legions of supporters you talk about spoke up to say "Wait, I want to see botnet crap on NANOG-L." Even if all 6 of your botnet-loving supporters spoke up, it would not change the fact that your botnet posts are off topic, unwanted, and disruptive. It's time for you to stop it. Please.
On 5/12/07, Albert Meyer <from_nanog@corenap.com> wrote:
I and numerous others (including some whom any reasonable NANOG-L poster would respect and listen to) have asked you repeatedly to stop trolling NANOG-L with this botnet crap. It is off-topic here. The last time you pulled this (starting
As frequent as Gadi is with his botnet posts, insecure and wide open CPE getting deployed across a large provider is definitely operational. srs
* Suresh Ramasubramanian:
As frequent as Gadi is with his botnet posts, insecure and wide open CPE getting deployed across a large provider is definitely operational.
And if Gadi's examples are not scary enoug for you, there are far more relevant vulnerabilities. It seems that the organization that assembles most of the firmware on those CPEs just takes the Sourceforge project with the smallest footprint they can find to implement a particular task. Not even a cursory code review takes place. As most of the software is GPLed, not just the firmware provider, but also the hardware manufacturer and the ISP itself could stop the deployment until the most egregious bugs have been fixed. Of course, you could argue that if Microsoft and Debian don't do this, why should ISPs? To me, the answer is that shipping vulnerable software is state of the art, but only if there is some kind of patch management appendix. Fortunately, there is a simple solution to this kind of problem: ISPs are very likely liable if they fail to alert customers about security problems, and do not provide updates in a timely manner. After a few painful incidents, the ISPs will learn, and either ship better software (unlikely) or implement some kind of patch management. With a bit of luck, the latter does not just shift back liability back to the customer, but also helps to parly solve the problem (in the sense that CPE attacks are less attractive).
On Sun, 13 May 2007, Florian Weimer wrote:
Fortunately, there is a simple solution to this kind of problem: ISPs are very likely liable if they fail to alert customers about security problems, and do not provide updates in a timely manner. After a few painful incidents, the ISPs will learn, and either ship better software (unlikely) or implement some kind of patch management. With a bit of luck, the latter does not just shift back liability back to the customer, but also helps to parly solve the problem (in the sense that CPE attacks are less attractive).
It won't solve the problem. ISPs will simply stop distributing CPE, and tell customers to buy CPE from their nearest electronics store (Best Buy, Radio Shack, or the equivilent in other countries). If you thought it was hard getting ISPs to patch CPE, try getting electronics stores to patch the CPE. Look at the ancient bugs in D-Link, Linksys, Netgear boxes that consumers haven't figured out how to patch for years. You really need to identify the sources and fix it there.
On Sun, 13 May 2007, Sean Donelan wrote:
On Sun, 13 May 2007, Florian Weimer wrote:
Fortunately, there is a simple solution to this kind of problem: ISPs are very likely liable if they fail to alert customers about security problems, and do not provide updates in a timely manner. After a few painful incidents, the ISPs will learn, and either ship better software (unlikely) or implement some kind of patch management. With a bit of luck, the latter does not just shift back liability back to the customer, but also helps to parly solve the problem (in the sense that CPE attacks are less attractive).
It won't solve the problem. ISPs will simply stop distributing CPE, and tell customers to buy CPE from their nearest electronics store (Best Buy, Radio Shack, or the equivilent in other countries). If you thought it was hard getting ISPs to patch CPE, try getting electronics stores to patch the CPE. Look at the ancient bugs in D-Link, Linksys, Netgear boxes that consumers haven't figured out how to patch for years.
You really need to identify the sources and fix it there.
"Passing the buck! Buck passer!" (see below - skip to Dilbert link) Not saying that you are wrong but... Ahh, these are out of our control, nor will they do anything if we don't. Might as well tell users not to patch their Windows systems as it's the responsibility of the store who sold them the computer. Yes, it could help if the stores did something. There is little to no financial incentive for ISPs to do something about this problem right now, even if it is currently under their direct control. Later on, when it is a problem - it will cost more. Today? Some will do someting, others won't. It surprises me how many do invest in this. Almost everything we do in Internet security operations has nothing to do with identifying the problem and fixing it. It's usually just about identifying the sympthoms and getting rid of them. It's like I sometimes tell law enforcement: "we can't afford to wait, we need to maintain our networks". We wait anyway and end up eating a sock. As to your suggestion here (quoting a /. user who wrote it down): Dilbert is in the Boss's office. Dilbert: I discovered a hole in our internet security. Boss: What?!! Boss: Good grief, man! How could you put a hole in our internet? Dilbert, angry: I didn't PUT it there, I FOUND it.. and it's not... Boss: It's your job to fix that hole. I want you to work 24-7! Dilbert: Actually, that's NOT my job. But I'll inform our network management group. Boss, yelling: PASSING THE BUCK!!! YOU'RE A BUCK PASSER!!! Dilbert: Forget it! There's no hole! It got better! Boss: That's more like it. Last panel, the boss is sitting alone smiling. Boss thinks: I fixed the internet. I found it on Google images: http://stderr.de/funstuff/dilbert_fixed_the_internet.jpg
Hi Gadi, reading all the email re off topic etc is wrong. If this issue is dealt with then transit bandwidth will be less, security will improve and the end user experience will be better. Great dilbert cartoon Colin
On Sun, 13 May 2007, Gadi Evron wrote:
There is little to no financial incentive for ISPs to do something about this problem right now, even if it is currently under their direct control. Later on, when it is a problem - it will cost more.
So, out of curiousity, could you define: "under their control" for me/us? I'm certainly think that weak CPE security is a problem, but I'm not sure I agree that the CPE which is purchased as part of your service and managed by you (you == random-customer in this example) is anyone's problem except yours. If you think that the lobbying groups inside consumer ISP's are going to let through some set of regulations that gets them on the hook without substantial rate increases you really need to re-evaluate that assumption :( Some of this comes back to making people understand that computer security is no different than anyother form of security (house, car, boat, wallet) the only thing that is different is it's age. -Chris
On Mon, 14 May 2007, Chris L. Morrow wrote:
On Sun, 13 May 2007, Gadi Evron wrote:
There is little to no financial incentive for ISPs to do something about this problem right now, even if it is currently under their direct control. Later on, when it is a problem - it will cost more.
So, out of curiousity, could you define: "under their control" for me/us? I'm certainly think that weak CPE security is a problem, but I'm not sure I agree that the CPE which is purchased as part of your service and managed by you (you == random-customer in this example) is anyone's problem except yours.
If you think that the lobbying groups inside consumer ISP's are going to let through some set of regulations that gets them on the hook without substantial rate increases you really need to re-evaluate that assumption :(
Some of this comes back to making people understand that computer security is no different than anyother form of security (house, car, boat, wallet) the only thing that is different is it's age.
No arguments here, thank you for the clarification.
-Chris
On Sun, 13 May 2007, Gadi Evron wrote:
"Passing the buck! Buck passer!" (see below - skip to Dilbert link)
I guess you missed my attempts 3 or 4 years ago at trying to establish some standards for CPE concerning security. I've been at this party for a long time, I know how the song ends.
Not saying that you are wrong but... Ahh, these are out of our control, nor will they do anything if we don't. Might as well tell users not to patch their Windows systems as it's the responsibility of the store who sold them the computer. Yes, it could help if the stores did something.
I spent about a year of my life working with Microsoft on getting patched versions of Windows in the pipeline and getting OEMs to regularly update their manufacturing copy being pre-installed on machines as they leave the factory. So yes, it did help to improve things in the pipeline.
On Sun, 13 May 2007, Sean Donelan wrote:
On Sun, 13 May 2007, Gadi Evron wrote:
"Passing the buck! Buck passer!" (see below - skip to Dilbert link)
I guess you missed my attempts 3 or 4 years ago at trying to establish some standards for CPE concerning security. I've been at this party for a long time, I know how the song ends.
Not saying that you are wrong but... Ahh, these are out of our control, nor will they do anything if we don't. Might as well tell users not to patch their Windows systems as it's the responsibility of the store who sold them the computer. Yes, it could help if the stores did something.
I spent about a year of my life working with Microsoft on getting patched versions of Windows in the pipeline and getting OEMs to regularly update their manufacturing copy being pre-installed on machines as they leave the factory. So yes, it did help to improve things in the pipeline.
Just a joke, Sean. What would you consider from your experience, the best way to make these third parties take responsibility?
On Mon, 14 May 2007, Gadi Evron wrote:
Just a joke, Sean. What would you consider from your experience, the best way to make these third parties take responsibility?
First, you need to identify the ODM making the software used in the CPE. -- Warning: Be careful signing up for UltraDNS services. Their terms are longer than they appear.
Sean Donelan wrote:
On Sun, 13 May 2007, Florian Weimer wrote:
Fortunately, there is a simple solution to this kind of problem: ISPs are very likely liable if they fail to alert customers about security problems, and do not provide updates in a timely manner. After a few painful incidents, the ISPs will learn, and either ship better software (unlikely) or implement some kind of patch management. With a bit of luck, the latter does not just shift back liability back to the customer, but also helps to parly solve the problem (in the sense that CPE attacks are less attractive).
It won't solve the problem. ISPs will simply stop distributing CPE, and tell customers to buy CPE from their nearest electronics store (Best Buy, Radio Shack, or the equivilent in other countries). If you thought it was hard getting ISPs to patch CPE, try getting electronics stores to patch the CPE. Look at the ancient bugs in D-Link, Linksys, Netgear boxes that consumers haven't figured out how to patch for years.
You really need to identify the sources and fix it there.
When your cpe costs $50 (to the consumer) it's not worth anyone's time (consumer, isp, manufacturer, store that sold it etc) to patch/upgrade the thing. If it's broken enough they'll eventually buy another one. or they'll buy another one because they decide they need some wazoo new feature, (802.11n, gigabit ethernet, p2p support, hard-disk etc)... The trick is insuring that when they do buy another one it;s tangibly better than the old one. Even if your cpe costs more (cisco 8xx) it's still not worth patching it if that is going to require external support (first time you call the tac you blow the profit on a cisco 800). Just remember, very few of these cpe devices existed 5 years ago, the probability that the same one's will be in use in 5 years seems pretty low. Deliver a compelling new technology platform and the users will upgrade en-masse (50mbit vdsl, ftfh, docsis 3 cable modems, fixed wimax, etc) It's my opinion that access isp's need to get out of the business of selling/delivering cpe because frankly the consumer will probably spend more on features and so forth, than the isp will when they lease you some crappy actiontec dsl router for 3-bucks a month. The isp's shoot themselves in the foot by shoveling the cheapest cpe they can out the door when the consumer would probably go out and pay for it if they felt like they weren't getting jacked.
Gadi Evron wrote:
[snip]
The previous unaddressed threat which most of us chose to ignore was spoofing. We all knew of it for a very long time, but some of us believed it did not pose a threat to the Internet or their networks for no other reason than "it is not currently being exploited" and "there are enough bots out there for spoofing to not be necessary". I still remember the bitter argument I had with Randy Bush over that one. This is a rare opportunity, let's not waste it.
We are all busy, but I hope some of you will have the time to look into this.
I am aware of and have assisted several ISPs, who spent some time and effort exploring this threat and in some cases acting on it. If anyone can share their experience on dealing with securing their infrastructure in this regard publicly, it would be much appreciated.
I don't know who the "us" is who you are referring to. One of the first things I did when I took over the management of the network at $DAYJOB was to tighten up the packet filtering at the edge of my network. That included fixing up the inbound and outbound filters: * blocking most "small services" inbound * blocking ports inbound used in widespread attacks * blocking multi-cast IP addresses inbound * blocking BOGON and RFC1918 source IP addresses inbound * blocking non-owned IP source addresses, including RFC1918, outbound * null-routing RFC1918 target addresses outbound (Under consideration but not yet implemented: null-routing BOGON target addresses) In my research into my new job, I got the impression that the above was considered one of the Best Current Practices for router configuration. I currently have a customer who is getting DDoSed by someone spoofing the source IP address in a TCP SYN flood. The problem us bad enough that I'm building a level-2 firewall (using a Linux box) to rate-limit TCP SYN to port 80 on his two IP addresses, and to raise an alarm when the incoming rate exceeds a threshold. When I ask my upstream where the SYN flood is entering *his* routers, the answer is "everywhere, I see these packets on every single upstream port I have." The last time I was able to do a packet capture and analysis during the flood, I found the source IP address of the packets that got through were evenly distributed across the IP address spectrum, with obvious notches in BOGON, RFC1918, and multicast IP ranges. (For those of you who like to build tools, I found using a FFT of the source addresses to be an excellent tool for analyzing traffic patterns.) So I don't have a problem sourcing such floods, because my ACLs block attempts to do so. I sure have problems sinking them.
Gadi, I appreciate your well thought out email but I sit here and wonder what exactly you are trying to accomplish with it? Are you just trying to shame the two ISPs listed publicly or are you trying to spark a discussion about something that many people here can't fix? Many businesses today are focused on driving revenue and fixing old CPE equipment doesn't generate revenue, it only ties up money and resources that can be used elsewhere to drive revenue. If I were you I would try to spin this problem in a way where you can show large ISPs by fixing CPE's it will free up network resources and staff which can be used elsewhere. The people that can fix these problems are usually unaware of them so try to educate those people. Write CEOs/CTOs/CSOs educating them and push the security teams for these companies to escalate these issues to their upper management (on that note I would say this type of discussion would be better suited for a security mailing list for the reason I stated before, many people here can't fix these problems). Simply stating that there is a problem and shunning ISPs with this problem isn't a fix for the problem, it just makes them ignore you and the problem. -Ross
On Wed, 16 May 2007, Ross Hosman wrote:
Gadi,
I appreciate your well thought out email but I sit here and wonder what exactly you are trying to accomplish with it? Are you just trying to shame the two ISPs listed publicly or are you trying to spark a discussion about something that many people here can't fix?
Many businesses today are focused on driving revenue and fixing old CPE equipment doesn't generate revenue, it only ties up money and resources that can be used elsewhere to drive revenue. If I were you I would try to spin this problem in a way where you can show large ISPs by fixing CPE's it will free up network resources and staff which can be used elsewhere.
The people that can fix these problems are usually unaware of them so try to educate those people. Write CEOs/CTOs/CSOs educating them and push the security teams for these companies to escalate these issues to their upper management (on that note I would say this type of discussion would be better suited for a security mailing list for the reason I stated before, many people here can't fix these problems).
Simply stating that there is a problem and shunning ISPs with this problem isn't a fix for the problem, it just makes them ignore you and the problem.
You are quite right. Thank you. I found some ways of showing several issues to be revenue-tied, such as blocking port 25, etc. This issue is something I am at a stage of exploring, and like it or not.. network operators are the ones who deal with this (on whatever level they do). I am unsure of where else to go with this, and if some ISPs do something for now, that is a step in the right direction until a better way shows itself. Whichever way we discover, for now, raising awareness is all I can think of. On a sarcastic evil tone, we may just plan to release a "fix" worm to harden all these devices world-wide. Right! Because that worked so well for us before. :>
-Ross
Gadi.
Ross Hosman wrote:
Gadi,
I appreciate your well thought out email but I sit here and wonder what exactly you are trying to accomplish with it? Are you just trying to shame the two ISPs listed publicly or are you trying to spark a discussion about something that many people here can't fix?
Many businesses today are focused on driving revenue and fixing old CPE equipment doesn't generate revenue, it only ties up money and resources that can be used elsewhere to drive revenue. If I were you I would try to spin this problem in a way where you can show large ISPs by fixing CPE's it will free up network resources and staff which can be used elsewhere.
The people that can fix these problems are usually unaware of them so try to educate those people. Write CEOs/CTOs/CSOs educating them and push the security teams for these companies to escalate these issues to their upper management (on that note I would say this type of discussion would be better suited for a security mailing list for the reason I stated before, many people here can't fix these problems).
Simply stating that there is a problem and shunning ISPs with this problem isn't a fix for the problem, it just makes them ignore you and the problem.
-Ross
Hi Ross, Gadi is talking about DTAG.de our biggest ISP in germany and quasi a monopoly. Gadi has reached the ears of the Pirates Party, a political party that fights monopolies. The hardware is very likely a branded version from AVM. They have no updates for the branded version, but you can unbrand it. Then you have a hardware that accepts open source firmware. Kind regards Peter and Karin -- Peter and Karin Dambier Cesidian Root - Radice Cesidiana Rimbacher Strasse 16 D-69509 Moerlenbach-Bonsweiher +49(6209)795-816 (Telekom) +49(6252)750-308 (VoIP: sipgate.de) mail: peter@peter-dambier.de mail: peter@echnaton.arl.pirates http://iason.site.voila.fr/ https://sourceforge.net/projects/iason/ http://www.cesidianroot.com/
participants (11)
-
Albert Meyer -
Chris L. Morrow -
Colin Johnston -
Florian Weimer -
Gadi Evron -
Joel Jaeggli -
Peter Dambier -
Ross Hosman -
Sean Donelan -
Stephen Satchell -
Suresh Ramasubramanian