This afternoon's panel about IoT's lack of security got me thinking... On the issue of ISPs unable to act on insecure devices because they can't detect the devices until they're compromised and then only have the largest hammer (full account ban) to act... What about some kind of requirement or convention that upon boot and successful attachment to the network (and maybe once a month thereafter), any IoT device must _by default_ emit a UDP packet to an anycast address reserved for the purpose which identifies the device model and software build. The ISP can capture traffic to that anycast address, compare the data against a list of devices known to be defective and, if desired, respond with a fail message. If the IoT device receives the fail message, it must try to report the problem to its owner and remove its default route so that it can only communicate on the local lan. The user can override the fail and if desired configure the device not to emit the init messages at all. But by default the ISP is allowed to disable the device by responding to the init message. Would have to cryptographically sign the fail message and let the device query the signer's reputation or something like that to avoid the obvious security issue. Obvious privacy issues to consider. Anyway, throwing it out there as a potential discussion starting point. The presentation on bandwidth policers... Seems like we could use some form of ICMP message similar to destination unreachable that provides some kind of arbitrary string plus the initial part of the dropped packet. One of the potential strings would be an explicit notice to the sender that packets were dropped and the bandwidth available. Yes, we already have ECN, but ECN tells the receiver about congestion, not the sender. More to the point, ECN can only be flagged on packets that are passed, not the packets that are dropped, so the policer would have to be complicated enough to note on the next packet that the prior packet was dropped. Also, ECN only advises that you're close to the limit not any information about the policer's target limit. This thought is not fully baked. Throwing it out for conversation purposes. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Mon, Feb 6, 2017 at 7:14 PM, joel jaeggli <joelja@bogus.com> wrote:
On 2/6/17 2:31 PM, William Herrin wrote:
This afternoon's panel about IoT's lack of security got me thinking...
Hi Joel, For clarification I was referring to this: http://nanog.org/meetings/abstract?id=3051 The long and short of the panel was: as an industry (device vendors and service providers both) it behooves us to voluntarily get on top of the IoT security problem before some catastrophic event requires the government to dictate the precise manner in which we will get on top of the problem.
I'm not sure how we get on top of the problem without offering an effective network kill switch to the nearest security-competent person. I think I'd prefer a user-disableable kill-switch used on a single piece of equipment to a kill switch for my entire Internet connection. The IPv6 SLAAC address suffers a rather worse case of the privacy problem since it allows the entire Internet to track your hardware, not just your local ISP. In any case, I thought "how do we fix this long term" could stand discussion on the list. Because yes, the IoT device vendors mostly produce trash and if (to borrow a phrase) it saves them a buck at retail they will keep producing trash. But we're the ones letting that trash cause nation-scale problems and when the regulatory hammer crashes down it's gonna hit us all. On Mon, Feb 6, 2017 at 7:10 PM, Michael Thomas <mike@mtcc.com> wrote:
Uh, yuck at many levels. Do you leak your cisco ios versions to the internet?
Hi Michael, I'm not aware of any Cisco IOS devices that qualify as IoT. Some lighter weight Cisco gear, yes. And no, I do not want to broadcast my information. But I'm professional who customizes my gear when I plug it in. I don't run with the defaults.
Do you really want the responsibility for the remote kill switch for IoT S&M gear?
I already have the kill switch for the customer's entire S&M transit link. I'd prefer to also have a smaller hammer whose use won't net me a furious call from Sales.
And of course, you're depending on rfc 3514, right?
Nope. I'll decide what's evil and what's not (more likely I'll pay a service to provide me a regularly updated database) and I depend only on a high enough percentage of the devices offering themselves up for that decision that it becomes impractical to construct another Mirai. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Mon, Feb 06, 2017 at 05:31:10PM -0500, William Herrin wrote:
I can think of at least four reasons why this idea must be killed immediately and permanently. This is off the top of my head *before* coffee, so I strongly suspect there are more. 1. An attacker who takes control of an IoT device can change the contents of that packet, cause it to be emitted, suppress it from being emitted, etc. 2. This will allow ISPs to build a database of which customers have which IOT devices. This is an appalling invasion of privacy. 3. This will allow ISPs to build a database of which customers have which IOT devices. This will create one-stop shopping for attackers. 4. It won't take long for this to be used as a DDoS vector. ---rsk
On Tue, Feb 7, 2017 at 5:26 AM, Rich Kulawiec <rsk@gsp.org> wrote:
Hi Rich, Immaterial. The point is to catch vulnerable devices before they're hacked. That can't always happen (even with customers and vendors engaged in best practice patching), but it need only happen often enough to limit the size of the resulting botnets.
2. This will allow ISPs to build a database of which customers have which IOT devices. This is an appalling invasion of privacy.
As I envision it, it's an opt-out system. Anyone who cares and knows enough to secure their network can turn it off for their devices. No permission or action from the ISP required to turn it off. In fact, you need only block packets to the well-known anycast address to disable it for all devices on your network.
3. This will allow ISPs to build a database of which customers have which IOT devices. This will create one-stop shopping for attackers.
Possible, though an ISP which retains the data opens itself to a class action lawsuit if it can't keep control of it. Nevertheless, let's not oversell the privacy implications: most of the IoT devices we're talking about phone home anyway. They're tied to this or that cloud service rather than accepting local access. An ISP can't learn as much information by watching who they phone home to (deep packet inspection), but they can learn an awful lot.
4. It won't take long for this to be used as a DDoS vector.
If the ISP can't keep control of its security head-end then sure, at least until the ISP regains control. Regardless, I encourage you to suggest alternate solutions which don't run afoul of the problems you mention. The problem isn't going away. Entirely cutting off customers doesn't work because there are too many impacted customers and ISPs don't have the resources or the will to do so. Demanding that trash vendors not produce trash won't get it done either. When you eliminate responsible device hardening, cutting customer connections and the kill switch, what's left? Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
" any IoT device must _by default_ emit a UDP packet to an anycast address reserved for the purpose which identifies the device model and software build. " Any semi-competent attacker will simply alter the way the network stack on the device works to make it _not_ look like an IoT device for the purposes of this check, rendering it moot. "If the IoT device receives the fail message, it must try to report the problem to its owner and remove its default route so that it can only communicate on the local lan." If the vendor isn't thinking about security already, as evidenced by the current issues with these devices, how are they to be convinced to add code to make said device do something exceptionally specific? I believe it's important to remember that network endpoints will never be sure fully secure. They will never be fully in compliance with $standard . They will never always follow best practices. This is not to say we should do nothing (obligatory 'Perfect is the enemy of good' comment) but it's something to think about when discussing possibilities. On Tue, Feb 7, 2017 at 6:56 AM, William Herrin <bill@herrin.us> wrote:
On Tue, Feb 7, 2017 at 8:13 AM, Tom Beecher <beecher@beecher.cc> wrote:
Hi Tom, Again, the idea is to remediate devices with insecure software -before- they're breached. Obviously once breached the attacker can present any profile he wants save that it has to emit packets consistent with his desired attack.
Because it's consistent with the carelessly throw-together free open source software process the trash vendors currently follow and when the industry promotes its "look for the safe network logo" program they won't want to have problems with their retailers over a trivially added bit of free software.
You can say that again. On Tue, Feb 7, 2017 at 8:58 AM, Ray Soucy <rps@maine.edu> wrote:
Hi Ray, I can speak to that with authority, but again my working theory is that adding the kill switch is consistent with the carelessly throw-together free open source software process the trash vendors currently follow.
One of us is missing something. The customer isn't scanning his network (if he was, we wouldn't have the problem), and from the outside of an SPI firewall how is the ISP supposed to do that?
The customer buys cool stuff he finds at CostCo. Seeking different behavior seems unlikely to succeed to me. On Tue, Feb 7, 2017 at 9:50 AM, Rich Kulawiec <rsk@gsp.org> wrote:
Well that's a silly misdirection Rich. Manufacturer's misbehavior is a whole different class of problem than equipment breached by a third party after deployment. Unless you claim a convenient way to merge solutions, take it to your own thread.
I'm not estimating the threat at all, just stating the precondition for a kill switch to be a denial of service vector.
And I've heard this song before too: If you choose not to decide you still have made a choice. Neither one of us will like the government solution, so if you have an idea that hasn't already demonstrated its failure, let's hear it. When there's no consensus on an optimal solution, we move forward by trying ideas until something sticks. That's progress, including the attempts which fail. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Tue, Feb 07, 2017 at 06:56:40AM -0500, William Herrin wrote:
This won't work on the majority of devices: they're pre-compromised at the factory. By the time you see the first packet from them, IF you see the first packet from them, it will already be too late. And a lot of them are deliberately designed and built to conduct attacks, e.g.: Vizio tracked and sold your TV viewing habits without consent https://www.engadget.com/2017/02/06/vizio-smart-tv-viewing-history-settlemen... What a nice favor to do for attackers: Vizio spent its time and money putting this in place for them, so they didn't have to. This is going to get worse -- MUCH worse -- as the few flimsy consumer protections in place are systematically dismantled and the agencies charged with enforcing them defunded.
As I envision it, it's an opt-out system.
No. In fact, HELL NO. Too many ISPs have already put absolutely clinching proof on the table that they will silently opt-in customers to all manner of tracking, surveillance, and security/privacy attacks. (I'm looking at you, Verizon, at the moment.) Opt-out is inherently abusive.
Possible, though an ISP which retains the data opens itself to a class action lawsuit if it can't keep control of it.
First, that's a meaningless remedy. Even if someone has the financial resources to sue an ISP, they're going to wind up quietly settling years later, the lawyers will get rich(er), the settlement will be sealed, and they'll just keep right on doing it. See the Vizio case above. Did anybody go to prison? No. Did it cost them anything? Not really: the fine is just chump change. Will Vizio do it again? OF COURSE they will, they'd be crazy not to. All they have to do is wait a little while, rename it, shuffle things around a bit, and they'll be fine. Second, the most likely outcome here is that the ISP will use this data to spy on its users and market to them. The next outcome is that someone inside the ISP will figure out that this is a potential revenue source and start selling it, under the argument that consumers didn't opt-out, therefore they WANT their private data sold. So let's not pretend that the delusional fiction of a successful class-action lawsuit is a deterrent. It's just a belly laugh for the executives and a windfall for the attorneys.
If the ISP can't keep control of its security head-end then sure, at least until the ISP regains control.
I think you're severely underestimating the threat. And note that even though you're just thinking about the problem from the perspective of ISPs, they're not the only ones affected. There are a lot of attack vectors available to anyone who's already on the inside.
Regardless, I encourage you to suggest alternate solutions which don't run afoul of the problems you mention. The problem isn't going away.
I'm aware of that. But this is not the way. I've seen this movie WAY too many times: 1. We must do something 2. This is something. 3. Let's do this. 4. We have done something. 5. Congrats and awards all around, everybody. ---rsk
In a recent article ( https://www.schneier.com/blog/archives/2017/02/security_and_th.html), Bruce Schneier sums up the IoT security mitigation issue quite nicely in this paragraph: "The market can't fix this because neither the buyer nor the seller cares. The owners of the webcams and DVRs used in the denial-of-service attacks don't care. Their devices were cheap to buy, they still work, and they don't know any of the victims of the attacks. The sellers of those devices don't care: They're now selling newer and better models, and the original buyers only cared about price and features. There is no market solution, because the insecurity is what economists call an externality: It's an effect of the purchasing decision that affects other people. Think of it kind of like invisible pollution." - Ed Lopez -- Ed Lopez | Security Architect | Corsa Technology Email: ed.lopez@corsa.com Mobile: +1.703.220.0988 www.corsa.com sent from my iPad ... I apologize for any auto-correct errors
On Tue, Feb 07, 2017 at 10:01:29PM +0000, Ed Lopez quoted Bruce Schneier:
This is precisely correct. The only way to change this is to make *our* problem *their* problem. Let me remind everyone of one of very best things ever said on this mailing list: If you give people the means to hurt you, and they do it, and you take no action except to continue giving them the means to hurt you, and they take no action except to keep hurting you, then one of the ways you can describe the situation is "it isn't scaling well". --- Paul Vixie This movie has been playing here 24x7 for the last few decades with the spam problem (among others): most operations which emit it will take absolutely no action of any kind until/unless it stops being *our* problem and starts being *their* problem. Having observed and studied that particular issue since it existed to be observed and studied, I've concluded that the only thing that has ever worked effectively is blacklisting: that is, the revocation of access/service privileges. And that's because it makes *our* problem *their* problem. Everything else is just avoidance and evasion. It's feel-good stuff that might, on a good day, deal with the symptoms of the problem -- but that's all it is. And dealing with symptoms isn't bad, per se: it makes the patient feel better. But it should never be accepted as a substitute for dealing with the underlying problem. Now we can either spend another couple more decades trying to tapdance around this or we can learn the lesson that's been taught to us thousands of times over many years and just cut to the chase. So how about if we save some time? If IOT-driven attacks and abuse are coming from X, then that needs to be made X's problem. Because right now X has no idea that this is happening, and even if told, will take no action because it's not X's problem. So make it X's problem. And I don't just mean "X", the person who bought some badly-designed poorly-engineered rushed-to-market never-tested piece of shiny new cruft that was pre-compromised at the factory and hijacked by attackers the moment it went live: I mean "X" the vendor who pulled that stunt in order to make a quick buck and then dumped it on us. We, for many values of "we" are not obligated to provide services to that vendor. (I do recognize that some are, due to contractual agreements.) We should cut them off until/unless they recall all those devices, get them removed from service, and solve what is now *their* problem to *our* satisfaction. I strongly suspect that it'll only take a few pointed lessons in order for the message that this conduct is unacceptable to be communicated in a language they understand. I don't like this. In a better world, vendors would be far more responsible, professional, and ethical. But we don't live in that world. We live in one where they will happily dump toxic waste on the Internet as fast as they can shovel it -- as long as it's not their problem. We need to make it their problem. ---rsk
On Wed, Feb 8, 2017 at 10:12 AM, Rich Kulawiec <rsk@gsp.org> wrote:
How? Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Wed, Feb 8, 2017 at 7:22 AM, William Herrin <bill@herrin.us> wrote:
The devices are trivially compromised (just log in with the default root password). So here's a modest proposal: log in as root and brick the device. This will encourage the consumer to seek a solution. When 100k consumers all discover their devices broke at the same time, they'll file a class-action lawsuit against the manufacturer, or at least never buy from them again. Market forces then solve the problem naturally, both for that manufacturer and for others who don't want the same fate. I realize there are drawbacks (including legal implications) to this method (which is why I'm posting from a personal, not work, account). But I challenge anyone to propose another solution that will work as well. Most other proposals I've heard depend on individual ISPs to take action, or governments to regulate devices and hope that foreign manufacturers care, or .... Damian
On Wed, Feb 8, 2017 at 11:30 AM, Damian Menscher <menscher@gmail.com> wrote:
Okay, so within the confines of lawful activity, how? 'Cause I'm guessing that coordinated criminal activity is going to be a community non-starter. At least when it's this unambiguous. ;) Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On Wed, 2017-02-08 at 08:30 -0800, Damian Menscher wrote:
So here's a modest proposal: log in as root and brick the device.
I strongly suspect that when the problem gets bad *enough*, someone will do exactly that. Yes, it is illegal in many places. Since when has the fact that any particular act is illegal been sufficient to deter *everyone*? People still drive while drunk. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) iEYEAREKAAYFAlibzdIACgkQL6j7milTFsH/WgCdEvde+zMvm8lRUyx2ay3EltZT 97kAn3Hl2tjPe2eUqiagDXxlE75OO/Xg =W+Cq -----END PGP SIGNATURE-----
Having spent the last few months systematically scanning ~700k of these hosts, Im thinking the following could be considered: As an ISP, scan your customers netrange, and notify customers with known vulnerable devices. With regards to the current Mirai threat, theres only a handful of devices that are the most critical importance. IE, biggest fraction of the infected host pie. Maybe someday I'll get around to parsing my database and auto-emailing the abuse emails of the affected netranges. That was my intention..... but dayjob got in the way. This breaks down however when you look at the geographic distribution of infected devices. Most are in Asian countries, so there would need to be more cooperation among network operators there. On Wed, Feb 8, 2017 at 6:03 PM, Carl Byington <carl@five-ten-sg.com> wrote:
On Wed, 08 Feb 2017 22:19:01 -0800, clinton mielke said:
Yup! All the mapping Ive done is over port 80. Id have a lot more than I currently have if I was looking at other ports, probably.
Wow. How does this work if more than one IoPT(*) device is in play in the home network, especially from different manufacturers? (*) Internet of Pwned Things.
It probably doesn't account for those situations. In the case of security products, it's also likely that multiple devices are hosting port 80 But it doesn't matter too much. Having this kind of data helps us prioritize what devices have the biggest chunk of the infected pie. On Feb 9, 2017 12:04 PM, <valdis.kletnieks@vt.edu> wrote:
Virgin Media in the UK do this for Mirai-infected or susceptible devices already. What they send out: https://twitter.com/2sec4u/status/825337376692121601 Quite interesting approach. If more consumers were aware of this, they may do something about it.. although.. people are lazy. :(
That being said, I think if other ISPs took virgins lead then we can start getting this population of devices reduced. The hard part is getting overseas ISPs to help with the problem. Most inbound infectious scanning traffic appears to come from China and Vietnam. I need to create some better aggregate statistics. On Feb 10, 2017 5:48 AM, "Marco Slater" <marco@marcoslater.com> wrote:
On Wed, Feb 08, 2017 at 08:30:15AM -0800, Damian Menscher wrote:
No. It's never a good idea to respond to abuse with abuse. Not only is it unethical and probably illegal (IANAL, this is not legal advice) but it won't take more than a day for someone to figure out that this is happening and use some variety of misdirection to cause third parties to target devices that aren't actually part of the problem. ---rsk
On Thu, Feb 9, 2017 at 12:04 PM, Rich Kulawiec <rsk@gsp.org> wrote:
Hi Rich, On that we agree. Vigilantism is a non-starter.
Is there some way an industry association could overcome this? Perhaps have some trivial way to assign each model of IoT device some kind of integer and have the device report the integer instead of its plain text manufacturer and hardware model number? Where the assigned integer is intentionally not published by the industry association though of course trivially determinable by anyone who owns one of the devices. Wouldn't especially impair building a database of vulnerable devices but it would raise the bar for trying to turn the self-reporting in to business intelligence. Particularly if industry association rules forbid retaining a record of device self-reports on pain of whatever. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
On Thu, 09 Feb 2017 14:54:26 -0500, William Herrin said:
Or anybody who knows how to use the internet to look for reports of owners who have issues. All it takes is one smarter than the average bear user posting "I've got a MobyWombat 3000 light bulb, and it keeps sending 1193432542 to some server someplace...."
Wouldn't especially impair building a database of vulnerable devices but it would raise the bar for trying to turn the
If it doesn't *heavily* impair building a database of vulnerable devices, it's not a solution to the problem under discussion.
On February 9, 2017 at 12:04 rsk@gsp.org (Rich Kulawiec) wrote:
Ok but what if you broke in and fixed their security w/o breaking the user experience? Would a vendor, presented with a good demo, sign off on that? If so isn't it just a mandatory patch? -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*
On Tue, Feb 7, 2017 at 3:27 PM, Randy Bush <randy@psg.com> wrote:
Hi Randy, I'd expect a tattler kill switch to take maybe a tenth of that from the anycast notification when the nic comes up to the ISPs response that it is known to be vulnerable and should disconnect. Regards, Bill Herrin -- William Herrin ................ herrin@dirtside.com bill@herrin.us Owner, Dirtside Systems ......... Web: <http://www.dirtside.com/>
I think the fundamental problem here is that these devices aren't good network citizens in the first place. The odds of getting them to add functionality to support a new protocol are even likely than getting them to not have open services externally IMHO. Couldn't a lot of this be caught by proactive vulnerability scanning and working with customers to have an SPI firewall in place, or am I missing something? Historically residential ISP CPE options have been terrible. If you could deliver something closer to user expectations you would likely see much more adoption and less desire to rip and replace. Ideally a cloud-managed device so that the config wouldn't need to be rebuilt in the event of a hardware swap. On Mon, Feb 6, 2017 at 5:31 PM, William Herrin <bill@herrin.us> wrote:
-- Ray Patrick Soucy Senior Cyber Security Engineer Networkmaine, University of Maine System US:IT 207-581-3526
On Tuesday, 7 February, 2017 06:59, Ray Soucy said:
I do not permit "cloud managed" devices on my network unless the "cloud" also belongs to me and is located on my network (in other words, a good old fashioned server on my network run by me). No ISP is permitted to put "cloud" or even remotely configured (by anyone who is not me) devices on my network. Such devices go on THEIR network not MY network. If they malfunction or get hacked, the problem is THEIRS not MINE. Such a policy ensures that I am entirely and exclusively responsible for the good behaviour of the equipment on MY network. If I were to permit devices managed by NOT-ME on MY network, then I would not be responsible. Therefore such filth should stay on NOT-MY network. So the CPE equipment owned, managed and configured by the ISP is on the ISP network, not my network. The demarc is the ethernet connection between the ISP network and MY network. The ISP cannot configure nor touch anything on MY network, nor I on THEIRS. As for "cloud" crap, anything that even mentions the work "cloud" on the box or glossy brochure gets an immediate 10,000,000 point penalty applied to ensure that it is forever off the consideration list. If someone is opposed to this policy and cannot live with it, either a network carrier or ISP, product vendor or whatever, I really do not give a rats butt. I will simply go do business with someone who has more sense.
On Tue, Feb 07, 2017 at 08:58:46AM -0500, Ray Soucy wrote:
Ideally a cloud-managed device so that the config wouldn't need to be rebuilt in the event of a hardware swap.
That opens them to a class breach: instead of one getting compromised they *all* get compromised. Better to save the configuration to cheap local media like a USB stick. Yes, it could get lost, but that doesn't break or compromise the device, and it only affects one device. ---rsk
participants (16)
-
bzs@theworld.com
-
Carl Byington
-
clinton mielke
-
Damian Menscher
-
Ed Lopez
-
joel jaeggli
-
Keith Medcalf
-
Marco Slater
-
Michael Thomas
-
Randy Bush
-
Ray Soucy
-
Rich Kulawiec
-
Richard
-
Tom Beecher
-
valdis.kletnieks@vt.edu
-
William Herrin