They were hosting him for free, and like insurance, I can assure you if you are consistently using a service, and not covering the costs of that service you won't be a client for long. This is the basis for AUP/client contracts and have been going back to the days when we all offered only dialup internet. On Fri, Sep 23, 2016 at 3:01 PM, Mike Hammett <nanog@ics-il.net> wrote:

I believe the article says they were being hosted for free.

----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com

Midwest-IX http://www.midwest-ix.com

----- Original Message -----

From: "Grant Ridder" <shortdudey123@gmail.com> To: nanog@nanog.org Sent: Friday, September 23, 2016 12:58:44 PM Subject: Krebs on Security booted off Akamai network after DDoS attack proves pricey

Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off- akamai-network-after-ddos-attack-proves-pricey/

"Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size."

-Grant

FWIW, we have offered to help. No word so far. We're more than willing to step in front of the cannon pointed his way. ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak <saper@saper.info> wrote:

On Fri, 23 Sep 2016, jim deleskie wrote:

...

They were hosting him for free, and like insurance, I can assure you if you are consistently using a service, and not covering the costs of that service you won't be a client for long. This is the basis for AUP/client contracts and have been going back to the days when we all offered only dialup internet.

Does being a victim of a DDoS constitute a breach of AUP?

Marcin Cieślak

We routinely mitigate L7s. Matthew is also on the record saying we've seen and mitigated similar attacks to this one (based on available information about this attack). ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D On Fri, Sep 23, 2016 at 12:26 PM, Patrick W. Gilmore <patrick@ianai.net> wrote:

Is CloudFlare able to filter Layer 7 these days? I was under the impression CloudFlare was not able to do that.

There have been a lot of rumors about this attack. Some say reflection, others say Layer 7, others say .. other stuff. If it is Layer 7, how are you going to ‘step in front of the cannon’? Would you just pass through all the traffic?

I realize Matthew is always happy for publicity (hell, the whole planet is aware of that). But if your system cannot actually do the required task, I’m not sure your company should give you credit for offering a service the user cannot use.

-- TTFN, patrick

...

On Sep 23, 2016, at 3:16 PM, Justin Paine via NANOG <nanog@nanog.org> wrote:

FWIW, we have offered to help. No word so far. We're more than willing to step in front of the cannon pointed his way.

____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D

On Fri, Sep 23, 2016 at 11:58 AM, Marcin Cieslak <saper@saper.info> wrote:

...

On Fri, 23 Sep 2016, jim deleskie wrote:

...

They were hosting him for free, and like insurance, I can assure you if you are consistently using a service, and not covering the costs of that service you won't be a client for long. This is the basis for AUP/client contracts and have been going back to the days when we all offered only dialup internet.

Does being a victim of a DDoS constitute a breach of AUP?

Marcin Cieślak

On Fri, 23 Sep 2016, Patrick W. Gilmore wrote:

Is CloudFlare able to filter Layer 7 these days? I was under the impression CloudFlare was not able to do that.

There have been a lot of rumors about this attack. Some say reflection, others say Layer 7, others say .. other stuff. If it is Layer 7, how are you going to ÿÿstep in front of the cannonÿÿ? Would you just pass through all the traffic?

Anycast + load balancers + high powered varnish? ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

On Fri, 23 Sep 2016, Christopher Morrow wrote:

On Fri, Sep 23, 2016 at 9:24 PM, Jon Lewis <jlewis@lewis.org> wrote:

...

On Fri, 23 Sep 2016, Patrick W. Gilmore wrote:

Is CloudFlare able to filter Layer 7 these days? I was under the

...

impression CloudFlare was not able to do that.

There have been a lot of rumors about this attack. Some say reflection, others say Layer 7, others say .. other stuff. If it is Layer 7, how are you going to ÿÿstep in front of the cannonÿÿ? Would you just pass through all the traffic?

Anycast + load balancers + high powered varnish?

notionally (because I have been paying zero attention to this) jon's suggesting: 1) setup a crapload of nginx/squid/etc configured tightly for things to be accessed behind them 2) ecmp to them across several layers (assume 32 ecmp at each layer, call it 4 layers get craploads of machines running) 3) change over the dns 4) profit--

eh? If you can eat the PPS, you can spray across enough tcp listeners, you can weed out the chaff and start filtering in the 'application'... perhaps also run a 'low bandwidth' version of the target site...

hey look, we invented prolexic.

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around 50G per site, and at that level, filtering the obvious crap gets much more reasonable. Then, doing the layer 7 scrubbing of the less obvious crap is more easily dealt with than a single site receiving 600G of attack traffic. I haven't actually done this (specifically for DDoS mitigation)...just speculating as to how it might easily be done given sufficient resources. The trouble is, the attackers have virtually unlimited bandwidth, and aren't constrained by having to pay for the bandwidth. ---------------------------------------------------------------------- Jon Lewis, MCP :) | I route | therefore you are _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

...

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :)

People who've tried it say it works fine. Routes don't flap that often.

On Sat, Sep 24, 2016 at 12:28 PM, Bill Woodcock <woody@pch.net> wrote:

...

On Sep 24, 2016, at 7:47 AM, John Levine <johnl@iecc.com> wrote:

...

...

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :)

People who've tried it say it works fine.

It’s worked fine for 28 years, for me.

boy, it'd sure be nice if there were some 'science' and 'measurement' behind such statements. Didn't k-root do some anycast studies ~8-10 years back? -chris (note I'm totally a believer in anycast for tcp in the 'right' circumstances, but often it feels like talking to climate-change-deniers when proffering it as a solution)

On Sat, Sep 24, 2016 at 2:43 PM, Niels Bakker <niels=nanog@bakker.net> wrote:

* morrowc.lists@gmail.com (Christopher Morrow) [Sat 24 Sep 2016, 18:55 CEST]:

...

boy, it'd sure be nice if there were some 'science' and 'measurement' behind such statements. Didn't k-root do some anycast studies ~8-10 years back?

Not k-root but CacheFly 2006: https://www.nanog.org/meetings /nanog37/presentations/matt.levine.pdf

that's not the one I was thinking of, this is: <https://www.nanog.org/meetings/nanog39/presentations/larson.pdf> which references your presentation, nice! and is about J-root, not K-root, but mentions Lorenzo's work on K-root studies... In anycase, both seem to say that 'tcp anycast works fine' (inside some set of parameters). thanks! -chris

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45 On Google now.  ____________ Justin Paine Head of Trust & Safety CloudFlare Inc. PGP: BBAA 6BCE 3305 7FD6 6452 711557B6 0114 DE0B 314D On Sat, Sep 24, 2016 at 2:17 PM -0700, "Brett Watson" <brett@the-watsons.org> wrote:

...

that's not the one I was thinking of, this is:

which references your presentation, nice! and is about J-root, not K-root, but mentions Lorenzo's work on K-root studies... In anycase, both seem to say that 'tcp anycast works fine' (inside some set of parameters).

Right… and we’ve known this since about… ? 1996?

And of course on windows ipconfig /flushdns Still I had to wait for my corporate caching servers to update; I think the TTL on the old A record was an hour. On Sat, Sep 24, 2016 at 9:51 PM, Jared Mauch <jared@puck.nether.net> wrote:

...

On Sep 24, 2016, at 9:28 PM, Justin Paine via NANOG <nanog@nanog.org> wrote:

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45

I recommend running this command (or similar):

rndc flushname krebsonsecurity.com

if you still see 127.0.0.1

- Jared

And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?). https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth <jra@baylink.com> wrote:

----- Original Message -----

...

From: "Jay Farrell via NANOG" <nanog@nanog.org>

...

And of course on windows ipconfig /flushdns

Still I had to wait for my corporate caching servers to update; I think the TTL on the old A record was an hour.

Are big eyeball networks still flooring A record TTLs on resolution?

Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On 09/25/2016 07:32 AM, Jay R. Ashworth wrote:

...

From: "Jay Farrell via NANOG" <nanog@nanog.org>

...

And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/ Well, given how few contributions we've gotten at bcp38.info in the last, what, 4 years, yeah, I guess so...

Yeah, right. I looked at BCP38.info, and there is very little concrete information. I've been slogging through the two RFCs, 2827 and 3794, and find it tough sledding to extract the nuggets to put into my firewall and routing table. One of the more interesting new additions to my systems is this, to the routing tables: ip route add blackhole 0.0.0.0/32 # broadcast (deprecated) ip route add blackhole 0.0.0.0/8 # “this” ip route add blackhole 10.0.0.0/8 # private network ip route add blackhole 100.64.0.0/10 # carrier-grade NAT ip route add blackhole 127.0.0.0/8 # loopback ip route add blackhole 169.254.0.0/16 # link-local ip route add blackhole 172.16.0.0/12 # private network ip route add blackhole 192.0.0.0/24 # IANA special purpose registry ip route add blackhole 192.0.2.0/24 # TEST-NET ip route add blackhole 192.88.99.0/24 # 6to4 anycast relay (optional) ip route add blackhole 192.168.0.0/16 # private network ip route add blackhole 198.18.0.0/15 # inter-network testing ip route add blackhole 198.51.100.0/24 # TEST-NET-2 ip route add blackhole 203.0.113.0/24 # TEST-NET-3 ip route add blackhole 224.0.0.0/4 # Multicast (all the implied routes from interface definitions that are inserted automatically into the routing table by the operating system) ip route add <downstream-subnet> via <gateway> dev <interface> ip route add <downstream-subnet> via <gateway> dev <interface> (Has this been published anywhere before? I haven't found any yet.) Notice I've omitted 255.255.255.255, because I have quite a bit of software that uses this broadcast address, and it needs to get onto my intranets; I just have to be sure that the edge link doesn't let it out to the world. One such program that uses broadcast is Dropbox. This routing table addition forms the nucleus from which I can derive most of the source-address input packet filtering (adding 255.255.255.255), as well as destination-address output filtering with code, instead of having to compile the rules by hand. Then there are other packet filters: * TCP “Lamp test/XMAS” et. al. (mask, set) --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE --tcp-flags FIN,SYN,RST,ACK,URG FIN,SYN,RST,ACK,URG --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG --tcp-flags FIN,ACK FIN --tcp-flags PSH,ACK PSH --tcp-flags URG,ACK URG --tcp-flags SYN,FIN SYN,FIN --tcp-flags SYN,RST SYN,RST --tcp-flags FIN,RST FIN,RST * DNS flood * DNS amplification * NTP flood * NTP amplification * ICMP flood * IGRP flood * protocol/port “small services” connections And I'm just getting started. It helps that I have a few books talking about the design of firewalls that discuss some of these filters. For DNS amplification, I'm using these IPTABLES rules:

# Limit UDP DNS "root NS" queries (must come before generic stream ACCEPT) # 012000010000000000010000020001 -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|012000010000000000010000020001|" --algo bm --from 20 --to 1550 -m recent --set --name dnsrootquery --rsource -A INPUT -i eth0 -p udp -m udp --dport 53 -m string --hex-string "|012000010000000000010000020001|" --algo bm --from 20 --to 1550 -m recent --rcheck --seconds 20 --hitcount 3 --name dnsrootquery --rsource -m comment --comment "UDP DNS ROOT flood" -j DROP

(Another possibility would be to read the DNS hints file for all the root servers, and impose rate-limiting to those IP addresses...but I didn't think of that before and so it's not part of the current DNS server firewall. But the filters above seem to work quite well -- it's been a long time since my ISP upstream's uplink has been nailed up solid.) In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY filtering system -- BSD, Linux, Cisco.

On Sun, 25 Sep 2016 21:19:31 -0700, Hugo Slabbert said:

Linux: From /etc/sysctl.conf:

# Uncomment the next two lines to enable Spoof protection (reverse-path=20 # filter) # Turn on Source Address Verification in all interfaces to # prevent some spoofing attacks net.ipv4.conf.default.rp_filter=1 net.ipv4.conf.all.rp_filter=1

Unfortunately, the net.ipv6 equivalents for those do not yet seem to be a thing on Linux.

See net/ipv6/netfilter/ip6t_rpfilter.c Also, note that a lot of net.ipv4.conf variables also apply to ipv6 (though checking the source tree, this isn't one of them, unless it's via a macro that some quick grepping didn't find...)

On Sun, 25 Sep 2016, Stephen Satchell wrote:

Yeah, right. I looked at BCP38.info, and there is very little concrete information.

Yeah, it's pretty naked. But how-to isn't the usual stumbling block, as has been pointed out in this thread there needs to be the will to spend resources setting it up and thus committing future resources to maintenance.

I've been slogging through the two RFCs, 2827 and 3794, and find it tough sledding to extract the nuggets to put into my firewall and routing table. One of the more interesting new additions to my systems is this, to the routing tables:

A list of martian addresses is useful to avoid sending to or accepting from weird places but it isn't useful for BCP38 purposes, of ensuring a node only uses address(es) assigned to it as the source address in the packets it creates/sends. BGP38 checking is not done by the node itself, though that is not entirely unreasonable. Enforcement is performed by the network as close to the point of the node's attachment as possible -- failures should be discarded or perhaps returned as prohibited and possibly even sampled for use by network staff to work on remediation. In some cases it's very simple and effective to just filter toward your upstream(s), i.e., allow this area's addresses and drop/reject/log the rest.

(Has this been published anywhere before? I haven't found any yet.)

Cymru has lists in various formats and levels of (de)aggregation and detail that you can easily turn into those commands, though there's no martians-only lists for IPv6. You might even use one of the "fullbogon" lists to block at a very detailed level if you have sufficient resources or tools that keep needs light, e.g., ipset.

In short, I have yet to see a "cookbook" for BGP38 filtering, for ANY filtering system -- BSD, Linux, Cisco.

For some it's just uRPF, strict or loose as your needs demand, which their router can already perform, e.g., interface your-template/or/range-of-interfaces/etc ip verify unicast reverse-path or ip verify unicast source reachable-via rx or ip verify unicast source reachable-via any Which for Linux is controlled by the net.ipv4.conf.if.rp_filter sysctl key where "if" can be "default", "all" or a specific interface name and a setting of 1 does strict checking while 2 does loose. Or there's always plain old packet filters, with varying degrees of ease or annoyance, as tightly (per customer applied to incoming packets received on their interface) or simply (leaving the pop) as you please and makes sense. /mark

On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org> wrote:

And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Yeh, bcp38 is not a viable solution. As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle. A solution is aggregating the telemetry of source IP addresses in the botnet and assigning blame and liability to the owners of the IP addresses / host ASN. The networks can then use AUP to shutdown the bot members. As where http://openntpproject.org/ was a proactive approach, Kreb's data can be reactive approach. And since the data is evidence of a crime, the network operators can enforce the AUP. The attack did happen. This ip was involved. Remediation is required.

From there, the host ASN can

On Sun, Sep 25, 2016 at 12:43 AM, Jay R. Ashworth <jra@baylink.com <javascript:;>> wrote:

...

----- Original Message -----

...

From: "Jay Farrell via NANOG" <nanog@nanog.org <javascript:;>>

...

And of course on windows ipconfig /flushdns

Still I had to wait for my corporate caching servers to update; I think the TTL on the old A record was an hour.

Are big eyeball networks still flooring A record TTLs on resolution?

Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com <javascript:;> Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

----- Original Message -----

From: "Ca By" <cb.list6@gmail.com>

On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org> wrote:

...

And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Yeh, bcp38 is not a viable solution.

As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle.

No; things which are not implemented anywhere generally don't move the needle. You're confusing cause and effect here, I think. You give no evidence that *pervasive implementation of 38* would *not* move the needle, and that's where we are right now: we do not have anything that looks like "pervasive implementation". *Ten* people could solve this problem. Tomorrow. The chief engineers of the top 10 US eyeball providers could simply sit down and say "let's go do this thing". And better than 80% of the potential sources would just vanish off the face of the internet. Do I need to go do research, and name these 10 people? :-) Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

You don't need complete adoption to reduce the attacks. If ASes representing 25% of the current spoofed traffic implemented BCP38, then guess what, there's 25% less of an attack. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Ca By" <cb.list6@gmail.com> To: "Jay R. Ashworth" <jra@baylink.com> Cc: "North American Network Operators' Group" <nanog@nanog.org> Sent: Sunday, September 25, 2016 10:13:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On Sunday, September 25, 2016, Jay R. Ashworth <jra@baylink.com> wrote:

----- Original Message -----

...

From: "Ca By" <cb.list6@gmail.com <javascript:;>>

...

On Sunday, September 25, 2016, Jay Farrell via NANOG <nanog@nanog.org <javascript:;>> wrote:

...

And of course Brian Krebs has a thing or two to say, not the least is which to push for BCP38 (good luck with that, right?).

https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/

Yeh, bcp38 is not a viable solution.

As long as their is one spoof capable network on the net, the problem will not be solved. While bcp38 is a true bcp, it is not a solution. It will not, and has not, moved the needle.

No; things which are not implemented anywhere generally don't move the needle.

It is implemented many places in fact.

You're confusing cause and effect here, I think.

I will argue you are confused.

You give no evidence that *pervasive implementation of 38* would *not* move the needle, and that's where we are right now: we do not have anything that looks like "pervasive implementation".

*Ten* people could solve this problem. Tomorrow.

The chief engineers of the top 10 US eyeball providers could simply sit down and say "let's go do this thing". And better than 80% of the potential sources would just vanish off the face of the internet.

Assume every network in the usa implements bcp38. This simply means no spoofs source from usa. Every packet is sent from the usa using a valid origin. Assume also 50% of networks in Europe and Asia and the Southern Hemisphere do bcp38 too. Great. The result is the needle has not moved at all. CC nodes in the non bcp38 locations will send spoofed packets destinations is comcast and att with a source of krebs. Result? Comcast and att cpe responds with crap to krebs. Ddos success despite bcp38 in all of usa.

Do I need to go do research, and name these 10 people? :-)

Cheers, -- jra -- Jay R. Ashworth Baylink jra@baylink.com <javascript:;> Designer The Things I Think RFC 2100 Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

On Sunday, September 25, 2016, John Levine <johnl@iecc.com> wrote:

...

...

Yeh, bcp38 is not a viable solution.

Krebs said this DDoS came from insecure IoT devices, of which there are a kazillion, with the numbers growing every day. Why would they need to spoof IPs? How would BCP38 help?

R's, John

Worth reading to level set https://www.internetsociety.org/sites/default/files/01_5.pdf The attack is triggered by a few spoofs somewhere in the world. It is not feasible to stop this. The attack traffic that blows up to 600gbs is from traceable iot crap , the victim knows who is sending the packers (iot crap) and the access network (comcast, att ...) has the AUP authority to shut it down. One by one. Or automated. Please see https://www.ietf.org/rfc/rfc6561.txt

On Sun 2016-Sep-25 17:01:55 -0400, John R. Levine <johnl@iecc.com> wrote:

...

https://www.internetsociety.org/sites/default/files/01_5.pdf

The attack is triggered by a few spoofs somewhere in the world. It is not feasible to stop this.

That paper is about reflection attacks. From what I've read, this was not a reflection attack. The IoT devices are infected with botware which sends attack traffic directly. Address spoofing is not particularly useful for controlling botnets.

But that's not only remaining use of source address spoofing in direct attacks, no? Even if reflection and amplification are not used, spoofing can still be used for obfuscation.

For example, the Conficker botnet generated pseudo-random domain names where the bots looked for control traffic.

...

Please see https://www.ietf.org/rfc/rfc6561.txt

Uh, yes, we're familiar with that. We even know the people who wrote it. It could use an update for IoT since I get the impression that in many cases the only way for a nontechnical user to fix the infection is to throw the device away.

Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly

-- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com pgp key: B178313E | also on Signal

In message <20160926155649.14061.qmail@ary.lan>, "John Levine" writes:

...

...

That paper is about reflection attacks. From what I've read, this was not a reflection attack. The IoT devices are infected with botware which sends attack traffic directly. Address spoofing is not particularly useful for controlling botnets.

But that's not only remaining use of source address spoofing in direct attacks, no? Even if reflection and amplification are not used, spoofing can still be used for obfuscation.

I agree that it would be nice if more networks did ingress filtering, but if you're expecting a major decrease in evil, you will be disappointed.

At this point it's mostly useful for identifying the guilty or negligent parties afterwards.

R's, John

Actually for ISP's that do it, it can becomes a source of intelligence on where the compromised / misconfigured machines are. A good ISP would be informing their customers that they are seeing anomalous traffic. With IPv6 there is likely to be more of this traffic visible as home NATs wont be masking it. Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

In message <03DC1038-024A-4D9F-AC5B-3E88CDF56246@cable.comcast.com>, "Livingood, Jason" writes:

On 9/26/16, 7:09 PM, "NANOG on behalf of Mark Andrews" <on behalf of marka@isc.org> wrote:

...

A good ISP would be informing their customers that they are seeing anomalous traffic.

Therein lies the problem if the traffic does not look anomalous I suppose. But even if it does look unusual, ISPs would be asking consumers to trash/update/turn off a lot of devices in time like when every home has 10s or 100s of these devices. ISP: Dear customer, looks like one of your light switches is sending spam. Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and

;-)

Jason

Dear customer, we are seeing <xxxx> traffic coming from your network. If you need help isolating the source of the traffic here are a few companies in your city that can help you. <list of companies> This is not a exhaustive list. Support -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews <marka@isc.org> wrote:

Giving them real time access to the anomalous traffic log feed for their residence would also help. They or the specialist they bring in will be able to use that to trace back the problem.

wouldn't this work better as a standard bit of CPE software capability? wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE and kept for ~30mins (just guessing) in a circular buffer be 'good enough' to present a pretty clear UI to the user? ip/mac/vendor sending (webtraffic|email|probes) to destination-name [checkbox] <repeat> select those youd' like to block [clickhere] This really doesn't seem hard, to present in a fairly straight forward manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something similar to this approach... but on the other hand: "At least my ISP isn't snooping on all my traffic"

On 27 Sep 2016, at 6:58, Christopher Morrow wrote:

wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE and kept for ~30mins (just guessing) in a circular buffer be 'good enough' to present a pretty clear UI to the user?

+1 for this capability in CPE. OTOH, it will be of no use whatsoever to the user. Providing the user with access to anomalous traffic feeds won't help, either. Users aren't going to call in some third-party service/support company, either. It call comes down to the network operator, one way or another. There's no separation in the public mind of 'my network' from 'the Internet' that is analogous to the separation between 'the power company' and 'the electrical wiring in my house/apartment' (and even in that space, the conceptual separation often isn't present). ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 11:43, Mark Andrews wrote:

Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different.

Washing machines aren't a utility. Internet is viewed as a utility.

Actually I don't believe that. They do know what machines they have have connected to their home network. Boxes don't magically connect. Every machine was explictly connected.

First of all, not every devices was explicitly connected by the user. Think set-top boxes/DVRs. Secondly, users connect things an then don't think about them, don't remember credentials, had a horrible ordeal (from their perspective) connecting said devices and then promptly forgot how to administer them. Thirdly, expecting users to troubleshoot which of their devices is emanating bad traffic is unrealistic. The only effective consumer remediation efforts we've seen to date have been broadband access ISPs proactively scanning their customer networks and contacting them when exploitable devices and compromised PCs have been found. Although it's a lot of work, that kind of thing can be done for CPE broadband routers; it can't be done for the things sitting behind those devices, which are doing NAT/firewalling. The partial exception is PCs, because everyone thinks of those when they think of 'the Internet'. And the fact that even their lightbulbs are being connected now - i.e., the huge proliferation of connected devices - militates against user troubleshooting, as well. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 12:14, Mark Andrews wrote:

I'm yet to see a set top box, DVR, TV, games console, phone, etc. that didn't require selecting the WiFi SSID or require you to plug in a ethernet cable.

I've 'seen' tens of millions of them, worldwide. You're generalizing your particular connection/personal provisioning model.

As I said, they don't magically connect to the network. Someone did something to permit them to connect.

That someone quite often isn't the end-user. And as noted previously in this thread, even when users themselves do this, they promptly forget how they did it, lose the documentation, etc.

Why do you think people are incapable of calling in someone to help them fix a known issue.

1. Because they demonstrably don't. 2. Because it's not perceived as a 'computer problem' - it's perceived as an 'Internet problem', and the 'Internet technician' = the broadband access operator's help-desk. 3. Going along with the line of reasoning you've expressed, it seems that the user should call a 'lightbulb technician' when his Internet-enabled lightbulb is causing a problem. Do you really think that's realistic? 4. In most cases, the user won't have any idea which connected device is causing the problem. Expecting the user to determine this by trial-and-error is unrealistic; most people don't even understand how to troubleshoot electrical problems by trial-and-error, much less Internet-related problems. You are a self-selected specialist, and understand all these things and have a DIY attitude, because you're an expert in this field. Most people aren't experts in this field. Ask yourself how many people set up and use 2FA for any online service which supports it, on their own initiative (i.e., not having a bank ship them a pre provisioned dongle). The number of people capable of doing this troubleshooting for themselves is roughly equivalent to the number of people who've successfully set up 2FA on their own initiative. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

hi,

From: NANOG <nanog-bounces@nanog.org> on behalf of Mike Hammett <nanog@ics-il.net> Sent: 27 September 2016 16:30 Cc: nanog@nanog.org Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey

You must not support end users.

haha...i read that wrong. I read it as a command, rather than an observation! ;-) alan

In message <ED450BED-5C57-4B90-A8BD-7160015B893A@puck.nether.net>, Jared Mauch writes:

...

On Sep 27, 2016, at 12:43 AM, Mark Andrews <marka@isc.org> wrote:

Why not? You call a washing machine mechanic when the washing machine plays up. This is not conceptually different.

Mark,

Your logic is infallible here, but the equivalencies are not. If I drive on the road and it’s bumpy, I would complain to the road people, but some people will take their car to the shop and says it shakes.

When you are a toll-free call away from a complaint, often this barrier of proof is quite high. I recall something that Vijay said when he was still at AOL, if the customer phones in for support they lost all profit on the customer for the lifetime of the customer.

Given that most people make decisions based on lowest cost (which isn’t always lowest or best due to marketing, promos, etc) the barrier for burden of proof is set such that a carrier must prove to a non-technical user it’s their fault.

This proof is tough, not impossible, but look at your EDNS project, the problems are real and often can’t be easily addressed.

Actually, EDNS shows they can be addressed. Firewall vendors are changing the defaults to allow through all packets that match the test classes. DNS vendors are fixing their products to properly handle packets with EDNS extension. DNS hosters are fixing their deployed firewalls and servers. Soon I'll be asking, my local opposition MP if she can ask why the DNS servers for *.gov.au aren't compliant with the standard after reporting to the DNS operators that they are broken. I suspect she will have fun with having more material to fling around. I'm having to reduce the parallelism of the test runs because the packets are being answered. Fixing EDNS is basically a education issue. Raise the awareness until it becomes something one can't ignore. Go look at the TLD graphs. Almost all the TLD operators have fixed their firewalls / servers. If Microsoft and Go Daddy fix their servers most of the incorrect echoing EDNS options and EDNS flags will disappear. Both have been informed. Microsoft about 2 years ago when we let them know that their servers have issues with EDNS, this included both the servers they ship in Windows and the servers answering DNS queries for Microsoft domains. They where reminded a year ago. Go Daddy was informed very recently (via email). Note that COOKIE is echoed. Also you can't report this to Microsoft using the email address listed below which was designed for reporting issues like this. Microsoft wants you to create a account or use twitter (which also requires an account to be created). You will note the DNS COOKIES are on by default. BIND 9.11.0 will be sending its queries with a DNS COOKIE option present. All your servers need to cope. % dig boimi.gov. @ns1-06.azure-dns.com soa ; <<>> DiG 9.11.0rc2 <<>> boimi.gov. @ns1-06.azure-dns.com soa ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54172 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 2 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ; COOKIE: ddcbdd73de5d5ef8 (echoed) ;; QUESTION SECTION: ;boimi.gov. IN SOA ;; ANSWER SECTION: boimi.gov. 3600 IN SOA ns1-06.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300 ;; ADDITIONAL SECTION: ns1-06.azure-dns.com. 3600 IN A 40.90.4.6 ;; Query time: 141 msec ;; SERVER: 40.90.4.6#53(40.90.4.6) ;; WHEN: Wed Sep 28 07:11:15 EST 2016 ;; MSG SIZE rcvd: 152 % % dig microsoft.com @ns1.msft.net ; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 7450 ;; flags: qr rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ; COOKIE: 5a294c21d4ac66a3 (echoed) ;; QUESTION SECTION: ;microsoft.com. IN A ;; Query time: 269 msec ;; SERVER: 2620:0:30::53#53(2620:0:30::53) ;; WHEN: Wed Sep 28 07:05:34 EST 2016 ;; MSG SIZE rcvd: 54 % dig microsoft.com @ns1.msft.net +nocookie ; <<>> DiG 9.11.0rc2 <<>> microsoft.com @ns1.msft.net +nocookie ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26221 ;; flags: qr aa rd; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;microsoft.com. IN A ;; ANSWER SECTION: microsoft.com. 3600 IN A 23.96.52.53 microsoft.com. 3600 IN A 191.239.213.197 microsoft.com. 3600 IN A 104.40.211.35 microsoft.com. 3600 IN A 104.43.195.251 microsoft.com. 3600 IN A 23.100.122.175 ;; Query time: 425 msec ;; SERVER: 2620:0:30::53#53(2620:0:30::53) ;; WHEN: Wed Sep 28 07:05:39 EST 2016 ;; MSG SIZE rcvd: 122 % Mark

- Jared -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

On 27 Sep 2016, at 12:17, Sam Silvester wrote:

or call their electricity retailer/distributer

This is the problematic case that is, unfortunately, the default. People tend to view anything related to 'the Internet' as a utility, and for consumers and SMBs, they typically have a single provider, just as they typically do for electricity and water. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 22:49, Florian Weimer wrote:

Most people over here have at least two providers of water and Internet (although the second one is perhaps sufficient for brushing your teeth, but certainly not for a shower or a bath).

That's not a common arrangement in much of the world, however. Especially the Internet part. ;> ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 21:48, Brielle Bruns wrote:

You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring.

It's important to keep in mind that in the not-so-distant future, their 'machines' will include every article of clothing they own, every can of soda in their refrigerator, ever major (and many minor) components of their automobiles, every blade in their windowshades, etc. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:

All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea.

Yes, but how do they determine that a given device is vulnerable? ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

It's not just consumers that need to understand this. Manufacturers of Things are right now on a steep learning curve. Consider that thermostat, for just a moment. In The Gold Old Days, before it had a network interface, the manufacturer cared about a handful of things like at what temperature to turn the heat or A/C on maybe with some adjustments for time of day or day or week. And that was it. That is their domain of expertise. Not security. Now the Internet looks like a new shiny object that promises to provide some cool new world capabilities, like letting people adjust the temp while they're away, or using weather forecasts to manage hysteresis effects. And so, the manufacturer initially thinks, we'll add an interface to the product, and a little bit of code, and we're done. Now the manufacturer has stepped outside their domain of expertise, and doesn't have a full understanding of the risks that need to be addressed. We as experts in this domain can help by informing manufacturers of those risks. Eliot On 9/27/16 6:05 PM, Patrick W. Gilmore wrote:

On Sep 27, 2016, at 11:49 AM, Roland Dobbins <rdobbins@arbor.net> wrote:

...

On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:

...

All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea. Yes, but how do they determine that a given device is vulnerable? Easy: Can you ping it? It’s vulnerable.

:-)

Hey, I said we would have to educate them. I did not say how that would happen.

Assuming all devices are vulnerable isn't a bad start. -- Keith Stokes

On Sep 27, 2016, at 11:04 AM, Roland Dobbins <rdobbins@arbor.net> wrote:

...

On 27 Sep 2016, at 22:37, Patrick W. Gilmore wrote:

All the more reason to educate people TODAY on why having vulnerable devices is a Very Bad Idea.

Yes, but how do they determine that a given device is vulnerable?

----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 27 Sep 2016, at 22:46, Brielle Bruns wrote:

I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers.

They can *see* the unruly children, but *choose* to ignore them. That's the difference. Keep in mind, most of the folks on this list are not representative of the average consumer in terms of the skill-sets which are relevant in this problem space. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On 9/27/16 11:18 AM, Brielle Bruns wrote:

On 9/27/16 10:05 AM, Roland Dobbins wrote:

...

...

I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers.

They can *see* the unruly children, but *choose* to ignore them. That's the difference.

I call shenanigans on providers not seeing their unruly users. They have no problems with bandwidth caps, or doing the dirty work of copyright police, both of which require some level of network monitoring per customer.

Or even better example - the providers who monetize customer browsing/shopping habits using third party network device? Or SiteFinder like services with their DNS? Providers have no problems monitoring, intercepting, mangling. I know people here will say, "Well, I'm not like that/don't believe in that!", but we're not talking about technical decisions - but decisions made by people who's job is to squeeze as much profit out of the customers as possible. There is no financial incentive or penalty for preventing/limiting your customers from harming others. If there was, I don't think we'd be having this discussion. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

On Tue, 27 Sep 2016, Brielle Bruns wrote:

I don't see how this is a problem exactly? If people want to buy devices that connect to their home network, they need to be aware of what these devices can do, and it is their responsibility.

I understand that is what you want. What you might like. What we all would like. People taking responsibility for their impact on others. Unfortunately people plug things in, and if they work for them, they don't even think about how what they are doing might affect anyone else. In some cases, they don't even care. They've got soccer games and work and TV shows and kids and family. Who has time to become an expert in Internet security? Google is doing a great job of annoying or alerting customers to potential issues, such as the red lock icon on their email, indicating that the email was sent unencrypted. The user gets worried (oooh, a red lock, that must be bad, I'm going to yell at someone to fix it for me) and the service provider jumps to improve the Internet, ideally. FreeBSD updated their default config so you have to proactively remove email encryption. If we are truly worried about IoT and consumers contributing to the downfall of the Internet, force the consumer router manufacturers and third party firmware folks to implement whatever is necessary to make filters and blocking the default. 90%+ of consumers don't change any settings, beyond the SSID and Wifi Password, and those who do might take the responsibility you want. Get the ISPs to realize that secure-by-default consumer routers that they distribute saves them millions/billions of dollars annually in customer service and security personnel. Secure-by-default routers means cost-savings. Get ISPs to pressure manufacturers to implement measures to protect their own network and the Internet from the non-network-admin consumer. We tech folk need to do this for the Internet citizens who don't know, don't care, or don't have time to mess with it.

If Timmy Numbnuts doesn't understand that plugging in a random device he found at Goodwill to his network could potentially carry liabilities, then he will keep doing it.

Timmy Numbnuts needs to be protected from himself, so when he plugs in that device, it doesn't do any harm to anyone but his own network. He'd have to proactively turn off features or filters on his Router in order to harm others.

I point to the current trend of parents watching and smiling, doing nothing as their kids destroy people's stores and restaurants. ISPs are literally doing the exact same thing when it comes to coddling their customers.

Automation and default configs means customers don't have to do anything, nor think about it. They are protected both FROM harm from the Internet and FROM harming the Internet, at least by default. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman@angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

"who from my experience tend to be the least experienced and network knowledgeable people running a customer network" Also most likely to have built their network from scratch out of pure need (perhaps for themselves) rather than someone cashing in on a trend. No offense meant (though surely someone took it) either way. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Brielle Bruns" <bruns@2mbit.com> To: nanog@nanog.org Sent: Tuesday, September 27, 2016 9:48:24 AM Subject: Re: Krebs on Security booted off Akamai network after DDoS attack proves pricey On 9/26/16 10:05 PM, Roland Dobbins wrote:

+1 for this capability in CPE.

OTOH, it will be of no use whatsoever to the user. Providing the user with access to anomalous traffic feeds won't help, either.

Users aren't going to call in some third-party service/support company, either.

You start cutting off users or putting them into a walled garden until they fix their machines, and they will start caring. This will only work if all providers including cable, DSL and *shudders* WISPs (hate to be blunt, but who from my experience tend to be the least experienced and network knowledgeable people running a customer network) do it so customer's can't just switch networks and 'make the problem go away'. I use escalating price increases and delays in service/repair time on some of my consulting customers who do things I warned them to be more careful about. It takes time, but when $cost starts to become prohibitive, they stop and think. And the ones that never learn... Well, that's more $$$ in my pocket for the effort that I would normally charge otherwise. -- Brielle Bruns The Summit Open Source Development Group http://www.sosdg.org / http://www.ahbl.org

On Sep 26, 2016, at 7:58 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

On Mon, Sep 26, 2016 at 7:49 PM, Mark Andrews <marka@isc.org> wrote:

...

Giving them real time access to the anomalous traffic log feed for their residence would also help. They or the specialist they bring in will be able to use that to trace back the problem.

wouldn't this work better as a standard bit of CPE software capability? wouldn't something as simple as netflow/sflow/ipfix synthesized on the CPE and kept for ~30mins (just guessing) in a circular buffer be 'good enough' to present a pretty clear UI to the user?

ip/mac/vendor sending (webtraffic|email|probes) to destination-name [checkbox] <repeat>

select those youd' like to block [clickhere]

This really doesn't seem hard, to present in a fairly straight forward manner... sure 'all cpe' (or 'a bunch of cpe') have to adopt something similar to this approach... but on the other hand: "At least my ISP isn't snooping on all my traffic"

The UBNT Edgerouter series has this. You can get fancy graphs and application breakdown. Scroll down and check the images: https://help.ubnt.com/hc/en-us/articles/204951104-EdgeMAX-Deep-Packet-Inspec... You can see the hosts that are doing traffic and the destinations. They even have a model that takes a SFP so you can use it as CPE for FTTH. - Jared

Therein lies the problem if the traffic does not look anomalous I suppose. But even if it does look unusual, ISPs would be asking consumers to trash/update/turn off a lot of devices in time – like when every home has 10s or 100s of these devices. ISP: Dear customer, looks like one of your light switches is sending spam. Customer: Which one? I have 25 light switches. And 25 smart bulbs. And 3 smart TVs, and 3 smart thermostats, and 6 cameras, and…

That's why turning them off has to be mandatory if the ISP can't mitigate the traffic in real time. Sorry, but something in your house is attacking strangers. Once you figure out what it is, here's a handy list of links to the ongoing class action suits against the manufacturers. Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly

* Eliot Lear:

As some on this thread know, I've been working with the folks who make light bulbs and switches. They fit a certain class of device that is not general purpose, but rather are specific in nature. For those devices it is possible for the manufacturers to inform the network what the communication pattern of the device is designed to be. This may be extraordinarily broad or quite narrow, depending on the device. Conveniently, the technology for describing much of this dates back to the paleolithic era in the form of access lists. That is what manufacturer usage descriptions are about. (Yep- MUD. There go my marketing credentials). They're slightly abstracted for adaptation to local deployments. There's a draft and we authors are soliciting comments.

What's the end goal here? Charge extra if I'm connecting a TV instead of a light bulb? I'm not convinced that expected traffic profiles are the right answer. We already have that in the server hosting market, and it does constraint the types of services you can run on hosted servers (for the hosting providers who does this). I'm wary of the network putting severe constraints on application architecture, way beyond what is dictated by current technology. NAT more or less killed servers on consumer networks, and this kind of traffic profiling has begun to kill clients on server networks.

The service providers has a strong role to play here, since they drive standards for connectivity. Having some access to residential CPE in particular for this purpose, I believe, is very helpful because by doing so not only can SPs protect others, but can also provide lateral protection within the home. As I wrote upthread, if consumers come to see smart lightbulbs not just as useful, but also as a concern, they may wish their SPs to help them. That's the internalizing of an externality that I see possible, and maybe even probable over time.

Most service providers appear to be struggling to maintain up-to-date software on their CPEs (and their infrastructure), and following recommended configuration practices as they evolve. This does not give me confidence that they could perform device management at consumer scale. Do we know how much the average consumer trusts their ISP? Would they want their ISP to control the digital (and increasingly, physical) doors in their home? My own outlook is rather biased, unfortunately.

* Eliot Lear:

Not my end goal. My end goal is that consumers have a means to limit risk in their home environments, and service providers have a means to deliver that to them.

They already have, with today's technology. It's just not a mass-market business. Consumers either have to educate themselves (which is not that hard), and service providers need to provide actual service, instead charging a fee for access to a computer system. There is little interest in this, however. There's a comparable business case for providing managed PCs to consumers, and I'm not sure if any such companies are still left.

...

I'm not convinced that expected traffic profiles are the right answer. We already have that in the server hosting market, and it does constraint the types of services you can run on hosted servers (for the hosting providers who does this). I'm wary of the network putting severe constraints on application architecture, way beyond what is dictated by current technology. NAT more or less killed servers on consumer networks, and this kind of traffic profiling has begun to kill clients on server networks.

The whole point of MUD is to leave control in the hands of those who have developed and have to support Things. It is not simply for the SP to decide what traffic is ok, or to charge more for it, but to respect the wishes of the developers. That may be sufficient to stop a lot of bad things from happening to a lot of Things.

Nobody respects what developers want, otherwise we wouldn't have any shipping products at all. What I'm trying to say: Cutting corners is more often a non-development decision. If you can ship today without any security, or at some unknowable date in the future, with additional security features whose impact may not matter, things usually head for the earlier shipping date. I used to be frustrated by such decisions, but over the past few years, I've come to realize that most of us have so little data on the effectiveness of security features that mandates for them are essentially arbitrary.

And again, this is the wrong way to look at it. The consumer should always get final say. They're the customer. This is a chance for the manufacturer of the device they're using to explain how the device is supposed to behave on the network.

If we want to make consumers to make informed decisions, they need to learn how things work up to a certain level. And then current technology already works. (Sorry that I'm not inclined to read upon the specs—I do wonder how this an improvement over UPnP.)

I just bought a $20 Lacrosse remote RF temperature sensor hub for home, the GW-1000U. It does the usual IoT things: after you plug it in, it gets a DHCP address and phones home, then you register it using a smartphone on the same LAN, which I'm guessing finds the device via a broadcast and then configures the hub with my Lacrosse account info. All communication is thereafter through the cloud. It set itself up quite conveniently and efficiently, and now will start charging me $12/year for alerts and texts. An acceptable business model. Except the thing is a teaming mass of security vulnerabilities. How much authentication went on in this process? None. I captured the thing's packets in my firewall's onboard sniffer from the get go. All data is exchanged as plaintext on port 80. The protocol is completely undocumented, but I've since discovered that at least one enterprising tinkerer has reverse engineered it so people can bypass the manufacturer's monetization model. The device accepts TCP connections on 22, 80, and 443. Theoretically I can't see why it ever needs ongoing inbound connections, so this seems to be a security concession made by the maker. Also, it appears to support SSL, but uses plaintext. Why? Because it's easier to debug in the early deployments, I'll wager. But the thing has been out for years and they're still not using encryption, even though the device apparently has the ability. As a knowledgable consumer (and security researcher) I'll overcome these shortcomings by putting this device on its own VLAN with extensive firewalling. Still, I can't be sure it won't be malicious, or get exploited through the cloud. And VLANs have their own security weaknesses, despite my using pricey enterprise hardware at home. My point is that if an expert has to expend several hours of highly technical labor to "responsibly" use a $20 IoT sensor, and use enterprise-grade IT gear and methods to gain even a modicum of safety, then what hope do Ma and Pa Kettle have? This is not a consumer education problem, unless we think consumers should also learn thermodynamics in order to drive, the Bernoulli principle in order to be airline passengers, and biochemistry to cook their own food. It's clearly a giant screw-up by manufacturers who could easily spread the cost of best-practice security measures across a large customer base. That they don't shows lack of moral character, and nothing else. -mel beckman

On Oct 9, 2016, at 7:03 AM, John R. Levine <johnl@iecc.com> wrote:

...

On Sun, 9 Oct 2016, Florian Weimer wrote:

If we want to make consumers to make informed decisions, they need to learn how things work up to a certain level. And then current technology already works.

I think it's fair to say that security through consumer education has been a failure every time anyone has tried it. Why do you think this would be any different?

...

There is little interest in this, however. There's a comparable business case for providing managed PCs to consumers, and I'm not sure if any such companies are still left.

There's at least two large ones: Microsoft and Apple. Try installing Windows 10 without letting Microsoft update and reconfigure the software any time they want, any way they want.

Expecting consumers to evaluate the security behavior of their lightbulbs and their refrigerator is absurd. We need to figure out how to have the devices and routers configure themselves so the devices can do what they need to do without doing what we really don't want them to do.

Regards, John Levine, johnl@iecc.com, Primary Perpetrator of "The Internet for Dummies", Please consider the environment before reading this e-mail. https://jl.ly

Stephen, But they don’t, in fact, allow such a console. And I don’t think such a thing is even a good idea on IoT devices, because permitting inbound connections is a pathway to exploitation. As I noted in my post, I’ve put it on its own VLAN, which is better than a DMZ: no inbound access at all, and no access to any other network or devices. I only permit port 80 outbound to the Lacrosse cloud server, and will get notified of any other traffic. But this is a wired device, which made it easier to sequester. If it were WiFi my task would have been much harder, and most IoT devices do seem to be WiFi. -mel

On Oct 9, 2016, at 8:33 AM, Stephen Satchell <list@satchell.net> wrote:

On 10/09/2016 07:31 AM, Mel Beckman wrote:

...

remote RF temperature sensor hub for home, the GW-1000U. ... The device accepts TCP connections on 22, 80, and 443. Theoretically I can't see why it ever needs ongoing inbound connections, so this seems to be a security concession made by the maker. Also, it appears to support SSL, but uses plaintext. Why? Because it's easier to debug in the early deployments, I'll wager. But the thing has been out for years and they're still not using encryption, even though the device apparently has the ability.

I could see one reason, and one reason only: to allow the customer to use a "control panel" with a local computer, smartphone app, or tablet app to set capabilities, options, and preferences. That said, the manufacturer probably thought that the sensor would be shielded from the Internet by a Wireless Access Point with NAT, so that there would be no direct exposure (in theory) to inbound connections from the outside world.

For IPv4, this is barely tolerable. For IPv6, not so much.

As a developer, I can tell you that "easier to debug in the early deployments" means that the later deployments won't be locked down until the manufacturer gets a fine, judgement, or other monetary hit.

Would you put this thing on a DMZ? I thought not... :)

On Sun, 09 Oct 2016 14:31:54 -0000, Mel Beckman said:

I just bought a $20 Lacrosse remote RF temperature sensor hub for home, the GW-1000U. It does the usual IoT things: after you plug it in, it gets a DHCP address and phones home, then you register it using a smartphone on the same LAN, which I'm guessing finds the device via a broadcast and then configures the hub with my Lacrosse account info. All communication is thereafter through the cloud.

That last sentence is sub-optimal. And as you note, things go downhill from there....

On Sun, 09 Oct 2016 18:05:20 -0000, Mel Beckman said:

I don't know why it's "sub optimal" to use the cloud from an isolated network. Can you elaborate?

Why should something out in the cloud have any part of the communication, other than perhaps telling your cellphone the current address of your widget? (And *that* should probably have a standardized protocol/service rather than every vendor rolling their own. Hello, IETF?) And even *that* can be bypassed if you cellphone is able to talk to your home network directly.

The idea behind IoT is that devices collect data, but the power to process that data, and archive it, is in the cloud. -mel beckman

On Oct 9, 2016, at 11:30 AM, "Valdis.Kletnieks@vt.edu" <Valdis.Kletnieks@vt.edu> wrote:

On Sun, 09 Oct 2016 18:05:20 -0000, Mel Beckman said:

...

I don't know why it's "sub optimal" to use the cloud from an isolated network. Can you elaborate?

Why should something out in the cloud have any part of the communication, other than perhaps telling your cellphone the current address of your widget?

(And *that* should probably have a standardized protocol/service rather than every vendor rolling their own. Hello, IETF?)

And even *that* can be bypassed if you cellphone is able to talk to your home network directly.

Elsewhere, for decades, I've bemoaned the fact that keyboards (etc) don't have credit card swipes (perhaps today "and chip readers") so with some care on the part of the software someone could prove they likely have physical access to the card. But it would be very useful in this IoT problem. You power up a new device, it won't enable until you run some web (e.g.) interface. At that point you swipe a card which generates a hash which secures the IoT device from further config until it's presented again. The device can have the usual reset to factory config button for the case of lost cards. It needn't even be an active credit card. It could be an old spent gift card. It could even be a free card that comes right in the box tho that might invite predictability, but maybe a basket of cards to use at the checkout counter "take one you'll need it for setup". The software just has to be able to read the magstripe or chip and use the info to generate a reasonably secure hash which is stored (preferably in the device.) Need to reconfig, open the window, swipe the same card. Hotel safes often use this approach as an alternative to PIN entry. The device doesn't store any info about the card directly, only the hash. And as I said it could be most anything that looks like a credit card and has a readable mag stripe. The user doesn't have to come up with a password and can't use the device until a hash is stored. But, alas, no swipes... -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*

On October 9, 2016 at 20:07 mel@beckman.org (Mel Beckman) wrote:

Barry,

The problem isn't authentication during initial installation, since that can be done using SSL and a web login to the cloud service. The problem is that vendors aren't even using minimal security protections, such as SSL, and then leaving devices open to inbound connections, which is bad even behind a firewall (because viruses typically scan LANs for these vulnerable devices). These are the devices exploited by hackers to become DDoS attack vectors.

It helps solve the bad (including manufacturer's default) password problem which was one of the attack vectors. The proposal only forces this to be used during initial installation and configuration (and any reconfig) arguing that it so lowers the threshold, just swipe a magstripe, there's really no excuse. And eliminates the owner choosing a password for the device, bad or otherwise. But, again, alas no swipe hardware. Big historical error I think but rectifying is feasible. -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*

On October 9, 2016 at 20:24 mel@beckman.org (Mel Beckman) wrote:

You might as well wish for fingerprint readers. It's not going to happen, and thus can't be remedied. But there are already acceptable COTS solutions that need no special hardware. IoT vendors simply have to use them.

Ok, let them go for that, or not. But I well remember proposed spam mitigations back in 2000 being just as forcefully shot down because IT WOULD TAKE A DECADE TO IMPLEMENT THAT!!! That was 16 years ago. It's a dumb sort of thing to waste people's time posting, whether it's attractive or not will stand on its own merits and not someone arguing that in their opinion (based on who knows what) market penetration is insurmountable. And many smart phones do have fingerprint readers and have for a few years now, so do quite a few keyboards. My trackpad on my 4 year old laptop has one (Aspire 8930G.) Your point is several years too late already. -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*

Rich, Thanks for the nice confirmation. My dabbling in internet governance topics has taught me (I guess) that the real challenge is to eschew easy approaches such as shutting off sites as a remedy. The hard work is trying to come up with effective measures which are anything but take downs / blocking -- those should be an absolute last resort at the end of some well-defined and transparent process. Obviously at some extreme point a site has gone so rogue it's just an act of self-defense. But that's the extreme case and still needs a process even if an emergency, short-circuited process. But for sites which imagine themselves to be responsibly managed but fall down on that job sufficiently to merit a response -- my favorite saying in life: EVERYONE forgives themselves! -- there's a need to structure proportionate and effective responses to failings ranging from warnings to actions. And to define clearly what those failings are. For example everyone might not agree that letting 1% of their traffic be spam or otherwise malicious traffic without opposition is even a problem worth exerting effort over. Ok, is it 2%? 0.1%? What is the threshold we can all live with? Or is a percentage just a bad idea and it's the effect which needs to be measured and judged? I suspect a contractual approach might be more productive, as one example. There are other possibilities. -- -Barry Shein Software Tool & Die | bzs@TheWorld.com | http://www.TheWorld.com Purveyors to the Trade | Voice: +1 617-STD-WRLD | 800-THE-WRLD The World: Since 1989 | A Public Information Utility | *oo*

On Saturday, September 24, 2016, Justin Paine via NANOG <nanog@nanog.org> wrote:

DNS Results for query A krebsonsecurity.comAnswer:krebsonsecurity.com 157 IN A 130.211.45.45

On Google now.

Next question. Will google use the information from the telemetry, rumored to be webcams, to defang the bot ? Like post an informative message that the source network is hosting a bad actor (same nat ipv4, /25, ... ). Or , work with the access networks (Comcast, rcs/rds, china telecom) to disconnect and manage the abusers ?

On Sep 24, 2016, at 8:47 AM, John Levine <johnl@iecc.com> wrote:

...

...

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :)

People who've tried it say it works fine. Routes don't flap that often.

It’s not just about route flap. Imagine the following. For any two any cast points A,B, one can draw a simple Venn diagram of two circles with equal radii overlapping to form an OGIVE. Consider that everyone in the nonintersecting portion of circle A will reach server A without issue. Likewise, everyone in the nonintersecting portion of circle B will reach server B without issue. However, for some subset of those within the OGIVE, it’s entirely likely that they will, instead, be broken by ECMP to both A and B. Here’s where it gets tricky… The people running A and B are unlikely to ever know because of the layers between the end user trapped in the OGIVE and the people running A and B. Most likely, the end users will suffer in silence or go to another website for their needs. If this is a small enough fraction of users, then it won’t be statistically noticeable drop in overall traffic and A,B may never know. For those few end-users that may actually attempt to resolve the issue in some meaningful way, most likely they will call their ISP rather than the administrators of A,B and if their ISP does anything, rather than bug A,B, they will most likely simple make routing more deterministic for this site for this end-user. This is the nature of any cast and how any cast problems with TCP get solved (or don’t in most cases). It’s safe to ignore the silent minority that cannot really tell what is happening in most cases, but that doesn’t mean it “works” for any standard I would consider valid. Owen

Assuming all transit providers your packets may traverse on the way to all of your customers is the kind of thing that leads to me quoting Mr. Bush… “I encourage my competitors to try this.” Owen

On Sep 25, 2016, at 6:32 PM, Mark Andrews <marka@isc.org> wrote:

In message <A3E266A0-2E94-4623-9300-2E15FE574BD6@delong.com>, Owen DeLong writes:

...

...

On Sep 24, 2016, at 8:47 AM, John Levine <johnl@iecc.com> wrote:

...

...

Well...by anycast, I meant BGP anycast, spreading the "target" geographically to a dozen or more well connected/peered origins. At that point, your ~600G DDoS might only be around

anycast and tcp? the heck you say! :)

People who've tried it say it works fine. Routes don't flap that often.

It’s not just about route flap.

Imagine the following. For any two any cast points A,B, one can draw a simple Venn diagram of two circles with equal radii overlapping to form an OGIVE.

Consider that everyone in the nonintersecting portion of circle A will reach server A without issue. Likewise, everyone in the nonintersecting portion of circle B will reach server B without issue. However, for some subset of those within the OGIVE, it’s entirely likely that they will, instead, be broken by ECMP to both A and B.

Here’s where it gets tricky…

The people running A and B are unlikely to ever know because of the layers between the end user trapped in the OGIVE and the people running A and B. Most likely, the end users will suffer in silence or go to another website for their needs. If this is a small enough fraction of users, then it won’t be statistically noticeable drop in overall traffic and A,B may never know. For those few end-users that may actually attempt to resolve the issue in some meaningful way, most likely they will call their ISP rather than the administrators of A,B and if their ISP does anything, rather than bug A,B, they will most likely simple make routing more deterministic for this site for this end-user.

This is the nature of any cast and how any cast problems with TCP get solved (or don’t in most cases).

It’s safe to ignore the silent minority that cannot really tell what is happening in most cases, but that doesn’t mean it “works” for any standard I would consider valid.

Owen

Which is why sane operators carefully choose where they deploy ECMP or include hashes of the source and destination addresses into the distribution of traffic over otherwise equal paths.

Mark -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: marka@isc.org

Not at all. I refered to AUP's as a way people remove you from a service when you use more of it then you are paying for. On Fri, Sep 23, 2016 at 3:58 PM, Marcin Cieslak <saper@saper.info> wrote:

On Fri, 23 Sep 2016, jim deleskie wrote:

...

They were hosting him for free, and like insurance, I can assure you if you are consistently using a service, and not covering the costs of that service you won't be a client for long. This is the basis for AUP/client contracts and have been going back to the days when we all offered only dialup internet.

Does being a victim of a DDoS constitute a breach of AUP?

Marcin Cieślak

While we are on topic of DDOS, it looks like it's quite a storm now. According to this WHT post [1], some large server providers were recently attacked, and many are still being attacked with quite a large bandwidth, ie 1Tbps attacks against OVH. [2], [3] Regards, Filip [1] http://www.webhostingtalk.com/showthread.php?t=1599694 [2] https://twitter.com/olesovhcom/status/778019962036314112 [3] https://twitter.com/olesovhcom/status/778830571677978624 On 23.9.2016 20:02, Chris Adams wrote:

Once upon a time, Grant Ridder <shortdudey123@gmail.com> said:

...

Didn't realize Akamai kicked out or disabled customers

Any business is likely to kick out customers that cost them much more than they are being paid (under relevant contract terms of course). Since his blog was being hosted for free, it isn't surprising that Akamai told him they couldn't do that anymore.

It certainly isn't fair to expect Akamai (and their paying customers) to deal with that.

My gigabit pipe was also DDOS attacked the same day my name appeared in Brian’s story. -mel

On Sep 23, 2016, at 11:02 AM, Alex Wacker <alex@alexwacker.com> wrote:

To be fair, he was getting the service for free. I wouldn’t really call that a paying customer. Still not great from a PR standpoint though.

-- Alex Wacker

On September 23, 2016 at 2:00:10 PM, Grant Ridder (shortdudey123@gmail.com) wrote:

Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-aft...

"Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size."

-Grant

On Fri, 23 Sep 2016, Mike wrote:

On 09/23/2016 11:30 AM, Seth Mattinen wrote:

...

On 9/23/16 10:58, Grant Ridder wrote:

...

Didn't realize Akamai kicked out or disabled customers http://www.zdnet.com/article/krebs-on-security-booted-off-akamai-network-aft...

"Security blog Krebs on Security has been taken offline by host Akamai Technologies following a DDoS attack which reached 665 Gbps in size."

So ultimately the DDoS was successful, just in a different way.

~Seth

More technical information about the characteristics of these attacks would be very interesting such as the ultimate sources of the attack traffic (compromised home pc's?), the nature of the traffic (dns / ssdp amplification?), whether it was spoofed source (BCP38-adverse), and whether the recent takedown the vDOS was really complete or if it's likely someone else gained control of the C&C servers that controlled it's assets?

At least for the OVH case there is a bit of info: https://twitter.com/olesovhcom/status/779297257199964160 "This botnet with 145607 cameras/dvr (1-30Mbps per IP) is able to send

1.5Tbps DDoS. Type: tcp/ack, tcp/ack+psh, tcp/syn."

c'ya sven-haegar -- Three may keep a secret, if two of them are dead. - Ben F.

On Sep 23, 2016, at 5:24 PM, Hugo Slabbert <hugo@slabnet.com> wrote:

Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet...

my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ‘clean’ traffic back to origin sites. - Jared

A similar GRE attack was used against the Olympics: "Once the Olympics got under way, LizardStresser along with a few other botnets ramped up their attack against organizations affiliated with the Olympics. The DDoS campaign launched attack traffic using the lesser-known IP protocol Generic Routing Encapsulation (GRE).” -mel On Sep 23, 2016, at 2:39 PM, Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>> wrote: On Fri 2016-Sep-23 17:29:59 -0400, Jared Mauch <jared@puck.nether.net<mailto:jared@puck.nether.net>> wrote: On Sep 23, 2016, at 5:24 PM, Hugo Slabbert <hugo@slabnet.com<mailto:hugo@slabnet.com>> wrote: Please tell me why I can't spoof source IPs on a stateless protocol like GRE. If he specifically meant you can't spoof a source, hit a reflector, and gain amplification, sure, but I see zero reason why GRE can't have spoofed source IPs. It bothered me sufficiently that I wrote up some spit-balling ideas about reflecting GRE using double encapsulation[2]. Very rough and untested, but apparently I got a bee in my bonnet... my guess is the GRE traffic was harder to filter because many providers use GRE to deliver ‘clean’ traffic back to origin sites. But those tunnels generally wouldn't terminate on addresses within the protected block. If I'm protecting 192.0.22.0/24 for you, things would get confused if my GRE tunnel dest is somewhere in 192.0.22.0/24 because I would have a route for 192.0.22.0/24 across the GRE tunnel (either static or more conventionally learned via BGP). I'd bank it on either a provider touchdown or another unprotected block, otherwise I'd have a recursive routing issue where the tunnel destination would get routed through the GRE tunnel. I've been there with bouncy-bouncy tunnels on service turn-up when that was flubbed. Alternatively, I could pin /32s for the tunnel destinations and still learn the shorter protected block, but even so I should then know which source (my) and dest (customer's) IPs to exclude explicitly from the GRE filtering. If the attackers were hitting the GRE tunnel destination and spoofing the tunnel source that would make things harder, but that's starting to get into rather intimate knowledge of the scrubber's and customer's setup. I could still probably filter on e.g. TTLs or drop GRE further up to the northern edge on input rather than output, but agreed that is starting to get trickier... - Jared -- Hugo Slabbert | email, xmpp/jabber: hugo@slabnet.com<mailto:hugo@slabnet.com> pgp key: B178313E | also on Signal