I question whether there is *any* high volume hoster out there that has a reputation for successfully addressing abuse issues coming from their customer base, and cuts off services... By high volume hoster I define it as companies where anybody with a credit card can buy a $2 to $15/month VPS/VM in a fully automated process. OVH just happens to be one of the largest and probably ranks in the top 10 worldwide by number of hypervisors and VPS. I doubt whether any of their 30-40 competitors that are smaller than them do much better, considering the ratio of clued and attentive staff to VMs. On Tue, Feb 27, 2018 at 12:47 PM, Ca By <cb.list6@gmail.com> wrote:

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from- port-11211/

Also, policer all UDP all the time... UDP is unsafe at any speed.

On Tue, Feb 27, 2018 at 12:28 PM Barry Greene <bgreene@senki.org> wrote:

...

Hello Fellow NANOGer,

If you have not already seen it, experiences it, or read about it, working to head off another reflection DOS vector. This time it is memcached on port 11211 UDP & TCP. There are active exploits using these ports. Reflection attacks and the memcached is not new. We know how reflection attacks work (send a spoofed packet to a device and have it reflected back (yes please deploy source address validation and BCP 38).

Operators are asked to review their networks and consider updating their Exploitable Port Filters (Infrastructure ACLs) to track or block UDP/TCP port 11211 for all ingress and egress traffic. If you do not know about iACLs or Explorable port filters, you can use this white paper details and examples from peers on Exploitable Port Filters: http://www.senki.org/operators-security-toolkit/ filtering-exploitable-ports-and-minimizing-risk-to-and- from-your-customers/

Enterprises are also asked to update their iACLs, Exploitable Port Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress and egress traffic.

Deploying these filters will help protect your network, your organization, your customers, and the Internet.

Ping me 1:1 if you have questions.

Sincerely,

-- Barry Raveendran Greene Security Geek helping with OPSEC Trust Mobile: +1 408 218 4669 E-mail: bgreene@senki.org

---------------------------- Resources on memcached Exploit (to evaluate your risk):

More information about this attack vector can be found at the following:

• JPCERT – memcached のアクセス制御に関する注意喚起 (JPCERT-AT-2018-0009) http://www.jpcert.or.jp/at/2018/at180009.html • Qrator Labs: The memcached amplification attacks reaching 500 Gbps

https://medium.com/@qratorlabs/the-memcached- amplification-attack-reaching-500-gbps-b439a7b83c98 • Arbor Networks: memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations

https://www.arbornetworks.com/blog/asert/memcached- reflection-amplification-description-ddos-attack- mitigation-recommendations/ • Cloudflare: Memcrashed – Major amplification attacks from UDP port 11211

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from- port-11211/ • Link11: New High-Volume Vector: Memcached Reflection Amplification Attacks

https://www.link11.com/en/blog/new-high-volume-vector- memcached-reflection-amplification-attacks/ • Blackhat Talk: The New Page of Injections Book: Memcached Injections by Ivan Novikov

https://www.blackhat.com/docs/us-14/materials/us-14-Novikov- The-New-Page-Of-Injections-Book-Memcached-Injections-WP.pdf • Memcache Exploit http://niiconsulting.com/checkmate/2013/05/memcache-exploit/

On 2018-02-27, Ca By <cb.list6@gmail.com> sent:

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed.

Hi, DigitalOcean here. We've taken steps to mitigate this attack on our network. Also, we've only seen udp/11211 being a problem. I'd be interested to hear of anyone seeing tcp/11211 attacks. -- Chip Marshall <chip@2bithacker.net> http://2bithacker.net/

On 28 Feb 2018, at 5:26, Ca By wrote:

Just udp.

This Arbor Threat Summary discusses the TCP issue, as well, FWIW: <https://www.arbornetworks.com/blog/asert/memcached-reflection-amplification-description-ddos-attack-mitigation-recommendations/> 'It should also be noted that memcached priming queries can also be directed towards TCP/11211 on abusable memcached servers. TCP is not currently considered a high-risk memcached reflection/amplification transport as TCP queries cannot be reliably spoofed.' We also recommend implementing situationally-appropriate network access policies at the IDC edge which disallow unwanted UDP/11211 as well as TCP/11211 from reaching abusable memcached deployments. ----------------------------------- Roland Dobbins <rdobbins@arbor.net>

On Tue, Feb 27, 2018 at 09:52:54PM +0000, Chip Marshall wrote:

On 2018-02-27, Ca By <cb.list6@gmail.com> sent:

...

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed.

Hi, DigitalOcean here. We've taken steps to mitigate this attack on our network.

NTT too has deployed rate limiters on all external facing interfaces on the GIN backbone - for UDP/11211 traffic - to dampen the negative impact of open memcached instances on peers and customers. The toxic combination of 'one spoofed packet can yield multiple reponse packets' and 'one small packet can yield a very big response' makes the memcached UDP protocol a fine example of double trouble with potential for severe operational impact. Kind regards, Job

Question: Since we cannot count on everyone to follow BCP 38 or investigate their abuse@, I was thinking about the feasibility of using filtering to prevent spoofing from peers’ networks. With the exception of a few edge cases, would it be possible to filter inbound traffic allowing only packets with source addresses matching the peer’s BGP announcement? Theoretically it should be a two way street to any address I can receive from, thus if I can’t send to it, I shouldn't be receiving from it. I realize this is not currently a feature of any router (to my knowledge), but could it be implemented into some NOSs fairly easily and jerry-rigged into others for the time being. This would allow us to peer with OVH et al, and not worry as much. Furthermore, whereas BCP 38 is reliant upon everybody, this could significantly reduce amplification attacks with even just a handful of adopters. —Todd

On Feb 28, 2018, at 6:52 PM, Job Snijders <job@ntt.net> wrote:

On Tue, Feb 27, 2018 at 09:52:54PM +0000, Chip Marshall wrote:

...

On 2018-02-27, Ca By <cb.list6@gmail.com> sent:

...

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed.

Hi, DigitalOcean here. We've taken steps to mitigate this attack on our network.

NTT too has deployed rate limiters on all external facing interfaces on the GIN backbone - for UDP/11211 traffic - to dampen the negative impact of open memcached instances on peers and customers.

The toxic combination of 'one spoofed packet can yield multiple reponse packets' and 'one small packet can yield a very big response' makes the memcached UDP protocol a fine example of double trouble with potential for severe operational impact.

Kind regards,

Job

Question: Since we cannot count on everyone to follow BCP 38 or investigate their abuse@, I was thinking about the feasibility of using filtering to prevent spoofing from peers’ networks.

With the exception of a few edge cases, would it be possible to filter inbound traffic allowing only packets with source addresses matching the peer’s BGP announcement? Theoretically it should be a two way street to any address I can receive from, thus if I can’t send to it, I shouldn't be receiving from it. I realize this is not currently a feature of any router (to my knowledge), but could it be implemented into some NOSs fairly easily and jerry-rigged into others for the time being. you can build ACLs from IRR objects  just like you can build prefix

On 3/1/18 10:57 AM, Todd Crane wrote: lists. for customers this is just as feasible as controlling what routes you accept from them. unlike URPF strict  this will not cause an outage every time a prefix is withdrawn but traffic continues to flow (which is a normal and healthy part of doing maintenance). on the other hand it means your prefix lists will have to be rather up to date and your peers will likely have to be very up to date with their customers. since failures for inclusion will cause black holes. you can't do this on on and MLPE exchange where you export routes unless you want to cause an outage everytime a new peer is added. there is also the problem of the number of cam slots these IP acls chew up. bgpq3 -P AS-FOO

This would allow us to peer with OVH et al, and not worry as much. Furthermore, whereas BCP 38 is reliant upon everybody, this could significantly reduce amplification attacks with even just a handful of adopters. The source addresses of attack traffic associated with for example memcached (and in fact virtually all reflection attacks) are not spoofed.

—Todd

...

On Feb 28, 2018, at 6:52 PM, Job Snijders <job@ntt.net> wrote:

...

On 2018-02-27, Ca By <cb.list6@gmail.com> sent:

...

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed. Hi, DigitalOcean here. We've taken steps to mitigate this attack on our network. NTT too has deployed rate limiters on all external facing interfaces on

On Tue, Feb 27, 2018 at 09:52:54PM +0000, Chip Marshall wrote: the GIN backbone - for UDP/11211 traffic - to dampen the negative impact of open memcached instances on peers and customers.

The toxic combination of 'one spoofed packet can yield multiple reponse packets' and 'one small packet can yield a very big response' makes the memcached UDP protocol a fine example of double trouble with potential for severe operational impact.

Kind regards,

Job

Le 2018-03-02 22:07, Barry Raveendran Greene a écrit :

Hi Todd,

What you are describing is uRPF VRF mode. This was phase 3 of the uRPF work. Russ White and I worked on it while at Cisco.

Given that you are setting up prefix filters with your peers, you can add to the peering agreement that you will only accept packets whose source addresses matches the prefixes sent.

You then take that BGP session, feed that into a VRF on the interface, and run uRPF against that VRF. If a source address does not match, drop.

If the BGP session adds more routes, those automatically update the VRF "white list" for the uRPF.

It was build to scale. Not sure where it is at in the code or the hardware. Ask Cisco.

Barry

PS - So yes, you can do uRPF on your peering sessions. It was coded and deployed back in 2006.

On Mar 1, 2018, at 13:57, Todd Crane <todd@toddcrane.com> wrote:

Question: Since we cannot count on everyone to follow BCP 38 or investigate their abuse@, I was thinking about the feasibility of using filtering to prevent spoofing from peers' networks.

With the exception of a few edge cases, would it be possible to filter inbound traffic allowing only packets with source addresses matching the peer's BGP announcement? Theoretically it should be a two way street to any address I can receive from, thus if I can't send to it, I shouldn't be receiving from it. I realize this is not currently a feature of any router (to my knowledge), but could it be implemented into some NOSs fairly easily and jerry-rigged into others for the time being.

This would allow us to peer with OVH et al, and not worry as much. Furthermore, whereas BCP 38 is reliant upon everybody, this could significantly reduce amplification attacks with even just a handful of adopters.

--Todd

On Feb 28, 2018, at 6:52 PM, Job Snijders <job@ntt.net> wrote:

On Tue, Feb 27, 2018 at 09:52:54PM +0000, Chip Marshall wrote: On 2018-02-27, Ca By <cb.list6@gmail.com> sent: Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed. Hi, DigitalOcean here. We've taken steps to mitigate this attack on our network.

NTT too has deployed rate limiters on all external facing interfaces on the GIN backbone - for UDP/11211 traffic - to dampen the negative impact of open memcached instances on peers and customers. The toxic combination of 'one spoofed packet can yield multiple reponse packets' and 'one small packet can yield a very big response' makes the memcached UDP protocol a fine example of double trouble with potential for severe operational impact. Kind regards, Job Hope one day the 3rd mode of uRPF will be something else than a plan ... uRPF is not very usefull when multi homed. And as far as I know, multi homed networks are increasing as fast as PNI development ;) -- FABIEN VINCENT _@beufanet_

This is just stupid. OVH is one of the largest server providers in the world - of course they will be at the top of that list. What exactly should they do, according to you? Why should people de-peer them? Regards, Filip Hruska

On 28 Feb 2018 at 1:13 am, <Dan Hollis> wrote:

OVH does not suprise me in the least. Maybe this is finally what it will take to get people to de-peer them. -Dan On Tue, 27 Feb 2018, Ca By wrote: > Please do take a look at the cloudflare blog specifically as they name and > shame OVH and Digital Ocean for being the primary sources of mega crap > traffic > > https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port... > > Also, policer all UDP all the time... UDP is unsafe at any speed. > > > On Tue, Feb 27, 2018 at 12:28 PM Barry Greene wrote: > >> Hello Fellow NANOGer, >> >> If you have not already seen it, experiences it, or read about it, working >> to head off another reflection DOS vector. This time it is memcached on >> port 11211 UDP & TCP. There are active exploits using these ports. >> Reflection attacks and the memcached is not new. We know how reflection >> attacks work (send a spoofed packet to a device and have it reflected back >> (yes please deploy source address validation and BCP 38). >> >> Operators are asked to review their networks and consider updating their >> Exploitable Port Filters (Infrastructure ACLs) to track or block UDP/TCP >> port 11211 for all ingress and egress traffic. If you do not know about >> iACLs or Explorable port filters, you can use this white paper details and >> examples from peers on Exploitable Port Filters: >> http://www.senki.org/operators-security-toolkit/filtering-exploitable-ports-... >> >> Enterprises are also asked to update their iACLs, Exploitable Port >> Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress >> and egress traffic. >> >> Deploying these filters will help protect your network, your organization, >> your customers, and the Internet. >> >> Ping me 1:1 if you have questions. >> >> Sincerely, >> >> -- >> Barry Raveendran Greene >> Security Geek helping with OPSEC Trust >> Mobile: +1 408 218 4669 >> E-mail: bgreene@senki.org >> >> ---------------------------- >> Resources on memcached Exploit (to evaluate your risk): >> >> More information about this attack vector can be found at the following: >> >> • JPCERT – memcached のアクセス制御に関する注意喚起 (JPCERT-AT-2018-0009) >> http://www.jpcert.or.jp/at/2018/at180009.html >> • Qrator Labs: The memcached amplification attacks reaching 500 >> Gbps >> >> https://medium.com/@qratorlabs/the-memcached-amplification-attack-reaching-5... >> • Arbor Networks: memcached Reflection/Amplification Description >> and DDoS Attack Mitigation Recommendations >> >> https://www.arbornetworks.com/blog/asert/memcached-reflection-amplification-... >> • Cloudflare: Memcrashed – Major amplification attacks from UDP >> port 11211 >> >> https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port... >> • Link11: New High-Volume Vector: Memcached Reflection >> Amplification Attacks >> >> https://www.link11.com/en/blog/new-high-volume-vector-memcached-reflection-a... >> • Blackhat Talk: The New Page of Injections Book: Memcached >> Injections by Ivan Novikov >> >> https://www.blackhat.com/docs/us-14/materials/us-14-Novikov-The-New-Page-Of-... >> • Memcache Exploit >> http://niiconsulting.com/checkmate/2013/05/memcache-exploit/ >> >

On Tue, Feb 27, 2018 at 4:29 PM Filip Hruska <fhr@fhrnet.eu> wrote:

This is just stupid.

OVH is one of the largest server providers in the world - of course they will be at the top of that list. What exactly should they do, according to you?

They should have rough norms enforced on their traffic behavior , especially around udp. If they do 1tbs of udp, they should police to 3tbs or something similar. Why should people de-peer them?

Abuse. But abuse happens, failure to fix chronic is a reason to depeer... not one off. Personally, i do not peer with ovh because i need my upstream to rtbh their traffic.

Regards, Filip Hruska

On 28 Feb 2018 at 1:13 am, <Dan Hollis <goemon@sasami.anime.net>> wrote:

OVH does not suprise me in the least.

Maybe this is finally what it will take to get people to de-peer them.

-Dan

On Tue, 27 Feb 2018, Ca By wrote:

...

Please do take a look at the cloudflare blog specifically as they name and shame OVH and Digital Ocean for being the primary sources of mega crap traffic

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port...

Also, policer all UDP all the time... UDP is unsafe at any speed.

On Tue, Feb 27, 2018 at 12:28 PM Barry Greene wrote:

...

Hello Fellow NANOGer,

If you have not already seen it, experiences it, or read about it, working to head off another reflection DOS vector. This time it is memcached on port 11211 UDP & TCP. There are active exploits using these ports. Reflection attacks and the memcached is not new. We know how reflection attacks work (send a spoofed packet to a device and have it reflected back (yes please deploy source address validation and BCP 38).

Operators are asked to review their networks and consider updating their Exploitable Port Filters (Infrastructure ACLs) to track or block UDP/TCP port 11211 for all ingress and egress traffic. If you do not know about iACLs or Explorable port filters, you can use this white paper details and examples from peers on Exploitable Port Filters: http://www.senki.org/operators-security-toolkit/filtering-exploitable-ports-...

...

...

Enterprises are also asked to update their iACLs, Exploitable Port Filters, and Firewalls to track or block UDP/TCP port 11211 for all ingress and egress traffic.

Deploying these filters will help protect your network, your organization, your customers, and the Internet.

Ping me 1:1 if you have questions.

Sincerely,

-- Barry Raveendran Greene Security Geek helping with OPSEC Trust Mobile: +1 408 218 4669 E-mail: bgreene@senki.org

---------------------------- Resources on memcached Exploit (to evaluate your risk):

More information about this attack vector can be found at the following:

• JPCERT – memcached のアクセス制御に関する注意喚起 (JPCERT-AT-2018-0009) http://www.jpcert.or.jp/at/2018/at180009.html • Qrator Labs: The memcached amplification attacks reaching 500 Gbps

https://medium.com/@qratorlabs/the-memcached-amplification-attack-reaching-5... • Arbor Networks: memcached Reflection/Amplification Description and DDoS Attack Mitigation Recommendations

https://www.arbornetworks.com/blog/asert/memcached-reflection-amplification-... • Cloudflare: Memcrashed – Major amplification attacks from UDP port 11211

https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port... • Link11: New High-Volume Vector: Memcached Reflection Amplification Attacks

https://www.link11.com/en/blog/new-high-volume-vector-memcached-reflection-a... • Blackhat Talk: The New Page of Injections Book: Memcached Injections by Ivan Novikov

https://www.blackhat.com/docs/us-14/materials/us-14-Novikov-The-New-Page-Of-... • Memcache Exploit http://niiconsulting.com/checkmate/2013/05/memcache-exploit/

On Wed, Feb 28, 2018 at 12:29:54AM +0000, Filip Hruska wrote:

OVH is one of the largest server providers in the world - of course they will be at the top of that list.

Of course not. The larger an operation, the greater its responsibility to the rest of the Internet -- because the more damage it can and will do if it's not professionally operated. Moreover, the larger the operation the easier abuse control is and the less tolerance any of us should have for failure. I don't want to hear any clueless whining from ignorant newbies about how-oh-so-terribly-hard-it-is-with-a-billion-users. If you don't know how run it, or you're too lazy, cheap, and stupid to run it, you should never have built it: you're simply not good enough. ---rsk

On Tue, Feb 27, 2018 at 04:13:23PM -0800, Dan Hollis wrote:

OVH does not suprise me in the least.

Maybe this is finally what it will take to get people to de-peer them.

Let's hope so. There are two, and only two possibilities. 1. They know what's going on in their operation. In that case, they're fully aware of all the spam, phishing, attacks, abuse, etc., and are choosing to actively support and encourage them because it's profitable. 2. They don't know what's going on in their operation. In that case, they're amazingly ignorant and stunningly incompetent -- because we all know what's going on in it and we're not even there. But in either case, the operational impact on us is exactly the same and so is the solution. ---rsk

I want to add one software vendor, who is major contributor to ddos attacks. Mikrotik till now shipping their quite popular routers, with wide open DNS recursor, that don't have even mechanism for ACL in it. Significant part of DNS amplification attacks are such Mikrotik recursors. They don't care till now. On 2018-02-28 14:31, Job Snijders wrote:

Dear all,

Before the group takes on the pitchforks and torches and travels down to the hosting providers' headquarters - let's take a step back and look at the root of this issue: the memcached software has failed both the Internet community and its own memcached users.

It is INSANE that memcached is/was[1] shipping with default settings that make the daemon listen and respond on UDP on INADDR_ANY. Did nobody take notes during the protocol wars where we were fodder for all the CHARGEN & NTP ordnance?

The memcached software shipped with a crazy default that required no authentication - allowing everyone to interact with the daemon. This is an incredibly risky proposition for memcached users from a confidentiality perspective; and on top of that the amplification factor is up to 15,000x. WHAT?!

And this isn't even new information, open key/value stores have been a security research topic for a number of years, these folks reported that in the 2015/2016 time frame they observed more than 100,000 open memcached instances: https://aperture-labs.org/pdf/safeconf16.pdf

Vendors need to ensure that a default installation of their software does not pose an immediate liability to the user itself and those around them. No software is deployed in a vacuum.

A great example of how to approach things is the behavior of the PowerDNS DNS recursor: this recursor - out of the box - binds to only 127.0.0.1, and blocks queries from RFC 1918 space. An operator has to consciously perform multiple steps to make it into the danger zone. This is how things should be.

Kind regards,

Job

[1]: https://github.com/memcached/memcached/commit/dbb7a8af90054bf4ef51f5814ef7ce...

ps. promiscuous defaults are bad, mmkay? Ask your BGP vendor for RFC 8212 support today! :-)

They ship it by default firewalled from the Internet. ----- Mike Hammett Intelligent Computing Solutions Midwest Internet Exchange The Brothers WISP ----- Original Message ----- From: "Denys Fedoryshchenko" <denys@visp.net.lb> To: nanog@nanog.org Sent: Wednesday, February 28, 2018 6:42:37 AM Subject: Re: New Active Exploit: memcached on port 11211 UDP & TCP being exploited for reflection attacks I want to add one software vendor, who is major contributor to ddos attacks. Mikrotik till now shipping their quite popular routers, with wide open DNS recursor, that don't have even mechanism for ACL in it. Significant part of DNS amplification attacks are such Mikrotik recursors. They don't care till now. On 2018-02-28 14:31, Job Snijders wrote:

Dear all,

Before the group takes on the pitchforks and torches and travels down to the hosting providers' headquarters - let's take a step back and look at the root of this issue: the memcached software has failed both the Internet community and its own memcached users.

It is INSANE that memcached is/was[1] shipping with default settings that make the daemon listen and respond on UDP on INADDR_ANY. Did nobody take notes during the protocol wars where we were fodder for all the CHARGEN & NTP ordnance?

The memcached software shipped with a crazy default that required no authentication - allowing everyone to interact with the daemon. This is an incredibly risky proposition for memcached users from a confidentiality perspective; and on top of that the amplification factor is up to 15,000x. WHAT?!

And this isn't even new information, open key/value stores have been a security research topic for a number of years, these folks reported that in the 2015/2016 time frame they observed more than 100,000 open memcached instances: https://aperture-labs.org/pdf/safeconf16.pdf

Vendors need to ensure that a default installation of their software does not pose an immediate liability to the user itself and those around them. No software is deployed in a vacuum.

A great example of how to approach things is the behavior of the PowerDNS DNS recursor: this recursor - out of the box - binds to only 127.0.0.1, and blocks queries from RFC 1918 space. An operator has to consciously perform multiple steps to make it into the danger zone. This is how things should be.

Kind regards,

Job

[1]: https://github.com/memcached/memcached/commit/dbb7a8af90054bf4ef51f5814ef7ce...

ps. promiscuous defaults are bad, mmkay? Ask your BGP vendor for RFC 8212 support today! :-)

I don’t agree that making RFC-1918 limitations a default in any daemon makes any sense whatsoever. First, there are plenty of LANs out there that don’t use RFC-1918. Second, RFC-1918 doesn’t apply to IPv6 at all, and (fortunately) hardly anyone uses ULA (the IPv6 analogue to RFC-1918). I do agree that listening on all addresses might not be the best default, but building assumptions about RFC-1918 into anything (other than the assumption that they’re generally a pretty bogus way to do IP) makes little sense to me. Owen

On Mar 1, 2018, at 10:32 AM, Eric Kuhnke <eric.kuhnke@gmail.com> wrote:

On the other side: VM/VPS providers have a template based image that they use for every type and subtype of operating system it's possible to auto-provision. For example Ubuntu Server Xenial AMD64 or Debian Jessie or Stretch AMD64.

It's important that VM/VPS providers don't push fresh images that have anything vulnerable, abusable or exploitable enabled by default. Not all VM/VPS providers do this. Standard sane configuration choices apply such as bind9 not being a recursive resolver by default.

If individual Debian, CentOS, Ubuntu or whatever other distro packages for memcached or any other daemon have "listen on all interfaces = yes" or "listen on non-RFC1918 IP ranges = yes" turned on in their respective configurations, that would be an issue to take up with the package maintainers for the daemons.

On Wed, Feb 28, 2018 at 4:31 AM, Job Snijders <job@ntt.net> wrote:

...

Dear all,

Before the group takes on the pitchforks and torches and travels down to the hosting providers' headquarters - let's take a step back and look at the root of this issue: the memcached software has failed both the Internet community and its own memcached users.

It is INSANE that memcached is/was[1] shipping with default settings that make the daemon listen and respond on UDP on INADDR_ANY. Did nobody take notes during the protocol wars where we were fodder for all the CHARGEN & NTP ordnance?

The memcached software shipped with a crazy default that required no authentication - allowing everyone to interact with the daemon. This is an incredibly risky proposition for memcached users from a confidentiality perspective; and on top of that the amplification factor is up to 15,000x. WHAT?!

And this isn't even new information, open key/value stores have been a security research topic for a number of years, these folks reported that in the 2015/2016 time frame they observed more than 100,000 open memcached instances: https://aperture-labs.org/pdf/safeconf16.pdf

Vendors need to ensure that a default installation of their software does not pose an immediate liability to the user itself and those around them. No software is deployed in a vacuum.

A great example of how to approach things is the behavior of the PowerDNS DNS recursor: this recursor - out of the box - binds to only 127.0.0.1, and blocks queries from RFC 1918 space. An operator has to consciously perform multiple steps to make it into the danger zone. This is how things should be.

Kind regards,

Job

[1]: https://github.com/memcached/memcached/commit/ dbb7a8af90054bf4ef51f5814ef7ceb17d83d974

ps. promiscuous defaults are bad, mmkay? Ask your BGP vendor for RFC 8212 support today! :-)

this is sort of why openbsd listens only on 127.0.0.1/::1 by default, right? it's the only sane choice for 'fresh out of the box' network daemons: "Yes, it's running, yes I can healthcheck it locally to prove it's running"

amidst all the hysterical pontification, i am having trouble finding any release which has, by default, a port 11211 listener on any interface. randy

On Thu, Mar 1, 2018 at 5:50 PM, Christopher Morrow <morrowc.lists@gmail.com> wrote:

pre install of memcache on a (debianXXX)

$ cat /etc/debian_version 9.3 (cut/paste fail before click-submit)

Abort. morrowc@build:~$ netstat -anA inet | grep LIST tcp 0 0 192.110.255.61:53 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5433 0.0.0.0:* LISTEN

run: apt-get install memcached

now: morrowc@build:~$ netstat -anA inet | grep LIST tcp 0 0 192.110.255.61:53 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:5433 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN

fargh.

On Thu, Mar 1, 2018 at 5:38 PM, Randy Bush <randy@psg.com> wrote:

...

...

this is sort of why openbsd listens only on 127.0.0.1/::1 by default, right? it's the only sane choice for 'fresh out of the box' network daemons: "Yes, it's running, yes I can healthcheck it locally to prove it's running"

amidst all the hysterical pontification, i am having trouble finding any release which has, by default, a port 11211 listener on any interface.

randy

The defaults for Zimbra seem to be to listen everywhere all the time. amidst all the hysterical pontification, i am having trouble finding any release which has, by default, a port 11211 listener on any interface.

sorry, i should have said "any operating system release" yes, you can install memcached yes, you can install some j random container which has memcached yes, you can shoot yourself in the foot; welcome to the internet my point was merely that the hysteria and grandstanding can cost a lot of ops a bunch of time. and folk should be aware that normal, simple, vanilla environments will not be a source of reflection. of course, they might be a target :) randy

hyperbole. sad and embarrassing to say, but it’s just another damned day of the internet security rolling disaster. there will be more. there will be worse. and screaming wolf will only make folk inured (excuse the american idiom). randy

On 03/01/2018 02:55 PM, Royce Williams wrote:

pstream, until two days ago, the default was to listen on all interfaces.

https://github.com/memcached/memcached/wiki/ReleaseNotes156

The package maintainers were (thankfully) injecting additional sanity.

Yes, they did, in commit dbb7a8af. Here is the commit comment:

disable UDP port by default

As reported, UDP amplification attacks have started to use insecure internet-exposed memcached instances. UDP used to be a lot more popular as a transport for memcached many years ago, but I'm not aware of many recent users.

Ten years ago, the TCP connection overhead from many clients was relatively high (dozens or hundreds per client server), but these days many clients are batched, or user fewer processes, or simply anre't worried about it.

While changing the default to listen on localhost only would also help, the true culprit is UDP. There are many more use cases for using memcached over the network than there are for using the UDP protocol.

--------------------------------- memcached.c --------------------------------- index 88a5f2e..7178666 100644

But then you look at the changes in that commit: what makes this a less-than-ideal change is that they didn't modify the default configuration file to include "-U 0". By defaulting their settings.udpport to zero in the C code, they stop-punch the astonishment factor. By not changing the distribution sysconfig file, though, they open a pitfall for those people who use UDP. The problem? They could have put a warning in the default file so that people who add OPTIONS="U 11211" would be told to firewall that UDP port from the public internet at large. (Then there is the case of people deploying memcached in the cloud, which would incur additional difficulties. But that's another argument...)

Are you insane. ISPs should never use RFC 1918 addresses for stuff that talks to their customers. They have no way of knowing which addresses the customers are using. Traffic from RFC 1918 addresses should be dropped by any sane border router which all routers connecting to a ISP are. -- Mark Andrews

On 2 Mar 2018, at 22:49, Bjørn Mork <bjorn@mork.no> wrote:

Owen DeLong <owen@delong.com> writes:

...

I don’t agree that making RFC-1918 limitations a default in any daemon makes any sense whatsoever.

+1

One of the more annoying anti-features I know of in this regard is the dnsmasq rebind "protection". It claims to protect web browsers on the LAN against DNS rebind attacks. But the implementation does not consider which adresses are used on the LAN at all. It simply blocks any A record pointing to an RFC1918 address, making a few bogus assumptions: - IPv4 LAN addresses are selected from RFC1918 - RFC1918 addresses are never used on the WAN side of a CPE - Noone use IPv6 on their LAN

It's hard to know how many users have been bitten by the first one. You'd have to depend on this rebind "protection" in the first place, and that would be.... stupid.

But the second assumption regularily bites end users when their ISP provides some ISP internal service using RFC1918 addresses. Which of course is fine.

The anti-feature has been enabled by default in OpenWrt for a long time, ref https://wiki.openwrt.org/doc/uci/dhcp#all_options , which means that there is a large user base having this enabled without knowing.

...

First, there are plenty of LANs out there that don’t use RFC-1918.

Second, RFC-1918 doesn’t apply to IPv6 at all,

Could you try to explain that to the OpenWrt guys? Thanks

Bjørn

The ones I know do so on private VLANs (or ATM circuits on DSL) so anyway unrelated to any client’s address space. Also, french triple play ISPs use RFC1918 space for IPTV but again isolated of any customer network so doesn’t really matter.

On 2 Mar 2018, at 22:18, K. Scott Helms <kscotthelms@gmail.com> wrote:

I won't comment on the sanity of doing so, but _many_ service providers use EMTAs, ATAs, and other voice devices over RFC1918 space back to their core.