Not just run the updates -- all private keys should be changed too, on the assumption that they've been compromised already. THAT is going to be the crappy part of this. - Pete On 4/8/2014 1:13 AM, David Hubbard wrote:

RHEL and CentOS both have patches out as of a couple hours ago, so run those updates! CentOS' mirrors do not all have it yet, so if you are updating, make sure you get the 1.0.1e-16.el6_5.7 version and not older.

David

-----Original Message----- From: Paul Ferguson [mailto:fergdawgster@mykolab.com] Sent: Tuesday, April 08, 2014 1:07 AM To: NANOG Subject: Fwd: Serious bug in ubiquitous OpenSSL library: "Heartbleed"

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg

Begin forwarded message:

...

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerability -revealed-7000028166/

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf 3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e =aAzE -----END PGP SIGNATURE-----

Hi, I was wondering why most of my secure services didn't show up as vulnerable... ----- It do not seems to affect those services that require a valid user certificate. aka, in apache 2.2 SSLVerifyClient Require SSLVerifyDepth 1 (up to 10) I couldn't find a way to use the HB before satisfying the verify. I might be wrong. ----- Alain Hebert ahebert@pubnix.net PubNIX Inc. 50 boul. St-Charles P.O. Box 26770 Beaconsfield, Quebec H9W 6G7 Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443 On 04/08/14 08:18, David Hubbard wrote:

Don't forget to restart every daemon that was using the old library as well, or just reboot.

-----Original Message----- From: Peter Kristolaitis [mailto:alter3d@alter3d.ca] Sent: Tuesday, April 08, 2014 1:19 AM To: nanog@nanog.org Subject: Re: Serious bug in ubiquitous OpenSSL library: "Heartbleed"

Not just run the updates -- all private keys should be changed too, on the assumption that they've been compromised already. THAT is going to be the crappy part of this.

- Pete

On 4/8/2014 1:13 AM, David Hubbard wrote:

...

RHEL and CentOS both have patches out as of a couple hours ago, so run

...

those updates! CentOS' mirrors do not all have it yet, so if you are updating, make sure you get the 1.0.1e-16.el6_5.7 version and not older.

David

-----Original Message----- From: Paul Ferguson [mailto:fergdawgster@mykolab.com] Sent: Tuesday, April 08, 2014 1:07 AM To: NANOG Subject: Fwd: Serious bug in ubiquitous OpenSSL library: "Heartbleed"

I'm really surprised no one has mentioned this here yet...

FYI,

- ferg

Begin forwarded message:

...

...

...

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerabilit y -revealed-7000028166/

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

...

Randy Bush <randy@psg.com> writes:

you might like (thanks smb, or was it sra)

openssl s_client -connect google\.com:443 -tlsextdebug 2>&1| grep 'server extension "heartbeat" (id=15)' || echo safe

protip: you have to run this from a device that actually is running 1.0.x, i.e. supports the heartbeat extension. your desktop mac (running 0.9.8y if you're running mavericks and haven't stomped on it via ports; homebrew is a keg only install) WILL NOT SUFFICE and will just sit there quietly until the http server times out (60 seconds in my case) and then echo "safe" even when you're not. -r

The updated CentOS openssl binaries haven't patched the underlying bug, but they have disabled the heartbeat functionality. By doing so, they've disabled the attack vector. Once upstream releases a fix, they will re-enable the heartbeat function with the working patch. And yes, don't forget to restart any linked services after updating. -richard On Tue, Apr 8, 2014 at 9:03 AM, Michael Thomas <mike@mtcc.com> wrote:

Just as a data point, I checked the servers I run and it's a good thing I didn't reflexively update them first. On Centos 6.0, the default openssl is 1.0.0 which supposedly doesn't have the vulnerability, but the ones queued up for update do. I assume that redhat will get the patched version soon but be careful!

Mike

On 04/07/2014 10:06 PM, Paul Ferguson wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg

Begin forwarded message:

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in

...

ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerability- revealed-7000028166/

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf 3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e =aAzE -----END PGP SIGNATURE-----

Lots of tools available. I'm with ferg, surprised more haven't been mentioned here. Tools to check for the bug: • on your own box: https://github.com/musalbas/heartbleed-masstest/blob/master/ssltest.py • online: http://filippo.io/Heartbleed/ (use carefully as they might log what you check) • online: http://possible.lv/tools/hb/ • offline: https://github.com/tdussa/heartbleed-masstest <--- Tobias Dussa, also Takes a CSV file with host names for input and ports as parameter • offline: http://s3.jspenguin.org/ssltest.py • offline: https://github.com/titanous/heartbleeder List of vulnerable Linux distributions: <http://www.circl.lu/pub/tr-21/>. Anyone have any more? -- TTFN, patrick On Apr 08, 2014, at 12:11 , Jonathan Lassoff <jof@thejof.com> wrote:

For testing, I've had good luck with https://github.com/titanous/heartbleeder and https://gist.github.com/takeshixx/10107280

Both are mostly platform-independent, so they should be able to work even if you don't have a modern OpenSSL to test with.

Cheers and good luck (you're going to need it), jof

On Tue, Apr 8, 2014 at 5:03 PM, Michael Thomas <mike@mtcc.com> wrote:

...

Just as a data point, I checked the servers I run and it's a good thing I didn't reflexively update them first. On Centos 6.0, the default openssl is 1.0.0 which supposedly doesn't have the vulnerability, but the ones queued up for update do. I assume that redhat will get the patched version soon but be careful!

Mike

On 04/07/2014 10:06 PM, Paul Ferguson wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg

Begin forwarded message:

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in

...

ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerability- revealed-7000028166/

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf 3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e =aAzE -----END PGP SIGNATURE-----

On 04/08/2014 10:16 AM, Patrick W. Gilmore wrote:

Lots of tools available. I'm with ferg, surprised more haven't been mentioned here.

Tools to check for the bug: • on your own box: https://github.com/musalbas/heartbleed-masstest/blob/master/ssltest.py • online: http://filippo.io/Heartbleed/ (use carefully as they might log what you check) • online: http://possible.lv/tools/hb/ • offline: https://github.com/tdussa/heartbleed-masstest <--- Tobias Dussa, also Takes a CSV file with host names for input and ports as parameter • offline: http://s3.jspenguin.org/ssltest.py • offline: https://github.com/titanous/heartbleeder

List of vulnerable Linux distributions: <http://www.circl.lu/pub/tr-21/>.

Anyone have any more?

Thanks for the expanded list, I had some of these already. I'm not comfortable in letting some online code that I can't see test my site though. --John

Me <jschiel@flowtools.net> writes:

Thanks for the expanded list, I had some of these already. I'm not comfortable in letting some online code that I can't see test my site though.

If that's true, you might want to consider immediately disconnecting your systems from the Internet and never re-connecting them. After all, theres a lot of online unseen code testing your site already whether you like it or not. -r

Here's the only way to keep a system safe from Internet hackers: http://goo.gl/ZvGrXw [google images] -j

On 04/08/2014 10:28 PM, Matt Palmer wrote:

On Wed, Apr 09, 2014 at 12:18:00AM -0500, jamie rishaw wrote:

...

Here's the only way to keep a system safe from Internet hackers:

http://goo.gl/ZvGrXw [google images]

/me is disappointed that wasn't a pair of scissors

... or a backhoe

On 2014-04-08 21:57, bmanning wrote:

On Tue, Apr 08, 2014 at 11:46:31PM -0400, Rob Seastrom wrote:

...

If that's true, you might want to consider immediately disconnecting your systems from the Internet and never re-connecting them. After all, theres a lot of online unseen code testing your site already whether you like it or not.

Diodes....

Is it wrong that I read that like this? http://jima.us/201404/diodes.jpg (Sorry.) Jima

On Apr 09, 2014, at 11:26 , Me <jschiel@flowtools.net> wrote:

On 04/08/2014 09:46 PM, Rob Seastrom wrote:

...

If that's true, you might want to consider immediately disconnecting your systems from the Internet and never re-connecting them. After all, theres a lot of online unseen code testing your site already whether you like it or not.

-r

Sending someone to a site with obscure TLDs of .io or .lv doesn't help in these situations. This is a perfect opportunity for someone to set up a drive by site to drop malware on someone's computer.

I'm not saying these sites did that but in order to see the code, someone would have to visit the site first. I personally would use wget instead of a browser for sites like these and did so in this situation.

And yes, your point is not lost on me, there are tons of sites that have obfuscated code and malware running on them, I know that.

In the list of tools were several sites with code you could download, review, and run locally on your machine to test against the bug. However, I trust some of the sites listed. My new favorite is <https://sslanalyzer.comodoca.com/>, since it takes ports other than 443 and gives back a lot of info. -- TTFN, patrick

On 04/09/2014 09:39 AM, Niels Bakker wrote:

* jschiel@flowtools.net (Me) [Wed 09 Apr 2014, 17:26 CEST]:

...

Sending someone to a site with obscure TLDs of .io or .lv doesn't help in these situations. This is a perfect opportunity for someone to set up a drive by site to drop malware on someone's computer.

Yes, because obviously .com registrations are limited to good people only. *eyeroll*

-- Niels.

Guess you didn't read my last sentence, I know the .coms and .orgs, and such don't all belong to the good folks. *sigh*. --John

On 04/09/2014 09:59 AM, Niels Bakker wrote:

Then why single out the .io and .lv's? Maybe you missed the trend (by now a few years old) to get domains in those and similar ccTLD's for startups? Why even try to portray them as less trusted, as you plainly did in the quoted paragraph?

-- Niels.

No, I didn't miss it, it's been a long time in coming. I happened to point .io and .lv because those are the ones that were shared on list and are not common, at least not to me. My concern mostly is with those that disregard what the link name is and just click blindly because it's a link into the issue that is affecting them that day. Instead of picking on a specific tld, I should have more clearly stated I'd rather folks do this type of checking with code that can run locally. This way we can validate the code once and run as many times as we want. Relying on a web page doesn't always work because that site may be overloaded or the site owner hits some limit and the page is not available so you have to go validate the code from yet another site. I did have some versions of the code that was shared and I thanked the OP for the other versions. --John

1.0.1 was not deployed until RHEL 6.5. RedHat released patches for RHEL last night, and CentOS followed suit a few minutes later. -----Original Message----- From: Michael Thomas [mailto:mike@mtcc.com] Sent: Tuesday, April 08, 2014 12:03 PM To: nanog@nanog.org Subject: Re: Fwd: Serious bug in ubiquitous OpenSSL library: "Heartbleed" Just as a data point, I checked the servers I run and it's a good thing I didn't reflexively update them first. On Centos 6.0, the default openssl is 1.0.0 which supposedly doesn't have the vulnerability, but the ones queued up for update do. I assume that redhat will get the patched version soon but be careful! Mike On 04/07/2014 10:06 PM, Paul Ferguson wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg

Begin forwarded message:

...

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerabilit y-revealed-7000028166/

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf 3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e =aAzE -----END PGP SIGNATURE-----

You can still potentially access all the same information since it all goes through the load balancer. Interesting bits of info are things like Cookie: headers being sent by clients and sitting in a buffer. Try one of the testing tools mentioned and see if you can see any info from other clients. It's almost like having remote tcpdump on the web server - you can copy down the in-memory process image. -Laszlo On Apr 8, 2014, at 7:12 PM, "Frank Bulk" <frnkblk@iname.com> wrote:

If we would front our HTTPS services with a (OpenSSL vulnerable) load-balancer that does the SSL work and we just use HTTP to the service, will that mitigate information loss that's possible with this exploit? Or will the OpenSSL code on the load-balancer also store or "cache" content?

Frank

-----Original Message----- From: Paul Ferguson [mailto:fergdawgster@mykolab.com] Sent: Tuesday, April 08, 2014 12:07 AM To: NANOG Subject: Fwd: Serious bug in ubiquitous OpenSSL library: "Heartbleed"

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256

I'm really surprised no one has mentioned this here yet...

FYI,

- - ferg

Begin forwarded message:

...

From: Rich Kulawiec <rsk@gsp.org> Subject: Serious bug in ubiquitous OpenSSL library: "Heartbleed" Date: April 7, 2014 at 9:27:40 PM EDT

This reaches across many versions of Linux and BSD and, I'd presume, into some versions of operating systems based on them. OpenSSL is used in web servers, mail servers, VPNs, and many other places.

Writeup: Heartbleed: Serious OpenSSL zero day vulnerability revealed

http://www.zdnet.com/heartbleed-serious-openssl-zero-day-vulnerability-revea led-7000028166/

...

Technical details: Heartbleed Bug http://heartbleed.com/

OpenSSL versions affected (from link just above): OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable OpenSSL 1.0.1g is NOT vulnerable (released today, April 7, 2014) OpenSSL 1.0.0 branch is NOT vulnerable OpenSSL 0.9.8 branch is NOT vulnerable

- -- Paul Ferguson VP Threat Intelligence, IID PGP Public Key ID: 0x54DC85B2 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iF4EAREIAAYFAlNDg9gACgkQKJasdVTchbIrAAD9HzKaElH1Tk0oIomAOoSOvfJf 3Dvt4QB54os4/yewQQ8A/0dhFZ/YuEdA81dkNfR9KIf1ZF72CyslSPxPvkDcTz5e =aAzE -----END PGP SIGNATURE-----