FFR you can use this to verify the site itself is good or not: https://www.sslshopper.com/ssl-checker.html (there are others, this is just what I have bookmarked) Josh Luthman Office: 937-552-2340 Direct: 937-552-2343 1100 Wayne St Suite 1337 Troy, OH 45373 On Fri, Mar 27, 2015 at 11:35 AM, Mike <mike-nanog@tiedyenetworks.com> wrote:

I'd like to thank everyone for their kind responses. One person who responded off list and bothered to look at the returned certificates pointed out, and correctly it seems, that my original setup was missing an intermediate certificate. The site was returning 'valid ssl' and all browsers got the green lock and offsite ssl tests came back ok, but apparently the missing intermediate means it would have had to have been fetched and that was the part that was failing at the customer site. Once I put the intermediate certificate in there, the customer site was able to access https without fail. I have not had an opportunity yet to examine in detail the config of the meraki router there but it's either a routing problem or a DPI problem. If I get an answer I'll post again with my results.

Thanks all.

Mike-

I believe the SSLLabs Analyzer should have pointed out an "Extra Download" in the cert chain. That was the hint that there was an intermediate cert that a client would have to go find on it's own because it wasn't included with your server cert. https://community.qualys.com/thread/12831 On 3/27/2015 12:34 PM, Mike wrote:

On 03/27/2015 08:43 AM, Josh Luthman wrote:

...

FFR you can use this to verify the site itself is good or not:

https://www.sslshopper.com/ssl-checker.html (there are others, this is just what I have bookmarked)

Thanks. Previously while diagnosing this however, I used some others similar and they all were saying I was ok. For example, https://www.ssllabs.com/ssltest/analyze.html and one other I forget now. I am surprised this problem was not being pointed out.

Glad you figured that out. I've used three SSL evaluation websites to help me with intermediate certificate issues: https://www.ssllabs.com/ssltest/analyze.html (will show the names and details of the certs, missing or not https://www.wormly.com/test_ssl (quick SSL tester, will point out if intermediate certificate is missing) https://www.digicert.com/help/ (will show a green chain link between certs when they're all there *and* in order) Frank -----Original Message----- From: NANOG [mailto:nanog-bounces@nanog.org] On Behalf Of Mike Sent: Friday, March 27, 2015 10:36 AM Cc: nanog@nanog.org Subject: FIXED - Re: Broken SSL cert caused by router? I'd like to thank everyone for their kind responses. One person who responded off list and bothered to look at the returned certificates pointed out, and correctly it seems, that my original setup was missing an intermediate certificate. The site was returning 'valid ssl' and all browsers got the green lock and offsite ssl tests came back ok, but apparently the missing intermediate means it would have had to have been fetched and that was the part that was failing at the customer site. Once I put the intermediate certificate in there, the customer site was able to access https without fail. I have not had an opportunity yet to examine in detail the config of the meraki router there but it's either a routing problem or a DPI problem. If I get an answer I'll post again with my results. Thanks all. Mike-

On 3/28/15 9:05 AM, Mike wrote:

I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of these either caught the problem either. Although I still do not fully understand the dependencies involved, it seems that if my server was failing to supply the full certificate chain, and the browser was compensating for it by (attempting?) to load the missing certificate from elsewhere, and this Meraki router was somehow able to confound that process, that would be an issue worthy of exploring more. I certainly don't blame these ssl check sites but clearly theres more checks needed.

The Qualsys site (https://www.ssllabs.com/ssltest/analyze.html) will report whether or not the server supplied the intermediate cert. But I agree with you that the other tools should make a bigger deal about it if the server doesn't supply it. FWIW, it's been the CW to do this for some time now, as there are systems like the one you've run into that were designed before intermediate certs were commonplace, and don't know how to handle them. I've also experienced situations where an enterprise purchases a DV certificate to be used on an offline system, and while that system has access to the "root" CA certs, it cannot retrieve the intermediate cert. Having the end system supply the intermediate cert as well solves this issue. The method of supplying the intermediate cert is simple, just append the intermediate certificate to the end of the file with your server certificate (the .crt file). Any reasonably modern software will handle that transparently, and provide the intermediate cert along with the server cert when doing its business. hope this helps, Doug -- I am conducting an experiment in the efficacy of PGP/MIME signatures. This message should be signed. If it is not, or the signature does not validate, please let me know how you received this message (direct, or to a list) and the mail software you use. Thanks!

On 03/28/2015 01:50 PM, Matt Palmer wrote:

On Sat, Mar 28, 2015 at 09:05:38AM -0700, Mike wrote:

...

...

Glad you figured that out.

I've used three SSL evaluation websites to help me with intermediate certificate issues: https://www.ssllabs.com/ssltest/analyze.html (will show the names and details of the certs, missing or not https://www.wormly.com/test_ssl (quick SSL tester, will point out if intermediate certificate is missing) https://www.digicert.com/help/ (will show a green chain link between certs when they're all there *and* in order) I went back to Frank's list and did some additional testing. I have a different server which was set up the same way as the previous one discussed, and I thought I would use the above tools and see if my problem would have been identified by any of them. I am sorry to report, no, none of

On 03/27/2015 10:34 AM, Frank Bulk wrote: these either caught the problem either. Are you able to share the URL of the misconfigured site? It would be interesting to examine exactly what's going on.

- Matt

SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt I have actually fixed it. What was going on seems to be this - I have a new godaddy certificate for *.mydomain.com, and that is what I installed. However, the certificate chain I supplied was missing some intermediate godaddy certificate. Originally, it appeared I was missing 'gdig2.crt', and once installed, that fixed some clients including the ones behind the meraki router. But then there were also some older clients this did not fix (a macos 10.4 something for example). So I went back and installed gd_bundle-g2-g1.crt in it's place, and that seems to have finally done it. I apologize for the diminishing lack of operational content. It just seems that these ssl tests should be tightened up and perhaps some additional tools deployed out there to help us less knowledgeable folks 'get it right'. Mike-

SSLCertificateChainFile /etc/ssl/certs/gd_bundle-g2-g1.crt

I have actually fixed it.

Yeah, that's always it. Back in the good aulde days all of the SSL certs one might buy were signed directly by the CA, but now more often than not there are intermediate certs, and a valid cert needs to be accompanied by all of the intermediate certs between it and the CA. What makes debugging hard is that browsers try to be helpful. If a server doesn't provide the intermediate certs, but the browser happens to have them in its cache from some other site, well, close enough and the SSL works. But if some other browser doesn't happen to have them, you lose. So if your SSL is flaky, check those intermediate certs first. R's, John

It might be filtering the CRL or OCSP verification for the SSL certificate. For GoDaddy I think this would be: http://crl.godaddy.com/ http://ocsp.godaddy.com/ http://certificates.godaddy.com/ We ran into this when OS X changed how it handles SSL a few years back, our captive portal was presenting a splash page in place of Thawte OCSP and crashing the SSL keychain process. The work-around was either to respond with a TCP RST for these requests or to allow them through. On Thu, Mar 26, 2015 at 11:57 PM, Lewis,Mitchell T. <ml-nanog@techcompute.net> wrote:

Meraki Access Points are interesting devices.

I have found they cause issues with Linux firewalls if the merakis are not configured "correctly".

Meraki Access Points do content inspections which I have found can cause produce symptoms similar to yours, although I have not experienced what you are describing. Since the MX64W is both an Access Point & security gateway, it has some additional content inspection/intelligence for it's security appliance role on top of the functions it performs as an access point, the same functions which are found in Meraki standalone access points as well.

I am not sure what the specifics are as I do not use Meraki security appliances but it is worth checking. I have found with Meraki that items in the control panel/dashboard are not always labeled the best so I have found it is usually worth putting in a ticket with them and/or a call to them to see what they think (1-888-490-0918).

Mitchell T. Lewis Mlewis@Techcompute.Net : www.linkedin.com/in/mlewiscc Mobile: (203)816-0371 PGP Fingerprint: 79F2A12BAC77827581C734212AFA805732A1394E Public PGP Key

A computer will do what you tell it to do, but that may be much different from what you had in mind. ~Joseph Weizenbaum

----- Original Message -----

From: "Mike" <mike-nanog@tiedyenetworks.com> To: nanog@nanog.org Sent: Thursday, March 26, 2015 6:38:55 PM Subject: Broken SSL cert caused by router?

Hi,

I have a very odd problem.

We've recently gotten a 'real' ssl certificate from godaddy to cover our domain (*.domain.com) and have installed it in several places where needed for email (imap/starttls and etc) and web. This works great, seems ok according to various online TLS certificate checkers, and I get the green lock when testing using my own browsers and such.

I have a customer however that uses our web mail system now secured with ssl. I myself and many others use it and get the green lock. But, whenever any station at the customer tries using it, they get a broken lock and 'your connection is not private'. The actual error displayed below is 'cert_authority_invalid' and it's "Go Daddy Secure Certificate Authority - G2". And it gets worse - whenever I go to the location and use my own laptop, the very one that 'works' when at my office, I ALSO get the error. AND EVEN WORSE - when I connect to my cell phone provided hotspot, the error goes away!

As weird as this all sounds, I got it nailed down to one device - they have a Cisco/Meraki MX64W as their internet gateway - and when I remove that device from the chain and go 'straight' out to the internet, suddenly, the certificate problem goes away entirely.

How is this possible? Can anyone comment on these devices and tell me what might be going on here?

Mike-

-- Ray Patrick Soucy Network Engineer University of Maine System T: 207-561-3526 F: 207-561-3531 MaineREN, Maine's Research and Education Network www.maineren.net