Tag Archives: ssl

Puppet CRL Time Errors

Puppet is much loved for it’s clear meaningful messages when something goes wrong, made even more delightful when you combine it with the lovely error messages thrown out by OpenSSL.

Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server
certificate B: certificate verify failed: [CRL is not yet valid for
/CN=host.example.com]

This error indicates that the certificate is failing to validate since the clock between the node and the puppet master differs. In my case, the clock on the node was far behind the master due to a VirtualBox clock drift issue.

In this case, it was a simple case of re-syncing the clock to resolve the issue. However if the master had been generating certs with the clock far in the future, I would have needed to re-generate my node certificates entirely as the certs would also be incorrect.

WordPress & SSL Fixes

I’ve been using WordPress for this blog for a number of years now – at some point I realised that whilst writing my own code is fun, there’s no need to reinvent yet-another-fucking-blog-platform and ended up selecting WordPress to use for my content, on the basis of it’s strong and active development and community.

Generally it’s pretty good, but there are times it disappoints, such as WordPress expecting servers to have FTP for unpacking updates and plugins (it’s 2013 guys, SFTP at least!), excessively setting cookies which makes caching layers more complex and doing stupid stuff with storing full URLs inside the database for page links and image resources.

The latter has been impacting me in particular. Visitors to my site have had the option of using HTTP or HTTPS (SSL secured) access methods for some time, but annoyingly whenever I posted an article with images, WordPress includes all the images using http://. This mixed content type prevents browsers from showing the lock icon (best case) or throws up a nasty error (worst case) depending on the browser and it’s level of concern for user safety for mismatched content.

Dubious Firefox is dubious about this site.

Dubious Firefox is dubious about this site, no lock icon of security here!

Despite having accessed the site on https://, WordPress still uses http:// for my images.

Despite having accessed the site on https://, WordPress still uses http:// for my images.

I could work around this by setting the WordPress base URL for my site to be https://www.jethrocarr.com, but then images served at the unsecured http:// site would also be served via SSL, which is just adding pointless load to the server (not that SSL termination really adds much load these days, but damnit, I’m being a purist here!).

I was hoping that it was a misconfiguration of my WordPress setup, but reading online it seems that this is a known issue with WordPress and a whole bunch of modules, hacks and themes have sprung up to fix/workaround the issue…

Of course there’s an easier way – fix it at the webserver layer! Both Nginx and Apache have modules to do substitutions in page content on load, for Nginx there’s HttpSubModule and for Apache there is mod_substitute. In my case with stock Apache 2.2 on CentOS 5, I was able to fix the whole issue by adding the following to my SSL vhost configuration:

# Fix SSL URLs thanks to WordPress hardcoding http:// links to images :'(
<Location />
    AddOutputFilterByType SUBSTITUTE text/html
    Substitute "s|http://www.example.com|https://www.example.com|"
</Location>

Following this, things look much better:

The lock icon of browser approval!

The lock icon of browser approval!

All media files are now https://, not http://

All media files are now https://, not http://

Technically this substitution will have some level of performance impact, as it has to process the generated HTML content and check for strings to replace, but the impact is so low that I wasn’t able to measure it amongst the usual variation of page response times – and it’s not going to be anywhere as slow as mod_php and WordPress itself anyway. ;-)

Finally, if you haven’t already, you probably want to change the following in wp-config.php:

define('FORCE_SSL_ADMIN', true);

This forces all WordPress logins and wp-admin activities to take place under HTTPS which is a pretty good idea if you ever post to your blog from an unsecured network.

SSL Intermediate CA Bundles with Amazon

When configuring SSL services, generally you need to set a certificate, a private key and the CA bundle containing the intermediate certificate(s), which is often a bundle of several different certificates.

For example, https://www.jethrocarr.com‘s configuration looks like:

SSLEngine on
SSLCertificateFile jethrocarr.com.crt
SSLCertificateKeyFile jethrocarr.com.key
SSLCertificateChainFile startssl.intermediate.ca.crt

When your browser connects, it doesn’t trust jethrocarr.com.crt, but it checks it against the certificates that have signed it in startssl.intermediate.ca.crt – and those certificates are signed by the CA that your browser trusts.

This means that the CAs can protect their root certificates which are trusted by the browser much more securely and sign their certificates with intermediates than can be revoked and easily(ish) replaced should the need arise.

Generally this works fine from the end user perspective, although there are sometimes issues when an sysadmin forgets to add the intermediate CA bundle and doesn’t immediately notice as sometimes some browsers work fine whilst others fail depending whether or not they already trust the intermediates.

 

Today I ran into a different new issue where Amazon Web Services is fussy about the order of the certificates in that bundle when adding a certificate to an Elastic Load Balancer for SSL termination.

Any attempt to upload my certificate was met with “Invalid Public Key Certificate”, which didn’t make a lot of sense as I was certain that my certificates were OK. It was easy to verify and prove this, using OpenSSL:

$ openssl rsa -noout -modulus -in example.com.key | openssl md5
(stdin)= 30e1b6cb4168117b7923392ca536c701

$ openssl x509 -noout -modulus -in example.com.crt | openssl md5
(stdin)= 30e1b6cb4168117b7923392ca536c701

$ openssl verify -verbose -CAfile cabundle.crt example.com.crt 
example.com.crt: OK

This proved that my certificates were all correct so the fault was Amazon-side. A post on their forums helped me “fix” the issue, by adjusting the order of my CA bundle, which subsequently fixed the error.

So is this a bug with Amazon? It’s tricky to say – there are several posts online which state that the order is important for some systems, but not for all. Clearly anything based around OpenSSL doesn’t care, as it was able to verify my out-of-order CA bundle happily enough.

As one does with issues like this, I dug into RFC 3280 which details how the certificate path validation should occur. Section 6.1 (Basic Path Validation) details that the path validation process is actually outside the specification, but then goes on and defines how the validation could occur, with the order of the certificates being implied, but not stated outright.

The primary goal of path validation is to verify the binding between
a subject distinguished name or a subject alternative name and
subject public key, as represented in the end entity certificate,
based on the public key of the trust anchor.  This requires obtaining
a sequence of certificates that support that binding.  The procedure
performed to obtain this sequence of certificates is outside the
scope of this specification.

To meet this goal, the path validation process verifies, among other
things, that a prospective certification path (a sequence of n
certificates) satisfies the following conditions:

   (a)  for all x in {1, ..., n-1}, the subject of certificate x is
   the issuer of certificate x+1;

   (b)  certificate 1 is issued by the trust anchor;

   (c)  certificate n is the certificate to be validated; and

   (d)  for all x in {1, ..., n}, the certificate was valid at the
   time in question.

Following the above, the specification goes on into detail different ways the path can be validated, which also imply that the certificates should be read in and then sorted by software, but it doesn’t actually state exactly.

Sadly with the way that this specification is written it’s not clear, which means the only 100% certain way to ensure nothing is unhappy is to order the CA bundles file in the correct order, which is something I would expect the SSL provider to do when they provide you with the files.

Attack vectors for personal computers

The sad thing is, I ran out of space to keep adding arrows.

Click image to enlarge

I’ve put together a (very) simplified overview of various attack vectors for an end user’s personal computer. For a determined attacker with the right resources, all of the above is potentially possible, although whether an attacker would go to this much effort, would depend on the value of the data you have vs the cost of obtaining it.

By far the biggest risk is you, the end user – strong, unique passwords and following practices such as disk encryption and not installing software from questionable vendors is the biggest protection from a malicious attacker and will protect against most of the common attacks including physical theft and remote attacks via the internet.

Where is gets nasty is when you’re up against a more determined attacker who can get hardware access to install keyloggers, can force a software vendor to push a backdoored software patch to your system via an update channel (ever wondered what the US government could make Microsoft distribute via Windows Update for them?), or has the knowledge on how to pull of an advanced attack such as putting your entire OS inside a hypervisor by attacking UEFI itself.

Of course never forget the biggest weakness – beating a user with a wrench until they give up their password is a lot cheaper than developing a sophisticated exploit if someone just wants access to some existing data.

Why SSL is really ISL

Secure transmission of data online is extremely important to avoid attackers intercepting data or claiming to be a site that they are not. To provide this, a technology called SSL/TLS (and commonly seen in the form of https://) was developed to provide security between an end user and a remote system to guarantee no interception or manipulation of data can occur during communications.

SSL is widely used for a huge number of websites ranging from banks, companies, and even this blog, as well as other protocols such as IMAP/SMTP (for Email) and SSH (for remote server shell logins).

As a technology, we generally believe that the cryptography behind SSL is not currently breakable (excluding attacks against some weaker parts of older versions) and that there’s no way of intercepting traffic by breaking the cryptography. (assuming that no organisation has broken it and is staying silent on that fact for now).

However this doesn’t protect users against an attack by the attacker going around SSL and attacking at other weak points outside the cryptography, such as the validation of certificate identity or by avoiding SSL and faking connections for users which is a lot easier than having to break strong cryptography.

 

Beating security measures by avoiding them entirely

It’s generally much easier to beat SSL by simply avoiding it entirely. A technology aware user will always check for the https:// URL and for the lock symbol in their browser, however even us geeks sometimes forget to check when in a hurry.

By intercepting a user’s connection, an attacker can communicate with the secure https:// website, but then replay it to the user in an unencrypted form. This works perfectly and unless the user explicitly checks for the https:// and lock symbol, they’ll never notice the difference.

Someone is going to be eating a lot of instant noodles for the next few years...

If you see this, you’re so, so, fucked. :-(

Anyone on the path of your connection could use this attack, from the person running your company network, a flatmate on your home LAN with a hacked router or any telecommunications company that your traffic passes through.

However it’s not particularly sophisticated attack and a smart user can detect when it’s taking place and be alerted due to a malicious network operator trying to attach them.

 

“Trusted” CAs

Whilst SSL is vendor independent, by itself SSL only provides security of transmission between yourself and a remote system, but it doesn’t provide any validation or guarantee of identity.

In order to verify whom we are actually talking to, we rely on certificate authorities – these are companies which for a fee validate the identify of a user and sign an SSL certificate with their CA. Once signed, as long as your browser trusts the CA, your certificate will be accepted without warning.

Your computer and web browsers have an inbuilt list of hundreds of CAs around the world which you trust to be reliable validators of security – as long as none of these CAs sign a certificate that they shouldn’t, your system remains secure. If any one of these CAs gets broken into or ordered by a government to sign a certificate, then an attacker could fake any certificate they want to and intercept your traffic without you ever knowing.

I sure hope all these companies have the same strong validation processes and morals that I'd have when validating certificates.

I sure hope all these companies have the same strong validation processes and morals that I’d have when validating certificates…

You don’t even need to be a government to exploit this – if you can get access to the end user device that’s being used, you can attack it.

A system administrator in a company could easily install a custom CA into all the desktop computers in the company By doing this, that admin could then generate certificates faking any website they want, with the end user’s computer accepting the certificate without complaint.

The user can check for the lock icon in their browser and feel assured of security, all whilst having their data intercepted in the background by a malicious admin. This is possible due to the admin-level access to the computer, but what about external companies that don’t have access to your computer, such as Internet Service Providers (ISPs) or an organisation like government agencies*? (* assuming they don’t have a backdoor into your computer already).

An ISP’s attack options are limited since they can’t fake a signed certificate without help from a CA. There’s also no incentive to do so, stealing customer data will ensure you don’t remain in business for a particularly long time. However an ISP could be forced to do so by a government legal order and is a prime place for the installation of interception equipment.

A government agency could use their own CA to sign fake certificates for a target they wish to intercept, or force a CA in their jurisdiction to sign a certificate for them with their valid trusted CA if they didn’t want to give anything away by signing certificates under their organisation name. (“https://wikileaks.org signed by US Government” doesn’t look particularly legit to a cautious visitor, but would “https://wikileaks.org signed by Verisign” raise your suspicions?)

If you’re doubting this is possible, keep in mind that your iOS or Android devices already trust several government CAs, and even browsers like Firefox include multiple government CAs.

http://support.apple.com/kb/ht5012

http://support.apple.com/kb/ht5012

Even the open source kids are pwned

It’s getting even easier for a government to intercept if desired –  more and more countries are legislating lawful interception requirements into ISP legalisation, requiring ISPs to intercept a customer’s traffic and provide it to a government authority upon request.

This could easily include an ISP being legally required to route traffic for a particular user(s) through a sealed devices provided by the government which could be performing any manner of attacks, including serving up intercepted SSL certs to users.

 

So I’m fucked, what do I do now?

Having established that by default, your SSL “secured” browser is probably prime for exploit, what is the fix?

Fortunately the trick of redirecting users to unsecured content is getting harder with browsers using methods such as Extended Validation certs to make the security more obvious – but unless you actually check that the site you’re about to login to is secure, all these improvements are meaningless.

Some extensions such as HTTPS Everywhere have limited whitelists of sites that it knows should always be SSL secured, but it’s not a complete list and can’t be relied on 100%. Generally the only true fix for this issue, is user education.

The CA issue is much more complex – one can’t browse the web without certificate authorities being trusted by your browser so you can’t just disable them. There are a couple approaches you could consider and it really depends whether or not you are concerned about company interception or government interception.

If you’re worried about company interception by your employer, the only true safe guard is not using your work computer for anything other than work. I don’t do anything on my work computer other than my job, I have a personal laptop for any of my stuff and in addition to preventing a malicious sysadmin from installing a CA, by using a personal machine also gives me legal protections against an employer reading my personal data on the grounds of it being on a company owned device.

If you’re worried about government interception, the level you go to protect against it depends whether you’re worried about interception in general, or whether you must ensure security to some particular site/system at all cost.

A best-efforts approach would be to use a browser plugin such as Certificate Patrol, which alerts when a certificate on a site you have visited has changed – sometimes it can be legitimate, such as the old one expiring and a new one being registered, or it could be malicious interception taking place. It would give some warning of wide spread interception, but it’s not an infallible approach.

A more robust approach would be to have two different profiles of your browser installed. The first for general web browsing and includes all the standard CAs. The second would be a locked down one for sites where you must have 100% secure transmission.

In the second browser profile, disable ALL the certificate authorities and then install the CAs or the certificates themselves for the servers you trust. Any other certificates, including those signed by normally trusted CAs would be marked as untrusted.

In my case I have my own CA cert for all my servers, so can import this CA into an alternative browser profile and can safely verify that I’m truly taking to my servers directly, as no other CA or cert will ever be accepted.

 

I’m a smart user and did all this, am I safe now?

Great stuff, you’re now somewhat safer, but a determined attacker still has several other approaches they could take to exploit you, by going around SSL:

  1. Malicious software – a hacked version of your browser could fake anything it wants, including accepting an attackers certificate but reporting it as the usual trusted one.
  2. A backdoored operating system allows an attacker to install any software desired. For example, the US Government could legally force Apple, Google or Microsoft to distribute some backdoor malware via their automatic upgrade systems.
  3. Keyboard & screen logging software stealing the output and sending it to the attacker.
  4. Some applications/frameworks are just poorly coded and accept any SSL certificate, regardless of whether or not it’s valid, these applications are prime to exploit.
  5. The remote site could be attacked, so even though your communications to it are secured, someone else could be intercepting the data on the server end.
  6. Many, many more.

 

The best security is awareness of the risks

There’s never going to be a way to secure your system 100%  – but by understanding some of the attack vectors, you can decide what is a risk to you and make the decision about what you do and don’t need to secure against.

You might not care about any of your regular browsing being intercepted by a government agency, in that case, keep browsing happily.

You might be worried about this since you’re in the region of an oppressive regime, or are whistle-blowing government secrets, in which case you may wish to take steps to ensure your SSL connections aren’t being tampered with.

And maybe you decided that you’re totally screwed and resort to communicating by meeting your friends in person in a Faraday cage in the middle of nowhere. It all comes down to the balance of risk and usability you wish to have.

Firefox Mobile for Android CAs

I’ve been using Firefox Mobile on Android for a while (thanks to the fact that it means I can use Firefox Sync between my laptop and mobile to share data). Overall it’s pretty good and the last few releases have fixed up a lot of the past stability issues and UI problems, it’s in a pretty decent state now.

One of the unfortunate problems I’ve had with it until recently is that the application was refusing to import custom certificate authorities. Whilst Android has it’s own CA store, add on browsers (inc Firefox Mobile) can have their own CA stores and the manageability of these can vary a lot.

In the case of Firefox Mobile, the ability to manage certificates was not ported across from the desktop version, meaning that none of my web applications would validate against my custom CA.

However as a passable solution, it’s now possible to import the CA file by downloading a PEM version of the CA certificate in the browser. Just upload a copy of the PEM formatted certificate to a webserver and download the file with the browser to install.

Installing CAs into Firefox Mobile (PEM formatted file).

Installing CAs into Firefox Mobile (PEM formatted file).

Now the biggest problem left is sites and applications that have poorly written user agent detection and assume that the only mobile devices that possibly exist are devices that have the iPhone or stock Android user agent. :-( *glares at Atlassian in particular*

Custom CA certificates & Android

With the number of servers I have internally, I have setup my own Certificate Authority and sign all my internal SSL certificates against this private CA.

This offers the useful advantage of being able to import the one CA certificate into all my devices and then being able to validate all connections to remote systems – if you run more than one or two personal servers, I’d highly recommend this approach – certificate signing takes a little bit of getting used to, but it’s a good skill to have.

As I want to access a number of systems via my Android mobile, I needed to import this CA file – the following instructions were followed with ICS release 4.0.3, however it may apply to earlier/later releases as well.

If you’ve followed most typical instructions for building your CA, you will have an PEM encoded CA certificate file in ASCII format. This is fine for import into most browsers and desktop OSes, however Android is particularly fussy with it’s input and requires a binary format only.

You can convert the CA PEM format file with the following command:

openssl x509 -inform PEM -outform DER -in CA.pem -out CA.crt

Then transfer the generated CA.crt file to the sdcard – easiest is via adb:

adb push CA.crt /sdcard/

Once done, you will be able to tell Android to install the CA file via Settings -> Security -> Credential Storage and selecting “Install from storage” and following prompts.

To verify functionality, easiest test is to access an https website signed with your CA certificate via the browser.

Some commenters have had issues - here is me importing a valid CA cert in DER format.

Some commenters have had issues – here is me importing a valid CA cert in DER format.