Although Docker 1.13 brings it’s usual load of features the most prohiminent one is certainly the secret management. There are many nice articles describing this feature so instead of presenting this feature, this article will focus on a solution to use Docker secrets with Letsencrypt.
Docker Secrets aim at providing a simple and secure way to store and use confidential data such as password, private keys, etc. The API is pretty simple:
- a new command
docker secretto create, delete, list or inspect the secrets
- new options to
docker service createand
docker service updateto control the container’s access to the secrets.
The secret creation is quite straightforward as
docker secret create --help will show:
When a secret is created Docker will guarantee to keep it encrypted when sending it to any remote node (encryption in transit) or when stored on the server filesystem (encryption at rest). The secrets are stored in the Raft log which is encrypted and replicated to all the managers in the cluster.
It is important to know that the log encryption is only performed by Docker 1.13 and above which means that if one of the cluster managers is running docker 1.12, the secrets can be stored uncrypted in this server filesystem.
secret subcommands are really as simple as this one with a couple of things worth noting:
- Secrets are immutable: you can create a secret, delete a secret but you cannot change it.
- A secret cannot be deleted while a service is using it.
Container Access to Secrets
Docker secrets implementation is strongly tied to swarm as the storage in the Raft log might suggest. As a consequence, secret are not available to plain old containers but only to services. At runtime, an in-memory filesystem will be mounted inside the container on
/run/secrets and will contain one file for each secret the services has been given access to.
A service can be given access to a secret throught the
--secret option of the
docker service create command. For instance:
docker service create --secret cartman will create a
/run/secrets/cartman file inside the container. A more complex version of this option allows to specify the name of the secret file and specify the owner, group and the permissions of the secret file (have a look at Create a service with secret for more information).
docker service update commands have
--service-add options (more information here).
Before talking about the integration with Docker secrets, let’s have a closer look at how Letsencrypt is actually working.
There are many ways of using Letsencrypt in this article I’ll be considering only the use of Cerbot. To add more restriction, I’ve only tested this with the standalone plugin while I’m pretty sure it’ll also work with the webroot plugin as well.
Obtaining a Certificate
If you are running a server with a public connection, getting the first certificate from Letsencrypt is quite easy:
In the current directory you’ll find a
etc directory containing the certificates, the private keys and some miscellaneous configuration files used by letsencrypt. For instance the
etc/live/hostname will contain everything you need to configure a web server with the newly generated certificates as indicated in the
Renewing the certificates
Letsencrypt create certificate with a very short validity: 90 days. The idea is to leverage the automated system to renew frequently your web servers’ certificate. Renewal is even easier that the creation as it can be done with:
--force-renewaloption in addition to
--stagingin order to renew the certificate even if we are not withing 30 days of the expiration date.
At this point you can notice that symbolic links in the
etc/live/hostname directory have been updated to reflect the certificate renewal. If you look closer you’ll see that the files are merely symbolic links to files located under
First Integration with Secrets
In order to test the integration I created a version of Nginx with SSL enabled and using key and certificate located under
/run/secrets to be compatible with Docker Secrets:
You can build your own image or use the pre-built ggtools/test-nginx-ssl.
Creating the Secrets
We are going to create two secrets:
Creating the Service
The next step is to create a service using these secret:
Thanks to the source/target syntax, the secret names can be mapped to the file names expected by the image.
At this point, this should be working and the nginx container could be accessed from a browser. There should have been a security warning from the browser but as we used the staging environment this is completely normal. Should we celebrate then? Naaaah. There’s a small issue with this setting: upgrading the certificate will be complicated as secrets are immutable and cannot be deleted until removed from all services. That’ll be mean that when the certificate is renewed, both secrets will have to be removed from all services.
Improving the Integration
We have seen that the files from the
etc/live/<hostname> directory are symbolic links to files in the
etc/archive/<hostname> directory. If we look at this directory we’ll find that Letsencrypt is actually versioning the certificates and files. Which is exactly what we need to implement secret rotation.
Creating the Secrets
We are still going to create two secrets but we will use the files from
etc/archive/<hostname> and will add a version to the secret names:
When renewing the certificate, Letsencrypt will create new files with a new version number. Will will then create new versionned secret:
Creating the Service
Not much difference at this point as we are only referencing the versionned secrets:
When renewing a certificate services could be updated one at the time since the old secrets will still exist:
This command will be completed using the normal service update mechanism and will stop and restart the service’s tasks according to the update configuration.
Letsencrypt and Docker secrets have a very similar philosophy with versionned immutable files/secrets which means that setting up a (basic) automation would be really simple:
In a second step, we can inspect the service and automatically update the services using outdated certificates. The script will be a little be more complicated as we have to retrieve the full configuration to add the renewed certificates (target, uid, gid and mode). Also total automation might not be wanted as the service updates can be triggered at any time.
A second script will check if the services are using the latest version of the certificates. While the first script is pretty harmless since it only creates new secrets, this one should be used with more care as all updated services will be restarted. In order to mitigate this problem the script will only look for service with the
In a full automated environment, both scripts could be run from Cerbot using the