CINQby Victor Louisa
Recently I was asked to give a demo about PKI using HashiCorp Vault. Everyone with some history in Ops knows the painful manual work for renewing TLS certificates, putting it on the server and maybe even restarting your application or web server. This is the main reason why such certificates are often valid for one or even two years. So to show the power of having an API to manage your PKI and request certificates, I created a demo where you can completely automate the renewal of TLS certificates for your app on Kubernetes, using both Vault and Cert-Manager.
We will use Minikube to create a Kubernetes cluster where we can run Vault, Cert-Manager, Minikube's ingress controller and a webapp, where:
We will first setup the prerequisites for demoing this certificate renewal automation:
Once the prerequisites are met, we are ready for the demo part which consist of:
I won't go into all the details of these prerequisites, but discuss only what we need.
To install Vault we use Helm and just follow the docs from HashiCorp.
Wait for the vault-0 pod to get healthy and then unseal Vault. We write the Vault key to a file, so we can retrieve it when necessary. (This is clearly very insecure, but ok for this blog.)
We now have Vault up and running.
We will often want to execute Vault CLI commands and can use the vault-0 pod to execute those. For instance like this:
We enable Kubernetes Auth Method and connect our Kubernetes cluster with Vault. For now it is enough to know that
Note that these issuer service account and pki policy do not exist yet, that is fine, we will create them later.
We will deploy a webapp and expose it with ingress, so it can use our TLS certificate to receive traffic over HTTPS.
For our ingress we use the domain name demo.example.com and a Kubernetes secret demo-example-com-tls where we will store our certificate.
Note: This Kubernetes secret demo-example-com-tls doesn't exist yet, but that is fine, we will create it later.
This ingress will get an external IP address (this can take some time) and since we use Minikube with a vm driver, this external IP address is equal to the IP of the vm.
To make the demo more pretty, we will add this ip to our /etc/hosts file so we can browse to our domain.
Edit your hosts file with sudo vi /etc/hosts, like this:
You can then browse to https://demo.example.com, or use curl to go to our webapp.
The ingress controller will serve a default self-signed certificate since we did not configured any certificate yet.
With curl we can bypass this simple:
What do we have so far?
We have Vault running and connected to our Kubernetes cluster and we have a webapp exposed via ingress.
We are now ready to automate certificate renewal by:
We will enable PKI in Vault. For most settings the default values suffices for this demo, we only:
With this Vault pki role we define what kind of certificates we are allowed to request, in this case: certificates for example.com and all it's subdomains and, most importantly, with a maximum time to live of 3 minutes: max_ttl=3m.
Now that we have an api to request certificates, we can start using it.
We will deploy Cert-Manager and configure Vault to be the issuer of the certificates. When that is done, we can define our certificate and Cert-Manager will request and renew the certificate when it will expire.
We install Cert-Manager v1.3.0 with Helm and just follow the docs from Jetstack. This will install Cert-Manager in a separate namespace and also install it's custom resource definitions CRD's.
From the CRD's we will use Issuer and Certificate.
First with CRD Issuer, we can define the issuer of the certificates. Recall the Kubernetes Auth Method configuration, where we granted the Kubernetes service account issuer access to our pki api endpoints. Hence we create a service account issuer in Kubernetes. We then define the Issuer CRD with a.o.:
Second, with CRD Certificate, we can define our TLS certificate, with:
Cert-Manager can now request this certificate at our issuer. Once that is done, you will see that a new secret is created.
We are done! Let's check if we can see our new TLS certificate.
Small recap on what we have now:
When we now browse to https://demo.example.com, or use curl to go to our webapp, we still get an invalid certificate. But when we look closer to the certificate, we see that it is a certificate from Vault.
For Firefox, go to:
Note that:
Remember we defined Vault to only issue certificates with 'max_ttl=3m'. Apparantly Vault adds another 30 seconds to it, but we don't care for now. The cool thing is that the certificate needs to be renewed very soon. So we wait some time, refresh our browser and see that it has new values for valid from and valid to. The certificate is renewed, cool!
So what happened with all the painful manual work of renewing your certificate?
It is automated by Cert-Manager who leveraged the PKI api of Vault. Cert-Manager monitors the TTL of the certificate and renews the certificate for you at the defined issuer, which is Vault in this case. Other possible issuers are Let's Encrypt (nice demo) and Venafi which both can be used for internet exposed TLS endpoints. Vault suites more the use-case of intranet exposed endpoints with certificates of an internal PKI managed by Vault.
To finish the demo, and convince also the non techies, we can add the Vault root CA to our browser so the certificate is validated properly and we get a valid TLS padlock.
Recall that, when we created the root CA, we stored it in a file in the Vault container. We first copy the root CA to our local machine.
Then import it into our browser.
For Firefox, go to:
Now browse to 'https://demo.example.com' again and we will have a valid HTTPS connection and our TLS padlock!
These guides helped me to create this blog:
Thank you for reading my blog and I hope I have helped you get setup with using Vault and Cert-Manager to automate the renewal of your TLS certificates. You can reach out to me on LinkedIn of by e-mail if you have any questions.