Ik heb de afgelopen jaren veel met Kubernetes gewerkt bij onze klanten en ondertussen is het uitgegroeid tot mijn favoriete (container/cloud) platform. In onze DevOps unit bij CINQ hebben wij veel aandacht besteed aan Kubernetes met interne kennis-sessies, een reis naar KubeCon Europe '18 in Kopenhagen, ons eigen Kubernetes bootcamp-weekend in Friesland én een reis naar KubeCon in Detroit in 2022. 

Ik wilde graag mijn CKA (Certified Kubernetes Administrator) halen omdat ik in de breedste zin wilde weten hoe Kubernetes precies werkt, in plaats van alleen maar de nodige resources te gebruiken voor deployments. Denk hierbij aan de werking van de controlplane en het gebruiken van alle resources die Kubernetes tot je beschikking stelt. Dat zijn er namelijk nogal wat. Ook is het natuurlijk fijn om te weten dat je een bepaalde techniek beheerst.

Het CKA examen is een examen van 2 uur waar er van je verwacht wordt 15 tot 20 problemen/vraagstukken vanaf de commandline op te lossen in verschillende Kubernetes clusters. Een score van minimaal 66% is vereist om te slagen. 

Ik heb het examen pas ingepland toen ik er van overtuigd was dat ik de tijd zou hebben om een paar weken achter elkaar te kunnen studeren en te oefenen. Zeker ook omdat ik meerdere malen heb gehoord dat het een pittig examen is waar je goed gebruik moet maken van je tijd. 

Het examen kan je in je eigen huis/kantoor afnemen, waar je door een toezichthouder via de webcam en screensharing in de gaten wordt gehouden. Het online examen programma werkte prima. 

Hoe kun je jezelf nou zo goed mogelijk voorbereiden voor het examen? En welke punten hebben extra aandacht nodig? Hieronder geef ik 10 tips waardoor ik het examen met een score van 91% heb gehaald.

1. “Certified-kubernetes-administrator-with-practice-tests” 

Een alles omvattende cursus op Udemy door Mumshad Mannambeth. Zorg dat je de hele cursus hebt doorlopen en maak alle labs in Kodekloud. Alles wat in mijn examen werd gevraagd is behandeld in deze cursus.

Leer de kubectl commandline. Zorg dat je weet welk commando je waarvoor moet gebruiken, get, describe, logs, events, apply, etc.

Je mag de officiële Kubernetes documentatie erbij houden. Maak hier gebruik van door de cheatsheet erbij te pakken of snel te zoeken naar het onderwerp naar keuze.

Bouw een Kubernetes cluster handmatig vanaf de grond op. Alhoewel ik deze niet zo belangrijk vond als dat andere mensen die vonden, is het in ieder geval handig om het een keer te doen. Bovenstaande Udemy cursus heeft een guide om het op je eigen workstation met Vagrant en Virtualbox te doen. 

Bespaar jezelf wat tijd door de “imperative commands” te kunnen van simpele opdrachten. Denk aan: “kubectl run”, voor het deployen van simpele pods, of “kubectl expose” voor het aanmaken van een service die verwijst naar een bestaande pod. Dit zijn geen complexe opdrachten en zijn sneller uit te voeren op deze manier, in plaats van een YAML file te genereren of op zoek te gaan in de documentatie.

Zorg voor de complexere deployments dat je een YAML genereert, hergebruikt uit een eerdere opdracht of gebruikt vanuit de documentatie. Denk aan deployments met secrets of volumes. Gebruik hiervoor de voorbeelden in de documentatie en verander die naar wens.

Sla opdrachten over die je niet meteen weet, en maak deze later. Je hebt misschien de volle 2 uur nodig en elke punt is meegenomen. Vlag bepaalde vragen die je niet zeker weet om later op terug te komen.

Zorg dat je de Docker commando’s en concepten ook kent om te troubleshooten. Soms heb je geen kubectl tot je beschikking. Basic Linux kennis om te troubleshooten en opdrachten uit te voeren zoals: systemctl, nslookup, vi/vim, ps, ls, cp etc. 

Hetgeen waar CKA zich in onderscheid ten opzichte van CKAD is met name infrastructuur kennis. Zorg dat je gerelateerde opdrachten hieraan goed kunt maken. Denk aan ETCD backups, het gebruik van KubeADM, controlplane service configuratie en certificaten. Deze vragen zijn vaak veel punten waard en kosten meer tijd.

Controleer elke deployment of pods wanneer je deze hebt aangemaakt. Gebruik met de “get” of “describe” commands of jouw pods/deployments de juiste naam hebben, de juiste images gebruiken en überhaupt draaien en doen wat deze zouden moeten doen.

Met deze zorgvuldige voorbereiding en werkwijze tijdens het examen, heb ik het in één keer weten te halen. Heel veel succes met studeren en het examen. En als je vragen hebt, neem contact met ons op.

👩‍🏫In twee dagen Kubernetes Advanced training, worden onze DevOps engineers klaargestoomd voor het KCA examen.

Dag 1 zit er bijna op, tijd voor de vrijmibo! 🍻

@container_solutions #kubernetes #containersolutions #joure #cinqict #work #prep #workshop #devops #cinq

Met deze 10 tips ga je zeker slagen voor je Certified Kubernetes Administrator examen.

10 tips voor het halen van je Certified Kubernetes Administrator examen

10 tips voor het halen van je Certified Kubernetes Administrator examen (2023)

Site Reliability Engineering (SRE) is a new discipline which is quickly gaining traction within the IT landscape. First conceived by Google, it is now adopted mainly by large enterprises such as Netflix, LinkedIn, Target, Tesla and many more. It is known for a focus on automating manual tasks without enduring value (toil), treating operations as a software problem and its unique approach to systems reliability.

There are many ways to implement SRE. SRE is a multi-faceted discipline. You can create Service Level Objectives to define the level of service to deliver to users, focus on reducing toil with targeted measures, define a risk tolerance for a service and do release engineering to make releases streamlined and consistent.

These, amongst some other practices, primarily dictate a certain way of working. However, some practices are more focused on designing systems to be reliable, highly available and robust. In other words, I see two categories:

Practices which are primarily ways of working; They also have technical implications, but the largest part is how we interact as employees and/or set priorities in what we find important to work on.

Practices which are primarily ways of designing systems; The key aspect of these are designing an application according to certain guidelines such as reliability, availability, robustness, etc. Subsequently the application is created to function to reach these goals.

Not all practices are as easy to apply for a given business. A lot of Googles practices which it has written about in its SRE Handbook has stemmed from dealing with its growing pains. As the organization grew it required cleverer solutions so it could continue to deliver a good service while using resources effectively. Small to medium businesses however do not have some of the challenges and use resources differently. Some examples:

They often use off-the-shelf software as opposed to tailor-made solutions

Teams are more converged. They often operate within one region and have a smaller selection of services to deliver

They have a lesser variety of roles in a team

They have less resources to their disposal

Very highly skilled people tend to flock to large prestigious enterprises as opposed to smaller companies

Given the above differences, the way small to medium business can implement SRE differs. For example:

While using off the shelf software, everything comes at it is. Some practices of SRE dictate how to design and build applications according to certain standards but this is not something you do in this case

Even when software is written for systems administration, it is likely either a smaller application or a script, as opposed to larger complicated systems

Some of these system design principles can be complicated to grasp and to implement, such as distributed consensus and distributed periodic scheduling. The employees working at the company might not have the skills necessary to work with these principles

These kinds of businesses might not scale as fast and investments to a high degree of scalability might not pay off for them

Even though mostly off-the-shelf software is used, you can still apply a lot of the principles in SRE. Instead of using them to design your own software, try to understand the principles and buy software which is built with these principles in mind. For example, you can buy software which has high availability as a feature. 

Having the ability to create your own software gives more opportunities to reduce toil as you have more control over the way your infrastructure is managed. However, you can do a lot with scripts, pipelines or small applications. Doing a routine clean-up in an external system could easily be encapsulated in a Python script ran by a cronjob.

Many principles can be implemented with off-the-shelf software and widely available documentation. Service level objectives can be monitored in many monitoring solutions such as DataDog and Splunk. Having engineers be on-call effectively can also be done with tools such as VictorOps.

Your business might not benefit from all the scale advantages of companies like Google, but investments in scalability can still pay off. Using container orchestration for workloads means scaling applications happens in a consistent and controlled manner. Cloud solutions offer comparable solutions for VMs. Having good control over scaling also helps during maintenance as you can easily scale down a node and perform maintenance temporarily.

Google describes a lot of principles in their SRE Handbook. The base principles upon which they are based are universally applicable but the degree to which they can be implemented and how it is done is likely very different when comparing smaller businesses to large enterprises such as Google. Still, by choosing software which adheres to the principles described and provides features for utilizing SRE, using smaller pieces of software for automation and implementing practices to the appropriate degree, your business can also benefit from SRE.

Challenges of implementing SRE in small to medium businesses

Until recently, the Cloud Native Compute Foundation (CNCF) only provided certifications for very experienced hands-on Kubernetes skills, i.e. CKAD, CKA and CKS. In this article I will give you a quick overview of CNCF's latest addition: KCNA.

The existing credentials CKAD, CKA and CKS are well-known and respected in the Kubernetes community, but there were still two gaps:

There were no Kubernetes beginner certifications.

There were no certifications about any of the other 100 CNCF projects.

Therefore, CNCF created a new credential called Kubernetes and Cloud Native Associate (KCNA) which has been available since November 18th 2021 and aims for

I've already proven my Kubernetes skills by getting CKAD, so my interest was more in the part about the cloud native ecosystem. I consider it as a gap in my own knowledge. I know some tools very well, some little, but most of them not. Well, that probably holds for anybody since there are just so many CNCF projects:

Kubernetes is by far the biggest project, but some others have earned their place as well, like Prometheus, Helm, and Linkerd - as you can see in the image below.

Graduated project velocity, from the CNCF annual report 2021

By becoming a Kubernetes Certified Service Provider CINQ ICT earned some CNCF vouchers, so I used one to see what the Kubernetes and Cloud Native Essentials course and KCNA exam learns me about the rest of the cloud native ecosystem.

Kubernetes and Cloud Native Essentials - course

As expected, I was scanning and skipping most of the Kubernetes chapters. However, the content was well written, holistic and complete and provide a very good fundamental for Kubernetes beginners. Personally, I especially enjoyed the "additional resources" paragraphs, which contains links to some very good blogs and docs about all kind of Kubernetes fundamentals e.g. Docker v1.0 announcement, fundamentals of container networking, difference between OCI v.s. CRI and Docker v.s. Containerd, difference between containers and pods. Most of them go way deeper into Kubernetes than needed for the KCNA exam, but where therefor very interesting to me.

Putting my hopes on the final two chapters on Cloud Native Application Delivery and Observability, my hopes vanished quickly when I started reading them. The cloud native ecosystem is only touched lightly, with just a little extra attention to Prometheus basics. Other than that, it's more like an overview on which tool does what with minimal descriptions.

During the exam almost no questions about the rest of the cloud native ecosystem came by. Which makes KCNA even more just a beginners Kubernetes credential. 

All together it's nice that there is an entry-level Kubernetes certification now, it was definitely missing. For engineers that are starting their journey in the world of Kubernetes, this is definitely a great way to start. The part about cloud native ecosystem is disappointing. Also from a beginners point of view. So for CNCF, there is still a gap left and I hope it will be filled with some sort of certification on the cloud native ecosystem in the future. 

https://docs.linuxfoundation.org/tc-docs/certification/important-instructions-kcna

https://www.cncf.io/announcements/2021/11/18/kubernetes-and-cloud-native-essentials-training-and-kcna-certification-now-available/

https://training.linuxfoundation.org/training/kubernetes-and-cloud-native-essentials-lfs250/

https://www.cncf.io/cncf-annual-report-2021/

https://www.cncf.io/announcements/2021/02/24/cloud-native-computing-foundation-welcomes-47-new-members-at-the-start-of-2021/

Until recently, the Cloud Native Compute Foundation (CNCF) only provided certifications for very experienced hands-on Kubernetes skills, i.e. CKAD, CKA and CKSS. In this article I will give you a quick overview of CNCF's latest addition: KCNA.

Kubernetes and Cloud Native Associate (KCNA)

Site Reliability Engineering (SRE) is a new discipline which is quickly gaining traction within the IT landscape. First conceived by Google, it is now adopted mainly by large enterprises such as Netflix, LinkedIn, Target, Tesla and many more. It is known for a focus on automating manual tasks without enduring value (toil), treating operations as a software problem and its unique approach to systems reliability. You might hear of the advantages of SRE and want to introduce it to your organization. Perhaps you are a software engineer and want to convince your peers to try it out or you are a manager overseeing one or more teams and would like to introduce SRE to them. The question is, are you ready to start?  

In this article I would like to give you some tips and tricks in order to judge how ready your organization is for this SRE journey and if it is already adopted, determine the current level of SRE maturity.

As Google describes it, SRE is a superset of DevOps. Many of the SRE principles align with DevOps and build upon them. Having good adoption of DevOps principles is therefore an important factor in the success of implementing SRE. Is the DevOps maturity level relatively low, then you will likely benefit more by focusing on improving that first. 

Google describes a number of levels in automation, the higher levels (lower on the list) being more advanced:

No automation - all operational tasks are performed manually

Externally maintained system-specific automation - operational tasks are performed using external systems such as scripts, specific for this one system

Externally maintained generic automation - there are generic systems that can perform tasks on multiple different systems

Internally maintained system-specific automation - systems include automated operational tasks out-of-the-box

So, what makes the higher levels better than the lower levels? Why is it better for a system to be internally monitored than externally? If you use an external system to monitor another system, you are now managing two systems. This usually increases the effort to maintain this process. 

A system that is internally automated also has more possibilities to do so as it can directly tap into all of the functionality and therefore has more fine-grained control. If the automation is system specific as opposed to generic, the impact of this particular automated task is not as broad and use of these tasks scales more linear.

If you are low on this scale (less flexible and in lower amounts of automation), there will be more to be gained from automation but some aspects of SRE might not fully come into fruition just yet. SRE works best if there is a foundation of automation upon which to build. Additionally, hiring SRE engineers is easier as the profile of SRE engineers usually includes skills which are a match for more mature forms of automation. 

When it comes to SRE as a term, there are usually three definitions used. 

As a set of practices and principles - the SRE discipline

In practice, these definitions are often intertwined and mixed depending on the context of discussions.

These definitions are commonly used in a number of forms of adoption. Depending on structure of your organization and how much knowledge about SRE is present and some of its principles are perhaps already adopted.

Consulting - SRE maturity level is zero to non-existent. Consultants provide guidance and advice but do not actively participate in engineering activities.

Embedded - SRE maturity is at a low level. Consultants actively participate in engineering but also keep sharing knowledge and giving direction

Platform - For medium SRE maturity. A SRE-minded platform team takes care of all supporting services and/or infrastructure for services.  Knowledge about delivered services is limited

Slice and Dice - Again for medium SRE maturity. Services are divided into functional slices such as payment processing or the user-facing helpdesk. SRE engineers responsible for this slice have knowledge of the complete 'flow' of the service and can therefore better tailor to automation needs

Full SRE - High SRE maturity. The full scope of a particular service or set of services involves SRE engineers and all team members have an SRE mindset

For all maturity levels there are gains to be made from adopting SRE and the aforementioned forms tailor to these varying needs. The red line in these forms is that as the SRE maturity level progresses, the scope of SRE widens and more of the service delivery flow is imbued with SRE practices and principles. Choose the form that suits the maturity level to make the most out of SRE adoption. 

So, know knowing there are different forms to adopt based on the maturity level, how can you judge how high the SRE maturity level is? Assuming SRE is already implemented to some degree, there are some indicators.

There is executive buy-in on an SRE strategy. Investing in SRE means investing more in reliability than velocity if reliability is under a certain threshold. For a lot of teams it means more investment in automation than new features, in the hopes it pays itself off in the future. As executives usually have a say in high-level/abstract strategy, they have to agree to one which includes SRE for it to work.

SREs are involved early in the service design process and can exert influence. Designing a service to be reliable and observable, among other things, is easier in earlier stages of the design process than post-facto.

There is a broad focus on automating repetitive tasks. If you can distinguish many operational tasks which still require manual labor and are not on the agenda to be addressed, likely there is still room for improvement in terms of SRE adoption

Both current incidents are mitigated and future ones. Good SRE engineers practice anti-fragility: managing a service that is not only stable under normal operating conditions but also can recover easily when disaster strikes. Additionally, if monitoring is adequately implemented and operational tasks have a high degree of automation, solving incidents should be a lot easier.

The key take-away here is to inspect how well integrated SRE is in the process and how effectively they handle automation, incident response, the degree to which SREs are involved in the design process and whether there is executive buy-in. In other words, is SRE implemented well and is there is support within the organization. 

Next to the SRE maturity level as a whole, there is also a way to judge the progression in terms of handling reliability. Effectively handling reliability is a key component of any SRE strategy and how it is dealt with is a good indicator on SRE maturity.

Reactive - Incidents are responded to after they happen. There is rarely any lessons learned and investments to reduce the number of incidents is limited

Proactive - The possibility for incidents are identified before they can occur and measures are taken to prevent them from happening

Strategic - Efforts are made to include reliability as an aspect to be specifically addressed in architectures, products and processes

Visionary - Reaching beyond the scope of the organization and getting involved in the broader spectrum. For example: Writing papers, giving advice to other organizations and making videos.

The saying "it is better to prevent than to cure" also applies to reliability. So being at the very least proactive to incidents is a good move to make. However, it is not needed to be very high on the spectrum per-se. It requires major investments in order to become a visionary on reliability and the benefits might not outweigh the costs of doing so. 

The key takeaway here is that there are various ways to judge how your organization fares in terms of SRE maturity. Depending on the maturity level, there are different rates of progression and approaches to advance on this level. 

A more 'junior' organization might focus more on making automation more generic, hire consultants to help with determining SLOs and make their incident response more proactive. 

A 'senior' organization will focus on developing tooling which comes with its own automation, broaden the scope of their existing SRE teams to include more of the service delivery flow and share knowledge about SRE with other people. Regardless of the maturity level, there are gains to be had but the right approach for the right level will mean gains will be maximized.

https://cloud.google.com/blog/products/devops-sre/evaluating-where-your-team-lies-on-the-sre-spectrum

https://cloud.google.com/blog/products/devops-sre/the-five-phases-of-organizational-reliability

https://sre.google/sre-book/automation-at-google/

You might hear of the advantages of SRE and want to introduce it to your organization. Perhaps you are a software engineer and want to convince your peers to try it out or you are a manager overseeing one or more teams and would like to introduce SRE to them. The question is, are you ready to start?

Is your organization ready for SRE?

Sinds Docker Enterprise is overgenomen door Mirantis is de naamgeving van het productportfolio compleet aangepast. Docker Enterprise heet tegenwoordig Mirantis Container Runtime, Universal Controle Plane (UCP) heeft de naam Mirantis Kubernetes Engine (MKE) gekregen en Docker Trusted Registry (DTR) heet Mirantis Secure Registry (MSR). 

Voor een opdracht bij een klant waar de omgeving voor 95% bestaat uit Windows Server 2019 en .NET als ontwikkel taal wordt gebruikt, is Linux een vreemde eend. Wat de opdracht interessant maakt was dat .NET containers gingen landen op Windows Server 2019 Core.

Dit in combinatie met een nieuwe Linux omgeving, waar MCR\MKE en MSR op geïnstalleerd werd. Om dit voor elkaar te krijgen zijn we aan de slag gegaan met Ansible-playbooks. Binnen de organisatie had men kennis hiervan, dus het toepassen van infrastructuur als code was bekend.  In deze opdracht zijn verschillende Ansible-playbooks gemaakt, waaronder het hardenen van de Linux omgeving zodat voldaan werd aan het security beleid. De betreffende playbooks werden in een repository geplaatst zodat versie beheer kon worden toegepast.

Het onderstaande voorbeeld werd o.a. gebruikt voor het het opzetten van de verschillende MKE-Clusters, d.m.v. Ansible code. Door gebuikt te  maken van de playbook kun je met 1 druk op de knop een MKE single node cluster opzetten, inclusief MSR en een Linux worker Node. Hoe we dit gedaan hebben leg ik hieronder uit. 

Om met de tutorial aan de slag te gaan heb je een PC of laptop met minimaal 16GB geheugen nodig, Vagrant en VirtualBox dient geïnstalleerd te zijn.

git clone https://dennisvermeulen@bitbucket.org/dennisvermeulen/mirantis-cluster.git

cd mirantis-cluster/

mirantis-cluster % vagrant up 

Installatie MKE/MSR met behulp van Ansible

mirantis-cluster % vagrant ssh vm1

[vagrant@cinqmke ~]$ cd ansible-mke-msr-wor/

[vagrant@cinqmke ansible-mke-msr-wor]$ ansible-playbook -b -K -i vm_hosts site.yml

BECOME password: vagrant 

Nadat alles ingesteld is via Ansible krijg je de onderstaande output.

Nu kun je inloggen in Mirantis Kubernetes Engine (MKE) en Mirantis Secure Registry met username “admin” en password “welcome01”.

Kies Next en plak onderstaande code in het veld Add Application File.

version: '3.7'
services:
  nginx:
    image: nginx:latest
    ports:
      - target: 80
        published: 5000
        protocol: tcp
        mode: host

Kies Create, wacht en ga naar de volgende URL http://192.168.33.72:5000. Als het goed is zie je dan de bevestiging dat nginx draait. 

Door het bestand vars in de directory ansible-mke-msr-wor/group_vars/ aan te passen heb je grip over de verschillende versies. Zie hiervoor ook: https://docs.mirantis.com/containers/v3.1/dockeree-release-notes/index.html

Wanneer je gebruik wilt maken van een MKE multinode cluster pas je het vm_host bestand aan in de directory ansible-mke-msr-wor.

Ruim de omgeving op, ga hiervoor als volgt te werk.

[vagrant@cinqmke ansible-mke-msr-wor]$ exit
mirantis-cluster % vagrant destroy
    vm3: Are you sure you want to destroy the 'vm3' VM? [y/N] y
==> vm3: Forcing shutdown of VM...
==> vm3: Destroying VM and associated drives...
    vm2: Are you sure you want to destroy the 'vm2' VM? [y/N] y
==> vm2: Forcing shutdown of VM...
==> vm2: Destroying VM and associated drives...
    vm1: Are you sure you want to destroy the 'vm1' VM? [y/N] y
==> vm1: Forcing shutdown of VM...
==> vm1: Destroying VM and associated drives…


 Pas het bestand aan, zoals hieronder aangegeven.

 mirantis-cluster % vi ansible-mke-msr-wor/vm_hosts


### MKE
[mke_main]
cinqmke ip_addr='192.168.33.70'

[mke]
cinqmke ip_addr='192.168.33.70'
cinqmsr ip_addr='192.168.33.71'
cinqwor ip_addr='192.168.33.72'

### MSR
[msr_main]
#cinqmsr ip_addr='192.168.33.71'

[msr]
#cinqmsr ip_addr='192.168.33.71'

### WORKER
[worker]
#cinqwor ip_addr='192.168.33.72'

mirantis-cluster % vagrant up 

Opnieuw installeren m.b.v. Ansible en je krijgt een multinode cluster:

mirantis-cluster % vagrant ssh vm1
[vagrant@cinqmke ~]$ cd ansible-mke-msr-wor/
[vagrant@cinqmke ansible-mke-msr-wor]$ ansible-playbook -b -K -i vm_hosts site.yml
BECOME password: vagrant
 

Met behulp van Ansible playbooks kun je met 1 druk op de knop een volledig MKE/MSR single- of multinode cluster opstarten en dit allemaal zonder al te veel inspanning. Ik hoop dat je er met deze tutorial uit komt. Als je vragen hebt, neem gerust contact met me op via email. 

In deze tutorial ga je met behulp van Ansible een multi-node Mirantis Container Engine (voorheen Docker Enterprise) cluster opzetten.

Met Ansible een Mirantis Container Engine cluster opzetten

Recently I was asked to give a demo about PKI using HashiCorp Vault. Everyone with some history in Ops knows the painful manual work for renewing TLS certificates, putting it on the server and maybe even restarting your application or web server. This is the main reason why such certificates are often valid for one or even two years. So to show the power of having an API to manage your PKI and request certificates, I created a demo where you can completely automate the renewal of TLS certificates for your app on Kubernetes, using both Vault and Cert-Manager.

We will use Minikube to create a Kubernetes cluster where we can run Vault, Cert-Manager, Minikube's ingress controller and a webapp, where:

Vault will give us an API to request certificates for our own PKI.

Cert-Manager will manage the renewal of our certificate.

The ingress will expose our webapp on HTTPS.

We will first setup the prerequisites for demoing this certificate renewal automation:

Once the prerequisites are met, we are ready for the demo part which consist of:

Setup Cert-Manager and request certificate

Virtual Box. Ingress works pretty nice for Minikube with a vm driver. I use virtual box as a vm driver, but you can also choose other Minikube vm drivers.

I won't go into all the details of these prerequisites, but discuss only what we need.

Create Minikube cluster with ingress addon

minikube start --driver=virtualbox

minikube addons enable ingress


To install Vault we use Helm and just follow the docs from HashiCorp.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm install vault hashicorp/vault \
    --set "injector.enabled=false"

# Get vault pod
kubectl get po vault-0

Wait for the vault-0 pod to get healthy and then unseal Vault. We write the Vault key to a file, so we can retrieve it when necessary. (This is clearly very insecure, but ok for this blog.)

# Create Vault unseal key
kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 -format=json > /tmp/demo-keys.json
cat /tmp/demo-keys.json | jq -r ".unseal_keys_b64[]"

# Unseal Vault
VAULT_UNSEAL_KEY=$(cat /tmp/demo-keys.json | jq -r ".unseal_keys_b64[]")
kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEY

We will often want to execute Vault CLI commands and can use the vault-0 pod to execute those. For instance like this:

# Get Vault token from our file
VAULT_TOKEN=$(cat /tmp/demo-keys.json | jq -r ".root_token")

# Get a shell in vault-0 pod with VAULT_TOKEN as environment variable
kubectl exec vault-0 -ti -- /bin/sh -c "VAULT_TOKEN=$VAULT_TOKEN sh" 

# Vault CLI example
vault token lookup

# Exit pod
exit

We enable Kubernetes Auth Method and connect our Kubernetes cluster with Vault. For now it is enough to know that

We will grant the Kubernetes service account issuer access to Vault

In this pki policy, the endpoints of our PKI engine are defined.

Note that these issuer service account and pki policy do not exist yet, that is fine, we will create them later.

# Prepare Vault CLI in `vault-0` pod
VAULT_TOKEN=$(cat /tmp/demo-keys.json | jq -r ".root_token")
kubectl exec vault-0 -ti -- /bin/sh -c "VAULT_TOKEN=$VAULT_TOKEN sh"

# Enable Kubernetes Auth Method
vault auth enable kubernetes
vault write auth/kubernetes/config \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Policy pki: read access to pki paths
vault policy write pki - <<EOF
path "pki*"                        { capabilities = ["read", "list"] }
path "pki/roles/example-dot-com"   { capabilities = ["create", "update"] }
path "pki/sign/example-dot-com"    { capabilities = ["create", "update"] }
path "pki/issue/example-dot-com"   { capabilities = ["create"] }
EOF

# Create Kubernetes Auth Method role: Grant service account `issuer` access via the `pki` policy
vault write auth/kubernetes/role/issuer \
    bound_service_account_names=issuer \
    bound_service_account_namespaces=default \
    policies=pki

# Exit pod
exit 

We will deploy a webapp and expose it with ingress, so it can use our TLS certificate to receive traffic over HTTPS.

For our ingress we use the domain name demo.example.com and a Kubernetes secret demo-example-com-tls where we will store our certificate.

Note: This Kubernetes secret demo-example-com-tls doesn't exist yet, but that is fine, we will create it later.

# Deploy webapp
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
# Create service
kubectl expose deployment web --port=8080
kubectl get service web

# Create ingress
cat <<EOF | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - demo.example.com
    secretName: demo-example-com-tls
  rules:
  - host: demo.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 8080
EOF

This ingress will get an external IP address (this can take some time) and since we use Minikube with a vm driver, this external IP address is equal to the IP of the vm.

# Show ingress external IP and Minikube's IP
kubectl get ingress
minikube ip

To make the demo more pretty, we will add this ip to our /etc/hosts file so we can browse to our domain.
Edit your hosts file with sudo vi /etc/hosts, like this:

# PKI Demo 
192.168.99.128 demo.example.com

You can then browse to https://demo.example.com, or use curl to go to our webapp.

The ingress controller will serve a default self-signed certificate since we did not configured any certificate yet.

# Ignore invalid cert
curl --insecure https://demo.example.com
# Show tls handshake details
curl -kivL https://demo.example.com

We have Vault running and connected to our Kubernetes cluster and we have a webapp exposed via ingress.

We are now ready to automate certificate renewal by:

We will enable PKI in Vault. For most settings the default values suffices for this demo, we only:

Give the root CA certificate a name which we can recognize: Demo Root Certificate Authority

With this Vault pki role we define what kind of certificates we are allowed to request, in this case: certificates for example.com and all it's subdomains and, most importantly, with a maximum time to live of 3 minutes: max_ttl=3m.

# Prepare Vault CLI in `vault-0` pod
VAULT_TOKEN=$(cat /tmp/demo-keys.json | jq -r ".root_token")
kubectl exec vault-0 -ti -- /bin/sh -c "VAULT_TOKEN=$VAULT_TOKEN sh"

# Enable PKI secret engine
vault secrets enable pki

# Generate root CA (and write to file)
vault write -format=json pki/root/generate/internal \
    common_name="Demo Root Certificate Authority" > /tmp/demo-root-ca.json
cat /tmp/demo-root-ca.json

# Configure pki api endpoints
vault write pki/config/urls \
    issuing_certificates="http://vault.default:8200/v1/pki/ca" \
    crl_distribution_points="http://vault.default:8200/v1/pki/crl"

# Create `pki` role to create certificates for example.com and all subdomains and with a maximum time to live of 3 minutes.
vault write pki/roles/example-dot-com \
    allowed_domains=example.com \
    allow_subdomains=true \
    max_ttl=3m

# Exit pod
exit

Now that we have an api to request certificates, we can start using it.

We will deploy Cert-Manager and configure Vault to be the issuer of the certificates. When that is done, we can define our certificate and Cert-Manager will request and renew the certificate when it will expire.

We install Cert-Manager v1.3.0 with Helm and just follow the docs from Jetstack. This will install Cert-Manager in a separate namespace and also install it's custom resource definitions CRD's.

kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install Cert-Manager 
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.3.0 \
  --set installCRDs=true

# View pods
kubectl get pods --namespace cert-manager

# View CRD's
kubectl get customresourcedefinitions.apiextensions.k8s.io

From the CRD's we will use Issuer and Certificate.

First with CRD Issuer, we can define the issuer of the certificates. Recall the Kubernetes Auth Method configuration, where we granted the Kubernetes service account issuer access to our pki api endpoints. Hence we create a service account issuer in Kubernetes. We then define the Issuer CRD with a.o.:

Connect it to the Vault server http://vault.default (to the pki/sign/example-dot-com uri in particular)

Authenticate via the token of our issuer service account

# Create service account
kubectl create serviceaccount issuer

# Token of service account `issuer`
ISSUER_SECRET_REF=$(kubectl get serviceaccount issuer -o json | jq -r ".secrets[].name")

# Define issuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://vault.default
    path: pki/sign/example-dot-com
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: issuer
        secretRef:
          name: $ISSUER_SECRET_REF
          key: token
EOF

# Show issuer
kubectl get issuer

Second, with CRD Certificate, we can define our TLS certificate, with:

Name of the Kubernetes secret, where the certificate needs to be stored: demo-example-com-tls

Specify at which issuer the certificate needs to be requested: vault-issuer

Define properties of our certificate:
undefined

# Define certificate
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: demo-example-com
  namespace: default
spec:
  secretName: demo-example-com-tls
  issuerRef:
    name: vault-issuer
  commonName: demo.example.com
  dnsNames:
  - demo.example.com
EOF

# View certificate CRD
kubectl get certificate

Cert-Manager can now request this certificate at our issuer. Once that is done, you will see that a new secret is created.

# See the request in the events
kubectl describe certificate demo-example-com

# New secret
kubectl get secrets demo-example-com-tls

We are done! Let's check if we can see our new TLS certificate.

Cert-Manager using this pki api to request a certificate

the certificate is stored as a Kubernetes secret demo-example-com-tls

Ingress gets demo-example-com-tls and uses it to enable TLS on our webapp

When we now browse to https://demo.example.com, or use curl to go to our webapp, we still get an invalid certificate. But when we look closer to the certificate, we see that it is a certificate from Vault.

In this blog our engineer Eric shows you how to automate the renewal of TLS certificates with Vault and Cert-Manager, running on a Kubernetes cluster.

How to automate the renewal of TLS certificates with Vault and Cert-Manager on Kubernetes

Automate the renewal of TLS certificates with Vault and Cert-Manager

Recently I presented a webinar about CI/CD and GitLab. It contained a demo of the Auto DevOps feature of GitLab. Connected to a Kubernetes cluster, this Auto DevOps feature really impressed me. Setting this up appeared to be rather easy. Hence I would like to share how you can do this, so you can play around with this Auto DevOps feature of GitLab yourself.

Why should you experience the power of GitLab's Auto DevOps feature with Kubernetes yourself?

Because, when you push just a simple Java Spring Boot helloworld app to your GitLab repository. The Auto DevOps feature gives you out-of-the-box an impressive, full fledged, very powerful standard pipeline. This pipeline isn't only complete in terms of testing, scanning, security and performance. But it also deploys your code on your Kubernetes cluster and can expose it at a public domain with TLS enabled.

Before we start, there are a couple of prerequisites. You need two accounts and three CLI's:

Eksctl: simple CLI to create an EKS cluster

Optional: Registered domain in AWS. Adding a domain name to this setup really makes the Auto DevOps feature shine.

Remark: There are minor costs involved in AWS for this setup, e.g. EKS, Load Balancer, route53.

Connected to a Kubernetes cluster, the GitLab Auto DevOps feature really impressed me. Setting this up appeared to be rather easy. This post shows you how to do it.

Onze laatste blogs over DevOps

10 tips voor het halen van je Certified Kubernetes Administrator examen (2023)

Challenges of implementing SRE in small to medium businesses

Kubernetes and Cloud Native Associate (KCNA)

Is your organization ready for SRE?

Met Ansible een Mirantis Container Engine cluster opzetten

Automate the renewal of TLS certificates with Vault and Cert-Manager

The power of GitLab Auto DevOps and Kubernetes

Sign up for our free webinar: the Benefits of CI/CD

Installing a Kubernetes cluster and connecting it to Lens

GitHub Actions vs Azure Pipelines

Learning Kubernetes with the Game of Pods