{"id":9655,"date":"2020-02-29T13:25:00","date_gmt":"2020-02-29T12:25:00","guid":{"rendered":"https:\/\/blog.mi.hdm-stuttgart.de\/?p=9655"},"modified":"2023-08-06T21:44:17","modified_gmt":"2023-08-06T19:44:17","slug":"image-editor-on-kubernetes-with-kompose-minikube-k3s-k3sup-and-helm-part-2","status":"publish","type":"post","link":"https:\/\/blog.mi.hdm-stuttgart.de\/index.php\/2020\/02\/29\/image-editor-on-kubernetes-with-kompose-minikube-k3s-k3sup-and-helm-part-2\/","title":{"rendered":"Kubernetes: from Zero to Hero with Kompose, Minikube, k3sup and Helm \u2014 Part 2: Hands-On"},"content":{"rendered":"

This is part two of our series on how we designed and implemented a scalable, highly-available and fault-tolerant microservice-based Image Editor. This part depicts how we went from a basic Docker Compose setup to running our application on our own \u00bbbare-metal\u00ab Kubernetes cluster.<\/p>\n

<\/p>\n

Part 1<\/a> of this series explains the design choices we’ve made to get here.<\/p>\n

The complete source code and configuration files along with setup instructions for our project can be found in GitLab<\/a>.<\/p>\n

\n

In this post we will dive into Kubernetes and Helm<\/a> \u2014 the<\/em> package manager for Kubernetes \u2014 and how we went from a basic microservice deployment with Docker Compose to a fully working and production-ready Kubernetes cluster with K3s<\/em> and k3sup<\/em>.<\/p>\n

Initially, while we were developing our various microservice components (see our previous blog post<\/a>), we decided to package each service to a Docker container. This went quite smooth as we already had some experience in working with Docker. We ended up with 11 Dockerfiles, one for every of our microservices: auth manager<\/a>, conversion manager<\/a>, the two converters (converter-deepdream<\/a> and converter-basic<\/a>), job manager<\/a>, the image storage<\/a> and storage service<\/a>, web frontend<\/a>, the TLS terminating proxy<\/a>, the request annotation proxy<\/a> and the API gateway<\/a>.<\/p>\n

A Dockerfile describes the steps required to build and package one or multiple apps into a Docker image. Docker images can be instantiated to a Docker container which then runs the application(s).<\/p>\n

FROM golang:alpine AS builder\n\nWORKDIR \/auth\n\nRUN apk add --no-cache make\n\nCOPY .\/config\/ .\/config\/\nCOPY .\/server\/ .\/server\/\nCOPY .\/vendor\/ .\/vendor\/\nCOPY .\/go.mod .\/go.mod\nCOPY .\/go.sum .\/go.sum\nCOPY .\/main.go .\/main.go\nCOPY .\/Makefile .\/Makefile\n\nRUN make\n\n# ---\n\nFROM alpine:latest\n\nWORKDIR \/auth\n\nRUN addgroup -g 1000 -S auth && \n    adduser -u 1000 -S auth -G auth\n\nCOPY --from=builder \/auth\/bin\/server .\n\nUSER auth\n\nCMD [ "\/auth\/server" ]<\/code><\/pre>\n
(Exemplary illustration of a Dockerfile<\/code> leveraging multi-stage builds<\/a>, taken from auth\/Dockerfile<\/a>)<\/i><\/p>\n
To orchestrate our various Docker containers we decided to use Docker Compose, a tool which amongst other features helps in making some capabilities of Docker available to configure via a simple docker-compose.yml<\/code> file, for example to launch containers in a specified order, define their service dependencies, manage port mappings and implement a restart policy.<\/p>\n
version: "3"\n\nservices:\n  auth:\n    build:\n      context: ${AUTH:-.}\n      dockerfile: Dockerfile\n    image: waa.edu.rndsh.it:5000\/auth:latest\n    restart: unless-stopped\n    ports:\n      - "8443:8080\/tcp"\n    env_file:\n      - ${AUTH_ENV:-.env}\n    cap_drop:\n      - ALL<\/code><\/pre>\n
(Exemplary illustration of a docker-compose.yml<\/code> file, taken from auth\/docker-compose.yml<\/a>)<\/i><\/p>\n
This was great for our development setup which allowed our developers to work independently of each other by running the entire application on their local development machine without any external dependencies.<\/p>\n
Once the backend of our application became usable with a web frontend, we decided it was time to make the app available on a public server so it could be used by our testers \u2014 finally it was time to explore Kubernetes!<\/p>\n
Diving into Kubernetes<\/h1>\n
At first, Kubernetes was quite intimidating to us when we only knew a bare minimum of what it can be used for. It was a beast we needed to tear apart and explore the parts each on their own.
The official Kubernetes documentation<\/a> and tutorials<\/a> already gave a good introduction to what Kubernetes does and doesn’t try to achieve.<\/p>\n
\n
To deploy our application on Kubernetes, we first needed to move from our Docker Compose files to the equivalent Kubernetes Resources<\/a>.<\/p>\n
Introducing Kompose, the official Docker Compose to Kubernetes converter<\/h2>\n
While exploring the repos of the Kubernetes GitHub organization<\/a>, we found a tool which helps in migrating existing Docker Compose files to Kubernetes: Kompose<\/a>. The procedure was quite straightforward:<\/p>\n
$ kompose convert --chart --controller=deployment -f docker-compose.yml<\/code><\/pre>\n
We ran this command for each of our 11 docker-compose.yml<\/code> files. Out came a total of around 30 Kubernetes resource files. Kompose did its job impressively well! The tool not only reliably converted the well-known Docker Compose directives image<\/code><\/a>, ports<\/code><\/a> and restart<\/code><\/a>, but also converted less frequently used directives such as env_file<\/code><\/a>, cap_drop<\/code><\/a> \/ cap_add<\/code><\/a> and even volumes<\/code><\/a> to the equivalent Kubernetes syntax \/ resource type counterpart. It was super helpful and gave a headstart into becoming familiar with resource files \u2014 the first step into mastering the Kubernetes-Fu.<\/p>\n
Unfortunately, some changes to the resource files as produced by Kompose were still required. The version of Kompose we were using wasn’t completely compatible with the then-recent release of Kubernetes which we used inside Minikube. We also dropped<\/a> all references to io.kompose<\/code> \u2014 they were no longer of any to us since we were already experienced enough to manage the files on our own.<\/p>\n
We needed to add a imagePullSecrets<\/code> policy to every Deployment<\/code> resource and configure the referenced Docker Registry secret as we are pulling the Docker images from our own login-protected registry:<\/p>\n
imagePullSecrets:\n- name: regcred<\/code><\/pre>\n
(Excerpt of auth\/kubernetes\/auth\/templates\/deployment.yaml<\/code><\/a>. The two lines instruct Kubernetes to use the regcred<\/code> secret to authenticate to the Docker Registry before pulling the image.)<\/i><\/p>\n
$ kubectl create secret docker-registry regcred \n  --docker-server=waa.edu.rndsh.it:5000 \n  --docker-username=$REGISTRY_USERNAME --docker-password=$REGISTRY_PASSWORD<\/code><\/pre>\n
(Configuring the regcred<\/code> secret via command line)<\/i><\/p>\n
Once the resource files were adjusted, it was time to test them \u2014 we needed a Kubernetes development cluster.<\/p>\n
Setting up a private Kubernetes development cluster with Minikube<\/h2>\n
Minikube probably is the<\/i> most widely used one-node Kubernetes cluster solution for development. And it became quite clear why: it was super easy to set up; With only a single command and a bit of time for the bootstrapping process we had a Kubernetes development cluster up and running on our local machines.<\/p>\n
# Create and launch Minikube VM\n$ minikube start --memory='10240mb' --cpus=8\n\ud83d\ude04 minikube v1.7.3 on Darwin\n\u2728 Automatically selected the virtualbox driver\n\ud83d\udcbf Downloading VM boot image ...\n\ud83d\udd25 Creating virtualbox VM (CPUs=8, Memory=10240MB, Disk=20000MB) ...\n\ud83d\udc33 Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...\n\ud83d\udcbe Downloading kubeadm v1.17.3\n\ud83d\udcbe Downloading kubelet v1.17.3\n\ud83d\udcbe Downloading kubectl v1.17.3\n\ud83d\ude80 Launching Kubernetes ...\n\ud83c\udf1f Enabling addons: default-storageclass, storage-provisioner\n\u231b Waiting for cluster to come online ...\n\ud83c\udfc4 Done! kubectl is now configured to use "minikube"<\/code><\/pre>\n
Minikube even automatically merged its kubeconfig<\/code> to ~\/.kube\/config<\/code> and switched the kubectl<\/code> context accordingly \u2014 we were ready to get started!<\/p>\n
# Check cluster version\n$ kubectl version --short\nClient Version: v1.17.3\nServer Version: v1.17.3\n\n# \u2026 success!<\/code><\/pre>\n
As a first step, to check whether our Kubernetes resource files were producing the desired state in our cluster, we kubectl apply<\/code>‘ed all of them as follows:<\/p>\n
$ find . -name '*.yaml' -not -name 'Chart.yaml' -not -name 'values*.yaml' \n  -exec cat {} ; -exec echo '---' ; | kubectl apply -f -\nconfigmap\/auth-env created\ndeployment.apps\/auth created\nservice\/auth created\ndeployment.apps\/conversion-manager created\nservice\/conversion-manager created\ndeployment.apps\/converter-basic created\nservice\/converter-basic created\nconfigmap\/converter-deepdream-env created\ndeployment.apps\/converter-deepdream created\nservice\/converter-deepdream created\ndeployment.apps\/frontend created\nservice\/frontend created\nconfigmap\/imagestorage-env created\ndeployment.apps\/imagestorage created\nservice\/imagestorage created\ndeployment.apps\/ingress-backend created\nservice\/ingress-backend created\ningress.networking.k8s.io\/ingress-frontend created\nconfigmap\/ingress-middleware-env created\ndeployment.apps\/ingress-middleware created\nservice\/ingress-middleware created\nconfigmap\/jobmanager-env created\ndeployment.apps\/jobmanager created\nservice\/jobmanager created\nconfigmap\/storageservice-env created\ndeployment.apps\/storageservice created\nservice\/storageservice created<\/code><\/pre>\n
We could see all our microservices spin up inside the cluster using kubectl get events --all-namespaces -w<\/code> \u2014 in real-time!
After a while, all deployments were READY<\/code> (verified with kubectl get all -o wide --all-namespaces<\/code>).<\/p>\n
To access the inside of the cluster from outside (our host machine), we needed to enable the Minikube ingress<\/code> addon via minikube addons enable ingress<\/code>. Minikube then made our application available to the world outside of the cluster.<\/p>\n
minikube ip<\/code> shows the IP of the Minikube cluster, and curl<\/code>ing it returned our web frontend (\ud83c\udf89). To our surprise, the app0<\/sup> was already working fine \u2014 just as on our Docker Compose-managed cluster:<\/p>\n
$ curl -Is $(minikube ip) | head -n1\nHTTP\/1.1 200 OK<\/code><\/pre>\n
It was time to extend the resource files with more awesome features.<\/p>\n
0<\/sup>: all services without additional dependencies such as a database<\/i><\/p>\n
Making the Kubernetes application highly-available<\/h2>\n
While skimming through the whole Kubernetes documentation<\/a>, we compiled a list of interesting directives we wanted to implement for our resource files. Amongst them were the readinessProbe<\/code><\/a> and livenessProbe<\/code><\/a> directives for Deployment<\/code> resources.<\/p>\n
By default, when the most basic unit of Kubernetes \u2014 a pod \u2014 has launched, Kubernetes will instantly begin to route traffic to it, even if the application running inside the pod is not ready to handle any traffic yet. This would make requests reaching such an “unhealthly” pod return a 502 Bad Gateway<\/code> response.<\/p>\n
That’s where readinessProbe<\/code> comes into play. It defines a check which needs to complete successfully before Kubernetes routes any traffic to the pod. For each of our deployments we created such a directive to instruct Kubernetes to either wait for a TCP port to become available or for a HTTP server to return a 2xx<\/code> response at a specified path:<\/p>\n
readinessProbe:\n  httpGet:\n    path: \/\n    port: 8080\n    periodSeconds: 2<\/code><\/pre>\n
(Exemplary configuration of a readinessProbe<\/code> directive commanding Kubernetes to wait for a HTTP server on port 8080 of the pod to become available and answer a request to \/<\/code> with a 2xx<\/code> response before beginning to route traffic to the pod. Taken from frontend\/kubernetes\/frontend\/templates\/deployment.yaml<\/a>)<\/i><\/p>\n
Similarly, livenessProbe<\/code>s are used to check if a pod is still<\/em> running correctly. Once a liveness check fails for a specific number of times (the failureThreshold<\/code>, default: 3, source<\/a>), Kubernetes automatically deletes (“restarts”) the pod and removes it from its internal service Load Balancer so traffic doesn’t reach it any longer.
Those two directives are configured in no more than 10 lines<\/a> \u2014 Kubernetes handles the rest of all the magic!<\/p>\n
Next, it was time to configure minimum and maximum hardware resource limits for our deployments. Setting those limits was super easy, too:<\/p>\n
resources:\n  requests:\n    memory: "256Mi"\n    cpu: "0.4"\n  limits:\n    memory: "512Mi"\n    cpu: "0.8"<\/code><\/pre>\n
(Exemplary configuration of a resources<\/code> directive, taken from converter-deepdream\/kubernetes\/converter-deepdream\/templates\/deployment.yaml<\/a>)<\/i><\/p>\n
requests<\/code> specifies the minimum hardware resources required to run a pod, while limits<\/code> configures the maximum amount of hardware resources available to a pod. The values were initially either chosen arbitrarily or by looking at the output of kubectl top pods<\/code> which gave a rough estimate of how the values needed to be configured.<\/p>\n
The memory limit can be expressed in various units (megabyte, gigabyte, etc., see here<\/a>), while the definition of the cpu<\/code> unit is a bit more complex:<\/p>\n
Quoting the documentation<\/a>, one “cpu”, in Kubernetes, is equivalent to:
– 1 AWS vCPU
– 1 GCP Core
– 1 Azure vCore
– 1 IBM vCPU
– 1 Hyperthread on a bare-metal Intel processor with Hyperthreading<\/p>\n
We played around a bit with various numbers and ended up configuring 0.1 \u2013 0.8 units of cpu<\/code> for our services.<\/p>\n
A scary discovery was that Kubernetes pods \u2014 just like Docker containers \u2014 are run with root privileges by default. We reconfigured our deployments accordingly to run the pods as non-root:<\/p>\n
securityContext:\n  capabilities:\n    drop:\n      - ALL\n  allowPrivilegeEscalation: false\n  runAsUser: 18443<\/code><\/pre>\n
(Excerpt of auth\/kubernetes\/auth\/templates\/deployment.yaml<\/code><\/a>, allowPrivilegeEscalation: false<\/code> and runAsUser: 18443<\/code> prevent running the pod with root privileges)<\/i><\/p>\n
As a last step, to make the number of pod replicas of a service scale up or down automatically based on their current load, we created a HorizontalPodAutoscaler<\/a> resource for every microservice:<\/p>\n
apiVersion: autoscaling\/v1\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: conversion-manager\nspec:\n  scaleTargetRef:\n    apiVersion: apps\/v1\n    kind: Deployment\n    name: conversion-manager\n  minReplicas: 2\n  maxReplicas: 8\n  targetCPUUtilizationPercentage: 50<\/code><\/pre>\n
(Exemplary illustration of a HorizontalPodAutoscaler<\/code> resource, taken from conversion-manager\/kubernetes\/conversion-manager\/templates\/hpa.yaml<\/code><\/a>)<\/i><\/p>\n
This basic file defines a minimum and maximum number of running replicas of a pod, and a CPU threshold (targetCPUUtilizationPercentage<\/code>) which tells Kubernetes when to launch a new, or terminate an existing pod instance.<\/p>\n
Now that the Kubernetes development cluster was running fine on a single node with Minikube, we needed a staging \/ production cluster with more than just one node.<\/p>\n
Entering bare-metal Kubernetes cluster solutions<\/h2>\n
There are a number of solutions available to obtain access to a production-grade Kubernetes cluster. Probably the<\/em> most basic way is to simply get a Managed Kubernetes cluster from Google Cloud’s Kubernetes Engine (GKE<\/a>), Amazon AWS’ Elastic Kubernetes Service (EKS<\/a>), Microsoft Azure’s Kubernetes Service (AKS<\/a>), IBM Cloud<\/a>, DigitalOcean<\/a> or one of the other countless providers<\/a>.<\/p>\n
Other solutions encompass:<\/p>\n