Autoscaling of Docker Containers in Google Kubernetes Engine

The name Kubernetes comes originally from the Greek word for helmsman. It is the person who steers a ship or boat. Representing a steering wheel, the Kubernetes logo was most likely inspired by it. [1] 

The choice of the name can also be interpreted to mean that Kubernetes (the helmsman) steers a ship that contains several containers (e.g. docker containers). It is therefore responsible for bringing these containers safely to their destination (to ensure that the journey goes smoothly) and for orchestrating them.

Apparently Kubernetes was called Seven of Nine within Google. The Star Trek fans under us should be familiar with this reference. Having 7 spikes, there might be a connection between the logo and this name. [2]

This blog post was created during the master lecture System Engineering and Management. In this lecture we deal with topics that are of interest to us and with which we would like to conduct experiments. We have already worked with docker containers very often and appreciate the advantages. Of course we have also worked with several containers within a closed system orchestrated by Docker-Compose configurations. Nevertheless, especially in connection with scaling and the big companies like Netflix or Amazon, you hear the buzzword Kubernetes and quickly find out that a distribution of a system to several nodes requires a platform such as Kubernetes.

Who is this blog post for? 

If you know what Docker Containers are and how they are built and used, the next step may be to create multiple instances of them and distribute the load. Of course this is possible with a Docker-Compose configuration, but the containers will only run on one computer at a time. So as soon as the computer reaches its physical limits, it is absolutely necessary to distribute the services and the instances thereof to several nodes. This is exactly what Kubernetes is suitable for, as it is used to orchestrate containers in a distributed system. In this blog post we will start exactly at this point and assume that you have basic experience with Docker and Kubernetes. The focus is on scaling containers within a Kubernetes cluster.

Of course all others who found their way to this blog post are also invited to read the blog post in order to learn more about Kubernetes, the autoscaling of services in docker containers plus the visualization of their utilization.

We will share and discuss our experiences about what we have built, what we have learned and difficulties we have encountered during our experiments. 

What is our goal?

First of all we just wanted to start with Kubernetes, experiment and build something with as little effort as possible and see results. But even the first question can be a challenge. Where to start? 

• Hosted Kubernetes
  Google GKE, Microsoft Azure, Amazon EKS, IBM Cloud Kubernetes Service, Apache Cloudstack and there are many more. 
• Test playground
  Magic Sandbox, Play with Kubernetes or Docker Desktop.

One can easily be overwhelmed by all these possibilities just to get started. Anyway, we wanted to expand our knowledge of Kubernetes and experiment a little. So we thought about a small use case. This should be kept as simple as possible so that it can be explained in a comprehensible way in this post.

Use Case

Imagine yourself being a backend developer or in a dev ops position and that you want to solve the following use case: 

Kubernetes Autoscaling

First you want to create a service (e.g. a simple hello world website) and containerize this service with Docker. Then you select a Hosted Kubernetes service of your choice and use it to provision and manage your cluster. Once it is set up and running, it should be able to scale based on CPU/memory usage or any custom metrics. Users requesting your service should then be redirected to the new services created by Kubernetes to balance the load.

For testing purposes and visual feedback it would be great to trigger/emulate high CPU/memory-usage to visually see (e.g. monitoring dashboard) the app scaling in both horizontal & vertical directions.

What are we not doing?

It won’t be a fully functioning production-ready step-by-step setup. We are just experimenting and the goal is to learn as much as possible.

Choice of Cloud Provider

As already mentioned, it can be difficult to choose the right provider. The choice of providers is huge and continues to increase. Among them are “playgrounds” like Magic Sandbox, Play with Kubernetes or Docker Desktop. These are however very limited in their functionality. Therefore, they were not considered for our experiment. To use Kubernetes without restrictions, we have to use “Hosted Kubernetes” providers, which does not make the choice easier. There are many platforms available, e.g.:

• Google Kubernetes Engine (GKE)
• Microsoft Azure Kubernetes Service (AKS)
• Amazon Elastic Kubernetes Service (EKS)
• IBM Cloud Kubernetes Service
• Alibaba Container Service for Kubernetes (ACK)
• DigitalOcean Kubernetes
• and many more…

Finally, we chose Google Kubernetes Engine because Kubernetes was originally developed by Google, so we assumed that if there were updates or new features, they would be directly supported by the GKE. Furthermore, we have worked a lot with other Google Cloud services like Firebase, Cloud Functions and Cloud Storage in the past. We hope that this will allow us to easily integrate and interact with Google products in the future. GKE delivers the Standard Metric API by default and does not need to be explicitly implemented. Google’s Kubernetes Engine can be tested over 12 months with a budget of $300, which we also took into consideration in our decision.

Scaling

In order to understand the scaling of Docker Containers within a Kubernetes cluster, different scaling options are presented first, which can be applied to various systems independently of Kubernetes. The Scale Cube is suitable for illustrating the different possibilities. [3, p. 8ff.]

X-Axis

A scale on the x-axis is called horizontal scaling.  This involves duplicating an existing application and distributing the requests to the available instances with the help of a load balancer. Figure 2 shows three instances of an application whose requests are distributed with the help of a load balancer.

Z-Axis

An extension of a system in Z-direction works similar to scaling along the x-axis, because the application is also duplicated. However, the load balancer does not forward the requests arbitrarily, but based on an attribute of the request such as a userId. Figure 3 also shows three instances of an application.  In contrast to x-axis scaling, the requests against these instances are forwarded by a load balancer based on the userId, which is known to the system through prior authentication of the user.

Y-Axis

If a system is functionally broken down into individual services, this is referred to as scaling in the y-direction. These services often have several instances, which also means scaling in the x-axis or the y-axis. Figure 4 shows a system scaled in x- y- and z-direction.

NestJS Application

Since we need at least one container, we decided to develop a small REST API with NestJS. We use NestJS as we have already gained experience with it. Basically any framework like ExpressJS (JavaScript), Flask (Python), Spring Boot (Java) etc. is suitable to create an easy (REST-)API.

To test the horizontal scaling by memory and CPU, we created two endpoints. The first endpoint tests the memory usage by appending a string to an array a few thousand times. The CPU test calculates roots in a loop to drive up the CPU load.

For scaling using custom metrics, we have created an endpoint that returns the static metric sem_metric with the value 5.

Monitoring

To visualize our monitoring data we used Prometheus and Grafana as mentioned in the previous chapter. Basically Prometheus is there to retrieve/collect and store the metrics. Grafana on the other hand retrieves the data from Prometheus and visualizes the metrics in a dashboard. Thereby the workloads and scales are displayed in a clear and simple way. [5, 6]

Prometheus belongs to the CNCF (Cloud Native Computing Foundation) Graduates and Grafana CNCF Silver-Member [7]. This was one reason for choosing these services.

Custom Scaler

To scale services in a more controlled way, scaling the Pods based on CPU and memory is usually not sufficient. Therefore Kubernetes offers a possibility to create own metrics with the help of the HPA. Afterwards it is possible to scale based on these metrics.

In order for the HPA to access own metrics, these must be exported to the Kubernetes Custom Metrics API. To achieve this, the Kubernetes API must be extended to include the Custom Metrics API. The Aggregation Layer is responsible for extending the Kubernetes API.

In our example, the metrics from the pods are already collected by Prometheus. To export the custom metrics, we use the Prometheus adapter from “directxman”, which we installed over Helm. Helm is package manager for Kubernetes. The installation via Helm has the advantage that the authorization of the adapter in the aggregation layer does not have to be set up manually.

Although the Prometheus adapter from Helm is already preconfigured, it must be adapted to our application. This is done by specifying the URL and port from which the metrics are to be obtained (see line 2 in gist below). We have also adapted the query to capture our metrics (line 8).

After the Prometheus adapter is installed, the custom metrics are collected. The following figure shows the process flow. Our applications (Nest JS Pods) provide metrics that are collected by Prometheus. With the help of the Prometheus Adapter our metrics are exported from Prometheus to the Custom Metrics API of Kubernetes. The Horizontal Pod Autoscaler accesses our metrics via this API and customizes the deployment. Afterwards,, the required number of pods of this deployment are provided.

To check if the custom metric is used, the command kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 can be used. If the setup is done correctly, the custom metric will appear in the output:

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
   ...
    {
      "name": "pods/sem_metric",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    ...
  ]
}

Finally, a new entry for your own metric must be made in the HPA YAML under the item metrics. As type we choose “Pods” (line 13 in the gist below), as we have assigned the metric to a Pod. As name we use sem_metric. Since the type is Pod, only AverageValue can be selected as target type (line 18). The average of all pods is used to calculate the scaling. The AvarageValue (line 19) is the value that should be reached. More details about the scaling algorithm can be found in the Kubernetes documentation.

After creating the HPA, the current status of the autoscaler can be retrieved:

There are two values under the item Targets. The left value is retrieved from the Pods using the Custom Metrics API, so it is the actual value. The right value defines the state which is declared in the HPA YAML file and therefore expected.

Problems that were noticed

While experimenting especially with the CPU and memory metrics and our NestJS applications that are designed to provoke exactly these loads, we noticed when testing the memory usage that the node server shoots itself off after a certain amount of time. This made testing a bit more difficult, but at this point you can clearly see that Kubernetes simply restarts the pod after it was no longer available.

Experimenting sometimes felt like a black box, because there was a lot of trial and error and the monitoring was very slow. We were never able to see exactly at what point it was scaling, because the visualization of the data only showed a change after an indefinite time. Sometimes faster and sometimes slower than usual.

Even though we were able to take a lot with us, we did not reach a point within this semester where we felt very confident so that we could use Kubernetes in real world projects.