KISS, DRY ‘n SOLID — Yet another Kubernetes System built with Ansible and observed with Metrics Server on arm64

“Roles are units of organization in Ansible. Assigning a role to a group of hosts (or a set of groups, or host patterns, and so on) implies that they should implement a specific behavior. A role may include applying certain variable values, certain tasks, and certain handlers – or just one or more of these things. Because of the file structure associated with a role, roles become redistributable units that allow you to share behavior among playbooks – or even with other users.”[11] What has already been mentioned is that roles have a specific structure. Ansible can easily initialize this structure with the following command: ansibile-galaxy init kubernetes.

../roles/kubernetes/
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── README.md
├── tasks<span style="color: silver">
│   ├── config-helm.yml
│   ├── config-kubernetes-dashboard.yml
│   ├── config-master-create-token.yml
│   ├── config-master.yml
│   ├── config-metrics-server.yml
│   ├── config-nginx.yml
│   ├── config-worker.yml
│   ├── install.yml</span>
│   ├── main.yml<span style="color: silver">
│   └── uninstall.yml</span>
├── templates<span style="color: silver">
│   ├── kubernetes-dashboard-service-nodeport.yml.j2
│   ├── kubernetes-dashboard.yml.j2
│   ├── metrics-server.yml.j2
│   ├── nginx_values.yml.j2</span>
├── tests
│   ├── inventory
│   └── test.yml
└── vars
    └── main.yml

There’s no magic; with the init command, all files are empty 😉 Let’s fill them with some tasks. For this purpose, Ansible uses Yet Another Modeling Language (YAML) Syntax to describe; the task/main.yml is mainly used to install or uninstall Kubernetes:

---
# role handling for special inventory
- fail:
    msg: "Please provide role_action=[install|uninstall|config-master|config-worker|config-nginx|config-metrics-server|config-kubernetes-dashboard]"
  when: role_action is not defined

# tasks file to install kubernetes
- include: install.yml
  when: role_action == 'install'

# tasks to configure master
- include: config-master.yml
  when: role_action in ['install', 'config-master'] and ansible_hostname in groups['master']

# tasks to add/configure worker
- include: config-worker.yml
  when: role_action in ['install', 'config-worker'] and ansible_hostname in groups['worker']

- include: config-nginx.yml
  when: role_action in ['install', 'config-master', 'config-nginx'] and ansible_hostname in groups['master']

# tasks to add/configure metrics incl dashboard
- include: config-metrics-server.yml
  when: role_action in ['install', 'config-master', 'config-metrics-server'] and ansible_hostname in groups['master']

- include: config-kubernetes-dashboard.yml
  when: role_action in ['install', 'config-master', 'config-kubernetes-dashboard'] and ansible_hostname in groups['master']

# uninstall
- include: uninstall.yml
  when: role_action == 'uninstall'

The configuration goes along with the installation process, but could be also triggered by using the role_action config option. The conditional is used to reduce the number of roles, to differ between master and worker, to keep the file structure simple and clean, and to keep the option to execute individual configuration files anyway. [13]

What Kubernetes is and what it does is not covered here. Click here, to get more information about Kubernetes. Furthermore, to focus more on Ansible, individual scripts such as Docker installation are shortened or partially omitted. To install Kubernetes via Ansible on the cluster, we first create a new file called task/install.yml with the following procedure:

---
- name: get system architecture
  shell: dpkg --print-architecture
  register: sys_architecture

- name: check if Kubernetes already exist
  stat:
    path: /usr/bin/kubeadm
  register: package_exist

- name: remove swapfile from /etc/fstab
  mount:
    name: "{{ item }}"
    fstype: swap
    state: absent
  with_items:
    - swap
    - none

- name: disable swap
  command: swapoff -a
  when: ansible_swaptotal_mb > 0

- name: set kernelmodule for bridge interface
  command: "{{ item }}"
  with_items:
    - modprobe bridge
    - modprobe br_netfilter
  when: not package_exist.stat.exists

- name: add kernelparameter for bridge interface
  sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    state: present
  with_items:
    - { name: 'vm.swappiness', value: '0' }
    - { name: 'net.bridge.bridge-nf-call-iptables', value: '1' }
  when: not package_exist.stat.exists

- name: add apt Kubernetes signing key for Ubuntu
  apt_key:
    url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
    state: present
  when: not package_exist.stat.exists

- name: add Kubernetes repository
  apt_repository:
    repo: deb [arch={{ sys_architecture.stdout }}] https://packages.cloud.google.com/apt kubernetes-xenial main
    state: present
    filename: kubernetes
  when: not package_exist.stat.exists

- name: install Kubernetes binaries
  apt: 
    name: "{{ packages }}"
    state: present
    update_cache: yes
  vars:
    packages:
      - kubelet 
      - kubeadm 
      - kubectl
  when: not package_exist.stat.exists
...

Ansible has its own modules[14] ranging from packaging over network configurations to third party hooks. First of all, it’s important to know what kind of system architecture is used to install Kubernetes. To avoid a reinstallation, a query will check if Kubernetes is already installed. After that, some tasks are defined that are necessary for Kubernetes, e.g., disable the paging memory (swap). After the pre-required tasks, more tasks like adding the repository, install Kubernetes and restart the service daemon is defined. Ansible uses YAML, and YAML uses the mustache syntax {{ var }} for the data binding, so the variables in the curly braces are placeholders. [15]

Alrighty, just a few lines to install Kubernetes on multiple Ubuntu instances. It’s also possible to refactor this to support different Linux derivatives. The other file, task/uninstall.yml, works vice versa. An excerpt to uninstall Kubernetes could look like this:

...
- name: force reset kubernetes cluster
  command: "kubeadm reset -f"
...
- name: remove Kubernetes binaries
  apt: 
    name: "{{ packages }}"
    state: absent
    update_cache: yes
  vars:
    packages:
      - kubelet 
      - kubeadm 
      - kubectl
  when: not package_exist.stat.exists
...

In the first block, we see that the kubeadm command is executed. This causes the services to be stopped gracefully and further that symlinks, directories and config files are removed. The second block finally removes with state: absent all packages related to the APT (Advanced Package Tool) repository.

- hosts: cluster
  roles:
  - docker
  tags:
  - docker

- hosts: cluster
  roles:
  - kubernetes
  tags:
  - kubernetes

We start with the latest version at this point (v.3.5.2) of the Helm installation on the master, set up the Kubernetes directory, install Calico, a pod networking plugin, and create a token to connect the worker to the master. So, the configuration file tasks/config-master.yml looks like this:

---
- name: include config file to install Helm on master
  include_tasks: config-helm.yml

- name: initialize the Kubernetes cluster using kubeadm
  command: kubeadm init --apiserver-advertise-address={{ kube_api_addr }} --pod-network-cidr={{ kube_pod_cidr }} --token "{{ kube_token }}" --token-ttl "{{ kube_token_ttl }}"

- name: setup kubeconfig directory
  command: "{{ item }}"
  with_items:
   - mkdir -p ~/.kube
   - ln -s /etc/kubernetes/admin.conf ~/.kube/config

- name: generate Kubernetes token
  command: kubeadm token create
  register: "{{ kube_token }}"

...

- name: set kernelmodule for pod network 
  command: "{{ item }}"
  with_items:
    - modprobe ip_set
    - modprobe xt_set
  when: not package_exist.stat.exists

- name: install Calico pod network
  command: kubectl create -f https://docs.projectcalico.org/archive/v3.18/manifests/calico.yaml

At the top, the Kubernetes control plane will be initialized with some basic options. Kubernetes executes several tasks in the background, such as a pre-flight check, generating a self-signed CA or creating an admin.conf file, et cetera. [21]

---
role_action: install
kube_api_addr: 10.0.100.10
kube_pod_cidr: 172.16.0.0/16
kube_token: "dieswi.rdueb3r5chr1eben"
kube_token_ttl: "1h2m3s"

Back to the Kubernetes configuration, where the next step is to create a folder in the home directory with a symlink to the previously created admin.conf file. After the operation, the Kubernetes token is generated and held temporarily in a variable, as tokens are constantly regenerated. The last step is to install Calico. This is necessary so that, for instance, individual pods are assigned to an address to provide or exchange data over the internal decoupled network, managed via Container Network Interface (CNI).[23] Wait for a second, is there a special reason to use Calico instead of another pod network like Flannel? It depends on the specific purpose. In this case, it should cost nothing, the performance should be right, the resources consumption should be low, and it should also offer security features!

...
- name: add helm repository
  apt_repository:
    repo: "deb [arch={{ sys_architecture.stdout }}] https://baltocdn.com/helm/stable/debian/ all main"
    state: present
    filename: helm
  when: not package_exist.stat.exists
...
- name: add bitnami helm repository
  command: helm repo add bitnami https://charts.bitnami.com/bitnami
...

...
- name: create nginx config
  template:
    src: templates/nginx_values.yml.j2
    dest: "~/nginx_values.yml"
    owner: root
    group: root
    mode: 0644
...
- name: install nginx
  command: helm install nginx bitnami/nginx --values nginx_values.yml
...

An excerpt of the nginx_values.yml looks like this:

image:
  registry: docker.io<b>
  repository: arm64v8/nginx </b>
  tag: 1.19.7
  pullPolicy: IfNotPresent
  pullSecrets: []
  debug: false
…
service:
  type: NodePort
  port: 80
  httpsPort: 443
  nodePorts: <b>
    http: "30080"</b>
    https: ""
  targetPort:
    http: http
    https: https
  annotations: {}
  externalTrafficPolicy: Cluster
…

To authenticate the worker with the Kubernetes control plane, the command kubeadm join is used in combination with some options and variables:

- name: join worker "{{ ansible_hostname }}" to cluster
  shell: |
    kubeadm join --token "{{ kube_token }}" \
                 --discovery-token-unsafe-skip-ca-verification \
                 "{{ kube_api_addr }}":6443

- name: create metrics-server config
  template:
    src: templates/metrics-server.yml.j2
    dest: "~/metrics-server.yml"
    owner: root
    group: root
    mode: 0644

- name: install metrics-server
  command: kubectl apply -f ~/metrics-server.yml

$ kubectl top pod --namespace default
NAME                     		CPU(cores)   	MEMORY(bytes)   
nginx-657f9bdb8b-6cm55   	1m           	2Mi

---
- name: create kubernetes-dashboard config
  template:
    src: templates/kubernetes-dashboard.yml.j2
    dest: "~/kubernetes-dashboard.yml"
    owner: root
    group: root
    mode: 0644

- name: create kubernetes-dashboard config
  template:
    src: templates/kubernetes-dashboard-service-nodeport.yml.j2
    dest: "~/kubernetes-dashboard-service-nodeport.yml"
    owner: root
    group: root
    mode: 0644

- name: install kubernetes-dashboard
  command: kubectl apply -f ~/kubernetes-dashboard.yml

- name: install kubernetes-dashboard service nodeport
  command: kubectl apply -f ~/kubernetes-dashboard-service-nodeport.yml
...

For the Metrics Server and the Dashboard scraper, a new Ansible module template is used. The benefit of this module is that a task could use a customized configuration file. Roundabout, the content of the template files are not of interest at this point.

Nothing is installed and configured yet, so let’s provide it with: ansible-playbook -t kubernetes -e role_action=install uls-playbook.yml. After a few minutes of pulling multiple applications and containers and making settings, we can see in the Ansible output that everything went well.

A small check directly on the worker with kubectl -n kube-system get pods shows us that Calico is up and running and thus configured.

This also applies for kubectl get services: