Setting up a CI/CD pipeline in Gitlab

Introduction

For all my university software projects, I use the HdM Gitlab instance for version control. But Gitlab offers much more such as easy and good ways to operate a pipeline. In this article, I will show how we can use the CI/CD functionality in a university project to perform automated testing and an automated build process.

Prerequisites

Before we start, we need to set up an initial environment. Since we have already a Gitlab repository on the HdM Instance. We only need a Gitlab runner and a container registry to perform our task. The Runner will do all the “work” we define in the pipeline and the registry will store the built images.

INFO: The public Gitlab Instance has shared runners and a registry for each repository. On the HdM Instance, the registry is disabled and the shared runners have restricted rules. So we need to set them up ourself.

For the runner, we use an amazon ec2 instance and as registry, we use docker hub. If you like to have the same setup you need to have the following accounts:

• AWS Account (Eduction Account)
• Docker hub Account

Prepare Gitlab

The first step to create a pipeline is to enable the CI/CD features in Gitlab.

You can enable the feature under settings > general > Visibility, project features, permissions > pipelines.

A Pipeline in Gitlab is configured with a YAML file. The file called .gitlab-ci.yml. The YAML file defines the structure and determines what the Gitlab runner should execute.

Setup the runner

As mentioned before I use an AWS ec2 Instance for the runner. But you can also use any other VM or computer you like because we will use the runner inside docker.

Launch an ec2 instance and install docker. Amazon provides an easy setup for that.

Make sure docker is installed correctly with the hello-world image.

docker run hello-world

In order to configure the runner, a folder and other resources must be mounted.

docker run -d --name gitlab-runner --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

Now we can connect the runner to our repository. Therefore we must run the register command of the runner.

docker run --rm -t -i -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

Enter the Hdm Gitlab Instance URL.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
https://gitlab.mi.hdm-stuttgart.de 

Add your Runner Token. (You can find the token under settings > CI/CD > runners)

Please enter the gitlab-ci token for this runner
xxx

Add Tags and Description.

Please enter the gitlab-ci description for this runner
[hostname] gitlab-runner

Please enter the gitlab-ci tags for this runner (comma separated):
a-tag,another-tag

Enter the runner executor. (Use docker as executor)

Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell:
docker

Choose the default docker image for the docker executor. This image will be used if no one is defined in the YAML.

Please enter the Docker image (eg. ruby:2.1): 
docker:stable

Now you should see the runner online under the settings page.

Create a Pipeline

To create the pipline you could either create a file called .gitlab-ci.yml or you can press the “Set up CI/CD” Button under the Project dashboard.

Now let’s define the Pipeline. You can edit the YAML file directly in Gitlab.

First, we define our stages. For the tests, we will use 2 stages called test-api and test-web. The two stages run sequentially. The former one cover tests for a backend written in Go and the latter one for a vue.js frontend.

stages:
  - test-api
  - test-web
  - build
  - push

First, we define a base image inside the job with the image keyword. Since I use Go the image in the first job is golang:alpine. In the second stage, I use a node:alpine image.

In the script part, we need to prepare the environment for the tests. You could either make a docker image with all dependencies installed or install it during the job. After that, we only need to call the test command.

The tests in the first job are designed to be tested against a functional application with database access. With the services keyword, we can run a docker image during a job. Access to the running container is as usual with hostname container name and port.

test-api:
  image: golang:alpine
  stage: test-api
  services: 
    - redis:latest
  script:
    - apk update && apk add --no-cache git
    - cd api && go mod download
    - CGO_ENABLED=0 go test

test-web:
  image: node:alpine
  stage: test-web
  script: 
    - cd web && npm install 
    - npm test

Now commit the changes and watch if the tests run correctly. Under CI/CD Pipelines you can see the process.

If a stage has an error the whole pipeline will fail and all stages after will not run. So the next stage will only run if the test stages succeed.

Now we define the build stage.

This stage is very simple and good to reuse. Let’s break it down.

build-api:
  stage: build
  script:
    - docker info
    - echo "$REGISTRY_PWD" | docker login -u "$REGISTRY_USER" --password-stdin
    - docker pull "$REGISTRY_API":latest || true 
    - docker build --cache-from "$REGISTRY_API":latest -t "$REGISTRY_API":"$CI_COMMIT_SHA" ./api
    - docker push "$REGISTRY_API":"$CI_COMMIT_SHA"

The first step in the script part is to call docker info just, to reconstruct the build if anything fails. After that we login to docker hub with our credentials. For that, we use the environment variables in Gitlab. You can enter environment variables under settings > CI/CD > variables.

NOTE: Make sure to protect the variables and the branch you use. Otherwise, other members can see your password.

To build a docker image in the pipeline we need the docker in docker (dind) service.

services:   
   - docker:dind 

We need to modify the runner configuration a bit to do that.

Open the mounted config file from the runner and set privileged to true.

NOTE: Docker in privileged mode disables all of the security mechanisms of containers. https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

Now we can build the images. You can pull your latest image first to use it as cache to speed up your build. Tag it with the predefined environment variable CI_COMMIT_SHA to identify the image for later processing. Afterward, you can push the image to the registry.

For the web build, you can copy the whole job and change only the environment variables and point to the correct dockerfile.

The last stage is the push stage. Here we only pull the image from the build stage and tag it again with meaningful tags. So the first part is the same as the build. After that, we tag it again with latest you can add other tags such as version numbers. This stage should be done only in the master branch. We can set the Git strategy to none because we don’t need to copy the code from the repository.

push-api:
  stage: push
  variables:
    GIT_STRATEGY: none
  script:
  - echo "$REGISTRY_PWD" | docker login -u "$REGISTRY_USER" --password-stdin
  - docker pull "$REGISTRY_API":"$CI_COMMIT_SHA"
  - docker tag "$REGISTRY_API":"$CI_COMMIT_SHA" "$REGISTRY_API":latest
  - docker push "$REGISTRY_API":latest
  only:
    - master

To complete your CI/CD process you can use a webhook to auto-deploy your application to your production server. You can find a very simple node.js webhook under my Gitlab account. The complete project is also available on Gitlab.