In this blog entry we take a look at Travis CI, Jenkins, Gitlab CI and Buildbot and evaluate their benefits and downsides when trying to build a content heavy project with it (e.g. games).
Requirements
To identify our requirements we first take a look at our typical project structure and tools. Game projects usually are quite large in size. This size usually comes from the checked in media and regarding file size consists mostly of classical media file types such as texture, audio and video files. Depending on the size of the project those sum up somewhere in the range between 5 to 30 gigabytes of workspace usage. Due to this size our first requirement would be a persistent or cached repository on the build node. Otherwise the checkout process could easily dominate the entire build time.
For our second requirement we need to consider our target platform. For almost every game project (except iOS mobile games) the primary platform would be windows. Even if the game is developed for consoles, the tools for those are usually available for windows as development platform. In addition to windows we may need our build system to be able to run on mac, if targeting iOS or the mac itself as a target platform. For linux we usually cover the most mainstram used distribution regarding games, which currently would be Ubuntu, followed by SteamOS. All other potential platforms (Consoles, Android, Windows Phone, HTML5) would be built on one of the already mentioned platforms.
The last requirement comes from the used tools to build and deploy our game. Tools for compiling source code are widely supported in most build systems out of the box or are at least available as plugins. However when building a game project the smallest part of the actual build chain is compiling source code or running unit tests. Most of the time is spent building assets which are processed by custom tools. Those tools are either specific to the used underlying engine or are completely custom tools. As we have a high number of them in our build chain, the chosen system should provide an easy interface to not only extend the system via plugins but also allow calling of custom tools directly.
With those three requirements in mind we now can take a look at the different build systems and evaluate them.
Travis CI
Travis CI is the first continuous system we take a look at. It offers easy integration with GitHub and a very simple configuration file. The standard version of Travis CI starts a virtual machine with a predefined docker container on which you can install additional dependencies and then run our build. After the build has been completed the vm will be stopped and discarded. Here we do run into our first problem. As a result of this behavior our repository will be cloned from scratch each time a build is started, increasing build times artificially.
One can acquire an Enterprise Travis CI license to install the service on their own hardware. While this allows to use custom images to get rid of the dependency installations on each run, it is still a container based build system, resulting in the full clone of the repository on each build.
To address the elephant in the room with this ci system, there is currently no windows support, which inevitably makes it unusable for game development. As mentioned before most build tools for games reside on the windows platform. There is no way to circumvent this dependency.
Gitlab CI
When using Gitlab either as self hosted service or using gitlab.com you get access to Gitlab CI as it is part of the installation. Gitlab CI uses basically the same configuration as Travis CI which is a simple YAML file, including some parameters for the environment, setting up stages and commands to run in each stage. A very simple single stage and single step gitlab-ci.yml file could look like this:
build_Shipping:
stage: build
script:
- >
call "%GITLAB_SCHACHT_ENGINE_PATH%\Engine\Build\BatchFiles\Build.bat"
Schacht
Win64
Shipping
"%CI_PROJECT_DIR%\Schacht.uproject"
-waitmutex
stages:
- buildBuilds are run on runners which are supported on all desktop platforms. During the installation of a runner the underlying execution engine can be chosen between shell, docker, docker-ssh, docker-machine, kubernetes, virtualbox, parallels and ssh. The easiest to use and for our purposes most practical is the plain shell execution engine. This engine simply runs the commands on the local machine using the specified shell environment. Docker, docker-ssh and docker-machine basically provide the same functionality as Travis CI does and inherits the same problems we described before. Virtualbox and Parallels offer basically the same as the docker executors but on a vm basis.
The repository is cached in between builds and will be updated on a build request, which is our desired behavior. On each build we get a clean version of our repository.
A minor problem currently exists in the usage of more than one build step and stage. The environment is cleaned up after a complete build step and build artifacts are uploaded to the server. This means to transfer any data to the next step it needs to be marked as an artifact, uploaded to the server and then downloaded again. This may work for smaller projects where build artifacts are below several hundred of megabytes. For bigger temporary artifacts which are discarded after the complete pipeline is completed anyway, this is just a huge slowdown of the build process. The only workaround for now is to just pack everything into a single build step and only use a single stage. However be aware that this ci system is rapidly evolving as of now and this might change over time.
Jenkins
Jenkins is probably the most widely known ci system. It offers an easy GUI based setup of itself and projects to be run. It runs on all desktop platforms in a java vm. Part of this easy setup is that the server itself provides worker so no additional setup is required.
The classic way of building projects in Jenkins was to install a bunch of plugins for the used build and version control systems and then create the necessary steps in the GUI. This is very easy to get started and getting a feel for the build system. However, when starting to create different builds where some steps are shared this becomes not longer maintainable. A solution to this problem was using another bunch of plugins allowing to trigger other builds on Jenkins from the current build, therefore creating a tree like structure of small build projects in Jenkins. This structure made it possible to re-use the small projects in other projects and re-gain some maintainability. On the other side it produces a invisible net of dependencies in-between the different projects and one must ensure that all projects run on the same node, if necessary.
Probably with the uprising of Travis CI and Gitlab CI, Jenkins introduced the Pipelines plugin, which allows creating build configuration files just like the mentioned ci systems. With those it is no longer necessary to fiddle with the Jenkins GUI and everything for the build is placed in a single, versionable configuration file. As bonus, if the workflow of how to build our project changes, the jenkinsfile can be changed in the same series of commits without breaking old builds.
node
{
env.ZIP_NAME = ZIP_NAME
env.FILE_PATH = FILE_PATH
stage 'Upload'
withCredentials([[$class: 'FileBinding', credentialsId: 'JenkinsPrivateKey', variable: 'CRED_FILE']])
{
bat '''
scp -o StrictHostKeyChecking=no -i "%CRED_FILE%" %FILE_PATH% buildserver@example.com:/builds/%ZIP_NAME%
ssh -o StrictHostKeyChecking=no -i "%CRED_FILE%" buildserver@example.com chmod 664 /builds/%ZIP_NAME%'''
}
}When no parallel execution is enabled on a project the same workspace will be used for every build, updating the repository but leaving the state as it was left behind from the previous build, if desired. This is beneficial for our type of projects as all cached build artifacts stay and when using a smart build system only changed files are build.
As already mentioned we can create plugins for jenkins to streamline the usage of some tools aside from just calling those directly via the shell command line.
Buildbot
The last system we take a look at in our series is a little bit different. Buildbot describes itself as a framework instead of a ready-to-go application. There is no standalone installer to be run and no GUI for configuration. Instead we get a python package to install (e.g. via pip) and need to create a basic configuration file written in python to setup the server. This is clearly meant to be used for bigger projects as the initial setup can take quite a while as everything from the vcs source, scheduling, status interface, worker, build chains and assigned workers need to be configured before the first run.
The biggest benefit from buildbot is that build scripts are written in python and therefore basically all available modules for python are usable and having no real restrictions as it is a full-blown programming language. For further extensions plugins can be written to simplify and streamline the used tools.
Another slight derivation from other buildsystems comes from the fact that usually only one project is configured per buildbot master. It is possible to configure one master to build for multiple projects but they will be mixed on the status pages. To use buildbot for multiple projects it is possible to run multiple, interconnected masters, each for one project.
As a python based it basically runs everywhere python can be run. I relies heavily on the twisted engine for python which makes it a bit harder to install on windows. The main problem installing on windows is getting all the dependencies for buildbot up and running, but after the initial setup it behaves exactly the same as on any other platform.
The workers are part of the installed buildbot package and are configured with a simple configuration file. On the worker the repository is cloned and the same workspace is used in the state as left behind from the previous build from the same build chain.
from buildbot.plugins import *
from buildbot.plugins import steps
from buildbot.reporters import gitlab
from gitlab import setupGitLabHooks
def buildSchachtConfig(config, credentials):
config = setupGitLabHooks(config, credentials)
lfsUninstall = steps.ShellCommand(command="git lfs uninstall")
lfsPull = steps.ShellCommand(command="git lfs pull")
importSchachtEnginePath = steps.SetPropertiesFromEnv(
variables=["SCHACHT_ENGINE_PATH"]
)
gitCheckout = steps.Git(
repourl="https://example.com/schicht-im/schacht.git"
.format(**credentials),
mode='incremental'
)
baseCommandParams = {
"engine_path": util.Property("SCHACHT_ENGINE_PATH"),
"project_path": util.Interpolate(
"%(prop:builddir)s\\build\\Schacht.uproject"),
"engine_type": "Installed",
"build_platform": "Windows",
}
buildCookRunBaseParams = {
# Disable Compiling of BuildScripts
# as this does not work on Rocket or Installed builds
"compile": False,
"build": True,
"cook": True,
"clean": True,
}
buildCookRunBaseParams.update(baseCommandParams)
config['builders'] = []
builderNames = []
compileOnlyBuilderNames = []
windows_workers = ["windows-worker1", "windows-worker2"]
# Create Build builders
for target_config in ["Development", "Shipping"]:
buildFactory = util.BuildFactory()
buildFactory.addStep(lfsUninstall)
buildFactory.addStep(gitCheckout)
buildFactory.addStep(lfsPull)
buildFactory.addStep(importSchachtEnginePath)
buildFactory.addStep(
steps.UEBuild(
target="Schacht",
target_config=target_config,
**baseCommandParams
)
)
name = "Build_{0}".format(target_config)
config['builders'].append(
util.BuilderConfig(
name=name,
workernames=windows_workers,
factory=buildFactory
)
)
compileOnlyBuilderNames.append(name)
# Create build editor builders
for target_config in ["DebugGame", "Development"]:
buildFactory = util.BuildFactory()
buildFactory.addStep(lfsUninstall)
buildFactory.addStep(gitCheckout)
buildFactory.addStep(lfsPull)
buildFactory.addStep(importSchachtEnginePath)
buildFactory.addStep(
steps.UEBuild(
target="SchachtEditor",
target_config=target_config,
**baseCommandParams
)
)
name = "BuildEditor_{0}".format(target_config)
config['builders'].append(
util.BuilderConfig(
name=name,
workernames=windows_workers,
factory=buildFactory
)
)
compileOnlyBuilderNames.append(name)
# Create BuildCookRun builders
for target_config in ["Shipping"]:
buildFactory = util.BuildFactory()
buildFactory.addStep(lfsUninstall)
buildFactory.addStep(gitCheckout)
buildFactory.addStep(lfsPull)
buildFactory.addStep(importSchachtEnginePath)
buildFactory.addStep(
steps.UEBuild(
target="SchachtEditor",
target_config="Development",
**baseCommandParams
)
)
buildFactory.addStep(
steps.BuildCookRun(
target_config=target_config,
**buildCookRunBaseParams
)
)
name = "BuildCookRun_{0}".format(target_config)
config['builders'].append(
util.BuilderConfig(
name=name,
workernames=windows_workers,
factory=buildFactory
)
)
builderNames.append(name)
gitlabStatus = gitlab.GitLabStatusPush(
credentials["private_token"],
baseURL="https://example.com/",
verbose=True)
def isCPPFile(change):
for file in change.files:
if file.endswith(".cpp") or file.endswith(".h"):
return True
return False
allBuilders = []
allBuilders.extend(builderNames)
allBuilders.extend(compileOnlyBuilderNames)
config['services'] = []
config['services'].append(gitlabStatus)
config['schedulers'] = []
config['schedulers'].append(schedulers.AnyBranchScheduler(
name="compileOnly",
treeStableTimer=None,
fileIsImportant=isCPPFile,
onlyImportant=True,
builderNames=compileOnlyBuilderNames))
config['schedulers'].append(schedulers.AnyBranchScheduler(
name="cook",
treeStableTimer=None,
builderNames=builderNames))
config['schedulers'].append(schedulers.ForceScheduler(
name="force",
builderNames=allBuilders))
return config
Conclusion
Looking back we’ve taken a closer look at 4 quite different continuous integration/deployment systems and evaluated their benefits and drawbacks for our use case.
Travis CI is clearly unusable for our requirements as there is no windows support as of now and each build will first re-clone the complete repository, unnecessarily increasing the already pretty long build times.
Gitlab CI is the newest of the presented build systems and for our usage scenario a little bit rough around the edges. However if gitlab is used as repository server anyway it may be worth a look as it is one system less to maintain and update.
Jenkins recent addition of pipelines makes it a maintainable and sturdy buildsystem once again. Without the pipelines I would not recommend it anymore, but with them it is definitely a good choice.
Buildbot is a framework and requires some initial setup time. But in my opinion this time is well spent as it gets rewarded with almost full control over everything. Buildbot is definitely my choice for bigger projects with longer development times and more complex build steps.
Leave a Reply
You must be logged in to post a comment.