Jenkbird – Quality & Testing – Part 4

Love and testing can be cruel.

– Bert on the difficulties in life

Quality and Testing – one of the most discussed and valuable topics software engineering has to offer. This blog post will cover all the relevant stuff related to quality and testing in regard to Continuous Integration and Jenkins. We will show you in detail, how you can automate your testing with Jenkins to ensure best possible software quality.

99% != 100%

We want to start this blog series with a thought-experiment given in the book „Continuous Integration“ by Paul M. Duvall [1]:
Imagine you have a software with three components and the reliability of these software components is 90% each (regardless on how this is measured). At the first glance you might say that the overall reliability of the software is 90% – but that’s wrong. Actually the reliability is 0,99^3=0.729 and therefore only 72,9%.

Now imagine having 100 components with 99% reliability, that’s 0,99^100=0,366 which equals a reliability of 36,6% .

What would you do with thinks like a bridge with 72,9% reliability? You probably would not consider using it… except you are one of those Red Bull guys. And that’s why we need to ensure quality in every little component, service or whatever we develop for our system. We want to dedicate this blog post to Quality Gates or Continuous Testing, because it’s an important part of Continuous Integration and the Agile Methodology. Why would someone develop agile without using the faster feedback to fix the bugs? That’s what it’s all about, right? Let’s test!

Talking about Quality and Testing in CI…

At first we want to mention the two most important points when talking about CI and Testing:

1. Integrate code often and let the build test itself as well as
2. do not test on the production environment, but on a clone (like TEST or DEV)

The first argument has the obvious advantage of automating quality assurance. The more often you deploy and test your code, the faster you can react to failures.  How often have you heard about „breaking down complex tasks into manageable junks“? It’s better to fix a few bugs immediately than trying to restructure a whole system some time later, isn’t it? Jeff Sutherland, the „father“ of Scrum and author of the Book „Scrum: The Art of Doing Twice the Work in Half the Time“ gives an example: statistics of a project by Palm a few year ago state, that fixing bugs on the same day is 24 times faster than fixing them three weeks later [2] (in our opinion one of the best books about SCRUM – very entertaining read about the history, benefits and other areas of use for SCRUM e.g. a pre-school). A bugfix which would take one hour on the same day will take three complete work days according to this statistic – and we think it’s way cooler to write some new stuff in the additional 23 hours you have than fixing something just to get the system back to work.

„Testing is for wimps, real men and women test in production… Oh, well…“ – it’s pretty much O.K. if it’s only your student project environment you just crashed, but a customer probably wouldn’t be satisfied. We don’t want to step any further in this topic, because we already talked about it in part 2 of our blog series, anyway: use different stages to improve quality! It is definitely a good habit  to ensure that you always have a stable version of your code.

Automating tests with Jenkins

Now that we have covered some basic theories of testing, we will create our first job to test our application! As we are using a small Laravel project as an exemplary project for deployment, we are going to create a PHPUnit test and execute it with our beloved Jenkins.

First we need to get PHPUnit. You have two possibilities to do so. Either you install it globally on your instance or you install it locally within your Laravel application. We are going to show you the second option.  You need to edit your composer.json file which is located at the root folder of your application. Insert the following line:

"require-dev": {
    ...
    "phpunit/phpunit": "5.0.*"
},

Now that you have listed PHPUnit in your dependency configuration you simply need to update composer and you’ve successfully installed PHPUnit. Now we get to the actual test. As you can see in the repository we created a basic test case:

class ExampleTest extends TestCase
{
    /**
     * A basic functional test example.
     *
     * @return void
     */
    public function testBasicExample()
    {
        $this->visit('/')
             ->see('Sesame Deployment Street');
    
    }  
}

It basically is a functional test which visits our application site and checks if the pattern “Sesame Deployment Street” matches any string within the page. If this is the case the test passes, otherwise it will fail. We want this test to be executed every time we deploy a version of our application to the server. For this we need to configure a new Job in Jenkins. Add a new item to your Jenkins as you’ve already seen in the previous post. In this job we act on the assumption that the application code is already in place and we can test it.

Just add a shell script to the job that switches to the directory which contains our code and executes PHPUnit. This should be enough for our small example.

#!/bin/sh
cd /var/www/html
vendor/bin/phpunit  --log-junit report.xml

As we are really lazy and do not want to manually click on a button to test our application, we schedule the test job every time one of our deploy jobs has been run. In order to do this we configure the test as a post action for the other build jobs. To do so select deploy_demo_branch and deploy_dev_branch
as Projects to watch under Build Triggers. Hit the save button and now every time one of the other builds is successfully completed Jenkins will run our test.

On our dev branch we manipulated our page to fail the test whereas in our demo branch the test should pass. You can play around with the deploy jobs and watch how the test job is automatically triggered. Feel like a super professional system administrator yet?

What’s coming up in the next episode?

Sadly our journey into the world of continuous integration and deployment comes to an end. The next post will be our last one, giving you a wrap up about everything we have discussed so far and presenting  some alternatives. See you soon!

Blogpost by Thomas Derleth (td036), Jörg Einfeldt (je051), Marc Stauffer (ms445)

[1] Continuous Integration: Improving Software Quality and Reducing Risk, Paul M.Duvall, 2007.
[2] Scrum: The Art of Doing Twice the Work in Half the Time, Jeff Sutherland, 2014.

Read more about Testing in our Tutorial part 5