Continuous Integration


What is "Continuous Integration", also known as CI?

It's a common misunderstanding to say that running tests automatically when you commit code is "CI". There are even some services that confuse the issue that way, like Travis CI.

No, Continuous Integration is something much much simpler:

Continuous Integration is a way of working, where your code is merged into the common repository often.

That's it. Simple, right? You don't create long-lived branches where you work for weeks and then try to merge to the master branch. You either work on very short-lived branches, or you have a long-lived one but merge from it into the master branch often.

Some of the benefit of continuous integration are:

  • Because you merge often, you have fewer conflicts
  • Because you merge often, your code is delivered to users faster
  • Because everyone merges often, coordination is improved (or rather, uncoordination is noticed)
  • Because you have to merge at sensible points, you will break down your tasks better.
  • Because you see the changes in action incrementally, you will avoid going in the wrong direction (you will pivot faster)

The part where automatic test runners come in is that you need to be confident to be able to do continuous integration. If every time you merge into master everything breaks, then you are just going to be frustrated.

If there is more than one person working on the code and everytime someone else merges everything breaks, you are going to be pissed off.

And since frustration and anger are unpleasant, we should endeavour to make continuous integration easier, using both tools and a mindset that are conductive towards continuous integration.

In the previous chapters we:

Let's combine those things.

We will configure GitLab CI so that whenever we push code to the public repository things are tested automatically. This includes testging when pushing code to the master branch, thus validating the state of our future releases, and also when pushing code to other branches, thus validating whether they are in a state that allows merging into master.

By having those tests run automatically, if our test coverage is good, we will have confidence that the code works and will be enabled to merge frequently into master. And then we will have continuous integration.

While the details are different for every CI system, and surely even for Gitlab CI things will change over time, the basic concepts are constant, and hopefully what you learn here will transfer well to whatever CI system you use.

Setting up Gitlab CI

The core of Gitlab CI is that the nice folks at Gitlab are letting us run stuff in their computers. If we accept their generous offer, we can run our tests there every time we do a change. Since our code is already in their servers and they know when we push changes to our repository, we just need to let them know what to run. We do that using a .gitlab-ci.yml file.

Notice the dot at the beginning of .gitlab-ci.yml! That is important:

  • Marks the file as "hidden" in Linux and OS/X
  • This won't work if you don't put it there :-)

Explaining the syntax for YAML files is beyond the scope of this book, but see here

Here's our starter CI configuration, which installs things then runs tests:

 1# Use a system image with python 3.7 installed
 2image: python:3.7
 4  script:
 5  # Install things we need for tests to run
 6  - apt-get update -qy
 7  - apt-get install -y python3-pip
 8  # Install our project's dependencies
 9  - pip3 install -r requirements.txt
10  # Setup our project
11  - python3 develop
12  # Run tests
13  - pytest

As soon as you push this into your repository, Gitlab will start running a "Job". You can see them by clicking on "CI/CD ▶ Jobs" in your repo's Gitlab page.

List of jobs in Gitlab CI

If you click on "Running" it would show what is running in the job. In this case it will start showing all the commands we asked it to run:

 1Running with gitlab-runner 11.5.0 (3afdaba6)
 2  on docker-auto-scale ed2dce3a
 3Using Docker executor with image ruby:2.5 ...
 4Pulling docker image ruby:2.5 ...
 5Using docker image sha256:7834f5f61ba80e65515163209a3f952fcd1d11f9ce4420ba63d952e5b52b77e1 for ruby:2.5 ...
 6Running on runner-ed2dce3a-project-7040109-concurrent-0 via runner-ed2dce3a-srm-1545402551-ae9208e5...
 7Cloning repository...
 8Cloning into '/builds/ralsina/boxes'...
 9Checking out 5f4c7059 as add-ci...
10Skipping Git submodules setup
11$ apt-get update -qy
12Get:1 stretch/updates InRelease [94.3 kB]
13Ign:2 stretch InRelease
14Get:3 stretch-updates InRelease [91.0 kB]
15Get:4 stretch Release [118 kB]

But at the very end, there will be an error:

1$ pytest
2/bin/bash: line 76: pytest: command not found
3ERROR: Job failed: exit code 1

In the end, .gitlab-ci.yml is not much more sophisticated than a shell script, and it is debugged in the same painful way, running it and seeing what breaks. In this case: no pytest, so we add it.

 1# Use a system image with python 3.7 installed
 2image: python:3.7
 4  script:
 5  # Install things we need for tests to run
 6  - apt-get update -qy
 7  - apt-get install -y python3-pip
 8  # Install our project's dependencies
 9  - pip3 install -r requirements.txt pytest
10  # Setup our project
11  - python3 develop
12  # Run tests
13  - pytest

And success! Sort of! Here is what I see in my Gitlab job:

 1$ pytest
 2============================= test session starts ==============================
 3platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
 4rootdir: /builds/ralsina/boxes, inifile:
 5collected 20 items
 7tests/ ..                                       [ 10%]
 8tests/ ....                                              [ 30%]
 9tests/ ......                                         [ 60%]
10tests/ ......F                                        [ 95%]
11tests/ .                                                   [100%]
13=================================== FAILURES ===================================
14____________________________ test_justify_overfull _____________________________
16tmpdir = local('/tmp/pytest-of-root/pytest-0/test_justify_overfull0')
18    def test_justify_overfull(tmpdir):
19        """If a line is overfull, it still should be justified."""
21        separation = 0.05
22        page = boxes.Box(0, 0, 30, 50)
23        # Our failing text
24        inp = tmpdir.mkdir("sub").join("hello.txt")
25        inp.write(
26            "take delight in vexing me. You have no compassion for my poor nerves.\”"
27        )
28        # Adjust widths:
29        row = deque(boxes.create_text_boxes(inp))
30        # Put side by side
31        boxes.fill_row(row, page, separation)
33        # Should be overwide
34>       assert row[-1].x > page.w
35E       assert 0 > 30
36E        +  where 0 = Box(0, 0, 0, 1, "\n").x
37E        +  and   30 = Box(0, 0, 30, 50, "x").w
39tests/ AssertionError
40===================== 1 failed, 19 passed in 0.49 seconds ======================
41ERROR: Job failed: exit code 1

But ... when I run this locally, it works!

Debugging CI

Earlier in this book I accused other books of lying because "The code seems rehearsed, there are no errors, everything progresses monotonically towards a lovely pyramid of code with no false starts and no wrong assumptions." ... this whole subsection is about the dirt of real life, where most things don't freaking work the first, or second, or third time, and figuring out how to make them work is an awesome skill.

If the same code does different things in different "places" it's because there is a difference between those environments. I want this code to work in both my machine and CI's, so I need to find out what is different.

Let's go over possibilities!

It could be the versions of the python packages we have installed:

Let's change our CI so we can see what is actually installed.

1  # Print versions of python packages
2  - pip3 freeze

And here is the relevant output:

 1$ pip3 freeze

Let's compare it with my own system's, where tests pass:


It seems my system is full of random stuff. One way to "fix" this is to use a clean virtual environment and try to run the tests there like our CI system is doing.

After some experimentation, the test still passes locally.

Could it be the python version? Looks like it's not that either:

  • CI: python 3.7.1
  • Local: python 3.7.1

Could it be the version of HarfBuzz? To see that, we need to add a command in our CI configuration to show us the versions of the system packages.

1# Install things we need for tests to run
2  - apt-get update -qy
3  - apt-get install -y python3-pip
4  - dpkg -l

And here is the output for harfbuzz in the CI system:

ii  libharfbuzz0b:amd64                1.4.2-1

And in my system:

harfbuzz 2.2.0-1

So, the library that is doing all the text positioning and shaping is a totally different version. That looks like a promising source for this error.

Whatever Gitlab is using to run our CI tests is much older than my local system (FYI Gitlab is running Debian Stretch), and our code doesn't notice, and then behaves in a subtly different way with those older versions.

That is not good. So, here we have to choose between three paths.

  1. Change the CI system to match my own by instaling a newer Harfbuzz
  2. Change my own system to match CI by installing an older Harfbuzz
  3. Create a CI-like system where I can find what the hell is wrong, then fix it and make it work in both places with both versions.

Option 1 is easy, but it means there is going to be some people for whom the test will just fail. Option 2 is out of the question because I like my system. Option 3 is what we will do.

Running a CI-like system

One thing to notice: you may not ever need or even want to do this. But if you want to, here is how.

Gitlab CI runs our system using docker and so can we.

I am not going to explain how to get docker running in your system, see here for a guide. Assuming it is running, you can do this to get a system much like Gitlab CI's:

 1$ docker run -ti -v $PWD:/boxes -w /boxes --network=host python:3.7 bash
 2Unable to find image 'python:3.7' locally
 33.7: Pulling from library/python
 454f7e8ac135a: Pull complete
 5d6341e30912f: Pull complete
 6087a57faf949: Pull complete
 75d71636fb824: Pull complete
 80c1db9598990: Pull complete
 9bfb904e99f24: Pull complete
1078a3d3a96a32: Pull complete
11885a0ed92c89: Pull complete
12dd7cc9ace242: Pull complete
13Digest: sha256:3870d35b962a943df72d948580fc66ceaaee1c4fbd205930f32e0f0760eb1077
14Status: Downloaded newer image for python:3.7

What that command does is start a container, which is sort of a virtual machine, just like Gitlab CI does:

  • docker run ... run a container
  • -ti ... give me an interactive terminal
  • -v $PWD:/boxes ... make the current directory be /boxes in the container
  • -w /boxes ... start in /boxes
  • --network=host ... use the host network because that will work
  • bash ... and run a bash shell please.

Now, inside that image, we can do the same things Gitlab does!

 1root@mybox:/boxes# apt-get -qy update
 2Ign:1 stretch InRelease
 3Get:2 stretch/updates InRelease [94.3 kB]
 4Get:3 stretch-updates InRelease [91.0 kB]
 5Get:4 stretch Release [118 kB]
 6Get:5 stretch/updates/main amd64 Packages [463 kB]
 7Get:6 stretch-updates/main amd64 Packages [5152 B]
 8Get:7 stretch Release.gpg [2434 B]
 9Get:8 stretch/main amd64 Packages [7089 kB]
10Fetched 7409 kB in 14s (499 kB/s)
11Reading package lists...
12root@mybox:/boxes# apt-get install -y python3-pip
14[Output omitted]
16root@mybox:/boxes# pip3 install -r requirements.txt
18[Output omitted]
20root@mybox:/boxes# python3 develop
22[Output omitted]
24root@mybox:/boxes# pytest
25=================== test session starts ===================
26platform linux -- Python 3.7.1, pytest-3.5.0, py-1.7.0, pluggy-0.6.0
27rootdir: /boxes/src/part3/code/ci/boxes, inifile:
28collected 20 items
30tests/ ..                                                                                                                                                                       [ 10%]
31tests/ ....                                                                                                                                                                              [ 30%]
32tests/ ......                                                                                                                                                                         [ 60%]
33tests/ ......F                                                                                                                                                                        [ 95%]
34tests/ .                                                                                                                                                                                   [100%]
36========================= FAILURES =========================
37__________________ test_justify_overfull ___________________
39tmpdir = local('/tmp/pytest-of-root/pytest-0/test_justify_overfull0')
41    def test_justify_overfull(tmpdir):
42        """If a line is overfull, it still should be justified."""
44        separation = 0.05
45        page = boxes.Box(0, 0, 30, 50)
46        # Our failing text
47        inp = tmpdir.mkdir("sub").join("hello.txt")
48        inp.write(
49            "take delight in vexing me. You have no compassion for my poor nerves.\”"
50        )
51        # Adjust widths:
52        row = deque(boxes.create_text_boxes(inp))
53        # Put side by side
54        boxes.fill_row(row, page, separation)
56        # Should be overwide
57>       assert row[-1].x > page.w
58E       assert 0 > 30
59E        +  where 0 = Box(0, 0, 0, 1, "\n").x
60E        +  and   30 = Box(0, 0, 30, 50, "x").w
62tests/ AssertionError
63=========== 1 failed, 19 passed in 0.77 seconds ============

And threre you go, we have the same bug but in a place where we can see it.

Just for convenience, we can save the current state of the container:

1$ docker ps
2CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
35c5b9f7e2c9c        python:3.7          "bash"              12 minutes ago      Up 12 minutes                           nervous_bassi
4$ docker commit 5c5b9f7e2c9c boxes-ci

And now, whenever I use the boxes-ci image in my system it's this one that looks a lot like Gitlab CI with my stuff installed in it.

I will not go into the shameful details of the bug, but here is the change that fixes it.


In this chapter we setup automated testing for our changes via Gitlab CI. Now, every commit we push, every branch we create, will run our tests. Since we are confident that our tests are righteous, when we see beautiful green checkmarks next to our code we will know that our changes are good.

And since the changes are good, then why not merge them into master? And thus, CI is achieved.

results matching ""

    No results matching ""