Continuous Integration
Intro
What is "Continuous Integration", also known as CI?
It's a common misunderstanding to say that running tests automatically when you commit code is "CI". There are even some services that confuse the issue that way, like Travis CI.
No, Continuous Integration is something much much simpler:
Continuous Integration is a way of working, where your code is merged into the common repository often.
That's it. Simple, right? You don't create long-lived branches where you work for weeks and then try to merge to the master branch. You either work on very short-lived branches, or you have a long-lived one but merge from it into the master branch often.
Some of the benefit of continuous integration are:
- Because you merge often, you have fewer conflicts
- Because you merge often, your code is delivered to users faster
- Because everyone merges often, coordination is improved (or rather, uncoordination is noticed)
- Because you have to merge at sensible points, you will break down your tasks better.
- Because you see the changes in action incrementally, you will avoid going in the wrong direction (you will pivot faster)
The part where automatic test runners come in is that you need to be confident to be able to do continuous integration. If every time you merge into master everything breaks, then you are just going to be frustrated.
If there is more than one person working on the code and everytime someone else merges everything breaks, you are going to be pissed off.
And since frustration and anger are unpleasant, we should endeavour to make continuous integration easier, using both tools and a mindset that are conductive towards continuous integration.
In the previous chapters we:
- Learned how to add tests to our code
- Learned a little about how to use git
- Learned how to share things in gitlab / github
- Packaged our app "The Right Way"
Let's combine those things.
We will configure GitLab CI so that whenever we push code to the public repository things are tested automatically. This includes testging when pushing code to the master branch, thus validating the state of our future releases, and also when pushing code to other branches, thus validating whether they are in a state that allows merging into master.
By having those tests run automatically, if our test coverage is good, we will have confidence that the code works and will be enabled to merge frequently into master. And then we will have continuous integration.
While the details are different for every CI system, and surely even for Gitlab CI things will change over time, the basic concepts are constant, and hopefully what you learn here will transfer well to whatever CI system you use.
Setting up Gitlab CI
The core of Gitlab CI is that the nice folks at Gitlab are letting us run stuff
in their computers. If we accept their generous offer, we can run our tests there
every time we do a change. Since our code is already in their servers and they
know when we push changes to our repository, we just need to let them know what
to run. We do that using a .gitlab-ci.yml file.
Notice the dot at the beginning of
.gitlab-ci.yml! That is important:
- Marks the file as "hidden" in Linux and OS/X
- This won't work if you don't put it there :-)
Explaining the syntax for YAML files is beyond the scope of this book, but see
here
Here's our starter CI configuration, which installs things then runs tests:
1# Use a system image with python 3.7 installed
2image: python:3.7
3test:
4 script:
5 # Install things we need for tests to run
6 - apt-get update -qy
7 - apt-get install -y python3-pip
8 # Install our project's dependencies
9 - pip3 install -r requirements.txt
10 # Setup our project
11 - python3 setup.py develop
12 # Run tests
13 - pytest
As soon as you push this into your repository, Gitlab will start running a "Job". You can see them by clicking on "CI/CD ▶ Jobs" in your repo's Gitlab page.
If you click on "Running" it would show what is running in the job. In this case it will start showing all the commands we asked it to run:
1Running with gitlab-runner 11.5.0 (3afdaba6)
2 on docker-auto-scale ed2dce3a
3Using Docker executor with image ruby:2.5 ...
4Pulling docker image ruby:2.5 ...
5Using docker image sha256:7834f5f61ba80e65515163209a3f952fcd1d11f9ce4420ba63d952e5b52b77e1 for ruby:2.5 ...
6Running on runner-ed2dce3a-project-7040109-concurrent-0 via runner-ed2dce3a-srm-1545402551-ae9208e5...
7Cloning repository...
8Cloning into '/builds/ralsina/boxes'...
9Checking out 5f4c7059 as add-ci...
10Skipping Git submodules setup
11$ apt-get update -qy
12Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
13Ign:2 http://deb.debian.org/debian stretch InRelease
14Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
15Get:4 http://deb.debian.org/debian stretch Release [118 kB]
16...
But at the very end, there will be an error:
1$ pytest
2/bin/bash: line 76: pytest: command not found
3ERROR: Job failed: exit code 1
In the end, .gitlab-ci.yml is not much more sophisticated than a shell script, and it is debugged in the same painful way, running it and seeing what breaks. In this case: no pytest, so we add it.
1# Use a system image with python 3.7 installed
2image: python:3.7
3test:
4 script:
5 # Install things we need for tests to run
6 - apt-get update -qy
7 - apt-get install -y python3-pip
8 # Install our project's dependencies
9 - pip3 install -r requirements.txt pytest
10 # Setup our project
11 - python3 setup.py develop
12 # Run tests
13 - pytest
And success! Sort of! Here is what I see in my Gitlab job:
1$ pytest
2============================= test session starts ==============================
3platform linux -- Python 3.7.1, pytest-4.0.2, py-1.7.0, pluggy-0.8.0
4rootdir: /builds/ralsina/boxes, inifile:
5collected 20 items
6
7tests/test_create_test_boxes.py .. [ 10%]
8tests/test_fill_row.py .... [ 30%]
9tests/test_is_breaking.py ...... [ 60%]
10tests/test_justify_row.py ......F [ 95%]
11tests/test_layout.py . [100%]
12
13=================================== FAILURES ===================================
14____________________________ test_justify_overfull _____________________________
15
16tmpdir = local('/tmp/pytest-of-root/pytest-0/test_justify_overfull0')
17
18 def test_justify_overfull(tmpdir):
19 """If a line is overfull, it still should be justified."""
20
21 separation = 0.05
22 page = boxes.Box(0, 0, 30, 50)
23 # Our failing text
24 inp = tmpdir.mkdir("sub").join("hello.txt")
25 inp.write(
26 "take delight in vexing me. You have no compassion for my poor nerves.\”"
27 )
28 # Adjust widths:
29 row = deque(boxes.create_text_boxes(inp))
30 # Put side by side
31 boxes.fill_row(row, page, separation)
32
33 # Should be overwide
34> assert row[-1].x > page.w
35E assert 0 > 30
36E + where 0 = Box(0, 0, 0, 1, "\n").x
37E + and 30 = Box(0, 0, 30, 50, "x").w
38
39tests/test_justify_row.py:179: AssertionError
40===================== 1 failed, 19 passed in 0.49 seconds ======================
41ERROR: Job failed: exit code 1
But ... when I run this locally, it works!
Debugging CI
Earlier in this book I accused other books of lying because "The code seems rehearsed, there are no errors, everything progresses monotonically towards a lovely pyramid of code with no false starts and no wrong assumptions." ... this whole subsection is about the dirt of real life, where most things don't freaking work the first, or second, or third time, and figuring out how to make them work is an awesome skill.
If the same code does different things in different "places" it's because there is a difference between those environments. I want this code to work in both my machine and CI's, so I need to find out what is different.
Let's go over possibilities!
It could be the versions of the python packages we have installed:
Let's change our CI so we can see what is actually installed.
1 # Print versions of python packages
2 - pip3 freeze
And here is the relevant output:
1$ pip3 freeze
2atomicwrites==1.2.1
3attrs==18.2.0
4docopt==0.6.2
5HarfPy==0.82
6more-itertools==4.3.0
7pluggy==0.8.0
8py==1.7.0
9pyparsing==2.3.0
10Pyphen==0.9.5
11pytest==4.0.2
12Python-FreeType==0.62
13six==1.12.0
14svgwrite==1.1.12
Let's compare it with my own system's, where tests pass:
1attrs==18.2.0
2black==18.4a0
3Click==7.0
4cssselect==1.0.3
5docopt==0.6.2
6docutils==0.14
7HarfPy==0.82
8lxml==4.2.5
9mistune==0.8.4
10more-itertools==4.3.0
11pluggy==0.6.0
12py==1.7.0
13Pygments==2.3.1
14pyliterate==0.1
15pyparsing==2.3.0
16Pyphen==0.9.5
17pytest==3.5.0
18Python-FreeType==0.62
19six==1.12.0
20svgwrite==1.1.12
It seems my system is full of random stuff. One way to "fix" this is to use a clean virtual environment and try to run the tests there like our CI system is doing.
After some experimentation, the test still passes locally.
Could it be the python version? Looks like it's not that either:
- CI: python 3.7.1
- Local: python 3.7.1
Could it be the version of HarfBuzz? To see that, we need to add a command in our CI configuration to show us the versions of the system packages.
1# Install things we need for tests to run
2 - apt-get update -qy
3 - apt-get install -y python3-pip
4 - dpkg -l
And here is the output for harfbuzz in the CI system:
ii libharfbuzz0b:amd64 1.4.2-1
And in my system:
harfbuzz 2.2.0-1
So, the library that is doing all the text positioning and shaping is a totally different version. That looks like a promising source for this error.
Whatever Gitlab is using to run our CI tests is much older than my local system (FYI Gitlab is running Debian Stretch), and our code doesn't notice, and then behaves in a subtly different way with those older versions.
That is not good. So, here we have to choose between three paths.
- Change the CI system to match my own by instaling a newer Harfbuzz
- Change my own system to match CI by installing an older Harfbuzz
- Create a CI-like system where I can find what the hell is wrong, then fix it and make it work in both places with both versions.
Option 1 is easy, but it means there is going to be some people for whom the test will just fail. Option 2 is out of the question because I like my system. Option 3 is what we will do.
Running a CI-like system
One thing to notice: you may not ever need or even want to do this. But if you want to, here is how.
Gitlab CI runs our system using docker and so can we.
I am not going to explain how to get docker running in your system, see here for a guide. Assuming it is running, you can do this to get a system much like Gitlab CI's:
1$ docker run -ti -v $PWD:/boxes -w /boxes --network=host python:3.7 bash
2Unable to find image 'python:3.7' locally
33.7: Pulling from library/python
454f7e8ac135a: Pull complete
5d6341e30912f: Pull complete
6087a57faf949: Pull complete
75d71636fb824: Pull complete
80c1db9598990: Pull complete
9bfb904e99f24: Pull complete
1078a3d3a96a32: Pull complete
11885a0ed92c89: Pull complete
12dd7cc9ace242: Pull complete
13Digest: sha256:3870d35b962a943df72d948580fc66ceaaee1c4fbd205930f32e0f0760eb1077
14Status: Downloaded newer image for python:3.7
15root@mybox:/boxes#
What that command does is start a container, which is sort of a virtual machine, just like Gitlab CI does:
docker run... run a container-ti... give me an interactive terminal-v $PWD:/boxes... make the current directory be/boxesin the container-w /boxes... start in/boxes--network=host... use the host network because that will workbash... and run abashshell please.
Now, inside that image, we can do the same things Gitlab does!
1root@mybox:/boxes# apt-get -qy update
2Ign:1 http://deb.debian.org/debian stretch InRelease
3Get:2 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]
4Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]
5Get:4 http://deb.debian.org/debian stretch Release [118 kB]
6Get:5 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [463 kB]
7Get:6 http://deb.debian.org/debian stretch-updates/main amd64 Packages [5152 B]
8Get:7 http://deb.debian.org/debian stretch Release.gpg [2434 B]
9Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7089 kB]
10Fetched 7409 kB in 14s (499 kB/s)
11Reading package lists...
12root@mybox:/boxes# apt-get install -y python3-pip
13
14[Output omitted]
15
16root@mybox:/boxes# pip3 install -r requirements.txt
17
18[Output omitted]
19
20root@mybox:/boxes# python3 setup.py develop
21
22[Output omitted]
23
24root@mybox:/boxes# pytest
25=================== test session starts ===================
26platform linux -- Python 3.7.1, pytest-3.5.0, py-1.7.0, pluggy-0.6.0
27rootdir: /boxes/src/part3/code/ci/boxes, inifile:
28collected 20 items
29
30tests/test_create_test_boxes.py .. [ 10%]
31tests/test_fill_row.py .... [ 30%]
32tests/test_is_breaking.py ...... [ 60%]
33tests/test_justify_row.py ......F [ 95%]
34tests/test_layout.py . [100%]
35
36========================= FAILURES =========================
37__________________ test_justify_overfull ___________________
38
39tmpdir = local('/tmp/pytest-of-root/pytest-0/test_justify_overfull0')
40
41 def test_justify_overfull(tmpdir):
42 """If a line is overfull, it still should be justified."""
43
44 separation = 0.05
45 page = boxes.Box(0, 0, 30, 50)
46 # Our failing text
47 inp = tmpdir.mkdir("sub").join("hello.txt")
48 inp.write(
49 "take delight in vexing me. You have no compassion for my poor nerves.\”"
50 )
51 # Adjust widths:
52 row = deque(boxes.create_text_boxes(inp))
53 # Put side by side
54 boxes.fill_row(row, page, separation)
55
56 # Should be overwide
57> assert row[-1].x > page.w
58E assert 0 > 30
59E + where 0 = Box(0, 0, 0, 1, "\n").x
60E + and 30 = Box(0, 0, 30, 50, "x").w
61
62tests/test_justify_row.py:179: AssertionError
63=========== 1 failed, 19 passed in 0.77 seconds ============
And threre you go, we have the same bug but in a place where we can see it.
Just for convenience, we can save the current state of the container:
1$ docker ps
2CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
35c5b9f7e2c9c python:3.7 "bash" 12 minutes ago Up 12 minutes nervous_bassi
4$ docker commit 5c5b9f7e2c9c boxes-ci
5sha256:636afbda6ecfc0d2f6c2efa23b541b258151c95cea9d1a7c709f32bd1bc58096
And now, whenever I use the boxes-ci image in my system it's this one that looks a lot like Gitlab CI with my stuff installed in it.
I will not go into the shameful details of the bug, but here is the change that fixes it.
Conclusion
In this chapter we setup automated testing for our changes via Gitlab CI. Now, every commit we push, every branch we create, will run our tests. Since we are confident that our tests are righteous, when we see beautiful green checkmarks next to our code we will know that our changes are good.
And since the changes are good, then why not merge them into master? And thus, CI is achieved.