Episode code: 001-04
Series: Building a Development Environment for Web Development
Date: 2026-05-21

Episode 4: Continuous Integration on GitLab

Description¶

In this video we introduce the concept of Continuous Integration (CI). CI will enable us to automatically test our software, and if everything works fine, build a new Docker image.

Video¶

Instructions¶

You might be wondering why we use GitLab instead of GitHub. There are a few reasons for this:

• GitLab's CI is more mature and feature-complete
• GitLab is open source, whereas GitHub is a proprietary, commercial service
• GitHub has been forcing Copilot on users recently
• GitHub is owned by Microsoft, a company known for their embrace, extend, and extinguish strategy; there is no reason to expect Microsoft will play nice long-term

Anyway, even if you want to work with GitHub, the ideas in this video will easily transfer to that environment.

Stages and Jobs¶

The CI pipeline will be divided into a set of stages. Stages run sequentially, that is, one after the other.

Each stage consists of one or more jobs. Unlike stages, jobs run in parallel. Jobs run in isolated Docker containers, each job starts a new container.

Jobs run a series of steps, so they are basically just shell scripts.

Planning Stages and Jobs¶

For our CI pipeline, we will focus on accomplishing these tasks:

• run static shell tests
• run static Python tests
• if static tests succeed, build a new Docker image, and run dynamic tests using docker-compose.
• if dynamic tests succeed, save the Docker image to a container registry

The two static tasks can run in parallel, so we will create separate jobs for them under a static stage.

Building the image, testing, and pushing the image to the container registry are all inter-dependent, so it makes no sense splitting them up. Consequently, we will make a dynamic stage with a single job.

Implementing the Static Stage¶

The CI pipeline is implemented using a YAML file named .gitlab-ci.yml.

1
2
3

stages:
    - static-stage
    - dynamic-stage

With this our two stages are defined, and they will be executed in the order listed above.

Let us go on and implement the first job in the static stage, that will do static shell tests.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

stages:
    - static-stage
    - dynamic-stage

static-shell:
  stage: static-stage
  image: alpine:latest
  before_script:
    - apk add shellcheck yamllint
  script:
    - shellcheck *.sh
    - yamllint *.yml

We define our first job static-shell, and assign the job to the static-stage(line 6). The job will make use of a standard alpine:latest Docker image (line 7).

The job defines two sections: before_script and script. These are different phases in the job life cycle and can be followed by an after_script phase as well.

The three sections define setup, main tasks, and clean up, and have a number of unique features, so they are not purely for semantic separation:

• the before_script overrides any global defaults defined under a top-level default section
• on failure or cancellation, the current step will finish, but remaining steps in before_script and script will not be executed, whereas the after_script section will always run; this makes after_script suitable for any clean up needed

Our before_script (setup) installs software packages needed to run the job. The script (main tasks) then run static shell tests, testing shell scripts and YAML files.

Because the job is running inside an ephemeral, non-persistent Docker container, we can skip the clean up step in this case. That would only be needed if something outside the Docker container needed cleaning up.

Let us go ahead and implement the second static job that will deal with Python static testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

stages:
    - static-stage
    - dynamic-stage

static-shell:
  stage: static-stage
  image: alpine:latest
  before_script:
    - apk add shellcheck yamllint
  script:
    - shellcheck *.sh
    - yamllint *.yml

static-python:
  stage: static-stage
  image: python:3.14.4-alpine3.22
  before_script:
    - apk add uv git
    - uv sync
    - uv pip install pip-audit
    - uv pip install git+https://github.com/psf/black
  script:
    - uv run pip-audit
    - uv run black .

This job makes use of a Docker image including a relevant version of Python (line 16). Again, the before_script sets up the necessary installations needed, and the script section does the actual main task.

There is nothing that should surprise you in this section, so we will carry on.

Implementing the Dynamic Stage¶

The dynamic stage has a lot more interesting things going on, so let us take a look.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

stages:
    - static-stage
    - dynamic-stage

static-shell:
  stage: static-stage
  image: alpine:latest
  before_script:
    - apk add shellcheck yamllint
  script:
    - shellcheck *.sh
    - yamllint *.yml

static-python:
  stage: static-stage
  image: python:3.14.4-alpine3.22
  before_script:
    - apk add uv git
    - uv sync
    - uv pip install pip-audit
    - uv pip install git+https://github.com/psf/black
  script:
    - uv run pip-audit
    - uv run black .

dynamic-build:
  stage: dynamic-stage
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - |
      echo "${CI_REGISTRY_PASSWORD}" | \
      docker login \
      --password-stdin \
      -u "${CI_REGISTRY_USER}" \
      -- \
      "${CI_REGISTRY}"
  script:
    - docker build --pull -t "${CI_REGISTRY_IMAGE}:latest" .
    - RTE=test docker-compose up --abort-on-container-exit --exit-code-from app
    - docker push "${CI_REGISTRY_IMAGE}:latest"

This time we use the docker:latest Docker image (line 28). This image includes the docker and docker-compose commands needed by this job.

The before_script logs in to the project's container registry. The "${CI_REGISTRY_PASSWORD}" is an environment variable provided by the GitLab CI system, allowing logging in to the registry with read and write access. This password is piped into stdin of the docker command, which is then configured to read the password from stdin.

The -u option provides the username via the "${CI_REGISTRY_USER}" environment variable provided by the CI system.

The -- syntax indicates that no more options will follow, so anything following will be arguments.

Finally, the "${CI_REGISTRY}" environment variable provided by the CI system sets the registry name to log into.

The script section of this job begins by building the Docker image based on the repository's Dockerfile (line 40). The --pull option first pulls any existing image from the container registry. You might wonder why are we pulling the registry image when we are building a new image anyway? The reason for doing this lies in Docker images' layered construction. Building a layer could potentially take a very long time, so reusing unchanged layers could save a lot of time. This is purely an optimization issue, but still a very important issue.

Next, we run docker compose up setting RTE=test, so test mode. We need to expand our entrypoint.sh script to handle this test mode, and we must also provide an env-test file with relevant environment variables. We will get back to this shortly, but let us first finish explaining the script section itself.

The docker-compose takes a couple of interesting options that are necessary for it to work in a CI pipeline environment, namely the --abort-on-container-exit and --exit-code-from app.

The first option, --abort-on-container-exit takes down docker-compose whenever a container exits. This means that if we design our entrypoint.sh to exit the app service container when we have finished testing the app, the CI pipeline will continue to the next step instead of sitting waiting for requests that will never come.

The second option, --exit-code-from app takes the exit code from the app service container and propagates it to the docker-compose command itself. Remember, in shell scripting, a non-zero exit code indicates an error. By selectively setting the exit code in the entrypoint.sh script we can then signal to the pipeline whether the testing succeeded or not. If the exit code is not zero, the pipeline will fail and stop. If it is zero, it will continue to the final step.

This final step (line 42) pushes the Docker image to the container registry. This will only happen if the test passed, so we will not end up with a bad image in the registry.

Implementing Development Mode¶

Before updating the entrypoint.sh script, let us recall how we left it in the Adapting Docker video.

1
2
3
4

#!/bin/sh

echo "**** ENTRYPOINT STARTED ****"
exec uv run guestbook.py

First, we will implement the development mode, as this is pretty easy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#!/bin/sh

echo "**** ENTRYPOINT STARTED ****"

case "${RTE}" in

    dev)
        echo "** Development mode."
        exec uv run guestbook.py
        ;;

    test)
        echo "** Test mode."
        ;;

    prod)
        echo "** Prod mode."
        ;;

esac

The case ... esac structure let us test the RTE variable and branch out depending on its value.

We already have an env-dev file, so we do not need to implement that.

Implementing Production Mode¶

The production mode is also quite easy to implement, however, in production we will no longer make use of the Flask web server, as this is only meant for use in development, so instead we will use the gunicorn Python Web Server Gateway Interface (WSGI) server instead.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/sh

echo "**** ENTRYPOINT STARTED ****"

case "${RTE}" in

    dev)
        echo "** Development mode."
        exec uv run guestbook.py
        ;;

    test)
        echo "** Test mode."
        ;;

    prod)
        echo "** Prod mode."
        exec uv run gunicorn -b 0.0.0.0:3000 guestbook:app
        ;;

esac

We then need to add the gunicorn package to the project dependencies using the uv tool.

uv add gunicorn

Because the docker-compose.yml file will now expect to be able to read a env-prod file, we need to implement this file also.

RTE=prod

WEB_HOST=0.0.0.0

POSTGRES_HOST=db
POSTGRES_DB=pgdb
POSTGRES_USER=pguser

PGHOST=db
PGPORT=5432
PGDATABASE=pgdb
PGUSER=pguser

Notice that the POSTGRES_PASSWORD and PGPASSWORD entries have been removed. This is obviously for security reasons. However, these variables must be made available to the services at run time.

We will make a video on how to handle the passwords in a secure way at a later point, but if you want to test the production mode you can set a dummy password for now.

Implementing Test Mode¶

Implementing test mode is going to be much more of a mouthful.

Our docker-compose.yml file expects an environment file called env-test when we run docker-compose up with RTE=test as we do on the CI pipeline. So let us go ahead an implement that file.

RTE=test

WEB_HOST=0.0.0.0

POSTGRES_HOST=db
POSTGRES_DB=pgdb
POSTGRES_USER=pguser
POSTGRES_PASSWORD=pgpassword

PGHOST=db
PGPORT=5432
PGDATABASE=pgdb
PGUSER=pguser
PGPASSWORD=pgpassword

Other than the RTE variable, the file is identical to the env-dev file.

Let us now turn our attention to the test mode implementation of the entrypoint.sh script. When we run test mode, we can not assume that the database will be present. In fact, this will never be the case running on the CI pipeline. However, we can not RTE=test docker-compose exec -it db sh and then run the provisioning script interactively in test mode, especially not on the CI pipeline.

This means that we will have to automate the provisioning for the test mode. Since entrypoint.sh is running on the app service, we will have to install the psql command into the guestbook service. Because we will only need the psql tool in test mode, we will install it ad-hoc into the app container rather than installing it into the guestbook image.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/bin/sh

echo "**** ENTRYPOINT STARTED ****"

case "${RTE}" in

    dev)
        echo "** Development mode."
        exec uv run guestbook.py
        ;;

    test)
        echo "** Test mode."

        apk add postgresql17-client

        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -f /dbfiles/provisioning.sql
        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -c "select * from book;"

        # more testing here

        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -c "drop table book;"

        ;;

    prod)
        echo "** Prod mode."
        exec uv run gunicorn -b 0.0.0.0:3000 guestbook:app
        ;;

esac

The code above takes care of installing the psql tool (line 15), running the provisioning script (line 17), testing that the book table has been created (line 18), and finally dropping the table (line 22). The reason for dropping the table is to ensure that the test will run under identical conditions every time, also if we run test mode locally.

Next, we need to start the gunicorn WSGI server, but now we face another problem: after starting the server, we need the script to continue and do the actual testing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!/bin/sh

echo "**** ENTRYPOINT STARTED ****"

case "${RTE}" in

    dev)
        echo "** Development mode."
        exec uv run guestbook.py
        ;;

    test)
        echo "** Test mode."

        apk add postgresql17-client curl grep

        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -f /dbfiles/provisioning.sql
        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -c "select * from book;"

        uv run gunicorn -b 0.0.0.0:3000 guestbook:app &
        sleep 10

        URL="http://nginx"

        pattern="hello"
        response="$(curl -X POST -d "tx=${pattern}&host=me" ${URL})"
        if echo "${response}" | grep -q "${pattern}"; then
            echo "Found ${pattern} in response."
        else
            echo "Error: ${pattern} not found in response."
            exit 1
        fi

        uv install pip-audit
        uv run pip-audit || exit 1

        psql -h db -U "${POSTGRES_USER}" -d "${POSTGRES_DB}" -c "drop table book;"

        ;;

    prod)
        echo "** Prod mode."
        exec uv run gunicorn -b 0.0.0.0:3000 guestbook:app
        ;;

esac

The solution is to put the server process in the background subshell using the & asynchronous operator (line 20). Because the server may take a little while to start up, we sleep for 10 seconds on the foreground shell; if we do not pause the shell this way, all the tests will run before the server is started and fail.

On line 23 to 32 we run a simple test. We set up an expected pattern that we know we would find in the response if the request succeeds, make the request, and check for the pattern. For this to work the curl and grep tools are needed, so those are installed in the 15.

Finally, on line 34 to 35 we install the pip-audit tool to test for any vulnerabilities in the installed Python packages. While we also do this on the static stage of the CI pipeline, the test mode might be used outside the pipeline, and it is of high importance, so the sooner we know the better.

There are many more things we could test, but this video is already getting very long, and the basic pattern can be repeated easily.

Separating Databases for Development, Test, and Production Mode¶

Before running the updated project, there is something we must consider.

When running test mode, we do not want to interfere with any data in the database. This could be very annoying in development mode, but outright catastrophic if someone ran RTE=test docker-compose up on the production server. For this reason, it would be a good idea to ensure that the database will be separate for the three modes.

Fortunately, this is a very simple fix. We can simply update the docker-compose.yml file as shown below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

services:

    db:
        image: postgres:latest
        volumes:
            - ./dbfiles:/dbfiles:ro
            - guestbookdb:/var/lib/postgresql
        env_file:
            - "env-${RTE}"
        networks:
            - dbnet
        healthcheck:
            test: ["CMD-SHELL", "pg_isready"]
            interval: 5s
            timeout: 5s
            retries: 5
            start_period: 30s

    app:
        image: guestbook:latest
        volumes:
            - ./dbfiles:/dbfiles
        env_file:
            - "env-${RTE}"
        networks:
            - dbnet
            - webnet
        depends_on:
            db:
                condition: service_healthy

    nginx:
        image: nginx:latest
        volumes:
            - ./nginx:/etc/nginx/conf.d:ro
        networks:
            - webnet
        ports:
            - "127.0.0.1:${HTTP_PORT:-80}:80"
            - "127.0.0.1:${HTTPS_PORT:-443}:443"
        depends_on:
            - app
        restart: unless-stopped

volumes:
    guestbookdb:
        name: "guestbookdb-${RTE}"

networks:
    dbnet:
    webnet:

With this change, the development database will be named guestbookdb-dev, the test mode database guestbookdb-test, and the production database guestbookdb-prod.

Also notice, line 21-22, the app service need access to the database provisioning script. This is because the app service will provision the database in test mode.

Running Test Mode Locally¶

The time has come to see if everything works. Let us run the test mode locally.

RTE=test docker-compose up --abort-on-container-exit --exit-code-from app

Docker should start in test mode and run the tests. Examine the log to see if any errors occurred.

Push To GitLab¶

Depending on whether you have already made a GitLab project you can push one of the following commands after adding and committing to git.

git push

git push --set-upstream git@gitlab.com:<your-gitlab-user-name>/<your-gitlab-project-name>.git master

If the push is accepted, you should see a message from the server with the command to set GitLab as a remote for the local repository. You should run this command.

git remote add origin git@gitlab.com:<your-gitlab-user-name>/<your-gitlab-project-name>.git

From the GitLab web site you can monitor the pipeline to see if everything works as intended.

Configure docker-compose.yml to Use the Registry Docker Image¶

We are not quite finished yet. We should set the Docker image of the app service in docker-compose.yml to use the registry image that we build on the pipeline.

Simply change image: guestbook:latest to image: registry.gitlab.com/<your-gitlab-user-name>/<your-gitlab-project-name>:latest.

Now you need to pull the Docker image from the GitLab registry, because that is the image specified in the docker-compose.yml file. You need to do a docker login similar to what we do on the CI pipeline, but this time the environment will not provide you with the variables. See Create a personal access token.

If you are not logged in to the GitLab container registry, run the command below.

echo "<your-personal-access-token>" | docker login --password-stdin -u <your-gitlab-user-name> -- registry.gitlab.com

Run your project again in test and development mode to see everything works as intended.

Summary¶

That was quite a journey, but we now have an automated CI pipeline, and we can easily extend the testing and fit it to another project.

In the next video in the series we will automate deployment, that is: when a new Docker image is created on the registry, it should be deployed on a server. Whether that be a test environment or the actual production environment is up to you, but the process is the same.