The Digital Cat - Docker

From Docker CLI to Docker Compose

2022-02-19T15:00:00+01:00

In this post I will show you how and why Docker Compose is useful, building a simple application written in Python that uses PostgreSQL. I think it is worth going through such an exercise to see how technologies that we might be already familiar with actually simplify workflows that would otherwise definitely be more complicated.

The name of the demo application I will develop is a very unimaginative whale, that shouldn't clash with any other name introduced by the tools I will use. Every time you see something with whale in it you know that I am referring to a value that you can change according to your setup.

Before we start, please create a directory to host all the files we will create. I will refer to this directory as the "project directory".

PostgreSQL¶

Since the application will connect to a PostgreSQL database the first thing we can explore is how to run that in a Docker container.

The official Postgres image can be found here, and I highly recommend taking the time to properly read the documentation, as it contains a myriad of details that you should be familiar with.

For the time being, let's focus on the environment variables that the image requires you to set.

Password

The first variable is POSTGRES_PASSWORD, which is the only mandatory configuration value (unless you disable authentication which is not recommended). Indeed, if you run the image without setting this value, you get this message

$ docker run postgres
Error: Database is uninitialized and superuser password is not specified.
       You must specify POSTGRES_PASSWORD to a non-empty value for the
       superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".

       You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
       connections without a password. This is *not* recommended.

       See PostgreSQL documentation about "trust":
       https://www.postgresql.org/docs/current/auth-trust.html

This value is very interesting because it's a secret. So, while I will treat it as a simple configuration value in the first stages of the setup, later we will need to discuss how to manage it properly.

Superuser

Being a production-grade database, Postgres allows you to specify users, groups, and permissions in a fine-grained fashion. I won't go into that as it's usually more a matter of database administration and application development, but we need to define at least the superuser. The default value for this image is postgres, but you can change it setting POSTGRES_USER.

Database name

If you do not specify the value of POSTGRES_DB, this image will create a default database with the name of the superuser.

A note of warning here. If you omit both the database name and the user you will end up with the superuser postgres and database postgres. The official documentation states that

After initialization, a database cluster will contain a database named
postgres, which is meant as a default database for use by utilities,
users and third party applications. The database server itself does not
require the postgres database to exist, but many external utility programs
assume it exists.

This mean that it is not ideal to use that as the database for our application. So, unless you are just trying out a quick piece of code, my recommendation is to always configure all three values: POSTGRES_PASSWORD, POSTGRES_USER, and POSTGRES_DB.

We can run the image with

$ docker run -d \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  postgres:13

As you can see I run the image in detached mode. This image is not meant to be interactive, as Postgres is by it's very nature a daemon. To connect in an interactive way we need to use the tool psql, which is provided by this image. Please note that I'm running postgres:13 only to keep the post consistent with what you will see if you read it in the future, you are clearly free to use any version of the engine.

The ID of the container is returned by docker run but we can retrieve it any time running docker ps. Using IDs is however pretty complicated, and looking at the command history is not immediately clear what you have been doing at a certain point in time. For this reason, it's a good idea to name the containers.

Stop the previous container and run it again with

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  postgres:13

Stopping containers

You can stop containers using docker stop ID. This gives containers a grace period to react to the SIGTERM signal, for example to properly close files and terminate connections, and then terminates it with SIGKILL. You can also force it to stop unconditionally using docker kill ID which sends SIGKILL immediately.

In either case, however, you might want to remove the container, that otherwise will be kept indefinitely by Docker. This can become a problem when containers are named, as you can't reuse a name that is currently assigned to a container.

To remove a container you have to run docker rm ID, but you can leverage the fact that both docker stop and docker kill return the ID of the container to pipe the termination and the removal

$ docker stop ID | xargs docker rm

Otherwise, you can use docker rm -f ID, which corresponds to docker kill followed by docker rm. If you name a container, however, you can use its name instead of the ID.

Now we can connect to the database using the executable psql provided in the image itself. To execute a command inside a container we use docker exec and this time we will specify -it to open an interactive session. psql uses by default the user name root, and the database with the same name as the user, so we need to specify both. The header informs me that the image is running PostgreSQL 13.5 on Debian.

$ docker exec -it whale-postgres psql -U whale_user whale_db
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=#

Postgres trust

You might be surprised by the fact that psql didn't ask for the password that we set when we run the container. This happens because the server trusts local connections, and when we run psql inside the container we are on localhost.

If you are curious about trust in Postgres you can see the configuration file with

$ docker exec -it whale-postgres \
  cat /var/lib/postgresql/data/pg_hba.conf

where you can spot the lines

# TYPE  DATABASE  USER  ADDRESS  METHOD

# "local" is for Unix domain socket connections only
local   all       all            trust

You can find more information about Postgres trust in the official documentation.

Here, I can list all the databases with \l. You can see all psql commands and the rest of the documentation at https://www.postgresql.org/docs/current/app-psql.html.

$ docker exec -it whale-postgres psql -U whale_user whale_db
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=# \l
                                    List of databases
   Name    |   Owner    | Encoding |  Collate   |   Ctype    |     Access privileges     
-----------+------------+----------+------------+------------+---------------------------
 postgres  | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | =c/whale_user            +
           |            |          |            |            | whale_user=CTc/whale_user
 template1 | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | =c/whale_user            +
           |            |          |            |            | whale_user=CTc/whale_user
 whale_db  | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | 
(4 rows)

whale_db=#

As you can see, the database called postgres has been created as part of the initialisation, as clarified previously. You can exit psql with Ctrl-D or \q.

If we want the database to be accessible from outside we need to publish a port. The image exposes port 5432 (see the source code), which tells us where the server is listening. To publish the port towards the host system we can add -p 5432:5432. Please remember that exposing a port in Docker basically means to add some metadata that informs the user of the image, but doesn't affect the way it runs.

Stop the container (you can use its name now) and run it again with

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 postgres:13

Running docker ps we can see that the container publishes the port now (0.0.0.0:5432->5432/tcp). We can double-check it with ss ("socket statistics")

$ ss -nulpt | grep 5432
tcp  LISTEN  0  4096  0.0.0.0:5432  0.0.0.0:*
tcp  LISTEN  0  4096     [::]:5432     [::]:*

Please note that usually ss won't tell you the name of the process using that port because the process is run by root. If you run ss with sudo you will see it

$ sudo ss -nulpt | grep 5432
tcp  LISTEN  0  4096  0.0.0.0:5432  0.0.0.0:*  users:(("docker-proxy",pid=1262717,fd=4))
tcp  LISTEN  0  4096     [::]:5432     [::]:*  users:(("docker-proxy",pid=1262724,fd=4))

Unfortunately, ss is not available on macOS. On that platform (and on Linux as well) you can use lsof with grep

$ sudo lsof -i -p -n | grep 5432
docker-pr 219643            root    4u  IPv4 2945982      0t0  TCP *:5432 (LISTEN)
docker-pr 219650            root    4u  IPv6 2952986      0t0  TCP *:5432 (LISTEN)

or directly using the option -i

$ sudo lsof -i :5432
COMMAND      PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
docker-pr 219643 root    4u  IPv4 2945982      0t0  TCP *:postgresql (LISTEN)
docker-pr 219650 root    4u  IPv6 2952986      0t0  TCP *:postgresql (LISTEN)

Please note that docker-pr in the output above is just docker-proxy truncated, matching what we saw with ss previously.

If you want to publish the container's port 5432 to a different port on the host you can just use -p ANY_NUMBER:5432. Remember however that port numbers under 1024 are privileged or well-known, which means that they are assigned by default to specific services (listed here).

This means that in theory you can use -p 80:5432 for your database container, exposing it on port 80 of your host. In practice this will result in a lot of headaches and a bunch of developers chasing you with spikes and shovels.

Now that we exposed a port we can connect to the database running psql in an ephemeral container. "Ephemeral" means that a resource (in this case a Docker container) is run just for the time necessary to serve a specific purpose, as opposed to "permanent". This way we can simulate someone that tries to connect to the Docker container from a different computer on the network.

Since psql is provided by the image postgres we can in theory run that passing the hostname with -h localhost, but if you try it you will be disappointed.

$ docker run -it postgres:13 psql -h localhost -U whale_user whale_db
psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (::1), port 5432 failed: Cannot assign requested address
        Is the server running on that host and accepting TCP/IP connections?

This is correct, as that container runs in a bridge network where localhost is the container itself. To make it work we need to run the container as part of the host network (that is the same network our computer is running on). This can be done with --network=host

$ docker run -it \
  --network=host postgres:13 \
  psql -h localhost -U whale_user whale_db
Password for user whale_user: 
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=#

Please note that now psql asks for a password (that you know because you set it when we run the container whale-postgres). This happens because the tool is not run on the same node as the database server any more, so PostgreSQL doesn't trust it.

Volumes¶

If we used a structured framework in Python, we could leverage an ORM like SQLAlchemy to map classes to database tables. The model definitions (or changes) can be captured into little scripts called migrations that are applied to the database, and those can also be used to insert some initial data. For this example I will go a simpler route, that is to initialise the database using SQL directly.

I do not recommend this approach for a real project but it should be good enough in this case. In particular, it will allow me to demonstrate how to use volumes in Docker.

Make sure the container whale-postgres is running (with or without publishing the port, it's not important at the moment). Connect to the container using psql and run the following two SQL commands (make sure you are connected to the database whale_db)

CREATE TABLE recipes (
  recipe_id INT NOT NULL,
  recipe_name VARCHAR(30) NOT NULL,
  PRIMARY KEY (recipe_id),
  UNIQUE (recipe_name)
);

INSERT INTO recipes 
    (recipe_id, recipe_name) 
VALUES 
    (1,'Tacos'),
    (2,'Tomato Soup'),
    (3,'Grilled Cheese');

This code creates a table called recipes and inserts 3 rows with an id and a name. The output of the above commands should be

CREATE TABLE
INSERT 0 3

You can double check that the database contains the table with \dt

whale_db=# \dt
           List of relations
 Schema |  Name   | Type  |   Owner    
--------+---------+-------+------------
 public | recipes | table | whale_user
(1 row)

and that the table contains three rows with a select.

whale_db=# select * from recipes;
 recipe_id |  recipe_name   
-----------+----------------
         1 | Tacos
         2 | Tomato Soup
         3 | Grilled Cheese
(3 rows)

Now, the problem with containers is that they do not store data permanently. While the container is running there are no issues, as a matter of fact you can terminate psql, connect, and run the select again, and you will see the same data.

If we stop the container and run it again, though, we will quickly realise that the values stored in the database are gone.

$ docker stop whale-postgres | xargs docker rm 
whale-postgres

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 postgres:13
4a647ebef78e32bb4733484a6e435780e17a69b643e872613ca50115d60d54ce

$ docker exec -it whale-postgres \
  psql -U whale_user whale_db -c "select * from recipes"
ERROR:  relation "recipes" does not exist
LINE 1: select * from recipes
                      ^

Containers have been created with isolation in mind, which is why by default nothing of what happens inside the container is connected with the host and is preserved when the container is destroyed.

As happened with ports, however, we need to establish some communication between containers and the host system, and we also want to keep data after the container has been destroyed. The solution in Docker is to use volumes.

There are three types of volumes in Docker: host, anonymous, and named. Host volumes are a way to mount inside the container a path on the host's filesystem, and while they are useful to exchange data between the host and the container, they also often have permissions issues. Generally speaking, containers define users whose IDs are not mapped to the host's ones, which means that the files written by the container might end up belonging to non-existing users.

Anonymous and named volumes are simply virtual filesystems created and managed independently from containers. These can be connected with a running container so the latter can use the data contained in them and store data that will survive its termination. The only difference between named an anonymous volumes is the name that allows you to easily manage them. For this reason, I think it's not really useful to consider anonymous volumes, which is why I will focus on named ones.

You can manage volumes using docker volume, that provides several subcommands such as create, and rm. You can then attach a named volume to a container when you run it using the option -v of docker run. This creates the volume if it's not already existing, so this is the standard way many of us create a volume.

Stop and remove the running Postgres container and run it again with a named volume

$ docker stop whale-postgres | xargs docker rm 
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13

This will create the volume named whale_dbdata and connect it to the path /var/lib/postgresql/data in the container that we are running. That path happens to be the one where Postgres stores the actual database, as you can see from the official documentation. There is a specific reason why I used the prefix whale_ for the name of the volume, which will be clear later when we will introduce Docker Compose.

docker ps doesn't give any information on volumes, so to see what is connected to your container you need to use docker inspect

$ docker inspect whale-postgres 
[...]
        "Mounts": [
            {
                "Type": "volume",
                "Name": "whale_dbdata",
                "Source": "/var/lib/docker/volumes/whale_dbdata/_data",
                "Destination": "/var/lib/postgresql/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
[...]

The value for "Source" is where the volume is stored in the host, that is on your computer, but generally speaking you can ignore that detail. You can see all volumes using docker volume ls (using grep if the list is long as it is in my case)

$ docker volume ls | grep whale
local     whale_dbdata

Now that the container is running and is connected to a volume, we can try to initialise the database again. Connect with psql using the command line we developed before and run the SQL commands that create the table recipes and insert three rows.

The whole point of using a volume is to make information permanent, so now terminate and remove the Postgres container, and run it again using the same volume. You can check that the database still contains data using the query shown previously.

$ docker rm -f whale-postgres 
whale-postgres
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13
893378f044204e5c1a87473a038b615a08ad08e5da9225002a470caeac8674a8
$ docker exec -it whale-postgres \
  psql -U whale_user whale_db \
  -c "select * from recipes"
 recipe_id |  recipe_name   
-----------+----------------
         1 | Tacos
         2 | Tomato Soup
         3 | Grilled Cheese
(3 rows)

Python application¶

Great! Now that we have a database that can be restarted without losing data we can create a Python application that interacts with it. Again, please remember that the goal of this post is to show what container orchestration is and how Docker compose can simplify it, so the application developed in this section is absolutely minimal.

I will first create an application and run it in the host, leveraging the port exposed by the container to connect to the database. Later, I will move the application in its own container.

To create the application, first create a Python virtual environment using your preferred method. I currently use pyenv (https://github.com/pyenv/pyenv).

pyenv virtualenv whale_docker
pyenv activate whale_docker

Now we need to put our requirements in a file and install them. I prefer to keep things tidy from day zero, so create the directory whaleapp in the project directory and inside it the file requirements.txt.

mkdir whaleapp
touch whaleapp/requirements.txt

The only requirement we have for this simple application is psycopg2, so I add it to the file and then install it. Since we are installing requirements is useful to update pip as well.

echo "psycopg2" >> whaleapp/requirements.txt
pip install -U pip
pip install -r whaleapp/requirements.txt

Now create the file whaleapp/whaleapp.py and put this code in it

whaleapp/whaleapp.py

import time

import psycopg2

connection_data = { 1
    "host": "localhost",
    "database": "whale_db",
    "user": "whale_user",
    "password": "whale_password",
}

while True:
    try:
        conn = None

        # Connect to the PostgreSQL server
        print("Connecting to the PostgreSQL database...")
        conn = psycopg2.connect(**connection_data) 2

        # Create a cursor
        cur = conn.cursor()

        # Execute the query
        cur.execute("select * from recipes") 3

        # Fetch all results
        results = cur.fetchall()
        print(results) 4

        # Close the connection
        cur.close()
    except (Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close() 5
            print("Database connection closed.")

    # Wait three seconds
    time.sleep(3)

As you can see the code is not complicated. The application is an endless while loop that every 3 seconds establishes a connection with the DB 2 using the configuration in 1. After this, the query select * from recipes is run 3 , all the results are printed on the standard output 4, and the connection is closed 5.

If the Postgres container is running and publishing port 5432, this application can be run directly on the host

$ python whaleapp.py 
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

and will go on indefinitely until we press Ctrl-C to stop it.

For the same reasons of isolation and security that we discussed previously, we want to run the application in a Docker container. This can be done pretty easily, but we will run into the same issues that we had when we where trying to run psql in a separate container. At the moment, the application tries to connect to the database on localhost, which is fine while the application is running on the host directly, but won't work any more once that is transported into a Docker container.

To face one problem at a time, let's first containerise the application and run it using the host network. Once this works, we can see how to solve the communication problem between containers.

The easiest way to containerise a Python application is to create a new image starting from the image python:3. The following Dockerfile goes into the application directory

whaleapp/Dockerfile

FROM python:3 1

WORKDIR /usr/src/app 2

COPY requirements.txt . 3
RUN pip install --no-cache-dir -r requirements.txt 4

COPY . . 5

CMD [ "python", "-u", "./whaleapp.py" ] 6

A Docker file contains the description of the layers that build an image. Here, we start from the official Python 3 image 1 (https://hub.docker.com/_/python), set a working directory 2, copy the requirements file 3 and install the requirements 4, then copy the rest of the application 5, and run the application 6. The Python option -u avoids output buffering, see https://docs.python.org/3/using/cmdline.html#cmdoption-u.

It is important to keep in mind the layered nature of Docker images, as this can lead to simple optimisation tricks. In this case, loading the requirements file and installing them creates a layer out of a file that doesn't change very often, while the layer created at 5 is probably changing very quickly while we develop the application. If we run something like

[...]

COPY . .

RUN pip install --no-cache-dir -r requirements.txt

CMD [ "python", "-u", "./app.py" ]

we would have to install the requirements every time we change the application code, as this would rebuild the COPY layer and thus invalidate the layer containing the RUN command.

Once the Dockerfile is in place we can build the image

$ cd whaleapp
$ docker build -t whaleapp .
Sending build context to Docker daemon  6.144kB
Step 1/6 : FROM python:3
 ---> 768307cdb962
Step 2/6 : WORKDIR /usr/src/app
 ---> Using cache
 ---> b00189756ddb
Step 3/6 : COPY requirements.txt .
 ---> a7aef12f562c
Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 153a3ca6a1b2
Collecting psycopg2
  Downloading psycopg2-2.9.3.tar.gz (380 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.9.3-cp39-cp39-linux_x86_64.whl size=523502 sha256=1a3aac3cf72cc86b63a3e0f42b9b788c5237c3e5d23df649ca967b29bf89ecf5
  Stored in directory: /tmp/pip-ephem-wheel-cache-ow3d1yop/wheels/b3/a1/6e/5a0e26314b15eb96a36263b80529ce0d64382540ac7b9544a9
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.9.3
WARNING: You are using pip version 20.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container 153a3ca6a1b2
 ---> b18aead1ef15
Step 5/6 : COPY . .
 ---> be7c3c11e608
Step 6/6 : CMD [ "python", "-u", "./app.py" ]
 ---> Running in 9e2f4f30b59e
Removing intermediate container 9e2f4f30b59e
 ---> b735eece4f86
Successfully built b735eece4f86
Successfully tagged whaleapp:latest

You can see the layers being built one by one (marked as Step x/6 here). Once the image has been build you should be able to see it in the list of images present in your system

$ docker image ls | grep whale
whaleapp  latest  969b15466905  9 minutes ago  894MB

Size of containers

You might want to observe 1 minute of silence meditating on the fact that we used almost 900 megabytes of space to run 40 lines of Python. As you can see benefits come with a cost, and you should not underestimate those. 900 megabytes might not seem a lot nowadays, but if you keep building images you will soon use up the space on your hard drive or end up paying a lot for the space on your remote repository.

By the way, this is the reason why Docker splits image into layers and reuses them. For now we can ignore this part of the game, but remember that keeping the system clean and removing past artefacts is important.

As I mentioned before we can run this image but we need to use the host network configuration.

$ docker run -it --rm --network=host --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Please note that I used --rm to make Docker remove the container automatically when it is terminated. This way I can run it again with the same name without having to explicitly remove the past container with docker rm.

Run containers in the same network¶

Docker containers are isolated from the host and from other containers by default. This however doesn't mean that they can't communicate with each other if we run them in a specific configuration. In particular, an important part in Docker networking is played by bridge networks.

Whenever containers are run in the same custom bridge network, Docker provides them DNS resolution using the container names. This means that we can make the application communicate with the database without having to run the former in the host network.

A custom network can be created using docker network

$ docker network create whale

As always, Docker will return the ID of the object it just created, but we can ignore it for now, as we can refer to the network by name.

Stop and remove the Postgres container, and run it again using the network whale

$ docker rm -f whale-postgres 
whale-postgres
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  --network=whale \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13

Please note that there is no need to publish the port 5432 in this setup, as the host doesn't need to access the container. Should this be a requirement, add the option -p 5432:5432 again.

As happened with volumes, docker ps doesn't give information about the network that containers are using, so you have to use docker inspect again

$ docker inspect whale-postgres 
[...]
        "NetworkSettings": {
            "Networks": {
                "whale": {
[...]

Docker network management

The command docker network can be used to change the network configuration of running containers.

You can disconnect a running container from a network with

$ docker network disconnect NETWORK_ID CONTAINER_ID

and connect it with

$ docker network connect NETWORK_ID CONTAINER_ID

You can see which containers are using a given network inspecting it

$ docker network inspect NETWORK_ID

Remember that disconnecting a container from a network makes it unreachable, so while it is good that we can do this on running containers, maintenance shall be always carefully planned to avoid unexpected downtime.

As I mentioned before, Docker bridge networks provide DNS resolution using the container's name. We can double check this running a container and using ping.

$ docker run -it --rm --network=whale whaleapp ping whale-postgres
PING whale-postgres (172.19.0.2) 56(84) bytes of data.
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=2 ttl=64 time=0.100 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=3 ttl=64 time=0.115 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=4 ttl=64 time=0.101 ms
^C
--- whale-postgres ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 80ms
rtt min/avg/max/mdev = 0.064/0.095/0.115/0.018 ms

What I did here was to run the image whaleapp that we built previously, but overriding the default command and running ping whale-postgres instead. This is a good way to check if a host can resolve a name on the network (dig is another useful tool but is not installed by default in that image).

As you can see the Postgres container is reachable and we also know that it currently runs with the IP 172.19.0.2. This value might be different on your system, but it will match the information you get if you run docker network inspect whale.

The point of all this talk about DNS is that we can now change the code of the Python application so that it connects to whale-postgres instead of localhost

connection_data = {
    "host": "whale-postgres",
    "database": "whale_db",
    "user": "whale_user",
    "password": "whale_password",
}

Once this is done, rebuild the image and run it in the whale network

$ docker build -t whaleapp .
[...]
$ docker run -it --rm --network=whale --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

You can also take the network directly from another container, which is a useful shortcut.

$ docker build -t whaleapp .
[...]
$ docker run -it --rm \
  --network=container:whale-postgres \
  --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Run time configuration¶

Hardcoding configuration values into the application is never a great idea, and while this is a very simple example it is worth pushing the setup a bit further to make it tidy.

In particular, we can replace the connection data host, database, and user with environment variables, which allow us to reuse the application configuring it at run time. For simplicity's sake I will store the password in an environment variable as well, and pass it in clear text when we run the container. See the box for more information about how to manage secret values.

Reading values from environment variables is easy in Python

import os
import time

import psycopg2

DB_HOST = os.environ.get("WHALEAPP__DB_HOST", None)
DB_NAME = os.environ.get("WHALEAPP__DB_NAME", None)
DB_USER = os.environ.get("WHALEAPP__DB_USER", None)
DB_PASSWORD = os.environ.get("WHALEAPP__DB_PASSWORD", None)

connection_data = {
    "host": DB_HOST,
    "database": DB_NAME,
    "user": DB_USER,
    "password": DB_PASSWORD,
}

Please note that I prefixed all environment variables with WHALEAPP__. This is not mandatory, and has no special meaning for the operating system. In my experience, complicated systems can have many environment variables, and using prefixes is a simple and effective way to keep track of which part of the system needs that particular value.

We already know how to pass environment variables to Docker containers as we did it when we run the Postgres container. Build the image again, and then run it passing the correct variables

$ docker build -t whaleapp .
[...]
$ docker run -it --rm --network=whale \
  -e WHALEAPP__DB_HOST=whale-postgres \
  -e WHALEAPP__DB_NAME=whale_db \
  -e WHALEAPP__DB_USER=whale_user \
  -e WHALEAPP__DB_PASSWORD=password \
  --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Managing secrets

A "secret" is a value that should never be shown in plain text, as it is used to grant access to a system. This can be a password or a private key such as the ones you have to run SSH, and as happens with everything related to security, managing them is complicated. Please keep in mind that security is hard and that the best attitude to have is: every time you think something in security is straightforward this means you got it wrong.

Generally speaking, you want secrets to be encrypted and stored in a safe place where access is granted to a narrow set of people. These secrets should be accessible to your application in a secure way, and it shouldn't be possible to access the secrets hosted in the memory of the application.

For example, many posts online show how you can use AWS Secrets Manager to store your secrets and access them from your application using jq to fetch them at run time. While this works, if the JSON secret contains a syntax error, jq dumps the whole value in the standard output of the application, which means that the logs contain the secret in plain text.

Vault is a tool created by Hashicorp that many use to store secrets needed by containers. It is interesting to read in the description of the image that with a specific configuration the container prevents memory from being swapped to disk, which would leak the unencrypted values. As you see, security is hard.

Orchestration tools always provide a way to manage secrets and to pass them to containers. For example, see Docker Swarm secrets, Kubernetes secrets, and secrets for AWS Elastic Container Service.

Enter Docker Compose¶

The setup we created in the past sections is good, but is far from being optimal. We had to create a custom bridge network and then start the Postgres and the application containers connected to it. To stop the system we need to terminate containers manually and to remember to remove them to avoid blocking the container name. We also have to manually remove the network if we want to keep the system clean.

The next step would then be to create a bash script, then to evolve it to a Makefile or similar solution. Fortunately, Docker provides a better solution with Docker Compose.

Docker Compose can be described as a single-host orchestration tool. Orchestration tools are pieces of software that allow us to deal with the problems described previously, such as starting and terminating multiple containers, creating networks and volumes, managing secrets, and so on. Docker Compose works in a single-host mode, so it's a great solution for development environment, while for production multi-host environments it's better to move to more advanced tools such as AWS ECS or Kubernetes.

Docker Compose reads the configuration of a system from the file docker-compose.yml (the default value, it can be changed) that captures all we did manually in the previous sections in a compact and readable way.

To install Docker Compose follow the instructions you find at https://docs.docker.com/compose/install/. Before we start using Docker Compose make sure you kill the Postgres container if you are still running it, and remove the network we created

$ docker rm -f whale-postgres 
whale-postgres
$ docker network remove whale
whale

Then create the file docker-compose.yml in the project directory (not the app directory) and put the following code in it

docker-compose.yml

version: '3.8'

services:

This is not a valid Docker Compose file, yet, but you can see that there is a value that specifies the syntax version and one that lists services. You can find the Compose file reference at https://docs.docker.com/compose/compose-file/, together with a detailed description of the various versions.

The first service we want to run is Postgres, and a basic configuration for that is

docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes: 2
      - dbdata:/var/lib/postgresql/data

volumes: 1
  dbdata:

As you can see, this file contains the environment variables that we passed to the Postgres container and the volume configuration. The final volumes 1 declares which volumes have to be present (so it creates them if they are not), while volumes 2 inside the service db creates the connection just like the option -v did previously.

Now, from the project directory, you can run Docker Compose with

$ docker-compose -p whale up -d
Creating network "whale_default" with the default driver
Creating whale_db_1 ... done

The option -p sets the name of the project, which otherwise would be by default that of the directory you are at the moment (which might or might not be meaningful), while the command up -d starts all the containers in a detached mode.

As you can see from the output, Docker Compose creates a (bridge) network called whale_default. Normally, you would see a message like Creating volume "whale_dbdata" with default driver as well, but in this case the volume is already present as we created it previously. Both the network and the volume are prefixed with PROJECTNAME_, and this is the reason why when we first created the volume I named it whale_dbdata. Keep in mind however that all these default behaviours can be customised in the Compose file.

If you run docker ps you will see that the container is named whale_db_1. This comes from the project name (whale_), the service name in the Compose file (db_) and the container number, which is 1 because at the moment we are running only one container for that service.

To stop the services you have to run

$ docker-compose -p whale down
Stopping whale_db_1 ... done
Removing whale_db_1 ... done
Removing network whale_default

As you can see from the output, Docker Compose stops and removes the container, then removes the network. This is very convenient, as it already removes a lot of the work we had to do manually earlier.

We can now add the application container to the Compose file

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes:
      - dbdata:/var/lib/postgresql/data
  app:
    build:
      context: whaleapp
      dockerfile: Dockerfile
    environment:
      WHALEAPP__DB_HOST: db
      WHALEAPP__DB_NAME: whale_db
      WHALEAPP__DB_USER: whale_user
      WHALEAPP__DB_PASSWORD: whale_password

volumes:
  dbdata:

This definition is slightly different, as the application container has to be built using the Dockerfile we created. Docker Compose allows us to store here the build configuration so that we don't need to pass al the options to docker build manually, but please note that configuring the build here doesn't mean that Docker Compose will build the image for you every time. You still need to run docker-compose -p whale build every time you need to rebuild it.

Please note that the variable WHALEAPP__DB_HOST is set to the service name, and not to the container name. Now, when we run Docker Compose we get

$ docker-compose -p whale up -d
Creating network "whale_default" with the default driver
Creating whale_db_1  ... done
Creating whale_app_1 ... done

and the output tells us that also the container whale_app_1 has been created this time. We can see the logs of a container with docker logs, but using docker-compose allows us to call services by name instead of by ID

$ docker-compose -p whale logs -f app
Attaching to whale_app_1
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.

Health checks and dependencies¶

You might have noticed that at the very beginning of the application logs there are some connection errors, and that after a while the application manages to connect to the database

$ docker-compose -p whale logs -f app
Attaching to whale_app_1
app_1  | Connecting to the PostgreSQL database...
app_1  | could not translate host name "db" to address: Name or service not known
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | could not translate host name "db" to address: Name or service not known
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | Connecting to the PostgreSQL database...
app_1  | could not connect to server: Connection refused
app_1  |        Is the server running on host "db" (172.31.0.3) and accepting
app_1  |        TCP/IP connections on port 5432?
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.

These errors come from the fact that the application container is up and running before the database is ready to serve connections. In a production setup this usually doesn't happen because the database is up and running much before the application gets deployed for the first time, and then runs (hopefully) without interruption. In a development environment, instead, such a situation is normal.

Please note that this might not happen in your setup, as this is tightly connected with the speed of Docker Compose and the containers. Time-sensitive bugs are one of the worst types to deal with, and this is the reason why managing distributed systems is hard. It is important that you realise that even though this might work now on your system, the problem is there and we need to find a solution.

The standard solution when part of a system depends on another is to create a health check that periodically tests the first service, and to start the second service only when the check is successful. We can do this in the Compose file using healthcheck and depends_on

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes:
      - dbdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 5
  app:
    build:
      context: whaleapp
      dockerfile: Dockerfile
    environment:
      WHALEAPP__DB_HOST: db
      WHALEAPP__DB_NAME: whale_db
      WHALEAPP__DB_USER: whale_user
      WHALEAPP__DB_PASSWORD: whale_password
    depends_on:
      db:
        condition: service_healthy

volumes:
  dbdata:

The health check for the Postgres container leverages the command line tool pg_isready that is successful only when the database is ready to accept connections, and tries every 10 seconds for 5 times. Now, when you run up -d this time you should notice a clear delay before the application is run, but the logs won't contain any connection error.

Final words¶

Well, this was a long one, but I hope you enjoyed the trip and you ended up having a better picture of what problems Docker Compose solve, along with a feeling of how complicated it might be to design an architecture. Everything we did was for a "simple" development environment with a couple of containers, so you can figure what is involved when we get to live environments.

Updates¶

2022-03-17: Thanks to my colleague Joanna Stadnik for a thorough review, for spotting typos, and for giving me several suggestions based on her experience. Thank you!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Cover picture by Verstappen Photography on Unsplash.

Flask project setup: TDD, Docker, Postgres and more - Part 3

2020-07-07T13:00:00+01:00

In this series of posts I explore the development of a Flask project with a setup that is built with efficiency and tidiness in mind, using TDD, Docker and Postgres.

Catch-up¶

In the first and second posts I created a Flask project with a tidy setup, using Docker to run the development environment and the tests, and mapping important commands in a management script, so that the configuration can be in a single file and drive the whole system.

In this post I will show you how to easily create scenarios, that is databases created on the fly with custom data, so that it is possible to test queries in isolation, either with the Flask application or with the command line. I will also show you how to define a configuration for production and give some hints for the deployment.

Step 1 - Creating scenarios¶

The idea of scenarios is simple. Sometimes you need to investigate specific use cases for bugs, or maybe increase the performances of some database queries, and you might need to do this on a customised database. This is a scenario, a Python file that populates the database with a specific set of data and that allows you to run the application or the database shell on it.

Often the development database is a copy of the production one, maybe with sensitive data stripped to avoid leaking private information, and while this gives us a realistic case where to test queries (e.g. how does the query perform on 1 million lines?) it might not help during the initial investigations, where you need to have all the data in front of you to properly understand what happens. Whoever learned how joins work in relational databases understands what I mean here.

In principle, to create a scenario we just need to spin up an empty database and to run the scenario code against it. In practice, things are not much more complicated, but there are a couple of minor issues that we need to solve.

First, I am already running a database for the development and one for the testing. The second is ephemeral, but I decided to setup the project so that I can run the tests while the development database is up, and the way I did it was using port 5432 (the standard Postgres one) for development and 5433 for testing. Spinning up scenarios adds more databases to the equation. Clearly I do not expect to run 5 scenarios at the same time while running the development and the test databases, but I make myself a rule to make something generic as soon I do it for the third time.

This means that I won't create a database for a scenario on port 5434 and will instead look for a more generic solution. This is offered me by the Docker networking model, where I can map a container port to the host but avoid assigning the destination port, and it will be chosen randomly by Docker itself among the unprivileged ones. This means that I can create a Postgres container mapping port 5432 (the port in the container) and having Docker connect it to port 32838 in the host (for example). As long as the application knows which port to use this is absolutely the same as using port 5432.

Unfortunately the Docker interface is not extremely script-friendly when it comes to providing information and I have to parse the output a bit. Practically speaking, after I spin up the containers, I will run the command docker-compose port db 5432 which will return a string like 0.0.0.0:32838, and I will extract the port from it. Nothing major, but these are the (sometimes many) issues you face when you orchestrate different systems together.

The new management script is

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess
import time
import shutil

import click
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development")

APPLICATION_CONFIG_PATH = "config"
DOCKER_PATH = "docker"


def app_config_file(config):
    return os.path.join(APPLICATION_CONFIG_PATH, f"{config}.json")


def docker_compose_file(config):
    return os.path.join(DOCKER_PATH, f"{config}.yml")


def configure_app(config):
    # Read configuration from the relative JSON file
    with open(app_config_file(config)) as f:
        config_data = json.load(f)

    # Convert the config into a usable Python dictionary
    config_data = dict((i["name"], i["value"]) for i in config_data)

    for key, value in config_data.items():
        setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand):
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def docker_compose_cmdline(commands_string=None):
    config = os.getenv("APPLICATION_CONFIG")
    configure_app(config)

    compose_file = docker_compose_file(config)

    if not os.path.isfile(compose_file):
        raise ValueError(f"The file {compose_file} does not exist")

    command_line = [
        "docker-compose",
        "-p",
        config,
        "-f",
        compose_file,
    ]

    if commands_string:
        command_line.extend(commands_string.split(" "))

    return command_line


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def compose(subcommand):
    cmdline = docker_compose_cmdline() + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def run_sql(statements):
    conn = psycopg2.connect(
        dbname=os.getenv("POSTGRES_DB"),
        user=os.getenv("POSTGRES_USER"),
        password=os.getenv("POSTGRES_PASSWORD"),
        host=os.getenv("POSTGRES_HOSTNAME"),
        port=os.getenv("POSTGRES_PORT"),
    )

    conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cursor = conn.cursor()
    for statement in statements:
        cursor.execute(statement)

    cursor.close()
    conn.close()


def wait_for_logs(cmdline, message):
    logs = subprocess.check_output(cmdline)
    while message not in logs.decode("utf-8"):
        time.sleep(0.1)
        logs = subprocess.check_output(cmdline)


@cli.command()
def create_initial_db():
    configure_app(os.getenv("APPLICATION_CONFIG"))

    try:
        run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])
    except psycopg2.errors.DuplicateDatabase:
        print(
            f"The database {os.getenv('APPLICATION_DB')} already exists and will not be recreated"
        )


@cli.command()
@click.argument("filenames", nargs=-1)
def test(filenames):
    os.environ["APPLICATION_CONFIG"] = "testing"
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = docker_compose_cmdline("up -d")
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline("logs db")
    wait_for_logs(cmdline, "ready to accept connections")

    run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])

    cmdline = ["pytest", "-svv", "--cov=application", "--cov-report=term-missing"]
    cmdline.extend(filenames)
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline("down")
    subprocess.call(cmdline)


@cli.group()
def scenario():
    pass


@scenario.command()
@click.argument("name")
def up(name): 1
    os.environ["APPLICATION_CONFIG"] = f"scenario_{name}"
    config = os.getenv("APPLICATION_CONFIG")

    scenario_config_source_file = app_config_file("scenario")
    scenario_config_file = app_config_file(config)

    if not os.path.isfile(scenario_config_source_file):
        raise ValueError(f"File {scenario_config_source_file} doesn't exist")
    shutil.copy(scenario_config_source_file, scenario_config_file) 3

    scenario_docker_source_file = docker_compose_file("scenario")
    scenario_docker_file = docker_compose_file(config)

    if not os.path.isfile(scenario_docker_source_file):
        raise ValueError(f"File {scenario_docker_source_file} doesn't exist")
    shutil.copy(docker_compose_file("scenario"), scenario_docker_file) 4

    configure_app(f"scenario_{name}")

    cmdline = docker_compose_cmdline("up -d") 5
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline("logs db")
    wait_for_logs(cmdline, "ready to accept connections")

    cmdline = docker_compose_cmdline("port db 5432") 6
    out = subprocess.check_output(cmdline)
    port = out.decode("utf-8").replace("\n", "").split(":")[1]
    os.environ["POSTGRES_PORT"] = port

    run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])

    scenario_module = f"scenarios.{name}"
    scenario_file = os.path.join("scenarios", f"{name}.py")
    if os.path.isfile(scenario_file): 7
        import importlib

        os.environ["APPLICATION_SCENARIO_NAME"] = name

        scenario = importlib.import_module(scenario_module)
        scenario.run()

    cmdline = " ".join( 8
        docker_compose_cmdline(
            "exec db psql -U {} -d {}".format(
                os.getenv("POSTGRES_USER"), os.getenv("APPLICATION_DB")
            )
        )
    )
    print("Your scenario is ready. If you want to open a SQL shell run")
    print(cmdline)


@scenario.command()
@click.argument("name")
def down(name): 2
    os.environ["APPLICATION_CONFIG"] = f"scenario_{name}"
    config = os.getenv("APPLICATION_CONFIG")

    cmdline = docker_compose_cmdline("down")
    subprocess.call(cmdline)

    scenario_config_file = app_config_file(config)
    os.remove(scenario_config_file)

    scenario_docker_file = docker_compose_file(config)
    os.remove(scenario_docker_file)


if __name__ == "__main__":
    cli()

where I added the commands scenario up 1 and scenario down 2. As you can see the function up first copies the files config/scenario.json 3 and docker/scenario.yml 4 (that I still have to create) into files named after the scenario.

Then I run the command up -d 5 and wait for the database to be ready, as I already do for tests. After that, it's time to extract the port of the container with some very simple Python string processing 6 and to initialise the correct environment variable.

Last, I import and execute the Python file 7 containing the code of the scenario itself and print a friendly message with the command line to run psql 8 to have a Postgres shell into the newly created database.

The function down simply tears down the containers and removes the scenario configuration files.

The two missing config files are pretty simple. The docker compose configuration is

docker/scenario.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "5432" 1
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
      APPLICATION_DB: ${APPLICATION_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_HOSTNAME: "db"
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    command: flask run --host 0.0.0.0
    volumes:
      - ${PWD}:/opt/code
    ports:
      - "5000"

Here you can see that the database is ephemeral, that the port on the host is automatically assigned 1, and that I also spin up the application (mapping it to a random port as well to avoid clashing with the development one).

The configuration file is

config/scenario.json

[
  {
    "name": "FLASK_ENV",
    "value": "development"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "development"
  },
  {
    "name": "POSTGRES_DB",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_USER",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_HOSTNAME",
    "value": "localhost"
  },
  {
    "name": "POSTGRES_PASSWORD",
    "value": "postgres"
  },
  {
    "name": "APPLICATION_DB",
    "value": "application"
  }
]

which doesn't add anything new to what I already did for development and testing.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Expose ports in docker-compose
Docker Compose port command - A command to print the port exposed by a container
psql - PostgreSQL interactive terminal

Scenario example 1

Let's have a look at a very simple scenario that doesn't do anything on the database, just to understand the system. The code for the scenario is

scenarios/foo.py

import os


def run():
    print("HEY! This is scenario", os.environ["APPLICATION_SCENARIO_NAME"])

When I run the scenario I get the following output

$ ./manage.py scenario up foo
Creating network "scenario_foo_default" with the default driver
Creating scenario_foo_db_1  ... done
Creating scenario_foo_web_1 ... done
HEY! This is scenario foo
Your scenario is ready. If you want to open a SQL shell run
docker-compose -p scenario_foo -f docker/scenario_foo.yml exec db psql -U postgres -d application

The command docker ps shows that my development environment is happily running alongside with the scenario

$ docker ps
CONTAINER ID  IMAGE             COMMAND                 [...]  PORTS                    NAMES
85258892a2df  scenario_foo_web  "flask run --host 0.…"  [...]  0.0.0.0:32826->5000/tcp  scenario_foo_web_1
a031b6429e07  postgres          "docker-entrypoint.s…"  [...]  0.0.0.0:32827->5432/tcp  scenario_foo_db_1
1a449d23da01  development_web   "flask run --host 0.…"  [...]  0.0.0.0:5000->5000/tcp   development_web_1
28aa566321b5  postgres          "docker-entrypoint.s…"  [...]  0.0.0.0:5432->5432/tcp   development_db_1

And the output of the command scenario up foo contains the string HEY! This is scenario foo that was printed by the file foo.py. We can also successfully run the suggested command

$ docker-compose -p scenario_foo -f docker/scenario_foo.yml exec db psql -U postgres -d application
psql (12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.

application=# \l
                                  List of databases
    Name     |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-------------+----------+----------+------------+------------+-----------------------
 application | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres    | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
 template1   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
(4 rows)

application=#

And inside the database we find the database application created explicitly for the scenario (the name is specified in config/scenario.json). If you don't know psql you can exit with \q or Ctrl-d.

Before tearing down the scenario have a look at the two files config/scenario_foo.json and docker/scenario_foo.yml. They are just copies of config/scenario.json and docker/scenario.yml but I think seeing them there might help to understand how the whole thing works. When you are done run ./manage.py scenario down foo.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Scenario example 2

Let's do something a bit more interesting. The new scenario is contained in scenarios/users.py

scenarios/users.py

from application.app import create_app
from application.models import db, User


app = create_app("development") 1


def run():
    with app.app_context():
        db.drop_all()
        db.create_all()

        # Administrator
        admin = User(email="admin@server.com")
        db.session.add(admin) 2

        # First user
        user1 = User(email="user1@server.com")
        db.session.add(user1)

        # Second user
        user2 = User(email="user2@server.com")
        db.session.add(user2)

        db.session.commit()

I decided to be as agnostic as possible in the scenarios, to avoid creating something too specific that eventually would not give me enough flexibility to test what I need. This means that the scenario has to create the app 1 and to use the database session explicitly 2, as I do in this example. The application is created with the configuration "development" 1. Remember that this is the Flask configuration that you find in application/config.py, not the one that is in config/development.json.

I can run the scenario with

$ ./manage.py scenario up users

and then connect to the database to find my users

$ docker-compose -p scenario_users -f docker/scenario_users.yml exec db psql -U postgres -d application
psql (12.3 (Debian 12.3-1.pgdg100+1))
Type "help" for help.

application=# \dt
         List of relations
 Schema | Name  | Type  |  Owner
--------+-------+-------+----------
 public | users | table | postgres
(1 row)

application=# select * from users;
 id |      email
----+------------------
  1 | admin@server.com
  2 | user1@server.com
  3 | user2@server.com
(3 rows)

application=# \q

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Step 2 - Simulating the production environment¶

As I stated at the very beginning of this mini series of posts, one of my goals was to run in development the same database that I run in production, and for this reason I went through the configuration steps that allowed me to have a Postgres container running both in development and during tests. In a real production scenario Postgres would probably run in a separate instance, for example on the RDS service in AWS, but as long as you have the connection parameters nothing changes in the configuration.

Docker actually allows us to easily simulate the production environment. If our notebook was connected 24/7 we might as well host the production there directly. Not that I recommend this nowadays, but this is how many important companies begun many years ago when cloud computing had not been here yet. Instead of installing a LAMP stack we configure containers, but the idea doesn't change.

I will then create a configuration that simulates a production environment and then give some hints on how to translate this into a proper production infrastructure. If you want to have a clear picture of the components of a web application in production read my post Dissecting a web stack that analyses them one by one.

The first component that we have to change here is the HTTP server. In development we use Flask's development server, and the first message that server prints is WARNING: This is a development server. Do not use it in a production deployment. Got it, Flask! A good choice to replace it is Gunicorn, so first of all I add it in the requirements

requirements/production.txt

Flask
flask-sqlalchemy
psycopg2
flask-migrate
gunicorn

Then I need to create a docker-compose configuration for production

docker/production.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${POSTGRES_PORT}:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile.production
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
      APPLICATION_DB: ${APPLICATION_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_HOSTNAME: "db"
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_PORT: ${POSTGRES_PORT}
    command: gunicorn -w 4 -b 0.0.0.0 wsgi:app 1
    volumes:
      - ${PWD}:/opt/code
    ports:
      - "8000:8000" 2

volumes:
  pgdata:

As you can see here the command that runs the application is slightly different 1. It exposes 4 processes (-w 4) on the container's address 0.0.0.0 loading the object app from the file wsgi.py (wsgi:app). As by default Gunicorn exposes port 8000 I mapped that 2 to the same port in the host.

Then I created the file Dockerfile.production that defines the production image of the web application

docker/Dockerfile.production

FROM python:3

ENV PYTHONUNBUFFERED 1

RUN mkdir /opt/code
RUN mkdir /opt/requirements
WORKDIR /opt/code

ADD requirements /opt/requirements
RUN pip install -r /opt/requirements/production.txt

The last thing I need is a configuration file

config/production.json

[
  {
    "name": "FLASK_ENV",
    "value": "production"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "production"
  },
  {
    "name": "POSTGRES_DB",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_USER",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_HOSTNAME",
    "value": "localhost"
  },
  {
    "name": "POSTGRES_PORT",
    "value": "5432"
  },
  {
    "name": "POSTGRES_PASSWORD",
    "value": "postgres"
  },
  {
    "name": "APPLICATION_DB",
    "value": "application"
  }
]

as you can notice this is not very different from the development one, as I just changed the values of FLASK_ENV and FLASK_CONFIG. Clearly this contains a secret that shouldn't be written in plain text, POSTGRES_PASSWORD, but after all this is a simulation of production. In a real environment secrets should be kept in an encrypted manager such as AWS Secrets Manager.

Remember that FLASK_ENV changes the internal settings of Flask, most notably disabling the debugger, and that FLASK_CONFIG=production loads the object ProductionConfig from application/config.py. That object is empty for the moment, but it might contain public configuration for the production server.

I can now build the image with

$ APPLICATION_CONFIG="production" ./manage.py compose build web

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Gunicorn - A Python WSGI HTTP Server

Step 3 - Scale up¶

Mapping the container port to the host is not a great idea, though, as it makes it impossible to scale up and down to serve more load, which is the main point of running containers in production. This might be solved in many ways in the cloud, for example in AWS you might run the container in AWS Fargate and register them in an Application Load Balancer. Another way to do it on a single host is to run a Web Server in front of your HTTP server, and this might be easily implemented with Docker Compose.

I will add nginx and serve HTTP from there, reverse proxying the application containers through docker-compose networking. First of all the new configuration for docker-compose

docker/production.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${POSTGRES_PORT}:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile.production
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
      APPLICATION_DB: ${APPLICATION_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_HOSTNAME: "db"
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_PORT: ${POSTGRES_PORT}
    command: gunicorn -w 4 -b 0.0.0.0 wsgi:app
    volumes:
      - ${PWD}:/opt/code
  nginx:
    image: nginx
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    ports:
      - 8080:8080

volumes:
  pgdata:

As you can see I added a service nginx that runs the default Nginx image, mapping a custom configuration file that I will create in a minute. The application container doesn't need any port mapping, as I won't access it directly from the host anymore. The Nginx configuration file is

docker/nginx/nginx.conf

worker_processes 1;

events { worker_connections 1024; }

http {

    sendfile on;

    upstream app { 1
        server web:8000;
    }

    server {
        listen 8080; 2

        location / {
            proxy_pass         http://app;
            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Host $server_name;
        }
    }
}

This is a pretty standard configuration, and in a real production environment I would add many other configuration values (most notably serving HTTPS instead of HTTP). The section upstream 1 leverages docker-compose networking referring to web, which in the internal DNS directly maps to the IPs of the service with the same name. The port 8000 comes from the default Gunicorn port that I already mentioned before. I won't run the nginx container as root on my notebook, so I expose port 8080 2 instead of the traditional 80 for HTTP, and this is also something that might be different in a real production environment.

I can at this point run

$ APPLICATION_CONFIG="production" ./manage.py compose up -d
Starting production_db_1    ... done
Starting production_nginx_1 ... done
Starting production_web_1   ... done

It's interesting to have a look at the logs of the nginx container, as Nginx by default prints all the incoming requests

$ APPLICATION_CONFIG="production" ./manage.py compose logs -f nginx
Attaching to production_nginx_1
[...]
nginx_1  | 172.30.0.1 - - [05/Jul/2020:10:40:44 +0000] "GET / HTTP/1.1" 200 13 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"

The last line is what I get when I visit localhost:8080 while the production setup is up and running.

Scaling up and down the service is now a breeze

$ APPLICATION_CONFIG="production" ./manage.py compose up -d --scale web=3
production_db_1 is up-to-date
Starting production_web_1 ... 
Starting production_web_1 ... done
Creating production_web_2 ... done
Creating production_web_3 ... done

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Nginx - An HTTP and reverse proxy server (and more)
Docker nginx - the official nginx Docker image
Docker Compose logs command - A command to print container logs
Docker Compose up command - The new way to scale containers in docker-compose

Bonus step - A closer look at Docker networking¶

I mentioned that Docker Compose creates a connection between services, and used that in the configuration of the nginx container, but I understand that this might look like black magic to some people. While I believe that this is actually black magic, I also think that we can investigate it a bit, so let's open the grimoire and reveal (some of) the dark secrets of Docker networking.

While the production setup is running we can connect to the nginx container and see what is happening in real time, so first of all I run a bash shell on it

$ APPLICATION_CONFIG="production" ./manage.py compose exec nginx bash

Once inside I can see my configuration file at /etc/nginx/nginx.conf, but this has not changed. Remember that Docker networking doesn't work as a templating engine, but with a local DNS. This means that if we try to resolve web from inside the container we should see multiple IPs. The command dig is a good tool to investigate the DNS, but it doesn't come preinstalled in the nginx container, so I need to run

root@33cbaea369be:/# apt update && apt install dnsutils

and at this point I can run it

root@33cbaea369be:/# dig web

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> web
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30539
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;web.                           IN      A

;; ANSWER SECTION:
web.                    600     IN      A       172.30.0.4
web.                    600     IN      A       172.30.0.6
web.                    600     IN      A       172.30.0.5

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Sun Jul 05 10:58:18 UTC 2020
;; MSG SIZE  rcvd: 78

root@33cbaea369be:/#

The command outputs 3 IPs, which correspond to the 3 containers of the service web that I am currently running. If I scale down (from outside the container)

$ APPLICATION_CONFIG="production" ./manage.py compose up -d --scale web=1

then the output of dig becomes

root@33cbaea369be:/# dig web

; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> web
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13146
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;web.                           IN      A

;; ANSWER SECTION:
web.                    600     IN      A       172.30.0.4

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11)
;; WHEN: Sun Jul 05 11:01:46 UTC 2020
;; MSG SIZE  rcvd: 40

root@33cbaea369be:/#

How to create the production infrastructure¶

This will be a very short section, as creating infrastructure and deploying in production are complex topics, so I want to just give some hints to stimulate your research.

AWS ECS is basically Docker in the cloud, and the whole structure can map almost 1 to 1 to the docker-compose setup, so it is worth learning. ECS can work on explicit EC2 instances that you manage, or in Fargate, which means that the EC2 instances running the containers are transparently managed by AWS itself.

Terraform is a good tool to create infrastructure. It has many limitations, mostly coming from its custom HCL language, but it's slowly becoming better (version 0.13 will finally allow us to run for loops on modules, for example). Despite its shortcomings, it's a great tool to create static infrastructure, so I recommend working on it.

Terraform is not the right tool to deploy your code, though, as that requires a dynamic interaction with the system, so you need to setup a good Continuous Integration system. Jenkins is a very well known open source CI, but I personally ended up dropping it because it doesn't seem to be designed for large scale systems. For example, it is very complicated to automate the deploy of a Jenkins server, and dynamic large scale systems should require zero manual intervention to be created. Anyway, Jenkins is a good tool to start with, but you might want to have a look at other products like CircleCI or Buildkite.

When you create your deploy pipeline you need to do much more than just creating the image and running it, at least for real applications. You need to decide when to apply database migrations and if you have a web front-end you will also need to compile and install the JavaScript assets. Since you don't want to have downtime when you deploy you will need to look into blue/green deployments, and in general to strategies that allow you to run different versions of the application at the same time, at least for short periods of time. Or for longer periods, if you want to perform A/B testing or zonal deployments.

Final words¶

This is the last post of this short series. I hope you learned something useful, and that it encouraged you to properly setup your projects and to investigate technologies like Docker. As always, feel free to send me feedback or questions, and if you find my posts useful please share them with whoever you thing might be interested.

Updates¶

2020-12-22 I reviewed the whole tutorial and corrected several typos

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Flask project setup: TDD, Docker, Postgres and more - Part 2

2020-07-06T13:00:00+01:00

In this series of posts I explore the development of a Flask project with a setup that is built with efficiency and tidiness in mind, using TDD, Docker and Postgres.

Catch-up¶

In the previous post we started from an empty project and learned how to add the minimal code to run a Flask project. Then we created a static configuration file and a management script that wraps the commands flask and docker-compose to run the application with a specific configuration.

In this post I will show you how to run a production-ready database alongside your code in a Docker container, both in your development setup and for the tests.

Step 1 - Adding a database container¶

A database is an integral part of a web application, so in this step I will add my database of choice, Postgres, to the project setup. To do this I need to add a service in the docker-compose configuration file

docker/development.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB} 1
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${POSTGRES_PORT}:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data 2
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
    command: flask run --host 0.0.0.0
    volumes:
      - ${PWD}:/opt/code
    ports:
      - "5000:5000"

volumes:
  pgdata: 3

The variables starting with POSTGRES_ are requested by the PostgreSQL Docker image. In particular, remember that POSTGRESQL_DB 1 is the database that gets created by default when you create the image, and also the one that contains data on other databases as well, so for the application we usually want to use a different one.

Notice also that I'm creating a persistent volume for the service db 2 3, so that the content of the database is not lost when we tear down the container. For this service I'm using the default image, so no build step is needed.

To orchestrate this setup we need to add those variables to the JSON configuration

config/development.json

[
  {
    "name": "FLASK_ENV",
    "value": "development"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "development"
  },
  {
    "name": "POSTGRES_DB",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_USER",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_HOSTNAME", 1
    "value": "localhost"
  },
  {
    "name": "POSTGRES_PORT",
    "value": "5432"
  },
  {
    "name": "POSTGRES_PASSWORD",
    "value": "postgres"
  }
]

These are all development variables so there are no secrets. In production we will need a way to keep the secrets in a safe place and convert them into environment variables. The AWS Secrets Manager for example can directly map secrets into environment variables passed to the containers, saving you from having to explicitly connect to the service with the API.

You may have noticed the variable POSTGRES_HOSTNAME 1, which is not used in the Docker Compose file. Generally we want the database to be accessible by utility scripts, so we want to record the host name where the database is running. As we will see shortly, other Docker containers do not need this, but migrations will.

We can run the commands ./manage.py compose up -d and ./manage.py compose down here to check that the database container works properly. Please note that the first time you run the command compose -d Docker will create the volume and build the Postgres image, and this might take some time.

CONTAINER ID  IMAGE       COMMAND                 ...  PORTS                   NAMES
9b5828dccd1c  docker_web  "flask run --host 0.…"  ...  0.0.0.0:5000->5000/tcp  docker_web_1
4440a18a1527  postgres    "docker-entrypoint.s…"  ...  0.0.0.0:5432->5432/tcp  docker_db_1

Now we need to connect the application to the database and to do this we can leverage flask-sqlalchemy. As we will use this at every stage of the life of the application, the requirement goes among the production ones. We also need psycopg2 as it is the library used to connect to Postgres.

requirements/production.txt

Flask
flask-sqlalchemy
psycopg2

Remember to run pip install -r requirements/development.txt to install the requirements locally and ./manage.py compose build web to rebuild the image.

At this point I need to create a connection string in the configuration of the application. The connection string parameters come from the same environment variables used to spin up the container db

application/config.py

import os

class Config(object):
    """Base configuration"""

    user = os.environ["POSTGRES_USER"]
    password = os.environ["POSTGRES_PASSWORD"]
    hostname = os.environ["POSTGRES_HOSTNAME"]
    port = os.environ["POSTGRES_PORT"]
    database = os.environ["APPLICATION_DB"] 1

    SQLALCHEMY_DATABASE_URI = (
        f"postgresql+psycopg2://{user}:{password}@{hostname}:{port}/{database}"
    )
    SQLALCHEMY_TRACK_MODIFICATIONS = False


class ProductionConfig(Config):
    """Production configuration"""


class DevelopmentConfig(Config):
    """Development configuration"""


class TestingConfig(Config):
    """Testing configuration"""

    TESTING = True

As you can see, here I use the variable APPLICATION_DB 1 and not POSTGRES_DB, so I need to specify that as well in the config file. The reason, as I mentioned before, is that we prefer to separate the default database, used by Postgres to manage all other databases, from the one used specifically by our application.

config/development.json

[
  {
    "name": "FLASK_ENV",
    "value": "development"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "development"
  },
  {
    "name": "POSTGRES_DB",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_USER",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_HOSTNAME",
    "value": "localhost"
  },
  {
    "name": "POSTGRES_PORT",
    "value": "5432"
  },
  {
    "name": "POSTGRES_PASSWORD",
    "value": "postgres"
  },
  {
    "name": "APPLICATION_DB",
    "value": "application"
  }
]

At this point the application container needs to access some of the Postgres environment variables and the APPLICATION_DB one

docker/development.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${POSTGRES_PORT}:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
      APPLICATION_DB: ${APPLICATION_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_HOSTNAME: "db"
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_PORT: ${POSTGRES_PORT}
    command: flask run --host 0.0.0.0
    volumes:
      - ${PWD}:/opt/code
    ports:
      - "5000:5000"

volumes:
  pgdata:

Please note that the container web receives the environment variables we pass to the Postgres container because it requires them to connect to the db. The variable POSTGRES_HOSTNAME is passed to give the application the address of the database, and thanks to Docker Compose internal DNS we can simply pass the name of the container. We could not pass the value localhost, as the application, which is running in a container, cannot access the host through that address (unless we use other network modes, which is not ideal).

Running compose now spins up both Flask and Postgres, but the application is not properly connected to the database yet.

Let's have a look inside the DB to see what our configuration created. First run ./manage.py compose up -d to spin up the containers, then connect to the Postgres DB with ./manage.py compose exec db psql -U postgres. Please note that we have to specify the user with -U. The default value is root, but we changed it to postgres with the variable POSTGRES_USER.

You should see a command line like

$ ./manage.py compose exec db psql -U postgres
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.

postgres=#

Also note that by default we are logging into the database called postgres, which was configured by the variable POSTGRES_DB. We can list the databases with \l

postgres=# \l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(3 rows)

postgres=#

Last, note that the application database configured with APPLICATION_DB is not present because we haven't created it yet. All the environment variables prefixed by POSTGRES_ are used automatically by the Docker image to perform the initial configuration, which is why the database postgres is already there.

You can exit psql with Ctrl-D or exit.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Postgres Docker image
Flask-SQLAlchemy - A Flask extension to work with SQLAlchemy
SQLAlchemy and PostgreSQL - The Python SQL Toolkit and ORM

Step 2 - Connecting the application and the database¶

To connect the Flask application with the database running in the container we need to initialise a SQLAlchemy object and add it to the application factory.

application/models.py

from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

application/app.py

from flask import Flask


def create_app(config_name):

    app = Flask(__name__)

    config_module = f"application.config.{config_name.capitalize()}Config"

    app.config.from_object(config_module)

    from application.models import db

    db.init_app(app)

    @app.route("/")
    def hello_world():
        return "Hello, World!"

    return app

A pretty standard way to manage the database in Flask is to use flask-migrate, that adds some commands that allow us to create migrations and apply them.

With flask-migrate you have to create the migrations folder once and for all with flask db init and then, every time you change your models, run flask db migrate -m "Some message" and flask db upgrade. As both db init and db migrate create files in the current directory we now face a problem that every Docker-based setup has to face: file permissions.

The situation is the following: the application is running in the Docker container as root, and there is no connection between the users namespace in the container and that of the host. The result is that if the Docker container creates files in a directory that is mounted from the host (like the one that contains the application code in our example), those files will result as belonging to root. While this doesn't make impossible to work (we usually can become root on our development machines), it is annoying to say the least. The solution is to run those commands from outside the container, but this requires the Flask application to be configured.

Fortunately I wrapped the command flask in the script manage.py, which loads all the required environment variables. Let's add flask-migrate to the production requirements

requirements/production.txt

Flask
flask-sqlalchemy
psycopg2
flask-migrate

Remember to run pip install -r requirements/development.txt to install the requirements locally and ./manage.py compose build web to rebuild the image. Please note that you need the executable pg_config and some other development tools installed in your system. If you get an error message from pip please check the documentation of your operating system to find out what to do to install the required packages. For Ubuntu Linux I had to run sudo apt install build-essential python3-dev libpq-dev.

Now we can initialise a Migrate object and add it to the application factory

application/models.py

from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate

db = SQLAlchemy()
migrate = Migrate()

application/app.py

from flask import Flask


def create_app(config_name):

    app = Flask(__name__)

    config_module = f"application.config.{config_name.capitalize()}Config"

    app.config.from_object(config_module)

    from application.models import db, migrate

    db.init_app(app)
    migrate.init_app(app, db)

    @app.route("/")
    def hello_world():
        return "Hello, World!"

    return app

I can now run the database initialisation script

$ ./manage.py flask db init
  Creating directory /home/leo/devel/flask-tutorial/migrations ...  done
  Creating directory /home/leo/devel/flask-tutorial/migrations/versions ...  done
  Generating /home/leo/devel/flask-tutorial/migrations/env.py ...  done
  Generating /home/leo/devel/flask-tutorial/migrations/README ...  done
  Generating /home/leo/devel/flask-tutorial/migrations/script.py.mako ...  done
  Generating /home/leo/devel/flask-tutorial/migrations/alembic.ini ...  done
  Please edit configuration/connection/logging settings in '/home/leo/devel/flask-tutorial/migrations/alembic.ini' before proceeding.

And, when we will start creating models we will use the commands ./manage.py flask db migrate and ./manage.py flask db upgrade. You will find a complete example at the end of this post.

For the time being let's have a brief look at what was created here. The command db init created the directory migrations and inside it some default configuration files and templates. The migration scripts will be created in the directory migrations/versions but at the moment that directory is empty, as we have no models and we run no migrations (only the initialisation of the system). No changes have been made to the database. The command db init can be run even without running containers (you can remove the directory migrations and try it).

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Flask-SQLAlchemy - A Flask extension to work with SQLAlchemy
Flask-Migrate - A Flask extension to handle database migrations with Alembic.

Step 3 - Testing setup¶

I want to use a TDD approach as much as possible when developing my applications, so I need to setup a good testing environment upfront, and it has to be as ephemeral as possible. It is not unusual in big projects to create (or scale up) infrastructure components explicitly to run tests, and through Docker and docker-compose we can easily do the same. Namely, I will:

Spin up a test database in a container without permanent volumes
Initialise it
Run all the tests against it
Tear down the container

This approach has one big advantage, which is that it requires no previous setup and can this be executed on infrastructure created on the fly. It also has disadvantages, however, as it can slow down the testing part of the application, which should be as fast as possible in a TDD setup. Tests that involve the database, however, should be considered integration tests, and not run continuously in a TDD process, which is impossible (or very hard) when using a framework that merges the concept of entity and database model. If you want to know more about this read the book that I wrote on the subject.

Another advantage of this setup is it that we might need other things during the test, e.g. Celery, other databases, other servers. They can all be created through the docker-compose file.

Generally speaking testing is an umbrella under which many different things can happen. As I will use pytest I can run the full suite, but I might want to select specific tests, mentioning a single file or using the powerful option -k that allows me to select tests by pattern-matching their name. For this reason I want to map the management command line to that of pytest.

Let's add pytest to the testing requirements, along with a couple of packages to monitor the test coverage

requirements/testing.txt

-r production.txt

pytest
coverage
pytest-cov

As you can see I also use the coverage plugin to keep an eye on how well I cover the code with the tests. Remember to run pip install -r requirements/development.txt to install the requirements locally and ./manage.py compose build web to rebuild the image.

Warning: before you change the script manage.py make sure you terminate all the running containers running ./manage.py compose down. The next version will change the naming convention for containers and you might end up with some stale containers and run into issues with the database.

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess
import time

import click


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development") 2


def configure_app(config): 1
    # Read configuration from the relative JSON file
    with open(os.path.join("config", f"{config}.json")) as f:
        config_data = json.load(f)

    # Convert the config into a usable Python dictionary
    config_data = dict((i["name"], i["value"]) for i in config_data)

    for key, value in config_data.items():
        setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand):
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def docker_compose_cmdline(config): 3
    configure_app(os.getenv("APPLICATION_CONFIG"))

    docker_compose_file = os.path.join("docker", f"{config}.yml")

    if not os.path.isfile(docker_compose_file):
        raise ValueError(f"The file {docker_compose_file} does not exist")

    return [
        "docker-compose",
        "-p", 4
        config,
        "-f",
        docker_compose_file,
    ]


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def compose(subcommand):
    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()

@cli.command()
@click.argument("filenames", nargs=-1)
def test(filenames):
    os.environ["APPLICATION_CONFIG"] = "testing" 5
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["up", "-d"]
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["logs", "db"]
    logs = subprocess.check_output(cmdline)
    while "ready to accept connections" not in logs.decode("utf-8"): 6
        time.sleep(0.1)
        logs = subprocess.check_output(cmdline)

    cmdline = ["pytest", "-svv", "--cov=application", "--cov-report=term-missing"]
    cmdline.extend(filenames)
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["down"]
    subprocess.call(cmdline)


if __name__ == "__main__":
    cli()

Notable changes are

The environment configuration code is now in the function configure_app 1. This allows me to force the variable APPLICATION_CONFIG inside the script 2 and then configure the environment, which saves me from having to call tests with APPLICATION_CONFIG=testing flask test.
Both commands flask and compose use the configuration development. Since that is the default value of the variable APPLICATION_CONFIG they just have to call the function configure_app.
The docker-compose command line is needed both in the commands compose and in test, so I isolated some code into a function called docker_compose_cmdline 3 which returns a list as needed by subprocess functions. The command line now uses also the option -p (project name) 4 to give a prefix to the containers. This way we can run tests while running the development server.
The command test forces APPLICATION_CONFIG to be testing 5, which loads the file config/testing.json, then runs docker-compose using the file docker/testing.yml (both file have not been created yet), runs the pytest command line, and tears down the testing database container. Before running the tests the script waits for the service to be available 6. Postgres doesn't allow connection until the database is ready to accept them.

config/testing.json

[
  {
    "name": "FLASK_ENV",
    "value": "production"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "testing"
  },
  {
    "name": "POSTGRES_DB",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_USER",
    "value": "postgres"
  },
  {
    "name": "POSTGRES_HOSTNAME",
    "value": "localhost" 2
  },
  {
    "name": "POSTGRES_PORT",
    "value": "5433" 1
  },
  {
    "name": "POSTGRES_PASSWORD",
    "value": "postgres"
  },
  {
    "name": "APPLICATION_DB",
    "value": "test"
  }
]

Note that here I specified the value 5433 for POSTGRES_PORT 1. This allows us to spin up the test database container while the development one is running, as that will use port 5432 and you can't have two different containers using the same port on the host. A more general solution could be to leave Docker pick a random host port for the container and then use that, but this requires a bit more code to be properly implemented, so I will come back to this problem when setting up the scenarios.

Also note that I set the variable POSTGRES_HOSTNAME to localhost in this file 2. We will run the tests on the local machine and not in a container, so we can't use the DNS provided by Docker Compose.

The last piece of setup that we need is the orchestration configuration for Docker Compose

docker/testing.yml

version: '3.4'

services:
  db:
    image: postgres
    environment:
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    ports:
      - "${POSTGRES_PORT}:5432"

Now we can run ./manage.py test and get

Creating network "testing_default" with the default driver
Creating testing_db_1 ... done
========================== test session starts =========================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
-- /home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir: .pytest_cache
rootdir: /home/leo/devel/flask-tutorial
plugins: cov-2.10.0
collected 0 items
Coverage.py warning: No data was collected. (no-data-collected)


----------- coverage: platform linux, python 3.7.5-final-0 -----------
Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
application/app.py         11     11     0%   1-21
application/config.py      13     13     0%   1-31
application/models.py       4      4     0%   1-5
-----------------------------------------------------
TOTAL                      28     28     0%

======================== no tests ran in 0.07s =======================
Stopping testing_db_1 ... done
Removing testing_db_1 ... done
Removing network testing_default

Note that the command first creates the testing database container testing_db_1, then runs pytest, and finally stops and remove the container. This is exactly what we wanted to achieve to run tests in isolation. At the moment, however there are no tests, and the testing database is empty.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

pytest - A full-featured Python testing framework
Useful pytest command line options

Step 4 - Initialise the testing database¶

When you develop a web application and then run it in production, you typically create the database once and then upgrade it through migrations. When running tests we need to create the database every time, so I need to add a way to run SQL commands on the testing database before I run pytest.

As running sql commands directly on the the database is often useful I will create a function that wraps the boilerplate for the connection. The command that creates the initial database at that point will be trivial.

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess
import time

import click
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development")


def configure_app(config):
    # Read configuration from the relative JSON file
    with open(os.path.join("config", f"{config}.json")) as f:
        config_data = json.load(f)

    # Convert the config into a usable Python dictionary
    config_data = dict((i["name"], i["value"]) for i in config_data)

    for key, value in config_data.items():
        setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand):
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def docker_compose_cmdline(config):
    configure_app(os.getenv("APPLICATION_CONFIG"))

    docker_compose_file = os.path.join("docker", f"{config}.yml")

    if not os.path.isfile(docker_compose_file):
        raise ValueError(f"The file {docker_compose_file} does not exist")

    return [
        "docker-compose",
        "-p",
        config,
        "-f",
        docker_compose_file,
    ]


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def compose(subcommand):
    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def run_sql(statements):
    conn = psycopg2.connect(
        dbname=os.getenv("POSTGRES_DB"),
        user=os.getenv("POSTGRES_USER"),
        password=os.getenv("POSTGRES_PASSWORD"),
        host=os.getenv("POSTGRES_HOSTNAME"),
        port=os.getenv("POSTGRES_PORT"),
    )

    conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cursor = conn.cursor()
    for statement in statements:
        cursor.execute(statement)

    cursor.close()
    conn.close()


@cli.command()
def create_initial_db(): 1
    configure_app(os.getenv("APPLICATION_CONFIG"))

    try:
        run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])
    except psycopg2.errors.DuplicateDatabase:
        print(
            f"The database {os.getenv('APPLICATION_DB')} already exists and will not be recreated"
        )


@cli.command()
@click.argument("filenames", nargs=-1)
def test(filenames):
    os.environ["APPLICATION_CONFIG"] = "testing"
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["up", "-d"]
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["logs", "db"]
    logs = subprocess.check_output(cmdline)
    while "ready to accept connections" not in logs.decode("utf-8"):
        time.sleep(0.1)
        logs = subprocess.check_output(cmdline)

    run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])

    cmdline = ["pytest", "-svv", "--cov=application", "--cov-report=term-missing"]
    cmdline.extend(filenames)
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline(os.getenv("APPLICATION_CONFIG")) + ["down"]
    subprocess.call(cmdline)


if __name__ == "__main__":
    cli()

As you can see I took the opportunity to write the command create_initial_db 1 as well, that just runs the very same SQL command that creates the testing database, but in any configuration I will use.

Before moving on I think it's time to refactor the file manage.py. Refactoring is not mandatory, but I feel like some parts of the script are not generic enough, and when I will add the scenarios I will definitely need my functions to be flexible.

The new script is

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess
import time

import click
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development")

APPLICATION_CONFIG_PATH = "config"
DOCKER_PATH = "docker"


def app_config_file(config): 1
    return os.path.join(APPLICATION_CONFIG_PATH, f"{config}.json")


def docker_compose_file(config): 2
    return os.path.join(DOCKER_PATH, f"{config}.yml")


def configure_app(config):
    # Read configuration from the relative JSON file
    with open(app_config_file(config)) as f:
        config_data = json.load(f)

    # Convert the config into a usable Python dictionary
    config_data = dict((i["name"], i["value"]) for i in config_data)

    for key, value in config_data.items():
        setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand):
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def docker_compose_cmdline(commands_string=None): 4
    config = os.getenv("APPLICATION_CONFIG")
    configure_app(config)

    compose_file = docker_compose_file(config)

    if not os.path.isfile(compose_file):
        raise ValueError(f"The file {compose_file} does not exist")

    command_line = [
        "docker-compose",
        "-p",
        config,
        "-f",
        compose_file,
    ]

    if commands_string:
        command_line.extend(commands_string.split(" "))

    return command_line


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def compose(subcommand):
    cmdline = docker_compose_cmdline() + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


def run_sql(statements):
    conn = psycopg2.connect(
        dbname=os.getenv("POSTGRES_DB"),
        user=os.getenv("POSTGRES_USER"),
        password=os.getenv("POSTGRES_PASSWORD"),
        host=os.getenv("POSTGRES_HOSTNAME"),
        port=os.getenv("POSTGRES_PORT"),
    )

    conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cursor = conn.cursor()
    for statement in statements:
        cursor.execute(statement)

    cursor.close()
    conn.close()


def wait_for_logs(cmdline, message): 3
    logs = subprocess.check_output(cmdline)
    while message not in logs.decode("utf-8"):
        time.sleep(0.1)
        logs = subprocess.check_output(cmdline)


@cli.command()
def create_initial_db():
    configure_app(os.getenv("APPLICATION_CONFIG"))

    try:
        run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])
    except psycopg2.errors.DuplicateDatabase:
        print(
            f"The database {os.getenv('APPLICATION_DB')} already exists and will not be recreated"
        )


@cli.command()
@click.argument("filenames", nargs=-1)
def test(filenames):
    os.environ["APPLICATION_CONFIG"] = "testing"
    configure_app(os.getenv("APPLICATION_CONFIG"))

    cmdline = docker_compose_cmdline("up -d")
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline("logs db")
    wait_for_logs(cmdline, "ready to accept connections")

    run_sql([f"CREATE DATABASE {os.getenv('APPLICATION_DB')}"])

    cmdline = ["pytest", "-svv", "--cov=application", "--cov-report=term-missing"]
    cmdline.extend(filenames)
    subprocess.call(cmdline)

    cmdline = docker_compose_cmdline("down")
    subprocess.call(cmdline)


if __name__ == "__main__":
    cli()

Notable changes:

I created two new functions app_config_file 1 and docker_compose_file 2 that encapsulate the creation of the file paths.
I isolated the code that waits for a message in the database container logs, creating the function wait_for_logs 3.
The command docker_compose_cmdline 4 now receives a string and converts it into a list internally. This way expressing commands is more natural, as it doesn't require the ugly list syntax that subprocess works with.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Psycopg – PostgreSQL database adapter for Python

Step 5 - Fixtures for tests¶

Pytest uses fixtures for tests, so we should prepare some basic ones that will be generally useful. First let's include pytest-flask, which provides already some basic fixtures

requirements/testing.txt

-r production.txt

pytest
coverage
pytest-cov
pytest-flask

Then add the fixtures app and database to the file tests/conftest.py. The first is required by pytest-flask itself (it's used by other fixtures) and the second one is useful every time you need to interact with the database itself.

tests/conftest.py

import pytest

from application.app import create_app
from application.models import db


@pytest.fixture
def app():
    app = create_app("testing")

    return app


@pytest.fixture(scope="function")
def database(app):
    with app.app_context():
        db.drop_all()
        db.create_all()

    yield db

Remember to create the empty file tests/__init__.py to make pytest correctly load the code.

As you can see, the fixture database uses the methods drop_all and create_all to reset the database. The reason is that this fixture is recreated for each function, and we can't be sure a previous function left the database clean. As a matter of fact, we might be almost sure of the opposite.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

pytest fixtures - One of the most powerful features of pytest
pytest-flask - A plugin for pytest that simplifies testing Flask applications
Flask-SQLAlchemy API

Bonus step - A full TDD example¶

Before wrapping up this post, I want to give you a full example of the TDD process that I would follow given the current state of the setup, which is already complete enough to start the development of an application. Let's pretend my goal is that of adding a User model that can be created with an id (primary key) and an email fields.

First of all I write a test that creates a user in the database and then retrieves it, checking its attributes

tests/test_user.py

from application.models import User


def test__create_user(database):
    email = "some.email@server.com"
    user = User(email=email)
    database.session.add(user)
    database.session.commit()

    user = User.query.first()

    assert user.email == email

Running this test results in an error, because the module User does not exist

$ ./manage.py test
Creating network "testing_default" with the default driver
Creating testing_db_1 ... done
======================================= test session starts ======================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 --
/home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir: .pytest_cache
rootdir: /home/leo/devel/flask-tutorial
plugins: flask-1.0.0, cov-2.10.0
collected 0 items / 1 error
============================================= ERRORS =============================================
___________________________ ERROR collecting tests/tests/test_user.py ___________________________
ImportError while importing test module '/home/leo/devel/flask-tutorial/tests/tests/test_user.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
venv3/lib/python3.7/site-packages/_pytest/python.py:511: in _importtestmodule
    mod = self.fspath.pyimport(ensuresyspath=importmode)
venv3/lib/python3.7/site-packages/py/_path/local.py:704: in pyimport
    __import__(modname)
venv3/lib/python3.7/site-packages/_pytest/assertion/rewrite.py:152: in exec_module
    exec(co, module.__dict__)
tests/tests/test_user.py:1: in <module>
    from application.models import User
E   ImportError: cannot import name 'User' from 'application.models'
	(/home/leo/devel/flask-tutorial/application/models.py)

----------- coverage: platform linux, python 3.7.5-final-0 -----------
Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
application/app.py         11      9    18%   6-21
application/config.py      14     14     0%   1-32
application/models.py       4      0   100%
-----------------------------------------------------
TOTAL                      29     23    21%

==================================== short test summary info ===================================
ERROR tests/tests/test_user.py
!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!
======================================= 1 error in 0.20s =======================================
Stopping testing_db_1 ... done
Removing testing_db_1 ... done
Removing network testing_default
$

I won't show here all the steps of the strict TDD methodology, and implement directly the final solution, which is

application/models.py

from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate

db = SQLAlchemy()
migrate = Migrate()


class User(db.Model):
    __tablename__ = "users"
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String, unique=True, nullable=False)

With this model the test passes

$ ./manage.py test
Creating network "testing_default" with the default driver
Creating testing_db_1 ... done
=================================== test session starts ==================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 --
/home/leo/devel/flask-tutorial/venv3/bin/python3
cachedir: .pytest_cache
rootdir: /home/leo/devel/flask-tutorial
plugins: flask-1.0.0, cov-2.10.0
collected 1 item

tests/test_user.py::test__create_user PASSED

----------- coverage: platform linux, python 3.7.5-final-0 -----------
Name                    Stmts   Miss  Cover   Missing
-----------------------------------------------------
application/app.py         11      1    91%   19
application/config.py      14      0   100%
application/models.py       8      0   100%
-----------------------------------------------------
TOTAL                      33      1    97%


==================================== 1 passed in 0.14s ===================================
Stopping testing_db_1 ... done
Removing testing_db_1 ... done
Removing network testing_default
$

Please not that this is a very simple example and that in a real case I would add some other tests before accepting this code. In particular we should check that the field email can be empty, and maybe also test some validation on that field.

Let's add a very simple route to use the newly created model

application/app.py

from flask import Flask
from application.models import User


def create_app(config_name):

    app = Flask(__name__)

    config_module = f"application.config.{config_name.capitalize()}Config"

    app.config.from_object(config_module)

    from application.models import db, migrate

    db.init_app(app)
    migrate.init_app(app, db)

    @app.route("/")
    def hello_world():
        return "Hello, World!"

    @app.route("/users")
    def users():
        num_users = User.query.count()
        return f"Number of users: {num_users}"

    return app

As you can see I didn't introduce anything too complicated. I import the model User and count the number of entries in its table. We will create the table in a minute with the migration that flask db migrate will create for us, so we expect this to just return a page that says "Number of users: 0", but it's a good demonstration that the connection with the database is working.

So, let's generate the migration in the database. Spin up the development environment with

$ ./manage.py compose up -d

If this is the first time I spin up the environment I have to create the application database and to initialise the migrations, so I run

$ ./manage.py create-initial-db

As we already initialised Alembic before we don't need to run the command db init. If you do, it will return Error: Directory migrations already exists and is not empty. Now I can create the migration with

$ ./manage.py flask db migrate -m "Initial user model"
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.autogenerate.compare] Detected added table 'users'
  Generating /home/leo/devel/flask-tutorial/migrations/versions/7a09d7f8a8fa_initial_user_model.py ...  done

As you can see from the output, this created the file migrations/versions/7a09d7f8a8fa_initial_user_model.py. The number 7a09d7f8a8fa is just an hex version of a UUID ,so it will be different for you, while the name comes from the commit message. The file itself contains SQLAlchemy code that changes the DB according to the code that we wrote in the application.

Finally I can apply the migration with

$ ./manage.py flask db upgrade
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> 7a09d7f8a8fa, Initial user model

At this point we can run ./manage.py compose exec db psql -U postgres again and see what happened to the database.

$ ./manage.py compose exec db psql -U postgres
psql (13.0 (Debian 13.0-1.pgdg100+1))
Type "help" for help.

postgres=# \l
                                  List of databases
    Name     |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   
-------------+----------+----------+------------+------------+-----------------------
 application | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 postgres    | postgres | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
 template1   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
             |          |          |            |            | postgres=CTc/postgres
(4 rows)

You see here that the database application configured with APPLICATION_DB has beed created. You can now connect to it and list the tables

postgres=# \c application
You are now connected to database "application" as user "postgres".
application=# \dt
              List of relations
 Schema |      Name       | Type  |  Owner   
--------+-----------------+-------+----------
 public | alembic_version | table | postgres
 public | users           | table | postgres
(2 rows)

The content of the table alembic_version shouldn't be surprising, as it's the UUID used for the migration

application=# select * from alembic_version;
 version_num  
--------------
 7a09d7f8a8fa
(1 row)

The table users contains the fields id and email according to the model that we wrote in Python

application=# \d users
                                 Table "public.users"
 Column |       Type        | Collation | Nullable |              Default              
--------+-------------------+-----------+----------+-----------------------------------
 id     | integer           |           | not null | nextval('users_id_seq'::regclass)
 email  | character varying |           | not null | 
Indexes:
    "users_pkey" PRIMARY KEY, btree (id)
    "users_email_key" UNIQUE CONSTRAINT, btree (email)

You can also open your browser and head to http://localhost:5000/users to see the new route in action. After this we can safely commit my code and move on with the next requirement.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Final words¶

I hope this post already showed you why a good setup can make the difference. The project is clean and wrapping the command in the management script plus the centralised config proved to be a good choice as it allowed me to solve the problem of migrations and testing in (what I think is) an elegant way. In the next post I'll show you how to easily create scenarios where you can test queries with only specific data in the database. If you find my posts useful please share them with whoever you thing might be interested.

Updates¶

2020-07-13 Vlad Pavlichek found and fixed a typo in the post, where manage.py was missing the extension .py. Thanks Vlad!

2020-12-22 I reviewed the whole tutorial and corrected several typos

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Flask project setup: TDD, Docker, Postgres and more - Part 1

2020-07-05T13:00:00+01:00

There are tons of tutorials on Internet that tech you how to use a web framework and how to create Web applications, and many of these cover Flask, first of all the impressive Flask Mega-Tutorial by Miguel Grinberg (thanks Miguel!).

Why another tutorial, then? Recently I started working on a small personal project and decided that it was a good chance to refresh my knowledge of the framework. For this reason I temporarily dropped the clean architecture I often recommend, and started from scratch following some tutorials. My development environment quickly became very messy, and after a while I realised I was very unsatisfied by the global setup.

So, I decided to start from scratch again, this time writing down some requirements I want from my development setup. I also know very well how complicated the deploy of an application in production can be, so I want my setup to be "deploy-friendly" as much as possible. Having seen too many project suffer from legacy setups, and knowing that many times such issues can be avoided with a minimum amount of planning, I thought this might be interesting for other developers as well. I consider this setup by no means better than others, it simply addresses different concerns.

What you will learn¶

This post contains a step-by-step description of how I set up a real Flask project that I am working on. It's important that you understand that this is just one of many possible setups, and that my choices are both a matter of personal taste and dictated by some goals that I will state in this section. Changing the requirements would clearly result in a change of the structure. The target of the post is then to show that the setup of a project can take into account many things upfront, without leaving them to an undetermined future when it will likely be too late to tackle them properly.

The requirements of my setup are the following:

Use the same database engine in production, in development and for tests
Run test on an ephemeral database
Run in production with no changes other that the static configuration
Have a command to initialise databases and manage migrations
Have a way to spin up "scenarios" starting from an empty database, to create a sandbox where I can test queries
Possible simulate production in the local environment

As for the technologies, I will use Flask, obviously, as the web framework. I will also use Gunicorn as HTTP server (in production) and Postgres for the database part. I won't show here how to create the production infrastructure, but as I work daily with AWS, I will take into account some of its requirements, trying however not to be too committed to a specific solution.

A general advice¶

Proper setup is an investment for the future. As we do in TDD, where we decide to spend time now (writing tests) to avoid spending tenfold later (to find and correct bugs), setting up a project requires time, and might frustrate the desire of "see things happen". Proper setup is a discipline that requires patience and commitment!

If you are ready to go, join me for this journey towards a great setup of a Flask application.

The golden rule¶

The golden rule of any proper infrastructural work is: there has to be a single source of information. The configuration of you project shouldn't be scattered among different files or repositories (not considering secrets, that have to be stored securely). The configuration has to be accessible and easy to convert into different formats to accommodate the needs of different tools. For this reason, the configuration should be stored in a static file format like JSON, YAML, INI, or similar, which can be read and processed by different programming languages and tools.

My format of choice for this tutorial is JSON, as it can be read by both Python and Terraform, and is natively used by ECS on AWS.

Step 1 - Requirements and editor¶

My standard structure for Python requirements uses 3 files: production.txt, development.txt, and testing.txt. They are all stored in the same directory called requirements, and are hierarchically connected.

requirements/production.txt

## This file is currently empty

requirements/testing.txt

-r production.txt

requirements/development.txt

-r testing.txt

There is also a final file requirements.txt that points to the production one.

requirements.txt

-r requirements/production.txt

As you can see this allows me to separate the requirements to avoid installing unneeded packages, which greatly speeds up the deploy in production and keeps things as essential as possible. Production contains the minimum requirements needed to run the project, testing adds to those the packages used to test the code, and development adds to the latter the tools needed during development. A minor shortcoming of this setup is that I might not need in development everything I need in production, for example the HTTP server. I don't think this is significantly affecting my local setup, though, and if I have to decide between production and development, I prefer to keep the former lean and tidy.

I have my linters already installed system-wide, but as I'm using black to format the code I have to configure flake8 to accept what I'm doing

.flake8

[flake8]
# Recommend matching the black line length (default 88),
# rather than using the flake8 default of 79:
max-line-length = 100
ignore = E231

This is clearly a very personal choice, and you might have different requirements. Take your time to properly configure the editor and the linter(s). Remember that the editor for a programmer is like the violin for the violinist. You need to know it, and to take care of it. So, set it up properly.

At this point I also create my virtual environment and activate it.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Installing packages in Python
flake8 - A tool for style guide enforcement
black - The uncompromising Python code formatter

Step 2 - Flask project boilerplate¶

As this will be a Flask application the first thing to do is to install Flask itself. That goes in the production requirements, as that is needed at every stage.

requirements/production.txt

Flask

Now, install the development requirements with

$ pip install -r requirements/development.txt

As we saw before, that file automatically installs the testing and production requirements as well.

Then we need a directory where to keep all the code that is directly connected with the Flask framework, and where we will start creating the configuration for the application. Create the directory application and the file config.py in it.

application/config.py

class Config(object):
    """Base configuration"""


class ProductionConfig(Config):
    """Production configuration"""


class DevelopmentConfig(Config):
    """Development configuration"""


class TestingConfig(Config):
    """Testing configuration"""

    TESTING = True

There are many ways to configure a Flask application, one of which is using Python objects. This allows me to leverage inheritance to avoid duplication (which is always good), so it's my method of choice.

It's important to understand the variables and the parameters involved in the configuration. As the documentation clearly states, FLASK_ENV and FLASK_DEBUG have to be initialised outside the application as the code might misbehave if they are changed once the engine has been started. Furthermore the variable FLASK_ENV can have only the two values development and production, and the main difference is in performances. The most important thing we need to be aware of is that if FLASK_ENV is development, then FLASK_DEBUG becomes automatically True. To sum up we have the following guidelines:

It's pointless to set DEBUG and ENV in the application configuration, they have to be environment variables.
Generally you don't need to set FLASK_DEBUG, just set FLASK_ENV to development.
Testing doesn't need the debug server turned on, so you can set FLASK_ENV to production during that phase. It needs TESTING set to True, though, and that has to be done inside the application.

We need now to create the application and to properly configure it.

application/app.py

from flask import Flask


def create_app(config_name): 1

    app = Flask(__name__)

    config_module = f"application.config.{config_name.capitalize()}Config" 2

    app.config.from_object(config_module)

    @app.route("/") 3
    def hello_world():
        return "Hello, World!"

    return app

I decided to use an application factory 1 that accepts a string config_name that is then converted into the name of the config object 2. For example, if config_name is development the variable config_module becomes application.config.DevelopmentConfig so that app.config.from_object can import it. I also added the standard "Hello, world!" route 3 to have a quick way to see if the server is working or not.

Last, we need something that initializes the application running the application factory and passing the correct value for the parameter config_name. The Flask development server can automatically use any file named wsgi.py in the root directory, and since WSGI is a standard specification using that makes me sure that any HTTP server we will use in production (for example Gunicorn or uWSGI) will be immediately working.

wsgi.py

import os

from application.app import create_app

app = create_app(os.environ["FLASK_CONFIG"])

Here, I decided to read the value of config_name from the variable FLASK_CONFIG. This is not a variable requested by the framework, but I decided to use the prefix FLASK_ anyway because it is tightly connected with the structure of the Flask application.

At this point we can happily run the Flask development server with

$ FLASK_CONFIG="development" flask run
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Please note that it says Environment: production because we haven't configured FLASK_ENV yet. If you head to http://127.0.0.1:5000/ with your browser you can see the greetings message.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Flask configuration documentation
Flask application factories
WSGI - The Python Web Server Gateway Interface
My post Dissecting a web stack includes a section on WSGI

Step 3 - Application configuration¶

As I mentioned in the introduction, I am going to use a static JSON configuration file. The choice of JSON comes from the fact that it is a widespread file format, accessible from many programming languages, included Terraform, which I plan to use to create my production infrastructure.

config/development.json

[
  {
    "name": "FLASK_ENV",
    "value": "development"
  },
  {
    "name": "FLASK_CONFIG",
    "value": "development"
  }
]

I obviously need a script that extracts variables from the JSON file and converts them into environment variables, so it's time to start writing my own manage.py file. This is a pretty standard concept in the world of Python web frameworks, a tradition initiated by Django. The idea is to centralise all the management functions like starting/stopping the development server or managing database migrations. As in flask this is partially done by the command flask itself, for the time being I just need to wrap it providing suitable environment variables.

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess

import click


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development") 1

# Read configuration from the relative JSON file
config_json_filename = os.getenv("APPLICATION_CONFIG") + ".json" 2
with open(os.path.join("config", config_json_filename)) as f:
    config = json.load(f)

# Convert the config into a usable Python dictionary
config = dict((i["name"], i["value"]) for i in config)

for key, value in config.items():
    setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand): 3
    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


cli.add_command(flask)

if __name__ == "__main__":
    cli()

Remember to make the script executable with

$ chmod 775 manage.py

As you can see I'm using click, which is the recommended way to implement Flask commands. As I might use it to customise subcommands of the flask main script, I decided to stick to one tool and use it for the script manage.py as well.

The variable APPLICATION_CONFIG 1 is the only one that I need to specify, and its default value is development. From that variable I infer the name of the JSON file with the full configuration 2 and load environment variables from that. The function flask 3 simply wraps the command flask provided by Flask so that I can run ./manage.py flask SUBCOMMAND to run it using the configuration development or APPLICATION_CONFIG="foobar" ./manage.py flask SUBCOMMAND to use the foobar one.

A clarification, to be sure you don't confuse environment variables with each other:

APPLICATION_CONFIG is strictly related to my project and is used only to load a JSON configuration file with the name specified in the variable itself.
FLASK_CONFIG is used to select the Python object that contains the configuration for the Flask application (see application/app.py and application/config.py). The value of the variable is converted into the name of a class.
FLASK_ENV is a variable used by Flask itself, and its values are dictated by it. See the configuration documentation mentioned in the resources of the previous section.

Now we can run the development server

$ ./manage.py flask run
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 172-719-201

Note that it now says Environment: development because of FLASK_ENV has been set to development in the configuration. As we did before, a quick visit to http://127.0.0.1:5000/ shows us that everything is up and running.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources:

Django's management script
Click - A Python package for creating command line interfaces

Step 4 - Containers and orchestration¶

There is no better way to simplify your development than using Docker.

There is also no better way to complicate your life than using Docker.

As you might guess, I have mixed feelings about Docker. Don't get me wrong, Linux containers are an amazing concept, and Docker is very useful. It's also a complex technology that sometimes requires a lot of work to get properly configured. In this case the setup will be pretty simple, but there is a major complication with using a database server that I will describe later.

Running the application in a Docker container allows me to isolate it and to simulate the way I will run it in production. I will use docker-compose, as I expect to have other containers running in my development setup (at least the database), so I can leverage the fact that the docker-compose configuration file can interpolate environment variables. Once again through the environment variable APPLICATION_CONFIG I will select the correct JSON file, load its values in environment variables and then run the docker-compose file.

First of all we need an image for the Flask application

docker/Dockerfile

FROM python:3

ENV PYTHONUNBUFFERED 1

RUN mkdir /opt/code
RUN mkdir /opt/requirements
WORKDIR /opt/code

ADD requirements /opt/requirements
RUN pip install -r /opt/requirements/development.txt

As you can see the requirements directory is copied into the image, so that Docker can run the command pip install at creation time. The whole code directory will be mounted live into the image at run time.

This clearly means that every time we change the development requirements we need to rebuild the image. This is not a complicated process, so I will keep it as a manual process for now. To run the image we can create a configuration file for docker-compose.

docker/development.yml

version: '3.4'

services:
  web:
    build:
      context: ${PWD}
      dockerfile: docker/Dockerfile
    environment:
      FLASK_ENV: ${FLASK_ENV}
      FLASK_CONFIG: ${FLASK_CONFIG}
    command: flask run --host 0.0.0.0
    volumes:
      - ${PWD}:/opt/code
    ports:
      - "5000:5000"

As you can see, the docker-compose configuration file can read environment variables natively. To run it we first need to add docker-compose itself to the development requirements.

requirements/development.txt

-r testing.txt

docker-compose

Install it with pip install -r requirements/development.txt, then build the image with

$ FLASK_ENV="development" FLASK_CONFIG="development" docker-compose -f docker/development.yml build web

This will take some time, as Docker has to download all the required layers and install the requirements.

We are explicitly passing environment variables here, as we have not wrapped docker-compose in the manage script yet. Once the image has been build, we can run it with the command up

$ FLASK_ENV="development" FLASK_CONFIG="development" docker-compose -f docker/development.yml up

This command gives us the following output

Creating network "docker_default" with the default driver
Creating docker_web_1 ... done
Attaching to docker_web_1
web_1  |  * Environment: development
web_1  |  * Debug mode: on
web_1  |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
web_1  |  * Restarting with stat
web_1  |  * Debugger is active!
web_1  |  * Debugger PIN: 234-361-737

You can stop the containers pressing Ctrl-C, which gracefully tears down the system. If you run the command up -d docker-compose will run as a daemon, leaving you the control of the current terminal. If docker-compose is running you can docker ps and you should see an output similar to this

CONTAINER ID  IMAGE       COMMAND                 ...   PORTS                   NAMES
c98f35635625  docker_web  "flask run --host 0.…"  ...   0.0.0.0:5000->5000/tcp  docker_web_1

If you need to explore the container you can login directly with

$ docker exec -it docker_web_1 bash

or with

$ FLASK_ENV="development" FLASK_CONFIG="development" docker-compose -f docker/development.yml exec web bash

In either case, you will end up in the directory /opt/code (which is the WORKDIR of the image), where the current directory in the host is mounted.

To tear down the containers, when running as daemon, you can run

$ FLASK_ENV="development" FLASK_CONFIG="development" docker-compose -f docker/development.yml down

Notice that the server now says Running on http://0.0.0.0:5000/, as the Docker container is using that network interface to communicate with the outside world. Since the ports are mapped, however, you can head to either http://localhost:5000 or http://0.0.0.0:5000 with your browser.

To simplify the usage of docker-compose, I want to wrap it in the script manage.py, so that it automatically receives environment variables, as their number is going to increase as soon as we add a database.

manage.py

#! /usr/bin/env python

import os
import json
import signal
import subprocess

import click

docker_compose_file = "docker/development.yml"
docker_compose_cmdline = ["docker-compose", "-f", docker_compose_file]


# Ensure an environment variable exists and has a value
def setenv(variable, default):
    os.environ[variable] = os.getenv(variable, default)


setenv("APPLICATION_CONFIG", "development")

# Read configuration from the relative JSON file
config_json_filename = os.getenv("APPLICATION_CONFIG") + ".json"
with open(os.path.join("config", config_json_filename)) as f:
    config = json.load(f)

# Convert the config into a usable Python dictionary
config = dict((i["name"], i["value"]) for i in config)

for key, value in config.items():
    setenv(key, value)


@click.group()
def cli():
    pass


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def flask(subcommand):
    cmdline = ["flask"] + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


@cli.command(context_settings={"ignore_unknown_options": True})
@click.argument("subcommand", nargs=-1, type=click.Path())
def compose(subcommand):
    cmdline = docker_compose_cmdline + list(subcommand)

    try:
        p = subprocess.Popen(cmdline)
        p.wait()
    except KeyboardInterrupt:
        p.send_signal(signal.SIGINT)
        p.wait()


if __name__ == "__main__":
    cli()

You might have noticed that the two functions flask and compose are basically the same code, but I resisted the temptation to refactor them because I know that the command compose will need some changes as soon as I add a database.

Now I can run ./manage.py compose up -d and ./manage.py compose down and have the environment variables automatically passed to the system.

Git commit

You can see the changes made in this step through this Git commit or browse the files.

Resources

Docker compose - A tool for defining and running multi-container Docker applications
Docker networking
Python subprocess module

Final words¶

That's enough for this first post. We started from scratch and added some boilerplate code for a Flask project, exploring what environment variables are used by the framework, then we added a configuration system, a management script, and finally we run everything in a Docker container. In the next post I will show you how to add a persistent database to the development setup and how to use an ephemeral one for the tests. If you find my posts useful please share them with whoever you thing might be interested.

Happy development!

Updates¶

2020-12-22 I reviewed the whole tutorial and corrected several typos

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.