The Digital Cat - Python

Data Partitioning and Consistent Hashing

2022-08-23T12:00:00+00:00

This post is an introduction to partitioning, a technique for distributed storage systems, and to consistent hashing, a specific partitioning algorithm that promotes even distribution of data, while allowing to dynamically change the number of servers and minimising the cost of the transition.

My interest in partitioning dates back to 2015 when I was following courses at the MongoDB university and learned about sharding, the name MongoDB uses for partitioning. I was fascinated by the topic and discovered the technique known as consistent hashing; I enjoyed it a lot, so much that I wrote a little demo in Python to understand it better. Later, I focused on other things and forgot the project completely, until recently, when David Eynon sent me a PR on GitHub to replace a deprecated testing library. So, I decided to brush up on my knowledge of consistent hashing and, as I often do on this blog, dump my thoughts in a post.

The topic of distributed storage and data processing is arguably rich and complicated, so while I will try to give a broader context to the concepts that I will introduce, I by no means intend to write a comprehensive guide to the subject matter. The audience of this post is developers who do not know what partitioning and consistent hashing are and want to take their first step into those topics.

Code syntax

You will find some code examples mentioned in the post, which are written using the Python notation. If you are not familiar with the language, these are the main rules

x**y means x^y, e.g. 2**3 => 8.
x//y means the integer division between x and y, e.g. 11//4 => 2.
x%y means the modulo operation (remainder of integer division), e.g. 11%4 => 3.

Rationale¶

When we design a system, we might want to scatter data among multiple sources to allow real concurrency of access and a more targeted optimisation.

For example, we might observe that in a given social media application there are two types of queries: some are very infrequent and involve tables related to personal data and the user profile, others are extremely frequent and pretty intensive, and are related to the content shared by the user. In this case, we might decide to store the tables related to the profile and the tables that are related to content in two different systems, A and B (here, the word system might be freely replaced by computer, database, storage system, or other similar components).

This means that the infrequent queries that fetch personal data will be served by system A, while the more frequent and intensive queries related to content will be served by system B.

Suddenly, we have the chance to deploy system B using more powerful and expensive hardware, or an architecture with better performances, without increasing the cost for tables that won't benefit from such an improvement as the ones stored in system A.

This is a standard approach in system design, and it requires the introduction of an additional layer of control that will route requests to the right source. This layer might be implemented in several places, for example:

in the code of our application, with conditional structures that query different data sources
in the framework that we are using for the application, for example in a middleware that automatically routes requests according to nature or the query
in a wrapper around the storage that hides the fact that data exists in two different systems

In the last case, this technique is usually called partitioning.

In this post, I will try to show the challenges we face when we partition data and focus on some of the algorithms that can be used to implement it, in particular on consistent hashing. Please note that, while some of these techniques are used by databases to provide internal partitioning, they have a wider range of applications and might come in handy in different contexts.

Design choices¶

Every design choice in a system depends on the requirements, and when it comes to data storage the most important factors are the nature of the data, its distribution, and the access patterns. Consider for example databases and Content Delivery Networks (CDNs): both are meant to store data, and the storage size of both can vary substantially. However, there are important differences between the two that greatly affect the design choices. Let's see some simple examples:

databases are meant to store data in a long-term fashion, while caches are by definition short-lived. This means that an important requirement for databases is data preservation, and we should do everything in our power to avoid losing parts of the database. A cache, conversely, holds data for a short time, either predetermined by the system or forced by a change in the data source. As you can see in this case we not only take data loss as part of the equation, but we get to the point where we trigger it on purpose.

applications often make use of range queries, which means that they retrieve sets of results spanning a range of values of one of the keys; for example, you might want to see all employers within a certain range of salaries, or all users that have more than a certain amount of followers. In such cases, it makes little sense to scatter data among different physical sources, thus making the retrieval more complicated and ultimately affecting performances. Databases see very often an access pattern of this type, while caches, being usually implemented as key/value stores, do not need to take this into account.

A practical example of partitioning¶

Let us consider a simple key/value store, for example a common address book where the key is the name of the contact and the value a rich document with their personal details. If multiple users access the store, chances are that the system will at a certain point struggle to serve all the requests, so we might want to partition the data to allow concurrent access. We can for example sort them alphabetically and split them in two, storing all values with a key that begins with the letters A-M in one server and the rest (keys N-Z) in the second one.

This might seem a good idea, but we will soon discover that performances are not great. Unfortunately, our address book doesn't contain the same number of people for each letter, as (for example) we know more people whose name starts with A or C than with X or Z.

That poses a problem, as our partitioning doesn't achieve the desired outcome, that of splitting requests evenly between the two servers. If we increase the number of partitions, serving smaller groups of letters, we will just worsen the problem, to the point where a partition might be completely empty and thus receive no traffic: since the problem comes from the data distribution, we need to find a way to change that property.

One way to deal with the problem is to change the boundaries of the partitions so that we get an almost even distribution of values among them. For example, we might store keys starting with A-B in the first partition, keys starting with C-D in the second, and all the rest in the third one.

The problem with such a strategy is that it is highly dependent on the actual data that we are storing. Not only does this mean the solution has to be customised for each use case (the partitions in the example might be good for one address book and completely wrong for another), but adding data to the storage might change the distribution and invalidate the solution.

Hash functions to the rescue¶

An interesting solution to the problem of distributing data evenly is represented by hash functions. As I explained in my post Introduction to hashing, good hash functions produce a highly uniform distribution, which makes them ideal in this case. Please note that hash functions can help with routing queries and not with storing data. Hashed values cannot replace the content, as they are not bijective, i.e. given two different inputs the output might be the same (collision), so they can only be used to decide where to store a piece of information.

We can at this point devise a storage strategy based on hash functions. We can divide the output space of the hash function (codomain) into a certain amount of partitions and be sure that each one of them will contain a similar amount of elements. For example, the hash function might output a 32-bit number, so we know that each hashed value will be between 0 and 2³² (4,294,967,295), and from here it's pretty straightforward to find partition boundaries. For example, we can create 16 partitions numbered 0 to 15, each one containing 2²⁸ hash values (268,435,456).

Routing is at this point very simple, as we can mathematically find the partition number given the hash. There are many ways to do this but two simple approaches are

using the integer division hash(k) // partition_size, e.g. hash(k) // 2**28. All keys from 0 to 268435455 end up in partition 0 (268435455 // 2**28), keys from 268435456 to 536870911 end up in partition 1, and so on.

using the modulo operator hash(k) % number_of_partitions, e.g. hash(k) % 16. This assigns values to partitions in a round robin fashion, where key 0 goes to partition 0 (0%16), key 1 to partition 1 (1%16), key 15 to partition 15 (15%16), and then starts again with key 16 which goes to partition 0 (16%16), and so on.

This architecture has the clear advantage that thanks to the properties of hash functions, data is scattered evenly among the partitions. This means that when we query the system, requests will also be divided evenly, thus giving us a good distribution of the load.

As we will see later, however, this is not a good approach for dynamic systems.

Partitioning use cases¶

Hash functions are definitely interesting but they are not the perfect solution in every case. Let's have a brief look at three different types of systems that might benefit from partitioning and discuss their specific requirements.

Load balancers

Pure load balancers solve a simple problem: to spread requests evenly across multiple identical servers. The key word here is "identical", as you cannot pick the wrong server, thus no routing can result in an error. However, spreading the load unevenly can result in performance loss, and possibly also service failure. For example, if a server gets overloaded queries might hit a timeout while waiting to be served.

For this reason, when load balancing is not content-aware, for example in a simple HTTP server scenario, round-robin partitioning is a good choice. The system just assigns new requests to servers on a rotation basis, which ensures perfectly even distribution. For example, this algorithm is the default choice for AWS Application Load Balancers.

Clearly, load balancers can be more complicated and feature-rich even without becoming content aware. The aforementioned AWS ALBs, for example, support also the "least outstanding requests" algorithm, which in simple words means choosing the server with the smallest workload.

Caches

Caches are systems that temporarily store data whose retrieval is expensive, either for the user or for the provider. For example, if a system runs a long query on a database caching the result will be beneficial both for the system and the database. For the former, because a repeated run will get the result much faster and for the latter because the load of the new query is zero.

Caches can be found everywhere and vary dramatically in size, but they are one of the best examples of systems that benefit from partitioning. As I mentioned before, their standard usage patterns don't include range queries and data loss (flushing) is part of their normal workflow.

A Content Delivery Network (CDN) is a specific type of cache that is distributed geographically. The purpose of the CDN nodes is to store content in a location that is physically near the users, thus increasing the performance of the system. This means that two geographically distinct nodes of a CDN contain the same values (replication), and the routing policy is solely based on the physical position of the user with respect to the node. Internally, each CDN node can be implemented using partitioning, though, which might speed up the performances of that specific node.

Databases

As for databases, I already mentioned that the most important problem is range queries or if you prefer, content-aware partitioning. In general, you can't partition a database without taking into account the content, or you will incur severe performance losses. So, when it comes to databases, partitioning has to be the result of a specific design and can't be applied regardless of the database schema.

To better understand the challenge, let's consider a simple database whose elements are employees with a name and a salary. Now, if we want to partition this database we have to choose a key for the partitioning itself. It might be the primary key, the name, or the salary, as these are the only values available in each record.

Say we use hash functions to partition the database and use the employee salary as a key. Because of the properties of hash functions, employees with the exact same salary will end up being stored in the same partition, but employees with similar salaries might end up in different ones. This depends on the number of partitions, clearly, but the main point is that records that are "near" (according to the selected key) now are potentially very far.

In the example above I used MD5 as the routing hash function, and you can reproduce the calculations using the following Python code

import hashlib

def hash_value(value):
    return int(hashlib.md5(str(value).encode("utf-8")).hexdigest(), 16)

# 57500283691658467528082923406379043196
hash_value(60000)

# 209589555716047624083879134729984902154
hash_value(60100)

# 12
hash_value(60000) % 16

# 10
hash_value(60100) % 16

Things do not go much better if we use the integer division. If we have 16 partitions, each one of them contains 2¹²⁴ values

# 2
hash_value(60000) // 2**124

# 9
hash_value(60100) // 2**124

Now, let's consider a query that selects all employees within a certain range of salaries. If the database is not partitioned, all records are kept on the same server, and if we optimised the system for such a query, the records will also be physically adjacent (e.g. stored in nearby memory addresses). This makes the query blazing fast, but if the database is partitioned the query has to collect values from multiple partitions which greatly penalises performance.

We can see a real example of this design challenge in the documentation of MongoDB, a non-relational database that supports partitioning (called sharding). MongoDB supports hashed sharding and ranged sharding. In their words

Hashed sharding uses either a single field hashed index or a compound hashed index as the shard key to partition data across your sharded cluster.

Range-based sharding involves dividing data into contiguous ranges determined by the shard key values. In this model, documents with "close" shard key values are likely to be in the same chunk or shard. This allows for efficient queries where reads target documents within a contiguous range. However, both read and write performance may decrease with poor shard key selection.

I highly recommend reading the two pages I linked above as they will give you a good idea of how a real system uses the concepts I introduced and what design challenges are involved when using partitioning.

Caching and scaling strategies¶

When we design distributed caches, an interesting problem we might face is that of scaling the system in and out to match the current load without wasting resources.

When the cache is under a light load we might want to run a small number of servers, but as soon as the number of requests increases we need to proportionally increase the number of cache nodes if we want to avoid a performance drop. This is usually not a big problem for partitioned databases, since in that case we change the number of partitions only occasionally to adjust performances or to increase the storage size, but caches like CDNs might need continuous adjustments during a single day.

Increasing or decreasing the number of nodes in a distributed cache might however be a pretty destructive action. Depending on the routing algorithm, if we add nodes (scale out) we might need to move data from existing ones to the newly added ones, and if we remove nodes (scale in) we will certainly lose the data contained in them. Both scenarios result in a (potentially massive) cache invalidation which can't be taken lightly.

The hash-based routing method presented in the previous section has terrible performances when it comes to scaling because any change in the number of servers impacts the key boundaries of the existing ones. Let's see a practical example of that and calculate the actual figures.

Scaling out with hash partitioning

Every time you consider a process or an algorithm you should have a look at how it behaves in the worst possible condition, to have a glimpse of what you might run into when you use it. For this reason, the following example considers a scale-out scenario in which all cache nodes are full. The best case is obviously when all nodes are empty, but in that case we don't need to scale out at all.

Let's consider a 32-bit hash function and 16 partitions numbered 0 to 15. Since the hash function space is 2³² (4,294,967,296), each partition will contain 2²⁸ hash values (268,435,456). Each node is full, which means that all the possible 2²⁸ slots are assigned to a cached item, that is some data stored in the server that corresponds to that partition. The system is using the integer division routing system.

If we scale out to 17 partitions, increasing the pool by just by 1 node, each node will now contain a smaller part of the global data space, as now we split it among more nodes. In particular, each node used to contain 1/16 of the global data (268,435,456), and will now contain 1/17 of it (approx. 252,645,135). Our biggest problem is now managing the transition between the initial setup and the new one.

The first node hosted 1/16 of the data space, the keys from 0 to 268435455. It will now contain 1/17 of the data space, the keys from 0 to 252645134. To simplify the example it is useful to convert everything into a common unit of measure: the node used to contain 17/272 of the space (1/16) and contains now 16/272 (1/17) of it.

This means that 1/272 of the whole data space has to be moved to the second node, corresponding to the keys from 252645135 to 268435455. It is important to note that these keys cannot be moved to the newly added node, but have to be moved to the second node because the algorithm we use maps keys to nodes in order.

This means that the second node will receive 1/272 of the whole data space. Since it originally already contained 17/272 of the whole space it should now theoretically contain 18/272 of it. However, as it happened for the first node, we want to balance the content and reduce it to 16/272, so now we have 2/272 of the whole space that we want to move to the third node.

So, we move 1/272 from node 1 to node 2, 2/272 from node 2 to node 3, 3/272 from node 3 to node 4, and going on with the example we end up moving 16/272 (1/17) from the 16th node to the 17th, which fills it with the correct amount of keys. However, in doing so we moved 136/272 (1/272 + 2/272 + 3/272 + ... + 16/272) of the data space between nodes, which is exactly 50% of it.

So, for any initial size and a scale out of 1 single node, we have to move 50 of the data stored in our cache, and it might only get worse by increasing the number of final nodes until we end up having to move almost 100 of it (in an extreme case). A similar effect plagues the scale-in action, where one or more nodes are removed from the pool, and the keys they contain have to be migrated to the remaining nodes, creating a ripple effect to redistribute the keys according to the algorithm.

Using a modulo routing strategy doesn't change things: as I mentioned before, the core issue is that the addition of new nodes changes the routing of the whole data space, requiring a massive migration of keys in the entire system.

A different approach¶

While the idea of using hash functions looked very promising, we quickly found that the trivial implementation has very poor performances in a dynamic setting. As we clearly saw in the previous section, the problem is that upon scaling more than half of the keys have to be moved across nodes, so if we could find a way to avoid this we could still use hash functions to scatter data uniformly across the nodes.

As you might have already figured out, the issue comes from the attempt to keep all nodes perfectly balanced. The modulo and integer division algorithms distribute keys evenly (as long as the hash function has a good diffusion), but this is a double-edged sword. The balance is extremely beneficial in a static environment, but it is also the Achilles heel of this architecture when we change the number of nodes.

When we design a system, requirements are paramount. Everything we add to the final product should be there to satisfy one or more requirements. However, often requirements clash with each other, and trying to implement all of them at once might lead to situations where there is no apparent solution. In such cases, it is useful to temporarily drop one or more requirements and investigate the options we have, and this is exactly what we can do in this case: maintaining balance is an important feature, but let's see what would happen if we didn't have that requirement.

If we don't care about balancing nodes we can solve the problem with a different approach. Instead of using the integer division to find the slot, we can keep a table of the minimum hash served by each slot and route requests according to that. Each row of the table will have a minimum hash and the node that serves them.

This means that when we increase the number of slots we can just drop a new slot anywhere and assign to it all the keys that fall under its domain. This means that the new node will become the owner of keys that belonged to another node as it happened before, but with an important difference. Now all keys come from another single node, and the amount of keys moved is a fraction of those contained in it (which is much less than half of the keys). In the worst case, we need to move all keys contained in a node, which once again is much less than half of the keys.

As you can see, this relieves the load of one single node. According to what we said before, we are not trying to balance the load of the whole cluster. If we could use this technique to cover multiple spaces with a single added node, though, we could relieve the load of more than one other node. In principle this is simple: we just need to add multiple rows with the same node to the table.

Pay attention to the fact that we added multiple rows, that is multiple partitions, but they are all served by the same physical node. This has several advantages:

It fills the new node with keys coming from several different nodes without rippling effects.
The key transfer load is spread among different nodes, noticeably hitting only the new node.

There is also an interesting turn of events: since keys for the new node are fetched from several different existing nodes, the process will keep the cluster balanced! This is a remarkable outcome: we temporarily dropped a requirement and found a solution that provides that exact requirement in a different way.

The key part of this new process is the idea that multiple partitions can be served by the same node. The only missing part at this point is a way to identify the new partitions (the sets of hashes) served by the new node in a deterministic way.

Consistent hashing¶

Finally, let me introduce consistent hashing as a technique to implement the process described above.

As we discussed in the previous section, the only missing part is an algorithm that produces a deterministic set of hash ranges for a single new node. These hash ranges represent the partitions served by that node and should be scattered across the whole hash space. It is important for them to be spread because this way they will each receive some keys from existing nodes, instead of migrating a bulk of keys from a single one. The more evenly spread, the better the distribution of the load and the more balanced the resulting cluster.

As we saw previously, any time we need to scatter data across a given space in a deterministic way, hash functions are a good choice, and they can be used in this case as well. The idea is simple: each partition of a node is assigned a name and this name is hashed with the same function used to hash the keys stored in the system. This will produce a deterministic value in the hash space, and that value will be the minimum value served by that partition. Thanks to diffusion the names of all partitions will generate different hash values that won't easily clash, and this is the way we generate the routing table.

Let's see an example, bearing in mind that the specific function can change among implementations.

For simplicity's sake, I used a custom hash function that outputs 28-bit hashes (7 hexadecimal digits). This makes it possible to compare hashes visually and simplifies the example. To do this I took the first 7 digits of the SHA1 hash with the following Python code

def hash_name(name):
    return int(hashlib.sha1(name.encode("utf-8")).hexdigest()[:7], 16)

thus creating a hash function whose values go from 0x0000000 to 0xfffffff. At the end of the post you will find the Python code that I used to generate the following routing tables, and you are free to experiment using different settings.

WARNING: this is not a good hash function! SHA1 produces 160 bits hashes, so taking the first 28 bits reduces the hash space to a microscopic fraction of the total, as we go from 2¹⁶⁰ total hashes to 2²⁸. Please keep in mind that this is done only to simplify the visualisation of the example.

All our nodes are called server-X with X being a letter of the English alphabet, thus giving us server-a, server-b, and so on. I decided to create 5 partitions per server, numbered from 0 to 4, which are generated appending -Y to the name, where Y is the number of the partition. For example:

server-a-0 -- hash --> 148456820
server-a-1 -- hash --> 57674441
server-a-2 -- hash --> 216250418
server-a-3 -- hash --> 30595746
server-a-4 -- hash --> 23746828

If we do this for two nodes (server-a and server-b) and then sort the results we will get a full routing table

 23746828 --> server-a-4 ( 6848918 hashes)
 30595746 --> server-a-3 (27078695 hashes)
 57674441 --> server-a-1 ( 3228787 hashes)
 60903228 --> server-b-2 (17957108 hashes)
 78860336 --> server-b-0 ( 7773725 hashes)
 86634061 --> server-b-4 (61822759 hashes)
148456820 --> server-a-0 (67793598 hashes)
216250418 --> server-a-2 (17304439 hashes)
233554857 --> server-b-3 (29289666 hashes)
262844523 --> server-b-1 ( 5590932 hashes)

Remember that the hashes in the routing table are the minimum hash served by that partition. For example, the first line tells us that all hashes from 23746828 are served by the partition server-a-4, while hashes from 30595746 are served by the partition server-a-3. This means that the partition server-a-4 serves 6848918 hashes (as you can read in the table). A key whose hash is 79249022 will be served by server-b-0

 60903228 --> server-b-2 (17957108 hashes)
 78860336 --> server-b-0 ( 7773725 hashes)
                     ^
                     |
 79249022 -----------+

 86634061 --> server-b-4 (61822759 hashes)
148456820 --> server-a-0 (67793598 hashes)

Since partitions are not physically separated, but are just virtual entities belonging to a node, the route table can be simplified to

 23746828 -- > server-a (37156400 hashes)
 60903228 -- > server-b (87553592 hashes)
148456820 -- > server-a (85098037 hashes)
233554857 -- > server-b (34880598 hashes)

What we achieved is remarkable, but there are still two problems. Let's have a look at a simple routing table for three nodes with 5 partitions each

3 nodes with 5 partitions each

 23746828 --> server-a (23267855 hashes)
 47014683 --> server-c (10659758 hashes)
 57674441 --> server-a ( 3228787 hashes)
 60903228 --> server-b (63557309 hashes)
124460537 --> server-c (23996283 hashes)
148456820 --> server-a (31382512 hashes)
179839332 --> server-c (36411086 hashes)
216250418 --> server-a (17304439 hashes)
233554857 --> server-b (15386579 hashes)
248941436 --> server-c (13903087 hashes)
262844523 --> server-b ( 5590932 hashes)

First, the lowest value is not 0, which means that there are some hashes (23,746,828 in this case) which are not served by any slot. Second, in general the distribution doesn't cover the space evenly, as some nodes receive too many keys compared to others. This second problem isn't actually visible in the setups I showed so far, but it becomes noticeable increasing the number of servers. For example, with two nodes we have this situation

2 nodes with 5 partitions each

server-a: 122254437 hashes
server-b: 146181018 hashes

while with 5 nodes it becomes

5 nodes with 5 partitions each

server-a: 64211359 hashes
server-b: 66179053 hashes
server-c: 57545779 hashes
server-d: 43217324 hashes
server-e: 37281940 hashes

As you can see, in the second case the load of server-e is 56% that of server-b.

The first problem is easily solved assigning the initial hashes to the last node, that is considering the hash space mapped on a circle. This means that for 2 nodes with 5 partitions each we have

Routing table of 2 nodes with 5 partitions each

Full routing table
        0 --> server-b-1 (23746828 hashes)
 23746828 --> server-a-4 (6848918 hashes)
 30595746 --> server-a-3 (27078695 hashes)
 57674441 --> server-a-1 (3228787 hashes)
 60903228 --> server-b-2 (17957108 hashes)
 78860336 --> server-b-0 (7773725 hashes)
 86634061 --> server-b-4 (61822759 hashes)
148456820 --> server-a-0 (67793598 hashes)
216250418 --> server-a-2 (17304439 hashes)
233554857 --> server-b-3 (29289666 hashes)
262844523 --> server-b-1 (5590932 hashes)

Simplified routing table
        0 -- > server-b (23746828 hashes)
 23746828 -- > server-a (37156400 hashes)
 60903228 -- > server-b (87553592 hashes)
148456820 -- > server-a (85098037 hashes)
233554857 -- > server-b (34880598 hashes)

where the partition server-b-1 contains the orphaned initial hashes.

The second problem is a matter of statistical approach. The hash function that we use to map the partition name to the key space cannot be controlled, as its diffusion property has been designed to avoid a regular spacing of values. However, if we increase the number of partitions we expect the hash function to spread values across the whole space. At that point, each partition will be assigned just a tiny key space, and the differences between partitions will be less noticeable. In other words, by increasing the number of partitions dramatically we should achieve a better distribution. Let's compare the results of 5 nodes with 2 partitions each

5 nodes with 2 partitions each

server-a 36500586
server-b 76678431
server-c 31738329
server-d 56183426
server-e 67334683

with the results of 5 nodes with 3000 partitions each

5 nodes with 3000 partitions each

server-a 53385222
server-b 53855877
server-c 53755762
server-d 53597662
server-e 53840932

There is clearly an upper limit to the number of partitions that we can create. If we create more partitions than the possible number of hashes we will end up having empty ones and incurring routing errors as some of them will clash, but this is a purely theoretical case: using standard real hash functions we generate hashes of at least 160 bits, which means a codomain of 2¹⁶⁰ possible values (more than 10⁴⁸). With 10,000 nodes (which is a considerable amount of servers in 2022) the threshold would be greater than 10⁴⁴ partitions per server.

So far, we achieved great results, but we already managed to properly partition the space with simple techniques. The real power of consistent hashing is in the way it behaves in a dynamic setting.

Consistent hashing and scaling¶

The interesting thing about consistent hashing is its amazing behaviour in a dynamic environment. As you might remember, the problem with hash partitioning was that a change in the number of nodes had ripple effects that resulted in a massive migration of at least half the keys.

With consistent hashing, when we add a new node we need to generate the hash values for that and put them in the routing table, and at that point we need to migrate the keys that fall under the domain of the newly created slots. Let's see an example before we discuss the performances.

The initial setup is 2 nodes with 5 partitions each

2 nodes with 5 partitions

Full routing table
        0 --> server-b-1 (23746828 hashes)
 23746828 --> server-a-4 (6848918 hashes)
 30595746 --> server-a-3 (27078695 hashes)
 57674441 --> server-a-1 (3228787 hashes)
 60903228 --> server-b-2 (17957108 hashes)
 78860336 --> server-b-0 (7773725 hashes)
 86634061 --> server-b-4 (61822759 hashes)
148456820 --> server-a-0 (67793598 hashes)
216250418 --> server-a-2 (17304439 hashes)
233554857 --> server-b-3 (29289666 hashes)
262844523 --> server-b-1 (5590932 hashes)

Simplified routing table
        0 -- > server-b (23746828 hashes)
 23746828 -- > server-a (37156400 hashes)
 60903228 -- > server-b (87553592 hashes)
148456820 -- > server-a (85098037 hashes)
233554857 -- > server-b (34880598 hashes)

Stats
server-a 122254437
server-b 146181018

TOTAL HASHES: 268435455/268435455

if we add one node we migrate to this new setup

3 nodes with 5 partitions

Full routing table
        0 --> server-b-1 (23746828 hashes)
 23746828 --> server-a-4 (6848918 hashes)
 30595746 --> server-a-3 (16418937 hashes)
 47014683 --> server-c-3 (10659758 hashes)
 57674441 --> server-a-1 (3228787 hashes)
 60903228 --> server-b-2 (17957108 hashes)
 78860336 --> server-b-0 (7773725 hashes)
 86634061 --> server-b-4 (37826476 hashes)
124460537 --> server-c-2 (23996283 hashes)
148456820 --> server-a-0 (31382512 hashes)
179839332 --> server-c-1 (25303093 hashes)
205142425 --> server-c-4 (11107993 hashes)
216250418 --> server-a-2 (17304439 hashes)
233554857 --> server-b-3 (15386579 hashes)
248941436 --> server-c-0 (13903087 hashes)
262844523 --> server-b-1 (5590932 hashes)

Simplified routing table
        0 -- > server-b (23746828 hashes)
 23746828 -- > server-a (23267855 hashes)
 47014683 -- > server-c (10659758 hashes)
 57674441 -- > server-a ( 3228787 hashes)
 60903228 -- > server-b (63557309 hashes)
124460537 -- > server-c (23996283 hashes)
148456820 -- > server-a (31382512 hashes)
179839332 -- > server-c (36411086 hashes)
216250418 -- > server-a (17304439 hashes)
233554857 -- > server-b (15386579 hashes)
248941436 -- > server-c (13903087 hashes)
262844523 -- > server-b ( 5590932 hashes)

Stats
server-a 75183593
server-b 108281648
server-c 84970214

TOTAL HASHES: 268435455/268435455

Let's have a closer look to what happens with server-c

Simplified routing table
        0 -- > server-b (23746828 hashes)
 23746828 -- > server-a (23267855 hashes) ----+ 10659758 hashes
                                              | from server-a
 47014683 -- > server-c (10659758 hashes) <---+
 57674441 -- > server-a ( 3228787 hashes)
 60903228 -- > server-b (63557309 hashes) ----+ 23996283 hashes
                                              | from server-b
124460537 -- > server-c (23996283 hashes) <---+
148456820 -- > server-a (31382512 hashes) ----+ 36411086 hashes
                                              | from server-a
179839332 -- > server-c (36411086 hashes) <---+
216250418 -- > server-a (17304439 hashes)
233554857 -- > server-b (15386579 hashes) ----+ 13903087 hashes
                                              | from server-b
248941436 -- > server-c (13903087 hashes) <---+
262844523 -- > server-b ( 5590932 hashes)

Globally, server-c receives 47,070,844 hashes from server-a and 37,899,370 hashes from server-b, which results in a migration of approximately 30% of the total hashes. As you can see there is no ripple effect here, as the boundaries of the existing partitions do not change.

Let's consider the performances in the worst case when we add one single node. If we are terribly unlucky (and we use a hash function with clear issues) each partition of the new node will cover completely a partition of an existing node. Assuming that the initial setup with N nodes created a balanced cluster, each node contains 1/Nth of the total keys, and in the worst case we need to move all of them from an existing node to the newly added one.

So, adding one node to a cluster of N nodes using consistent hashing results, in the worst case, in the migration of 1/Nth of the keys. In the previous example, then, we expected to migrate at most 50% of the keys (1/2), and we ended up migrating 30$ of them.

This is a terrific result. Not only it's much better than the previous one (at least 50% of the keys), but it gets better increasing the size of the cluster. In a cluster with 100 nodes, adding a node will result (in the worst case!) in the migration of 1/100 of the keys.

Source code¶

All routing tables shown in the post have been created with the following Python script. Please bear in mind that this is just demo code, so things haven't been optimised or designed particularly well. Feel free to change the hash function and the parameters of the script to experiment and see what consistent hashing can do.

consistent_hashing_demo.py

import hashlib
import itertools
import sys
import string

from operator import itemgetter

NUM_NODES = 3
NUM_PARTITIONS = 5


def hash_name(name):
    encoded_name = name.encode("utf-8")
    hash_encoded_name = hashlib.sha1(encoded_name).hexdigest()

    return int(hash_encoded_name[:7], 16)


def create_partitions(node_name, partitions):
    partition_hashes = []

    for partition_number in range(partitions):
        partition_name = f"{node_name}-{partition_number}"
        partition_hash = hash_name(partition_name)

        partition_hashes.append(
            {
                "min_hash": partition_hash,
                "partition_name": partition_name,
                "node_name": node_name,
            }
        )

    return partition_hashes


def create_routing_table(node_names, partitions):
    table = []

    for node_name in node_names:
        table.extend(create_partitions(node_name, partitions))

    table = sorted(table, key=itemgetter("min_hash"))

    return table


if NUM_NODES > len(string.ascii_lowercase):
    print("Too many servers")
    sys.exit(1)

nodes = [f"server-{i}" for i in string.ascii_lowercase[:NUM_NODES]]

routing_table = create_routing_table(nodes, NUM_PARTITIONS)
routing_table = [
    {
        "min_hash": 0,
        "partition_name": routing_table[-1]["partition_name"],
        "node_name": routing_table[-1]["node_name"],
    }
] + routing_table

routing_table_shift = routing_table[1:] + [
    {"min_hash": 0xFFFFFFF, "partition_name": "END"}
]

full_routing_table = []
for i, j in zip(routing_table, routing_table_shift):
    full_routing_table.append(
        {
            "min_hash": i["min_hash"],
            "partition_name": i["partition_name"],
            "node_name": i["node_name"],
            "served_hashes": j["min_hash"] - i["min_hash"],
        }
    )

print("Full routing table")
for r in full_routing_table:
    print(f'{r["min_hash"]:9} --> {r["partition_name"]} ({r["served_hashes"]} hashes)')

grouped_routing_table = itertools.groupby(
    full_routing_table, key=itemgetter("node_name")
)


simplified_routing_table = []
for r in grouped_routing_table:
    consecutive_partitions = list(r[1])

    simplified_routing_table.append(
        {
            "node_name": r[0],
            "min_hash": consecutive_partitions[0]["min_hash"],
            "served_hashes": sum([i["served_hashes"] for i in consecutive_partitions]),
        }
    )

print()
print("Simplified routing table")
for r in simplified_routing_table:
    print(f'{r["min_hash"]:9} -- > {r["node_name"]} ({r["served_hashes"]:8} hashes)')

print()
print("Stats")
stats = []
for node in nodes:
    slots = filter(lambda x: x["node_name"] == node, simplified_routing_table)
    total_hashes = sum([i["served_hashes"] for i in slots])
    stats.append({"node_name": node, "served_hashes": total_hashes})

for r in stats:
    print(r["node_name"], r["served_hashes"])

total_hashes = sum([i["served_hashes"] for i in stats])
print()
print(f"TOTAL HASHES: {total_hashes}/{2**28 - 1}")

Final words¶

I hope this long post was useful to introduce you to the topic of partitioning and in general to system design. As I mentioned, such concepts are currently in use by well-known systems, and still discussed as none of them is perfect, so it is worth understanding the fundamental issues before adopting a specific solution.

Resources¶

Martin Kleppmann, Designing Data-Intensive Applications, Chapter 6 "Partitioning", O’Reilly 2017 official site.
The Wikipedia article about consistent hashing.
A Guide to Consistent Hashing by Juan Pablo Carzolio.
The original article by David Karger et al.: "Consistent Hashing and Random Trees: Distributed Caching protocols for Relieving Hot Spots ont the World Wide Web".
An alternative algorithm by John Lamping and Eric Veach: "A Fast, Minimal Memory, Consistent Hash Algorithm".

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Photo by Alex Lvrs on Unsplash.

From Docker CLI to Docker Compose

2022-02-19T15:00:00+01:00

In this post I will show you how and why Docker Compose is useful, building a simple application written in Python that uses PostgreSQL. I think it is worth going through such an exercise to see how technologies that we might be already familiar with actually simplify workflows that would otherwise definitely be more complicated.

The name of the demo application I will develop is a very unimaginative whale, that shouldn't clash with any other name introduced by the tools I will use. Every time you see something with whale in it you know that I am referring to a value that you can change according to your setup.

Before we start, please create a directory to host all the files we will create. I will refer to this directory as the "project directory".

PostgreSQL¶

Since the application will connect to a PostgreSQL database the first thing we can explore is how to run that in a Docker container.

The official Postgres image can be found here, and I highly recommend taking the time to properly read the documentation, as it contains a myriad of details that you should be familiar with.

For the time being, let's focus on the environment variables that the image requires you to set.

Password

The first variable is POSTGRES_PASSWORD, which is the only mandatory configuration value (unless you disable authentication which is not recommended). Indeed, if you run the image without setting this value, you get this message

$ docker run postgres
Error: Database is uninitialized and superuser password is not specified.
       You must specify POSTGRES_PASSWORD to a non-empty value for the
       superuser. For example, "-e POSTGRES_PASSWORD=password" on "docker run".

       You may also use "POSTGRES_HOST_AUTH_METHOD=trust" to allow all
       connections without a password. This is *not* recommended.

       See PostgreSQL documentation about "trust":
       https://www.postgresql.org/docs/current/auth-trust.html

This value is very interesting because it's a secret. So, while I will treat it as a simple configuration value in the first stages of the setup, later we will need to discuss how to manage it properly.

Superuser

Being a production-grade database, Postgres allows you to specify users, groups, and permissions in a fine-grained fashion. I won't go into that as it's usually more a matter of database administration and application development, but we need to define at least the superuser. The default value for this image is postgres, but you can change it setting POSTGRES_USER.

Database name

If you do not specify the value of POSTGRES_DB, this image will create a default database with the name of the superuser.

A note of warning here. If you omit both the database name and the user you will end up with the superuser postgres and database postgres. The official documentation states that

After initialization, a database cluster will contain a database named
postgres, which is meant as a default database for use by utilities,
users and third party applications. The database server itself does not
require the postgres database to exist, but many external utility programs
assume it exists.

This mean that it is not ideal to use that as the database for our application. So, unless you are just trying out a quick piece of code, my recommendation is to always configure all three values: POSTGRES_PASSWORD, POSTGRES_USER, and POSTGRES_DB.

We can run the image with

$ docker run -d \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  postgres:13

As you can see I run the image in detached mode. This image is not meant to be interactive, as Postgres is by it's very nature a daemon. To connect in an interactive way we need to use the tool psql, which is provided by this image. Please note that I'm running postgres:13 only to keep the post consistent with what you will see if you read it in the future, you are clearly free to use any version of the engine.

The ID of the container is returned by docker run but we can retrieve it any time running docker ps. Using IDs is however pretty complicated, and looking at the command history is not immediately clear what you have been doing at a certain point in time. For this reason, it's a good idea to name the containers.

Stop the previous container and run it again with

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  postgres:13

Stopping containers

You can stop containers using docker stop ID. This gives containers a grace period to react to the SIGTERM signal, for example to properly close files and terminate connections, and then terminates it with SIGKILL. You can also force it to stop unconditionally using docker kill ID which sends SIGKILL immediately.

In either case, however, you might want to remove the container, that otherwise will be kept indefinitely by Docker. This can become a problem when containers are named, as you can't reuse a name that is currently assigned to a container.

To remove a container you have to run docker rm ID, but you can leverage the fact that both docker stop and docker kill return the ID of the container to pipe the termination and the removal

$ docker stop ID | xargs docker rm

Otherwise, you can use docker rm -f ID, which corresponds to docker kill followed by docker rm. If you name a container, however, you can use its name instead of the ID.

Now we can connect to the database using the executable psql provided in the image itself. To execute a command inside a container we use docker exec and this time we will specify -it to open an interactive session. psql uses by default the user name root, and the database with the same name as the user, so we need to specify both. The header informs me that the image is running PostgreSQL 13.5 on Debian.

$ docker exec -it whale-postgres psql -U whale_user whale_db
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=#

Postgres trust

You might be surprised by the fact that psql didn't ask for the password that we set when we run the container. This happens because the server trusts local connections, and when we run psql inside the container we are on localhost.

If you are curious about trust in Postgres you can see the configuration file with

$ docker exec -it whale-postgres \
  cat /var/lib/postgresql/data/pg_hba.conf

where you can spot the lines

# TYPE  DATABASE  USER  ADDRESS  METHOD

# "local" is for Unix domain socket connections only
local   all       all            trust

You can find more information about Postgres trust in the official documentation.

Here, I can list all the databases with \l. You can see all psql commands and the rest of the documentation at https://www.postgresql.org/docs/current/app-psql.html.

$ docker exec -it whale-postgres psql -U whale_user whale_db
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=# \l
                                    List of databases
   Name    |   Owner    | Encoding |  Collate   |   Ctype    |     Access privileges     
-----------+------------+----------+------------+------------+---------------------------
 postgres  | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | 
 template0 | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | =c/whale_user            +
           |            |          |            |            | whale_user=CTc/whale_user
 template1 | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | =c/whale_user            +
           |            |          |            |            | whale_user=CTc/whale_user
 whale_db  | whale_user | UTF8     | en_US.utf8 | en_US.utf8 | 
(4 rows)

whale_db=#

As you can see, the database called postgres has been created as part of the initialisation, as clarified previously. You can exit psql with Ctrl-D or \q.

If we want the database to be accessible from outside we need to publish a port. The image exposes port 5432 (see the source code), which tells us where the server is listening. To publish the port towards the host system we can add -p 5432:5432. Please remember that exposing a port in Docker basically means to add some metadata that informs the user of the image, but doesn't affect the way it runs.

Stop the container (you can use its name now) and run it again with

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 postgres:13

Running docker ps we can see that the container publishes the port now (0.0.0.0:5432->5432/tcp). We can double-check it with ss ("socket statistics")

$ ss -nulpt | grep 5432
tcp  LISTEN  0  4096  0.0.0.0:5432  0.0.0.0:*
tcp  LISTEN  0  4096     [::]:5432     [::]:*

Please note that usually ss won't tell you the name of the process using that port because the process is run by root. If you run ss with sudo you will see it

$ sudo ss -nulpt | grep 5432
tcp  LISTEN  0  4096  0.0.0.0:5432  0.0.0.0:*  users:(("docker-proxy",pid=1262717,fd=4))
tcp  LISTEN  0  4096     [::]:5432     [::]:*  users:(("docker-proxy",pid=1262724,fd=4))

Unfortunately, ss is not available on macOS. On that platform (and on Linux as well) you can use lsof with grep

$ sudo lsof -i -p -n | grep 5432
docker-pr 219643            root    4u  IPv4 2945982      0t0  TCP *:5432 (LISTEN)
docker-pr 219650            root    4u  IPv6 2952986      0t0  TCP *:5432 (LISTEN)

or directly using the option -i

$ sudo lsof -i :5432
COMMAND      PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
docker-pr 219643 root    4u  IPv4 2945982      0t0  TCP *:postgresql (LISTEN)
docker-pr 219650 root    4u  IPv6 2952986      0t0  TCP *:postgresql (LISTEN)

Please note that docker-pr in the output above is just docker-proxy truncated, matching what we saw with ss previously.

If you want to publish the container's port 5432 to a different port on the host you can just use -p ANY_NUMBER:5432. Remember however that port numbers under 1024 are privileged or well-known, which means that they are assigned by default to specific services (listed here).

This means that in theory you can use -p 80:5432 for your database container, exposing it on port 80 of your host. In practice this will result in a lot of headaches and a bunch of developers chasing you with spikes and shovels.

Now that we exposed a port we can connect to the database running psql in an ephemeral container. "Ephemeral" means that a resource (in this case a Docker container) is run just for the time necessary to serve a specific purpose, as opposed to "permanent". This way we can simulate someone that tries to connect to the Docker container from a different computer on the network.

Since psql is provided by the image postgres we can in theory run that passing the hostname with -h localhost, but if you try it you will be disappointed.

$ docker run -it postgres:13 psql -h localhost -U whale_user whale_db
psql: error: connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (::1), port 5432 failed: Cannot assign requested address
        Is the server running on that host and accepting TCP/IP connections?

This is correct, as that container runs in a bridge network where localhost is the container itself. To make it work we need to run the container as part of the host network (that is the same network our computer is running on). This can be done with --network=host

$ docker run -it \
  --network=host postgres:13 \
  psql -h localhost -U whale_user whale_db
Password for user whale_user: 
psql (13.5 (Debian 13.5-1.pgdg110+1))
Type "help" for help.

whale_db=#

Please note that now psql asks for a password (that you know because you set it when we run the container whale-postgres). This happens because the tool is not run on the same node as the database server any more, so PostgreSQL doesn't trust it.

Volumes¶

If we used a structured framework in Python, we could leverage an ORM like SQLAlchemy to map classes to database tables. The model definitions (or changes) can be captured into little scripts called migrations that are applied to the database, and those can also be used to insert some initial data. For this example I will go a simpler route, that is to initialise the database using SQL directly.

I do not recommend this approach for a real project but it should be good enough in this case. In particular, it will allow me to demonstrate how to use volumes in Docker.

Make sure the container whale-postgres is running (with or without publishing the port, it's not important at the moment). Connect to the container using psql and run the following two SQL commands (make sure you are connected to the database whale_db)

CREATE TABLE recipes (
  recipe_id INT NOT NULL,
  recipe_name VARCHAR(30) NOT NULL,
  PRIMARY KEY (recipe_id),
  UNIQUE (recipe_name)
);

INSERT INTO recipes 
    (recipe_id, recipe_name) 
VALUES 
    (1,'Tacos'),
    (2,'Tomato Soup'),
    (3,'Grilled Cheese');

This code creates a table called recipes and inserts 3 rows with an id and a name. The output of the above commands should be

CREATE TABLE
INSERT 0 3

You can double check that the database contains the table with \dt

whale_db=# \dt
           List of relations
 Schema |  Name   | Type  |   Owner    
--------+---------+-------+------------
 public | recipes | table | whale_user
(1 row)

and that the table contains three rows with a select.

whale_db=# select * from recipes;
 recipe_id |  recipe_name   
-----------+----------------
         1 | Tacos
         2 | Tomato Soup
         3 | Grilled Cheese
(3 rows)

Now, the problem with containers is that they do not store data permanently. While the container is running there are no issues, as a matter of fact you can terminate psql, connect, and run the select again, and you will see the same data.

If we stop the container and run it again, though, we will quickly realise that the values stored in the database are gone.

$ docker stop whale-postgres | xargs docker rm 
whale-postgres

$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 postgres:13
4a647ebef78e32bb4733484a6e435780e17a69b643e872613ca50115d60d54ce

$ docker exec -it whale-postgres \
  psql -U whale_user whale_db -c "select * from recipes"
ERROR:  relation "recipes" does not exist
LINE 1: select * from recipes
                      ^

Containers have been created with isolation in mind, which is why by default nothing of what happens inside the container is connected with the host and is preserved when the container is destroyed.

As happened with ports, however, we need to establish some communication between containers and the host system, and we also want to keep data after the container has been destroyed. The solution in Docker is to use volumes.

There are three types of volumes in Docker: host, anonymous, and named. Host volumes are a way to mount inside the container a path on the host's filesystem, and while they are useful to exchange data between the host and the container, they also often have permissions issues. Generally speaking, containers define users whose IDs are not mapped to the host's ones, which means that the files written by the container might end up belonging to non-existing users.

Anonymous and named volumes are simply virtual filesystems created and managed independently from containers. These can be connected with a running container so the latter can use the data contained in them and store data that will survive its termination. The only difference between named an anonymous volumes is the name that allows you to easily manage them. For this reason, I think it's not really useful to consider anonymous volumes, which is why I will focus on named ones.

You can manage volumes using docker volume, that provides several subcommands such as create, and rm. You can then attach a named volume to a container when you run it using the option -v of docker run. This creates the volume if it's not already existing, so this is the standard way many of us create a volume.

Stop and remove the running Postgres container and run it again with a named volume

$ docker stop whale-postgres | xargs docker rm 
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13

This will create the volume named whale_dbdata and connect it to the path /var/lib/postgresql/data in the container that we are running. That path happens to be the one where Postgres stores the actual database, as you can see from the official documentation. There is a specific reason why I used the prefix whale_ for the name of the volume, which will be clear later when we will introduce Docker Compose.

docker ps doesn't give any information on volumes, so to see what is connected to your container you need to use docker inspect

$ docker inspect whale-postgres 
[...]
        "Mounts": [
            {
                "Type": "volume",
                "Name": "whale_dbdata",
                "Source": "/var/lib/docker/volumes/whale_dbdata/_data",
                "Destination": "/var/lib/postgresql/data",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
[...]

The value for "Source" is where the volume is stored in the host, that is on your computer, but generally speaking you can ignore that detail. You can see all volumes using docker volume ls (using grep if the list is long as it is in my case)

$ docker volume ls | grep whale
local     whale_dbdata

Now that the container is running and is connected to a volume, we can try to initialise the database again. Connect with psql using the command line we developed before and run the SQL commands that create the table recipes and insert three rows.

The whole point of using a volume is to make information permanent, so now terminate and remove the Postgres container, and run it again using the same volume. You can check that the database still contains data using the query shown previously.

$ docker rm -f whale-postgres 
whale-postgres
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  -p 5432:5432 \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13
893378f044204e5c1a87473a038b615a08ad08e5da9225002a470caeac8674a8
$ docker exec -it whale-postgres \
  psql -U whale_user whale_db \
  -c "select * from recipes"
 recipe_id |  recipe_name   
-----------+----------------
         1 | Tacos
         2 | Tomato Soup
         3 | Grilled Cheese
(3 rows)

Python application¶

Great! Now that we have a database that can be restarted without losing data we can create a Python application that interacts with it. Again, please remember that the goal of this post is to show what container orchestration is and how Docker compose can simplify it, so the application developed in this section is absolutely minimal.

I will first create an application and run it in the host, leveraging the port exposed by the container to connect to the database. Later, I will move the application in its own container.

To create the application, first create a Python virtual environment using your preferred method. I currently use pyenv (https://github.com/pyenv/pyenv).

pyenv virtualenv whale_docker
pyenv activate whale_docker

Now we need to put our requirements in a file and install them. I prefer to keep things tidy from day zero, so create the directory whaleapp in the project directory and inside it the file requirements.txt.

mkdir whaleapp
touch whaleapp/requirements.txt

The only requirement we have for this simple application is psycopg2, so I add it to the file and then install it. Since we are installing requirements is useful to update pip as well.

echo "psycopg2" >> whaleapp/requirements.txt
pip install -U pip
pip install -r whaleapp/requirements.txt

Now create the file whaleapp/whaleapp.py and put this code in it

whaleapp/whaleapp.py

import time

import psycopg2

connection_data = { 1
    "host": "localhost",
    "database": "whale_db",
    "user": "whale_user",
    "password": "whale_password",
}

while True:
    try:
        conn = None

        # Connect to the PostgreSQL server
        print("Connecting to the PostgreSQL database...")
        conn = psycopg2.connect(**connection_data) 2

        # Create a cursor
        cur = conn.cursor()

        # Execute the query
        cur.execute("select * from recipes") 3

        # Fetch all results
        results = cur.fetchall()
        print(results) 4

        # Close the connection
        cur.close()
    except (Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close() 5
            print("Database connection closed.")

    # Wait three seconds
    time.sleep(3)

As you can see the code is not complicated. The application is an endless while loop that every 3 seconds establishes a connection with the DB 2 using the configuration in 1. After this, the query select * from recipes is run 3 , all the results are printed on the standard output 4, and the connection is closed 5.

If the Postgres container is running and publishing port 5432, this application can be run directly on the host

$ python whaleapp.py 
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

and will go on indefinitely until we press Ctrl-C to stop it.

For the same reasons of isolation and security that we discussed previously, we want to run the application in a Docker container. This can be done pretty easily, but we will run into the same issues that we had when we where trying to run psql in a separate container. At the moment, the application tries to connect to the database on localhost, which is fine while the application is running on the host directly, but won't work any more once that is transported into a Docker container.

To face one problem at a time, let's first containerise the application and run it using the host network. Once this works, we can see how to solve the communication problem between containers.

The easiest way to containerise a Python application is to create a new image starting from the image python:3. The following Dockerfile goes into the application directory

whaleapp/Dockerfile

FROM python:3 1

WORKDIR /usr/src/app 2

COPY requirements.txt . 3
RUN pip install --no-cache-dir -r requirements.txt 4

COPY . . 5

CMD [ "python", "-u", "./whaleapp.py" ] 6

A Docker file contains the description of the layers that build an image. Here, we start from the official Python 3 image 1 (https://hub.docker.com/_/python), set a working directory 2, copy the requirements file 3 and install the requirements 4, then copy the rest of the application 5, and run the application 6. The Python option -u avoids output buffering, see https://docs.python.org/3/using/cmdline.html#cmdoption-u.

It is important to keep in mind the layered nature of Docker images, as this can lead to simple optimisation tricks. In this case, loading the requirements file and installing them creates a layer out of a file that doesn't change very often, while the layer created at 5 is probably changing very quickly while we develop the application. If we run something like

[...]

COPY . .

RUN pip install --no-cache-dir -r requirements.txt

CMD [ "python", "-u", "./app.py" ]

we would have to install the requirements every time we change the application code, as this would rebuild the COPY layer and thus invalidate the layer containing the RUN command.

Once the Dockerfile is in place we can build the image

$ cd whaleapp
$ docker build -t whaleapp .
Sending build context to Docker daemon  6.144kB
Step 1/6 : FROM python:3
 ---> 768307cdb962
Step 2/6 : WORKDIR /usr/src/app
 ---> Using cache
 ---> b00189756ddb
Step 3/6 : COPY requirements.txt .
 ---> a7aef12f562c
Step 4/6 : RUN pip install --no-cache-dir -r requirements.txt
 ---> Running in 153a3ca6a1b2
Collecting psycopg2
  Downloading psycopg2-2.9.3.tar.gz (380 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.9.3-cp39-cp39-linux_x86_64.whl size=523502 sha256=1a3aac3cf72cc86b63a3e0f42b9b788c5237c3e5d23df649ca967b29bf89ecf5
  Stored in directory: /tmp/pip-ephem-wheel-cache-ow3d1yop/wheels/b3/a1/6e/5a0e26314b15eb96a36263b80529ce0d64382540ac7b9544a9
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.9.3
WARNING: You are using pip version 20.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removing intermediate container 153a3ca6a1b2
 ---> b18aead1ef15
Step 5/6 : COPY . .
 ---> be7c3c11e608
Step 6/6 : CMD [ "python", "-u", "./app.py" ]
 ---> Running in 9e2f4f30b59e
Removing intermediate container 9e2f4f30b59e
 ---> b735eece4f86
Successfully built b735eece4f86
Successfully tagged whaleapp:latest

You can see the layers being built one by one (marked as Step x/6 here). Once the image has been build you should be able to see it in the list of images present in your system

$ docker image ls | grep whale
whaleapp  latest  969b15466905  9 minutes ago  894MB

Size of containers

You might want to observe 1 minute of silence meditating on the fact that we used almost 900 megabytes of space to run 40 lines of Python. As you can see benefits come with a cost, and you should not underestimate those. 900 megabytes might not seem a lot nowadays, but if you keep building images you will soon use up the space on your hard drive or end up paying a lot for the space on your remote repository.

By the way, this is the reason why Docker splits image into layers and reuses them. For now we can ignore this part of the game, but remember that keeping the system clean and removing past artefacts is important.

As I mentioned before we can run this image but we need to use the host network configuration.

$ docker run -it --rm --network=host --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Please note that I used --rm to make Docker remove the container automatically when it is terminated. This way I can run it again with the same name without having to explicitly remove the past container with docker rm.

Run containers in the same network¶

Docker containers are isolated from the host and from other containers by default. This however doesn't mean that they can't communicate with each other if we run them in a specific configuration. In particular, an important part in Docker networking is played by bridge networks.

Whenever containers are run in the same custom bridge network, Docker provides them DNS resolution using the container names. This means that we can make the application communicate with the database without having to run the former in the host network.

A custom network can be created using docker network

$ docker network create whale

As always, Docker will return the ID of the object it just created, but we can ignore it for now, as we can refer to the network by name.

Stop and remove the Postgres container, and run it again using the network whale

$ docker rm -f whale-postgres 
whale-postgres
$ docker run -d \
  --name whale-postgres \
  -e POSTGRES_PASSWORD=whale_password \
  -e POSTGRES_DB=whale_db \
  -e POSTGRES_USER=whale_user \
  --network=whale \
  -v whale_dbdata:/var/lib/postgresql/data \
  postgres:13

Please note that there is no need to publish the port 5432 in this setup, as the host doesn't need to access the container. Should this be a requirement, add the option -p 5432:5432 again.

As happened with volumes, docker ps doesn't give information about the network that containers are using, so you have to use docker inspect again

$ docker inspect whale-postgres 
[...]
        "NetworkSettings": {
            "Networks": {
                "whale": {
[...]

Docker network management

The command docker network can be used to change the network configuration of running containers.

You can disconnect a running container from a network with

$ docker network disconnect NETWORK_ID CONTAINER_ID

and connect it with

$ docker network connect NETWORK_ID CONTAINER_ID

You can see which containers are using a given network inspecting it

$ docker network inspect NETWORK_ID

Remember that disconnecting a container from a network makes it unreachable, so while it is good that we can do this on running containers, maintenance shall be always carefully planned to avoid unexpected downtime.

As I mentioned before, Docker bridge networks provide DNS resolution using the container's name. We can double check this running a container and using ping.

$ docker run -it --rm --network=whale whaleapp ping whale-postgres
PING whale-postgres (172.19.0.2) 56(84) bytes of data.
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=2 ttl=64 time=0.100 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=3 ttl=64 time=0.115 ms
64 bytes from whale-postgres.whale (172.19.0.2): icmp_seq=4 ttl=64 time=0.101 ms
^C
--- whale-postgres ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 80ms
rtt min/avg/max/mdev = 0.064/0.095/0.115/0.018 ms

What I did here was to run the image whaleapp that we built previously, but overriding the default command and running ping whale-postgres instead. This is a good way to check if a host can resolve a name on the network (dig is another useful tool but is not installed by default in that image).

As you can see the Postgres container is reachable and we also know that it currently runs with the IP 172.19.0.2. This value might be different on your system, but it will match the information you get if you run docker network inspect whale.

The point of all this talk about DNS is that we can now change the code of the Python application so that it connects to whale-postgres instead of localhost

connection_data = {
    "host": "whale-postgres",
    "database": "whale_db",
    "user": "whale_user",
    "password": "whale_password",
}

Once this is done, rebuild the image and run it in the whale network

$ docker build -t whaleapp .
[...]
$ docker run -it --rm --network=whale --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

You can also take the network directly from another container, which is a useful shortcut.

$ docker build -t whaleapp .
[...]
$ docker run -it --rm \
  --network=container:whale-postgres \
  --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Run time configuration¶

Hardcoding configuration values into the application is never a great idea, and while this is a very simple example it is worth pushing the setup a bit further to make it tidy.

In particular, we can replace the connection data host, database, and user with environment variables, which allow us to reuse the application configuring it at run time. For simplicity's sake I will store the password in an environment variable as well, and pass it in clear text when we run the container. See the box for more information about how to manage secret values.

Reading values from environment variables is easy in Python

import os
import time

import psycopg2

DB_HOST = os.environ.get("WHALEAPP__DB_HOST", None)
DB_NAME = os.environ.get("WHALEAPP__DB_NAME", None)
DB_USER = os.environ.get("WHALEAPP__DB_USER", None)
DB_PASSWORD = os.environ.get("WHALEAPP__DB_PASSWORD", None)

connection_data = {
    "host": DB_HOST,
    "database": DB_NAME,
    "user": DB_USER,
    "password": DB_PASSWORD,
}

Please note that I prefixed all environment variables with WHALEAPP__. This is not mandatory, and has no special meaning for the operating system. In my experience, complicated systems can have many environment variables, and using prefixes is a simple and effective way to keep track of which part of the system needs that particular value.

We already know how to pass environment variables to Docker containers as we did it when we run the Postgres container. Build the image again, and then run it passing the correct variables

$ docker build -t whaleapp .
[...]
$ docker run -it --rm --network=whale \
  -e WHALEAPP__DB_HOST=whale-postgres \
  -e WHALEAPP__DB_NAME=whale_db \
  -e WHALEAPP__DB_USER=whale_user \
  -e WHALEAPP__DB_PASSWORD=password \
  --name whale-app whaleapp
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.
Connecting to the PostgreSQL database...
[(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
Database connection closed.

Managing secrets

A "secret" is a value that should never be shown in plain text, as it is used to grant access to a system. This can be a password or a private key such as the ones you have to run SSH, and as happens with everything related to security, managing them is complicated. Please keep in mind that security is hard and that the best attitude to have is: every time you think something in security is straightforward this means you got it wrong.

Generally speaking, you want secrets to be encrypted and stored in a safe place where access is granted to a narrow set of people. These secrets should be accessible to your application in a secure way, and it shouldn't be possible to access the secrets hosted in the memory of the application.

For example, many posts online show how you can use AWS Secrets Manager to store your secrets and access them from your application using jq to fetch them at run time. While this works, if the JSON secret contains a syntax error, jq dumps the whole value in the standard output of the application, which means that the logs contain the secret in plain text.

Vault is a tool created by Hashicorp that many use to store secrets needed by containers. It is interesting to read in the description of the image that with a specific configuration the container prevents memory from being swapped to disk, which would leak the unencrypted values. As you see, security is hard.

Orchestration tools always provide a way to manage secrets and to pass them to containers. For example, see Docker Swarm secrets, Kubernetes secrets, and secrets for AWS Elastic Container Service.

Enter Docker Compose¶

The setup we created in the past sections is good, but is far from being optimal. We had to create a custom bridge network and then start the Postgres and the application containers connected to it. To stop the system we need to terminate containers manually and to remember to remove them to avoid blocking the container name. We also have to manually remove the network if we want to keep the system clean.

The next step would then be to create a bash script, then to evolve it to a Makefile or similar solution. Fortunately, Docker provides a better solution with Docker Compose.

Docker Compose can be described as a single-host orchestration tool. Orchestration tools are pieces of software that allow us to deal with the problems described previously, such as starting and terminating multiple containers, creating networks and volumes, managing secrets, and so on. Docker Compose works in a single-host mode, so it's a great solution for development environment, while for production multi-host environments it's better to move to more advanced tools such as AWS ECS or Kubernetes.

Docker Compose reads the configuration of a system from the file docker-compose.yml (the default value, it can be changed) that captures all we did manually in the previous sections in a compact and readable way.

To install Docker Compose follow the instructions you find at https://docs.docker.com/compose/install/. Before we start using Docker Compose make sure you kill the Postgres container if you are still running it, and remove the network we created

$ docker rm -f whale-postgres 
whale-postgres
$ docker network remove whale
whale

Then create the file docker-compose.yml in the project directory (not the app directory) and put the following code in it

docker-compose.yml

version: '3.8'

services:

This is not a valid Docker Compose file, yet, but you can see that there is a value that specifies the syntax version and one that lists services. You can find the Compose file reference at https://docs.docker.com/compose/compose-file/, together with a detailed description of the various versions.

The first service we want to run is Postgres, and a basic configuration for that is

docker-compose.yml

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes: 2
      - dbdata:/var/lib/postgresql/data

volumes: 1
  dbdata:

As you can see, this file contains the environment variables that we passed to the Postgres container and the volume configuration. The final volumes 1 declares which volumes have to be present (so it creates them if they are not), while volumes 2 inside the service db creates the connection just like the option -v did previously.

Now, from the project directory, you can run Docker Compose with

$ docker-compose -p whale up -d
Creating network "whale_default" with the default driver
Creating whale_db_1 ... done

The option -p sets the name of the project, which otherwise would be by default that of the directory you are at the moment (which might or might not be meaningful), while the command up -d starts all the containers in a detached mode.

As you can see from the output, Docker Compose creates a (bridge) network called whale_default. Normally, you would see a message like Creating volume "whale_dbdata" with default driver as well, but in this case the volume is already present as we created it previously. Both the network and the volume are prefixed with PROJECTNAME_, and this is the reason why when we first created the volume I named it whale_dbdata. Keep in mind however that all these default behaviours can be customised in the Compose file.

If you run docker ps you will see that the container is named whale_db_1. This comes from the project name (whale_), the service name in the Compose file (db_) and the container number, which is 1 because at the moment we are running only one container for that service.

To stop the services you have to run

$ docker-compose -p whale down
Stopping whale_db_1 ... done
Removing whale_db_1 ... done
Removing network whale_default

As you can see from the output, Docker Compose stops and removes the container, then removes the network. This is very convenient, as it already removes a lot of the work we had to do manually earlier.

We can now add the application container to the Compose file

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes:
      - dbdata:/var/lib/postgresql/data
  app:
    build:
      context: whaleapp
      dockerfile: Dockerfile
    environment:
      WHALEAPP__DB_HOST: db
      WHALEAPP__DB_NAME: whale_db
      WHALEAPP__DB_USER: whale_user
      WHALEAPP__DB_PASSWORD: whale_password

volumes:
  dbdata:

This definition is slightly different, as the application container has to be built using the Dockerfile we created. Docker Compose allows us to store here the build configuration so that we don't need to pass al the options to docker build manually, but please note that configuring the build here doesn't mean that Docker Compose will build the image for you every time. You still need to run docker-compose -p whale build every time you need to rebuild it.

Please note that the variable WHALEAPP__DB_HOST is set to the service name, and not to the container name. Now, when we run Docker Compose we get

$ docker-compose -p whale up -d
Creating network "whale_default" with the default driver
Creating whale_db_1  ... done
Creating whale_app_1 ... done

and the output tells us that also the container whale_app_1 has been created this time. We can see the logs of a container with docker logs, but using docker-compose allows us to call services by name instead of by ID

$ docker-compose -p whale logs -f app
Attaching to whale_app_1
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.

Health checks and dependencies¶

You might have noticed that at the very beginning of the application logs there are some connection errors, and that after a while the application manages to connect to the database

$ docker-compose -p whale logs -f app
Attaching to whale_app_1
app_1  | Connecting to the PostgreSQL database...
app_1  | could not translate host name "db" to address: Name or service not known
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | could not translate host name "db" to address: Name or service not known
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | Connecting to the PostgreSQL database...
app_1  | could not connect to server: Connection refused
app_1  |        Is the server running on host "db" (172.31.0.3) and accepting
app_1  |        TCP/IP connections on port 5432?
app_1  | 
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.
app_1  | Connecting to the PostgreSQL database...
app_1  | [(1, 'Tacos'), (2, 'Tomato Soup'), (3, 'Grilled Cheese')]
app_1  | Database connection closed.

These errors come from the fact that the application container is up and running before the database is ready to serve connections. In a production setup this usually doesn't happen because the database is up and running much before the application gets deployed for the first time, and then runs (hopefully) without interruption. In a development environment, instead, such a situation is normal.

Please note that this might not happen in your setup, as this is tightly connected with the speed of Docker Compose and the containers. Time-sensitive bugs are one of the worst types to deal with, and this is the reason why managing distributed systems is hard. It is important that you realise that even though this might work now on your system, the problem is there and we need to find a solution.

The standard solution when part of a system depends on another is to create a health check that periodically tests the first service, and to start the second service only when the check is successful. We can do this in the Compose file using healthcheck and depends_on

version: '3.8'

services:
  db:
    image: postgres:13
    environment:
      POSTGRES_DB: whale_db
      POSTGRES_PASSWORD: whale_password
      POSTGRES_USER: whale_user
    volumes:
      - dbdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 5
  app:
    build:
      context: whaleapp
      dockerfile: Dockerfile
    environment:
      WHALEAPP__DB_HOST: db
      WHALEAPP__DB_NAME: whale_db
      WHALEAPP__DB_USER: whale_user
      WHALEAPP__DB_PASSWORD: whale_password
    depends_on:
      db:
        condition: service_healthy

volumes:
  dbdata:

The health check for the Postgres container leverages the command line tool pg_isready that is successful only when the database is ready to accept connections, and tries every 10 seconds for 5 times. Now, when you run up -d this time you should notice a clear delay before the application is run, but the logs won't contain any connection error.

Final words¶

Well, this was a long one, but I hope you enjoyed the trip and you ended up having a better picture of what problems Docker Compose solve, along with a feeling of how complicated it might be to design an architecture. Everything we did was for a "simple" development environment with a couple of containers, so you can figure what is involved when we get to live environments.

Updates¶

2022-03-17: Thanks to my colleague Joanna Stadnik for a thorough review, for spotting typos, and for giving me several suggestions based on her experience. Thank you!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Cover picture by Verstappen Photography on Unsplash.

How to write a Pelican theme for your static website

2021-03-25T14:00:00+01:00

I run The Digital Cat using a static site generator called Pelican, created by my friend Justin Mayer and actively maintained by him and other developers. I also gave some minor contributions to the project.

Since I started working on the blog in 2013 I run a great length to customise the theme that I use. I initially went for a pre-made Pelican theme, but I soon started to change small things, and eventually ended up creating a whole new theme that suits my needs.

Front-end development if not my forte, though, so I didn't want to start from scratch with HTML, CSS and JS. My knowledge of those tools is limited, and I have other interests, so I started from a free (CC BY 3.0) pre-made HTML5 template created by HTML5 UP. You can see a demo of the original template and compare it with what you see on this very page.

Encouraged by Justin, I decided to write down this initial guide on how to port a Pelican theme from a static template. I will show you how to start a blog from scratch, how to get a static template and how to make it usable by Pelican. Everything done step by step without skipping any passage. At the end of the post you will have a running blog with some demo articles, you will have learned how to use the Jinja language and the Pelican variables, and you will have an idea of what to do next to further customise your static website.

Let's start!

Initial setup¶

Let's create a blog called The Analog Fox, following Pelican's quickstart guide.

I created a virtual environment and installed Pelican as suggested, then run

mkdir theanalogfox
cd theanalogfox
pelican-quickstart

For this project I will only run the blog locally, so I didn't configure any specific way to publish it, neither properly set up a URL prefix. If you are about to create a real website please read Pelican's documentation about those settings.

> Where do you want to create your new web site? [.] 
> What will be the title of this web site? The Analog Fox
> Who will be the author of this web site? Leonardo Giordani
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., https://example.com   (Y/n) n
> Do you want to enable article pagination? (Y/n) 
> How many articles per page do you want? [10] 3
> What is your time zone? [Europe/Paris] 
> Do you want to generate a tasks.py/Makefile to automate generation and publishing? (Y/n) 
> Do you want to upload your website using FTP? (y/N) 
> Do you want to upload your website using SSH? (y/N) 
> Do you want to upload your website using Dropbox? (y/N) 
> Do you want to upload your website using S3? (y/N) 
> Do you want to upload your website using Rackspace Cloud Files? (y/N) 
> Do you want to upload your website using GitHub Pages? (y/N) 
Done. Your new project is available at /home/leo/devel/theanalogfox

If you run pelican -lr now and visit http://localhost:8000 with your browser you will see the first page of the blog rendered with the default theme.

Demo content¶

Before we venture into the jungle of Jinja templates it's worth creating some content. As this is a very boring activity I prepared a little script that you can run in the terminal.

#!/bin/bash

NUM_POSTS=20
CONTENT_DIR=content
LOREM_API=https://jaspervdj.be/lorem-markdownum/markdown.txt
IMAGES_API=https://placeimg.com/1000/341/animals

rm -fR content
mkdir -p content/images

for i in $(seq -w 1 ${NUM_POSTS})
do
    post_file=${CONTENT_DIR}/post${i}.markdown

    echo "Creating post ${i}"
    echo "Title: A sample article ${i}" >> ${post_file}
    echo "Date: 2021-03-${i}" >> ${post_file}
    echo "Category: News" >> ${post_file}
    echo "Tags: $(seq 1 20 | shuf | head -n3 | sed -r s,"^","tag", | paste -sd "," -)" >> ${post_file}
    echo "Image: post${i}.jpg" >> ${post_file}
    echo "Summary: Summary of post ${i}" >> ${post_file}
    echo >> ${post_file}

    curl -s ${LOREM_API} | sed -r s,"^#","##", >> ${post_file}

    curl -s ${IMAGES_API} > ${CONTENT_DIR}/images/post${i}.jpg
done

Save it as create_content.sh and give it execution permissions with chmod 775 create_content.sh. At this point you can run it with ./create_content.sh and it will create the directory content with 20 posts and an image for each of them. You can safely run it multiple times, it will automatically delete the previous output.

If you know bash feel free to hack the script to do something more complicated, but this very simple program does everything we need to work on Pelican themes.

Running pelican -lr and visiting http://localhost:8000 will now show a richer website.

The template¶

From now on I will make extensive use of the documentation at https://docs.getpelican.com/en/latest/themes.html#creating-themes, so please be sure to have that page open in your browser.

For this tutorial I will use the template "Future Imperfect" by HTML5 UP. The template can be seen in action at this page, and you can download it using the button in the top right corner of the page itself.

Please consider supporting HTML UP even only with a Tweet. Being a content creator myself I know how important it can be to receive any type of feedback from readers/users.

Let's have a quick look at the template before we dive into the core of the post. We have a navbar at the top of the screen, with a link to the homepage, several links to specific pages, a search button, and a menu. In the body of the page there is a sidebar on the left and a preview of the articles on the right.

The sidebar contains the title and the subtitle of the blog, two lists of posts, the about section, and some social buttons. The first list of posts features image, title, date, and the avatar of the author, while the second list has just a small thumbnail, title, and date. Each post in the main list shows the full image, title, subtitle, name and avatar of the author, publication date, a preview of the content of the article, and a button that links the full version of the article. Last, tags are listed at the bottom right, just next to the number of likes and comments.

Just to be clear from the start, I won't implement everything we see here in my Pelican theme. I won't touch the navbar, and I won't discuss likes and comments, which require external systems when it comes to static sites. I will also simplify the sidebar, using only one list of posts. Moreover, I will not preview the articles in the main page, but print the full content.

Unzip the template archive in a subdirectory of the blog directory called future-imperfect. The archive doesn't contains a root folder, so you need to create it explicitly.

Enter the theme directory and change the layout of the files to follow Pelican's requirements:

mv assets/ static
mv images/ static/
mkdir templates
mv *.html templates/

At this point edit the file pelicanconf.py in the main directory of the blog, adding the variable THEME

pelicanconf.py

PATH = 'content'

THEME = "future-imperfect"

TIMEZONE = 'Europe/Paris'

DEFAULT_LANG = 'en'

If you refresh the blog page now you will see that the output doesn't even have a working style sheet, but don't worry, Pelican is still working correctly. We are overriding Pelican's output with the file future-imperfect/templates/index.html, which is supposed to be a Jinja template, but being part of the HTML5 template is just injecting static content. In particular, the CSS/JS assets are not loaded correctly, as you can see.

Let's learn the first piece of syntax adjusting the CSS and JS links, then, so that we can at least have a good output to look at. We need to change the path assets/ with {{ SITEURL }}/theme/

future-imperfect/templates/index.html

<!DOCTYPE HTML>
<!--
    Future Imperfect by HTML5 UP
    html5up.net | @ajlkn
    Free for personal and commercial use under the CCA 3.0 license (html5up.net/license)
  -->
<html>
  <head>
    <title>Future Imperfect by HTML5 UP</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
    <link rel="stylesheet" href="{{ SITEURL }}/theme/css/main.css" />
  </head>

[...]

    <script src="{{ SITEURL }}/theme/js/jquery.min.js"></script>
    <script src="{{ SITEURL }}/theme/js/browser.min.js"></script>
    <script src="{{ SITEURL }}/theme/js/breakpoints.min.js"></script>
    <script src="{{ SITEURL }}/theme/js/util.js"></script>
    <script src="{{ SITEURL }}/theme/js/main.js"></script>

  </body>
</html>

We also need to correctly link images. Change any occurrence of images/ into {{ SITEURL}}/theme/images/, e.g.

    <div class="meta">
      <time class="published" datetime="2015-11-01">November 1, 2015</time>
      <a href="#" class="author"><span class="name">Jane Doe</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="single.html" class="image featured"><img src="{{ SITEURL }}/theme/images/pic01.jpg" alt="" /></a>

If you refresh the page after these changes you will see the template fully rendered (minus the images you saw in the demo, those are replaced by placeholders in the downloaded version).

A little trick: if you remove the comments at lines 177 and 487 you will get a nice recap of the graphical components of the template. I will not use them, so I removed lines 176-487, but remember that those cheat sheets can be very useful when trying to understand how a template works.

As I mentioned earlier, I also removed the third list of posts, as it doesn't add anything to what we will learn. You are clearly free to keep it and experiment with it.

Deep dive

How does {{ SITEURL }}/theme work?

Pelican's documentation on themes says

THEME_STATIC_DIR = 'theme'

Destination directory in the output path where Pelican will place the files collected from THEMESTATICPATHS. Default is theme.

the variable THEME_STATIC_PATHS is by default static, which is why we created that directory inside the theme.

As you can see all these paths are configurable, should you prefer different names.

Pelican variables¶

As I mentioned, we are currently overriding Pelican's output with a static template. What we want to do is to inject values known to Pelican into the template itself, be those static variables or more dynamic items like articles, tags, and images.

To do this, Pelican uses Jinja, a widely adopted template engine written in Python. If you want to fully understand how to create Pelican themes, then, you need to learn Jinja. Don't worry, it's not complicated, and since Jinja uses Python you will catch up very quickly. I won't get into details about the Jinja syntax that I will use, please check out the Jinja documentation if you have any doubts.

We actually already used Pelican's variables and Jinja templates when we prefixed links with {{ SITEURL }}. Aside from that, however, the first and simplest variable injection for our template are title and subtitle.

Title

The Pelican variable we are interested in is SITENAME, which has been initialised by the quickstart script as you can see in the configuration file

pelicanconf.py

SITENAME = "The Analog Fox"

We need to replace the static text with this variable three times: in the tag <title>, in the navigation bar and in the header at the top of the sidebar.

future-imperfect/templates/index.html

<html>
  <head>
    <title>{{ SITENAME }}</title>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
    <link rel="stylesheet" href="{{ SITEURL }}/theme/css/main.css" />
  </head>
  <body class="is-preload">

    <!-- Wrapper -->
    <div id="wrapper">

      <!-- Header -->
      <header id="header">
        <h1><a href="index.html">{{ SITENAME }}</a></h1>
        <nav class="links">
          <ul>
            <li><a href="#">Lorem</a></li>

[...]

      <!-- Sidebar -->
      <section id="sidebar">

        <!-- Intro -->
        <section id="intro">
          <a href="#" class="logo"><img src="{{ SITEURL }}/theme/images/logo.jpg" alt="" /></a>
          <header>
            <h2>{{ SITENAME }}</h2>
            <p>Another fine responsive site template by <a href="http://html5up.net">HTML5 UP</a></p>
          </header>
        </section>

Subtitle

Pelican provides support even for the subtitle, but that wasn't filled in by the setup script for us, so we need to create the variable in the configuration file

pelicanconf.py

SITENAME = "The Analog Fox"
SITESUBTITLE = "A great blog about old stuff"

Once this is done we can insert the variable at the top of the sidebar, just under the title

future-imperfect/templates/index.html

        <section id="intro">
          <a href="#" class="logo"><img src="{{ SITEURL }}/theme/images/logo.jpg" alt="" /></a>
          <header>
            <h2>{{ SITENAME }}</h2>
            <p>{{ SITESUBTITLE }}</p>
          </header>
        </section>

Marvellous! Now the page should show the title of the blog in the window header, announcing to the world the The Analog Fox is ready to take over the world of vintage!

OK, I might be a bit overexcited, but I love when plans come together ;)

Deep dive

Pelican passes the whole configuration file to the template, together with the parsed content of the site itself, so you are free to use any variable, should you need them, or to introduce new ones (which we will do in the next section).

For now, just to familiarise with the concept, you might try to add TIMEZONE under the subtitle

future-imperfect/templates/index.html

          <header>
            <h2>{{ SITENAME }}</h2>
            <p>{{ SITESUBTITLE }}</p>
            <p>{{ TIMEZONE }}</p>
          </header>

I don't think this specific change is really useful, but it's good to remember that all those variables are available.

The template has a section for social buttons under the sidebar, and this is a great use case for a bit of advanced usage of the configuration file.

Pelican has a native support for social links, as you can see from the SOCIAL variable in pelicanconf.py. For the sake of showing you that you are free to define custom variables in that file and use them I will however ignore it and go with something richer. The template uses nice icons to represent the links, so I'd like to include that information.

The section of the template that renders those buttons is

future-imperfect/templates/index.html

        <!-- Footer -->
        <section id="footer">
          <ul class="icons">
            <li><a href="#" class="icon brands fa-twitter"><span class="label">Twitter</span></a></li>
            <li><a href="#" class="icon brands fa-facebook-f"><span class="label">Facebook</span></a></li>
            <li><a href="#" class="icon brands fa-instagram"><span class="label">Instagram</span></a></li>
            <li><a href="#" class="icon solid fa-rss"><span class="label">RSS</span></a></li>
            <li><a href="#" class="icon solid fa-envelope"><span class="label">Email</span></a></li>
          </ul>
          <p class="copyright">&copy; Untitled. Design: <a href="http://html5up.net">HTML5 UP</a>. Images: <a href="http://unsplash.com">Unsplash</a>.</p>
        </section>

Lists are one of the most common patterns when writing templates, as they usually become just a simple for loop. let's start writing down the data, then we will learn how to render the buttons.

As I mentioned, I created a new variable CONTACTS

pelicanconf.py

# Social widget
SOCIAL = (
    ("You can add links in your config file", "#"),
    ("Another social link", "#"),
)

CONTACTS = [
    ("Twitter", "twitter", "https://twitter.com/theanalogfox"),
    ("Facebook", "facebook-f", "https://facebook.com/theanalogfox"),
    ("Instagram", "instagram", "https://www.instagram.com/theanalogfox/"),
    ("Email", "envelope", "info@theanalogfox.com"),
]

DEFAULT_PAGINATION = 3

that is a list of tuples, each one including the title of the button, the name of the icon (Font Awesome), and the link itself. Now we can replace the snippet of code above with this

future-imperfect/templates/index.html

        <!-- Footer -->
        <section id="footer">
          <ul class="icons">
            {% for name, icon, link in CONTACTS %}
            <li><a href="{{ link }}" class="icon brands fa-{{ icon }}"><span class="label">{{ name }}</span></a></li>
            {% endfor %}
          </ul>
          <p class="copyright">&copy; Untitled. Design: <a href="http://html5up.net">HTML5 UP</a>. Images: <a href="http://unsplash.com">Unsplash</a>.</p>
        </section>

As you can see, the core of the snippet is a for loop that uses Python's unpacking to assign name, icon, and link. The variables are then used directly in the HTML as we did before. Remember that Jinja doesn't understand HTML, it just blindly replaces the variables in a text file, which allows us to perform nice tricks like fa-{{ icon }} to use the right icon for each social button.

Articles¶

Now that we introduced loops and variables we have all the tools we need to work on the two lists of articles. Let's first change the list in the main body of the page, the one in the sidebar will then receive the very same treatment.

First of all, I reduced the static list of posts to a single one

future-imperfect/templates/index.html

      <!-- Main -->
      <div id="main">

        <!-- Post -->
        <article class="post">
          <header>
            <div class="title">
              <h2><a href="single.html">Magna sed adipiscing</a></h2>
              <p>Lorem ipsum dolor amet nullam consequat etiam feugiat</p>
            </div>
            <div class="meta">
              <time class="published" datetime="2015-11-01">November 1, 2015</time>
              <a href="#" class="author"><span class="name">Jane Doe</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
            </div>
          </header>
          <a href="single.html" class="image featured"><img src="{{ SITEURL }}/theme/images/pic01.jpg" alt="" /></a>
          <p>Mauris neque quam, fermentum ut nisl vitae, convallis maximus nisl. Sed mattis nunc id lorem euismod placerat. Vivamus porttitor magna enim, ac accumsan tortor cursus at. Phasellus sed ultricies mi non congue ullam corper. Praesent tincidunt sed tellus ut rutrum. Sed vitae justo condimentum, porta lectus vitae, ultricies congue gravida diam non fringilla.</p>
          <footer>
            <ul class="actions">
              <li><a href="single.html" class="button large">Continue Reading</a></li>
            </ul>
            <ul class="stats">
              <li><a href="#">General</a></li>
              <li><a href="#" class="icon solid fa-heart">28</a></li>
              <li><a href="#" class="icon solid fa-comment">128</a></li>
            </ul>
          </footer>
        </article>

        <!-- Pagination -->
        <ul class="actions pagination">
          <li><a href="" class="disabled button large previous">Previous Page</a></li>
          <li><a href="#" class="button large next">Next Page</a></li>
        </ul>

      </div>

Please note that your content will be slightly different as it has been randomly generated.

The list of Pelican variables we can access is available at https://docs.getpelican.com/en/latest/themes.html#index-html (variables for the page index.html) and https://docs.getpelican.com/en/latest/themes.html#article (attributes of Article objects).

future-imperfect/templates/index.html

        {% for article in articles %}
        <!-- Post -->
        <article class="post">
          <header>
            <div class="title">
              <h2><a href="single.html">{{ article.title }}</a></h2>
              <p>{{ article.summary }}</p>
            </div>
            <div class="meta">
              <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
              <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
            </div>
          </header>
          <a href="single.html" class="image featured"><img src="{{ SITEURL }}/theme/images/pic01.jpg" alt="" /></a>
          {{ article.content }}
          <footer>
            <ul class="actions">
              <li><a href="single.html" class="button large">Continue Reading</a></li>
            </ul>
            <ul class="stats">
              {% for tag in article.tags %}
              <li><a href="#">{{ tag.name }}</a></li>
              {% endfor %}
            </ul>
          </footer>
        </article>
        {% endfor %}

As you can see we can just loop over the page variable articles and read the attributes of object just like we usually do in Python, for example with {{ article.title }} or {{ article.summary }}.

Jinja filters are very powerful, and as you see I used them twice with dates to give them different formats. One for the internal representation of time {{ article.date | strftime('%Y-%m-%d') }} and one is the more visually pleasant ({{ article.date | strftime('%b %-d, %Y') }}). I don't normally use the American date format, but as the template did I kept it as a good example of what you can do with Jinja filters.

Last, you can nest for loops in other for loops, as I did with tags. You can see the internal structure of tags in the documentation. Please note that the string representation of tags (and other objects in Pelican) is the attribute name, so we might write

              {% for tag in article.tags %}
              <li><a href="#">{{ tag }}</a></li>
              {% endfor %}

and it would return the same output.

A note about the content of the article. As you can see {{ article.content }} prints the full article, so if you want to print a preview you need to truncate it. Unfortunately you can't just write something like {{ article.content[:600] }} to print 600 characters. Remember that Jinja works on the output of the readers already, which means that article. content is already HTML, and if you arbitrarily truncate it you leave some tags open, which disrupts the rendering of the page.

At the end of the post I will show you how you can easily solve this in Pelican, once we have a dedicated page for each article.

Pagination¶

What we did in the previous section prints all the articles in the blog on the same page. This is clearly not acceptable as the number of articles increases, and the standard solution for this is pagination.

Pelican fully supports it, and if you remember we set it up when we run pelican-quickstart. You can see the current number of articles per page in pelicanconf.py

DEFAULT_PAGINATION = 3

To leverage pagination we first need to replace the variable articles with articles_page.object_list

future-imperfect/templates/index.html

        {% for article in articles_page.object_list %}
        <!-- Post -->
        <article class="post">
          <header>
            <div class="title">
              <h2><a href="single.html">{{ article.title }}</a></h2>
              <p>{{ article.summary }}</p>

If you render the page in the browser now you'll see that it shows only the last 3 posts, which corresponds to the value of DEFAULT_PAGINATION. If you want you can try to change it and see it affecting the page.

To use pagination we also need to configure navigation buttons. The template already includes them after all the articles in the main body.

        <!-- Pagination -->
        <ul class="actions pagination">
          <li><a href="" class="disabled button large previous">Previous Page</a></li>
          <li><a href="#" class="button large next">Next Page</a></li>
        </ul>

We have three different cases when it comes to pagination. The first page should grey out or remove the "Previous page" button, the last page should do the same with the "Next page" button, and any page between the two should show both.

We can achieve it with the following code

future-imperfect/templates/index.html

        <!-- Pagination -->
        <ul class="actions pagination">
          {% if articles_page.has_previous() %}
          <li><a href="/{{ articles_previous_page.url }}" class="button large previous">Previous Page</a></li>
          {% else %}
	  <li><a href="" class="disabled button large previous">Previous Page</a></li>
	  {% endif %}
	  
	  {% if articles_page.has_next() %}
	  <li><a href="/{{ articles_next_page.url }}" class="button large next">Next Page</a></li>
	  {% else %}
	  <li><a href="#" class="disabled button large next">Next Page</a></li>
	  {% endif %}
	</ul>

The two functions articles_page.has_previous() and articles_page.has_next() can be used to know if there are pages before of after the current one, and if not we can use the class disabled to grey out the button. The link to the previous or next page is provided by articles_previous_page.url and articles_next_page.url respectively. Again, remember that you can learn everything about all these variables reading Pelican's documentation on themes.

Render the page in the browser and you will see that the "Next page" button leads to http://localhost:8000/index2.html, which is the second page of articles. Indeed, if you click it, you will see articles from 17 to 15, and so on until the last page of articles http://localhost:8001/index7.html that contains the first and the secodn articles of the blog (if you generated 20 articles at the beginning as I did).

Slicing¶

The sidebar is usually the best place to show a fixed set of posts like latest ones. We can easily achieve this slicing the list of articles. To do this let's first reduce the number of mini posts to one, so that we can introduce a loop

future-imperfect/templates/index.html

        <!-- Mini Posts -->
        <section>
          <div class="mini-posts">

            <!-- Mini Post -->
            <article class="mini-post">
              <header>
                <h3><a href="single.html">Vitae sed condimentum</a></h3>
                <time class="published" datetime="2015-10-20">October 20, 2015</time>
                <a href="#" class="author"><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
              </header>
              <a href="single.html" class="image"><img src="{{ SITEURL }}/theme/images/pic04.jpg" alt="" /></a>
            </article>

          </div>
        </section>

We can show the last 4 articles with this simple sintax

future-imperfect/templates/index.html

        <!-- Mini Posts -->
        <section>
          <div class="mini-posts">

	    {% for article in articles[:4] %}
            <!-- Mini Post -->
            <article class="mini-post">
              <header>
                <h3><a href="single.html">Vitae sed condimentum</a></h3>
                <time class="published" datetime="2015-10-20">October 20, 2015</time>
                <a href="#" class="author"><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
              </header>
              <a href="single.html" class="image"><img src="{{ SITEURL }}/theme/images/pic04.jpg" alt="" /></a>
            </article>
	    {% endfor %}

          </div>
        </section>

which will print the same static article 4 times because we are not using Pelican's variables yet. Applying the same changes we introduced for the main list of articles we finally get

future-imperfect/templates/index.html

        <!-- Mini Posts -->
        <section>
          <div class="mini-posts">

	    {% for article in articles[:4] %}
            <!-- Mini Post -->
            <article class="mini-post">
              <header>
                <h3><a href="single.html">{{ article.title }}</a></h3>
                <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
                <a href="#" class="author"><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
              </header>
              <a href="single.html" class="image"><img src="{{ SITEURL }}/theme/images/pic04.jpg" alt="" /></a>
            </article>
	    {% endfor %}
	    
          </div>
        </section>

And now you will see the proper titles and dates. Please note that this loop depends on articles, which doesn't change from page to page, so this list will be immutable across the blog.

Pictures¶

The script that created demo articles downloaded an image for each article, that can be used as featured image. There are many way to manage images in a website and I won't dive into that particular subject here. I will only show you a basic way to serve images that are not static, but part of the content.

The images are stored in content/images and this is also an arbitrary decision. The only thing you need to keep in mind is that Pelican's root is usually set in the directory content, so if you place files elsewhere it might be complicated to reach them.

Through the automated script that we used to generate content we wrote a specific metadata in each article, with the name of the relative image. For example

content/post01.markdown

Title: A sample article 01
Date: 2021-03-01
Category: News
Tags: tag6,tag11,tag8
Image: post01.jpg
Summary: Summary of post 01

[...]

This shows you that you can add whatever metadata you want to your posts and have them loaded into the object article. To show the image we just need to load the correct file instead of the placeholder, both in the main list

future-imperfect/templates/index.html

        {% for article in articles_page.object_list %}
        <!-- Post -->
        <article class="post">
          <header>
            <div class="title">
              <h2><a href="single.html">{{ article.title }}</a></h2>
              <p>{{ article.summary }}</p>
            </div>
            <div class="meta">
              <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
              <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
            </div>
          </header>
          <a href="single.html" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
          {{ article.content }}
          <footer>
            <ul class="actions">
              <li><a href="single.html" class="button large">Continue Reading</a></li>
            </ul>
            <ul class="stats">
              {% for tag in article.tags %}
              <li><a href="#">{{ tag.name }}</a></li>
              {% endfor %}
            </ul>
          </footer>
        </article>
        {% endfor %}

and in the sidebar

future-imperfect/templates/index.html

        <!-- Mini Posts -->
        <section>
          <div class="mini-posts">

	    {% for article in articles[:4] %}
            <!-- Mini Post -->
            <article class="mini-post">
              <header>
                <h3><a href="single.html">{{ article.title }}</a></h3>
                <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
                <a href="#" class="author"><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
              </header>
              <a href="single.html" class="image"><img src="images/{{ article.image }}" alt="" /></a>
            </article>
	    {% endfor %}
	    
          </div>
        </section>

Now the page is definitely more appealing!

Advanced techniques: extend and include¶

As in everything related to computer programming (and not only) you will soon discover that you are repeating yourself, and one of the top design advice that you should follow says: don't do it.

Jinja templates provide two tags to help you reduce duplicated code, namely extend and include. To see how they work let's focus our attention on the page of a specific article. When we click on an article we would like to go to a dedicated page, while all links at the moment load the same static page single.html.

The article page, however, won't be that different from the front page we just created. It might be, as you are free to completely change the style, but in general some elements will be in common, for example the navigation, the sidebar, and the footer.

In the following section I will show you how to use the Jinja tags extends and include to reuse parts of your theme.

Extends

Since we want to reuse some parts of the page we need to move them to a "common space". Create a new file called future-imperfect/templates/base.html and move the whole content of index.html into it. Then write this single statement in the now empty index.html

future-imperfect/templates/index.html

{% extends "base.html" %}

What we did is to tell Jinja that everything we do in index.html should happen on top of base.html (we'll soon discover the meaning of this "on top"). For now you might think about it as index.html copying the content of base.html.

The problem with this setup is that base.html doesn't have access to the variables that index.html has access to, as base.html is an arbitrary file and not something known to Pelican. Indeed the command pelican -lr that you are running in a terminal should give you this error

WARNING: Caught exception:
  | "'articles_page' is undefined".

Since that is something only index.html can provide we need to do something more than just copying the content of base.html, we need to also fill in some parts, which is exactly what you can do with block (see Jinja's documentation on template inheritance).

Grab the whole content of the <div id="main"> and move it in index.html wrapped in {% block content %}

future-imperfect/templates/index.html

{% extends "base.html" %}

{% block content %}
{% for article in articles_page.object_list %}
<!-- Post -->
<article class="post">
  <header>
    <div class="title">
      <h2><a href="single.html">{{ article.title }}</a></h2>
      <p>{{ article.summary }}</p>
    </div>
    <div class="meta">
      <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
      <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="single.html" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
  {{ article.content }}
  <footer>
    <ul class="actions">
      <li><a href="single.html" class="button large">Continue Reading</a></li>
    </ul>
    <ul class="stats">
      {% for tag in article.tags %}
      <li><a href="#">{{ tag.name }}</a></li>
      {% endfor %}
    </ul>
  </footer>
</article>
{% endfor %}

<!-- Pagination -->
<ul class="actions pagination">
  {% if articles_page.has_previous() %}
  <li><a href="/{{ articles_previous_page.url }}" class="button large previous">Previous Page</a></li>
  {% else %}
  <li><a href="" class="disabled button large previous">Previous Page</a></li>
  {% endif %}

  {% if articles_page.has_next() %}
  <li><a href="/{{ articles_next_page.url }}" class="button large next">Next Page</a></li>
  {% else %}
  <li><a href="#" class="disabled button large next">Next Page</a></li>
  {% endif %}
</ul>
{% endblock %}

At the same time fill the empty space you left in base.html with a call for that block

future-imperfect/templates/base.html

      <!-- Main -->
      <div id="main">
	{% block content %}{% endblock %}
      </div>

When this is done, reload the page and... nothing should have changed. Sorry, but this is one of those things that happen behind the scenes and that do not have any immediate benefit. However, Pelican shouldn't give you any error on the command line, which is comforting.

To see the benefit of what we did let's move on and create the page for the single article, article.html

future-imperfect/templates/article.html

{% extends "base.html" %}

As we said before base.html is supposed to be the common part of all pages, so article.html should start from that as well. To check how that page looks like now let's update the links from the articles in the main body. Open index.html and replace the URLs of each article

future-imperfect/templates/index.html

<article class="post">
  <header>
    <div class="title">
      <h2><a href="{{ article.url }}">{{ article.title }}</a></h2>
      <p>{{ article.summary }}</p>
    </div>
    <div class="meta">
      <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
      <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="{{ article.url }}" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
  {{ article.content }}
  <footer>
    <ul class="actions">
      <li><a href="{{ article.url }}" class="button large">Continue Reading</a></li>
    </ul>
    <ul class="stats">
      {% for tag in article.tags %}
      <li><a href="#">{{ tag.name }}</a></li>
      {% endfor %}
    </ul>
  </footer>
</article>

Now if you render the front page and click on the title of a post in the main column you will end up in the page dedicated to it, which has a worrying empty central column! Well, after all article.html extends base.html but doesn't provide anything for the block content, so let's fix this

future-imperfect/templates/article.html

{% extends "base.html" %}

{% block content %}
<!-- Post -->
<article class="post">
  <header>
    <div class="title">
      <h2><a href="{{ article.url }}">{{ article.title }}</a></h2>
      <p>{{ article.summary }}</p>
    </div>
    <div class="meta">
      <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
      <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="{{ article.url }}" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
  {{ article.content }}
  <footer>
    <ul class="actions">
      <li><a href="{{ article.url }}" class="button large">Continue Reading</a></li>
    </ul>
    <ul class="stats">
      {% for tag in article.tags %}
      <li><a href="#">{{ tag.name }}</a></li>
      {% endfor %}
    </ul>
  </footer>
</article>
{% endblock %}

I copied the content of the block from the loop in index.html. In this simple example they have the same code, but in a real production environment you might want to differentiate them. As you can see article.html provides the variable article out of the box. Please note that the page of each article doesn't show the pagination buttons, as they are not part of article.html.

Include

When you write a theme there are often snippets of code that you might need to repeat in different parts of the whole site. The sidebar is typically a good example of a collection of such snippets, like the "About" section or the list of latest posts that you might want to reuse somewhere else.

Jinja provides the tag include that allows you to easily inject the content of another template file into the current one. Let's see how it works moving the list of latest posts to a separate file.

Move the content of the mini posts section from base.html to a file called templates/includes/latest_posts.html

templates/includes/latest_posts.html

<!-- Mini Posts -->
<section>
  <div class="mini-posts">

    {% for article in articles[:4] %}
    <!-- Mini Post -->
    <article class="mini-post">
      <header>
        <h3><a href="single.html">{{ article.title }}</a></h3>
        <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
        <a href="#" class="author"><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
      </header>
      <a href="single.html" class="image"><img src="images/{{ article.image }}" alt="" /></a>
    </article>
    {% endfor %}

  </div>
</section>

And replace it with a call to include in base.html

templates/includes/base.html

        <!-- Intro -->
        <section id="intro">
          <a href="#" class="logo"><img src="{{ SITEURL }}/theme/images/logo.jpg" alt="" /></a>
          <header>
            <h2>{{ SITENAME }}</h2>
            <p>{{ SITESUBTITLE }}</p>
          </header>
        </section>

        {% include 'includes/latest_posts.html' %}

        <!-- About -->
        <section class="blurb">
          <h2>About</h2>
          <p>Mauris neque quam, fermentum ut nisl vitae, convallis maximus nisl. Sed mattis nunc id lorem euismod amet placerat. Vivamus porttitor magna enim, ac accumsan tortor cursus at phasellus sed ultricies.</p>
          <ul class="actions">
            <li><a href="#" class="button">Learn More</a></li>
          </ul>
        </section>

Again, reloading the page won't show anything different, but now we isolated a piece of code that shows the latest posts and we can reuse it in another part of the site which is not the sidebar. You can find the documentation of include at https://jinja.palletsprojects.com/en/2.11.x/templates/#include.

Deep dive

What is the difference between extends and include? Which one shall you use?

The difference between extending and including is the same difference that lies between a framework and a library in the context of programming languages. A framework provides the bulk of the system, and we provide the code for specific operations, while a library provides specific code that we need to put in a bigger picture. This about a Web framework like Django or Flask: they already "work" in the background, but until you provide the endpoints they don't do anything specific. At the same time, when you need to encrypt a password you import a library and use its functions.

In the same way extends means that another template is providing the main part of the page and that we provide the content of the blocks. When we use include the opposite happens, we get a specific snippet of code and insert it in some wider context.

As there are no limitations to the number of lines contained in a block or in an imported snippet, there is a certain amount of overlap between the two. Sometimes it might not be immediately clear which one to use, but in the vast majority of cases it should be straightforward.

Preview articles¶

Earlier in the post I mentioned a way to create the preview of articles, so now that we have two different pages it makes sense to implement that.

As I said before the preview of an article has to be created before the content is converted into HTML, or we might leave open tags. Pelican supports this mechanism out of the box, through the summary metadata. If you specify a summary (as I did through the random generation script), Pelican won't do anything, but id the summary is not present it will be initialised with a preview of the article.

The two variables involved in this process are SUMMARY_MAX_LENGTH and SUMMARY_END_SUFFIX (see the documentation), but we can accept the default values for our little project.

To see the summary in action let's open the last post (post20.markdown) and remove the line Summary: Summary of post 20. Then we need to adjust the code of the page to properly display the summary as content.

templates/includes/index.html

{% for article in articles_page.object_list %}
<!-- Post -->
<article class="post">
  <header>
    <div class="title">
      <h2><a href="{{ article.url }}">{{ article.title }}</a></h2>
                                                                  
    </div>
    <div class="meta">
      <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
      <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL}}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="{{ article.url }}" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
  {{ article.summary }}
  <footer>
    <ul class="actions">
      <li><a href="{{ article.url }}" class="button large">Continue Reading</a></li>
    </ul>
    <ul class="stats">
      {% for tag in article.tags %}
      <li><a href="#">{{ tag.name }}</a></li>
      {% endfor %}
    </ul>
  </footer>
</article>
{% endfor %}

In the article page, instead we want the content to be displayed, so we just need to remove the summary (which at this point is not a single line that can be easily displayed under the title.

templates/includes/article.html

{% block content %}
<!-- Post -->
<article class="post">
  <header>
    <div class="title">
      <h2><a href="{{ article.url }}">{{ article.title }}</a></h2>
                                                                  
    </div>
    <div class="meta">
      <time class="published" datetime="{{ article.date | strftime('%Y-%m-%d') }}">{{ article.date | strftime('%b %-d, %Y') }}</time>
      <a href="#" class="author"><span class="name">{{ article.author }}</span><img src="{{ SITEURL }}/theme/images/avatar.jpg" alt="" /></a>
    </div>
  </header>
  <a href="{{ article.url }}" class="image featured"><img src="images/{{ article.image }}" alt="" /></a>
  {{ article.content }}
  <footer>
    <ul class="actions">
      <li><a href="{{ article.url }}" class="button large">Continue Reading</a></li>
    </ul>
    <ul class="stats">
      {% for tag in article.tags %}
      <li><a href="#">{{ tag.name }}</a></li>
      {% endfor %}
    </ul>
  </footer>
</article>
{% endblock %}

Next steps¶

As I promised, I showed you how to use Jinja tags and Pelican variables, and we went from a purely static template to a rich dynamic one. There are several things that you might want to investigate and implement now:

Both index.html and article.html contain a great amount of common code (the part that displays the article). Try to work on that to isolate the shared code and remove the duplication. Remember that they are not exactly the same, so you might want to investigate the tag with provided by Jinja.
Tags and categories. Pelican creates pages with all the articles that belong to a specific category or tag, and they work exactly like the index page. The documentation of the variables available in those pages can be found at https://docs.getpelican.com/en/latest/themes.html#category-html and https://docs.getpelican.com/en/latest/themes.html#tag-html. You should try to create a page that lists all the articles with a certain tag and connect it to the tag button at the end of each article.
Pelican can create pages of content (as opposed to articles), and you might want to show those in the navigation bar. You can find the documentation of the Page objects at https://docs.getpelican.com/en/latest/themes.html#page. Create an "About me" page and link it there.
If you want to publish the website you probably want to minify assets to save bandwidth and reduce the loading time. Have a look at webassets (https://github.com/pelican-plugins/webassets), a plugin that makes assets management as easy as pie.
Speaking of plugins, Pelican has a huge number of them. The developers are in the process of transitioning them to a new better format, so you will find the main plugins at https://github.com/pelican-plugins and the rest at https://github.com/getpelican/pelican-plugins. I recommend having a look at sitemap, which is very easy to setup and provides an important tool for search engines to discover your site.
When it comes to deploying the static website you should definitely read Pelican's documentation, as the management scripts support several backends.

Last, but hopefully not least, the code of this blog is public and can be found at https://github.com/TheDigitalCatOnline/blog_source. While the overall setup contains a bit of legacy code (I started a while ago and many things have changed in the meanwhile), the theme editorial is pretty new, so feel free to have a look at it and to get inspired (https://github.com/TheDigitalCatOnline/blog_source/tree/master/editorial).

I hope this was useful. Remember that Pelican is open source, and you can always get in touch with the maintainers to ask questions or to submit enhancements. Also remember to drop a line on Twitter to the author of any template you will use (thanks are always welcome) and to link to the source on the website. Happy blogging!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

AWS Log Insights as CloudWatch metrics with Python and Terraform

2021-03-22T17:00:00+01:00

Recently I started using AWS CloudWatch Log Insights and I find the tool really useful to extract data about the systems I'm running without having to set up dedicated monitoring tools, which come with their own set of permissions, rules, configuration language, and so forth.

Log Insights allow you to query log outputs with a language based on regular expressions with hints of SQL and to produce tables or graphs of quantities that you need to monitor. For example, the system I am monitoring runs Celery in ECS containers that log received tasks with a line like the following

16:39:11,156 [32mINFO [0m [34m[celery.worker.strategy][0m [01mReceived task: lib.tasks.lists.trigger_list_log_notification[9b33b464-d4f9-4909-8d4e-1a3134fead97] [0m

In this case the specific function in the system that was triggered is lib.tasks.log_notification, and I'm interested in knowing which functions are called the most, so I can easily count them with

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) as number by task
 | sort number desc
 | limit 9

This gives me a nice table of the top 9 source functions and the number of task submitted for each, and the time frame can be adjusted with the usual CloudWatch controls

1 lib.tasks.lists.trigger_list_log_notification 4559
2 lib.tasks.notify.notify_recipient 397
3 lib.message._send_mobile_push_notification 353
4 lib.tasks.jobs.check_job_cutoffs 178
5 lib.tasks.notify.check_message_cutoffs 177
6 lib.tasks.notify.check_notification_retry 177
7 lib.tasks.notify.async_list_response 81
8 lib.tasks.hmrc_poll.govtalk_periodic_poll 59
9 lib.tasks.lists.recalculate_list_entry 56

Using time bins, quantities can also be easily plotted. For example, I can process and visualise the number of received tasks with

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) by bin(30s)

Unfortunately I quickly discovered an important limitation of Log Insights, that is queries are not metrics. Which also immediately implies that I can't set up alarms on those queries. As fun as it is to look at nice plots, I need something automatic that sends me messages or scales up systems in reaction to specific events such as "too many submitted tasks".

The standard solution to this problem suggested by AWS is to write a Lambda that runs the query and stores the value into a custom CloudWatch metric, which I can then use to satisfy my automation needs. I did it, and in this post I will show you exactly how, using Terraform, Python and Zappa, CloudWatch, and DynamoDB. At the end of the post I will also briefly discuss the cost of the solution.

The big picture¶

Before I get into the details of the specific tools or solutions that I decided to implement, let me have a look at the bigger picture. The initial idea is very simple: a Lambda function can run a specific Log Insights query and store the results in a custom metric, which can in turn be used to trigger alarms and other actions.

For a single system I already have 4 or 5 of these queries that I'd like to run, and I have multiple systems, so I'd prefer to have a solution that doesn't require me to deploy and maintain a different Lambda for each query. The maintenance can be clearly automated as well, but such a solution smells of duplicated code miles away, and if there is no specific reason to go down that road I prefer to avoid it.

Since Log Insights queries are just strings of code, however, we can store them somewhere and then simply loop on all of them within the same Lambda function. To implement this, I created a DynamoDB table and every element contains all the data I need to run each query, such as the log group that I want to investigate and the name of the target metric.

Terraform¶

In the following sections I will discuss the main components of the solution from the infrastructural point of view, showing how I created them with Terraform. The four main AWS services that I will use are: DynamoDB, Lambda, IAM, CloudWatch.

I put the bulk of the code in a module so that I can easily create the same structure for multiple AWS accounts. While my current setup is a bit more complicated that that, the structure of the code can be simplified as

+ common
  + lambda-loginsights2metrics
    + cloudwatch.tf
    + dynamodb.tf
    + iam.tf
    + lambda.tf
    + variables.tf
+ account1
  + lambda-loginsights2metrics
    + main.tf
    + variables.tf

Variables

Since I will refer to them in the following sections, let me show you the four variables I defined for this module.

First I need to receive the items that I need to store in the DynamoDB table

common/lambda-loginsights2metrics/variables.tf

variable "items" {
  type    = "list"
  default = []
}

I prefer to have a prefix in front of my components that allows me to duplicate them without clashes

common/lambda-loginsights2metrics/variables.tf

variable "prefix" {
  type    = "string"
  default = "loginsights2metrics"
}

The Lambda function will require a list of security groups that grant access to specific network components

common/lambda-loginsights2metrics/variables.tf

variable "security_groups" {
  type    = "list"
  default = []
}

Finally, Lambda functions need to be told which VPC subnets they can use to run

common/lambda-loginsights2metrics/variables.tf

variable "vpc_subnets" {
  type    = "list"
  default = []
}

Resources

Terraform variables.
An in-depth post that explains how to use variables in Terraform, by Sumeet Ninawe

DynamoDB

Let's start with the corner stone, which is the DynamoDB table that contains data for the queries. As DynamoDB is not a SQL database we don't need to define columns in advance. This clearly might get us into trouble later, so we need to be careful and be consistent when we write items, adding everything is needed by the Lambda code.

common/lambda-loginsights2metrics/dynamodb.tf

resource "aws_dynamodb_table" "loginsights2metrics" {
  name = "${var.prefix}-items"

  billing_mode = "PAY_PER_REQUEST"

  hash_key = "SlotName"

  attribute {
    name = "SlotName"
    type = "S"
  }
}

Speaking of items, I assume I will pass them when I call the module, so here I just need to loop on the input variable items

common/lambda-loginsights2metrics/dynamodb.tf

resource "aws_dynamodb_table_item" "item" {
  count = length(var.items)

  table_name = aws_dynamodb_table.loginsights2metrics.name
  hash_key   = aws_dynamodb_table.loginsights2metrics.hash_key

  item = jsonencode(element(var.items, count.index))
}

Since the query is written as a Terraform string and will be read from Python there are two small caveats here. To be consistent with Terraform's syntax we need to escape double quotes in the query, and to avoid fights with Python we need to escape backslashes. So for example a valid query like

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) as Value by bin(1m)

will be stored as

"parse @message /\\[celery\\.(?<source>[a-z.]+)\\].*Received task: (?<task>[a-z._]+)\\[/ | filter not isblank(source) | stats count(*) as Value by bin(1m)"

Another remark is that the Lambda I will write in Python will read data plotted with the name Value on bins of 1 minute, so the query should end with stats X as Value by bin(1m) where X is a specific stat, for example stats count(*) as Value by bin(1m).

The reason behind 1 minute is that the maximum standard resolution of CloudWatch metrics is 1 minute. Should you want more you need to have a look at CloudWatch High-Resolution Metrics.

Resources

IAM part 1

IAM roles are central in AWS. In this specific case we have the so-called Lambda execution role, which is the IAM role that the Lambda assumes when you run it. In AWS users or services (that is humans or AWS components) assume a role, receiving the permissions connected with it. To assume roles, however, they need to have a specific permission, a so-called trust policy.

Let's define a trust policy that allows the Lambda service to assume the role that we will define

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "trust" {
  statement {

    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"
      identifiers = [
        "lambda.amazonaws.com"
      ]
    }
  }
}

and after that the role in question

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role" "loginsights2metrics" {
  name               = var.prefix
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

To run, Lambdas need an initial set of permissions which can be found in the canned policy AWSLambdaVPCAccessExecutionRole. You can see the content of the policy in the IAM console or dumping it with aws iam get-policy and aws iam get-policy-version

$ aws iam get-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
{
    "Policy": {
        "PolicyName": "AWSLambdaVPCAccessExecutionRole",
        "PolicyId": "ANPAJVTME3YLVNL72YR2K",
        "Arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole",
        "Path": "/service-role/",
        "DefaultVersionId": "v2",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "Description": "Provides minimum permissions for a Lambda function to execute while accessing a resource within a VPC - create, describe, delete network interfaces and write permissions to CloudWatch Logs. ",
        "CreateDate": "2016-02-11T23:15:26Z",
        "UpdateDate": "2020-10-15T22:53:03Z"
    }
}
$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole --version-id v2
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents",
                        "ec2:CreateNetworkInterface",
                        "ec2:DescribeNetworkInterfaces",
                        "ec2:DeleteNetworkInterface",
                        "ec2:AssignPrivateIpAddresses",
                        "ec2:UnassignPrivateIpAddresses"
                    ],
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v2",
        "IsDefaultVersion": true,
        "CreateDate": "2020-10-15T22:53:03Z"
    }
}

Attaching a canned policy is just a matter of creating a specific aws_iam_role_policy_attachment resource

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy_attachment" "loginsights2metrics-" {
  role       = aws_iam_role.loginsights2metrics.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

Now that we have the IAM role and the basic policy we can assign custom permissions to it. We need to grant the Lambda permissions on other AWS components, namely CloudWatch to run Log Insights queries and to store metrics and DynamoDB to retrieve all the items from the queries table.

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "loginsights2metrics" {
  statement {
    actions = [
      "cloudwatch:PutMetricData",
      "cloudwatch:PutMetricAlarm",
      "logs:StartQuery",
      "logs:GetQueryResults",
      "logs:GetLogEvents",
    ]

    resources = ["*"]
  }

  statement {
    actions = [
      "dynamodb:Scan"
    ]

    resources = [aws_dynamodb_table.loginsights2metrics.arn]
  }
}

Through aws_iam_role_policy we can create and assign the policy out of a data structure

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy" "loginsights2metrics" {
  name   = var.prefix
  role   = aws_iam_role.loginsights2metrics.name
  policy = data.aws_iam_policy_document.loginsights2metrics.json
}

Resources

Lambda

We can now create the Lambda function container. I do not use Terraform as a deployer, as I think it should be used to define static infrastructure only, so I will use a dummy function here and later deploy the real code using the AWS CLI.

The dummy function can be easily created with

common/lambda-loginsights2metrics/lambda.tf

data "archive_file" "dummy" {
  type        = "zip"
  output_path = "${path.module}/lambda.zip"

  source {
    content  = "dummy"
    filename = "dummy.txt"
  }
}

The Lambda function is a bit more complicated. As I mentioned, I'll use Zappa to package the function, so the handler has to be "zappa.handler.lambda_handler". The IAM role given to the function is the one we defined previously, while memory_size and timeout clearly depend on the specific function. Lambdas should run in private networks, and I won't cover here the steps to create them. The AWS docs contains a lot of details on this topic, e.g. https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/.

The environment variables allow me to inject the name of the DynamoDB table so that I don't need to hardcode it. I also pass another variable, the Sentry DSN that I use in my configuration. This is not essential for the problem at hand, but I left it there to show how to pass such values.

common/lambda-loginsights2metrics/lambda.tf

resource "aws_lambda_function" "loginsights2metrics" {
  function_name = "loginsights2metrics"
  handler       = "zappa.handler.lambda_handler"
  runtime       = "python3.8"
  filename      = data.archive_file.dummy.output_path

  role        = aws_iam_role.loginsights2metrics.arn
  memory_size = 128
  timeout     = 300

  vpc_config {
    subnet_ids         = var.vpc_subnets
    security_group_ids = var.security_groups
  }

  environment {
    variables = {
      "SENTRY_DSN"     = "https://XXXXXX:@sentry.io/YYYYYY",
      "DYNAMODB_TABLE" = aws_dynamodb_table.loginsights2metrics.name
    }
  }

  lifecycle {
    ignore_changes = [last_modified, filename]
  }
}

Please note that I instructed Terraform to ignore changes to the two attributes last_modified and filename, and that I haven't used any source_code_hash. This way I can safely apply Terraform to change parameters like memory_size or timeout without affecting what I deployed with the CI.

Since I want to trigger the function from AWS CloudWatch Events I need to grant the service events.amazonaws.com the lambda:InvokeFunction permission.

common/lambda-loginsights2metrics/lambda.tf

resource "aws_lambda_permission" "loginsights2metrics" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.loginsights2metrics.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.rate.arn
}

Resources

IAM part 2

Since 2018 Lambdas have a maximum execution time of 15 minutes (900 seconds), which is more than enough for many services, but to be conservative I preferred to leverage Zappa's asynchronous calls and to make the main Lambda call itself for each query. The Lambda doesn't clearly call the same Python function (it's not recursive), but from AWS's point of view we have a Lambda that calls itself, so we need to give it a specific permission to do this.

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "loginsights2metrics_exec" {
  statement {
    actions = [
      "lambda:InvokeAsync",
      "lambda:InvokeFunction"
    ]

    resources = [aws_lambda_function.loginsights2metrics.arn]
  }
}

I could not define this when I defined the rest of the IAM components because this needs the Lambda to be defined, but the resource is in the same file. Terraform doesn't care about which resource we defined first and where we define it as long as there are no loops in the definitions.

We can now assign the newly created policy document to the IAM role we created previously

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy" "loginsights2metrics_exec" {
  name   = "${var.prefix}-exec"
  role   = aws_iam_role.loginsights2metrics.name
  policy = data.aws_iam_policy_document.loginsights2metrics_exec.json
}

Resources

aws_iam_policy_document documentation documentation")
aws_iam_role_policy documentation

CloudWatch

Whenever you need to run Lambdas (or other things) periodically, the standard AWS solution is to use CloudWatch Events, which work as the AWS cron system. CloudWatch Events are made of rules and targets, so first of all I defined a rule that gets triggered every 2 minutes

common/lambda-loginsights2metrics/cloudwatch.tf

resource "aws_cloudwatch_event_rule" "rate" {
  # Zappa requires the name to match the processing function
  name                = "main.loginsights2metrics"
  description         = "Trigger Lambda ${var.prefix}"
  schedule_expression = "rate(2 minutes)"
}

Please note that Zappa has a specific requirement for CloudWatch Events, so I left a comment to clarify this to my future self. The second part of the event is the target, which is the Lambda function that we defined in the previous section.

common/lambda-loginsights2metrics/cloudwatch.tf

resource "aws_cloudwatch_event_target" "lambda" {
  rule      = aws_cloudwatch_event_rule.rate.name
  target_id = "${var.prefix}-target"
  arn       = aws_lambda_function.loginsights2metrics.arn
}

Resources

Using the module

Now the module is finished, so I just need to create some items for the DynamoDB table and to call the module itself

account1/lambda-loginsights2metrics/main.tf

locals {
  items = [
    {
      "SlotName" : {
        "S" : "Celery Logs submitted tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery\\.(?<source>[a-z.]+)\\].*Received task: (?<task>[a-z._]+)\\[/ | filter not isblank(source) | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Submitted tasks"
      }
    },
    {
      "SlotName" : {
        "S" : "Celery Logs succeeded tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery.(?<source>[a-z\\._]+)].*Task (?<task>[a-z\\._]+)\\[.*\\] (?<event>[a-z]+)/ | filter source = \"app.trace\" | filter event = \"succeeded\" | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Succeeded tasks"
      }
    },
    {
      "SlotName" : {
        "S" : "Celery Logs retried tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery.(?<source>[a-z\\._]+)].*Task (?<task>[a-z\\._]+)\\[.*\\] (?<event>[a-z]+)/ | filter source = \"app.trace\" | filter event = \"retry\" | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Retried tasks"
      }
    }
  ]
}

I need to provide a security group for the Lambda, and in this case I can safely use the default one provided by the VPC

account1/lambda-loginsights2metrics/main.tf

data "aws_security_group" "default" {
  name   = "default"
  vpc_id = var.vpc_id
}

And I can finally call the module

account1/lambda-loginsights2metrics/main.tf

module "loginsights2metrics" {
  source = "../../common/lambda-loginsights2metrics"

  items           = local.items
  security_groups = [data.aws_security_group.default.id]
  vpc_subnets     = var.vpc_private_subnets
}

Please note that the variable vpc_private_subnets is a list of subnet names that I created in another module.

Resources

Python¶

As I mentioned before, the Python code of the Lambda function is contained in a different repository and deployed with the CI using Zappa. Given we are interacting with AWS I am clearly using Boto3, the AWS SDK for Python. The code was developed locally without Zappa's support, to test out the Boto3 functions I wanted to use, then quickly adjusted to be executed in a Lambda.

I think the code is pretty straightforward, but I left my original comments to be sure everything is clear.

import os
import time
import json
from datetime import datetime, timedelta

import boto3
from zappa.asynchronous import task

# CONFIG
logs = boto3.client("logs", region_name="eu-west-1")
cw = boto3.client("cloudwatch", region_name="eu-west-1")
dynamodb = boto3.resource("dynamodb", region_name="eu-west-1")


@task
def put_metric_data(item): 3
    slot_name = item["SlotName"]
    log_group = item["LogGroup"]
    cluster_name = item["ClusterName"]
    query = item["Query"]
    namespace = item["Namespace"]
    metric_name = item["MetricName"]

    # This runs the Log Insights query fetching data
    # for the last 15 minutes.
    # As we deal with logs processing it's entirely possible
    # for the metric to be updated, for example because
    # a log was received a bit later.
    # When we put multiple values for the same timestamp
    # in the metric CW can show max, min, avg, and percentiles.
    # Since this is an update of a count we should then always
    # use "max".
    start_query_response = logs.start_query( 4
        logGroupName=log_group,
        startTime=int((datetime.now() - timedelta(minutes=15)).timestamp()),
        endTime=int(datetime.now().timestamp()),
        queryString=query,
    )

    query_id = start_query_response["queryId"]

    # Just polling the API. 5 seconds seems to be a good
    # compromise between not pestering the API and not paying
    # too much for the Lambda.
    response = None
    while response is None or response["status"] == "Running":
        print(f"{slot_name}: waiting for query to complete ...")
        time.sleep(5)
        response = logs.get_query_results(queryId=query_id)

    # Data comes in a strange format, a dictionary of
    # {"field":name,"value":actual_value}, so this converts
    # it into something that can be accessed through keys
    data = []
    for d in response["results"]: 5
        sample = {}
        for i in d:
            field = i["field"]
            value = i["value"]

            sample[field] = value

        data.append(sample)

    # Now that we have the data, let's put them into a metric.
    for d in data:
        timestamp = datetime.strptime(d["bin(1m)"], "%Y-%m-%d %H:%M:%S.000")
        value = int(d["Value"])

        print(f"{slot_name}: putting {value} on {timestamp}")

        cw.put_metric_data( 6
            Namespace=namespace,
            MetricData=[
                {
                    "MetricName": metric_name,
                    "Dimensions": [{"Name": "Cluster", "Value": cluster_name}],
                    "Timestamp": timestamp,
                    "Value": value,
                    "Unit": "None",
                }
            ],
        )


def loginsights2metrics(event, context): 1
    with open("package_info.json", "r") as f:
        package_info = json.load(f)
        build_timestamp = int(package_info["build_time"])
        build_datetime = datetime.fromtimestamp(build_timestamp)

    print("###################################")
    print(
        "LogInsights2Metrics - Build date: "
        f'{build_datetime.strftime("%Y/%m/%d %H:%M:%S")}'
    )
    print("###################################")

    print(f'Reading task from DynamoDB table {os.environ["DYNAMODB_TABLE"]}')

    table = dynamodb.Table(os.environ["DYNAMODB_TABLE"])

    # This is the simplest way to get all entries in the table
    # The next loop will asynchronously call `put_metric_data`
    # on each entry.
    response = table.scan(Select="ALL_ATTRIBUTES") 2

    for i in response["Items"]:
        print(f"* Processing item {i['SlotName']}")
        put_metric_data(i)

So, when the Lambda is executed, the entry point is the function loginsights2metrics 1 which queries the DynamoDB table 2 and loops over all the items contained in it. The loop executes the function put_metric_data 3 which being a Zappa task runs it in a new Lambda invocation. This function runs the Log Insights query 4, adjusts Boto3's output 5, and finally puts the values in the custom metric 6.

The problem I mention in the comment just before I run logs.start_query is interesting. Log Insights are queries, and since they extract data from logs the result can change between two calls of the same query. This means that, since there is an overlap between calls (we run a query on the last 15 minutes every 2 minutes), the function will put multiple values in the same bin of the metric. This is perfectly normal, and it's the reason why CloudWatch allows you to show the maximum, minimum, average, or various percentiles of the same metric. When it comes to counting events, the number can only increase or stay constant in time, but never decrease, so it's sensible to look at the maximum. This is not true if you are looking at execution times, for example, so pay attention to the nature of the underlying query when you graph the metric.

The Zappa settings I use for the function are

zappa_settings.json

{
  "main": {
    "app_module": "main",
    "app_function": "main.loginsights2metrics",
    "runtime": "python3.8",
    "log_level": "WARNING",
    "xray_tracing": true,
    "exception_handler": "zappa_sentry.unhandled_exceptions"
  }
}

And the requirements are

requirements.txt

zappa
zappa-sentry

Please note that as I mentioned before zappa-sentry is not a strict requirement for this solution.

The code can be packaged and deployed with a simple bash script like

#!/bin/bash

VENV_DIRECTORY=venv
LAMBDA_PACKAGE=lambda.zip
REGION=eu-west-1
FUNCTION_NAME=loginsights2metrics

if [[ -d ${VENV_DIRECTORY} ]]; then rm -fR ${VENV_DIRECTORY}; fi
if [[ -f ${LAMBDA_PACKAGE} ]]; then rm -fR ${LAMBDA_PACKAGE}; fi

python -m venv ${VENV_DIRECTORY}
source ${VENV_DIRECTORY}/bin/activate

pip install -r requirements.txt

zappa package main -o ${LAMBDA_PACKAGE}

rm -fR ${VENV_DIRECTORY}

aws --region=${REGION} lambda update-function-code --function-name ${FUNCTION_NAME}  --zip-file "fileb://${LAMBDA_PACKAGE}"

Costs¶

I will follow here the AWS guide on Lambda pricing and the calculations published in 2018 by my colleague João Neves on his blog.

I assume the following:

The Lambda runs 4 queries, so we have 5 invocations (1 for the main Lambda and 4 asynchronous tasks)
Each invocation runs for 5 seconds. The current average time of each invocation in my AWS accounts is 4.6 seconds
I run the Lambda every 2 minutes

Requests: 5 invocations/event * 30 events/hour * 24 hours/day * 31 days/month = 111600 requests

Duration: 0.128 GB/request * 111600 requests * 5 seconds = 71424 GB-second

Total: $0.20 * 111600 / 10^6 + $0.0000166667 * 71424 ~= $1.22/month

As you can see, for applications like this it's extremely convenient to use a serverless solution like Lambda functions.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

First-class objects in Python - Higher-order functions, wrappers, and factories

2021-03-09T16:00:00+00:00

My new book "First-class objects in Python" is out! Grab your FREE copy here!

Mau: a lightweight markup language

2021-02-22T10:00:00+00:00

What is Mau?¶

Mau is a lightweight markup language heavily inspired by AsciiDoc that makes is very easy to write blog posts or books.

The main goal of Mau is to provide a customisable markup language, reusing the good parts of AsciiDoc and providing a pure Python 3 implementation.

You can find Mau's source code on GitHub.

Why not Markdown or AsciiDoc?¶

Markdown is a very good format, and I used it for all the posts in this blog so far. I grew increasingly unsatisfied, though, because of the lack of some features and the poor amount of customisation that it provides. When I wrote the second version of my book "Clean Architectures in Python" I considered using Markdown (through Pelican), but I couldn't find a good way to create tips and warnings. Recently, Python Markdown added a feature that allows to specify the file name for the source code, but the resulting HTML cannot easily be changed, making it difficult to achieve the graphical output I wanted.

AsciiDoc started as a Python project, but then was abandoned and eventually resurrected by Dan Allen with Asciidoctor. AsciiDoc has a lot of features and I consider it superior to Markdown, but Asciidoctor is a Ruby program, and this made it difficult for me to use it. In addition, the standard output of Asciidoctor is a nice single HTML page but again customising it is a pain. I had to struggle to add my Google Analytics code and a sitemap.xml to the book site.

I simply thought I could try to write my own tool, in a language that I know well (Python). It works, and I learned a lot writing it, so I'm definitely happy. I'd be delighted to know that this can be useful to other people, though.

Pelican¶

A reader for Mau source files is available in Pelican, you can find the code at https://github.com/getpelican/pelican-plugins/pull/1327. Simply add the code to your Pelican plugins directory and activate it adding "mau_reader" to PLUGINS in your file pelicanconf.py. The Mau reader processes only files with the .mau extension, so you can use Markdown/reStructuredText and Mau at the same time.

Development¶

If you are interested you can leave a star on the project on the GitHub page, start using it, or contribute ideas, code, bugfixes.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Public key cryptography: SSL certificates

2020-11-04T23:00:00+01:00

In the context of public key cryptography, certificates are a way to prove the identity of the owner of a public key.

While public key cryptography allows us to communicate securely through an insecure network, it leaves the problem of identity untouched. Once we established an encrypted communication we can be sure that the data we send and receive cannot be read or tampered with by third parties. But how can we be sure that the entity on the other side of the communication channel, with which we initiated the communication, is what it claims to be?

In other words, the messages cannot be read or modified by malicious third-parties, but what if we established communication with a malicious actor in the first place? Such a situation can arise during a man-in-the-middle attack, where the low-level network communication is hijacked by a malicious actor who pretends to be the desired recipient of the communication.

In the context of the Internet, and in particular of the World Wide Web, the main concern is that the server that provides services we log into (think of every service that has your personal or financial data like you bank, Google, Facebook, Netflix, etc.) is run by the company that we trust and not by an attacker who wants to steal our data.

In this post I will try to clarify the main components of the certificates system and to explain the meaning of the major acronyms and names that you might hear when you deal with this part of web development.

Clarification: SSL vs TLS¶

In the world of web development and infrastructure management, we normally speak of SSL protocol and of SSL certificates, but it has to be noted that SSL (Secure Sockets Layer) is the name of a deprecated protocol. The current implementation of the protocol used to secure web applications is TLS (Transport Layer Security).

The story of SSL and TLS is rich of events and spans 25 years since its inception by Taher Elgamal at Netscape. In short, SSL had 3 major versions (the first of which was never publicly used), and was replaced by TLS in 1999. TLS itself has gone through 3 revisions at the time of writing, TLS 1.3 being the latest version available.

The TLS/SSL nomenclature is one of many sources of confusion in the complicated world of security and applied cryptography. In this article I will use only the acronym TLS, but I went for SSL in the title because I wanted the subject matter to be recognisable also by developers that are not much into security and cryptography.

X.509 certificates¶

While the problem of the identity in an insecure network can be solved in several ways, the solution embraced to secure the World Wide Web is based on a standard called X.509. When we mention TLS certificates, we usually mean X.509 certificates used in a TLS connection, such as that created by HTTPS.

X.509 is the ITU-T standard used to represent certificates, and has been chosen to be the standard used in the TLS protocol. The standard doesn't only define the binary structure of the certificate itself, but it also defines procedures to revoke the certificates, and establishes a hierarchical system of certification known as certificate path, or certificate chain.

The structure of an X.509 certificate is expressed using ASN.1, a notation used natively by the PEM format (discussed here). You can read the full specification in RFC 2459, in particular Section 4 "Certificate and Certificate Extensions Profile". I will refer to this later when I will have a look at a real certificate.

Before I discuss how certificates solve the problem of identity (or ownership of a public key), let's clarify the relationship between them and HTTPS.

HTTPS stands for HTTP Secure, and the core of the protocol consists of running HTTP over TLS. When we access a web site with HTTPS the browser first establishes a TLS connection with the server and then communicates with it using pure HTTP. This means that the whole HTTP protocol is encrypted, as the secure channel is established outside it, and also means that, aside from the different URI scheme https:// instead of http://, there are no differences between the two protocols.

Certificates come into play when the browser establishes the TLS connection, which is why you need to set-up HTTPS as part of your infrastructure and not in your web application. By the time the HTTP requests reach your application they are already decrypted and accessible in plain text, as the HTTP protocol mandates. We usually say that we "terminate TLS" when a component of our infrastructure manages certificates and decrypts HTTPS into HTTP.

How do certificates work?¶

The X.509 standard establishes entities called Certificate Authorities (CAs), and creates a hierarchy of trust called chain between them. The idea is that there is a set of entities that are trusted worldwide by operating systems, browsers, and other network-related software, and that these entities can trust other entities, thus creating a trust network.

While the market of Certificate Authorities is dominated by three major commercial players (see the usage statistics) there are approximately 100 organisations operating worldwide, among which some non-profit ones. Not all of these are trusted by all operating systems or browsers, though.

The set of CAs trusted by an organisation is called root program. The Mozilla community runs a program that is independent from the hardware/software platform, aptly called Mozilla's CA Certificate Program and uses data contained in the Common CA Database (CCADB). Private companies such as Microsoft, Apple, and Oracle run their own root programs and software running on the respective platforms (Windows, macOS/iOS, Java) can decide to trust the CAs provided by those programs.

In the open-source world, the Mozilla root program is by far the most influential and important source of information, being used by other software packages and Linux distributions.

It is possible to create certificates that are not signed by any CA, and these are called self-signed certificates. Such certificates can be used with any software that relies on certificates, but it requires such a software to disable certificate checking with the Certificate Authorities. Self-signed certificates are obviously useful for testing purposes, but there are scenarios in which it might be desirable not to rely on the CAs and establish a private network of trust.

Example: CA root certificate¶

The certificates for root CAs that are part of the Mozilla root program can be retrieved from the Common CA Database web page, or can be seen in the Firefox source code directly. On a running Firefox browser you can open the Privacy & Security menu and click on "View Certificates" at the bottom of the page. The CAs are listed under the tab "Authorities".

The interesting thing you can do here is to export a CA certificate. If you do it Firefox will save it in a file with extension .crt, that contains data in PEM format. I exported the certificate for Amazon Root CA 1 and I ended up with the file AmazonRootCA1.crt. If, instead of exporting, you view the certificate, you will end up in a page that allows you to download the certificate and the chain, both in PEM format, in files with the extension .pem. As you see, you are not the only one who is confused.

I described the PEM format in a post on RSA keys so I won't repeat here the whole discussion about it. The RFC 7468 ("Textual Encodings of PKIX, PKCS, and CMS Structures") describes certificates in section 5. Section 4 mentions the module id-pkix1-e for Certificate, CertificateList, and SubjectPublicKeyInfo RFC 5280 ("Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile").

The identifier id-pkix1-e is part of a registry of objects to be used in ASN.1 data created in the framework of the Public-Key Infrastructure using X.509 (PKIX) Working Group, that defined the infrastructure around the X.509 certificates system. Basically it's a standard way to identify binary objects and their structure. You can see a full list of all the objects in RFC 7299 ("Object Identifier Registry for the PKIX Working Group"). Not a very exciting one to read, if you ask me.

I can dump the content of the Amazon Root CA 1 certificate with OpenSSL

$ openssl asn1parse -inform pem -in amazon-root-ca-1.pem
    0:d=0  hl=4 l= 833 cons: SEQUENCE
    4:d=1  hl=4 l= 553 cons: SEQUENCE
    8:d=2  hl=2 l=   3 cons: cont [ 0 ]
   10:d=3  hl=2 l=   1 prim: INTEGER           :02
   13:d=2  hl=2 l=  19 prim: INTEGER           :066C9FCF99BF8C0A39E2F0788A43E696365BCA
   34:d=2  hl=2 l=  13 cons: SEQUENCE
   36:d=3  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
   47:d=3  hl=2 l=   0 prim: NULL
   49:d=2  hl=2 l=  57 cons: SEQUENCE
   51:d=3  hl=2 l=  11 cons: SET
   53:d=4  hl=2 l=   9 cons: SEQUENCE
   55:d=5  hl=2 l=   3 prim: OBJECT            :countryName
   60:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :US
   64:d=3  hl=2 l=  15 cons: SET
   66:d=4  hl=2 l=  13 cons: SEQUENCE
   68:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
   73:d=5  hl=2 l=   6 prim: PRINTABLESTRING   :Amazon
   81:d=3  hl=2 l=  25 cons: SET
   83:d=4  hl=2 l=  23 cons: SEQUENCE
   85:d=5  hl=2 l=   3 prim: OBJECT            :commonName
   90:d=5  hl=2 l=  16 prim: PRINTABLESTRING   :Amazon Root CA 1
  108:d=2  hl=2 l=  30 cons: SEQUENCE
  110:d=3  hl=2 l=  13 prim: UTCTIME           :150526000000Z
  125:d=3  hl=2 l=  13 prim: UTCTIME           :380117000000Z
  140:d=2  hl=2 l=  57 cons: SEQUENCE
  142:d=3  hl=2 l=  11 cons: SET
  144:d=4  hl=2 l=   9 cons: SEQUENCE
  146:d=5  hl=2 l=   3 prim: OBJECT            :countryName
  151:d=5  hl=2 l=   2 prim: PRINTABLESTRING   :US
  155:d=3  hl=2 l=  15 cons: SET
  157:d=4  hl=2 l=  13 cons: SEQUENCE
  159:d=5  hl=2 l=   3 prim: OBJECT            :organizationName
  164:d=5  hl=2 l=   6 prim: PRINTABLESTRING   :Amazon
  172:d=3  hl=2 l=  25 cons: SET
  174:d=4  hl=2 l=  23 cons: SEQUENCE
  176:d=5  hl=2 l=   3 prim: OBJECT            :commonName
  181:d=5  hl=2 l=  16 prim: PRINTABLESTRING   :Amazon Root CA 1
  199:d=2  hl=4 l= 290 cons: SEQUENCE
  203:d=3  hl=2 l=  13 cons: SEQUENCE
  205:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  216:d=4  hl=2 l=   0 prim: NULL
  218:d=3  hl=4 l= 271 prim: BIT STRING
  493:d=2  hl=2 l=  66 cons: cont [ 3 ]
  495:d=3  hl=2 l=  64 cons: SEQUENCE
  497:d=4  hl=2 l=  15 cons: SEQUENCE
  499:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Basic Constraints
  504:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  507:d=5  hl=2 l=   5 prim: OCTET STRING      [HEX DUMP]:30030101FF
  514:d=4  hl=2 l=  14 cons: SEQUENCE
  516:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Key Usage
  521:d=5  hl=2 l=   1 prim: BOOLEAN           :255
  524:d=5  hl=2 l=   4 prim: OCTET STRING      [HEX DUMP]:03020186
  530:d=4  hl=2 l=  29 cons: SEQUENCE
  532:d=5  hl=2 l=   3 prim: OBJECT            :X509v3 Subject Key Identifier
  537:d=5  hl=2 l=  22 prim: OCTET STRING      [HEX DUMP]:04148418CC8534ECBC0C94942E08599CC7B2104E0A08
  561:d=1  hl=2 l=  13 cons: SEQUENCE
  563:d=2  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
  574:d=2  hl=2 l=   0 prim: NULL
  576:d=1  hl=4 l= 257 prim: BIT STRING

Let's read part of it using the aforementioned section 4 of RFC 5280.

The signed certificate is a sequence of three main components

   Certificate  ::=  SEQUENCE  {
        tbsCertificate       TBSCertificate,
        signatureAlgorithm   AlgorithmIdentifier,
        signatureValue       BIT STRING  }

and the TBSCertificate structure represents the unsigned certificate (TBS = To Be Signed)

TBSCertificate  ::=  SEQUENCE  {
        version         [0]  EXPLICIT Version DEFAULT v1,
        serialNumber         CertificateSerialNumber,
        signature            AlgorithmIdentifier,
        issuer               Name,
        validity             Validity,
        subject              Name,
        subjectPublicKeyInfo SubjectPublicKeyInfo,
        issuerUniqueID  [1]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        subjectUniqueID [2]  IMPLICIT UniqueIdentifier OPTIONAL,
                             -- If present, version MUST be v2 or v3
        extensions      [3]  EXPLICIT Extensions OPTIONAL
                             -- If present, version MUST be v3
        }

Comparing this with the output of OpenSSL we can find fields such as version

   10:d=3  hl=2 l=   1 prim: INTEGER           :02

which according to the documentation is 3 (binary 02). Many values are of type PRINTABLESTRING, so they are readable already in the ASN.1 dump.

The validity of the certificate is

  110:d=3  hl=2 l=  13 prim: UTCTIME           :150526000000Z
  125:d=3  hl=2 l=  13 prim: UTCTIME           :380117000000Z

and following section 4.1.2.5.1 of the RFC we find out that the certificate is valid between 26 May 2015 and 17 Jan 2038. You can easily read these values in the certificate page in the browser without getting an headache trying to decode ASN.1.

The CA signed the certificate using a certain algorithm. The algorithm identifier is repeated twice, first in the structure Certificate (signatureAlgorithm AlgorithmIdentifier) and then in the structure TBSCertificate (signature AlgorithmIdentifier). The two fields must have the same value.

   34:d=2  hl=2 l=  13 cons: SEQUENCE
   36:d=3  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
   47:d=3  hl=2 l=   0 prim: NULL

[...]

  561:d=1  hl=2 l=  13 cons: SEQUENCE
  563:d=2  hl=2 l=   9 prim: OBJECT            :sha256WithRSAEncryption
  574:d=2  hl=2 l=   0 prim: NULL

For this certificate, the algorithm used by Amazon is sha256WithRSAEncryption. This label is described in RFC 4055 ("Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile") as "PKCS #1 version 1.5 signature algorithm with SHA-256". The specific algorithm can be found in RFC 2313 ("PKCS #1: RSA Encryption Version 1.5"). As the name of the algorithm suggests, the certificate is first digested with SHA-256 and then encrypted using RSA and the private key of the signer.

Speaking of keys, the public key the CA used for the certificate can be found in the field subjectpublickeyinfo, which is again made of a field type AlgorithmIdentifier and a bit string with the value of the key. In this case the fields are

  205:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  216:d=4  hl=2 l=   0 prim: NULL
  218:d=3  hl=4 l= 271 prim: BIT STRING

The algorithm rsaEncryption is described in RFC 3279 ("Algorithms and Identifiers for the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"), section 2.3.1 as

      RSAPublicKey ::= SEQUENCE {
         modulus            INTEGER,    -- n
         publicExponent     INTEGER  }  -- e

(sic) or in RFC 8017 ("PKCS #1: RSA Cryptography Specifications Version 2.2")

RSAPublicKey ::= SEQUENCE {
    modulus           INTEGER,  -- n
    publicExponent    INTEGER   -- e
}

We can then use the option -strparse of the module asn1parse to find the actual values

$ openssl asn1parse -inform pem -in amazon-root-ca-1.pem -strparse 218
    0:d=0  hl=4 l= 266 cons: SEQUENCE
    4:d=1  hl=4 l= 257 prim: INTEGER           :B2788071CA78D5E371AF478050747D6ED8D78876F4
9968F7582160F97484012FAC022D86D3A0437A4EB2A4D036BA01BE8DDB48C80717364CF4EE8823C73EEB37F5B5
19F84968B0DED7B976381D619EA4FE8236A5E54A56E445E1F9FDB416FA74DA9C9B35392FFAB02050066C7AD080
B2A6F9AFEC47198F503807DCA2873958F8BAD5A9F948673096EE94785E6F89A351C0308666A14566BA54EBA3C3
91F948DCFFD1E8302D7D2D747035D78824F79EC4596EBB738717F2324628B843FAB71DAACAB4F29F240E2D4BF7
715C5E69FFEA9502CB388AAE50386FDBFB2D621BC5C71E54E177E067C80F9C8723D63F40207F2080C4804C3E3B
24268E04AE6C9AC8AA0D
  265:d=1  hl=2 l=   3 prim: INTEGER           :010001

As we already saw for RSA keys), OpenSSL has a specific module for important structures, and the X.509 certificates are definitely worth a module aptly called x509. using that we can easily decode any certificate

$ openssl x509 -inform pem -in amazon-root-ca-1.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            06:6c:9f:cf:99:bf:8c:0a:39:e2:f0:78:8a:43:e6:96:36:5b:ca
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Amazon, CN = Amazon Root CA 1
        Validity
            Not Before: May 26 00:00:00 2015 GMT
            Not After : Jan 17 00:00:00 2038 GMT
        Subject: C = US, O = Amazon, CN = Amazon Root CA 1
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b2:78:80:71:ca:78:d5:e3:71:af:47:80:50:74:
                    7d:6e:d8:d7:88:76:f4:99:68:f7:58:21:60:f9:74:
                    84:01:2f:ac:02:2d:86:d3:a0:43:7a:4e:b2:a4:d0:
                    36:ba:01:be:8d:db:48:c8:07:17:36:4c:f4:ee:88:
                    23:c7:3e:eb:37:f5:b5:19:f8:49:68:b0:de:d7:b9:
                    76:38:1d:61:9e:a4:fe:82:36:a5:e5:4a:56:e4:45:
                    e1:f9:fd:b4:16:fa:74:da:9c:9b:35:39:2f:fa:b0:
                    20:50:06:6c:7a:d0:80:b2:a6:f9:af:ec:47:19:8f:
                    50:38:07:dc:a2:87:39:58:f8:ba:d5:a9:f9:48:67:
                    30:96:ee:94:78:5e:6f:89:a3:51:c0:30:86:66:a1:
                    45:66:ba:54:eb:a3:c3:91:f9:48:dc:ff:d1:e8:30:
                    2d:7d:2d:74:70:35:d7:88:24:f7:9e:c4:59:6e:bb:
                    73:87:17:f2:32:46:28:b8:43:fa:b7:1d:aa:ca:b4:
                    f2:9f:24:0e:2d:4b:f7:71:5c:5e:69:ff:ea:95:02:
                    cb:38:8a:ae:50:38:6f:db:fb:2d:62:1b:c5:c7:1e:
                    54:e1:77:e0:67:c8:0f:9c:87:23:d6:3f:40:20:7f:
                    20:80:c4:80:4c:3e:3b:24:26:8e:04:ae:6c:9a:c8:
                    aa:0d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                84:18:CC:85:34:EC:BC:0C:94:94:2E:08:59:9C:C7:B2:10:4E:0A:08
    Signature Algorithm: sha256WithRSAEncryption
         98:f2:37:5a:41:90:a1:1a:c5:76:51:28:20:36:23:0e:ae:e6:
         28:bb:aa:f8:94:ae:48:a4:30:7f:1b:fc:24:8d:4b:b4:c8:a1:
         97:f6:b6:f1:7a:70:c8:53:93:cc:08:28:e3:98:25:cf:23:a4:
         f9:de:21:d3:7c:85:09:ad:4e:9a:75:3a:c2:0b:6a:89:78:76:
         44:47:18:65:6c:8d:41:8e:3b:7f:9a:cb:f4:b5:a7:50:d7:05:
         2c:37:e8:03:4b:ad:e9:61:a0:02:6e:f5:f2:f0:c5:b2:ed:5b:
         b7:dc:fa:94:5c:77:9e:13:a5:7f:52:ad:95:f2:f8:93:3b:de:
         8b:5c:5b:ca:5a:52:5b:60:af:14:f7:4b:ef:a3:fb:9f:40:95:
         6d:31:54:fc:42:d3:c7:46:1f:23:ad:d9:0f:48:70:9a:d9:75:
         78:71:d1:72:43:34:75:6e:57:59:c2:02:5c:26:60:29:cf:23:
         19:16:8e:88:43:a5:d4:e4:cb:08:fb:23:11:43:e8:43:29:72:
         62:a1:a9:5d:5e:08:d4:90:ae:b8:d8:ce:14:c2:d0:55:f2:86:
         f6:c4:93:43:77:66:61:c0:b9:e8:41:d7:97:78:60:03:6e:4a:
         72:ae:a5:d1:7d:ba:10:9e:86:6c:1b:8a:b9:59:33:f8:eb:c4:
         90:be:f1:b9

Now I'm pretty sure you want to kill me because I could have shown you this from the start. But I like to understand things, and the easy path doesn't always make everything clear. At any rate, here you have a way to read an X.509 certificate in PEM format.

Please note that in this certificate the Issuer and the Subject are the same entity, as this is a root certificate, which is signed by the same entity that creates it.

        Issuer: C = US, O = Amazon, CN = Amazon Root CA 1
[...]
        Subject: C = US, O = Amazon, CN = Amazon Root CA 1

Moreover, one of the version 3 extensions of the self-signed certificate is a basic constraint with the boolean CA set to true. It also has the extension Key Usage set to Digital Signature, Certificate Sign, CRL Sign, which means that the certificate can be used to sign other certificates.

Example: self-signed certificate¶

You can use OpenSSL to create a self-signed certificate using the module req that you would normally use to create certificate requests. As a self-signed certificate doesn't need approval, the module can directly output the certificate.

$ openssl req -x509 -newkey rsa:2048 -keyout self-signed-key.pem -out self-signed.pem -days 365 -nodes -subj '/CN=localhost'
Generating a RSA private key
....+++++
................+++++
writing new private key to 'self-signed-key.pem'
-----

(note that for simplicity's sake I specified the option -nodes that prevents the key to be protected with a password, but this is a bad practice). This command creates the two files I mentioned, self-signed-key.pem (the private key) and self-signed.pem.

We can read the certificate using the module x509

$ openssl x509 -inform pem -in self-signed.pem  -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            46:e5:2f:8e:42:82:43:b8:ac:88:cb:6d:0c:2f:71:28:a9:fe:00:ec
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = localhost
        Validity
            Not Before: Nov  3 00:23:34 2020 GMT
            Not After : Nov  3 00:23:34 2021 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b7:14:ef:3b:eb:8b:a9:40:18:c5:d2:eb:1d:4f:
                    5d:e4:a3:17:f3:df:ce:b7:d3:3f:52:58:eb:61:02:
                    a2:68:0a:cd:0f:97:ae:e0:a5:ac:a7:88:cf:a1:15:
                    0a:97:ca:e7:03:8a:a5:c0:66:38:ef:bb:59:4d:48:
                    17:db:a7:bd:fa:4b:50:2a:be:e9:5b:bb:59:65:71:
                    dc:99:73:9c:bc:4d:3b:42:97:91:e9:3b:1a:8a:9d:
                    cc:41:38:ba:8b:8f:df:65:ff:5b:1f:ef:8a:b7:c5:
                    93:07:ce:15:4c:13:72:78:59:64:9a:5b:95:20:b6:
                    b3:8e:aa:c3:29:c3:7f:28:39:43:81:59:e4:0f:26:
                    7c:3f:49:d2:06:05:d9:54:ab:09:65:96:01:cc:c2:
                    72:be:85:1f:40:ea:94:35:04:09:9d:87:eb:a1:90:
                    36:ce:d2:55:f9:ee:08:db:52:78:e8:70:d0:25:89:
                    13:8e:0f:9d:98:98:d1:4d:67:06:8f:8a:61:9e:3a:
                    73:89:aa:0a:0a:1b:05:a7:52:32:ef:1b:78:5a:5f:
                    4b:b6:c9:a7:4e:15:10:04:50:99:00:09:2f:60:8e:
                    aa:20:af:6b:ee:f5:60:0b:29:da:38:1c:b2:73:14:
                    99:a4:ee:5e:89:e6:77:0b:ba:cf:d3:5d:d7:a3:ea:
                    c4:bf
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier: 
                64:7B:C1:FC:99:74:56:B7:82:D1:4F:E7:2D:94:77:1A:09:52:26:5C
            X509v3 Authority Key Identifier: 
                keyid:64:7B:C1:FC:99:74:56:B7:82:D1:4F:E7:2D:94:77:1A:09:52:26:5C

            X509v3 Basic Constraints: critical
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         43:7b:0b:c8:98:b8:6f:72:af:39:4a:d9:76:ce:e3:9d:3a:c7:
         9f:14:b0:4f:20:0a:45:b3:b4:8c:e5:37:4c:bf:15:ad:8e:5c:
         45:4f:3e:b7:ef:8d:60:57:bb:6f:d9:5e:6a:d3:04:05:4a:ff:
         f2:66:b1:76:66:59:7e:24:89:0a:50:28:c9:d5:f5:7a:00:07:
         8a:79:9c:6e:53:43:66:e5:9a:10:d8:f8:e1:f2:c1:f1:17:d0:
         d2:9e:50:80:fe:2a:ca:08:b6:98:e9:b5:a4:82:23:31:45:35:
         33:da:2c:e3:fe:54:f2:bd:f2:61:91:f4:32:e3:7d:4c:3a:e5:
         3a:0f:cd:36:b0:8b:af:9f:8e:3d:0e:0b:a5:df:4a:3a:91:83:
         b3:b2:5f:3c:47:81:73:4f:a2:c1:49:06:75:17:25:fa:5a:8d:
         30:e5:55:7f:9c:3e:15:a8:b5:ab:f7:45:38:e3:76:8e:d4:0d:
         60:fc:42:17:3d:85:72:41:1d:53:9d:58:b0:e9:29:0c:e4:6b:
         14:c2:22:c4:d5:7b:de:36:da:df:d8:a0:4f:a4:0a:f2:3e:ca:
         7e:66:a6:10:38:97:24:73:5b:db:eb:0b:6c:a8:f8:37:15:2c:
         0e:b1:82:44:cc:fe:85:b0:cb:6c:26:4b:4a:70:33:dc:7e:f5:
         84:ba:07:db

As you can see this certificate has the same value in Issuer and Subject, as happened before for the Amazon Root one. It also has the flag CA set to true but it doesn't have the extension Key Usage meaning that this certificate can't be used to sign other certificates.

Example: this site's certificate¶

You can see TLS certificates and the chain of trust in action in this very website. Following the documentation of you browser (instructions for Firefox are here), you can see the certificate used by The Digital Cat. At the time of writing the blog is hosted on GitHub Pages, even tough I'm using a custom domain, and GitHub partnered with Let's Encrypt to provide certificates for such a configuration (details here).

Indeed, the certificate for thedigitalcatonline.com is provided by "Let's Encrypt Authority X3", which in turn is trusted by Digital Signature Trust Co. with its root CA "DST Root CA X3".

Let's have a look at the three certificates. The one for The Digital Cat is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:93:02:bb:9a:c9:ed:a5:c3:d1:16:00:8b:15:76:af:e5:d9
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
        Validity
            Not Before: Oct 22 04:53:28 2020 GMT
            Not After : Jan 20 04:53:28 2021 GMT
        Subject: CN = www.thedigitalcatonline.com
[...]
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                63:4E:15:85:56:5A:A4:94:02:C2:16:42:A4:A5:97:9A:38:02:57:97
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:www.thedigitalcatonline.com
[...]

And you can see that this time the Subject is www.thedigitalcatonline.com, but the Issuer is Let's Encrypt Authority X3. The certificate provided by the organisation Let's Encrypt is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Mar 17 16:40:46 2016 GMT
            Not After : Mar 17 16:40:46 2021 GMT
        Subject: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
[...]
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            Authority Information Access: 
                OCSP - URI:http://isrg.trustid.ocsp.identrust.com
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier: 
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier: 
                A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1
[...]

Here, the Subject is Let's Encrypt Authority X3 (the Issuer of the previous certificate), and the Issuer is DST Root CA X3. Last, the certificate provided by the organisation Digital Signature Trust Co. is

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
[...]
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10
[...]

As happened for the certificate Amazon Root CA 1 that we discussed before, this one is self-signed, having the same value for Subject and Issuer.

How to verify certificates with OpenSSL¶

To verify if a certificate is valid we can use the module verify of OpenSSL. By default, OpenSSL doesn't trust anything, and verify relies on a default path in the system to find root certificates. You can see the path running

$ openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

On Ubuntu 20.04, the directory /usr/lib/ssl/certs is a symbolic link to /etc/ssl/certs that is installed by the package ca-certificates which is linked to the Mozilla's CA Certificate Program (details on that package can be found in the source code).

So, if a root certificate is included in the Mozilla program, it is trusted by OpenSSL

$ openssl verify amazon-root-ca-1.pem 
amazon-root-ca-1.pem: OK

while a self-signed certificate is not

$ openssl verify self-signed.pem 
CN = localhost
error 18 at 0 depth lookup: self signed certificate
error self-signed.pem: verification failed

A non-root certificate can be verified specifying which root certificate signed it. So, the certificate for this website is not trusted automatically

$ openssl verify www-thedigitalcatonline-com.pem 
CN = www.thedigitalcatonline.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error www-thedigitalcatonline-com.pem: verification failed

But it is verified specifying the certificate for Let's Encrypt that signed it

$ openssl verify -CAfile lets-encrypt-x3.pem www-thedigitalcatonline-com.pem 
www-thedigitalcatonline-com.pem: OK

because the certificate lets-encrypt-x3.pem is signed by DST_Root_CA_X3.pem which is included in the Mozilla program, and thus included in my Linux distribution.

If I remove the default certificates path OpenSSL doesn't accept the certificate for Let's Encrypt any more

$ openssl verify -no-CApath -CAfile lets-encrypt-x3.pem www-thedigitalcatonline-com.pem
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
error 2 at 1 depth lookup: unable to get issuer certificate
error www-thedigitalcatonline-com.pem: verification failed

Low-level certificate validation process¶

Let's have a look at the signature process for x.509 certificates. The process depends on the specific algorithm used to sign the certificate, so I will use the certificate Amazon Root CA 1 as an example, leaving to the reader the investigation about other algorithms.

A signed certificate is made of two parts, the certificate itself and the signature. The signature contains an encrypted hash of the certificate, so the verification is done in three steps:

Decrypt the encrypted hash using the public key
Compute the hash of the certificate using the same algorithm
Compare the hashes

For the Amazon root certificate, we know the signature algorithm and value from the output of openssl x509

$ openssl x509 -inform pem -in amazon-root-ca-1.pem  -noout -text
[...]
    Signature Algorithm: sha256WithRSAEncryption
         98:f2:37:5a:41:90:a1:1a:c5:76:51:28:20:36:23:0e:ae:e6:
         28:bb:aa:f8:94:ae:48:a4:30:7f:1b:fc:24:8d:4b:b4:c8:a1:
         97:f6:b6:f1:7a:70:c8:53:93:cc:08:28:e3:98:25:cf:23:a4:
         f9:de:21:d3:7c:85:09:ad:4e:9a:75:3a:c2:0b:6a:89:78:76:
         44:47:18:65:6c:8d:41:8e:3b:7f:9a:cb:f4:b5:a7:50:d7:05:
         2c:37:e8:03:4b:ad:e9:61:a0:02:6e:f5:f2:f0:c5:b2:ed:5b:
         b7:dc:fa:94:5c:77:9e:13:a5:7f:52:ad:95:f2:f8:93:3b:de:
         8b:5c:5b:ca:5a:52:5b:60:af:14:f7:4b:ef:a3:fb:9f:40:95:
         6d:31:54:fc:42:d3:c7:46:1f:23:ad:d9:0f:48:70:9a:d9:75:
         78:71:d1:72:43:34:75:6e:57:59:c2:02:5c:26:60:29:cf:23:
         19:16:8e:88:43:a5:d4:e4:cb:08:fb:23:11:43:e8:43:29:72:
         62:a1:a9:5d:5e:08:d4:90:ae:b8:d8:ce:14:c2:d0:55:f2:86:
         f6:c4:93:43:77:66:61:c0:b9:e8:41:d7:97:78:60:03:6e:4a:
         72:ae:a5:d1:7d:ba:10:9e:86:6c:1b:8a:b9:59:33:f8:eb:c4:
         90:be:f1:b9

You can see the signed certificate binary values with cat amazon-root-ca-1.pem | tail -n+2 | head -n-1 | base64 -di | hexdump -ve '/1 "%02x "' -e '2/8 "\n"'. While we can recognise the signature in the last 256 bytes we can't easily separate the bytes with the signature algorithm. If we open the signed certificate with an ASN.1 parser, instead, we can easily find the binary value of the certificate part

30 82 03 41 30 82 02 29 a0 03 02 01 02 02 13 06
6c 9f cf 99 bf 8c 0a 39 e2 f0 78 8a 43 e6 96 36
5b ca 30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05
00 30 39 31 0b 30 09 06 03 55 04 06 13 02 55 53
31 0f 30 0d 06 03 55 04 0a 13 06 41 6d 61 7a 6f
6e 31 19 30 17 06 03 55 04 03 13 10 41 6d 61 7a
6f 6e 20 52 6f 6f 74 20 43 41 20 31 30 1e 17 0d
31 35 30 35 32 36 30 30 30 30 30 30 5a 17 0d 33
38 30 31 31 37 30 30 30 30 30 30 5a 30 39 31 0b
30 09 06 03 55 04 06 13 02 55 53 31 0f 30 0d 06
03 55 04 0a 13 06 41 6d 61 7a 6f 6e 31 19 30 17
06 03 55 04 03 13 10 41 6d 61 7a 6f 6e 20 52 6f
6f 74 20 43 41 20 31 30 82 01 22 30 0d 06 09 2a
86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30
82 01 0a 02 82 01 01 00 b2 78 80 71 ca 78 d5 e3
71 af 47 80 50 74 7d 6e d8 d7 88 76 f4 99 68 f7
58 21 60 f9 74 84 01 2f ac 02 2d 86 d3 a0 43 7a
4e b2 a4 d0 36 ba 01 be 8d db 48 c8 07 17 36 4c
f4 ee 88 23 c7 3e eb 37 f5 b5 19 f8 49 68 b0 de
d7 b9 76 38 1d 61 9e a4 fe 82 36 a5 e5 4a 56 e4
45 e1 f9 fd b4 16 fa 74 da 9c 9b 35 39 2f fa b0
20 50 06 6c 7a d0 80 b2 a6 f9 af ec 47 19 8f 50
38 07 dc a2 87 39 58 f8 ba d5 a9 f9 48 67 30 96
ee 94 78 5e 6f 89 a3 51 c0 30 86 66 a1 45 66 ba
54 eb a3 c3 91 f9 48 dc ff d1 e8 30 2d 7d 2d 74
70 35 d7 88 24 f7 9e c4 59 6e bb 73 87 17 f2 32
46 28 b8 43 fa b7 1d aa ca b4 f2 9f 24 0e 2d 4b
f7 71 5c 5e 69 ff ea 95 02 cb 38 8a ae 50 38 6f
db fb 2d 62 1b c5 c7 1e 54 e1 77 e0 67 c8 0f 9c
87 23 d6 3f 40 20 7f 20 80 c4 80 4c 3e 3b 24 26
8e 04 ae 6c 9a c8 aa 0d 02 03 01 00 01 a3 42 30
40 30 0f 06 03 55 1d 13 01 01 ff 04 05 30 03 01
01 ff 30 0e 06 03 55 1d 0f 01 01 ff 04 04 03 02
01 86 30 1d 06 03 55 1d 0e 04 16 04 14 84 18 cc
85 34 ec bc 0c 94 94 2e 08 59 9c c7 b2 10 4e 0a
08

The signature algorithm part is

30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00
03 82 01 01 00

and the ASN.1 parser tells us that those bytes represent an OBJECT IDENTIFIER which value is 2.16.840.1.101.3.4.2.1. Now, object identifiers are not complicated per se, they are just a way to identify algorithms and other well known components in ASN.1 structures. The description of the field signatureAlgorithm of an x.509 certificate mentions three other RFCs that contains descriptions of the available algorithms. In particular, RFC 4055 contains the description of PKCS #1 one-way hash functions, one of which is

id-sha256  OBJECT IDENTIFIER  ::=  { joint-iso-itu-t(2)
                     country(16) us(840) organization(1) gov(101)
                     csor(3) nistalgorithm(4) hashalgs(2) 1 }

You can see the values in the object identifier between parentheses. Since these are PKCS #1 (a.k.a. RSA) has functions, OpenSSL identifies it as sha256WithRSAEncryption (see again RFC 4055).

RSA encryption is described in RFC 2313 ("PKCS #1: RSA Encryption Version 1.5") and the signature algorithm based on RSA is described there in section 10. In particular, section 10.2 details the verification process, which is the one we are interested in. The steps are

Bit-string-to-octet-string conversion of the signature
RSA decryption
Digest decoding (ASN.1)
Message digesting and comparison

As for the signature conversion, the sentence

Specifically, assuming that the length in bits of the
signature S is a multiple of eight, the first bit of the signature
shall become the most significant bit of the first octet of the
encrypted data, and so on through the last bit of the signature,
which shall become the least significant bit of the last octet of the
encrypted data.

is a very verbose way to say that the signature is big-endian.

So, the hexadecimal value of the signature is

98f2375a4190a11ac57651282036230eaee628bbaaf894ae48a4307f1bfc248d
4bb4c8a197f6b6f17a70c85393cc0828e39825cf23a4f9de21d37c8509ad4e9a
753ac20b6a897876444718656c8d418e3b7f9acbf4b5a750d7052c37e8034bad
e961a0026ef5f2f0c5b2ed5bb7dcfa945c779e13a57f52ad95f2f8933bde8b5c
5bca5a525b60af14f74befa3fb9f40956d3154fc42d3c7461f23add90f48709a
d9757871d1724334756e5759c2025c266029cf2319168e8843a5d4e4cb08fb23
1143e843297262a1a95d5e08d490aeb8d8ce14c2d055f286f6c49343776661c0
b9e841d7977860036e4a72aea5d17dba109e866c1b8ab95933f8ebc490bef1b9

And reading the field Subject Public Key Info of the certificate we find the public key. Remember that this is a root certificate, so it is signed using the same key that it contains, which is not true in general.

The public key's modulus is

b2788071ca78d5e371af478050747d6ed8d78876f49968f7582160f97484012f
ac022d86d3a0437a4eb2a4d036ba01be8ddb48c80717364cf4ee8823c73eeb37
f5b519f84968b0ded7b976381d619ea4fe8236a5e54a56e445e1f9fdb416fa74
da9c9b35392ffab02050066c7ad080b2a6f9afec47198f503807dca2873958f8
bad5a9f948673096ee94785e6f89a351c0308666a14566ba54eba3c391f948dc
ffd1e8302d7d2d747035d78824f79ec4596ebb738717f2324628b843fab71daa
cab4f29f240e2d4bf7715c5e69ffea9502cb388aae50386fdbfb2d621bc5c71e
54e177e067c80f9c8723d63f40207f2080c4804c3e3b24268e04ae6c9ac8aa0d

and the exponent is 0x10001 (default choice).

RSA public-key signature decryption is performed with signature ^ exponent mod modulus, and this operation returns

1fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
fffffffffffffffffffff003031300d0609608648016503040201050004206fc
4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a

Once the padding is removed, we are left with an ASN.1 binary structure that represents the digest

DigestInfo ::= SEQUENCE {
  digestAlgorithm DigestAlgorithmIdentifier,
  digest Digest }

(see RFC 2313 - Section 10.1.2)

The value of Digest can be extracted with an ASN.1 parser or by taking the last 256 bits and is 6fc4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a.

At this point we need to process the certificate bytes (without signature) with the SHA-256 hash function and we will find a matching value of 6fc4b8ac3d2b52c08baf56255e43d22c762962e4facab01ace16d48ec008be0a.

This process (for the specific case of this certificate) can be easily done in Python

from cryptography import x509
from hashlib import sha256

certificate_pem_file = "amazon-root-ca-1.pem"

with open(certificate_pem_file, "rb") as f:
    certificate_pem = f.read()

certificate = x509.load_pem_x509_certificate(certificate_pem)

modulus = certificate.public_key().public_numbers().n
exponent = certificate.public_key().public_numbers().e

signature = int.from_bytes(certificate.signature, "big")

verification = pow(signature, exponent, modulus)

digest = bytes().fromhex(str(hex(verification))[-64:])

calculated_digest = sha256(certificate.tbs_certificate_bytes)

print(digest.hex() == calculated_digest.hexdigest())

This is arguably not the best Python code ever, but it's a simple way to demonstrate the process. As I said, this is far from being general, as it assumes the signature is sha256WithRSAEncryption, which might not be the case.

What I showed you here is what happens when we validate a root certificate. When we validate a non-root certificate the process is exactly the same (taking into account that the algorithms involved might be different), only the public key used to sign the certificate doesn't come from the certificate itself, but from the signer one. So, in the case of this blog, the certificate for www.thedigitalcat.com has a signature encrypted with the public key of Let's Encrypt. And the certificate for Let's Encrypt will be signed using the public key of Digital Signature Trust Co. This is what creates the chain of trust.

Algorithms used by root certificates¶

A quick scan of the certificates that are part of the Mozilla program reveals that the vast majority of them is using RSA to self-sign them

$ for i in /etc/ssl/certs/*.pem; do openssl x509 -inform pem -in ${i} -noout -text | grep -E "Public Key Algorithm"; done | sort | uniq -c
     25             Public Key Algorithm: id-ecPublicKey
    114             Public Key Algorithm: rsaEncryption

while part of them are using id-ecPublicKey which is the identifier of elliptic curves algorithms.

When it comes to signature algorithms, instead, there is more variety

$ for i in /etc/ssl/certs/*.pem; do openssl x509 -inform pem -in ${i} -noout -text | grep -E "^    Signature Algorithm"; done | sort | uniq -c
      7     Signature Algorithm: ecdsa-with-SHA256
     18     Signature Algorithm: ecdsa-with-SHA384
     47     Signature Algorithm: sha1WithRSAEncryption
     57     Signature Algorithm: sha256WithRSAEncryption
      9     Signature Algorithm: sha384WithRSAEncryption
      1     Signature Algorithm: sha512WithRSAEncryption

Even here, elliptic curves are slowly being adopted.

If you are using AWS, you can create certificates with ACM, the AWS Certificate Manager. Such certificates cannot be downloaded, they can only be attached to other AWS components. For this reason, the generation process requires you to create any request, as you might have to do with other authorities. Certificates created in the ACM are free.

Certificates created in the ACM can be attached to several AWS components, most notably Load Balancers, CloudFront, and API Gateway.

Traditionally, load balancers are the place where TLS is terminated for HTTPS, requiring a connection to port 443. While Application Load Balancers can do that, in 2019 AWS announced support for certificates in Network Load Balancers as well.

Let's encrypt¶

In an effort to push for HTTP encryption of any public server, the Internet Security Research Group founded in 2016 a non-profit CA named Let's Encrypt, which provides at no charge TLS certificates valid for 90 days. Such certificates can be renewed automatically as part of the setup (certbot) and represent a viable alternative to certificates issued by other CA, in particular for open source projects. This blog uses a certificate issued by Let's Encrypt (provided by GitHub Pages) and will thus expire in less than 3 months (but also automatically renewed).

Final words¶

I hope this post helped to clarify some of the most obscure points of certificates, that definitely bugged be when I first approached them. As always when standards are involved, the risk is to get lost in the myriad of documents where information is scattered, and not to realise that some (if not many) parts of the systems we run every day have a long history and thus a big burden of legacy code or nomenclature.

Resources¶

The Wikipedia article on TLS
The Wikipedia article on Certificate authority
The Wikipedia article on X.509
The Wikipedia article on Let's Encrypt
OpenSSL documentation: asn1parse, x509, verify
The Abstract Syntax Notation One ASN.1 interface description language
RFC 2313 - "PKCS #1: RSA Encryption Version 1.5"
RFC 2459 - "Internet X.509 Public Key Infrastructure Certificate and CRL Profile"
RFC 3279 - "Algorithms and Identifiers for the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
RFC 4055 - "Additional Algorithms and Identifiers for RSA Cryptography for use in the Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
RFC 5280 - "Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile"
RFC 7299 - "Object Identifier Registry for the PKIX Working Group"
RFC 7468 - "Textual Encodings of PKIX, PKCS, and CMS Structures"
RFC 8017 - "PKCS #1: RSA Cryptography Specifications Version 2.2"
RFC 8446 - "The Transport Layer Security (TLS) Protocol Version 1.3"
pyca/cryptography - The Python pyca/cryptography package

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

TDD in Python with pytest - Part 5

2020-09-21T10:30:00+02:00

This is the fifth and last post in the series "TDD in Python with pytest" where I develop a simple project following a strict TDD methodology. The posts come from my book Clean Architectures in Python and have been reviewed to get rid of some bad naming choices of the version published in the book.

You can find the first post here.

In this post I will conclude the discussion about mocks introducing patching.

Patching¶

Mocks are very simple to introduce in your tests whenever your objects accept classes or instances from outside. In that case, as shown in the previous sections, you just have to instantiate the class Mock and pass the resulting object to your system. However, when the external classes instantiated by your library are hardcoded this simple trick does not work. In this case you have no chance to pass a fake object instead of the real one.

This is exactly the case addressed by patching. Patching, in a testing framework, means to replace a globally reachable object with a mock, thus achieving the goal of having the code run unmodified, while part of it has been hot swapped, that is, replaced at run time.

A warm-up example¶

Clone the repository fileinfo that you can find here and move to the branch develop. As I did for the project simple_calculator, the branch master contains the full solution, and I use it to maintain the repository, but if you want to code along you need to start from scratch. If you prefer, you can clearly clone it on GitHub and make your own copy of the repository.

git clone https://github.com/lgiordani/fileinfo
cd fileinfo
git checkout --track origin/develop

Create a virtual environment following your preferred process and install the requirements

pip install -r requirements/dev.txt

You should at this point be able to run

pytest -svv

and get an output like

=============================== test session starts ===============================
platform linux -- Python XXXX, pytest-XXXX, py-XXXX, pluggy-XXXX --
fileinfo/venv3/bin/python3
cachedir: .cache
rootdir: fileinfo, inifile: pytest.ini
plugins: cov-XXXX
collected 0 items 

============================== no tests ran in 0.02s ==============================

Let us start with a very simple example. Patching can be complex to grasp at the beginning so it is better to start learning it with trivial use cases. The purpose of this library is to develop a simple class that returns information about a given file. The class shall be instantiated with the file path, which can be relative.

The starting point is the class with the method __init__. If you want you can develop the class using TDD, but for the sake of brevity I will not show here all the steps that I followed. This is the set of tests I have in tests/test_fileinfo.py

tests/test_fileinfo.py

from fileinfo.fileinfo import FileInfo


def test_init():
    filename = 'somefile.ext'
    fi = FileInfo(filename)
    assert fi.filename == filename


def test_init_relative():
    filename = 'somefile.ext'
    relative_path = '../{}'.format(filename)
    fi = FileInfo(relative_path)
    assert fi.filename == filename

and this is the code of the class FileInfo in the file fileinfo/fileinfo.py

fileinfo/fileinfo.py

import os


class FileInfo:
    def __init__(self, path):
        self.original_path = path
        self.filename = os.path.basename(path)

Git tag: first-version

As you can see the class is extremely simple, and the tests are straightforward. So far I didn't add anything new to what we discussed in the previous posts.

Now I want the method get_info to return a tuple with the file name, the original path the class was instantiated with, and the absolute path of the file. Pretending we are in the directory /some/absolute/path, the class should work as shown here

>>> fi = FileInfo('../book_list.txt')
>>> fi.get_info()
('book_list.txt', '../book_list.txt', '/some/absolute')

You can quickly realise that you have a problem writing the test. There is no way to easily test something as "the absolute path", since the outcome of the function called in the test is supposed to vary with the path of the test itself. Let us try to write part of the test

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)
    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, '???')

where the '???' string highlights that I cannot put something sensible to test the absolute path of the file.

Patching is the way to solve this problem. You know that the function will use some code to get the absolute path of the file. So, within the scope of this test only, you can replace that code with something different and perform the test. Since the replacement code has a known outcome writing the test is now possible.

Patching, thus, means to inform Python that during the execution of a specific portion of the code you want a globally accessible module/object replaced by a mock. Let's see how we can use it in our example

tests/test_fileinfo.py

from unittest.mock import patch

[...]

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath
        fi = FileInfo(original_path)
        assert fi.get_info() == (filename, original_path, test_abspath)

You clearly see the context in which the patching happens, as it is enclosed in a with statement. Inside this statement the module os.path.abspath will be replaced by a mock created by the function patch and called abspath_mock. So, while Python executes the lines of code enclosed by the statement with any call to os.path.abspath will return the object abspath_mock.

The first thing we can do, then, is to give the mock a known return_value. This way we solve the issue that we had with the initial code, that is using an external component that returns an unpredictable result. The line

tests/test_fileinfo.py

from unittest.mock import patch

[...]

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath
        fi = FileInfo(original_path)
        assert fi.get_info() == (filename, original_path, test_abspath)

instructs the patching mock to return the given string as a result, regardless of the real values of the file under consideration.

The code that make the test pass is

fileinfo/fileinfo.py

class FileInfo:
    [...]

    def get_info(self):
        return (
            self.filename,
            self.original_path,
            os.path.abspath(self.original_path)
        )

When this code is executed by the test the function os.path.abspath is replaced at run time by the mock that we prepared there, which basically ignores the input value self.original_path and returns the fixed value it was instructed to use.

Git tag: patch-with-context-manager

It is worth at this point discussing outgoing messages again. The code that we are considering here is a clear example of an outgoing query, as the method get_info is not interested in changing the status of the external component. In the previous post we reached the conclusion that testing the return value of outgoing queries is pointless and should be avoided. With patch we are replacing the external component with something that we know, using it to test that our object correctly handles the value returned by the outgoing query. We are thus not testing the external component, as it has been replaced, and we are definitely not testing the mock, as its return value is already known.

Obviously to write the test you have to know that you are going to use the function os.path.abspath, so patching is somehow a "less pure" practice in TDD. In pure OOP/TDD you are only concerned with the external behaviour of the object, and not with its internal structure. This example, however, shows that this pure approach has some limitations that you have to cope with, and patching is a clean way to do it.

The patching decorator¶

The function patch we imported from the module unittest.mock is very powerful, as it can temporarily replace an external object. If the replacement has to or can be active for the whole test, there is a cleaner way to inject your mocks, which is to use patch as a function decorator.

This means that you can decorate the test function, passing as argument the same argument you would pass if patch was used in a with statement. This requires however a small change in the test function prototype, as it has to receive an additional argument, which will become the mock.

Let's change test_get_info, removing the statement with and decorating the function with patch

tests/test_fileinfo.py

@patch('os.path.abspath')
def test_get_info(abspath_mock):
    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath

    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath)

Git tag: patch-with-function-decorator

As you can see the decorator patch works like a big with statement for the whole function. The argument abspath_mock passed to the test becomes internally the mock that replaces os.path.abspath. Obviously this way you replace os.path.abspath for the whole function, so you have to decide case by case which form of the function patch you need to use.

Multiple patches¶

You can patch more that one object in the same test. For example, consider the case where the method get_info calls os.path.getsize in addition to os.path.abspath in order to return the size of the file. You have at this point two different outgoing queries, and you have to replace both with mocks to make your class work during the test.

This can be easily done with an additional patch decorator

tests/test_fileinfo.py

@patch('os.path.getsize')
@patch('os.path.abspath')
def test_get_info(abspath_mock, getsize_mock):
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath

    test_size = 1234
    getsize_mock.return_value = test_size

    fi = FileInfo(original_path)
    assert fi.get_info() == (filename, original_path, test_abspath, test_size)

Please note that the decorator which is nearest to the function is applied first. Always remember that the decorator syntax with @ is a shortcut to replace the function with the output of the decorator, so two decorators result in

@decorator1
@decorator2
def myfunction():
    pass

which is a shorcut for

def myfunction():
    pass
myfunction = decorator1(decorator2(myfunction))

This explains why, in the test code, the function receives first abspath_mock and then getsize_mock. The first decorator applied to the function is the patch of os.path.abspath, which appends the mock that we call abspath_mock. Then the patch of os.path.getsize is applied and this appends its own mock.

The code that makes the test pass is

fileinfo/fileinfo.py

class FileInfo:
    [...]

    def get_info(self):
        return (
            self.filename,
            self.original_path,
            os.path.abspath(self.original_path),
            os.path.getsize(self.original_path)
        )

Git tag: multiple-patches

We can write the above test using two with statements as well

tests/test_fileinfo.py

def test_get_info():
    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    with patch('os.path.abspath') as abspath_mock:
        test_abspath = 'some/abs/path'
        abspath_mock.return_value = test_abspath

        with patch('os.path.getsize') as getsize_mock:
            test_size = 1234
            getsize_mock.return_value = test_size

            fi = FileInfo(original_path)
            assert fi.get_info() == (
                filename,
                original_path,
                test_abspath,
                test_size
            )

Using more than one with statement, however, makes the code difficult to read, in my opinion, so in general I prefer to avoid complex with trees if I do not really need to use a limited scope of the patching.

Checking call parameters¶

When you patch, your internal algorithm is not executed, as the patched method just return the values it has been instructed to return. This is connected to what we said about testing external systems, so everything is good, but while we don't want to test the internals of the module os.path, we want to be sure that we are passing the correct values to the external methods.

This is why mocks provide methods like assert_called_with (and other similar methods), through which we can check the values passed to a patched method when it is called. Let's add the checks to the test

tests/test_fileinfo.py

@patch('os.path.getsize')
@patch('os.path.abspath')
def test_get_info(abspath_mock, getsize_mock):
    test_abspath = 'some/abs/path'
    abspath_mock.return_value = test_abspath

    filename = 'somefile.ext'
    original_path = '../{}'.format(filename)

    test_size = 1234
    getsize_mock.return_value = test_size

    fi = FileInfo(original_path)
    info = fi.get_info() 

    abspath_mock.assert_called_with(original_path)
    getsize_mock.assert_called_with(original_path)
    assert info == (filename, original_path, test_abspath, test_size)

As you can see, I first invoke fi.get_info storing the result in the variable info, check that the patched methods have been called witht the correct parameters, and then assert the format of its output.

The test passes, confirming that we are passing the correct values.

Git tag: addding-checks-for-input-values

Patching immutable objects¶

The most widespread version of Python is CPython, which is written, as the name suggests, in C. Part of the standard library is also written in C, while the rest is written in Python itself.

The objects (classes, modules, functions, etc.) that are implemented in C are shared between interpreters, and this requires those objects to be immutable, so that you cannot alter them at runtime from a single interpreter.

An example of this immutability can be given easily using a Python console

>>> a = 1
>>> a.conjugate = 5
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object attribute 'conjugate' is read-only

Here I'm trying to replace a method with an integer, which is pointless per se, but clearly shows the issue we are facing.

What has this immutability to do with patching? What patch does is actually to temporarily replace an attribute of an object (method of a class, class of a module, etc.), which also means that if we try to replace an attribute in an immutable object the patching action will fail.

A typical example of this problem is the module datetime, which is also one of the best candidates for patching, since the output of time functions is by definition time-varying.

Let me show the problem with a simple class that logs operations. I will temporarily break the TDD methodology writing first the class and then the tests, so that you can appreciate the problem.

Create a file called logger.py and put there the following code

fileinfo/logger.py

import datetime


class Logger:
    def __init__(self):
        self.messages = []

    def log(self, message):
        self.messages.append((datetime.datetime.now(), message))

This is pretty simple, but testing this code is problematic, because the method log produces results that depend on the actual execution time. The call to datetime.datetime.now is however an outgoing query, and as such it can be replaced by a mock with patch.

If we try to do it, however, we will have a bitter surprise. This is the test code, that you can put in tests/test_logger.py

tests/test_logger.py

from unittest.mock import patch

from fileinfo.logger import Logger


@patch('datetime.datetime.now')
def test_log(mock_now):
    test_now = 123
    test_message = "A test message"
    mock_now.return_value = test_now

    test_logger = Logger()
    test_logger.log(test_message)
    assert test_logger.messages == [(test_now, test_message)]

When you try to execute this test you will get the following error

TypeError: can't set attributes of built-in/extension type 'datetime.datetime'

which is raised because patching tries to replace the function now in datetime.datetime with a mock, and since the module is immutable this operation fails.

Git tag: initial-logger-not-working

There are several ways to address this problem. All of them, however, start from the fact that importing or subclassing an immutable object gives you a mutable "copy" of that object.

The easiest example in this case is the module datetime itself. In the function test_log we tried to patch directly the object datetime.datetime.now, affecting the builtin module datetime. The file logger.py, however, does import datetime, so this latter becomes a local symbol in the module logger. This is exactly the key for our patching. Let us change the code to

tests/test_logger.py

@patch('fileinfo.logger.datetime.datetime')
def test_log(mock_datetime):
    test_now = 123
    test_message = "A test message"
    mock_datetime.now.return_value = test_now

    test_logger = Logger()
    test_logger.log(test_message)
    assert test_logger.messages == [(test_now, test_message)]

Git tag: correct-patching

If you run the test now, you can see that the patching works. What we did was to inject our mock in fileinfo.logger.datetime.datetime instead of datetime.datetime.now. Two things changed, thus, in our test. First, we are patching the module imported in the file logger.py and not the module provided globally by the Python interpreter. Second, we have to patch the whole module because this is what is imported by the file logger.py. If you try to patch fileinfo.logger.datetime.datetime.now you will find that it is still immutable.

Another possible solution to this problem is to create a function that invokes the immutable object and returns its value. This last function can be easily patched, because it just uses the builtin objects and thus is not immutable. This solution, however, requires changing the source code to allow testing, which is far from being optimal. Obviously it is better to introduce a small change in the code and have it tested than to leave it untested, but whenever is possible I try as much as possible to avoid solutions that introduce code which wouldn't be required without tests.

Mocks and proper TDD¶

Following a strict TDD methodology means writing a test before writing the code that passes that test. This can be done because we use the object under test as a black box, interacting with it through its API, and thus not knowing anything of its internal structure.

When we mock systems we break this assumption. In particular we need to open the black box every time we need to patch an hardcoded external system. Let's say, for example, that the object under test creates a temporary directory to perform some data processing. This is a detail of the implementation and we are not supposed to know it while testing the object, but since we need to mock the file creation to avoid interaction with the external system (storage) we need to become aware of what happens internally.

This also means that writing a test for the object before writing the implementation of the object itself is difficult. Pretty often, thus, such objects are built with TDD but iteratively, where mocks are introduced after the code has been written.

While this is a violation of the strict TDD methodology, I don't consider it a bad practice. TDD helps us to write better code consistently, but good code can be written even without tests. The real outcome of TDD is a test suite that is capable of detecting regressions or the removal of important features in the future. This means that breaking strict TDD for a small part of the code (patching objects) will not affect the real result of the process, only change the way we achieve it.

A warning¶

Mocks are a good way to approach parts of the system that are not under test but that are still part of the code that we are running. This is particularly true for parts of the code that we wrote, which internal structure is ultimately known. When the external system is complex and completely detached from our code, mocking starts to become complicated and the risk is that we spend more time faking parts of the system than actually writing code.

In this cases we definitely crossed the barrier between unit testing and integration testing. You may see mocks as the bridge between the two, as they allow you to keep unit-testing parts that are naturally connected ("integrated") with external systems, but there is a point where you need to recognise that you need to change approach.

This threshold is not fixed, and I can't give you a rule to recognise it, but I can give you some advice. First of all keep an eye on how many things you need to mock to make a test run, as an increasing number of mocks in a single test is definitely a sign of something wrong in the testing approach. My rule of thumb is that when I have to create more than 3 mocks, an alarm goes off in my mind and I start questioning what I am doing.

The second advice is to always consider the complexity of the mocks. You may find yourself patching a class but then having to create monsters like cls_mock().func1().func2().func3.assert_called_with(x=42) which is a sign that the part of the system that you are mocking is deep into some code that you cannot really access, because you don't know it's internal mechanisms.

The third advice is to consider mocks as "hooks" that you throw at the external system, and that break its hull to reach its internal structure. These hooks are obviously against the assumption that we can interact with a system knowing only its external behaviour, or its API. As such, you should keep in mind that each mock you create is a step back from this perfect assumption, thus "breaking the spell" of the decoupled interaction. Doing this makes it increasingly complex to create mocks, and this will contribute to keep you aware of what you are doing (or overdoing).

Final words¶

Mocks are a very powerful tool that allows us to test code that contains outgoing messages. In particular they allow us to test the arguments of outgoing commands. Patching is a good way to overcome the fact that some external components are hardcoded in our code and are thus unreachable through the arguments passed to the classes or the methods under analysis.

Updates¶

2021-03-06 GitHub user 4myhw spotted an inconsistency between the code on GitHub and the code in the post. Thanks!

2022-11-19 GitHub user rioj7 found and corrected a typo. Thanks!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

TDD in Python with pytest - Part 4

2020-09-17T11:30:00+02:00

This is the fourth post in the series "TDD in Python with pytest" where I develop a simple project following a strict TDD methodology. The posts come from my book Clean Architectures in Python and have been reviewed to get rid of some bad naming choices of the version published in the book.

You can find the first post here.

In this post I will discuss a very interesting and useful testing tool: mocks.

Basic concepts¶

As we saw in the previous post the relationship between the component that we are testing and other components of the system can be complex. Sometimes idempotency and isolation are not easy to achieve, and testing outgoing commands requires to check the parameters sent to the external component, which is not trivial.

The main difficulty comes from the fact that your code is actually using the external system. When you run it in production the external system will provide the data that your code needs and the whole process can work as intended. During testing, however, you don't want to be bound to the external system, for the reasons explained in the previous post, but at the same time you need it to make your code work.

So, you face a complex issue. On the one hand your code is connected to the external system (be it hardcoded or chosen programmatically), but on the other hand you want it to run without the external system being active (or even present).

This problem can be solved with the use of mocks. A mock, in the testing jargon, is an object that simulates the behaviour of another (more complex) object. Wherever your code connects to an external system, during testing you can replace the latter with a mock, pretending the external system is there and properly checking that your component behaves like intended.

First steps¶

Let us try and work with a mock in Python and see what it can do. First of all fire up a Python shell and import the library

>>> from unittest import mock

The main object that the library provides is Mock and you can instantiate it without any argument

>>> m = mock.Mock()

This object has the peculiar property of creating methods and attributes on the fly when you require them. Let us first look inside the object to get an idea of what it provides

>>> dir(m)
[
    'assert_any_call', 'assert_called_once_with',
    'assert_called_with', 'assert_has_calls',
    'attach_mock', 'call_args', 'call_args_list',
    'call_count', 'called', 'configure_mock',
    'method_calls', 'mock_add_spec', 'mock_calls',
    'reset_mock', 'return_value', 'side_effect'
]

As you can see there are some methods which are already defined into the object Mock. Let's try to read a non-existent attribute

>>> m.some_attribute
<Mock name='mock.some_attribute' id='140222043808432'>
>>> dir(m)
[
    'assert_any_call', 'assert_called_once_with',
    'assert_called_with', 'assert_has_calls',
    'attach_mock', 'call_args', 'call_args_list',
    'call_count', 'called', 'configure_mock',
    'method_calls', 'mock_add_spec', 'mock_calls',
    'reset_mock', 'return_value', 'side_effect',
    'some_attribute'
]

As you can see this class is somehow different from what you are used to. First of all, its instances do not raise an AttributeError when asked for a non-existent attribute, but they happily return another instance of Mock itself. Second, the attribute you tried to access has now been created inside the object and accessing it returns the same mock object as before.

>>> m.some_attribute
<Mock name='mock.some_attribute' id='140222043808432'>

Mock objects are callables, which means that they may act both as attributes and as methods. If you try to call the mock, it just returns another mock with a name that includes parentheses to signal its callable nature

>>> m.some_attribute()
<Mock name='mock.some_attribute()' id='140247621475856'>

As you can understand, such objects are the perfect tool to mimic other objects or systems, since they may expose any API without raising exceptions. To use them in tests, however, we need them to behave just like the original, which implies returning sensible values or performing real operations.

Simple return values¶

The simplest thing a mock can do for you is to return a given value every time you call one of its methods. This is configured setting the attribute return_value of a mock object

>>> m.some_attribute.return_value = 42
>>> m.some_attribute()
42

Now, as you can see the object does not return a mock object any more, instead it just returns the static value stored in the attribute return_value. Since in Python everything is an object you can return here any type of value: simple types like an integer of a string, more complex structures like dictionaries or lists, classes that you defined, instances of those, or functions.

Pay attention that what the mock returns is exactly the object that it is instructed to use as return value. If the return value is a callable such as a function, calling the mock will return the function itself and not the result of the function. Let me give you an example

>>> def print_answer():
...  print("42")
... 
>>> 
>>> m.some_attribute.return_value = print_answer
>>> m.some_attribute()
<function print_answer at 0x7f8df1e3f400>

As you can see calling some_attribute just returns the value stored in return_value, that is the function itself. This is not exactly what we were aiming for. To make the mock call the object that we use as a return value we have to use a slightly more complex attribute called side_effect.

Complex return values¶

The side_effect parameter of mock objects is a very powerful tool. It accepts three different flavours of objects: callables, iterables, and exceptions, and changes its behaviour accordingly.

If you pass an exception the mock will raise it

>>> m.some_attribute.side_effect = ValueError('A custom value error')
>>> m.some_attribute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/unittest/mock.py", line 939, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.6/unittest/mock.py", line 995, in _mock_call
    raise effect
ValueError: A custom value error

If you pass an iterable, such as for example a generator, a plain list, tuple, or similar objects, the mock will yield the values of that iterable, i.e. return every value contained in the iterable on subsequent calls of the mock.

>>> m.some_attribute.side_effect = range(3)
>>> m.some_attribute()
0
>>> m.some_attribute()
1
>>> m.some_attribute()
2
>>> m.some_attribute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/unittest/mock.py", line 939, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/usr/lib/python3.6/unittest/mock.py", line 998, in _mock_call
    result = next(effect)
StopIteration

As promised, the mock just returns every object found in the iterable (in this case a range object) one at a time until the generator is exhausted. According to the iterator protocol once every item has been returned the object raises the StopIteration exception, which means that you can safely use it in a loop.

Last, if you feed side_effect a callable, the latter will be executed with the parameters passed when calling the attribute. Let's consider again the simple example given in the previous section

>>> def print_answer():
...     print("42")       
>>> m.some_attribute.side_effect = print_answer
>>> m.some_attribute()
42

A slightly more complex example is that of a function with arguments

>>> def print_number(num):
...     print("Number:", num)
... 
>>> m.some_attribute.side_effect = print_number
>>> m.some_attribute(5)
Number: 5

As you can see the arguments passed to the attribute are directly used as arguments for the stored function. This is very powerful, especially if you stop thinking about "functions" and start considering "callables". Indeed, given the nature of Python objects we know that instantiating an object is not different from calling a function, which means that side_effect can be given a class and return a instance of it

>>> class Number:
...     def __init__(self, value):
...         self._value = value
...     def print_value(self):
...         print("Value:", self._value)
... 
>>> m.some_attribute.side_effect = Number
>>> n = m.some_attribute(26)
>>> n
<__main__.Number object at 0x7f8df1aa4470>
>>> n.print_value()
Value: 26

Asserting calls¶

As I explained in the previous post outgoing commands shall be tested checking the correctness of the message argument. This can be easily done with mocks, as these objects record every call that they receive and the arguments passed to it.

Let's see a practical example

from unittest import mock
import myobj


def test_connect():
    external_obj = mock.Mock()

    myobj.MyObj(external_obj)

    external_obj.connect.assert_called_with()

Here, the class myobj.MyObj needs to connect to an external object, for example a remote repository or a database. The only thing we need to know for testing purposes is if the class called the method connect of the external object without any parameter.

So the first thing we do in this test is to instantiate the mock object. This is a fake version of the external object, and its only purpose is to accept calls from the object MyObj under test and possibly return sensible values. Then we instantiate the class MyObj passing the external object. We expect the class to call the method connect so we express this expectation calling external_obj.connect.assert_called_with.

What happens behind the scenes? The class MyObj receives the fake external object and somewhere in its initialization process calls the method connect of the mock object. This call creates the method itself as a mock object. This new mock records the parameters used to call it and the subsequent call to its method assert_called_with checks that the method was called and that no parameters were passed.

In this case an object like

class MyObj():
    def __init__(self, repo):
        repo.connect()

would pass the test, as the object passed as repo is a mock that does nothing but record the calls. As you can see, the method __init__ actually calls repo.connect, and repo is expected to be a full-featured external object that provides connect in its API. Calling repo.connect when repo is a mock object, instead, silently creates the method (as another mock object) and records that the method has been called once without arguments.

The method assert_called_with allows us to also check the parameters we passed when calling. To show this let us pretend that we expect the method MyObj.setup to call setup(cache=True, max_connections=256) on the external object. Remember that this is an outgoing command, so we are interested in checking the parameters and not the result.

The new test can be something like

def test_setup():
    external_obj = mock.Mock()
    obj = myobj.MyObj(external_obj)
    obj.setup()
    external_obj.setup.assert_called_with(cache=True, max_connections=256)

In this case an object that passes the test can be

class MyObj():
    def __init__(self, repo):
        self._repo = repo
        repo.connect()

    def setup(self):
        self._repo.setup(cache=True, max_connections=256)

If we change the method setup to

    def setup(self):
        self._repo.setup(cache=True)

the test will fail with the following error

E           AssertionError: Expected call: setup(cache=True, max_connections=256)
E           Actual call: setup(cache=True)

Which I consider a very clear explanation of what went wrong during the test execution.

As you can read in the official documentation, the object Mock provides other methods and attributes, like assert_called_once_with, assert_any_call, assert_has_calls, assert_not_called, called, call_count, and many others. Each of those explores a different aspect of the mock behaviour concerning calls. Make sure to read their description and go through the examples.

A simple example¶

To learn how to use mocks in a practical case, let's work together on a new module in the simple_calculator package. The target is to write a class that downloads a JSON file with data on meteorites and computes some statistics on the dataset using the class SimpleCalculator. The file is provided by NASA at this URL.

The class contains a method get_data that queries the remote server and returns the data, and a method average_mass that uses the method SimpleCalculator.avg to compute the average mass of the meteorites and return it. In a real world case, like for example in a scientific application, I would probably split the class in two. One class manages the data, updating it whenever it is necessary, and another one manages the statistics. For the sake of simplicity, however, I will keep the two functionalities together in this example.

Let's see a quick example of what is supposed to happen inside our code. An excerpt of the file provided from the server is

[
    {
        "fall": "Fell",
        "geolocation": {
            "type": "Point",
            "coordinates": [6.08333, 50.775]
        },
        "id":"1",
        "mass":"21",
        "name":"Aachen",
        "nametype":"Valid",
        "recclass":"L5",
        "reclat":"50.775000",
        "reclong":"6.083330",
        "year":"1880-01-01T00:00:00.000"
    },
    {
        "fall": "Fell",
        "geolocation": {
            "type": "Point",
            "coordinates": [10.23333, 56.18333]
        },
        "id":"2",
        "mass":"720",
        "name":"Aarhus",
        "nametype":"Valid",
        "recclass":"H6",
        "reclat":"56.183330",
        "reclong":"10.233330",
        "year":"1951-01-01T00:00:00.000"
    }
]

So a good way to compute the average mass of the meteorites is

import urllib.request
import json

from simple_calculator.main import SimpleCalculator

URL = ("https://data.nasa.gov/resource/y77d-th95.json")

with urllib.request.urlopen(URL) as url:
    data = json.loads(url.read().decode())

masses = [float(d['mass']) for d in data if 'mass' in d]

print(masses)

calculator = SimpleCalculator()

avg_mass = calculator.avg(masses)

print(avg_mass)

Where the list comprehension filters out those elements which do not have a attribute mass. This code returns the value 50190.19568930039, so that is the average mass of the meteorites contained in the file.

Now we have a proof of concept of the algorithm, so we can start writing the tests. We might initially come up with a simple solution like

def test_average_mass():
    metstats = MeteoriteStats()

    data = metstats.get_data()

    assert metstats.average_mass(data) == 50190.19568930039

This little test contains, however, two big issues. First of all the method get_data is supposed to use the Internet connection to get the data from the server. This is a typical example of an outgoing query, as we are not trying to change the state of the web server providing the data. You already know that you should not test the return value of an outgoing query, but you can see here why you shouldn't use real data when testing either. The data coming from the server can change in time, and this can invalidate your tests.

Testing such a case becomes very simple with mocks. Since the class has a public method get_data that interacts with the external component, it is enough to temporarily replace it with a mock that provides sensible values. Create the file tests/test_meteorites.py and put this code in it

tests/test_meteorites.py

from unittest import mock

from simple_calculator.meteorites import MeteoriteStats


def test_average_mass():
    metstats = MeteoriteStats()

    metstats.get_data = mock.Mock()
    metstats.get_data.return_value = [
        {
            "fall": "Fell",
            "geolocation": {
                "type": "Point",
                "coordinates": [6.08333, 50.775]
            },
            "id":"1",
            "mass":"21",
            "name":"Aachen",
            "nametype":"Valid",
            "recclass":"L5",
            "reclat":"50.775000",
            "reclong":"6.083330",
            "year":"1880-01-01T00:00:00.000"},
        {
            "fall": "Fell",
            "geolocation": {
                "type": "Point",
                "coordinates": [10.23333, 56.18333]
            },
            "id":"2",
            "mass":"720",
            "name":"Aarhus",
            "nametype":"Valid",
            "recclass":"H6",
            "reclat":"56.183330",
            "reclong":"10.233330",
            "year":"1951-01-01T00:00:00.000"
        }
    ]

    result = metstats.average_mass(metstats.get_data())

    assert result == 370.5

When we run this test we are not testing that the external server provides the correct data. We are testing the process implemented by average_mass, feeding the algorithm some known input. This is not different from the first tests that we implemented: in that case we were testing an addition, here we are testing a more complex algorithm, but the concept is the same.

We can now write a class that passes this test. Put the following code in simple_calculator/meteorites.py alongside with main.py

simple_calculator/meteorites.py

import urllib.request
import json

from simple_calculator.main import SimpleCalculator

URL = ("https://data.nasa.gov/resource/y77d-th95.json")


class MeteoriteStats:
    def get_data(self):
        with urllib.request.urlopen(URL) as url:
            return json.loads(url.read().decode())

    def average_mass(self, data):
        calculator = SimpleCalculator()

        masses = [float(d['mass']) for d in data if 'mass' in d]

        return calculator.avg(masses)

As you can see the class contains the code we wrote as a proof of concept, slightly reworked to match the methods we used in the test. Run the test suite now, and you will see that the latest test we wrote passes.

Please note that we are not testing the method get_data. That method uses the function urllib.request.urlopen that opens an Internet connection without passing through any other public object that we can replace at run time during the test. We need then a tool to replace internal parts of our objects when we run them, and this is provided by patching, which will be the topic of the next post.

Git tag: meteoritestats-class-added

Final words¶

Mocks are very important, and as a Python programmer you need to know the subtleties of their implementation. Aside from the technical details, however, I believe it is mandatory to master the different types of tests that I discussed in the previous post, and to learn when to use simple assertions and when to pull a bigger gun like a mock object.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

TDD in Python with pytest - Part 3

2020-09-15T08:00:00+02:00

This is the third post in the series "TDD in Python from scratch" where I develop a simple project following a strict TDD methodology. The posts come from my book Clean Architectures in Python and have been reviewed to get rid of some bad naming choices of the version published in the book.

What I introduced in the previous two posts is commonly called "unit testing", since it focuses on testing a single and very small unit of code. As simple as it may seem, the TDD process has some caveats that are worth being discussed. In this chapter I discuss some aspects of TDD and unit testing that I consider extremely important.

Tests should be fast¶

You will run your tests many times, potentially you should run them every time you save your code. Your tests are the watchdogs of your code, the dashboard warning lights that signal a correct status or some malfunction. This means that your testing suite should be fast. If you have to wait minutes for each execution to finish, chances are that you will end up running your tests only after some long coding session, which means that you are not using them as guides.

It's true however that some tests may be intrinsically slow, or that the test suite might be so big that running it would take an amount of time which makes continuous testing uncomfortable. In this case you should identify a subset of tests that run quickly and that can show you if something is not working properly, the so-called "smoke tests", and leave the rest of the suite for longer executions that you run less frequently. Typically, the library part of your project has tests that run very quickly, as testing functions does not require specific set-ups, while the user interface tests (be it a CLI or a GUI) are usually slower. If your tests are well-structured you can also run just the tests that are connected with the subsystem that you are dealing with.

Tests should be idempotent¶

Idempotency in mathematics and computer science identifies processes that can be run multiple times without changing the status of the system. Since this latter doesn't change, the tests can be run in whichever order without changing their results. If a test interacts with an external system leaving it in a different state you will have random failures depending on the execution order.

The typical example is when you interact with the filesystem in your tests. A test may create a file and not remove it, and this makes another test fail because the file already exists, or because the directory is not empty. Whatever you do while interacting with external systems has to be reverted after the test. If you run your tests concurrently, however, even this precaution is not enough.

This poses a big problem, as interacting with external systems is definitely to be considered dangerous. Mocks, introduced in the next chapter, are a very good tool to deal with this aspect of testing.

Tests should be isolated¶

In computer science isolation means that a component shall not change its behaviour depending on something that happens externally. In particular it shouldn't be affected by the execution of other components in the system (spatial isolation) and by the previous execution of the component itself (temporal isolation). Each test should run as much as possible in an isolated universe.

While this is easy to achieve for small components, like we did with the class SimpleCalculator, it might be almost impossible to do in more complex cases. Whenever you write a routine that deals with time, for example, be it the current date or a time interval, you are faced with something that flows incessantly and that cannot be stopped or slowed down. This is also true in other cases, for example if you are testing a routine that accesses an external service like a website. If the website is not reachable the test will fail, but this failure comes from an external source, not from the code under test.

Mocks or fake objects are a good tool to enforce isolation in tests that need to communicate with external actors in the system.

External systems¶

It is important to understand that the above definitions (idempotency, isolation) depend on the scope of the test. You should consider external whatever part of the system is not directly involved in the test, even though you need to use it to run the test itself. You should also try to reduce the scope of the test as much as possible.

Let me give you an example. Consider a web application and imagine a test that checks that a user can log in. The login process involves many layers: the user inputs, the username and the password in a GUI and submits the form, the GUI communicates with the core of the application that finds the user in the DB and checks the password hash against the one stored there, then sends back a message that grants access to the user, and the GUI stores a cookie to keep the user logged in. Suppose now that the test fails. Where is the error? Is it in the query that retrieves the user from the DB? Or in the routine that hashes the password? Or is it just an issue in the connectivity between the application and the database?

As you can see there are too many possible points of failure. While this is a perfectly valid integration test, it is definitely not a unit test. Unit tests try to test the smallest possible units of code in your system, usually simple routines like functions or object methods. Integration tests, instead, put together whole systems that have already been tested and test that they can work together.

Too many times developers confuse integration tests with unit tests. One simple example: every time a web framework makes you test your models against a real database you are mixing a unit test (the methods of the model object work) with an integration one (the model object connects with the database and can store/retrieve data). You have to learn how to properly identify what is external to your system in the scope of a given test, so your tests can be focused and small.

Focus on messages¶

I will never recommend enough Sandi Metz's talk "The Magic Tricks of Testing" where she considers the different messages that a software component has to deal with. She comes up with 3 different origins for messages (incoming, sent to self, and outgoing) and 2 types (query and command). The very interesting conclusion she reaches is that you should only test half of them, and I believe this is one of the most useful results you can learn as a software developer. In this section I will shamelessly start from Sandi Metz's categorisations and give a personal view of the matter. I absolutely recommend to watch the original talk as it is both short and very effective.

Testing is all about the behaviour of a component when it is used, i.e. when it is connected to other components that interact with it. This interaction is well represented by the word "message", which has hereafter the simple meaning of "data exchanged between two actors".

We can then classify the interactions happening in our system, and thus to our components, by flow and by type (Sandi Metz speaks of origin and type).

Message flow¶

The flow is defined as the tuple (source, destination), that is where the message comes from and what is its destination. There are three different combinations that we are interested in: (outside, self), (self, self), and (self, outside), where self is the object we are testing, and outside is a generic object that lives in the system. There is a fourth combination, (outside, outside) that is not relevant for the testing, since it doesn't involve the object under analysis.

So (outside, self) contains all the messages that other parts of the system send to our component. These messages correspond to the public API of the component, that is the set of entry points the component makes available to interact with it. Notable examples are the public methods of an object in an object-oriented programming language or the HTTP endpoints of a Web application. This flow represents the incoming messages.

At the opposite side of the spectrum there is (self, outside), which is the set of messages that the component under test sends to other parts of the system. These are for example the external calls that an object does to a library or to other objects, or the API of other applications we rely on, like databases or Web applications. This flow describes all the outgoing messages.

Between the two there is (self, self), which identifies the messages that the component sends to itself, i.e. the use that the component does of its own internal API. This can be the set of private methods of an object or the business logic inside a Web application. The important thing about this last case is that while the component is seen as a black box by the rest of the system it actually has an internal structure and it uses it to run. This flow contains all the private messages.

Message type¶

Messages can be further divided according to the interaction the source requires to have with the target: queries and commands. Queries are messages that do not change the status of the component, they just extract information. The class SimpleCalculator that we developed in the previous section is a typical example of object that exposes query methods. Adding two numbers doesn't change the status of the object, and you will receive the same answer every time you call the method add.

Commands are the opposite. They do not extract any information, but they change the status of the object. A method of an object that increases an internal counter or a method that adds values to an array are perfect examples of commands.

It's perfectly normal to combine a query and a command in a single message, as long as you are aware that your message is changing the status of the component. Remember that changing the status is something that can have concrete secondary effect.

The testing grid¶

Combining 3 flows and 2 message types we get 6 different message cases that involve the component under testing. For each one of this cases we have to decide how to test the interaction represented by that flow and message type.

Incoming queries¶

An incoming query is a message that an external actor sends to get a value from your component. Testing this behaviour is straightforward, as you just need to write a test that sends the message and makes an assertion on the returned value. A concrete example of this is what we did to test the method add of SimpleCalculator.

Incoming commands¶

An incoming command comes from an external actor that wants to change the status of the system. There should be a way for an external actor to check the status, which translates into the need of having either a companion incoming query message that allows to extract the status (or at least the part of the status affected by the command), or the knowledge that the change is going to affect the behaviour of another query. A simple example might be a method that sets the precision (number of digits) of the division in the object SimpleCalculator. Setting that value changes the result of a query, which can be used to test the effect of the incoming command.

Private queries¶

A private query is a message that the component sends to self to get a value without affecting its own state, and it is basically nothing more than an explicit use of some internal logic. This happens often in object-oriented languages because you extracted some common logic from one or more methods of an object and created a private method to avoid duplication.

Since private queries use the internal logic you shouldn't test them. This might be surprising, as private methods are code, and code should be tested, but remember that other methods are calling them, so the effects of that code are not invisible, they are tested by the tests of the public entry points, although indirectly. The only effect you would achieve by testing private methods is to lock the tests to the internal implementation of the component, which by definition shouldn't be used by anyone outside of the component itself. This in turn, makes refactoring painful, because you have to keep redundant tests in sync with the changes that you do, instead of using them as a guide for the code changes like TDD wants you to do.

As Sandi Metz says, however, this is not an inflexible rule. Whenever you see that testing an internal method makes the structure more robust feel free to do it. Be aware that you are locking the implementation, so do it only where it makes a real difference businesswise.

Private commands¶

Private commands shouldn't be treated differently than private queries. They change the status of the component, but this is again part of the internal logic of the component itself, so you shouldn't test private commands either. As stated for private queries, feel free to do it if this makes a real difference.

Outgoing queries and commands¶

An outgoing query is a message that the component under testing sends to an external actor asking for a value, without changing the status of the actor itself. The correctness of the returned value, given the inputs, is not part of what you want to test, because that is an incoming query for the external actor. Let me repeat this: you don't want to test that the external actor return the correct value given some inputs.

This is perhaps one of the biggest mistakes that programmers make when they test their applications. Definitely it is a mistake that I made many times. We tend to introduce tests that, starting from the code of our component, end up testing different components.

Outgoing commands are messages sent to external actors in order to change their state. Since our component sends such messages to cause an effect in another part of the system we have to be sure that the sent values are correct. We do not want to test that the state of the external actor change accordingly, as this is part of the testing suite of the external actor itself (incoming command).

From this consideration it is evident that you shouldn't test the results of any outgoing query or command. Possibly, you should avoid running them at all, otherwise you will need the external system to be up and running when you run the test suite.

We want to be sure, however, that our component uses the API of the external actor in a proper way and the standard technique to test this is to use mocks, that is components that simulate other components. Mocks are an important tool in the TDD methodology and for this reason they are the topic of the next chapter.

| Flow     | Type    | Test? |
|----------|---------|-------|
| Incoming | Query   | Yes   |
| Incoming | Command | Yes   |
| Private  | Query   | Maybe |
| Private  | Command | Maybe |
| Outgoing | Query   | Mock  |
| Outgoing | Command | Mock  |

Final words¶

Since the discovery of TDD few things changed the way I write code more than these considerations on what I am supposed to test. Out of 6 different types of tests we discovered that 2 shouldn't be tested, 2 of them require a very simple technique based on assertions, and the last 2 are the only ones that requires an advanced technique (mocks). This should cheer you up, as for once a good methodology doesn't add new rules and further worries, but removes one third of them, even forbidding you to implement them!

In the next two posts I will discuss mocks and patches, two very important testing tools to have in your belt.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

The Digital Cat - Python

Data Partitioning and Consistent Hashing

Rationale¶

Design choices¶

A practical example of partitioning¶

Hash functions to the rescue¶

Partitioning use cases¶

Load balancers

Caches

Databases

Caching and scaling strategies¶

Scaling out with hash partitioning

A different approach¶

Consistent hashing¶

Consistent hashing and scaling¶

Source code¶

Final words¶

Resources¶

Feedback¶

From Docker CLI to Docker Compose

PostgreSQL¶

Password

Superuser

Database name

Volumes¶

Python application¶

Run containers in the same network¶

Run time configuration¶

Enter Docker Compose¶

Health checks and dependencies¶

Final words¶

Updates¶

Feedback¶

How to write a Pelican theme for your static website

Initial setup¶

Demo content¶

The template¶

Deep dive

Pelican variables¶

Title

Subtitle

Deep dive

Social buttons¶

Articles¶

Pagination¶

Slicing¶

Pictures¶

Advanced techniques: extend and include¶

Extends

Include

Deep dive

Preview articles¶

Next steps¶

Feedback¶

AWS Log Insights as CloudWatch metrics with Python and Terraform

The big picture¶

Terraform¶

Variables

Resources

DynamoDB

Resources

IAM part 1

Resources

Lambda

Resources

IAM part 2

Resources

CloudWatch

Resources

Using the module

Resources

Python¶

Costs¶

Feedback¶

First-class objects in Python - Higher-order functions, wrappers, and factories

Mau: a lightweight markup language

What is Mau?¶

Why not Markdown or AsciiDoc?¶

Pelican¶

Development¶