The Digital Cat - Rust

Call conventions, ownership, and mutability in Rust

2025-01-09T18:00:00+01:00

My roots as a developer are in C and Assembly, and I remember fondly the time spent to learn those languages using Borland TASM or Turbo C under MS-DOS. Those languages, and the technology that surrounded them, forced me to understand the low-level architecture of computers, and I still consider those details extremely interesting.

Rust, just like C and C++, is a low-level language that exposes details of the underlying architecture, and to use it proficiently requires a certain understanding of concepts like memory management that are mostly ignored by high-level language like Python or JavaScript.

This post has been written for beginners who are used to dynamically typed languages. I will explore the intricacies of passing arguments to Rust functions, trying to clarify exactly what happens behind the scenes and why the language provides syntactic devices like & or mut.

For the sake of clarity, I will split the discussion into three separate topics: call conventions, ownership, and mutability. However, it is important to keep in mind that those concepts are not independent, and that the separation is purely a strategy to make them more digestible.

Computer architecture¶

Throughout the article, I will mention and show what happens at CPU level when code is written in a certain way.

You do not need to have previous knowledge of computer architecture to follow those sections, as I will explain what happens step by step, but I highly recommend to eventually become familiar with some concepts like the stack and CPU calling conventions. You will find some useful links in the Resources section at the end of the article.

Assembly and machine code

Formally, machine code is the sequence of binary values that are given to the CPU: everything in machine code is just a binary number. Assembly is a slightly higher-level language that uses mnemonics and register names to make it simpler for humans to read and write machine code.

For all practical purposes, in this article we can assume that Assembly and machine code are the same thing and use the two words interchangeably.

Call conventions¶

When we define a function, we list a set of parameters, that is values that have to be given to the function when we call it. As Rust is a statically typed language, functions need to declare the type of each parameter, and arguments need to match that.

Arguments and parameters

Strictly speaking, arguments and parameters are different. However, this explanation from the Rust book looks like a good informal approach.

We can define functions to have parameters, which are special variables that are part of a function’s signature. When a function has parameters, we can call it providing concrete values for those parameters. Technically, the concrete values are called arguments, but in casual conversation, people tend to use the words parameter and argument interchangeably for either the variables in a function’s definition or the concrete values passed in when you call a function.

In Rust, there are many built-in types like integers, floats, booleans, and an infinite amount of user-defined types, but when we define function parameters there are two macro categories in which they can fall: values and references.

This is the first and most important concept to learn. We can call a function passing values in only two different ways, regardless of the data type. Depending on the language and the speaker, such techniques are named in different ways, and in this article I will use call by value and call by reference. We call these two different strategies call conventions.

Values and references

Let's review this important distinction between values and references before we jump into the details of function calling.

Consider this situation: I want to give a friend a card for their birthday. I can hand them the card itself, passing the physical object.

Let's consider a different scenario: a friend wants to borrow my car to go shopping. It would be crazy for me to try and get my car to hand it physically. I will probably just tell my friend where the car is parked.

These two examples are exactly what happens when a computer manages data. Ultimately, data is just bytes stored somewhere in memory (typically in RAM, but the same is true for caches or long-term storage like hard drives). When I want to send data to another component in the system I can either give the data itself (value) or tell the component where to find the value (reference).

Memory location

Technically, a memory location is called address. Ultimately, it's just a number that identifies a specific location in memory that hosts data. A pointer or a reference is the type of a variable that contains a memory address. It is usually an unsigned integer whose size depends on the computer architecture. For an Intel x86-64 machine it's a 64 bit number (8 bytes).

Strictly speaking, while C pointers are literally just an integer, Rust references are structures that contain additional data, but for the sake of simplicity in this post we will ignore the difference. If you want to dig into this you can read the documentation.

This means that, strictly speaking, arguments can be passed only by value. References are values (addresses) that are used to find other values.

Why two different ways?

Why do computer provide two ways to pass arguments to functions?

Let's consider a real world example. I won a huge sum of money at the national lottery, and decided to get a safe deposit box at the bank and put the money there. To make sure I don't forget the code to open the safe I write it on a piece of paper. One day I need to get some money from the safe, but I cannot go myself, so I ask a friend to go, open the safe, and get the money for me. To do this, I have to give this person the piece of paper with the code, and they will give it back once they are done. As this happens multiple times, I decide that it might be simpler to copy the code on another piece of paper and to give that to them.

Everything works well, until one day the bank asks us to change the code every time we open the safe, as a security measure. Now, the one who opens the safe has to come up with a new code and update their own piece of paper. This means that the copy owned by the other person is instantly invalidated. Assuming we don't have any other way to communicate the change, we need to review our strategy. In this case there is only one way to deal with the situation. We need a single copy of the piece of paper and each one of us knows where it is hidden. This way, the one who changes the code can go and update the only copy of the note in existence.

At CPU level, when we call a function (the friend) we might need to pass data (the code to open the safe). As long as data is not changed we can safely copy it into the function's memory space (copy the code on another piece of paper), but as soon as the value changes we need a different method that allows the function to change the original value (the only copy of the note). We can pass the location of the data (the place where the note is hidden) so that the function can use it and change it.

In the first two parts of the article we will deal with read-only data, so for a while it will look like there is no need to use references. In the third part, we will introduce mutability that will finally justify them.

Pass by value in Rust

The following code shows how we can pass function arguments by value in Rust

struct Item {
    pub value: u16,
}

fn process(item: Item) -> u16{
    item.value 3
}

fn main() {
    let i = Item { value: 42 }; 1

    process(i); 2
}

As you can see, the code is extremely simple. The function main creates a variable of type Item 1 and calls the function process 2. This in turn extracts the field value 3 and returns it (which is ignored by main).

The function prototype accepts an argument of type Item. This means that the function wants to receive the physical data (the birthday card).

In this case we say that the caller passes the variable i by value.

Behind the scenes

Traditionally, we assume that the compiler will byte copy the value into the function stack, and this is still a valid mental model. However, remember that the compiler's task is to optimise code, so it might decide to do something completely different at machine language level.

We can see what happens in this case using the amazing Compiler Explorer. Please note that to use it you need to make the function main public, as the compiler expects a library, not an executable. See the section "Disassemble Rust" at the end of the post if you want to do it locally.

Using rustc 1.83.0, the code above becomes

example:process:
        mov     ax, di 6
        ret 7

example:main:
        push    rax 1
        mov     edi, 42 3
        call    example:process 5
        pop     rax 2
        ret

For those unfamiliar with (Intel) Assembly, let me clarify what happens here:

First of all, there is no concept of struct in machine code. Here, the compiler figures out there is only one field so the struct i is just the 16 bit value 42.
In example::main the code saves 1 and restores 2 the 64-bit register rax, which is used by convention to store a function's return value (low-level calling convention).
The value of the variable i is stored in the register edi 3. Here, the compiler decided not to use the stack, as the value of the variable is ultimately just a single u16 that can be hosted by a register.
The function example:process is called 5. The code of the function in Rust is very simple, and it needs only to extract the field value. In Assembly, there is no field, and the return value of the function is basically just its input. The function stores the output in the register ax 6 which is the lower 16-bit part of rax.
The function returns 7.

Leaving aside the complexity of the low-level architecture and conventions, the main concept we need to retain is: passing parameters by value copies the value of the arguments.

Again, keep in mind that the compiler writes machine code with optimisation in mind, so what we do in Rust is not always reflected in what happens at CPU level. Later, we will see an example of this.

Pass by reference in Rust

The following code is a slight modification of the previous example and shows how we can pass function arguments by reference in Rust

struct Item {
    pub value: u16,
}

fn process(item: &Item) -> u16{ 1
    item.value
}

pub fn main() {
    let i = Item { value: 42 };

    process(&i); 2
}

As you can see, the only difference is that the function process accepts &Item 1, that is a reference to Item. The function call is modified accordingly, passing &i 2, which is the reference to i.

Here, we are doing what we did when we lent a car. Instead of handing the car itself we tell the recipient where to find it.

Automatic referencing and dereferencing

The code above might surprise those who are used to code in C/C++, as the function process receives &Item but then reads one of the field as if the variable was an Item.

Formally, we should first dereference the pointer, which in Rust can be done with *item and then access the field. Indeed, the following code works

fn process(item: &Item) -> u16{
    (*item).value
}

In C/C++ the syntax (*item).value can be written item->value, but Rust doesn't have the arrow operator.

Rust has a feature called automatic referencing and dereferencing that can automatically add &, &mut, or * to match the required data type. In this case, it automatically transforms item into *item.

It is extremely important to remember that Rust silently adjusts calls. The feature is useful, as it simplifies the syntax of the language, but if we want to understand what happens behind the scenes we need to be aware of this behaviour.

You can read more about automatic referencing and dereferencing in chapter 5.3 of the Rust book.

Behind the scenes

This time, the mental model is that we are handing the function the location of our data, without copying it to new memory locations. Let's see what happens at CPU level.

Again, using the Compiler Explorer and rustc 1.83.0, the code above becomes

example:process:
        mov     ax, word ptr [rdi] 6
        ret 7

example:main:
        push    rax 1
        mov     word ptr [rsp + 6], 42 3
        lea     rdi, [rsp + 6] 4
        call    example:process 5
        pop     rax 2
        ret

The machine code is clearly different from the previous version. Let's have a deep look:

As happened before, the code saves 1 and restores 2 the 64-bit register rax, which is used by convention to store a function's return value (calling convention).
The value 42 is pushed onto the stack 3. This call looks complicated because of byte alignment, but it basically moves the value 42 at the stack address stored into the stack register rsp. Let's decompose it
- In Intel Assembly mov [rsp], 42 would move the value 42 to the address stored in the register rsp. The value is not moved into the register. Rather, the value in the register is used as an address.
- The CPU is very picky when it comes to the size of data that we want to move, so the code clarifies that we want to treat the address as the space that hosts 2 bytes (a word that corresponds to 16 bits) with mov word ptr [rsp], 42.
- Last, the Intel calling convention wants the stack pointer rsp to be constantly aligned to a 16-byte boundary. The function main already pushed rax, which advanced the stack pointer by 8 bytes, and we are going to store 2 bytes (word), so we are missing 6 bytes to keep the stack pointer aligned. This is the reason why the code above uses [rsp + 6] instead of just [rsp].
The address stored at rsp + 6 is loaded into the register rdi 4. The instruction lea (Load Effective Address) calculates the result of [rsp + 6] as an address and loads it into rdi. This is different from what mov would do, which is to copy the value at that address.
As before, the function process is called 5.
The code of the function is different as well because now rdi doesn't contain a value but the address of the value. Thus, the function uses word ptr [rdi] 6 to move the 16 bits value at that address into ax (which is still the conventional register for the return value) and returns 7.

Once again, the Assembly code is complicated because of conventions and low-level architectural details, but the main concept is: passing parameters by address gives the function the location of the data and not the data itself.

The compiler's role

I mentioned several times that the compiler's task is to optimise code, which means that the machine code might or might not correspond to what we wrote in Rust (or any other high level language). A simple example is the following

struct LargeItem {
    value: [u16; 1024 * 1024], 2
}

fn process(item: LargeItem) -> u16 {
    item.value[0]
}

pub fn main() {
    let li = LargeItem {
        value: [0; 1024 * 1024],
    };

    process(li); 1
}

Here, we are passing the variable li by value 1, but the size of the type is rather huge. It's a struct of 2 MiB (2 bytes * 2³²) 2 which cannot be stored into a register. This means that, even though in Rust we pretend we pass the value, the compiler is forced to call the function passing the address. The relevant part of the machine code this time is

example:process:
        mov     ax, word ptr [rdi]
        ret

example:main:
        # Calls to memset and memcpy
        # to set up the large struct
        ...
        lea     rdi, [rsp + 8] 1
        call    example:process
        ...
        ret

where I omitted the rest of the code that deals with the memory initialisation of the large struct.

As you can see, the compiler ignores our "directive" to pass by value and uses the address of the struct 1 as it did with the Rust code that uses references. The function extracts the first value of the array item.value, so the machine code reads the value at the address 2.

The bottom line is: both passing by value and passing by reference can result in the same machine code.

This might sound surprising, but if you think about it, that's exactly the reason why we use a compiler and why it's such a fascinating and important piece of software.

As I mentioned before, the reason why we should pass arguments by reference instead of by value will be clear once we introduce mutability. To get there, however, we first need to discuss another features of the language: ownership.

Ownership¶

This is the second aspect of function calls that I want to discuss, together with call conventions and mutability. Once again, such concepts are interconnected and are separated here just for the sake of clarity.

To discuss ownership, let's once again have a look at real world examples.

In the first scenario, I have a car (once again!) that I don't use any more, so I sell it. From today, I cannot drive that car any more, as it belongs to someone else.

The second situation is: I own a holiday home in a beautiful place, and a friend asked me to use it for a coupe of weeks. The ownership of the house doesn't change, but for a while it will be used by another person.

The third case is similar to a previous example. I annotated the address of a shop on a piece of paper. A friend expresses interest in the same shop, so I copy the note and give it to them. Now, we both have the same information in two different physical locations.

The difference between the cases is clear. Once the car is sold it's not part of my possession any more. I can't drive it, but at the same time I don't have to pay insurance or to deal with it when it's time to demolish it. In the second case, the house is still mine, and I am responsible for council tax and repairs, but for a while it will be given to another. In the third case, each one of us owns their own copy of the information and is responsible for the data.

These three cases can be connected with the way function arguments are treated in Rust. When we pass variables to a function we need to consider the ownership of those variables (actually, the ownership of the data stored in those variables).

In Rust, each argument can be passed in one of the three possible ways:

Copy: we make a copy of value and end up with two owners (like the address of the shop).
Move: we transfer ownership (like selling the car).
Borrow: we lend the value to a function but we want to have it back (like we did with the house).

It is important to understand that ownership is an additional check introduced by the Rust compiler, and that there is no such a thing at CPU level.

Copy: two values and two owners

Let's consider the following code

fn process(value: u16) -> u16 {
    value * 2
}

fn main() {
    let v: u16 = 42; 1

    process(v); 2

    println!("Value {}", v); 3
}

Here, we initialise a value v 1 and we pass it by value to the function process 2.

The value of v is copied into the variable value when the function is called. This means that when we call process there are two independent areas of memory:

The area labelled v which is owned by main.
The area labelled value which is owned by process.

Both areas of memory (variables) contain the same value initially, but they are independent so they can change without affecting each other. We will see an example when we discuss mutability later.

There are two important facts to consider here:

We can use v 3 after we called process. As we copied the value into the function, our variable is still accessible.
The variable has been copied automatically by Rust. This happens because v is a simple type and implements the trait Copy out of the box.

The Copy trait

In Rust, the Copy trait is associated with bitwise copy, that is a trivial memory copy between memory areas. Going back to real world examples we can think of a photocopy of a document, where the two copies are exactly identical at the end of the process.

This is not always what we want and this is where the trait Clone comes into play. I won't discuss it further in this article, make sure you read what the Rust book says about Memory and Allocation.

Some simple types in Rust implement the trait Copy out of the box: integers, floats, booleans, and char are among those. If you are unsure, you can always check the documentation. For example, u32 implements Copy as stated here.

It's important to remember that a struct doesn't implement Copy automatically. This was the case originally, but it was changed around 2014, and you can read a long and detailed explanation in RFC #19.

Move: transfer ownership

What happens when a variable is passed by value and its type doesn't implement the Copy trait? In Rust, the variable is moved to the function and the ownership is transferred (selling the car).

Let's have a look at the following code

struct Item {
    pub value: u16,
}

fn process(item: Item) -> u16 {
    item.value
}

fn main() {
    let i = Item { value: 42 };

    process(i);

    // This fails: value moved to process()
    println!("Item value {}", i.value);
}

The compiler won't accept it and will return the following error

    |
    |     let i = Item { value: 42 };
    |         - move occurs because `i` has type `Item`,
    |           which does not implement the `Copy` trait
    |
    |     process(i);
    |             - value moved here
...
    |     println!("Item value {}", i.value);
    |                               ^^^^^^^ value borrowed
    |                                       here after move
    |

After what we said in the previous sections, I believe the error messages are very clear.

When we called process passing i by value we gave up ownership of the variable, so it is not acceptable to use it after that instruction. We are basically trying to drive the car after we sold it.

Implementing Copy for structs

As you see, the Copy trait is explicitly mentioned. If we implement it for Item we should be able to go back to the previous case, with two copies of the value (i in main and item in process). When we define a struct that contains only types that implement Copy we can ask Rust to implement the trait for us using #[derive(Copy, Clone)].

#[derive(Copy, Clone)]
struct Item {
    pub value: u16,
}

fn process(item: Item) -> u16 {
    item.value
}

fn main() {
    let i = Item { value: 42 };

    process(i);

    // This succeeds: value copied to process()
    println!("Item value {}", i.value);
}

This code compiles as now Item can be copied when passed as an argument to process. The only field of Item is a u16, a type that implements the trait Copy, so it is sufficient to derive the trait to implement it for the structure.

To prove that ownership is a protection mechanism that exists only in Rust, it is interesting to compare the Assembly code for the two cases of a struct that doesn't implement Copy and for one that does. The code is exactly the same for both cases (we saw it in the first example of the article).

Borrow: lend the value to a function

So far we saw two cases that happen when we pass arguments by value. We learned however that we can also pass arguments by reference, so let's have a look at what happens in that case.

struct Item {
    pub value: u16,
}

fn process(item: &Item) -> u16 {
    item.value
}

fn main() {
    let i = Item { value: 42 };

    process(&i);

    println!("Item value {}", i.value);
}

Here, we pass to the function a reference to the value, just like we did with the house in the example. We are basically telling the function that it is allowed to access the value for a while, but that we retain ownership. The code compiles without errors.

This seems to be exactly what happened when we passed by value a type that implements Copy, so what is the difference? As we said before, when data is read-only passing arguments by value and by reference might produce identical results, and using references might look like an unnecessary complication.

Sooner or later, however, we will need to change the values of our variables. Time to discuss mutability.

Mutability¶

In Rust, variables are immutable unless they are declared as mutable. However, as it happened for ownership, it is important to remember that at CPU level everything is mutable.

Mutability, like ownership, is a feature that the compiler introduces to help us to write code that is more correct. Forcing us to declare a variable as mutable gives us the chance to ask ourselves if we need that and to avoid bugs. Mutability and ownership work together to ensure that we write safer code.

While the role of mut in front of variables is usually simple to grasp, mutable references might prove more complicated. For this reason, to discuss mutability we will consider separately the case of arguments passed by value and arguments passed by reference.

Mutability and arguments passed by value

Looking back at two of the real world examples in the previous section, we can quickly understand what happens if we introduce mutability with parameters passed by value.

When I sell the car (pass by value, move) or give away the address of the shop (pass by value, copy), the new owner is free to do whatever they want with the object they receive. They can decide to paint the car, or to burn the note, and those actions are not affecting me at all.

The same happens with Rust variables. If the variable is moved (because it doesn't implement Copy), the ownership is transferred to the function, which is free to do whatever it wants.

struct Item {
    pub value: u16,
}

fn process(mut item: Item) {
    item.value += 1;
}

fn main() {
    let i = Item { value: 42 };

    // Variable moved
    process(i);
}

If the variable is copied (because it implements Copy) the function receives a copy of the value, and once again it's free to do whatever it wants with it.

#[derive(Copy, Clone)]
struct Item {
    pub value: u16,
}

fn process(mut item: Item) {
    item.value += 1;
}

fn main() {
    let i = Item { value: 42 };

    // Variable copied
    process(i);

    // i.value is 42
    println!("Value {}", i.value);
}

Here, we can access i after the call because it implements Copy, but the function changed the value of the copied value and not that of the original variable.

The two examples show what we said at the beginning of the article: passing arguments by value doesn't allow us to change the original variables. In Rust, when a variable is passed by value mutability can be ignored.

Declaring an argument as mutable

There is an important point to clarify. Let's have a look at the following code

fn process(mut value: u16) {
    value += 1;
}

fn main() {
    let i: u16 = 42;

    process(i);
}

As you see, the variable i is passed by value. This means that the function process is free to do whatever it wants with the argument value. However, i is not mutable, so how can the function increment value?

The syntax mut value in the function signature means: create a local mutable variable that will host a value coming from the outside.

This is paramount to understand: mut value doesn't mean that the argument passed in main has to be mutable. It just means that the argument will be hosted by a mutable local variable.

As a matter of fact, the following code compiles

fn process(mut value: u16) {
    value += 1;
}

fn main() {
    let mut i: u16 = 42;

    process(i);
}

But the compiler gives us a warning

warning: variable does not need to be mutable
    |
    |     let mut i: u16 = 42;
    |         ----^
    |         |
    |         help: remove this `mut`

Mutable arguments by reference

When a friend borrows my house, I am expecting them to vacate it after a while, and its management is my responsibility. While they live there, I might not like the fact that they paint the walls or change furniture. I need to be clear if I am giving them an object that they can alter or not.

This is why in Rust we can pass arguments to functions by reference in two different ways. We can pass a normal reference (&) or a mutable reference (&mut).

Mutable references

The name "mutable reference" can be misleading, as &mut is a reference that allows to mutate the referenced variable. The reference itself, as any other variable, is immutable unless stated otherwise.

We already saw how to pass a normal reference

fn process(value: &u16) -> u16 {
    *value
}

fn main() {
    let i: u16 = 42;

    process(&i);

    // i is 42
    println!("Item value {}", i);
}

Please note that everything is coherent here. The variable i is not mutable (because we don't need it), and is passed by (immutable) reference to the function. This means that the function receives a reference to a value and Rust knows that it is not allowed to change the referenced value.

If we want to change the value inside the function we need to alter the code in many places

fn process(value: &mut u16) { 1
    *value += 1
}

fn main() {
    let mut i: u16 = 42; 3

    process(&mut i); 2

    // i is 43
    println!("Item value {}", i);
}

The function process needs to receive &mut u16 1 because we want to change the referenced value. This means that main has to call the function passing &mut i 2 and not &i, as the types need to be coherent. We cannot create a mutable reference to an immutable value, though, so i has to be mutable as well 3.

The functional way¶

There is a last option that is a standard strategy in functional languages, where often variables cannot be declared as mutable at all.

struct Item {
    pub value: u16,
}

fn process(mut item: Item) -> Item { 2
    item.value += 1;
    item  3
}

fn main() {
    let mut i = Item { value: 42 };

    // Variable moved
    // Reassigned to retain ownership
    i = process(i); 1

    println!("Value {}", i.value)
}

Here, we pass the variable i by value 1, which will copy or move the variable (depending on Copy). The function process declares the parameter as mutable 2 and changes its value. Then, it returns the same type it accepted 3 and the caller reassigns the old variable 1, that at this point has to be mutable.

A quick recap¶

As you can see, it is very hard to separate call conventions, ownership, and mutability. They are all different aspects of function arguments in Rust, and they work together to ensure the code is safe. Before wrapping up, it might be useful to summarise how we decide the correct strategy.

Do you want to modify the original value?

YES: Variable has to be mutable. Pass by mutable reference OR return and reassign.
NO: Do you need to retain ownership?
- YES: Implement Copy. Pass by value OR pass by reference.
- NO: Pass by value OR pass by reference.

Disassemble Rust¶

If you want to disassemble the Rust code on your machine (for example to explore a different architecture) you can follow these steps.

Install cargo-binutils.
Create a project with Cargo: cargo new proto
Open Cargo.toml and turn off debug for the profile dev

Cargo.toml

[package]
name = "proto"
version = "0.1.0"
edition = "2021"

[dependencies]

[profile.dev]
debug=0

Use llvm-objdump options and awk to get the output you want. For example

$ cargo objdump -- \
   --disassemble \
   --x86-asm-syntax=intel \
   --demangle \
   --no-show-raw-insn \
   --no-print-imm-hex \
   | awk -v RS="" '/^[[:xdigit:]]+ <proto::/'

The awk script is useful to isolate the functions in your code only, skipping the boilerplate that the compiler has to put into an executable. Make sure you mention the correct name of your script if you use something else than proto.

Final words¶

This was quite a ride! I hope it was useful, it definitely was to me to clarify in my head the available options and the reasons behind them. Happy coding!

Resources¶

I highly recommend to watch this video by James Sherman that explains in detail how function code is called at CPU level.
Make sure you are familiar with the following concepts:
- The stack
- The calling convention of a specific CPU, such as the x86 calling convention.

Free book "Rust Projects - Write a Redis Clone"

2024-12-24T10:00:00+00:00

I just published version 1.1.0 of my free book "Rust Projects - Write a Redis Clone".

You can download the book from Leanpub!

The book follows the Redis challenge on CodeCrafters and contains a 40% discount for any type of paid membership.

In this version I added a section to chapter 5 to restore some tests that I left commented in one of the previous sections during a refactoring. I also added a final section to address compiler warnings and tidy up the code.

A Rust to-do list CLI app - Part 1

2024-02-14T18:00:00+01:00

This blog was born as a place where I could share what I discovered while I was learning new technologies and concepts. Remaining faithful to this manifesto, I decided to start writing some posts about Rust, as I recently joined the vibrant community behind this language. My roots are in C and Assembly, so I feel at home with Rust that looks to me like a proper modern version of C. As such, I'm supremely interested in the ideas behind it, and in particular I want to focus on low-level data representation, structures, and memory usage.

After having read the manual and implemented several snippets, I decided to try to implement a more complete application and to annotate the journey here. While I was looking for a tutorial I found this useful post by Claudio Restifo, where he develops a simple to-do list management application.

So, I decided to implement the same using Claudio's solution when I was stuck or to compare his strategy with mine, as it is always extremely useful to see how another coder tackles certain challenges. Thanks Claudio! However, as I'm a big fan of TDD, I'd like to follow that approach, which is something that Claudio doesn't do in his post.

Please keep in mind that these are my first steps with the language, so consider what you read here as the work of a beginner (as I am, with this language). I'm more than happy to receive advice or corrections, so feel free to get in touch if you see anything that can be done in a better way. In the post, you will find annotations that highlight the major topics that I think a Rust programmer should be familiar with.

Requirements¶

The requirements I set for the application are:

Manage a list of entries. Each entry can be in state "to be done" or "done".
Provide commands to view, add, delete and mark items as "done" or "to be done".
Can save and retrieve data from a file. The file has a default name that can be changed with an option.

This is an extremely basic application, so the command line is: todo [OPTIONS] COMMAND [KEY].

I expect the interaction with the tool to be something like

$ todo list
# TO DO

* Write post
* Buy milk
* Have fun

# DONE

* Feed the cat

$ todo add "Update CV"
$ todo mark-done "Buy milk"
$ todo list
# TO DO

* Write post
* Have fun
* Update CV

# DONE

* Feed the cat
* Buy milk

Initial setup¶

Starting a new Rust project is extremely simple with Cargo:

$ cargo new todo-cli

This will create the required structure in a new directory and create two files: Cargo.toml and src/main.rs. The latter will contain some placeholder code that we can use to check our setup

main.rs

fn main() {
    println!("Hello, world!");
}

We can run the code with cargo run or build it into a stand-alone executable with cargo build. This will compile the code in debug mode by default and put the executable in the directory target/debug. Cargo can be used to run tests as well (cargo test).

Cargo

Cargo [docs] is the Rust package manager and the default solution to manage dependencies, compile packages and in general to manage your code. It's highly recommended to learn at least the basics of this powerful tool.

See the source code

CLI management¶

Command line interfaces are typically not part of the classic TDD cycle, as they should be part of integration tests. Now, the definition that the Rust community uses for integration tests is

Integration tests are external to your crate and use only its public interface in the same way any other code would. Their purpose is to test that many parts of your library work correctly together.

So, the integration they consider here is that between multiple parts of a library. What I am referring to here is more properly system integration tests, where we test the public interface of a whole tool. Long story short, I will not write tests for the CLI commands.

In the aforementioned post, Claudio Restifo suggests we can read command line arguments using std::env::args() directly with something like

fn main() {
    let action = std::env::args().nth(1)
        .expect("Please specify an action");
    let item = std::env::args().nth(2)
        .expect("Please specify an item");

    println!("{:?}, {:?}", action, item);
}

Modules

In Rust a module can be used directly as long as it is part of the current project. The standard library is clearly visible by default, while other modules have to be declared in the file Cargo.toml. It is then perfectly acceptable to write let action = std::env::args()....

Use declarations, however, can import other modules into the current namespace, to make the code more readable.

The method nth [docs] returns (not too surprisingly) the nth element of an iterator.

Iterators

The Rust documentation contains a very useful section on iterators [docs].

The function std::env::args [docs] (used to access the command line arguments in the traditional Unix fashion) returns Args [docs], which implements the trait Iterator.

As it happens in object-oriented programming languages (which Rust is not), the expression "implements an interface" is often simplified to "is". So, colloquially speaking, we can say that std::env::Args is an Iterator [docs].

The prototype of nth is

fn nth(&mut self, n: usize) -> Option<Self::Item>

and it mentions Option<Self::Item> as the return type. The type Option provides a method expect [docs] that returns either the content of the Some value or panics, printing the given message in the backtrace.

Option

Option [docs] and Result [docs] are a versatile way to manage optional results (either something or nothing) and results (either something good or an error), and are among the most important structures to learn in Rust.

Running the code above with cargo run produces the following output, where we can see the message set by the first call to expect.

    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/todo-cli`
thread 'main' panicked at src/main.rs:80:42:
Please specify an action
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Better CLI management with Clap¶

Clap stands for Command Line Argument Parser and is a nice crate that simplifies the creation of advanced command line interfaces. I installed it using

$ cargo add clap --features derive

as detailed in the documentation and my code is now

use clap::Parser;

#[derive(Parser)]
struct Cli {
    command: String,
    key: String,
}

fn main() {
    let args = Cli::parse();

    println!("Command line: {} {}", args.command, args.key);
}

Clap allows me to add long and short options as well, so later I will use it to specify the database file name. For now, however, this is enough.

derive

The attribute derive [docs] is another cornerstone of the language and is used everywhere. The machinery behind it is not trivial, but I recommend getting used to the syntax and the standard use cases.

See the source code

A simple list of elements¶

From Claudio's post I got the idea of using a hash map for the list of items. That's a simple and effective solution, in particular given the fact that Rust provides the collection type out of the box.

As I want to use TDD, I begin with a test. In Rust, we put tests and code in the same file (but for integration tests between modules), so I can write a simple test at the bottom of the file to check that a TodoList type exists and can be initialised.

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn init_todo() {
        let todo = TodoList::new();
    }
}

TDD

TDD is one of my favourite methodologies and I'm happy to see that Rust allows me to follow it. I can't recommend TDD enough! The Rust book contains a pretty detailed chapter on how to write tests.

Clearly, when I run cargo test I get a compile error. Let's implement the type then

use std::collections::HashMap;

[...]

struct TodoList {
    // true = to do, false = done
    items: HashMap<String, bool>,
}

As you see, I had to write a comment as a reminder of the meaning of the boolean values. I also suspect that I will need to use the type HashMap<String, bool> multiple times, so I will probably end up creating a type alias of some sort.

To initialise such structure I have to create an implementation of the function new

Version 1

impl TodoList {
    fn new() -> TodoList {
        let items: HashMap<String, bool> =
            HashMap::<String, bool>::new();

        TodoList { items: items }
    }
}

struct and impl

Rust is not an object-oriented programming language, so it uses plain structs to encapsulate data. The Rust book has a full chapter on struct and impl.

Thanks to type inference, the explicit definition of types after the call to HashMap is not needed and I can write

Version 2

        let items: HashMap<String, bool> = HashMap::new();

Version 3

        let items = HashMap::<String, bool>::new();

For such a simple initialisation, I might also write directly

Version 4

        TodoList {
            items: HashMap::<String, bool>::new(),
        }

However, I will soon replace the ::new() with something more complicated that reads a file, so I decided to keep version 2. This code passes the test I wrote, so it's good enough for now.

At this point I can also initialise the list in the main function

struct TodoList {
    // true = to do, false = done
    items: HashMap<String, bool>,
}

impl TodoList {
    fn new() -> TodoList {
        let items: HashMap<String, bool> = HashMap::new();

        TodoList { items: items }
    }
}

fn main() {
    let args = Cli::parse();

    let todo = TodoList::new();

    println!("Command line: {} {}", args.command, args.key);
}

Please note that I'm not being too strict with dead code here and the compile will complain about unused variables and fields. I like this, and I won't add underscores to silence the warnings since they are a good reminder of what I still have to implement.

See the source code

Adding items¶

A good improvement at this point would be to create a method to add items to the list. First, the mandatory test

    #[test]
    fn add_item() {
        let mut todo = TodoList::new();
        todo.add(String::from("Something to do"));
        assert_eq!(todo.items.get("Something to do"), Some(&true))
    }

The type HashMap provides a method called insert [docs] which is exactly what I need

impl TodoList {

    ...

    fn add(&mut self, key: String) {
        self.items.insert(key, true);
    }
}

And once again this code passes the test, so I consider it good enough.

self and Self

In Rust self is a keyword [docs] and not just a name as it happens in Python. Rust considers self of type Self [docs], which is the type we are implementing in a trait or impl block.

The code fn add(&mut self, key: String) { above is equivalent to fn add(self: &mut Self, key: String) {. However, self cannot be renamed to something like foo, as Rust is expecting a parameter with that specific name.

References and mutability

I found confusing, at first, that in Rust we usually call &mut a mutable reference. In my head, I always translate it into a reference to mutable data as this helps me to remember what I am doing here.

In short, in Rust we need to declare explicitly when we intend to consider a value mutable using the keyword mut [docs], and this is valid also when we pass arguments to functions. If we decide to borrow data instead of moving it, we can use references, that in C terms are equivalent to protected pointers. We can also pass a reference to data that we intend to mutate, which is where &mut comes into play.

However, as I mentioned I think it's important to understand that the reference (a pointer) is not mutating. The data referenced by it is.

See the source code

Multiple additions and updates¶

If I add the same key multiple times I want the list to contain only one occurrence, so I test this.

    #[test]
    fn add_item_already_exist() {
        let mut todo = TodoList::new();
        todo.add(String::from("Something to do"));
        todo.add(String::from("Something to do"));
        assert_eq!(todo.items.get("Something to do"), Some(&true));
        assert_eq!(todo.items.len(), 1);
    }

The test passes already, thanks to the properties of the hash map.

I also want the second insertion not to update the value of the existing element, and in this case the test is

    #[test]
    fn add_item_does_not_change_value() {
        let mut todo = TodoList::new();
        todo.add(String::from("Something to do"));

        if let Some(x) = todo.items.get_mut("Something to do") {
            *x = false;
        }

        todo.add(String::from("Something to do"));
        assert_eq!(todo.items.get("Something to do"), Some(&false));
        assert_eq!(todo.items.len(), 1);
    }

I have to manually change the value inside the map using get_mut [docs] that returns a mutable reference to the value. This test doesn't pass, as insert actually updates the existing value.

At the time of writing the method try_insert of HashMap is experimental, so I implemented a custom solution

use std::collections::hash_map::Entry;

[...]


    fn add(&mut self, key: String) {
        if let Entry::Vacant(entry) = self.items.entry(key) {
            entry.insert(true);
        }
    }

Here, I'm basically checking if an entry for key is vacant (does not exist) and I create it only in that case. This code passes all tests.

if let

I consider if let [docs] a very powerful piece of syntax. I care only about one of the possible outcomes, so I don't want to waste time defining it in a full-fledged match.

See the source code

Marking items¶

The second method I want to add is mark that allows me to set the value of the boolean corresponding to a given key. This will be used to flag an item as "done" or "to be done". The test is

    #[test]
    fn mark_item() {
        let mut todo = TodoList::new();
        todo.add(String::from("Something to do"));
        todo.mark(String::from("Something to do"), false);
        assert_eq!(todo.items.get("Something to do"), Some(&false))
        todo.mark(String::from("Something to do"), true);
        assert_eq!(todo.items.get("Something to do"), Some(&true))
    }

Here, I can follow the same strategy I used in the test add_item_does_not_change_value

impl TodoList {

    ...

    fn mark(&mut self, key: String, value: bool) {
        if let Some(x) = self.items.get_mut(&key) {
            *x = value;
        }
    }
}

What if the key is not in the list, though? The function get_mut returns an Option, but mark should signal with a Result that something didn't work. I can test this with

    #[test]
    fn mark_item_does_not_exist() {
        let mut todo = TodoList::new();
        assert_eq!(
            todo.mark(String::from("Something to do"), false),
            Err(String::from("Something to do"))
        );
    }

The new version of the function is then

impl TodoList {

    ...

    fn mark(&mut self, key: String, value: bool) -> Result<String, String> {
        let x = self.items.get_mut(&key).ok_or(&key)?;
        *x = value;

        Ok(key)
    }
}

The method ok_or [docs] converts an Option into a Result, so I just call ? to propagate the error.

The question mark operator

The operator ? is one of the best features of Rust, and it's explained in this chapter of the Rust Book. I find it such a simple yet extremely powerful way to deal with error propagation.

See the source code

Listing items¶

At this point I want to add the method list that allows me to see the items contained in TodoList. I'd like to separate the logic from the presentation so the method will return two lists of items, one for each value of the connected boolean.

This means that the output of the method should in my opinion be a tuple of iterators, one on the items with state "to be done" and one on the ones in state "done".

Iterators

Iterators are a big thing in Rust, and I can understand why as they definitely boost performances saving memory. The Rust book has a chapter on them, and there is clearly plenty of documentation for the relative trait.

I start with tests as usual

    #[test]
    fn list_items() {
        let mut todo = TodoList::new();
        todo.add(String::from("Something to do"));
        todo.add(String::from("Something else to do"));
        todo.add(String::from("Something done"));
        todo.mark(String::from("Something done"), false);

        let (todo_items, done_items) = todo.list();

        let todo_items: Vec<String> = todo_items.map(|x| x.clone()).collect();
        let done_items: Vec<String> = done_items.map(|x| x.clone()).collect();

        assert!(todo_items.iter().any(|e| e == "Something to do"));
        assert!(todo_items.contains(&String::from("Something else to do")));
        assert_eq!(todo_items.len(), 2);
        assert!(done_items.contains(&String::from("Something done")));
        assert_eq!(done_items.len(), 1);
    }

There is a lot to say here, and please remember the caveat that I'm not sure what I'm doing is the best thing.

I add some elements to the list and mark one as done, then I call the method list to get two iterators and test them. However, iterators can be traversed only once, so to test them properly I prefer to convert them into vectors using the method collect [docs].

To generate the two iterators I will probably use HashMap::iter [docs], which means they will have an element type &String, as we are interested in the item key.

As far as I can tell, there are several different strategies I can use here.

I can generate vectors of &String using the elements directly from the iterators and then use the method Vec::contains [docs]. However, the latter wants to receive a reference to the searched value, which means that I would end up with

let todo_items: Vec<&String> = todo_items.collect();
assert!(todo_items.contains(&&String::from("Something else to do")));

While this is perfectly reasonable in terms of memory consumption and performances, the double && is a bit ugly. So, considering that I'm writing a test, where performances are not the major concern, I'd prefer to simplify the syntax. I can create a vector of String values and check them

let todo_items: Vec<String> = todo_items.cloned().collect();
assert!(todo_items.contains(&String::from("Something else to do")));

The syntax todo_items.cloned() is equivalent to todo_items.map(|x| x.clone()) and leverages the implicit dereferencing of x. Here, copied() cannot be used as String doesn't implement the trait Copy.

A good alternative to contains is any, which however works on iterators. A final version of the code is then

let todo_items: Vec<String> = todo_items.cloned().collect();
assert!(todo_items.iter().any(|e| e == "Something to do"));

Which is also more elegant since it uses the comparison between a String (which is the iterator item type) and an &str (the right side). At this point my test is

#[test]
fn list_items() {
    let mut todo = TodoList::new();
    todo.add(String::from("Something to do"));
    todo.add(String::from("Something else to do"));
    todo.add(String::from("Something done"));
    todo.mark(String::from("Something done"), false);

    let (todo_items, done_items) = todo.list();

    let todo_items: Vec<String> = todo_items.cloned().collect();
    let done_items: Vec<String> = done_items.cloned().collect();

    assert!(todo_items.iter().any(|e| e == "Something to do"));
    assert!(todo_items.iter().any(|e| e == "Something else to do"));
    assert_eq!(todo_items.len(), 2);
    assert!(done_items.iter().any(|e| e == "Something done"));
    assert_eq!(done_items.len(), 1);
}

An implementation of the method list that passes this test is

    fn list(&self) ->
       (impl Iterator<Item = &String>, impl Iterator<Item = &String>) {
        (
            self.items.iter().filter(|x| *x.1 == true).map(|x| x.0),
            self.items.iter().filter(|x| *x.1 == false).map(|x| x.0),
        )
    }

Here, the powerful keyword impl declares that whatever comes out of that function implements the Iterator trait with an element type String. The code uses iter [docs] to create an iterator on the elements of the hash map (element type (&String, &bool), then uses map [docs] to extract the first element of each tuple. All in all, the function returns a tuple of Map [docs] which is a type that implements Iterator.

See the source code

Exposing commands on the CLI¶

It's time to expose the methods I implemented on the CLI. I realised that commands like add and mark-done require a second argument (the key), other commands like list don't.

So, the first change is to make the key argument optional.

#[derive(Parser)]
struct Cli {
    command: String,

    key: Option<String>,
}

Purely to have something to play with, I will also add some values to the list in main. This is temporary, as long as I don't implement a file storage mechanism.

fn main() {
    let args = Cli::parse();

    let mut todo = TodoList::new();

    todo.add("Something to do".to_string());
    todo.add("Something else to do".to_string());
    todo.add("Something done".to_string());
    todo.mark("Something done".to_string(), false).unwrap();
}

Last, the command-method binding part. A match construct is the best option in this case, something like

match args.command.as_str() {
    "add" => ...,
    "mark-done" => ...,
    "list" => ...
}

match

The match control flow construct is a blessing that comes directly from functional programming, where pattern matching is an important tool. The Rust book has a chapter dedicated to it and a chapter on the pattern syntax.

However, since each method has a different return type, I need the whole construct to return a uniform Result that can be used to print a meaningful state message at the end of the execution.

The code I wrote is the following

fn main() {
   ...

   let result = match args.command.as_str() {
        "add" => match args.key {
            Some(key) => {
                todo.add(key);
                Ok(())
            }
            None => Err("Key cannot be empty!".to_string()),
        },
        "mark-done" => match args.key {
            Some(key) => todo
                .mark(key, false)
                .map_err(|e| format!("Invalid key {}", e))
                .and(Ok(())),
            None => Err("Key cannot be empty!".to_string()),
        },
        "list" => {
            let (todo_items, done_items) = todo.list();

            println!("# TO DO");
            println!();
            todo_items.for_each(|x| println!(" * {}", x));

            println!();

            println!("# DONE");
            println!();
            done_items.for_each(|x| println!(" * {}", x));

            Ok(())
        }
        cmd => Err(format!("Command {} not recognised", cmd)),
    };

    match result {
        Err(e) => println!("ERROR: {}", e),
        Ok(_) => println!("SUCCESS"),
    }
}

Option and Result

It's paramount to learn how to convert Option [docs] into Result [docs] and vice versa, as well as how to convert a Result type into a different one. Being familiar with functions like map_err [docs] or and [docs] will drastically change the quality of your Rust code.

See the source code

Tidy up¶

At this point I went through the code and fixed some of the warning the compiler was still giving me. These all come from the tests, where I created the todo variable but never used it, and where I ignored the results returned by calls of todo.mark. There, I used unwrap [docs] as I'm happy for that to panic if something goes wrong.

See the source code

Final words¶

What a journey so far! It's really true that you can't consider a language learned until you start from scratch and try to use it to implement a real application. Well, it's not over yet, I'm still missing an important part which is the file storage.

If you have comments, suggestions, or corrections, please let me know! I am more than happy to learn something new from other coders and to publish updates to the post.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.