The Digital Cat - functional programming

First-class objects in Python - Higher-order functions, wrappers, and factories

2021-03-09T16:00:00+00:00

My new book "First-class objects in Python" is out! Grab your FREE copy here!

Clojure sequential data types for Python programmers

2015-09-22T19:00:00+01:00

I have been working with Python for more than fifteen years and I developed very big systems with this language, learning to know and love both its power and its weaknesses, admiring its gorgeous object-oriented system and exploring some of its darkest corners.

Despite being so committed to this language (both for personal choices and for business reasons) I never stopped learning new languages, trying to get a clear picture of new (or old) paradigms, to explore new solutions to old problems and in general to get my mind open.

Unfortunately introducing a new language in your business is always very risky, expensive and many times not a decision of yours only. You are writing software that other may have to maintain, so either you evangelize your team and persuade other to join you or you will be forced to stay with the languages shared by them. This is not bad in itself, but sometimes it is a big obstacle for the adoption of a new language.

One of the languages I fell in love (but never used in production) is LISP. I strongly believe that everyone calls themselves a programmer shall be somehow familiar with LISP, as well as with Assembly language. The reasons are many and probably do not fit well with the title of this article so I will save them for another, more theoretical post, and move on.

Clojure is a LISP implementation that runs on the JVM, thus being available on a wide number of platforms. As any other LISP its syntax is a step lower in the abstraction tree: while other languages provide their syntax and then a conversion to an AST (Abstract Syntax Tree), LISPs provide only their AST syntax. This has the great advantage of homoiconicity, something that they share with Assembly and other languages like Prolog, but comes at the cost of a slightly more difficult syntax, at least for the novice.

In this post I want to review the sequential data types provided by Clojure, trying to introduce them to programmers accustomed to Python data types. Anyway, while the nomenclature and the examples will be specific to Python, I think the discussion can be easily extended to other dynamic languages such as Ruby.

Three collection types for the programming-kings¶

In Clojure, just like in Python, you find three types of basic collection types, that is types of data that hold together other types of data, also called compound data types. These are sequences, sets and maps.

Python provides two types of basic sequences, tuples and lists, while maps are called dictionaries. Those are present in Python since its first versions. Sets have been introduced with Python 2.3 in 2003, at first with a dedicated set module, which was then deprecated with Python 2.6 in favour of the built-in type with the same name. From Python 2.7 sets get also a dedicated short syntax similar to that of dictionaries (e.g. {'just', 'a', 'set').

The two basic Python sequences differ mainly for a small but very important detail: mutability. In Python tuples are immutable, while lists can be modified. Python lists behave mostly like C arrays: elements can be appended to and extracted from the right end. Apart from the mutability, Python lists and tuples expose a common API which we may address as the Python sequence API or protocol.

An object that follows the sequence API in Python has the following traits, among the others:

Elements are indexed starting from 0 up to the total number of elements minus 1.
The sequence can be traversed in order, that is, elements can be visited one by one and each repeated visit of the whole collection presents the elements in the same order.

A derived feature which is provided by sequences is that they can be sliced, i.e. part of them can be extracted as a collection of the same type.

These features are paramount because they allow Python to define a common interface for loops. This is obviously shared with mostly every programming language, but Clojure implements it in a slightly different way. A Python class shall implement some "magic" methods to expose the sequence API. To get a starting picture of the steps involved read this post, where I discuss the implementation of a binary number type in Python.

A matter of definitions¶

The Clojure counterpart of the Python sequence protocol is the seq API and it consists of two simple functions: first and rest. Their behaviour is the following

If the collection is not empty (first collection) returns the first element, otherwise returns nil.
If the collection is not empty (rest collection) returns a new sequence containing all the elements of collection except the first one, in the same order. If the collection is empty rest returns an empty collection.

This API is the same we can find in functional languages like Erlang or Scala, where most of the loops are implemented as (tail-) recursive function calls that consume the sequence one element at a time, leaving the rest of the sequence to the next recursion.

Pay attention to an important difference in Clojure's handling of empty sequences, compared to other LISPs and Python itself. While those treat empty sequences as logically false, Clojure doesn't, so be careful when you deal with this part of it coming from a different language. Remember that in Clojure everything is true except nil and false.

Given the definition of the seq API, in Clojure we call sequence everything behaves according to it. Since Clojure is implemented on the JVM this "behaviour" matches a very specific Java interface, which in this case is ISeq (see the source definition here).

The concept of being an ordered set of values in Clojure is represented by the word sequential, which corresponds to the Java interface Sequential (source code). Last, the concept of being a collection is represented in Clojure by means of the Java interface IPersistenCollection (source code).

Given these definitions it is immediately clear what difference is there between the seq?, sequential? and coll? functions. They tell you if the given parameter implements the relative Java interface.

The function seq, instead, performs a conversion of its input into a sequence, that is something that implements the ISeq interface. This function always returns nil if the input is an empty collection, which allows Clojure to adhere to the standard LISP technique called nil-punning, or in simpler words loop termination condition check. Every collection in Clojure provides a meaningful output for the seq function, but some of them provide more than one function to return a sequence. For example maps provide a way to get the sequence of both keys and values through the aptly named keys and vals functions.

Vectors and lists¶

Clojure provides two sequential data types just like Python, but they do not differ for the mutability property, because in Clojure every collection is immutable. Instead, they differ for their behaviour as queues (where a new element goes and where a popped element comes from).

These two data types are vectors and lists, and as happens in other LISPs lists are very important for the language syntax itself. Their main difference between the two types, which is immediately clear when using them, is that while vectors grow on the right end, lists grow on the left one. This will be important when discussing append functions like conj.

Let me first start with some details on mutability and persistence.

Mutability and persistence¶

Clojure's collections are immutable, which means that you cannot change the content of one of their instances. This is a feature provided in Python by means of the tuple data type only. As happens in other languages like Erlang, Clojure's immutability of collections is a trampoline to concurrency safety, something the language provides as a feature rather than a limitation.

Clojure collections are also persistent. The original meaning of this word refers to the possibility of accessing the old version of a given structure after having changed it. Since structures in Clojure are immutable you are forced to create a new variable every time you want to change something, which implies that the "old version" is still available. That is, immutability implies persistence. In Clojure, moreover, structures share values as long as this is consistent, which is exactly what Python does with the references concept (see this post).

Vectors in Clojure¶

Clojure's vectors are very similar to Python lists. The first thing that makes them similar (apart from the square brackets syntax) is that they grow on the right end.

user=> (conj [1 2 3 4] 5)
[1 2 3 4 5]

>>> l = [1,2,3,4]
>>> l.append(5)
>>> l
[1, 2, 3, 4, 5]

From the right end you can also easily get values

user=> (peek [1 2 3 4 5])
5

>>> l = [1,2,3,4,5]
>>> l
[1, 2, 3, 4, 5]
>>> l.pop()
5
>>> l
[1, 2, 3, 4]

So Clojure vectors and Python lists are LIFO queues (Last In First Out), otherwise known as stacks. In Clojure peek takes the role of the classic pop operation, but remember that being collections immutable, what you get is a new value. That is, while in Python pop() modifies the list, in Clojure peek doesn't alter the vector.

Subvectors¶

Just like Python lists, vectors can be sliced with the subvec function

user=> (subvec [:a :b :c :d :e] 2 5)
[:c :d :e]

Subvectors are however different from Python list slices. When you slice a list in Python you get a new object containing part of the same references you can find in the source list. That is, slicing a list actually builds a new list copying the reference of each object contained in the slice.

Let us briefly review this important concept. For this example I build a list of mutable objects, in this case dictionaries.

>>> l = [{'a': 1, 'b': 2}, {'c': 3, 'd': 4}, {'e': 5, 'f': 6}]
>>> l
[{'b': 2, 'a': 1}, {'c': 3, 'd': 4}, {'f': 6, 'e': 5}]

then take a slice of it, a list containing the second element only

>>> s = l[1:2]
>>> s
[{'c': 3, 'd': 4}]

The original list l and the slice s are different objects, as shown by their memory address

>>> id(l)
3069706412
>>> id(s)
3061925100

but there is still a link between them, as you can see if you change the value of the element contained in the slice. This change affects the original list too

>>> s[0]['c'] = 100
>>> s
[{'c': 100, 'd': 4}]
>>> l
[{'b': 2, 'a': 1}, {'c': 100, 'd': 4}, {'f': 6, 'e': 5}]

In Clojure a subvector is a complete reference to the source vector, with a special annotation of the slice boundaries. This gives you a very fast way to create subvectors, which is independent from the size of the vector itself (aka O(1) or constant time). On the other hand this gives you a structure which is a bit slower when accessed, due to the math involved in computing the actual offset. This handicap however does not propagate to subvectors of subvectors, so it can be considered a (little) constant slowdown of every subvector.

Appending values (push)¶

You find two different functions that can append values to a Clojure vector, into and conj, implementing the push operation of classic stacks. Clojure is not an object-oriented language, so these functions are not methods of the vector type. Those methods however behave in a different way according to the destination type.

The into function, according to the official documentation conjoins every item of the second argument (the from collection) into the first argument (the to collection). This means that every item of the incoming collection is appended to the destination, according to the way that collection manages the append operation. For example, vectors grow on the right, so performing an into will append elements on the right end of the vector itself. This means also that into may be used to convert one type of collection into another.

Conversely, the conj function conjoins the whole second argument (conj accepts more than two arguments but let's deal with the simpler case now)

user=> (conj [1 2 3 4] [5 6])
[1 2 3 4 [5 6]]

Just like into, conj relies on the destination type to perform the actual append operation.

In Python you can find a similar situation if you consider the two methods extend() and append() of lists. The first one expects an argument that exposes the sequential protocol, and pushes into the destination list each of its elements

>>> l = [1, 2, 3]
>>> l
[1, 2, 3]
>>> l.extend([4, 5, 6])
>>> l
[1, 2, 3, 4, 5, 6]

On the other hand, append() pushes into the list the whole argument, just like conj does for vectors in Clojure, and thus is mostly used to append single elements, for example in for loops.

>>> l = [1, 2, 3]
>>> l.append([4, 5, 6])
>>> l
[1, 2, 3, [4, 5, 6]]

Getting values (pop)¶

The pop operation of classic stacks is also implemented by two different functions, peek and pop. The first one returns the last element of the vector, being thus the Clojure's equivalent of Python's pop() as shown in a previous example. Clojure's pop, instead, removes the last element and returns the resulting vector.

user=> (pop [1 2 3 4 5])
[1 2 3 4]

The Python equivalent could be a slice

>>> [1,2,3,4,5][:-1]
[1, 2, 3, 4]

The last function will return the same value as peek, but is very inefficient for vectors. The official documentation gives an example for a vector of ten thousand elements where last is around 25 times slower than peek. This is mainly due to the fact that last scans the whole sequence to find the last element, that is works in linear time O(n) (see the source code).

A final note about looking for values. The Python in operator has no direct equivalent for Clojure vectors. Beware that the contains? function does check the collection keys (indices of the vector) and not the values. The official documentation of contains? gives some hints about this problem. I strongly suggest you to check them.

Random access¶

Clojure vectors and Python lists are both arrays of indexed values, which means that they are optimized for random access. In Python you may just index the list

>>> l = ["a", "list", "of", "strings"]
>>> l[2]
'of'

where the indexing syntax works also for slices. In Clojure you can use the nth function

user=> (nth ["a" "vector" "of" "strings"] 2)
"of"

For vectors the get function maps to nth, since vectors are after all ordered dictionaries with the position indices as keys. Vectors in Clojure can also be directly used as functions, which aliases the use of nth

user=> (["a" "vector" "of" "strings"] 2)
"of"

Lists in Clojure¶

Being Clojure a LISP, lists are of paramount importance and a complete discussion about them will surely span more than a small post on a blog. So in this section I will try to stay focused on the differences with Python lists (and Clojure vectors), rather than give you a complete overview of this incredibly powerful tool. The most important thing I will not review here is that lists in Clojure are part of the syntax itself and in the following section I will only consider lists as data containers. The fact that LISPs consider code as data is outside the scope of this work.

Being vectors the Clojure collection that implement classic stacks (LIFO queues), and being stacks one of the most used data structures in computer science, it is no surprise that a lot of sequential data processing in Clojure is done through vectors.

As already stated the big difference between vectors and lists in Clojure is the fact that lists grow on the left end. Let us review the behaviour of peek and conj on lists compared to that of the same functions on vectors

user=> (peek '(1 2 3 4))
1
user=> (peek [1 2 3 4])
4
user=> (conj '(2 3 4) 1)
(1 2 3 4)
user=> (conj [2 3 4] 1)
[2 3 4 1]
user=> (pop '(2 3 4))
(3 4)
user=> (pop [2 3 4])
[2 3]

I think this is simple to remember and easy to deal with in algorithms. As you can see lists behave somehow like mirrored vectors. Beware, however, that functions such as first and last still work in the same way. That is, lists are not reversed vectors.

As happened for vectors there are some performance considerations to do for lists. Even for lists the last function performs in linear time, but while vectors can use peek to perform the same service, lists cannot. This is a big asymmetry and one of the reasons you could prefer vectors over lists for your data. While accessing the first element is blazing fast both for lists and vectors, accessing the last can be very fast for vectors through the peek function but is performed in linear time on lists.

One of the biggest drawbacks of Clojure lists used like arrays is that they are not optimized for random access, while vectors are. That is, the nth function performs in linear time on lists since, just like last, it scans the whole list one element at a time. Vectors, on the other hand, are indexed, which means that random access is performed in constant time.

As you can see there is not much to say about Clojure lists as data containers (here data is used more in a Python sense): there are simply very few reasons to prefer lists over vectors as data containers.

Final words¶

Python and Clojure are very different languages, so this was not an attempt to find "migration rules" from Python code to Clojure. I just wanted to review what Python programmers shall expect from Clojure sequential data types, expressing it in a familiar way.

I wish you a happy Clojure time!

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 16-20

2015-05-13T16:00:00+01:00

Problem 16¶

The problem¶

P16 (**) Drop every Nth element from a list.

Example:

scala> drop(3, List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k))
res0: List[Symbol] = List('a, 'b, 'd, 'e, 'g, 'h, 'j, 'k)

Mapping¶

A first simple approach is to divide the list in small groups of up to N elements and drop the last element from each of them. This is easily achieved with the grouped() method of List types and the application of a flat mapping.

def drop[A](n: Int, l: List[A]):List[A] = {
    l.grouped(n).flatMap { _.take(n - 1) }.toList
}

The grouped() method returns an iterator, which is, as in other languages, something that can visit the elements contained in a collection and yield them. When an iterator is required to be consumed all at once you can call the toList() method to convert it into a List.

scala> val l = List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k)
l: List[Symbol] = List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k)

scala> l.grouped(3)
res0: Iterator[List[Symbol]] = non-empty iterator

scala> l.grouped(3).toList
res1: List[List[Symbol]] = List(List('a, 'b, 'c), List('d, 'e, 'f), List('g, 'h, 'i), List('j, 'k))

Iterator type has a lot of methods in common with List type, most notably flatMap(). This last method is fed with a simple anonymous function that, using take(), keeps the first n - 1 elements.

A different approach may be followed using the zipWithIndex() method that packs each element of the list with its index (starting from 0)

scala> l zipWithIndex
res5: List[(Symbol, Int)] = List(('a,0), ('b,1), ('c,2), ('d,3), ('e,4), ('f,5), ('g,6), ('h,7), ('i,8), ('j,9), ('k,10))

After this transformation, we may apply the filter() method to get the right elements with a selection based on the modulo operation.

The list is now made of Tuple2 elements, as tuples in Scala aren't a collection just like lists (or like Python tuples). For performance reasons, they are implemented with a different type according to the length, like in this case. To index a Tuple2 element we may use its _1 and _2 attributes (pay attention that tuples are indexed starting from 1).

The filter anonymous function just checks if the index (the second element of the tuple) is a multiple of n. Then, with map() we keep the first element only (the original element); the anonymous function { _._1 } is a shortcut form for { e => e._1 }

def dropZip[A](n: Int, l: List[A]):List[A] = {
    l.zipWithIndex filter { e => (e._2 + 1) % n != 0 } map { _._1 }
}

The recursive solution¶

A recursive solution can be written based on this simple algorithm: the number N is used as a countdown, keeping elements until it reaches the value 1. Then the current element is discarded and the process continues with a countdown reset.

def dropRec[A](n: Int, l: List[A]):List[A] = {
    def _dropRec[A](c: Int, res: List[A], rem: List[A]):List[A] = (c, rem) match {
        case (_, Nil) => res
        case (1, _::tail) => _dropRec(n, res, tail)
        case (_, h::tail) => _dropRec(c -1, res:::List(h), tail)
    }
    _dropRec(n, List(), l)
}

As you can see the case (1, _::tail) restarts the process ignoring the current head of the list, while the more generic (_, h::tail) case stores the element in the result list.

Problem 17¶

The problem¶

P17 (*) Split a list into two parts.

Example:

scala> split(3, List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k))
res0: (List[Symbol], List[Symbol]) = (List('a, 'b, 'c),List('d, 'e, 'f, 'g, 'h, 'i, 'j, 'k))

Solution¶

List objects provide a bunch of methods to interact with them, thus splitting may make use of several different ones. Very simple and straightforward solutions come from the splitAt() method

def split[A](n: Int, l: List[A]): (List[A], List[A]) = l.splitAt(n)

using the two take() and drop() methods

def split[A](n: Int, l: List[A]): (List[A], List[A]) = {
    (l.take(n), l.drop(n))
}

or using take() and takeRight()

def split[A](n: Int, l: List[A]):(List[A], List[A]) = {
    (l.take(n), l.takeRight(l.length - n))
}

A list can be splitted also using recursion, however, using the given split position as a counter

def split[A](n: Int, l: List[A]):(List[A], List[A]) = {
    def _split[A](c: Int, res: List[A], rem: List[A]):(List[A],List[A]) = (c, rem) match {
        case (_, Nil) => (res, Nil)
        case (0, rem) => (res, rem)
        case (c, h::tail) => _split(c - 1, res:::(List(h)), tail)
    }
    _split(n, List(), l)
}

where I basically keep extracting the head from rem appending it to res until the counter is 0.

Problem 18¶

The problem¶

Pxx (**) Extract a slice from a list. Given two indices, I and K, the slice is the list containing the elements from and including the Ith element up to but not including the Kth element of the original list. Start counting the elements with 0.

Example:

scala> slice(3, 7, List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k))
res0: List[Symbol] = List('d, 'e, 'f, 'g)

Solution¶

As for the previous problem we may make use of the List type methods. The aptly named slice()

def slice[A](start: Int, end: Int, l: List[A]): List[A] =
    l.slice(start, end)

or a combination of drop() and take()

def slice[A](i: Int, k: Int, l: List[A]):List[A] = {
    l take k drop i
}

The recursive solution first drops the required number of elements from the head of the list. When the first index is exhausted the function starts dropping elements from the tail (using init()). The number of elements to be dropped is computed as k - i when the internal function is called.

def slice[A](i: Int, k: Int, l: List[A]):List[A] = {
    def _slice[A](cl: Int, cr: Int, rem: List[A]):List[A] = (cl, cr, rem) match {
        case (0, 0, rem) => rem
        case (0, cr, rem) => _slice(0, cr - 1, rem.init)
        case (cl, cr, h::rem) => _slice(cl - 1, cr, rem)
    }
    _slice(i, k - i, l)
}

Problem 19¶

The problem¶

P19 (**) Rotate a list N places to the left.

Example:

scala> rotate(3, List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k))
res0: List[Symbol] = List('d, 'e, 'f, 'g, 'h, 'i, 'j, 'k, 'a, 'b, 'c)

scala> rotate(-2, List('a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i, 'j, 'k))
res1: List[Symbol] = List('j, 'k, 'a, 'b, 'c, 'd, 'e, 'f, 'g, 'h, 'i)

Solution¶

Rotating a list has some caveats. First one must consider wrapping, so the first thing I did is to calculate the actual shift with the modulo operation. The second thing is that rotation can happen in both directions, and the case of negative shift can be converted to a positive one just adding the shift to the length of the list (being the shift negative this is actually a subtraction). Eventually we can perform the rotation which requires a composition of the output of drop() and take()

def rotate[A](n: Int, l: List[A]):List[A] = {
    val wrapn = if (l.isEmpty) 0 else n % l.length
    if (wrapn < 0) rotate(l.length + n, l)
    else l.drop(wrapn):::l.take(wrapn)
}

Problem 20¶

The problem¶

P20 (*) Remove the Kth element from a list. Return the list and the removed element in a Tuple. Elements are numbered from 0.

scala> removeAt(1, List('a, 'b, 'c, 'd))
res0: (List[Symbol], Symbol) = (List('a, 'c, 'd),'b)

Solution¶

There are two edge conditions in this problem. The first is when the list is empty, and the second is when we are asking for an index outside the list. Since an empty list has length 0 both conditions can be summarized checking if the requested index is greater or equal than the length of the list (the "or equal" part comes from the fact that lists are indexed starting from 0).

The simplest solution uses take() and drop()

def removeAt[A](n: Int, l: List[A]):(List[A], A) = {
    if (l.length <= n ) throw new NoSuchElementException
    (l.take(n):::l.drop(n).tail, l(n))
}

The problem with boundaries comes from the use of tail(), that cannot be applied to an empty list, while take() and drop() seamlessly work on empty lists and negative indexes. So using the take() counterpart takeRight() we could write

def removeAt[A](n: Int, l: List[A]):(List[A], A) = {
    (l.take(n):::l.takeRight(l.length - n), l(n))
}

which automatically raises an IndexOutOfBoundsException when the requested index is outside the list. If we want a NoSuchElementException, however, we have to wrap the code with a try statement.

We can also use mapping, even if in this case its use is probably overkill

def removeAt[A](n: Int, l: List[A]):(List[A], A) = {
    (l.zipWithIndex filter { e => e._2 != n } map { _._1 }, l(n))
}

this solution raises an IndexOutOfBoundsException when the index is outside the list, just like the previous one.

A recursive solution may use the index as a countdown value to get elements until the requested one pops up, then skipping it and getting the rest of the list.

def removeAtR[A](n: Int, l: List[A]):(List[A], A) = {
    def _removeAtR[A](k: Int, res: List[A], rem: List[A]):(List[A], A) = (k, rem) match {
        case (0, h::tail) => return (res:::tail, h)
        case (k, Nil) => throw new NoSuchElementException
        case (k, h::tail) => return _removeAtR(k - 1, res:::List(h), tail)
    }
    return _removeAtR(n, List(), l)
}

Final considerations¶

Problem 16 introduced me to new list methods filter() and zipWithIndex(). I met iterators and tuples for the first time and learned how to deal with them on a basic level. Problem 17 introduced the new splitAt() method and let me review take(), takeRight() and drop(). The recursive solution makes use of everything has been already discovered solving the previous problems. With problem 18 I learned a new method of the List type, slice(). The recursive solution is pretty straightforward but matching a tuple of three values requires some attention. I appreciate the syntactic sugar that allows to drop parenthesis when calling a function with a single argument, it makes the call resemble natural language. Problems 19 and 20 are just applications of the same methods.

Feedback¶

The GitHub issues page is the best place to submit corrections.

Python decorators: metaprogramming with style

2015-04-23T13:00:00+01:00

This post is the result of a lot of personal research on Python decorators, meta- and functional programming. I want however to thank Bruce Eckel and the people behind the open source book "Python 3 Patterns, Recipes and Idioms" for a lot of precious information on the subject. See the Resources section at the end of the post to check their work.

Is Python functional?¶

Well, no. Python is a strong object-oriented programming language and is not really going to mix OOP and functional like, for example, Scala (which is a very good language, by the way).

However, Python provides some features taken from functional programming. Generators and iterators are one of them, and Python is not the only non pure functional programming language to have them in their toolbox.

Perhaps the most distinguishing feature of functional languages is that functions are first-class citizens (or first-class objects). This means that functions can be passed as an argument to other functions or can be returned by them. Functions, in functional languages, are just one of the data types available (even if this is a very rough simplification).

Python has three important features that allows it to provide a functional behaviour: references, function objects and callables.

References¶

Python variables share a common nature: they are all references. This means that variables are not typed per se, being pure memory addresses, and that functions do not declare the incoming data type for arguments (leaving aside gradual typing). Python polymorphism is based on delegation, and incoming function arguments are expected to provide a given behaviour, not a given structure.

Python functions are thus ready to accept every type of data that can be referenced, and functions can.

Read this post to dive into delegation-based polymorphism and references in Python.

Functions objects¶

Since Python pushes the object-oriented paradigm to its maximum, it makes a point of always following the tenet everything is an object. So Python functions are objects as you can see from this simple example

>>> def f():
...  pass
... 
>>> type(f)
<class 'function'>
>>> type(type(f))
<class 'type'>
>>> type(f).__bases__
(<class 'object'>,)
>>>

Given that, Python does nothing special to treat functions like first-class citizens, it simply recognizes that they are objects just like any other thing.

Callables¶

While Python has the well-defined function class seen in the above example, it relies more on the presence of the __call__ method. That is, in Python any object can act as a function, provided that it has this method, which is invoked when the object is "called".

This will be crucial for the discussion about decorators, so be sure that you remember that we are usually more interested in callable objects and not only in functions, which, obviously, are a particular type of callable objects (or simply callables).

The fact that functions are callables can also be shown with some simple code

>>> def f():
...  pass
... 
>>> f.__call__
<method-wrapper '__call__' of function object at 0xb6709fa4>

Metaprogramming¶

While this is not a post on languages theory, it is worth spending a couple of words about metaprogramming. Usually "programming" can be seen as the task of applying transformations to data. Data and functions can be put together by an object-oriented approach, but they still are two different things. But you soon realize that, as you may run some code to change data, you may also run some code to change the code itself.

In low level languages this can be very simple, since at machine level everything is a sequence of bytes, and changing data or code does not make any difference. One of the most simple examples that I recall from my x86 Assembly years is the very simple self obfuscation code found is some computer viruses. The code was encrypted with a XOR cipher and the first thing the code itself did upon execution was to decrypt its own code and then run it. The purpose of such tricks was (and is) to obfuscate the code such that it would be difficult for an antivirus to find the virus code and remove it. This is a very primitive form of metaprogramming, since it recognizes that for Assembly language there is no real distinction between code and data.

In higher lever languages such as Python achieving metaprogramming is no more a matter of changing byte values. It requires the language to treat its own structures as data. Every time we are trying to alter the behaviour of a language part we are actually metaprogramming. The first example that usually comes to mind are metaclasses (probably due to the "meta" word in their name), which are actually a way to change the default behaviour of the class creation process. Classes (part of the language) are created by another part of the language (metaclasses).

Decorators¶

Metaclasses are often perceived as a very tricky and dangerous thing to play with, and indeed they are seldom required in Python, with the most notable exception (no pun intended) being the Abstract Base Classes provided by the collections module.

Decorators, on the other side, are a feature loved by many experienced programmers and after their introduction the community has developed a big set of very interesting use cases.

I think that the first approach to decorators is often difficult for beginners because the functional version of decorators are indeed a bit complex to understand. Luckily, Python allows us to write decorators using classes too, which make the whole thing really easy to understand and write, I think.

So I will now review Python decorators starting from their rationale, then looking at class-based decorators without arguments, class-based decorators with arguments, and finally moving to function-based decorators.

Rationale¶

What are decorators, and why should you learn how to define and use them?

Well, decorators are a way to change the behaviour of a function or a class, so they are actually a way of metaprogramming, but they make it a lot more accessible than metaclasses do. Decorators are, in my opinion, a very natural way of altering functions and classes.

Moreover, with the addition of some syntactic sugar, they are a very compact way to both make changes and signal that those changes have been made.

The best syntactic form of a decorator is the following

@dec
def func(*args, **kwds):
    pass

where dec is the name of the decorator and the function func is said to be decorated by it. As you can see any reader can quickly identify that the function has a special label attached, thus being altered in its behaviour.

This form, however, is just a simplification of the more generic form

def func(*args, **kwds):
    pass
func = dec(func)

But what actually ARE the changes that you may want do to functions or classes? Let us stick for the moment to a very simple task: adding attributes. This is by no means a meaningless task, since there are many practical use cases that make use of it. Let us first test how we can add attributes to functions in plain Python

>>> def func():
...  pass
... 
>>> func.attr = "a custom function attribute"
>>> func.attr
'a custom function attribute'

and to classes

>>> class SomeClass:
...  pass
... 
>>> SomeClass.attr = "a custom class attribute"
>>> SomeClass.attr
'a custom class attribute'
>>> s = SomeClass()
>>> s.attr
'a custom class attribute'

As you can see adding attributes to a class correctly results in a class attribute, which is thus shared by any instance of that class (check this post for some explanations about class attributes and sharing).

Class-based decorators without arguments¶

As already explained, Python allows you to call any object (as you do with functions) as long as it provides the __call__() method. So to write a class-based decorator you just need to create an object that defines such a method.

When used as a decorator, a class is instantiated at decoration time, that is when the function is defined, and called when the function is called.

class CustomAttr:
    def __init__(self, obj):
        self.attr = "a custom function attribute"
        self.obj = obj

    def __call__(self):
        self.obj()

As you can see there is already a lot of things that shall be clarified. First of all the class, when used as a decorator, is initialized with the object that is going to be decorated, here called obj (most of the time it is just called f for function, but you know that this is only a special case).

While the __init__() method is called at decoration time, the __call__() method of the decorator is called instead of the same method of the decorated object. In this case (decorator without arguments), the __call__() method of the decorator does not receive any argument. In this example we just "redirect" the call to the original function, which was stored during the initialization step.

So you see that in this case we have two different moments in which we may alter the behaviour of the decorated objects. The first is at its definition and the second is when it is actually called.

The decorator can be applied with the simple syntax shown in a previous section

@CustomAttr
def func():
    pass

When Python parses the file and defines the function func the code it executes under the hood is

def func():
    pass

func = CustomAttr(func)

according to the definition of decorator. This is why the class shall accept the decorated object as a parameter in its __init__() method.

Note that in this case the func object we obtain after the decoration is no more a function but a CustomAttr object

>>> func
<__main__.CustomAttr object at 0xb6f5ea8c>

and this is why in the __init__() method I attached the attr attribute to the class instance self and not to obj, so that now this works

>>> func.attr
'a custom function attribute'

This replacement is also the reason why you shall also redefine __call__(). When you write func() you are not executing the function but calling the instance of CustomAttr returned by the decoration.

Class-based decorators with arguments¶

This case is the most natural step beyond the previous one. Once you started metaprogramming, you want to do it with style, and the first thing to do is to add parametrization. Adding parameters to decorators has the only purpose of generalizing the metaprogramming code, just like when you write parametrized functions instead of hardcoding values.

There is a big caveat here. Class-based decorators with arguments behave in a slightly different way to their counterpart without arguments. Specifically, the __call__() method is run during the decoration and not during the call.

Let us first review the syntax

class CustomAttrArg:
    def __init__(self, value):
        self.value = value

    def __call__(self, obj):
        obj.attr = "a custom function attribute with value {}".format(self.value)
        return obj

@CustomAttrArg(1)
def func():
    pass

Now the __init__() method shall accept some arguments, with the standard rules of Python functions for named and default arguments. The __call__() method receives the decorated object, which in the previous case was passed to the __init__() method.

The biggest change, however is that __call__() is not run when you call the decorated object, but immediately after __init__() during the decoration phase. This results in the following difference: while in the previous case the decorated object was no more itself, but an instance of the decorator, now the decorated objects becomes the return value of the __call__() method.

Remember that, when you call the decorated object, you are now actually calling what you get from __call__() so be sure of returning something meaningful.

In the above example I stored one single argument in __init__(), and this argument is passed to the decorator when applying it to the function. Then, in __call__(), I set the attribute of the decorated object, using the stored argument, and return the object itself. This is important, since I have to return a callable object.

This means that, if you have to do something complex with the decorated object, you may just define a local function that makes use of it and return this function. Let us see a very simple example

class CustomAttrArg:
    def __init__(self, value):
        self.value = value

    def __call__(self, obj):
        def wrap():
            # Here you can do complex stuff
            obj()
            # Here you can do complex stuff
        return wrap

@CustomAttrArg(1)
def func():
    pass

Here the returned object is no more the decorated one, but a new function wrap() defined locally. It is interesting to show how Python identifies it

>>> @CustomAttrArg(1)
... def func():
...     pass
... 
>>> func
<function CustomAttrArg.__call__.<locals>.wrap at 0xb70185cc>

This pattern enables you to do every sort of things with the decorated object. Not only to change it (adding attributes, for example), but also pre- or post- filtering its results. You may start to understand that what we called metaprogramming may be very useful for everyday tasks, and not only for some obscure wizardry.

Decorators and prototypes¶

If you write a class-based decorator with arguments, you are in charge of returning a callable object of choice. There is no assumption on the returned object, even if the usual case is that the returned object has the same prototype as the decorated one.

This means that if the decorated object accepts zero arguments (like in my example), you usually return a callable that accepts zero arguments. This is however by no means enforced by the language, and through this you may push the metaprogramming technique a bit. I'm not providing examples of this technique in this post, however.

Function-based decorators¶

Function-based decorators are very simple for simple cases and a bit trickier for complex ones. The problem is that their syntax can be difficult to grasp at first sight if you never saw a decorator. They are indeed not very different from the class-based decorators with arguments case, as they define a local function that wraps the decorated object and return it.

The case without arguments is always the simplest one

def decorate(f):
    def wrap():
        f()
    return wrap

@decorate
def func():
    pass

This behaves like the equivalent case with classes. The function is passed as an argument to the decorate() function by the decoration process that calls it passing the decorated object. When you actually call the function, however, you are actually calling wrap().

As happens for class-based decorators, the parametrization changes the calling procedure. This is the code

def decorate(arg1):
    def wrap(f):
        def _wrap(arg):
            f(arg + arg1)
        return _wrap
    return wrap

@decorate(1)
def func(arg):
    pass

As you see it is not really straightforward, and this is the reason I preferred to discuss it as the last case. Recall what we learned about class-based decorators: the first call to the decorate() function happens when the decorator is called with an argument. Thus @decorate(1) calls decorate() passing 1 as arg1, and this function returns the wrap() local function.

This second function accepts another function as an argument, and indeed it is used in the actual decoration process, which can be represented by the code func = wrap(func). This wrap() function, being used to decorate func(), wants to return a compatible object, that is in this case a function that accepts a single argument. This is why, in turn, wrap() defines and returns a _wrap() local function, which eventually uses both the argument passed to func() and the argument passed to the decorator.

So the process may be summarized as follows (I will improperly call func_dec the decorated function to show what is happening)

@decorator(1) returns a wrap() function (that knows the argument)
func is redefined as func_dec = wrap(func) becoming _wrap()
When you call func_dec(arg) Python executes _wrap(arg) which calls the original func() passing 1 + arg as argument

Obviously the power of the decorator concept is that you are not dealing with func_dec() but with func() itself, and all the "magic" happens under the hood.

If you feel uncomfortable with function-based decorators don't worry, as they are indeed a bit awkward. I usually stick to function based decorators for the simple cases (setting class attributes, for example), and move to class-based ones when the code is more complex.

Example¶

A good example of the power of decorators which comes out of the box is functools.total_ordering. The functools module provides a lot of interesting tools to push the functional approach in Python, most notably partial() and partialmethod(), which are however out of the scope of this post.

The total_ordering decorator (documented here) wants to make an object provide a full set of comparison ordering methods starting from a small set of them. Comparison methods are those methods Python calls when two objects are compared. For example when you write a == b, Python executes a.__eq__(b) and the same happens for the other five operators > (__gt__), < (__lt__), >= (__ge__), <= (__le__) and != (__ne__).

Mathematically all those operators may be expressed using only one of them and the __eq__() method, for example __ne__ is !__eq__ and __lt__ is __le__ and !__eq__. This decorator makes use of this fact to provide the missing methods for the decorated object. A quick example

class Person:
    def __init__(self, name):
        self.name = name

    def __eq__(self, other):
        return self.name == other.name

    def __lt__(self, other):
        return self.name < other.name

This is a simple class that defines the == and < comparison methods.

>>> p1 = Person('Bob')
>>> p2 = Person('Alice')
>>> p1 == p2
False
>>> p1 < p2
False
>>> p1 >= p2
Traceback (most recent call last):
  File "/home/leo/prova.py", line 103, in <module>
    p1 >= p2
TypeError: unorderable types: Person() >= Person()

A big warning: Python doesn't complain if you try to perform the > and != comparisons but lacking the dedicated methods it does perform a "standard" comparison. This means that, as the documentation states here, "There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false."

With the total_ordering decorator, however, all six comparisons become available

import functools

@functools.total_ordering
class Person:
    def __init__(self, name):
        self.name = name

    def __eq__(self, other):
        return self.name == other.name

    def __lt__(self, other):
        return self.name < other.name

>>> p1 = Person('Bob')
>>> p2 = Person('Alice')
>>> p1 == p2
False
>>> p1 != p2
True
>>> p1 > p2
True
>>> p1 < p2
False
>>> p1 >= p2
True
>>> p1 <= p2
False

Final words¶

Decorators are a very powerful tool, and they are worth learning. This may be for you just the first step into the amazing world of metaprogramming or just an advanced technique you may use to simplify your code. Whatever you do with them be sure to understand the difference between the two cases (with and without arguments) and don't avoid function-based decorators just because their syntax is a bit complex.

Resources¶

Many thanks to Bruce Eckel for his three posts which have been (I think) the source for the page on Python 3 Patterns, Recipes and Idioms (the latter is still work in progress). Update: as Bruce stated "the Python3 Patterns book is kind of a failed project". So beware that information contained there is not going to be updated. It is however still a good starting point for Python studies and investigations.

A good source of advanced decorators can be found at the Python Decorator Library, and a lot of stuff may be found on Stackoverflow under the python-decorators tag.

Graham Dumpleton wrote a very interesting a in-depth analysis of Python decorators here.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 15 - Duplicate the elements of a list a given number of times

2015-04-14T12:30:00+01:00

The problem¶

P15 (**) Duplicate the elements of a list a given number of times.

Example:

scala> duplicateN(3, List('a, 'b, 'c, 'c, 'd))
res0: List[Symbol] = List('a, 'a, 'a, 'b, 'b, 'b, 'c, 'c, 'c, 'c, 'c, 'c, 'd, 'd, 'd)

Initial thoughts¶

Problem 12 already taught me to use fill() to build lists of repeated elements so this could be a good occasion to use it again.

Solution¶

This problem is a generalization of the previous problem 14 which required to duplicate the elements of a list. There we could build the list into an anonymous function with the List(e, e) syntax.

For this problem we have to use a more general solution, which is the fill() method of the List object (not class), already discovered in problem 12. This method is expressed in a curried form, thus the two argument applications.

Again, since the result of fill() is a list, we have to flatten it through flatMap().

def duplicateN[A](n: Int, l: List[A]):List[A] = {
    l flatMap { e => List.fill(n)(e) }
}

Scala allows us to express the anonymous function in a shorter way, making use of the underscore wild card.

def duplicateN2[A](n: Int, l: List[A]):List[A] = {
    l flatMap { List.fill(n)(_) }
}

Final considerations¶

Previous problems gave me all I needed to solve this one. I used anonymous functions and mapping, which are a very important component of Scala. I learned how to simplify an anonymous function using the underscore.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 14 - Duplicate the elements of a list

2015-04-14T12:00:00+01:00

The problem¶

P14 (*) Duplicate the elements of a list.

Example:

scala> duplicate(List('a, 'b, 'c, 'c, 'd))
res0: List[Symbol] = List('a, 'a, 'b, 'b, 'c, 'c, 'c, 'c, 'd, 'd)

Initial thoughts¶

Once again (see problem 12) a perfect task for flatMap() since for each element a list must be produced, but the resulting list shall be flat.

Solution¶

The function passed to flatMap() this time shall return a list with two elements for each element of the source list.

def duplicate[A](l:List[A]):List[A] = {
    l flatMap { e => List(e, e) }
}

Final considerations¶

This problem didn't involve new concepts, but allowed me to test again mapping and anonymous functions.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 13 - Run-length encoding of a list (direct solution)

2015-04-14T11:30:00+01:00

The problem¶

P13 (**) Run-length encoding of a list (direct solution). Implement the so-called run-length encoding data compression method directly. I.e. don't use other methods you've written (like P09's pack); do all the work directly.

Example:

scala> encodeDirect(List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e))
res0: List[(Int, Symbol)] = List((4,'a), (1,'b), (2,'c), (2,'a), (1,'d), (4,'e))

Initial thoughts¶

Obviously the issue shouldn't be solved just copying all stuff from problem 09 and problem 10 inside a single file. There shall be a way to build a single-function solution.

Spanning¶

Scala lists provide a span() method that splits the list in two. It scans all elements in order, storing them into a list while the given predicate (a function) is true. When the predicate is false span() stops and returns the two resulting lists.

It seems to be the perfect candidate for our job. We just have to recursively apply it obtaining two lists. The first list is made by equal elements, and can be reduced to a tuple, the second list is given to the recursive call.

def encodeDirect[A](l: List[A]):List[(Int, A)] = {
    def _encodeDirect(res: List[(Int, A)], rem: List[A]):List[(Int, A)] = rem match {
        case Nil => res
        case ls => {
            val (s, r) = rem span { _ == rem.head }
            _encodeDirect(res:::List((s.length, s.head)), r)
        }
    }
    _encodeDirect(List(), l)
}

The predicate given to span() is _ == rem.head, an anonymous function that tests if the current element is equal to the first element. While this is always true for the first element, by definition, other elements can be equal or not, and the job of span() is to find where to split the list.

So example given in the problem runs through the following steps

scala> var rem = List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e)
rem: List[Symbol] = List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e)

scala> var (res, rem1) = rem span { _ == rem.head }
res: List[Symbol] = List('a, 'a, 'a, 'a)
rem1: List[Symbol] = List('b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e)

scala> var (res, rem2) = rem1 span { _ == rem1.head }
res: List[Symbol] = List('b)
rem2: List[Symbol] = List('c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e)

scala> var (res, rem3) = rem2 span { _ == rem2.head }
res: List[Symbol] = List('c, 'c)
rem3: List[Symbol] = List('a, 'a, 'd, 'e, 'e, 'e, 'e)

scala> var (res, rem4) = rem3 span { _ == rem3.head }
res: List[Symbol] = List('a, 'a)
rem4: List[Symbol] = List('d, 'e, 'e, 'e, 'e)

scala> var (res, rem5) = rem4 span { _ == rem4.head }
res: List[Symbol] = List('d)
rem5: List[Symbol] = List('e, 'e, 'e, 'e)

scala> var (res, rem6) = rem5 span { _ == rem5.head }
res: List[Symbol] = List('e, 'e, 'e, 'e)
rem6: List[Symbol] = List()

Final considerations¶

I learned another very useful method to split Scala lists, span(). I also used a complex expression in a case statement, as already done in problems 09 and 10.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 12 - Decode a run-length encoded list

2015-04-14T11:00:00+01:00

The problem¶

P12 (**) Decode a run-length encoded list. Given a run-length code list generated as specified in problem P10, construct its uncompressed version.

Example:

scala> decode(List((4, 'a), (1, 'b), (2, 'c), (2, 'a), (1, 'd), (4, 'e)))
res0: List[Symbol] = List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e)

Initial thoughts¶

The problem is simple, being just a reverse version of the last two problems. The target is to build a list from another one, so it should be possible to find both a recursive and a functional solution. The only problem is that we may easily build a list for each element, but we shall end up with a flat list. So flatMap() from problem 07 will come in play again.

The recursive solution¶

Since I decided to use flatMap(), which basically acts like map() but then inserts the resulting list into the target list (indeed flattening it), I have to write a function that converts a (Int, A) tuple into a List[A], where A is a type of choice.

The only issue in building lists in Scala is that the ::: operator works with two lists, so we have to build a list with the current head to be able to append it.

def decode[A](l: List[(Int, A)]):List[A] = {
    def _expand(res: List[A], rem:(Int, A)):List[A] = rem match {
        case (0, _) => res
        case (n, h) => _expand(res:::List(h), (n -1, h))
    }

    l flatMap { e => _expand(List(),e) }
}

The last call is a mapping. Each element is mapped into a list by the _expand() function, while flatMap() does the flattening job.

The functional solution¶

There is a simpler way that makes use of a function of the List object. Object and classes are two different entities in Scala, and objects are singletons that encapsulate constructor (and de-constructor) methods.

The fill() method is documented here, and we are interested now in the one-dimensional version. As you can see from the documentation, this function is curried (see problem 04) and accepts as first parameter the length of the list and as second parameter the element to put into the list itself.

We are just interested in repeating an element this time, so the function invocation is very simple

def decode2[A](l: List[(Int, A)]):List[A] = {
    l flatMap { e => List.fill(e._1)(e._2) }
}

Scala tuples may be indexed using the _index notation, starting from 1 (while lists are indexed from 0). Here, just like in the recursive solution, we have to use flatMap() to flatten the resulting list.

Final considerations¶

Writing this solution I renewed my knowledge of flatmapping, learned that Scala classes have companion objects, learned how to create lists and how to index tuples.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 11 - Modified run-length encoding

2015-04-14T10:30:00+01:00

The problem¶

P11 (*) Modified run-length encoding. Modify the result of problem P10 in such a way that if an element has no duplicates it is simply copied into the result list. Only elements with duplicates are transferred as (N, E) terms.

Example:

scala> encodeModified(List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e))
res0: List[Any] = List((4,'a), 'b, (2,'c), (2,'a), 'd, (4,'e))

Initial thoughts¶

The solution shall be a modification of that of problem 10, and the only difference is that this time we do not produce the same type of result for each element.

Choices¶

If the source element is a list with more than an element, we produce a tuple (Int, A), where A is the actual type of elements in the list. If the source element is a list with a single element, we just produce that element, so the type is A.

In Scala this situation may be represented by means of Either, Left and Right. Good explanations of this matter may be found here and here.

Given that, the solution is straightforward

def encode[A](l: List[A]):List[Either[A, (Int, A)]] = {
    utils.packer.pack(l) map {
        e => e match {
            case e if e.length == 1 => Left(e.head)
            case e => Right((e.length, e.head))
        }
    }
}

I prefer to express conditions through pattern matching, but the same code may be expressed with an if/else statement.

Final considerations¶

This time I learned how to use the three types Either, Left and Right. Reading the suggested posts I also dug into Option, Some and None, and the three similar types introduced with Scala 2.10 Try, Success and Failure.

Feedback¶

The GitHub issues page is the best place to submit corrections.

99 Scala Problems 10 - Run-length encoding of a list.

2015-04-14T10:00:00+01:00

The problem¶

P10 (*) Run-length encoding of a list. Use the result of problem P09 to implement the so-called run-length encoding data compression method. Consecutive duplicates of elements are encoded as tuples (N, E) where N is the number of duplicates of the element E.

Example:

scala> encode(List('a, 'a, 'a, 'a, 'b, 'c, 'c, 'a, 'a, 'd, 'e, 'e, 'e, 'e))
res0: List[(Int, Symbol)] = List((4,'a), (1,'b), (2,'c), (2,'a), (1,'d), (4,'e))

Initial thoughts¶

The problem explicitly states "use the result of problem 09" (here) so I think it's time for me to learn how to make and import libraries. The problem itself seems to be rather simple to solve.

The recursive solution¶

Given the availability of the pack() function from problem 09 the recursive solution just has to walk the packed list and convert each element (a list of equal elements) into a tuple. This is the tail recursive version

import utils.packer

def encode[A](l: List[A]):List[(Int, A)] = {
    def _encode(res: List[(Int, A)], rem: List[List[A]]):List[(Int, A)] = rem match {
        case Nil => res
        case h::tail => _encode(res:::List((h.length, h.head)), tail)
    }
    _encode(List(), utils.packer.pack(l))
}

Note that I use the utils.packer.pack() function imported from a package which contains the code developed for problem 09.

To obtain the availability of the pack() function I first created a package. In Scala a package may be declared with the package keyword, but the functions cannot be declared at module-level, they have to be converted into methods of an object. So the content of the utils.scala file is

package utils

object packer {
    def pack[A](l: List[A]):List[List[A]] = {
        def _pack(res: List[List[A]], rem: List[A]):List[List[A]] = rem match {
            case Nil => res
            case ls => {
                val (s, r) = rem span { _ == rem.head }
                _pack(res:::List(s), r)
            }
        }
        _pack(List(), l)
    }
}

In Scala, an object is a singleton, and it is used without instancing it. So the pack() method may be used in this file as packer.pack(). This file is then compiled with the Scala compiler

$ scalac utils.scala

producing a utils directory with the following files

$ ls utils
packer$$anonfun$1.class
packer.class
packer$.class

This package may be directly used by any file in the same directory (e.g. through scala program.scala). If you plan to use it from another directory use the -classpath switch to include the right directory where the utils package may be found. (As an UNIX programmer, I hate single-dash Java long options, but they are here to stay)

Mapping¶

Mapping is, just like folding, a functional technique, because it applies a function to the elements of some collection. In Scala, the map() function of the List type produces a new list processing each element of the original list with the given function.

The function to process elements is very simple, since it has to pack together the length of the element (which is a list) and its head.

import utils.packer

def encode[A](l: List[A]):List[(Int, A)] = {
    utils.packer.pack(l) map { e => (e.length, e.head) }
}

The expression e => (e.length, e.head) is an anonymous function that maps an element into a tuple.

Final considerations¶

With this problem I met packages for the first time, learned how to compile and add paths through classpath. The functional solution introduced me to mapping.

Feedback¶

The GitHub issues page is the best place to submit corrections.