The Digital Cat - Terraform

Design and implement a flexible VPC on AWS

2023-10-16T11:00:00+01:00

Designing networks is not an easy task, and while cloud computing removes the hassle (and also a bit the fun) of moving around switches and cables, it left untouched the complexity of planning a good structure. But, what does "good structure" mean?

I think this is crucial question in engineering. A well-designed (or well-architected) system cannot be defined once and for all, because its nature, structure, and components depend on the requirements. Hence the usual answer: "it depends".

In this post I want to give an example of some business requirements and of the structure of a network that might satisfy them.

For the implementation I will work with AWS VPC, which has several advantages. First of all, AWS is one of the major cloud providers, and this might help beginners to better understand how it works. Second, most of the components of a VPC are free of charge, which means that anyone can apply the structure I will show without having to pay. The only component that AWS will charge you for is a NAT gateway, but the price is around 0.045 $/hour, which means that with a single dollar you can enjoy a trip on a well-architected VPC for approximately 22 hours. After that time you can always remove the NAT and keep working on the free part of your VPC.

A quick recap of IP and CIDRs¶

You should be familiar with the IP protocol to use VPC effectively, but if your knowledge is rusty I give you a quick recap.

IPv4 addresses are made of 32 bits, thus spanning 2³² values, between 0 and 4,294,967,295 (2³²-1). To simplify their usage, we split IPv4 addresses into 4 chunks of 8 bits (octets) and convert each into a decimal number which is thus between 0 and 255 (2⁸-1). The classic form of an IPv4 address is thus A.B.C.D, e.g. 1.2.3.4, 134.32.175.52, 255.255.0.0.

When considering ranges of addresses, giving the first and last address might be tedious and difficult to read. The CIDR notation was introduced to simplify this. A CIDR (Classless Inter-Domain Routing) is expressed in the form A.B.C.D/N, where N is a number of bits between 0 and 32 and represents how many bits of the address remain fixed. A CIDR like 134.73.28.196/32 represents only the address 134.73.28.196, as 32 bits out of 32 are fixed. Conversely, the CIDR 0.0.0.0/0 represents all IPv4 addresses as 0 bits out of 32 are fixed.

The range of addresses corresponding to a CIDR is in general not easy to compute manually, but those corresponding to the 4 octets are trivial

The CIDR A.B.C.D/32 corresponds to the address A.B.C.D.
The CIDR A.B.C.0/24 corresponds to the addresses between A.B.C.0 and A.B.C.255 (255 addresses, 2^32-24 or 2⁸). Here, the first 24 bits (the 3 octets A, B, and C) are fixed.
The CIDR A.B.0.0/16 corresponds to the addresses between A.B.0.0 and A.B.255.255 (65,536 addresses, 2^32-16 or 2¹⁶). Here, the first 16 bits (the 2 octets A and B) are fixed.
The CIDR A.0.0.0/8 corresponds to the addresses between A.0.0.0 and A.255.255.255 (16,777,216 addresses, 2^32-8 or 2²⁴). Here, the first 8 bits (the octet A) are fixed.

Please note that by convention we set the variable octets to 0. The CIDR A.B.C.0/24 is exactly the same as the CIDR A.B.C.D/24, as the octet D is not fixed. For this reason it's deceiving and useless to set it. For example, I would never write 153.23.95.34/24, as this means all addresses between 153.23.95.0 and 153.23.95.255, so the final 34 is just misleading. 153.23.95.0/24 is much better in this case.

You can use the IP Calculator by Krischan Jodies to explore CIDRs.

As the number of IPv4 addresses quickly proved to be insufficient we developed IPv6, but in the meantime we also created private network spaces. In IPv4 there are 3 different ranges of addresses that are considered "private", which means that they can be duplicated and that they are not reachable from the Internet. The difference between public and private addresses is the same between "London, UK" (there is only one in the world) and "kitchen" (every house has one).

The three private ranges in IPv4 are:

192.168.0.0/16 - 65536 addresses between 192.168.0.0 and 192.168.255.255
172.16.0.0/12 - 1,048,576 addresses between 172.16.0.0 and 172.31.255.255 (this is not easily computed manually because 12 is not a multiple of 8)
10.0.0.0/8 - 16,777,216 addresses between 10.0.0.0 and 10.255.255.255

This means that IP addresses like 192.168.6.1, 172.17.123.45, and 10.34.168.20 are all private. Take care of the second range, as it goes from 172.16 to 172.31, so an address like 172.32.123.45 is definitely public.

Now that your knowledge of IP addresses has been fully restored we can dive into network design.

Requirements¶

As I mentioned in the introduction, the most important part of a system design are requirements, both the present and the future ones.

If you design a road, it is crucial to understand how many vehicles will travel on it per minute (or per hour, day, month) and the type of vehicle. I'm not an expert of highway engineering, but I'm sure a road for mining trucks has to be different from a cycle lane, and the same is true for a computer system. Surely you want to store information in a database, but the size and the type of it depend on the amount of data you have, the usage pattern, the required reliability, and so on.

We are designing a network that will host cloud computing resources such as computing instances, databases, load balancers, and so on. I will globally refer to them as resources or instances, without paying to much attention to the concrete nature of each of them. From the networking point of view they are all just a bunch of network cards.

As an example, we have the following business requirements for a company called ZooSoft:

There are currently three main products: Alligator Accounting, Barracuda Blogging, Coyote CAD.
There might be more products in the future, we are in the first design stages of Dragonfly Draw and Echidna Email.
We need four environments for each product: Live, Staging, Demo, UAT.
- Live is the application accessed by clients
- Staging is a clone of Live that is used to run extensive pre-release tests and to perform initial support debugging
- Demo runs the application with the same configuration as Live but with fake data, used to showcase the application to new customers
- UAT contains on-demand instances used by developers and QA to test new features
Some data or services are shared among the products, and the infrastructure team needs a space where to deploy their tools.

Initial analysis¶

As you see, I highlighted some of the most important points we need to keep in mind.

There are currently 3 products. Not a single one, not 1 hundred. It is important to understand this number because we probably want to have separate spaces for each product, with different teams working on each one. If the company had one single product we might expect it to create a new one in the future, but it might not be that urgent to have space to grow. On the other hand, if the company had already 100 products we might want to design things with a completely different approach.
There might be more products in the future. Again, it is important to have a good idea of the future requirements, as most of the problems of a system will come when the usage patterns change. It's generally a good idea to leave space for growth, but overdoing it might lead to a waste of resources and ultimately money. Understanding the growth expectation is paramount to find a good balance between inflexibility and waste of resources.
There are 4 different usage patterns for each application, each one with its own requirements. The Live environment clearly needs a lot of power and redundancy to provide a stable service for users, while environments like UAT and Demo will certainly have more relaxed parameters in terms of availability or reliability.
We need space to deploy internal tools used to monitor the application and to test new solutions. The architecture of the application might change in the future so we need space to try out different structures and products.

In general, it's a good idea to isolate anything that doesn't need to be shared across teams or products, as it reduces the risk of errors and exposes less resources to attacks. In AWS, the concept of account allows us to completely separate environments at the infrastructure level. Resources in separate accounts can still communicate, but this requires a certain amount of work to set up the connection, which ultimately promotes isolation.

So, the initial idea might be to give each product a different account. However, we also have 4 different environments for each product, and given the relative simplicity involved in the creation of an AWS account it sounds like a good idea to have one of them for each combination of product and environment. AWS provides a tool called Control Tower that can greatly simplify the creation and management of accounts, which makes this choice even more reasonable.

A VPC (Virtual Private Network) is, as the name suggests, a private network that allows different products to use the same IP address pool without clashing, which is not new to anyone is familiar with private IP address spaces. This means that we could easily create in each account a VPC with a CIDR 10.0.0.0/8 that grants 2²⁴ (more than 16M) different IP addresses, plenty enough to host instances and databases for any type of application.

However, it might be useful in the future to connect different VPCs, for example to perform data migrations, and this is done in AWS through VPC peering. In simple words, this is a way to create a single network out of two different VPCs, but it can't be done if the two VPCs have overlapping CIDR blocks. This means that while we keep VPCs separate in different accounts, we might also want to assign different CIDRs to each one.

Avoiding overlap clearly reduces the size of a VPC, so let's have a look at some figures to have an idea of what we can create.

If we assign to each account a CIDR 10.X.0.0/16, with X being a number assigned to the specific account, we can create up to 256 different accounts (from 10.0.0.0/16 to 10.255.0.0/16). Out of an abundance of caution we might reserve the first 10 CIDRs for future use and internal needs, which leaves us with 246 non-overlapping CIDRs (from 10.10.0.0/16 to 10.255.0.0/16). This means that we have space for several combinations of product/environment, for example we might have up to 41 products with 6 environments each or 30 products with 8 environments each (with leftovers).

Since at the moment we have 3 products with 4 environments each, this choice looks reasonable. At the same time, a /16 CIDR grants us space for 216 (65536) resources, which again looks more than enough to host a standard web application.

Assignment plan¶

To simplify the schema, let's grant space for 20 products and group CIDRs by environment. This means we will have 20 CIDRs for Live environments, 20 for Staging, and so on. The assignment plan is then

10.0.0.0/16  reserved
...
10.9.0.0/16  reserved

10.10.0.0/16 alligator-accounting-live
10.11.0.0/16 barracuda-blogging-live
10.12.0.0/16 coyote-cad-live
...

10.30.0.0/16 alligator-accounting-staging
10.31.0.0/16 barracuda-blogging-staging
10.32.0.0/16 coyote-cad-staging
...

10.50.0.0/16 alligator-accounting-demo
10.51.0.0/16 barracuda-blogging-demo
10.52.0.0/16 coyote-cad-demo
...

10.70.0.0/16 alligator-accounting-uat
10.71.0.0/16 barracuda-blogging-uat
10.72.0.0/16 coyote-cad-uat
...

10.250.0.0/16 infrastructure-team
...
10.255.0.0/16 infrastructure-team

As I mentioned, the initial CIDRs are reserved for future use, but we also kept the final 6 CIDRs for the needs of the infrastructure team. Keep in mind that this is only an example and that we are clearly free to change any of these figure to match our needs more closely. Each one of these CIDRs will be assigned to a specific account.

Should we create new products we will continue with the same pattern, e.g.

10.0.0.0/16  reserved
...

10.10.0.0/16 alligator-accounting-live
10.11.0.0/16 barracuda-blogging-live
10.12.0.0/16 coyote-cad-live
10.13.0.0/16 dragonfly-draw-live
...

10.30.0.0/16 alligator-accounting-staging
10.31.0.0/16 barracuda-blogging-staging
10.32.0.0/16 coyote-cad-staging
10.33.0.0/16 dragonfly-draw-staging
...

10.50.0.0/16 alligator-accounting-demo
10.51.0.0/16 barracuda-blogging-demo
10.52.0.0/16 coyote-cad-demo
10.53.0.0/16 dragonfly-draw-demo
...

10.70.0.0/16 alligator-accounting-uat
10.71.0.0/16 barracuda-blogging-uat
10.72.0.0/16 coyote-cad-uat
10.73.0.0/16 dragonfly-draw-uat
...

10.250.0.0/16 infrastructure-team
...

We are planning to use IaC tools to implement this, but it's nevertheless interesting to spot the patterns in this schema that make it easier to debug network connections.

All environments of a certain type belong to a specific range, so an address like 10.15.123.456 is definitely in a Live environment. At the same time, IP addresses across the same product have the same final digit in the second part of the address, so if 10.12.456.789 is a Live instance, the corresponding Staging instance will have an address like 10.32.X.Y.

While this is not crucial, I wouldn't underestimate the value of a regular structure that can give precious information at a glance. While debugging during an emergency things like this might be a blessing.

The last thing to note is that in this schema the 160 CIDRs between 10.90.0.0/16 and 10.249.0.0/16 are not allocated. This might give you a better idea of how wide a 10.0.0.0/8 network space is! Such accounts can be used to host up to other 8 environments for each product.

Address space¶

Let's focus on a single CIDR in the form 10.N.0.0/16. As we know this provides 65536 addresses (2¹⁶) that we need to split into subnets. In AWS, subnets correspond to different Availability Zones, that are "distinct locations within an AWS Region that are engineered to be isolated from failures in other Availability Zones" (from the docs). In other words, they are separate data centres built so that if one blows up the others should be unaffected. I guess this depends on the size of the explosion, but within reason this is the idea.

So, each account gets 65536 addresses (/16), split into:

1 public subnet for the NAT gateway to live in (nat).
3 private subnets for the computing resources (private_a, private_b, private_c).
3 public subnets for the load balancer (public_a, public_b, public_c).
1 public subnet for the bastion instance (bastion).

Now, if you are not familiar with subnets they are the simplest of concepts. You get the address space of a network (say for example 10.10.0.0/16, that is the addresses from 10.10.0.0 to 10.10.255.255) and split it into chunks. The fact that each chunk is assigned to a different data centre is an AWS addition and is not part of the definition of subnet in principle. However, the reason behind subnetting is exactly to create small physical networks that are therefore more efficient. If two computers are on the same subnet the routing of IP packets exchanged by them is simpler and thus faster. For similar reasons, and to increase security, it's a good idea to keep your subnets as small as possible.

In this case, we might create subnets in a /23 space (512 addresses each), which looks wide enough to host the web applications of ZooSoft. Before we have a look at the actual figures let me clarify what this means. I assume each application (Alligator Accounting, Barracuda Blogging, and so on) has been containerised, maybe using ECS or EKS, which however means that there are EC2 instances behind the scenes running the containers. If we are using Fargate we do not provide EC2 instances and in that case we might set up our network in a different way.

EC2 instances are computers, and they all have at least one network interface, which corresponds to an IP address. So, when I say that a subnet contains 512 addresses I mean that in a single subnet I can run up to 507 EC2 instances (remember that AWS reserves some addresses, see https://docs.aws.amazon.com/vpc/latest/userguide/subnet-sizing.html). Assuming instances with 8 GiB of memory each (e.g. m7g.large) and containers that require 1 GiB of memory we can easily host 3042 containers (507*6) leaving 2 GiB for each instance to host newly created containers (for example to run blue-green deployments). These are clearly examples and you have to adapt them to the requirements of your specific application, but I hope you get an idea of how to roughly estimate this sort of quantities.

Remember that in AWS the difference between public and private networks is only in the gateway they are connected to. Public networks are connected to an Internet Gateway and thus are reachable from Internet, while private networks are either disconnected from Internet or connected with a NAT, which allows them to access Internet but not to be accessed from outside.

The bastion subnet might or might not be useful. In general, bastion hosts are very secure instances that can be accessed using SSH, and from which you can access the rest of the instances. As from the point of view of security they are a weak point of the whole infrastructure you might not want to have them or replace them with more ephemeral solutions. In any case, I left the network there as an example of a space that hosts tools not directly connected with the application.

Let's have a deeper look at the figures. A /16 space can be split into 128 /23 spaces (2^23-16), but given the list of subnets I showed before we need only 8 of them, which leaves again a lot of space for further expansion, and there are two types of expansion we might consider. One is increasing the number of subnets, the other is increasing the size of subnets themselves. With the amount of space granted by the current size of the networks we have plenty of options to cover both cases. We might reach a good balance between the size of the network and the number of networks increasing the potential size to /21 (2048 addresses), which grants us space for 32 subnetworks.

Here, I show a possible schema for the account alligator-accounting-live that is granted the space 10.10.0.0/16.

NAME       CIDR           ADDRESSES                     NUM ADDRESSES

reserved   10.10.0.0/21   (10.10.0.0  - 10.10.7.255)    {2048}

nat        10.10.8.0/23   (10.10.8.0  - 10.10.9.255)    {512}
	   expandable to
           10.10.8.0/21   (10.10.8.0  - 10.10.15.255)   {2048}
           PUBLIC

reserved   10.10.16.0/21  (10.10.10.0 - 10.10.15.255)   {2048}

reserved   10.10.24.0/21  (10.10.10.0 - 10.10.31.255)   {2048}

private-a  10.10.32.0/23  (10.10.32.0 - 10.10.33.255)   {512}
	   expandable to
           10.10.32.0/21  (10.10.32.0 - 10.10.39.255)   {2048}

private-b  10.10.40.0/23  (10.10.40.0 - 10.10.41.255)   {512}
	   expandable to
           10.10.40.0/21  (10.10.40.0 - 10.10.47.255)   {2048}

private-c  10.10.48.0/23  (10.10.48.0 - 10.10.49.255)   {512}
	   expandable to
           10.10.48.0/21  (10.10.48.0 - 10.10.55.255)   {2048}

reserved   10.10.56.0/21  (10.10.56.0 - 10.10.63.255)   {2048}
           RESERVED PRIVATE 4

reserved   10.10.63.0/21  (10.10.63.0 - 10.10.71.255)   {2048}
           RESERVED PRIVATE 5

...

reserved   10.10.128.0/21 (10.10.128.0 - 10.10.135.255) {2048}
           RESERVED PRIVATE 13

public-a   10.10.136.0/23 (10.10.136.0 - 10.10.137.255) {512}
	   expandable to
           10.10.136.0/21 (10.10.136.0 - 10.10.143.255) {2048}
           PUBLIC

public-b   10.10.144.0/23 (10.10.144.0 - 10.10.145.255) {512}
	   expandable to
           10.10.144.0/21 (10.10.144.0 - 10.10.151.255) {2048}
           PUBLIC

public-c   10.10.152.0/23 (10.10.152.0 - 10.10.153.255) {512}
	   expandable to
           10.10.152.0/21 (10.10.152.0 - 10.10.159.255) {2048}
           PUBLIC

reserved   10.10.160.0/21 (10.10.160.0 - 10.10.167.255) {2048}
           RESERVED PUBLIC 4

reserved   10.10.168.0/21 (10.10.167.0 - 10.10.175.255) {2048}
           RESERVED PUBLIC 5

...

reserved   10.10.232.0/21 (10.10.232.0 - 10.10.239.255) {2048}
           RESERVED PUBLIC 13

bastion    10.10.240.0/23(10.10.240.0 - 10.10.241.255)  {512}
	   expandable to
           10.10.240.0/21 (10.10.240.0 - 10.10.247.255) {2048}
           PUBLIC

reserved   10.10.248.0/21(10.10.248.0 - 10.10.255.255)  {2048}

Routing¶

Routing of each VPC is very simple:

All resources in the private subnets will be routed into the NAT to grant them Internet access but to isolate them from outside.
All resources in the public subnets will be routed into the default Internet Gateway.
The NAT network has to be public, so it is routed into the default Internet Gateway.
The bastion subnet is public, so it routed into the default Internet Gateway.

The NAT is a device that translates internet addresses, hiding the internal ones through some clever hacking of TCP/IP. This means that it has to live in a public network so that it can access the Internet.

The bastion (if present) is a machine that can be accessed from Internet, so it has to be in a public subnet. It's customary to grant access to the bastion to a specific set of IPs (e.g. the personal IPs of some developers) but this is done through Security Groups.

Relevant figures¶

In summary the current design grants us the following:

246 non-overlapping accounts
20 different products each with 12 environments
32 subnets for each account
512 addresses per subnet, upgradable to 2048 without overlapping

A simple Terraform module¶

The following code is a simple Terraform module intended to showcase how to create a well-designed VPC with that tool. I decided to avoid using complex loops or other clever hacks to keep it simple and accessible to anyone might be moving their first steps into AWS, VPC, network design, and Terraform. You are clearly free to build on top of it and to come up with a different or more clever implementation.

Remember that the NAT is the only resource that is not free of charge, so don't leave it up and running if don't use it. Don't be afraid of creating one and having a look in the AWS console though.

I assume the following files are all created in the same directory that I will conventionally call modules/vpc.

VPC

modules/vpc/vpc.tf

resource "aws_vpc" "main" {
  cidr_block = "${var.cidr_prefix}.0.0/16"

  tags = {
    Name = var.name
  }
}

modules/vpc/variables.tf

variable "cidr_prefix" {
  description = "The first two octets of the CIDR, e.g. 10.10 (will become 10.10.0.0/16)"
  type        = string
}

variable "name" {
  description = "The name of this VPC and the prefix/tag for its related resources"
  type        = string
}

When you call the module you will have to pass these two variables, e.g.

alligator-accounting-live/vpc/main.tf

module "vpc" {
  source = "../../modules/vpc"

  cidr_prefix = "10.10"
  name        = "alligator-accounting-live"
}

Internet Gateway

modules/vpc/gateway.tf

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id

  tags = {
    Name = var.name
  }
}

Subnets

modules/vpc/subnets.tf

resource "aws_subnet" "nat" {
  vpc_id     = aws_vpc.main.id
  cidr_block = "${var.cidr_prefix}.8.0/23"

  availability_zone = "eu-west-1a"

  tags = {
    Name = "${var.name}-nat"
  }
}

resource "aws_subnet" "private_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.32.0/23"
  availability_zone = "eu-west-1a"

  tags = {
    Name = "${var.name}-private-a"
    Tier = "private"
  }
}

resource "aws_subnet" "private_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.40.0/23"
  availability_zone = "eu-west-1b"

  tags = {
    Name = "${var.name}-private-b"
    Tier = "private"
  }
}

resource "aws_subnet" "private_c" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.48.0/23"
  availability_zone = "eu-west-1c"

  tags = {
    Name = "${var.name}-private-c"
    Tier = "private"
  }
}

resource "aws_subnet" "public_a" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.136.0/23"
  availability_zone = "eu-west-1a"
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.name}-public-a"
    Tier = "public"
  }
}

resource "aws_subnet" "public_b" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.144.0/23"
  availability_zone = "eu-west-1b"
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.name}-public-b"
    Tier = "public"
  }
}

resource "aws_subnet" "public_c" {
  vpc_id            = aws_vpc.main.id
  cidr_block        = "${var.cidr_prefix}.152.0/23"
  availability_zone = "eu-west-1c"
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.name}-public-c"
    Tier = "public"
  }
}

resource "aws_subnet" "bastion" {
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "${var.cidr_prefix}.240.0/23"
  availability_zone       = "eu-west-1a"
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.name}-bastion"
    Tier = "bastion"
  }
}

We need to associate the public subnets with the Internet Gateway.

modules/vpc/gateway.tf

[...]

resource "aws_route_table" "main" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name = "${var.name} Internet Gateway"
  }
}


resource "aws_route_table_association" "public_a_to_igw" {
  subnet_id      = aws_subnet.public_a.id
  route_table_id = aws_route_table.main.id
}

resource "aws_route_table_association" "public_b_to_igw" {
  subnet_id      = aws_subnet.public_b.id
  route_table_id = aws_route_table.main.id
}

resource "aws_route_table_association" "public_c_to_igw" {
  subnet_id      = aws_subnet.public_c.id
  route_table_id = aws_route_table.main.id
}

resource "aws_route_table_association" "bastion_to_igw" {
  subnet_id      = aws_subnet.bastion.id
  route_table_id = aws_route_table.main.id
}

NAT Gateway

We need a NAT to grant private networks access to the Internet.

modules/vpc/nat.tf

resource "aws_eip" "nat_gateway" {
  vpc = true

  tags = {
    Name = var.name
  }
}

resource "aws_route_table_association" "nat_to_igw" {
  subnet_id      = aws_subnet.nat.id
  route_table_id = aws_route_table.main.id
}

resource "aws_nat_gateway" "nat_gateway" {
  allocation_id = aws_eip.nat_gateway.id
  subnet_id     = aws_subnet.nat.id

  depends_on = [aws_internet_gateway.main]

  tags = {
    Name = var.name
  }
}

resource "aws_route_table" "nat_gateway" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gateway.id
  }

  tags = {
    Name = "${var.name} NAT Gateway"
  }
}

resource "aws_route_table_association" "private_a_to_nat" {
  subnet_id      = aws_subnet.private_a.id
  route_table_id = aws_route_table.nat_gateway.id
}

resource "aws_route_table_association" "private_b_to_nat" {
  subnet_id      = aws_subnet.private_b.id
  route_table_id = aws_route_table.nat_gateway.id
}

resource "aws_route_table_association" "private_c_to_nat" {
  subnet_id      = aws_subnet.private_c.id
  route_table_id = aws_route_table.nat_gateway.id
}

Subnet groups¶

As an optional step, you might want to create subnet groups for RDS. In AWS, you can create RDS instances in public networks out of the box, but if you want to put them in a private network (and you want to put them there) you need to build a subnet group. See the documentation.

modules/vpc/subnets.tf

resource "aws_db_subnet_group" "rds_group" {
  name = "rds_private"
  subnet_ids = [
    aws_subnet.private_a.id,
    aws_subnet.private_b.id,
    aws_subnet.private_c.id
  ]

  tags = {
    Name = "${var.name} RDS subnet group"
  }
}

Final words¶

I hope this was a useful and interesting trip into network design. As an example, this might sound trivial and simple compared to what is needed in certain contexts, but it is definitely a good setup that you can build on. I think VPC is often overlooked as it is assumed developers are familiar with networks. As networking is a crucial part of a system and will pop up in other technologies like Docker or Kubernetes, I recommend any mid-level or senior developer to make sure they are familiar with the main concepts of IP. Happy learning!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Photo by Nathan Dumlao on Unsplash.

AWS Log Insights as CloudWatch metrics with Python and Terraform

2021-03-22T17:00:00+01:00

Recently I started using AWS CloudWatch Log Insights and I find the tool really useful to extract data about the systems I'm running without having to set up dedicated monitoring tools, which come with their own set of permissions, rules, configuration language, and so forth.

Log Insights allow you to query log outputs with a language based on regular expressions with hints of SQL and to produce tables or graphs of quantities that you need to monitor. For example, the system I am monitoring runs Celery in ECS containers that log received tasks with a line like the following

16:39:11,156 [32mINFO [0m [34m[celery.worker.strategy][0m [01mReceived task: lib.tasks.lists.trigger_list_log_notification[9b33b464-d4f9-4909-8d4e-1a3134fead97] [0m

In this case the specific function in the system that was triggered is lib.tasks.log_notification, and I'm interested in knowing which functions are called the most, so I can easily count them with

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) as number by task
 | sort number desc
 | limit 9

This gives me a nice table of the top 9 source functions and the number of task submitted for each, and the time frame can be adjusted with the usual CloudWatch controls

1 lib.tasks.lists.trigger_list_log_notification 4559
2 lib.tasks.notify.notify_recipient 397
3 lib.message._send_mobile_push_notification 353
4 lib.tasks.jobs.check_job_cutoffs 178
5 lib.tasks.notify.check_message_cutoffs 177
6 lib.tasks.notify.check_notification_retry 177
7 lib.tasks.notify.async_list_response 81
8 lib.tasks.hmrc_poll.govtalk_periodic_poll 59
9 lib.tasks.lists.recalculate_list_entry 56

Using time bins, quantities can also be easily plotted. For example, I can process and visualise the number of received tasks with

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) by bin(30s)

Unfortunately I quickly discovered an important limitation of Log Insights, that is queries are not metrics. Which also immediately implies that I can't set up alarms on those queries. As fun as it is to look at nice plots, I need something automatic that sends me messages or scales up systems in reaction to specific events such as "too many submitted tasks".

The standard solution to this problem suggested by AWS is to write a Lambda that runs the query and stores the value into a custom CloudWatch metric, which I can then use to satisfy my automation needs. I did it, and in this post I will show you exactly how, using Terraform, Python and Zappa, CloudWatch, and DynamoDB. At the end of the post I will also briefly discuss the cost of the solution.

The big picture¶

Before I get into the details of the specific tools or solutions that I decided to implement, let me have a look at the bigger picture. The initial idea is very simple: a Lambda function can run a specific Log Insights query and store the results in a custom metric, which can in turn be used to trigger alarms and other actions.

For a single system I already have 4 or 5 of these queries that I'd like to run, and I have multiple systems, so I'd prefer to have a solution that doesn't require me to deploy and maintain a different Lambda for each query. The maintenance can be clearly automated as well, but such a solution smells of duplicated code miles away, and if there is no specific reason to go down that road I prefer to avoid it.

Since Log Insights queries are just strings of code, however, we can store them somewhere and then simply loop on all of them within the same Lambda function. To implement this, I created a DynamoDB table and every element contains all the data I need to run each query, such as the log group that I want to investigate and the name of the target metric.

Terraform¶

In the following sections I will discuss the main components of the solution from the infrastructural point of view, showing how I created them with Terraform. The four main AWS services that I will use are: DynamoDB, Lambda, IAM, CloudWatch.

I put the bulk of the code in a module so that I can easily create the same structure for multiple AWS accounts. While my current setup is a bit more complicated that that, the structure of the code can be simplified as

+ common
  + lambda-loginsights2metrics
    + cloudwatch.tf
    + dynamodb.tf
    + iam.tf
    + lambda.tf
    + variables.tf
+ account1
  + lambda-loginsights2metrics
    + main.tf
    + variables.tf

Variables

Since I will refer to them in the following sections, let me show you the four variables I defined for this module.

First I need to receive the items that I need to store in the DynamoDB table

common/lambda-loginsights2metrics/variables.tf

variable "items" {
  type    = "list"
  default = []
}

I prefer to have a prefix in front of my components that allows me to duplicate them without clashes

common/lambda-loginsights2metrics/variables.tf

variable "prefix" {
  type    = "string"
  default = "loginsights2metrics"
}

The Lambda function will require a list of security groups that grant access to specific network components

common/lambda-loginsights2metrics/variables.tf

variable "security_groups" {
  type    = "list"
  default = []
}

Finally, Lambda functions need to be told which VPC subnets they can use to run

common/lambda-loginsights2metrics/variables.tf

variable "vpc_subnets" {
  type    = "list"
  default = []
}

Resources

Terraform variables.
An in-depth post that explains how to use variables in Terraform, by Sumeet Ninawe

DynamoDB

Let's start with the corner stone, which is the DynamoDB table that contains data for the queries. As DynamoDB is not a SQL database we don't need to define columns in advance. This clearly might get us into trouble later, so we need to be careful and be consistent when we write items, adding everything is needed by the Lambda code.

common/lambda-loginsights2metrics/dynamodb.tf

resource "aws_dynamodb_table" "loginsights2metrics" {
  name = "${var.prefix}-items"

  billing_mode = "PAY_PER_REQUEST"

  hash_key = "SlotName"

  attribute {
    name = "SlotName"
    type = "S"
  }
}

Speaking of items, I assume I will pass them when I call the module, so here I just need to loop on the input variable items

common/lambda-loginsights2metrics/dynamodb.tf

resource "aws_dynamodb_table_item" "item" {
  count = length(var.items)

  table_name = aws_dynamodb_table.loginsights2metrics.name
  hash_key   = aws_dynamodb_table.loginsights2metrics.hash_key

  item = jsonencode(element(var.items, count.index))
}

Since the query is written as a Terraform string and will be read from Python there are two small caveats here. To be consistent with Terraform's syntax we need to escape double quotes in the query, and to avoid fights with Python we need to escape backslashes. So for example a valid query like

parse @message /\[celery\.(?<source>[a-z.]+)\].*Received task: (?<task>[a-z._]+)\[/
 | filter not isblank(source)
 | stats count(*) as Value by bin(1m)

will be stored as

"parse @message /\\[celery\\.(?<source>[a-z.]+)\\].*Received task: (?<task>[a-z._]+)\\[/ | filter not isblank(source) | stats count(*) as Value by bin(1m)"

Another remark is that the Lambda I will write in Python will read data plotted with the name Value on bins of 1 minute, so the query should end with stats X as Value by bin(1m) where X is a specific stat, for example stats count(*) as Value by bin(1m).

The reason behind 1 minute is that the maximum standard resolution of CloudWatch metrics is 1 minute. Should you want more you need to have a look at CloudWatch High-Resolution Metrics.

Resources

IAM part 1

IAM roles are central in AWS. In this specific case we have the so-called Lambda execution role, which is the IAM role that the Lambda assumes when you run it. In AWS users or services (that is humans or AWS components) assume a role, receiving the permissions connected with it. To assume roles, however, they need to have a specific permission, a so-called trust policy.

Let's define a trust policy that allows the Lambda service to assume the role that we will define

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "trust" {
  statement {

    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"
      identifiers = [
        "lambda.amazonaws.com"
      ]
    }
  }
}

and after that the role in question

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role" "loginsights2metrics" {
  name               = var.prefix
  assume_role_policy = data.aws_iam_policy_document.trust.json
}

To run, Lambdas need an initial set of permissions which can be found in the canned policy AWSLambdaVPCAccessExecutionRole. You can see the content of the policy in the IAM console or dumping it with aws iam get-policy and aws iam get-policy-version

$ aws iam get-policy --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole
{
    "Policy": {
        "PolicyName": "AWSLambdaVPCAccessExecutionRole",
        "PolicyId": "ANPAJVTME3YLVNL72YR2K",
        "Arn": "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole",
        "Path": "/service-role/",
        "DefaultVersionId": "v2",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "Description": "Provides minimum permissions for a Lambda function to execute while accessing a resource within a VPC - create, describe, delete network interfaces and write permissions to CloudWatch Logs. ",
        "CreateDate": "2016-02-11T23:15:26Z",
        "UpdateDate": "2020-10-15T22:53:03Z"
    }
}
$ aws iam get-policy-version --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole --version-id v2
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents",
                        "ec2:CreateNetworkInterface",
                        "ec2:DescribeNetworkInterfaces",
                        "ec2:DeleteNetworkInterface",
                        "ec2:AssignPrivateIpAddresses",
                        "ec2:UnassignPrivateIpAddresses"
                    ],
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v2",
        "IsDefaultVersion": true,
        "CreateDate": "2020-10-15T22:53:03Z"
    }
}

Attaching a canned policy is just a matter of creating a specific aws_iam_role_policy_attachment resource

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy_attachment" "loginsights2metrics-" {
  role       = aws_iam_role.loginsights2metrics.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

Now that we have the IAM role and the basic policy we can assign custom permissions to it. We need to grant the Lambda permissions on other AWS components, namely CloudWatch to run Log Insights queries and to store metrics and DynamoDB to retrieve all the items from the queries table.

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "loginsights2metrics" {
  statement {
    actions = [
      "cloudwatch:PutMetricData",
      "cloudwatch:PutMetricAlarm",
      "logs:StartQuery",
      "logs:GetQueryResults",
      "logs:GetLogEvents",
    ]

    resources = ["*"]
  }

  statement {
    actions = [
      "dynamodb:Scan"
    ]

    resources = [aws_dynamodb_table.loginsights2metrics.arn]
  }
}

Through aws_iam_role_policy we can create and assign the policy out of a data structure

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy" "loginsights2metrics" {
  name   = var.prefix
  role   = aws_iam_role.loginsights2metrics.name
  policy = data.aws_iam_policy_document.loginsights2metrics.json
}

Resources

Lambda

We can now create the Lambda function container. I do not use Terraform as a deployer, as I think it should be used to define static infrastructure only, so I will use a dummy function here and later deploy the real code using the AWS CLI.

The dummy function can be easily created with

common/lambda-loginsights2metrics/lambda.tf

data "archive_file" "dummy" {
  type        = "zip"
  output_path = "${path.module}/lambda.zip"

  source {
    content  = "dummy"
    filename = "dummy.txt"
  }
}

The Lambda function is a bit more complicated. As I mentioned, I'll use Zappa to package the function, so the handler has to be "zappa.handler.lambda_handler". The IAM role given to the function is the one we defined previously, while memory_size and timeout clearly depend on the specific function. Lambdas should run in private networks, and I won't cover here the steps to create them. The AWS docs contains a lot of details on this topic, e.g. https://aws.amazon.com/premiumsupport/knowledge-center/internet-access-lambda-function/.

The environment variables allow me to inject the name of the DynamoDB table so that I don't need to hardcode it. I also pass another variable, the Sentry DSN that I use in my configuration. This is not essential for the problem at hand, but I left it there to show how to pass such values.

common/lambda-loginsights2metrics/lambda.tf

resource "aws_lambda_function" "loginsights2metrics" {
  function_name = "loginsights2metrics"
  handler       = "zappa.handler.lambda_handler"
  runtime       = "python3.8"
  filename      = data.archive_file.dummy.output_path

  role        = aws_iam_role.loginsights2metrics.arn
  memory_size = 128
  timeout     = 300

  vpc_config {
    subnet_ids         = var.vpc_subnets
    security_group_ids = var.security_groups
  }

  environment {
    variables = {
      "SENTRY_DSN"     = "https://XXXXXX:@sentry.io/YYYYYY",
      "DYNAMODB_TABLE" = aws_dynamodb_table.loginsights2metrics.name
    }
  }

  lifecycle {
    ignore_changes = [last_modified, filename]
  }
}

Please note that I instructed Terraform to ignore changes to the two attributes last_modified and filename, and that I haven't used any source_code_hash. This way I can safely apply Terraform to change parameters like memory_size or timeout without affecting what I deployed with the CI.

Since I want to trigger the function from AWS CloudWatch Events I need to grant the service events.amazonaws.com the lambda:InvokeFunction permission.

common/lambda-loginsights2metrics/lambda.tf

resource "aws_lambda_permission" "loginsights2metrics" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.loginsights2metrics.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.rate.arn
}

Resources

IAM part 2

Since 2018 Lambdas have a maximum execution time of 15 minutes (900 seconds), which is more than enough for many services, but to be conservative I preferred to leverage Zappa's asynchronous calls and to make the main Lambda call itself for each query. The Lambda doesn't clearly call the same Python function (it's not recursive), but from AWS's point of view we have a Lambda that calls itself, so we need to give it a specific permission to do this.

common/lambda-loginsights2metrics/iam.tf

data "aws_iam_policy_document" "loginsights2metrics_exec" {
  statement {
    actions = [
      "lambda:InvokeAsync",
      "lambda:InvokeFunction"
    ]

    resources = [aws_lambda_function.loginsights2metrics.arn]
  }
}

I could not define this when I defined the rest of the IAM components because this needs the Lambda to be defined, but the resource is in the same file. Terraform doesn't care about which resource we defined first and where we define it as long as there are no loops in the definitions.

We can now assign the newly created policy document to the IAM role we created previously

common/lambda-loginsights2metrics/iam.tf

resource "aws_iam_role_policy" "loginsights2metrics_exec" {
  name   = "${var.prefix}-exec"
  role   = aws_iam_role.loginsights2metrics.name
  policy = data.aws_iam_policy_document.loginsights2metrics_exec.json
}

Resources

aws_iam_policy_document documentation documentation")
aws_iam_role_policy documentation

CloudWatch

Whenever you need to run Lambdas (or other things) periodically, the standard AWS solution is to use CloudWatch Events, which work as the AWS cron system. CloudWatch Events are made of rules and targets, so first of all I defined a rule that gets triggered every 2 minutes

common/lambda-loginsights2metrics/cloudwatch.tf

resource "aws_cloudwatch_event_rule" "rate" {
  # Zappa requires the name to match the processing function
  name                = "main.loginsights2metrics"
  description         = "Trigger Lambda ${var.prefix}"
  schedule_expression = "rate(2 minutes)"
}

Please note that Zappa has a specific requirement for CloudWatch Events, so I left a comment to clarify this to my future self. The second part of the event is the target, which is the Lambda function that we defined in the previous section.

common/lambda-loginsights2metrics/cloudwatch.tf

resource "aws_cloudwatch_event_target" "lambda" {
  rule      = aws_cloudwatch_event_rule.rate.name
  target_id = "${var.prefix}-target"
  arn       = aws_lambda_function.loginsights2metrics.arn
}

Resources

Using the module

Now the module is finished, so I just need to create some items for the DynamoDB table and to call the module itself

account1/lambda-loginsights2metrics/main.tf

locals {
  items = [
    {
      "SlotName" : {
        "S" : "Celery Logs submitted tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery\\.(?<source>[a-z.]+)\\].*Received task: (?<task>[a-z._]+)\\[/ | filter not isblank(source) | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Submitted tasks"
      }
    },
    {
      "SlotName" : {
        "S" : "Celery Logs succeeded tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery.(?<source>[a-z\\._]+)].*Task (?<task>[a-z\\._]+)\\[.*\\] (?<event>[a-z]+)/ | filter source = \"app.trace\" | filter event = \"succeeded\" | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Succeeded tasks"
      }
    },
    {
      "SlotName" : {
        "S" : "Celery Logs retried tasks"
      },
      "LogGroup" : {
        "S" : "mycluster/celery",
      },
      "ClusterName" : {
        "S" : "mycluster"
      },
      "Query" : {
        "S" : "parse @message /\\[celery.(?<source>[a-z\\._]+)].*Task (?<task>[a-z\\._]+)\\[.*\\] (?<event>[a-z]+)/ | filter source = \"app.trace\" | filter event = \"retry\" | stats count(*) as Value by bin(1m)",
      },
      "Namespace" : {
        "S" : "Custom"
      },
      "MetricName" : {
        "S" : "Retried tasks"
      }
    }
  ]
}

I need to provide a security group for the Lambda, and in this case I can safely use the default one provided by the VPC

account1/lambda-loginsights2metrics/main.tf

data "aws_security_group" "default" {
  name   = "default"
  vpc_id = var.vpc_id
}

And I can finally call the module

account1/lambda-loginsights2metrics/main.tf

module "loginsights2metrics" {
  source = "../../common/lambda-loginsights2metrics"

  items           = local.items
  security_groups = [data.aws_security_group.default.id]
  vpc_subnets     = var.vpc_private_subnets
}

Please note that the variable vpc_private_subnets is a list of subnet names that I created in another module.

Resources

Python¶

As I mentioned before, the Python code of the Lambda function is contained in a different repository and deployed with the CI using Zappa. Given we are interacting with AWS I am clearly using Boto3, the AWS SDK for Python. The code was developed locally without Zappa's support, to test out the Boto3 functions I wanted to use, then quickly adjusted to be executed in a Lambda.

I think the code is pretty straightforward, but I left my original comments to be sure everything is clear.

import os
import time
import json
from datetime import datetime, timedelta

import boto3
from zappa.asynchronous import task

# CONFIG
logs = boto3.client("logs", region_name="eu-west-1")
cw = boto3.client("cloudwatch", region_name="eu-west-1")
dynamodb = boto3.resource("dynamodb", region_name="eu-west-1")


@task
def put_metric_data(item): 3
    slot_name = item["SlotName"]
    log_group = item["LogGroup"]
    cluster_name = item["ClusterName"]
    query = item["Query"]
    namespace = item["Namespace"]
    metric_name = item["MetricName"]

    # This runs the Log Insights query fetching data
    # for the last 15 minutes.
    # As we deal with logs processing it's entirely possible
    # for the metric to be updated, for example because
    # a log was received a bit later.
    # When we put multiple values for the same timestamp
    # in the metric CW can show max, min, avg, and percentiles.
    # Since this is an update of a count we should then always
    # use "max".
    start_query_response = logs.start_query( 4
        logGroupName=log_group,
        startTime=int((datetime.now() - timedelta(minutes=15)).timestamp()),
        endTime=int(datetime.now().timestamp()),
        queryString=query,
    )

    query_id = start_query_response["queryId"]

    # Just polling the API. 5 seconds seems to be a good
    # compromise between not pestering the API and not paying
    # too much for the Lambda.
    response = None
    while response is None or response["status"] == "Running":
        print(f"{slot_name}: waiting for query to complete ...")
        time.sleep(5)
        response = logs.get_query_results(queryId=query_id)

    # Data comes in a strange format, a dictionary of
    # {"field":name,"value":actual_value}, so this converts
    # it into something that can be accessed through keys
    data = []
    for d in response["results"]: 5
        sample = {}
        for i in d:
            field = i["field"]
            value = i["value"]

            sample[field] = value

        data.append(sample)

    # Now that we have the data, let's put them into a metric.
    for d in data:
        timestamp = datetime.strptime(d["bin(1m)"], "%Y-%m-%d %H:%M:%S.000")
        value = int(d["Value"])

        print(f"{slot_name}: putting {value} on {timestamp}")

        cw.put_metric_data( 6
            Namespace=namespace,
            MetricData=[
                {
                    "MetricName": metric_name,
                    "Dimensions": [{"Name": "Cluster", "Value": cluster_name}],
                    "Timestamp": timestamp,
                    "Value": value,
                    "Unit": "None",
                }
            ],
        )


def loginsights2metrics(event, context): 1
    with open("package_info.json", "r") as f:
        package_info = json.load(f)
        build_timestamp = int(package_info["build_time"])
        build_datetime = datetime.fromtimestamp(build_timestamp)

    print("###################################")
    print(
        "LogInsights2Metrics - Build date: "
        f'{build_datetime.strftime("%Y/%m/%d %H:%M:%S")}'
    )
    print("###################################")

    print(f'Reading task from DynamoDB table {os.environ["DYNAMODB_TABLE"]}')

    table = dynamodb.Table(os.environ["DYNAMODB_TABLE"])

    # This is the simplest way to get all entries in the table
    # The next loop will asynchronously call `put_metric_data`
    # on each entry.
    response = table.scan(Select="ALL_ATTRIBUTES") 2

    for i in response["Items"]:
        print(f"* Processing item {i['SlotName']}")
        put_metric_data(i)

So, when the Lambda is executed, the entry point is the function loginsights2metrics 1 which queries the DynamoDB table 2 and loops over all the items contained in it. The loop executes the function put_metric_data 3 which being a Zappa task runs it in a new Lambda invocation. This function runs the Log Insights query 4, adjusts Boto3's output 5, and finally puts the values in the custom metric 6.

The problem I mention in the comment just before I run logs.start_query is interesting. Log Insights are queries, and since they extract data from logs the result can change between two calls of the same query. This means that, since there is an overlap between calls (we run a query on the last 15 minutes every 2 minutes), the function will put multiple values in the same bin of the metric. This is perfectly normal, and it's the reason why CloudWatch allows you to show the maximum, minimum, average, or various percentiles of the same metric. When it comes to counting events, the number can only increase or stay constant in time, but never decrease, so it's sensible to look at the maximum. This is not true if you are looking at execution times, for example, so pay attention to the nature of the underlying query when you graph the metric.

The Zappa settings I use for the function are

zappa_settings.json

{
  "main": {
    "app_module": "main",
    "app_function": "main.loginsights2metrics",
    "runtime": "python3.8",
    "log_level": "WARNING",
    "xray_tracing": true,
    "exception_handler": "zappa_sentry.unhandled_exceptions"
  }
}

And the requirements are

requirements.txt

zappa
zappa-sentry

Please note that as I mentioned before zappa-sentry is not a strict requirement for this solution.

The code can be packaged and deployed with a simple bash script like

#!/bin/bash

VENV_DIRECTORY=venv
LAMBDA_PACKAGE=lambda.zip
REGION=eu-west-1
FUNCTION_NAME=loginsights2metrics

if [[ -d ${VENV_DIRECTORY} ]]; then rm -fR ${VENV_DIRECTORY}; fi
if [[ -f ${LAMBDA_PACKAGE} ]]; then rm -fR ${LAMBDA_PACKAGE}; fi

python -m venv ${VENV_DIRECTORY}
source ${VENV_DIRECTORY}/bin/activate

pip install -r requirements.txt

zappa package main -o ${LAMBDA_PACKAGE}

rm -fR ${VENV_DIRECTORY}

aws --region=${REGION} lambda update-function-code --function-name ${FUNCTION_NAME}  --zip-file "fileb://${LAMBDA_PACKAGE}"

Costs¶

I will follow here the AWS guide on Lambda pricing and the calculations published in 2018 by my colleague João Neves on his blog.

I assume the following:

The Lambda runs 4 queries, so we have 5 invocations (1 for the main Lambda and 4 asynchronous tasks)
Each invocation runs for 5 seconds. The current average time of each invocation in my AWS accounts is 4.6 seconds
I run the Lambda every 2 minutes

Requests: 5 invocations/event * 30 events/hour * 24 hours/day * 31 days/month = 111600 requests

Duration: 0.128 GB/request * 111600 requests * 5 seconds = 71424 GB-second

Total: $0.20 * 111600 / 10^6 + $0.0000166667 * 71424 ~= $1.22/month

As you can see, for applications like this it's extremely convenient to use a serverless solution like Lambda functions.

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Emacs Configuration for Python/JavaScript, Terraform and blogging

2020-07-18T15:30:00+01:00

(image from https://commons.wikimedia.org/wiki/File:Gnu-listen-half.jpg)

I have been an Emacs user for a long time. There is no specific reason why I started using Emacs: it was available in the RedHat 6.0 distribution that I found in a magazine in 1999, and with which I started my journey in the open source world. It was mentioned in some Linux guide I read at the time, so it became my editor.

I'm not into flame wars, in particular about editors. If I don't like a software/operating system/language/whatever, I just don't use it, and at the same time I'm not scared to test alternatives, even though I'm not actively looking to replace tools that work. Admittedly, at the time I didn't properly configure my Emacs for years, in particular because the C language support was very good out of the box, and that was what I needed, so when I started programming in Python not everything was optimal.

One day a new colleague showed me Sublime Text and PyCharm. I don't like IDEs that much, they are too integrated, so the PyCharm wasn't very attractive, but Sublime Text is a very good editor, it's fast and has a lot of features (multiple cursors blew my mind when I first discovered them) and so it became my editor of choice for some years. In time, however I started growing increasingly dissatisfied with it, and with some alternatives that I tested like Atom. The main reason is that modern editor rely too much on the mouse: many people are happy with this, in particular because they use trackpads, but I honestly can't get use to them, and I simply don't want to take my hands off the keyboards while I code because I want to change tab, reorganise the screen, open a file, and so on.

So I went back to Emacs, and started from scratch to configure it in order to match my needs. In this post I want to describe what I did, so that newcomers might be helped to setup their own editor. Emacs is incredibly customisable, and this is one of its main strengths, but a proper setup takes time. Please don't consider configuring your editor a waste of time. If you are a programmer, the editor is your main tool, and you have to take care of it: like any other editor, Emacs has pros and cons, and a proper setup minimises the impact of the shortcomings on your daily work.

Requirements¶

As always, the choice of tools and their configuration depends on the requirements, so these are the ones I have at the moment.

I need to create keyboard shortcuts for everything I do often in the editor, I should be able to work without the mouse
Multiple cursors are a very useful feature and I am used to have them, so there should be an implementation similar to the one I found in Sublime Text
Python/JavaScript/Terraform syntax highlighting and formatting/linting
As I have some projects written with React, JSX syntax highlighting is also needed
Markdown syntax highlighting with spell-checking for my blogger activity

Preamble¶

The Emacs configuration file is ~/.emacs, and when you install the editor you might get a default minimal version of it. Whenever part of the configuration is changed from the menus, Emacs writes the changes in the .emacs, using the custom-set-variables function. That file also contains the list of packages that I have installed, so I moved the custom-set-variables invocation to a separate file, .emacs-custom. This way Emacs doesn't have to change the main file whenever I install or remove a package and when I customise the face (colours and fonts).

For packages, I'm using the standard MELPA configuration

;; Added by Package.el.  This must come before configurations of
;; installed packages.  Don't delete this line.  If you don't want it,
;; just comment it out by adding a semicolon to the start of the line.
;; You may delete these explanatory comments.
(package-initialize)

(setq custom-file "~/.emacs-custom")
(load custom-file)

;; Load package system and add MELPA repository.
(require 'package)
(add-to-list
 'package-archives
 ;; '("melpa" . "https://stable.melpa.org/packages/") ; many packages won't show if using stable
 '("melpa" . "https://melpa.milkbox.net/packages/")
 t)

Global settings¶

In Emacs you can define new functions and assign them to key combinations, and this is usually a major part of the setup. As I often need to work on the configuration file, at least initially, the first function I defined is one that allows me to quickly open the .emacs file. In the comments I'm following Emacs key notation

;; Open .emacs with C-x c
(defun dotemacs () (interactive) (switch-to-buffer (find-file-noselect "~/.emacs")))
(global-set-key (kbd "C-x c") 'dotemacs)

I like the visual line mode to wraps long lines

;; Visual line mode everywhere, please
(global-visual-line-mode t)

I also want to always see line numbers

;; Display line numbers
(global-display-line-numbers-mode)

I use C-End a lot to reach the end of the file, but often I press C-<next> (Ctrl-PgDown) by mistake. That is by default associated with scroll-left which is disabled, but Emacs opens a buffer to warn me. I don't want that, so I just unset the key combination

;; I keep pressing C-next by mistake...
(global-unset-key (kbd "C-<next>"))

I might not be a purist here. I used M-w and C-y for many years, but I also use other software during my day, and now the Ctrl-c/x/v combinations are universally implemented. It's really hard to remap my brain every time I go back to Emacs, so I prefer to remap Emacs. Welcome Cua Mode, which also maps undo to C-z

;; Set C-c, C-x, C-v just to be in sync with the rest of the world
(cua-mode t)

I like Emacs to highlight the matching parenthesis whenever I'm on an opening or closing one. Please note that the word "parenthesis" here refers to what are more generally called brackets in English, which includes parentheses or round brackets (), curly brackets or curly braces {}, and square brackets []

;; Show the matching parenthesis
(show-paren-mode 1)

Speaking of brackets, I like to have a way to automatically match the opened ones. This is good in Lisp because there are more parentheses than words, but also in other languages like Python when dealing with complex data structures. Long story short, I mapped C-] to the auto-closing feature called syntactic-close (installed as a package)

;; Syntactic close
;; https://github.com/emacs-berlin/syntactic-close
(global-set-key (kbd "C-]") 'syntactic-close)

Minibuffer is a very important part of Emacs, not only when you open files but also when running commands. I like the Ivy completion, so I installed it with MELPA and activate it. I want to have a quick access to the commands that I run previously (history), but since <down> and <up> are already used to navigate the suggestions I mapped the history functions to their Shift version.

;; Ivy completion
;; https://github.com/abo-abo/swiper
(ivy-mode 1)
(define-key ivy-minibuffer-map (kbd "S-<up>") #'ivy-previous-history-element)
(define-key ivy-minibuffer-map (kbd "S-<down>") #'ivy-next-history-element)

The standard behaviour of C-Del in Emacs is far too greedy for me. I'm used to have a combination that deletes all spaces if I'm on a space, and the word if I'm on a word, so I found this

;; https://stackoverflow.com/questions/17958397/emacs-delete-whitespaces-or-a-word
(defun kill-whitespace-or-word ()
  (interactive)
  (if (looking-at "[ \t\n]")
      (let ((p (point)))
        (re-search-forward "[^ \t\n]" nil :no-error)
        (backward-char)
        (kill-region p (point)))
    (kill-word 1)))
(global-set-key (kbd "C-<delete>") 'kill-whitespace-or-word)

Last, as I have a rotating wallpaper with screenshots from my favourite films I like to see them behind the editor and the terminal, so I enable a little bit of transparency. I like this to be callable as an interactive function as I might want to remove the transparency, for example when sharing the screen, as that way the text might be easier to read.

;; Set transparency of emacs
;; https://www.emacswiki.org/emacs/TransparentEmacs
(defun transparency (value)
  "Sets the transparency of the frame window. 0=transparent/100=opaque"
  (interactive "nTransparency Value 0 - 100 opaque:")
  (set-frame-parameter (selected-frame) 'alpha value))
(set-frame-parameter (selected-frame) 'alpha '90)

Keyboard mapping¶

I'm an Italian native speaker, so I often write texts in that language for my personal blog or for other reasons. In Italian, we use the accented letters à, è, é, ì, ò, ù a lot ("is" is just "è", "why" and "because" are "perché", almost all future tenses end with an accented letter, and so on), so I like to have a proper mapping. Emacs has a very powerful mode for Latin alphabet diacritics, which is latin-postfix, where you can write "è" typing e followed by backticks, so that was my first choice. While this is very powerful, and allows me to input almost everything I need also for other languages like German, on the long run I found it very difficult to use efficiently.

First of all, I'm used to the Italian keymap, I know it by heart as I know the English keyboard mapping, so I tend to press keys that are supposed to insert the accented letters directly. Moreover, other software like the browser work with a system keyboard mapping (that Emacs ignores), so again, I need to rewire my brain every time I go back to the editor. Last, latin-postfix is far too generic and has mappings for letters like ą or ę which are created typing "a," which is a combination that we have every time we type a comma in Italian given that almost all words end in a vowel. This forces me to type "a,," whenever I need "a,", which is far too different from the normal system I'm used to.

The input method italian-keyboard unfortunately doesn't map very well on a standard modern English keyboard, so I decided to just write my own mapping using the quail system, which is unsurprisingly not very well documented. I'm sorry to say it, but multilingual input has been and still is one of the great forgotten and messy topics in computer science, in particular in the open source world. Let's not even mention Chinese and other languages not based on an alphabet.

; Define a proper mapping for the Italian keyboard over the English one
(quail-define-package
 "italian-english-keyboard" "Latin-1" "IT@" t
 "Italian (Italiano) input method for modern English keyboards"
 nil t t t t nil nil nil nil nil t)

;; \|  1!  2"  3£  4$  5%  6&  7/  8(  9)  0=  '?  ì^
;;      qQ  wW  eE  rR  tT  yY  uU  iI  oO  pP  èé  +*
;;       aA  sS  dD  fF  gG  hH  jJ  kK  lL  òç  à°  ù§
;;    <>  zZ  xX  cC  vV  bB  nN  mM  ,;  .:  -_

;; èè -> È Originally CapsLock+è
;; àà -> # Originally AltGr+à
;; òò -> @ Originally AltGr+ò
;; ìì -> ~ Originally AltGr+ì

(quail-define-rules
 ("`" ?\\) ("¬" ?|)
 ("^" ?&)
 ("&" ?/)
 ("*" ?\()
 ("(" ?\))
 (")" ?=)
 ("-" ?') ("_" ??)
 ("=" ?ì) ("+" ?^) ("==" ?~)

 ("[" ?è) ("{" ?é) ("[[" ?È)
 ("]" ?+) ("}" ?*)

 (";" ?ò) (":" ?ç) (";;" ?@)
 ("'" ?à) ("@" ?\°) ("''" ?#)
 ("#" ?ù) ("~" ?§)

 ("\\" ?<) ("|" ?>)
 ("<" ?\;)
 (">" ?:)
 ("/" ?-) ("?" ?_)
)

The standard Italian keyboard allows to input È activating CapsLock, pressing the key for è and deactivating CapsLock. This because Shift-è inputs é. I always found that system incredibly awkward, not to mention that it is impossible (as far as I understand) to create such a combination using quail, so I mapped it to èè, since that combination doesn't exist in Italian.

Another issue is that the Italian keyboard makes use of the right Alt key (AltGr) to input third level characters like #, @, and ~. Again, I don't know if and how I can express these in quail so I went for a slight variation of that. Instead of typing AltGr+à for # I will type àà and so on. This is not a perfect mapping, but considering that I don't use those character that much when writing notes or blog posts I can accept it.

Lines and regions¶

This section is about shortcuts to interact with lines and regions. First of all something that comments the regions I highlighted or just the line I'm on. This is a good example of something that Emacs doesn't provide out of the box, maybe surprisingly, but that is easily implemented. OK, "easily" might be an overstatement, but this function is a good starting point if you want to learn how Emacs Lisp works. If you are not interested you can just copy and paste it and call it a day

;; Define C-/ to comment and uncomment regions and lines
(defun comment-or-uncomment-line-or-region ()
    "Comments or uncomments the region or the current line if there's no active region."
    (interactive)
    (let (beg end)
        (if (region-active-p)
            (setq beg (region-beginning) end (region-end))
            (setq beg (line-beginning-position) end (line-end-position)))
        (comment-or-uncomment-region beg end)
        (next-line)))
(global-set-key (kbd "C-/") 'comment-or-uncomment-line-or-region)

I want to be able to move up and down lines or regions using Ctrl-Shift-up/down, that is to drag stuff

;; Drag-stuff - Drag lines and regions
(drag-stuff-global-mode 1)
;; Use C-S-up/down
(setq drag-stuff-modifier '(control shift))
(define-key drag-stuff-mode-map (drag-stuff--kbd 'up) 'drag-stuff-up)
(define-key drag-stuff-mode-map (drag-stuff--kbd 'down) 'drag-stuff-down)

I also want to be able to duplicate the current line or region with Ctrl-Shift-d. This function is a mixture of several solutions I found on Internet, and I'm not 100% satisfied with it, as when I duplicate a region it also duplicates the last line of the region itself, even though the line is not highlighted. Again, it might be surprising that Emacs doesn't provide a solution out-of-the-box, but this way I can implement the behaviour I prefer. The fact that I'm not able to implement the exact behaviour (yet!) is part of the game, I think.

;; Duplicate line with C-S-d
(defun duplicate-current-line-or-region (arg)
  "Duplicates the current line or region ARG times.
If there's no region, the current line will be duplicated. However, if
there's a region, all lines that region covers will be duplicated."
  (interactive "p")
  (let (beg end (origin (point)))
    (if (and mark-active (> (point) (mark)))
        (exchange-point-and-mark))
    (setq beg (line-beginning-position))
    (if mark-active
        (exchange-point-and-mark))
    (setq end (line-end-position))
    (let ((region (buffer-substring-no-properties beg end)))
      (dotimes (i arg)
        (goto-char end)
        (newline)
        (insert region)
        (setq end (point)))
      (goto-char (+ origin (* (length region) arg) arg)))))
(global-set-key (kbd "C-S-d") 'duplicate-current-line-or-region)

Multiple cursors! Emacs has an implementation of multiple cursors, that is slightly different from the Sublime Text one, but that can be customised to my needs. Most notably, multiple cursors don't work perfectly with some parts of the editor like TAB for indentation, which is not automatically applied to all of them. So, this part of the setup is not perfect, but it's definitely usable and effective.

Ctrl-d creates a second cursor on the next repetition of the current word, Ctrl-c Ctrl-d marks all the repetitions in the text, Ctrl-Alt-up/down add a new cursor on the previous/following line, and last Ctrl-Alt-d marks all the lines in a region

;; Multiple cursors
(global-set-key (kbd "C-d") 'mc/mark-next-like-this-word)
(global-set-key (kbd "C-c C-d") 'mc/mark-all-words-like-this)
(global-set-key (kbd "C-M-<up>") 'mc/mark-previous-lines)
(global-set-key (kbd "C-M-<down>") 'mc/mark-next-lines)
(global-set-key (kbd "C-M-d") 'mc/edit-lines)

How many times do I need to highlight the whole buffer? More than I imagined, so I defined the classic Ctrl-a shortcut

;; C-a marks the whole buffer
(global-set-key (kbd "C-a") 'mark-whole-buffer)

Windows and buffers¶

As everyone else, when I code I need to keep multiple files open, and often to have them side-by-side. Emacs provides out of the box the window split shortcuts Ctrl-2 (horizontal split) and Ctrl-3 (vertical split), but I need a shortcut to quickly move between them. Enter Windmove, which by default uses Shift-<arrow> combinations, which I instead want to keep for highlighting regions, so I remapped it to Alt-<arrow>

;; Windmove - move between windows use ALT
(windmove-default-keybindings 'meta)
;; Unset Shift-arrow keybindings
(global-unset-key (vector (list 'shift 'left)))
(global-unset-key (vector (list 'shift 'right)))
(global-unset-key (vector (list 'shift 'up)))
(global-unset-key (vector (list 'shift 'down)))

Next I need to move between buffers (open files not currently visible) Emacs used buffers also for its own internal logs, and all those buffers have a name surrounded by *, so I filter them out when I move through open files

;; Skip system buffers when cycling
(set-frame-parameter (selected-frame) 'buffer-predicate
             (lambda (buf) (not (string-match-p "^*" (buffer-name buf)))))

To move between buffers I map previous-buffer and next-buffer to Ctrl-Super-left/right (Super here is my left Win key, which is between Ctrl and Alt)

;; Map previous and next buffer to C-s-
(global-set-key (kbd "C-s-<left>") 'previous-buffer)
(global-set-key (kbd "C-s-<right>") 'next-buffer)

Oh, I love the golden ratio mode that splits window giving more space to the current one

;; Golden ratio mode
(require 'golden-ratio)
(golden-ratio-mode 1)

Spell checking¶

When I will learn how to properly spell inconceivably and counter-intuitive I might get rid of this. In the meanwhile a good spell check is a blessing from heaven. I haven't used function keys that much so far, and since I'm used to the standard Sublime Text mapping F6 I reused that.

;; Spell check
(global-set-key (kbd "<f6>") 'flyspell-mode)

;; Spell check with hunspell
(when (executable-find "hunspell")
  (setq-default ispell-program-name "hunspell")
  (setq ispell-really-hunspell t))

By the way, I misspell simpler words all the time, no need to get into inconceivably counter-intuitive words. And I'm sure there will be typos in this text, no matter how many times I check and read it =)

Python development¶

At last, let's get into development! Emacs has a very good package for Python, called Elpy.

;; Enable Elpy for Python development
;; https://elpy.readthedocs.io/en/latest/
(setq elpy-rpc-python-command "python3")
(elpy-enable)

There are some adjustment needed to prevent the mode to override the Windmove and other binding

;; Prevent Elpy from overriding Windmove shortcuts
(eval-after-load "elpy"
  '(cl-dolist (key '("M-<up>" "M-<down>" "M-<left>" "M-<right>"))
     (define-key elpy-mode-map (kbd key) nil)))

;; Prevent Elpy from overriding standard cursor movements
(eval-after-load "elpy"
  '(cl-dolist (key '("C-<left>" "C-<right>"))
     (define-key elpy-mode-map (kbd key) nil)))

Since I use black to format Python code I run it on save. Plus I have a shortcut to fix the syntax in the buffer at any time. Since the syntax document is PEP8 C-8 is a nice combination.

;; Run black on save
(add-hook 'elpy-mode-hook (lambda ()
  (add-hook 'before-save-hook 'elpy-black-fix-code nil t)))

;; Set C-8 to format Python code
(global-set-key (kbd "C-8") 'elpy-black-fix-code)

JavaScript development¶

Let's go full-stack and add some JavaScript and JSX to the recipe. The rjsx-mode is perfect for React development and similar things

;; JSX
;;https://github.com/felipeochoa/rjsx-mode
(add-to-list 'auto-mode-alist '("\\.js\\'" . rjsx-mode))
;; Don't override my beloved multiple cursors
(with-eval-after-load 'rjsx-mode
  (define-key rjsx-mode-map (kbd "C-d") nil))

As I will install Node.js modules locally in node_modules I want code formatters and other tools to run the binaries stored there

;; Use binaries in node_modules
;; https://github.com/codesuki/add-node-modules-path
(add-hook 'js2-mode-hook 'add-node-modules-path)

And also JS need to be beautified, at least when I have to read and understand it. I use prettier and prettier-emacs

;; Pretty JS code
;; https://github.com/prettier/prettier-emacs
(require 'prettier-js)
(add-hook 'js2-mode-hook 'prettier-js-mode)

Other languages¶

Other languages I use are less demanding in term of customisation (generally because they are simpler or less general-purpose). The Terraform mode automatically fixes the syntax of the file on save

;; Terraform
;; https://github.com/emacsorphanage/terraform-mode
(require 'terraform-mode)
(add-to-list 'auto-mode-alist '("\\.tf\\'" . terraform-mode))
(add-hook 'terraform-mode-hook #'terraform-format-on-save-mode)

;; YAML
(require 'yaml-mode)
(add-to-list 'auto-mode-alist '("\\.ya?ml\\'" . yaml-mode))

Files and projects¶

My setup is still a work in progress here, I'm testing Projectile, Prescient, and Treemacs. So far, only Projectile made it to the configuration file, but I'm not using it that much yet. I think I might use a good sidebar with Git integration through colours and icons, but I still have to find something I feel comfortable with.

;; Projectile
;; https://docs.projectile.mx/projectile/usage.html
(require 'projectile)
(define-key projectile-mode-map (kbd "s-p") 'projectile-command-map)
(projectile-mode +1)
(setq projectile-completion-system 'ivy)

The custom file¶

As I mentioned when I discussed the preamble the file .emacs-custom contains the customisation of variables and faces, and this is what I have

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(ansi-color-faces-vector
   [default default default italic underline success warning error])
 '(custom-enabled-themes (quote (tango-dark)))
 '(default-input-method (quote italian-english-keyboard))
 '(inhibit-startup-screen t)
 '(js-indent-level 2)
 '(package-selected-packages
   (quote
    (syntactic-close ivy projectile rjsx-mode js2-mode golden-ratio
     add-node-modules-path prettier-js tide elpy jinja2-mode drag-stuff
     markdown-mode fcitx command-log-mode yaml-mode terraform-mode multiple-cursors))))
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(flymake-error ((t (:foreground "orange red" :background "yellow"))))
 '(js2-external-variable ((t (:foreground "orange red" :background "yellow")))))

As you can see I have a default theme tango-dark. The default alternate input method activated by C-\ is the one I defined previously, italian-english-keyboard. JavaScript uses 2 spaces for indentation, and last I changed some colours to make errors in Python and JavaScript stand out very clearly.

Final words¶

Any configuration is a work in progress, as it changes with the personal preferences and the requirements, but this has been my configuration for a while now, so it seems pretty stable. I hope this might help whoever wants to try emacs or who is already using it but struggles to properly configure it. Happy coding!

Feedback¶

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.