In-Line Route Conflicts with Terraform and AWS

Terraform is an excellent tool for the declarative construction of resources in AWS. This is done by creating a plan out of one or more Terraform files, providing valid credentials, and running terraform apply. Snazzy!

I often include AWS networking resources in my Terraform plans. This includes AWS route tables for subnets and transit gateways. However, route tables can become a little confusing for new users of Terraform. Especially when you read the disclaimer about using a route table with in-line routes:

At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources.
Terraform Documentation

In this post, I’ll break down the construction of route tables, how to use in-line routes, and when to avoid in-line routes. Then, I’ll provide an alternative design using separate route table and route statements. Finally, I’ll describe a use case where using an in-line route provides an attractive solution.

The Terraform AWS Route Table Resource

The aws_route_table resource is used to create a new route table. A route table resource lives within an AWS Virtual Private Cloud (VPC) to supply routing information to one or more VPC subnets. The code is minimal; only a VPC id value must be supplied to build a new route table resource. Optionally, tags can be supplied to provide metadata on the resource, such as the name.

resource "aws_route_table" "vpc-rtb-private" {
  vpc_id = "vpc-0123456789"
  tags = {
    Name = "prod-vpc-rtb-private-01"
  }
}

This will create a new route table for the VPC matching vpc-0123456789 in AWS and name it prod-vpc-rtb-private-01.

A new route table comes to life! It doesn’t do very much right now. The only route is the local target that matches the CIDR of the VPC.

Adding Routes to the Route Table

Let’s suppose you want to make sure the route table resource contains a route to a NAT gateway that lives in a public subnet within the VPC. There are two ways to accomplish this:

In-Line: Edit the route table resource and add the route statement “in-line”.
Separate: Create a new aws_route resource that adds a route to the route table in a separate statement.

In-Line Route Statements in a Route Table

Wouldn’t it be easier to create the route table and also supply the route to your NAT gateway all in one resource statement? Sure! This is what the code would look like:

resource "aws_route_table" "vpc-rtb-private" {
  vpc_id = "vpc-03abfdcf031815266"
  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = "nat-0123456789"
  }  
  tags = {
    Name = "prod-vpc-rtb-private-01"
  }
}

And this is the execution plan:

This works. I now have a route table with the local target and a default route to the NAT gateway.

In-Line Route Statement Conflicts

The problem with this type of resource is that the entire route table and all route entries are now part of a single, combined resource. If you desire to add more routes to the route table in the future, the plan will see this as a configuration drift and purge the added routes.

In this example, I’ve added an additional route to the route table to reach my transit gateway. Any traffic matching 10.120.8.0/21 is now targeting the transit gateway.

Running the Terraform plan reveals the issue. Because the route table was created with an in-line route, the combined object has been altered from the desired state. The execution plan will remove all of the routes and replace them with the in-line route statement. If you are using multiple plans to control routing, this is not a viable solution.

I advocate for using separate statements for the route table resource and route resource in situations where change is anticipated or even possible, especially when multiple plans will be used across the infrastructure landscape.

Separate Route Table and Route Statements

Here is what a Terraform plan with separated route table and route statements might look like. The aws_route_table resource creates the route table while the aws_route resource adds a route to the route table. Because both statements are in the same plan, I’m referencing the route table id by using aws_route_table.vpc-rtb-private.id in the route statement.

resource "aws_route_table" "vpc-rtb-private" {
  vpc_id = "vpc-0123456789"
  tags = {
    Name = "prod-vpc-rtb-private-01"
  }
}
resource "aws_route" "vpc-rtb-private-tgw" {
  route_table_id            = aws_route_table.vpc-rtb-private.id
  destination_cidr_block    = "0.0.0.0/0"
  nat_gateway_id            = "nat-0123456789"
}

This ensures that the route table is treated as a unique resource and all routes added are also treated as unique resources. Making changes to the route table in other Terraform plans, or even by hand (in emergency!), will not cause a conflict with the route table resource.

Use Case for In-Line Route Statements

Knowing how in-line route statements work, there may be design scenarios where this is a viable solution. Perhaps your requirements are to construct a single resource that is immutable to configuration drift. Using in-line route statements makes this possible and tightly controls the configuration of the route table.

For example, in-line routes make sense for a test environment that has a route table in which the routes need to be enforced exactly as originally specified. If any deviation is found, such as from new routes being added manually, a subsequent terraform apply will surgically remove those routes.

Summary

In this post, I described the construction of route tables, how to use in-line routes, and why it’s a good idea to avoid in-line routes. I then provided an example with separate route table and route statements. Finally, a use case was presented showing where an in-line route provides a viable solution.