Continuous Integration with GitHub Actions and Terraform

In my earlier Terraform Plans, Modules, and Remote State post, I described the evolution from a simple Terraform plan to a more complex module with remote state. However, each step was performed at the console using the Terraform CLI. While this works, it would be advantageous to leverage a Continuous Integration (CI) workflow to plan and apply my changes in a more automated and collaborative manner.

In this post, I’m exploring GitHub Actions as a CI workflow that will build and maintain a backend Amazon Web Sevices (AWS) Simple Storage Service (S3) bucket to store Terraform state files based on this example. I’ll start by generating a new GitHub repository, then write the GitHub Workflow files, and finally start testing the CI workflow and introduce a small change.

Update 2020-05-28: I talk about GitHub Actions on the Day Two Cloud podcast. Have a listen!

If you’d like to skip ahead, all of the source code and examples supplied in this post are available in this repository.

I’d like to share a special thank you to Kyle Ruddy at HashiCorp for publishing this Automate Infrastructure Provisioning Workflows with the GitHub Action for Terraform blog post – it was helpful! I plan to kick the tires with Terraform Cloud in an upcoming article.

Introduction to Actions

GitHub Actions is a hosted runner service provided by GitHub. Any user can write individual tasks, called actions, and put them together into a workflow. These workflows can trigger off numerous events, such as pull requests, comments, labels, releases, and so forth. I think of it as having a box of LEGO bricks that can be put together as needed; I can build a rocket ship or a pirate ship as my heart desires.

Users are free to write their own actions or consume them from the GitHub Marketplace. For example, the action that performs code checkout is written by GitHub and is on the Marketplace. For a more in-depth introduction to GitHub Actions, I suggest reading the Getting started with GitHub Actions documentation.

For the purpose of this article, I am using GitHub Actions to construct a workflow to provide CI-like functionality. This is not, however, the only use case.

Creating a GitHub Repository

To begin, I need to setup a GitHub repository to store my Terraform configuration and run various GitHub Actions as part of a workflow. I use Terraform and the GitHub provider to build and maintain my Wahl Network repositories.

The resource below constructs the repository with the correct license, topics, and description:

resource "github_repository" "github-action-terraform" {
  name          = "github-action-terraform"
  description   = "An example of Continuous Integration with GitHub Actions and HashiCorp Terraform"
  private       = false
  homepage_url  = "https://wahlnetwork.com"
  has_projects  = false
  has_wiki      = false
  has_downloads = false
  license_template = "mit"
  topics = ["example", "public", "ci", "continuous-integration", "terraform", "github", "github-actions"]
}

Once applied, a new GitHub repository named github-action-terraform exists.

Storing Secrets in the GitHub Repository

Terraform requires credentials to access the backend S3 bucket and AWS provider. I use the Terraform GitHub provider to push secrets into my GitHub repositories from a variety of sources, such as encrypted variable files or HashiCorp Vault.

A simplified example of this is shown below:

resource "github_actions_secret" "github-action-terraform-access-key" {
  repository       = "github-action-terraform"
  secret_name      = "AWS_ACCESS_KEY_ID"
  plaintext_value  = "ABCDEFG"
}
resource "github_actions_secret" "github-action-terraform-secret-key" {
  repository       = "github-action-terraform"
  secret_name      = "AWS_SECRET_ACCESS_KEY"
  plaintext_value  = "1234567890"
}

Once run, the GitHub repository contains the two secrets used to construct AWS credentials needed by the future CI workflow.

Next, it’s time to write the workflow files.

Writing the GitHub Workflow Files

The entirety of GitHub Workflows are driven by yaml files. These files contain information on when to trigger a run, what actions to perform, and other requirements for CI. The files are stored in .github/workflows.

In my scenario, I want two things to happen:

1. Plan: When a pull request is received, a GitHub Workflow is triggered to perform a security audit, construct AWS credentials, load the Terraform CLI, and perform an init and plan using the proposed files in the pull request. Finally, I want a comment on the pull request to show the results of the plan step.
2. Apply: When a push is received in the master branch, I can assume that the proposal was accepted and the new code should be applied. I again wish to construct AWS credentials and load the Terraform CLI. However, the final steps will be to init and apply the configuration.

Each of these steps will be represented with a unique workflow file that contains numerous actions.

The first yaml file I’ve written is tf-plan.yml. The file includes copious amounts of comments that guide users through the steps. It will trigger any time a pull request is received by the repository.

The second yaml file I’ve written is tf-apply.yml. It also contains copious amounts of comments to guide users through the steps. It will only trigger when a push is made to the master branch.

Note: I highly suggest using specific versions with GitHub Actions. For example, I use actions/[email protected] to specify the release version instead of actions/[email protected]. This helps avoid a breaking change effecting my code.

With these two workflow files written, it is time to make the initial commit into the repository.

Loading The Initial Commit

I’ve started the process of loading my configuration files by making a branch named init and pushing it up to the repository. GitHub provides a notification stating that a new branch is detected and offers to create a pull request, which I do.

Once the pull request is generated, a new check is triggered. This is the tf-plan.yml file being triggered by the pull request event.

I can view the workflow process in real time by visiting the Actions menu or by clicking the Details button next to the queued check. This shows all of the individual actions that comprise this workflow.

Checking for Terraform Security Vulnerabilities

An additional benefit of using a CI workflow is adding tests. In this scenario, I’ve added a step leveraging tfsec to scan for static code vulnerabilities. In the example below, tfsec warns against creating an AWS S3 bucket without logging enabled. This will halt and fail the workflow unless I provide an ignore comment to accept the warning.

Posting Comments to the Pull Request

During the workflow, the action named Run – Terraform Comment will post a comment to the pull request with the results of the plan step. Because this is an existing Terraform configuration, there are no changes needed. I can now accept this pull request and merge it into the master branch.

Merging the Pull Request into Master

Once the pull request is merged into master, the tf-apply.yml workflow is triggered. This is because code has been pushed into the master branch.

There are no changes to the resources because the plan is already current. The repository now contains an up-to-date copy of the Terraform plan, has correct and valid AWS credentials, and has the ability to plan and apply the configuration. I will now introduce a small amount of change.

Changing the Terraform Configuration

As a huge fan of The Hitchhiker’s Guide to the Galaxy, I tend to drop little easter eggs into my code. In this scenario, I want to add a new tag to my AWS S3 bucket. The key is Answer and the value is 42.

To keep things simple, I am editing the main.tf file directly from GitHub as shown below:

Rather than commit the change directly to the master branch, I’ve created a new branch named chore-update-s3-bucket-tag and submitted a pull request. This triggers the tf-plan.yml workflow, along with a comment from the GitHub Actions bot showing the change.

This looks good to me! I’ll accept the pull request and merge it into master. I then delete the chore branch for good hygiene.

Applying the Terraform Configuration Change

The tf-apply.yml workflow is once again in the spotlight because a new push has been detected to the master branch. This time, there is a small change to make: my new tag needs to be added. The GitHub Actions log shows that the change was made successfully.

I validate that the change exists on the AWS S3 bucket by logging into the console and inspected the tags.

Further changes can be made by myself or others in a collaborative manner following the same process.

Public Security Concerns

In this scenario, I have made my GitHub repository public so that readers can see a real example of working code. I have also disabled GitHub Actions to prevent any abuse of my Actions credits, deleted / masked some of the comments, and purged the GitHub Actions log.

In reality, I don’t think it is wise to make the repository public. It would make more sense to invite collaborators and form teams with specific permissions to view and interact with the repository. The risk of a secret or semi-secret value being published to the public is too high.

I highly advise starting with a private GitHub repository to learn the ropes.

Billing

A “GitHub Free” subscription provides 2000 minutes per month of free Actions runtime for workflows executed in private repositories. There is no charge for Actions in a public repository. This may change beyond the date of this posting; check here for current quotas.

Minutes are billed at 1x, 2x, and 10x multipliers for Linux, Windows, and macOS runners, respectively. I use Linux runners works for my workflows as it is the cheapest option and also an acceptable choice for my needs.

The Wahl Network organization falls in the range of 150 – 200 minutes consumed per month on average.

Summary

In this post, I explored using GitHub Actions as a CI workflow that could build and maintain a backend Amazon Web Sevices (AWS) Simple Storage Service (S3) bucket for Terraform state files. I started by generating a new GitHub repository, then wrote the GitHub Workflow files, and finally started testing the CI workflow and introduced a small, fun change.

GitHub Actions is still a relatively new and untested service with frequent updates, changes, and improvements. It has been a pleasure working with this feature from the initial release and seeing the features and use cases evolve. Kudos to the GitHub team! ♥

Next Steps

Please accept a crisp high five for reaching this point in the post!

If you’d like to learn more about Continuous Integration, or other modern technology approaches, head over to the Guided Learning page.