Solving Azure Connection Internal Server Errors

I leverage a global transit network design for my Azure cloud resources. This requires the creation of a Virtual WAN, Virtual Hub, and numerous Virtual Networks (VNets). Connecting these components allows traffic to flow between them so long as the correct security controls are in place.

All of the cloud resources are constructed and maintained using Terraform coupled with GitLab CI for Continuous Integration (CI). I highly suggest reading Continuous Integration Fundamentals if you’re new to the concept.

Iterating upon the Terraform code requires testing using an isolated, ephemeral Azure region that often excludes the VPN gateway to save time and cost; a VPN gateway takes 30 minutes or more to create! Each VNet module executed by Terraform completes with a Virtual Hub connection resource, as shown below:

resource "azurerm_virtual_hub_connection" "vhub" {
  name                                           = "cn-to-${var.azure_name_suffix}"
  virtual_hub_id                                 = var.parent_azurerm_virtual_hub_id
  remote_virtual_network_id                      = azurerm_virtual_network.vnet.id
  hub_to_vitual_network_traffic_allowed          = true
  vitual_network_to_hub_gateways_traffic_allowed = true

Note: Yes, “virtual” is misspelled as “vitual.” The issue is tracked here. Correcting the spelling is a breaking change.

The Mysterious Internal Server Error

Running the code when no VPN gateway exists within the Virtual Hub yielded a mysterious error.

Error: Error waiting for addition of Connection "connection_name" to Virtual Hub "virtual_hub_name" (Resource Group "resource_group_name"): Code="InternalServerError" Message="An error occurred." Details=[]
on vnet\main.tf line 33, in resource "azurerm_virtual_hub_connection" "vhub":
33: resource "azurerm_virtual_hub_connection" "vhub" {
virtual_hub_id = var.parent_azurerm_virtual_hub_id
remote_virtual_network_id = azurerm_virtual_network.vnet.id

The error messages provides a few clues:

Most “Internal Server Error” codes mean a fault on the server side. This rules out most of the client side troubleshooting components that I would normally suspect.
Details being blank (null) leads me to believe that I’m passing configuration values that are not supported, but are not being handled gracefully by the server’s error framework.

The culprit is vitual_network_to_hub_gateways_traffic_allowed. This parameter must be false or excluded when connecting a VNet to a Virtual Hub that lacks a VPN gateway. I encountered the error before reading deeper into the provider documentation here. Other cloud providers tend to surface a more helpful error message or flip the parameter value on the server side.

Commenting out the parameter is an immediate workaround that worked. Later, I switched to using a conditional statement predicated on the existence of a azurerm_vpn_gateway to determine the boolean value of the vitual_network_to_hub_gateways_traffic_allowed parameter.