Tracking Infrastructure with SSM and Terraform

Terraform is the most popular cross cloud,
infrastructure as code tool. Not only does Terraform specify what is to be
deployed, it tracks what was deployed. This is really useful when all of your
infrastructure is managed centrally.

It’s not always practical or desirable to track all resources in a single
account in the same project. For example low level infrastructure, such as VPCs
or SSO configuration, change infrequently. On the other hand applications might
be redeployed dozens of time per day.

Both applications and low level infrastructure can be managed by Terraform.
There are also cases when the low level infrastructure is managed with Terraform
while other services are managed using other tooling such as
CDK,
CloudFormation or
Pulumi.

Teams need a way of sharing configuration, without resorting to hard coded
strings. When using AWS, SSM Parameter
Store
(aka SSM Params) helps support an eclectic mix of environment management
tooling. Values such as ARNs or other identifiers can be stored in SSM. Other
tools and applications can pull in these parameters and use them. It’s even
possible to use the parameters across accounts, which is useful with shared
resources.

Your environment might contain multiple subnets. While it is possible to lookup
the subnets using tags, these conventions can vary from team to team. Instead
the subnets can be stored as SSM Params. This partial snippet shows how to store
these values:

variable "tags" {
  description = "Tags to apply to all resources."
  type        = map(string)

  default = {
    todo = "Add real values here"
  }
}

variable "vpc_cidr" {
  description = "CIDR block used by the VPC. Must be a /16 for the dynamic subnet creation to work."
  type        = string
  default     = "10.0.0.0/16"
}

data "aws_availability_zones" "all" {}

locals {
  az_names = toset(data.aws_availability_zones.all.names)
}

resource "aws_vpc" "application" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true

  tags = merge(
    var.tags,
    {
      Name = "application"
    }
  )
}

resource "aws_subnet" "private" {
  for_each = local.az_names

  vpc_id            = aws_vpc.application.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, index(local.az_names, each.value) + 8)
  availability_zone = each.value

  tags = merge(
    var.tags,
    {
      "Name" = "application-private-${each.value}"
    }
  )
}

resource "aws_subnet" "public" {
  for_each = local.az_names

  vpc_id            = aws_vpc.application.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, index(local.az_names, each.value))
  availability_zone = each.value

  tags = merge(
    var.tags,
    {
      "Name" = "application-public-${each.value}"
    }
  )
}

resource "aws_ssm_parameter" "subnets" {
  name = "/vpc/application/subnets"
  type = "String"
  value = jsonencode({
    "private" : [for subnet in aws_subnet.private : subnet.id],
    "public" : [for subnet in aws_subnet.public : subnet.id],
  })

  tags = var.tags
}

In an application pipeline these values can be retrieved and validated using
code like this:

# Get the value from SSM
data "aws_ssm_parameter" "subnets" {
  name = "/vpc/application/subnets"
}

# Use a local to avoid decoding the string multiple times
locals {
  subnets = jsondecode(data.aws_ssm_parameter.subnets.insecure_value)
}

# Validate the subnets, this also makes it easy to access
# the VPC ID and other properties associated with the subnet
data "aws_subnet" "public" {
  for_each = toset(local.subnets.public)

  id = each.key
}

This can be used with other resources. For example a GitHub
Action might need to publish web content
to a S3 bucket. The target bucket can be fetched from SSM, rather than the
target being tracked using GitHub secrets. This ensures the permissions are also
configured properly for pushing changes to the bucket.

resource "aws_s3_bucket" "website" {
  bucket = "website-example.com"

  tags = var.tags
}

# Rest of S3 config goes here

resource "aws_ssm_parameter" "website_bucket" {
  name  = "/resources/website_bucket_arn"
  type  = "String"
  value = aws_s3_bucket.website.arn

  tags = var.tags
}

Once you have OIDC properly configured for
AWS,
in the build you can fetch the bucket ARN by running the command:
aws ssm get-parameter --name /resources/website_bucket_arn --output text --query Parameter.Value

This pattern can be used for storing the security group IDs for VPC endpoints.
This works well when you allow all traffic from the VPC to access the endpoints,
the use the egress controls on the security groups used on other resources such
as Lambdas and ECS instances. The values used on Lambda security groups. This
snippet can be used in your application Terraform configuration.

# Fetch the VPC endpoint security group ID
data "aws_ssm_parameter" "sec_group_vpce_sqs" {
  name = "/vpc/application/sec_groups/vpce_sqs"
}

# If the VPCe is the same account we can validate the ID
data "aws_security_group" "vpce_sqs" {
  id = data.aws_ssm_parameter.sec_group_vpce_sqs
}

resource "aws_security_group" "my_lambda" {
  name        = "my-lambda"
  description = "Controls for my-lambda"

  tags = merge(
    var.tags,
    {
      "Name" = "my-lambda"
    }
  )
}

resource "aws_security_group_rule" "my_lambda_to_sqs" {
  type      = "egress"
  from_port = 443
  to_port   = 443
  protocol  = "tcp"

  security_group_id = aws_security_group.my_lambda.id

  source_security_group_id = data.aws_security_group.vpce_sqs.id
}