stuff I figured out the hard way

Defaults:

  • Lock state with use_lockfile in the S3 bucket; no DynamoDB table.
  • Validate every constrained input with a validation block.
  • Give every workload its own cloud identity through workload identity; never mount cloud keys into pods.
  • Pin every OIDC trust - workloads and CI - to an exact subject: sub/aud on AWS, attribute_condition on GCP, subject on Azure.
  • Use for_each over a map, not count, for sets of similar resources.
  • Rename with a moved block, never destroy-and-recreate.

Targets AWS provider 6.x, Google 7.x and AzureRM 4.x. Everything here passes terraform validate.

Remote state with S3-native locking

Use use_lockfile (Terraform 1.11+). It locks state with a file in the bucket itself, so no DynamoDB table is needed.

terraform {
  required_version = ">= 1.11"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }

  backend "s3" {
    bucket       = "my-company-terraform-state"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-central-1"
    encrypt      = true
    use_lockfile = true
  }
}

Validate inputs early

Add a validation block to constrained variables. A bad value then fails at plan, not halfway through an apply.

variable "environment" {
  type        = string
  description = "Deployment environment."

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

A locked-down S3 bucket

Make every bucket private, versioned, encrypted and self-cleaning. Set prevent_destroy so terraform destroy fails instead of deleting the data.

resource "aws_s3_bucket" "this" {
  bucket = "my-company-${var.environment}-artifacts"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  = aws_s3_bucket.this.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
    bucket_key_enabled = true # far fewer KMS calls, lower cost
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "this" {
  bucket = aws_s3_bucket.this.id

  rule {
    id     = "expire-old-versions"
    status = "Enabled"
    filter {}

    noncurrent_version_expiration {
      noncurrent_days = 30
    }
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

For cost-driven lifecycle rules, see taming runaway database backups.

Workload identity: pods to cloud APIs without keys

Give each Kubernetes service account its own cloud identity, scoped to one namespace and name. Never mount cloud access keys into pods. Every cloud does this by trusting the cluster’s OIDC issuer; only the wiring differs.

CloudMechanismKubernetes side
AWSEKS Pod Identity (default) or IRSAPod Identity: nothing. IRSA: eks.amazonaws.com/role-arn annotation
GCPWorkload Identity Federation for GKENothing - grant IAM to the service account’s principal directly
AzureAKS Workload Identityazure.workload.identity/client-id annotation and azure.workload.identity/use: "true" pod label

AWS: EKS Pod Identity

Default for EKS. The trust policy is the same for every role - no OIDC issuer, no conditions to get wrong. Requires the eks-pod-identity-agent add-on.

data "aws_iam_policy_document" "pod_identity_trust" {
  statement {
    actions = ["sts:AssumeRole", "sts:TagSession"]

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "my_app" {
  name               = "${var.environment}-my-app"
  assume_role_policy = data.aws_iam_policy_document.pod_identity_trust.json
}

resource "aws_eks_pod_identity_association" "my_app" {
  cluster_name    = var.cluster_name
  namespace       = "my-namespace"
  service_account = "my-app"
  role_arn        = aws_iam_role.my_app.arn
}

AWS: IRSA

Use when Pod Identity isn’t available - Fargate, self-managed clusters, or tools that only support IRSA. Look the issuer and provider up from the cluster name instead of passing them in. Check aud as well as sub so tokens meant for other audiences are rejected.

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_iam_openid_connect_provider" "eks" {
  url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}

locals {
  # IAM condition keys use the issuer without the scheme
  oidc_issuer = trimprefix(data.aws_eks_cluster.this.identity[0].oidc[0].issuer, "https://")
}

data "aws_iam_policy_document" "irsa_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [data.aws_iam_openid_connect_provider.eks.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_issuer}:sub"
      values   = ["system:serviceaccount:my-namespace:my-app"]
    }

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_issuer}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "my_app" {
  name               = "${var.environment}-my-app"
  assume_role_policy = data.aws_iam_policy_document.irsa_trust.json
}

Using terraform-aws-modules/eks? Its oidc_provider (already without https://) and oidc_provider_arn outputs replace both lookups. For role chaining on top of IRSA, see Hoop + EKS access via IAM role chaining.

GCP: Workload Identity Federation for GKE

Enable the workload pool on the cluster, then grant roles straight to the Kubernetes service account’s principal. No Google service account, no annotation.

data "google_project" "this" {}

resource "google_container_cluster" "this" {
  # ...
  workload_identity_config {
    workload_pool = "${data.google_project.this.project_id}.svc.id.goog"
  }
}

resource "google_storage_bucket_iam_member" "my_app" {
  bucket = "my-company-artifacts"
  role   = "roles/storage.objectViewer"
  member = "principal://iam.googleapis.com/projects/${data.google_project.this.number}/locations/global/workloadIdentityPools/${data.google_project.this.project_id}.svc.id.goog/subject/ns/my-namespace/sa/my-app"
}

The principal uses the project number, not the ID. A few Google APIs don’t accept these principals yet; for those, fall back to a Google service account that the Kubernetes service account impersonates.

Azure: AKS Workload Identity

The cluster needs oidc_issuer_enabled and workload_identity_enabled. Create a user-assigned managed identity, federate it with the service account, then grant it roles.

data "azurerm_kubernetes_cluster" "this" {
  name                = var.cluster_name
  resource_group_name = var.resource_group_name
}

resource "azurerm_user_assigned_identity" "my_app" {
  name                = "${var.environment}-my-app"
  resource_group_name = var.resource_group_name
  location            = var.location
}

resource "azurerm_federated_identity_credential" "my_app" {
  name                = "my-app"
  resource_group_name = var.resource_group_name
  parent_id           = azurerm_user_assigned_identity.my_app.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = data.azurerm_kubernetes_cluster.this.oidc_issuer_url
  subject             = "system:serviceaccount:my-namespace:my-app"
}

resource "azurerm_role_assignment" "my_app" {
  scope                = var.storage_account_id
  role_definition_name = "Storage Blob Data Reader"
  principal_id         = azurerm_user_assigned_identity.my_app.principal_id
}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: my-namespace
  annotations:
    azure.workload.identity/client-id: "<azurerm_user_assigned_identity.my_app.client_id>"
---
# In the pod template:
metadata:
  labels:
    azure.workload.identity/use: "true"

CI to the cloud without stored keys

Exchange GitHub’s OIDC token for short-lived cloud credentials; never store cloud keys in repository secrets. Every cloud trusts https://token.actions.githubusercontent.com - the trust must be pinned to a repository and branch, or any GitHub repository can use it.

CloudTrust objectPin it with
AWSIAM OIDC provider + role trust policysub condition
GCPWorkload identity pool + providerattribute_condition
AzureFederated credential on a managed identityExact subject match

AWS

resource "aws_iam_openid_connect_provider" "github" {
  url            = "https://token.actions.githubusercontent.com"
  client_id_list = ["sts.amazonaws.com"]
}

data "aws_iam_policy_document" "github_trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.github.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringLike"
      variable = "token.actions.githubusercontent.com:sub"
      values   = ["repo:my-org/my-repo:ref:refs/heads/main"]
    }
  }
}

resource "aws_iam_role" "github_deploy" {
  name                 = "github-deploy"
  assume_role_policy   = data.aws_iam_policy_document.github_trust.json
  max_session_duration = 3600
}

GCP

Google rejects GitHub providers without an attribute_condition. Grant roles straight to the repository’s principal set; no Google service account needed.

resource "google_iam_workload_identity_pool" "github" {
  workload_identity_pool_id = "github"
}

resource "google_iam_workload_identity_pool_provider" "github" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.github.workload_identity_pool_id
  workload_identity_pool_provider_id = "github"

  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.repository" = "assertion.repository"
    "attribute.ref"        = "assertion.ref"
  }
  attribute_condition = "assertion.repository == 'my-org/my-repo' && assertion.ref == 'refs/heads/main'"

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

resource "google_project_iam_member" "github_deploy" {
  project = data.google_project.this.project_id
  role    = "roles/artifactregistry.writer"
  member  = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github.name}/attribute.repository/my-org/my-repo"
}

Azure

A standard federated credential matches the token’s subject exactly - no wildcards. Add one credential per branch or environment that deploys.

resource "azurerm_user_assigned_identity" "github_deploy" {
  name                = "github-deploy"
  resource_group_name = var.resource_group_name
  location            = var.location
}

resource "azurerm_federated_identity_credential" "github_main" {
  name                = "github-main"
  resource_group_name = var.resource_group_name
  parent_id           = azurerm_user_assigned_identity.github_deploy.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = "https://token.actions.githubusercontent.com"
  subject             = "repo:my-org/my-repo:ref:refs/heads/main"
}

resource "azurerm_role_assignment" "github_deploy" {
  scope                = var.resource_group_id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.github_deploy.principal_id
}

The workflow side

permissions:
  id-token: write   # required to request the OIDC token
  contents: read

steps:
  # AWS
  - uses: aws-actions/configure-aws-credentials@v6
    with:
      role-to-assume: arn:aws:iam::123456789012:role/github-deploy
      aws-region: eu-central-1

  # GCP
  - uses: google-github-actions/auth@v3
    with:
      workload_identity_provider: projects/123456789/locations/global/workloadIdentityPools/github/providers/github

  # Azure
  - uses: azure/login@v3
    with:
      client-id: ${{ vars.AZURE_CLIENT_ID }}
      tenant-id: ${{ vars.AZURE_TENANT_ID }}
      subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}

Watch the subject format. A job that uses a GitHub environment gets repo:my-org/my-repo:environment:prod as its subject, not the branch ref. Pin to whichever form your deploy jobs actually produce, on every cloud.

Many similar resources, and refactoring without destroying

  • Use for_each over a map, not count. Resources are keyed by name, so removing one entry doesn’t shift the others.
  • Rename with a moved block. It renames the resource in state instead of destroying and recreating it.
variable "buckets" {
  type = map(object({
    versioning = bool
  }))
  default = {
    logs    = { versioning = false }
    backups = { versioning = true }
  }
}

resource "aws_s3_bucket" "named" {
  for_each = var.buckets
  bucket   = "my-company-${var.environment}-${each.key}"
}

# Was a standalone resource; now part of the map
moved {
  from = aws_s3_bucket.logs
  to   = aws_s3_bucket.named["logs"]
}

Run terraform plan after adding a moved block. It must show the move and no destroy.