Deploying containerized applications to AWS can be complex, involving multiple interconnected services: ECS task definitions, load balancers, target groups, listener rules, and DNS configuration. While you could manage these with CLI commands, Infrastructure as Code (IaC) with Terraform provides versioning, reproducibility, and team collaboration.

In this guide, I'll show you how to deploy any containerized application to AWS ECS with Application Load Balancer and custom domain configuration—all defined in Terraform.

What We'll Build

By the end of this tutorial, you'll have Terraform code that creates:

• ✅ ECS Task Definition (containerized app blueprint)

• ✅ ECS Service (manages running containers)

• ✅ ALB Target Group (health checks & routing)

• ✅ ALB Listener Rule (custom domain routing)

• ✅ Route53 DNS Record (points domain to ALB)

• ✅ CloudWatch Log Group (centralized logging)

  Best part: All infrastructure is versioned, reviewable, and reproducible!

  Architecture Overview

  User Request (https://myapp.yourdomain.com)

           ↓

  Route53 DNS (Terraform managed)

           ↓

  Application Load Balancer

           ↓

  ALB Listener Rule (Terraform managed)

           ↓

  Target Group (Terraform managed)

           ↓

   ECS Service (Terraform managed)

           ↓

   ECS Tasks (Docker Containers)

Prerequisites

• Terraform installed (v1.0+)

• AWS CLI configured with credentials

• Docker image in ECR or Docker Hub

• Existing AWS infrastructure:

• VPC with subnets

• ECS cluster

• Application Load Balancer

• Route53 hosted zone

  Project Structure

  terraform/

  ├── main.tf                 # Main resource definitions

  ├── variables.tf            # Input variables

  ├── outputs.tf              # Output values

  ├── terraform.tfvars        # Variable values (gitignored)

  ├── versions.tf             # Provider versions

  └── data.tf                 # Data sources (existing resources)

Step 1: Set Up Terraform Configuration

versions.tf - Provider Configuration

 terraform {

    required_version = ">= 1.0"

    required_providers {

      aws = {

        source  = "hashicorp/aws"

        version = "~> 5.0"

      }

    }

    # Optional: Remote state backend

    backend "s3" {

      bucket         = "my-terraform-state"

      key            = "ecs/my-app/terraform.tfstate"

      region         = "us-east-1"

      encrypt        = true

      dynamodb_table = "terraform-state-lock"

    }

  }

  provider "aws" {

    region = var.aws_region


    default_tags {

      tags = {

        Environment = var.environment

        Project     = var.project_name

        ManagedBy   = "Terraform"

      }

    }

  }

Why remote backend?

• ✅ Team collaboration (shared state)

• ✅ State locking (prevents conflicts)

• ✅ Encryption at rest

• ✅ Version history

  variables.tf - Input Variables

  variable "aws_region" {

    description = "AWS region"

    type        = string

    default     = "us-east-1"

  }

  variable "environment" {

    description = "Environment name (e.g., prod, staging)"

    type        = string

  }

  variable "project_name" {

    description = "Project name for resource naming"

    type        = string

  }

  variable "app_name" {

    description = "Application name"

    type        = string

  }

  variable "app_image" {

    description = "Docker image URL"

    type        = string

  }

  variable "app_port" {

    description = "Port the application listens on"

    type        = number

    default     = 8080

  }

  variable "app_cpu" {

    description = "CPU units for the task (1024 = 1 vCPU)"

    type        = number

    default     = 256

  }

  variable "app_memory" {

    description = "Memory for the task (MiB)"

    type        = number

    default     = 512

  }

  variable "desired_count" {

    description = "Number of tasks to run"

    type        = number

    default     = 2

  }

  variable "health_check_path" {

    description = "Health check endpoint"

    type        = string

    default     = "/"

  }

  variable "health_check_matcher" {

    description = "Expected HTTP status codes"

    type        = string

    default     = "200"

  }

  variable "domain_name" {

    description = "Custom domain for the app (e.g., myapp.example.com)"

    type        = string

  }

  variable "hosted_zone_name" {

    description = "Route53 hosted zone (e.g., example.com)"

    type        = string

  }

  variable "vpc_id" {

    description = "VPC ID where resources will be created"

    type        = string

  }

  variable "ecs_cluster_name" {

    description = "Name of existing ECS cluster"

    type        = string

  }

  variable "alb_arn" {

    description = "ARN of existing Application Load Balancer"

    type        = string

  }

  variable "alb_listener_arn" {

    description = "ARN of ALB HTTPS listener (port 443)"

    type        = string

  }

  variable "environment_variables" {

    description = "Environment variables for the container"

    type        = map(string)

    default     = {}

  }

  variable "secrets" {

    description = "Secrets from AWS Secrets Manager"

    type = list(object({

      name      = string

      valueFrom = string

    }))

    default = []

  }

  terraform.tfvars - Variable Values

  aws_region      = "us-east-1"

  environment     = "production"

  project_name    = "my-company"

  app_name        = "my-app"

  # Docker image

  app_image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest"

  # Container configuration

  app_port        = 8080

  app_cpu         = 512

  app_memory      = 1024

  desired_count   = 2

  # Health check

  health_check_path    = "/health"

  health_check_matcher = "200,302"  # Accept 200 OK and 302 redirects

  # DNS

  domain_name       = "myapp.example.com"

  hosted_zone_name  = "example.com"

  # Existing infrastructure

  vpc_id              = "vpc-0123456789abcdef0"

  ecs_cluster_name    = "production-cluster"

  alb_arn             = "arn:aws:elasticloadbalancing:us-east-1:123456789012:loadbalancer/app/production-alb/abc123"

  alb_listener_arn    = "arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/production-alb/abc123/def456"

  # Environment variables

  environment_variables = {

    ENV           = "production"

    LOG_LEVEL     = "info"

    PORT          = "8080"

  }

  # Secrets (stored in AWS Secrets Manager)

  secrets = [

    {

      name      = "DATABASE_URL"

      valueFrom = "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-url-abc123"

    },

    {

      name      = "API_KEY"

      valueFrom = "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/api-key-def456"

    }

  ]

🔒 Important: Add terraform.tfvars to .gitignore - it contains sensitive values!

Step 2: Data Sources for Existing Resources

data.tf - Query Existing Infrastructure

 # Get existing ECS cluster

  data "aws_ecs_cluster" "main" {

    cluster_name = var.ecs_cluster_name

  }

  # Get existing ALB

  data "aws_lb" "main" {

    arn = var.alb_arn

  }

  # Get Route53 hosted zone

  data "aws_route53_zone" "main" {

    name         = var.hosted_zone_name

    private_zone = false

  }

  # Get VPC

  data "aws_vpc" "main" {

    id = var.vpc_id

  }

  # Get current AWS account

  data "aws_caller_identity" "current" {}

  # Get current AWS region

  data "aws_region" "current" {}

Why use data sources?

• ✅ Reference existing infrastructure without hardcoding ARNs

• ✅ Validate resources exist before creating new ones

• ✅ Get dynamic values (like ALB DNS name)

  Step 3: Main Infrastructure Resources

  main.tf - Core Resources

 # ============================================================

  # CloudWatch Log Group for Container Logs

  # ============================================================

  resource "aws_cloudwatch_log_group" "app" {

    name              = "/ecs/${var.environment}/${var.app_name}"

    retention_in_days = 30

    tags = {

      Name = "${var.environment}-${var.app_name}-logs"

    }

  }

  # ============================================================

  # IAM Role for ECS Task Execution

  # ============================================================

  resource "aws_iam_role" "ecs_task_execution" {

    name = "${var.environment}-${var.app_name}-ecs-task-execution"

    assume_role_policy = jsonencode({

      Version = "2012-10-17"

      Statement = [

        {

          Action = "sts:AssumeRole"

          Effect = "Allow"

          Principal = {

            Service = "ecs-tasks.amazonaws.com"

          }

        }

      ]

    })

  }

  resource "aws_iam_role_policy_attachment" "ecs_task_execution" {

    role       = aws_iam_role.ecs_task_execution.name

    policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"

  }

  # Additional policy for Secrets Manager access

  resource "aws_iam_role_policy" "secrets_access" {

    count = length(var.secrets) > 0 ? 1 : 0

    name = "${var.environment}-${var.app_name}-secrets-access"

    role = aws_iam_role.ecs_task_execution.id

    policy = jsonencode({

      Version = "2012-10-17"

      Statement = [

        {

          Effect = "Allow"

          Action = [

            "secretsmanager:GetSecretValue",

            "kms:Decrypt"

          ]

          Resource = [for secret in var.secrets : secret.valueFrom]

        }

      ]

    })

  }

  # ============================================================

  # IAM Role for ECS Task (Application Runtime)

  # ============================================================

  resource "aws_iam_role" "ecs_task" {

    name = "${var.environment}-${var.app_name}-ecs-task"

    assume_role_policy = jsonencode({

      Version = "2012-10-17"

      Statement = [

        {

          Action = "sts:AssumeRole"

          Effect = "Allow"

          Principal = {

            Service = "ecs-tasks.amazonaws.com"

          }

        }

      ]

    })

  }

  # Add custom policies for your app (e.g., S3 access, DynamoDB, etc.)

  # resource "aws_iam_role_policy" "app_permissions" { ... }

  # ============================================================

  # ECS Task Definition

  # ============================================================

  resource "aws_ecs_task_definition" "app" {

    family                   = "${var.environment}-${var.app_name}"

    network_mode             = "bridge"  # Use "awsvpc" for Fargate

    requires_compatibilities = ["EC2"]   # Use ["FARGATE"] for Fargate

    cpu                      = var.app_cpu

    memory                   = var.app_memory

    execution_role_arn       = aws_iam_role.ecs_task_execution.arn

    task_role_arn            = aws_iam_role.ecs_task.arn

    container_definitions = jsonencode([

      {

        name      = var.app_name

        image     = var.app_image

        cpu       = var.app_cpu

        memory    = var.app_memory

        essential = true

        portMappings = [

          {

            containerPort = var.app_port

            hostPort      = 0  # Dynamic port mapping (use var.app_port for Fargate)

            protocol      = "tcp"

          }

        ]

        environment = [

          for key, value in var.environment_variables : {

            name  = key

            value = value

          }

        ]

        secrets = var.secrets

        logConfiguration = {

          logDriver = "awslogs"

          options = {

            "awslogs-group"         = aws_cloudwatch_log_group.app.name

            "awslogs-region"        = data.aws_region.current.name

            "awslogs-stream-prefix" = var.app_name

          }

        }

        healthCheck = {

          command = [

            "CMD-SHELL",

            "curl -f http://localhost:${var.app_port}${var.health_check_path} || exit 1"

          ]

          interval    = 30

          timeout     = 5

          retries     = 3

          startPeriod = 60

        }

      }

    ])

    tags = {

      Name = "${var.environment}-${var.app_name}"

    }

  }

  # ============================================================

  # ALB Target Group

  # ============================================================

  resource "aws_lb_target_group" "app" {

    name                 = "${var.environment}-${var.app_name}-tg"

    port                 = var.app_port

    protocol             = "HTTP"

    vpc_id               = var.vpc_id

    target_type          = "instance"  # Use "ip" for Fargate

    deregistration_delay = 30

    health_check {

      enabled             = true

      healthy_threshold   = 2

      unhealthy_threshold = 3

      timeout             = 5

      interval            = 30

      path                = var.health_check_path

      protocol            = "HTTP"

      matcher             = var.health_check_matcher

    }

    tags = {

      Name = "${var.environment}-${var.app_name}-tg"

    }

  }

  # ============================================================

  # ALB Listener Rule

  # ============================================================

  resource "aws_lb_listener_rule" "app" {

    listener_arn = var.alb_listener_arn

    priority     = 100  # Adjust as needed (lower = higher priority)

    action {

      type             = "forward"

      target_group_arn = aws_lb_target_group.app.arn

    }

    condition {

      host_header {

        values = [var.domain_name]

      }

    }

    tags = {

      Name = "${var.environment}-${var.app_name}-rule"

    }

  }

  # ============================================================

  # ECS Service

  # ============================================================

  resource "aws_ecs_service" "app" {

    name            = "${var.environment}-${var.app_name}"

    cluster         = data.aws_ecs_cluster.main.id

    task_definition = aws_ecs_task_definition.app.arn

    desired_count   = var.desired_count



    # Launch type (EC2 or FARGATE)

    launch_type = "EC2"  # Change to "FARGATE" if using Fargate

    # Deployment configuration

    deployment_maximum_percent         = 200

    deployment_minimum_healthy_percent = 50

    # Load balancer configuration

    load_balancer {

      target_group_arn = aws_lb_target_group.app.arn

      container_name   = var.app_name

      container_port   = var.app_port

    }

    # Placement constraints (EC2 only)

    placement_constraints {

      type = "distinctInstance"

    }

    # Depends on listener rule to avoid race condition

    depends_on = [aws_lb_listener_rule.app]

    tags = {

      Name = "${var.environment}-${var.app_name}"

    }

    lifecycle {

      ignore_changes = [desired_count]  # Allow manual scaling without Terraform drift

    }

  }

  # ============================================================

  # Route53 DNS Record

  # ============================================================

  resource "aws_route53_record" "app" {

    zone_id = data.aws_route53_zone.main.zone_id

    name    = var.domain_name

    type    = "A"

    alias {

      name                   = data.aws_lb.main.dns_name

      zone_id                = data.aws_lb.main.zone_id

      evaluate_target_health = false

    }

  }

  Step 4: Outputs

  outputs.tf - Export Important Values

  output "task_definition_arn" {

    description = "ARN of the ECS task definition"

    value       = aws_ecs_task_definition.app.arn

  }

  output "service_name" {

    description = "Name of the ECS service"

    value       = aws_ecs_service.app.name

  }

  output "target_group_arn" {

    description = "ARN of the target group"

    value       = aws_lb_target_group.app.arn

  }

  output "cloudwatch_log_group" {

    description = "CloudWatch log group name"

    value       = aws_cloudwatch_log_group.app.name

  }

  output "app_url" {

    description = "Application URL"

    value       = "https://${var.domain_name}"

  }

  output "dns_name" {

    description = "DNS record created"

    value       = aws_route53_record.app.fqdn

  }

Step 5: Deploy Your Infrastructure

Initialize Terraform

cd terraform/

terraform init

What this does:

• Downloads AWS provider plugins

• Initializes remote backend (if configured)

• Prepares working directory

  Plan Changes

  terraform plan -out=tfplan

  What this shows:

• Resources to be created (green +)

• Resources to be modified (yellow ~)

• Resources to be destroyed (red -)

• Total changes

  Review carefully! This is your preview before applying changes.

  Apply Changes

terraform apply tfplan

What happens:

1. Creates CloudWatch log group

2. Creates IAM roles and policies

3. Registers ECS task definition

4. Creates ALB target group

5. Creates ALB listener rule

6. Creates ECS service (launches containers)

7. Creates Route53 DNS record

   Typical completion time: 2-5 minutes

   Verify Deployment

  # Check outputs

  terraform output

  # Check service status

  aws ecs describe-services \

    --cluster production-cluster \

    --services $(terraform output -raw service_name)

  # Check target health

  aws elbv2 describe-target-health \

    --target-group-arn $(terraform output -raw target_group_arn)

  # View logs

  aws logs tail $(terraform output -raw cloudwatch_log_group) --follow

Advanced: Terraform Modules

For reusability across multiple apps, create a module:

Module Structure

  modules/

  └── ecs-app/

      ├── main.tf

      ├── variables.tf

      ├── outputs.tf

      └── README.md

  environments/

  ├── production/

  │   ├── main.tf          # Uses module

  │   ├── variables.tf

  │   └── terraform.tfvars

  └── staging/

      ├── main.tf

      ├── variables.tf

      └── terraform.tfvars

modules/ecs-app/main.tf

Move all resources from previous main.tf into this module.

environments/production/main.tf - Use Module

  module "my_app" {

    source = "../../modules/ecs-app"

    aws_region       = var.aws_region

    environment      = "production"

    project_name     = "my-company"

    app_name         = "my-app"

    app_image        = "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:v1.2.3"

    app_port         = 8080

    desired_count    = 3

    domain_name       = "myapp.example.com"

    hosted_zone_name  = "example.com"



    vpc_id            = data.aws_vpc.main.id

    ecs_cluster_name  = "production-cluster"

    alb_arn           = data.aws_lb.production.arn

    alb_listener_arn  = data.aws_lb_listener.https.arn



    environment_variables = {

      ENV = "production"

    }

  }

  module "another_app" {

    source = "../../modules/ecs-app"



    # Different configuration for another app

    app_name = "api-service"

    app_port = 3000

    # ...

  }

Benefits:

• ✅ Deploy multiple apps with same pattern

• ✅ Consistent configuration across environments

• ✅ Easy to maintain and update

• ✅ Reusable across projects

  Deployment Workflow with CI/CD

  GitHub Actions Example

  .github/workflows/deploy.yml:

name: Deploy to ECS

  on:

    push:

      branches: [main]

  env:

    AWS_REGION: us-east-1

    ECR_REPOSITORY: my-app

  jobs:

    deploy:

      runs-on: ubuntu-latest

      steps:

        - name: Checkout code

          uses: actions/checkout@v3

        - name: Configure AWS credentials

          uses: aws-actions/configure-aws-credentials@v2

          with:

            aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}

            aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

            aws-region: ${{ env.AWS_REGION }}

        - name: Login to Amazon ECR

          id: login-ecr

          uses: aws-actions/amazon-ecr-login@v1

        - name: Build, tag, and push image

          env:

            ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}

            IMAGE_TAG: ${{ github.sha }}

          run: |

            docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .

            docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

            echo "IMAGE=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_ENV

        - name: Setup Terraform

          uses: hashicorp/setup-terraform@v2

          with:

            terraform_version: 1.5.0

        - name: Terraform Init

          working-directory: ./terraform

          run: terraform init

        - name: Terraform Plan

          working-directory: ./terraform

          run: |

            terraform plan \

              -var="app_image=${{ env.IMAGE }}" \

              -out=tfplan

        - name: Terraform Apply

          working-directory: ./terraform

          run: terraform apply -auto-approve tfplan

        - name: Wait for deployment

          run: |

            aws ecs wait services-stable \

              --cluster production-cluster \

              --services my-app

Deployment Process

1. Push to main branch → Triggers workflow

2. Build Docker image → Tag with Git SHA

3. Push to ECR → Store image

4. Terraform plan → Show changes

5. Terraform apply → Update infrastructure

6. Wait for stability → Ensure deployment succeeds

   Managing Updates

   Update Application Code

   1. Update app_image in terraform.tfvars

  app_image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:v1.2.4"

2. Plan and apply

  terraform plan -out=tfplan

  terraform apply tfplan

What Terraform does:

• Creates new task definition revision (:2)

• Updates ECS service to use new task definition

• ECS performs rolling deployment (zero downtime)

  Scale Application

  Update desired_count in terraform.tfvars

  desired_count = 5

  Apply changes

  terraform apply -var="desired_count=5"

  Update Environment Variables

  Edit terraform.tfvars

  environment_variables = {

  ENV = "production"

  LOG_LEVEL = "debug" # Added

  NEW_FEATURE_FLAG = "true" # Added

  }

  Apply

  terraform apply

  Important: Changing environment variables creates a new task definition and triggers deployment.

  State Management Best Practices

  Remote State with S3

  terraform {

    backend "s3" {

      bucket         = "my-company-terraform-state"

      key            = "production/ecs/my-app/terraform.tfstate"

      region         = "us-east-1"

      encrypt        = true

      dynamodb_table = "terraform-state-lock"

      # Enable versioning on S3 bucket

      # Enable encryption

      # Enable bucket logging

    }

  }

Create State Backend

Create S3 bucket

  aws s3 mb s3://my-company-terraform-state --region us-east-1

Enable versioning

  aws s3api put-bucket-versioning \

    --bucket my-company-terraform-state \

    --versioning-configuration Status=Enabled

Enable encryption

  aws s3api put-bucket-encryption \

    --bucket my-company-terraform-state \

    --server-side-encryption-configuration '{

      "Rules": [{

        "ApplyServerSideEncryptionByDefault": {

          "SSEAlgorithm": "AES256"

        }

      }]

    }'

Create DynamoDB table for locking

  aws dynamodb create-table \

    --table-name terraform-state-lock \

    --attribute-definitions AttributeName=LockID,AttributeType=S \

    --key-schema AttributeName=LockID,KeyType=HASH \

    --billing-mode PAY_PER_REQUEST

State Commands

View current state

terraform state list

Show specific resource

terraform state show aws_ecs_service.app

Import existing resource

terraform import aws_ecs_service.app arn:aws:ecs:...

Remove resource from state (doesn't delete)

terraform state rm aws_ecs_service.app

Move resource to different address

terraform state mv aws_ecs_service.app aws_ecs_service.renamed

Troubleshooting Common Issues

Issue 1: Task Definition Already Exists

Error:

Error: creating ECS Task Definition: ClientException: Family already exists

Solution: Import existing task definition or use a different family name

terraform import aws_ecs_task_definition.app my-app

Issue 2: Target Group In Use

Error:

Error: deleting Target Group: ResourceInUse: Target group is in use

Solution: Remove listener rule first, then target group. Terraform handles this with depends_on.

Issue 3: Service Won't Stabilize

Symptoms: Terraform times out waiting for service to become stable

Check:

Service events

aws ecs describe-services --cluster production-cluster --services my-app

Target health

aws elbv2 describe-target-health --target-group-arn arn:...

Container logs

aws logs tail /ecs/production/my-app --follow

Common causes:

• Health check path returning wrong status code

• Container port mismatch

• Security group blocking traffic

• Container crashing on startup

  Issue 4: DNS Not Resolving

  Check:

  Verify record created

 aws route53 list-resource-record-sets \

    --hosted-zone-id Z1234567890ABC \

    --query "ResourceRecordSets[?Name=='myapp.example.com.']"

Test DNS resolution

dig myapp.example.com

nslookup myapp.example.com

Cost Optimization

1. Right-Size Resources

   Monitor CloudWatch metrics

   Adjust based on actual usage

   app_cpu = 256 # Start small

   app_memory = 512 # Increase if needed

2. Use Spot Instances (Non-Production)

   In ECS capacity provider

   capacity_providers = ["FARGATE_SPOT"]

   70% cheaper than Fargate on-demand

3. Log Retention

  resource "aws_cloudwatch_log_group" "app" {

    retention_in_days = 7  # vs 30 or 90

  }

4. Cleanup Unused Resources

   Remove old task definition revisions

  aws ecs list-task-definitions --family-prefix my-app --status INACTIVE

Security Best Practices

1. Use Secrets Manager

   Never put secrets in environment variables!

   Use secrets parameter instead:

  secrets = [

    {

      name      = "DATABASE_PASSWORD"

      valueFrom = aws_secretsmanager_secret.db_password.arn

    }

  ]

2. Least Privilege IAM

   Task role - only what app needs

  resource "aws_iam_role_policy" "app" {

    policy = jsonencode({

      Statement = [

        {

          Effect = "Allow"

          Action = ["s3:GetObject"]

          Resource = ["arn:aws:s3:::my-bucket/*"]

        }

      ]

    })

  }

3. Enable Container Insights

  resource "aws_ecs_cluster" "main" {

    setting {

      name  = "containerInsights"

      value = "enabled"

    }

  }

4. Network Isolation

   Use awsvpc network mode with private subnets

  network_mode = "awsvpc"

  network_configuration {

    subnets         = var.private_subnet_ids

    security_groups = [aws_security_group.app.id]

  }

Terraform vs AWS CLI: Comparison

  | Aspect             | AWS CLI                  | Terraform                               |

  |--------------------|--------------------------|-----------------------------------------|

  | Reproducibility    | Manual re-execution      | Declarative, version-controlled         |

  | Team Collaboration | Difficult (manual docs)  | Easy (code review, shared state)        |

  | Rollback           | Manual, error-prone      | terraform apply previous version        |

  | Drift Detection    | None                     | terraform plan shows drift              |

  | Dependencies       | Manual ordering          | Automatic dependency graph              |

  | Documentation      | Separate                 | Infrastructure as code IS documentation |

  | Learning Curve     | Moderate (many commands) | Steeper initially, easier long-term     |

  | Multi-cloud        | AWS only                 | Works across providers                  |

Key Takeaways

1. Infrastructure as Code: Terraform makes infrastructure reproducible, version-controlled, and collaborative

2. Modules: Reuse common patterns across multiple apps and environments

3. Remote State: Essential for team collaboration and state locking

4. Variables: Separate configuration from code for environment-specific values

5. Secrets Management: Never hardcode secrets - use Secrets Manager

6. CI/CD Integration: Automate deployments with GitHub Actions or similar

7. Incremental Adoption: Can import existing resources with terraform import

   Next Steps

8. Set up remote state backend (S3 + DynamoDB)

9. Create reusable modules for common patterns

10. Implement CI/CD pipeline with Terraform

11. Add auto-scaling with Application Auto Scaling

12. Enable Container Insights for monitoring

13. Implement blue/green deployments with CodeDeploy

14. Add WAF rules to ALB for security

    Resources

15. https://registry.terraform.io/providers/hashicorp/aws/latest/docs

16. https://docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/

17. https://www.terraform-best-practices.com/

18. https://github.com/terraform-aws-modules/terraform-aws-ecs

Questions or suggestions? Drop a comment below!

Found this helpful? Share with your team and follow for more DevOps content!

In modern DevOps practices, managing secrets securely is one of the most critical aspects of infrastructure management. Whether it's API keys, database credentials, or JWT private keys, these sensitive pieces of information need to be handled with care. In this article, I'll walk through how I implemented a comprehensive secrets management solution using Google Cloud Platform's Key Management Service (KMS) and Secret Manager, automated with Terraform and integrated into a CI/CD pipeline.

The Challenge

When managing multiple services in a cloud environment, you face several challenges:

- Storing hundreds of secrets securely

- Maintaining different configurations for staging and production environments

- Ensuring secrets are encrypted at rest and in transit

- Automating secret rotation and deployment

- Providing secure access to applications without exposing credentials

The Architecture

My solution leverages several GCP services and tools:

┌─────────────────┐
│   Developer     │
│   Encrypts      │
│   Secrets       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   GCP KMS       │
│  (Encryption)   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Terraform     │
│   Variables     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Terraform     │
│   Decrypts &    │
│   Provisions    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  GCP Secret     │
│   Manager       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│External Secrets │
│   Operator      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Kubernetes    │
│     Pods        │
└─────────────────┘

How Encryption Works

Step 1: Encrypting Secrets with GCP KMS

The first layer of security comes from Google Cloud KMS. Before any secret enters our repository, it's encrypted using environment-specific KMS keys:

echo -n "your-secret-value" | gcloud kms encrypt \
  --location=global \
  --keyring=stage-keyring \
  --key=key-name \
  --plaintext-file=- \
  --ciphertext-file=- \
  --project=your-project-id \
  | base64

This command:

1. Takes your plaintext secret

2. Encrypts it using a specific KMS key ring and key

3. Outputs the ciphertext in base64 format

The resulting encrypted value looks something like:

CiQA0TkZJRZqLSLT2w+OLxu1imQozdof+xBGTjepJbwrUW/bynESPQBCuzVB...

Step 2: Storing Encrypted Values in Terraform

The encrypted values are stored in Terraform variable files (`*.tfvars`). Here's the structure:

services_config = {

  "hge-django" = {

    secrets = {

      AWS_ACCESS_KEY_ID     = "CiQA0TkZJRZqLSLT2w+OLxu1imQo..."

      AWS_SECRET_ACCESS_KEY = "CiQA0TkZJUeuYXTwfli7eaNdDg9s..."

      DATABASE_URL          = "CiQA0TkZJRJ/hBTBsbb4UFp+O3JG..."

      API_HOST              = "https://api.com/api"  # Plaintext

    }

    encrypted_keys = [

      "AWS_ACCESS_KEY_ID",

      "AWS_SECRET_ACCESS_KEY",

      "DATABASE_URL"

    ]

  }

}

Notice how:

- Sensitive values are encrypted (AWS keys, database URLs)

- Non-sensitive values can remain in plaintext (API endpoints)

- The encrypted_keys array tells Terraform which values need decryption

The Terraform Magic

Dynamic Decryption

The heart of the system is in the Terraform configuration. Here's how it works:

locals {

  # Flatten and decrypt secrets for all services

  decrypted_secrets = {
    for service_name, config in var.services_config : service_name => {
      for key, value in config.secrets : key => (
        contains(config.encrypted_keys, key) ?
        data.google_kms_secret.secrets["${service_name}-${key}"].plaintext :
        value
      )
    }
  }

  # Prepare encrypted secrets for KMS decryption

  encrypted_secrets = merge([
    for service_name, config in var.services_config : {
      for key in config.encrypted_keys :
      "${service_name}-${key}" => {
        service = service_name
        key     = key
        value   = config.secrets[key]
      }
    }
  ]...)
}

# Decrypt using KMS

data "google_kms_secret" "secrets" {
  for_each = local.encrypted_secrets
  crypto_key = "projects/${var.project_id}/locations/global/keyRings/${var.environment}-keyring/cryptoKeys/${var.environment}-key"
  ciphertext = each.value.value

}

This Terraform code:

1. Iterates through all service configurations

2. Identifies which secrets are encrypted

3. Decrypts them using GCP KMS

4. Combines decrypted and plaintext values

Provisioning to Secret Manager

Once decrypted, the secrets are provisioned to GCP Secret Manager:

module "secret-manager" {
  source     = "GoogleCloudPlatform/secret-manager/google"
  version    = "~> 0.5"
  project_id = var.project_id
  secrets = nonsensitive(flatten([
    for service_name, secrets in local.decrypted_secrets : [
      for key, value in secrets : {
        name        = "${service_name}-${lower(key)}"
        secret_data = value
      }
      if value != ""
    ]
  ]))
  secret_accessors_list = [
    "group:gcp-organization-admins@company.com"
  ]
}

This creates secrets in GCP Secret Manager with:

- Consistent naming: service-name-secret-key

- Proper access controls

- Automatic versioning

The Workflow

Here's the complete workflow for adding a new secret:

1. Developer encrypts the secret using GCP KMS

2. Updates the Terraform variables file with the encrypted value

3. Creates a pull request for review

4. CI runs Terraform plan to validate changes

5. After approval and merge, CI/CD deploys to staging

6. Manual approval triggers production deployment

7. External Secrets Operator syncs to Kubernetes

8. Applications access secrets from Kubernetes secrets

Security Best Practices

1. Encryption at Rest

All sensitive values are encrypted before entering version control. Even if someone gains access to the repository, they can't decrypt without KMS access.

2. Principle of Least Privilege

- KMS keys are environment-specific

- Secret Manager access is controlled via IAM groups

- Applications only access secrets they need

3. Audit Trail

- GCP Cloud Audit Logs track all KMS operations

- Secret Manager maintains access logs

- Terraform state is stored securely in GCS

4. Separation of Concerns

- Developers can add secrets without accessing production systems

- DevOps team controls deployment

- Applications never see encryption keys

5. Automated Rotation

With everything in Terraform, rotating secrets is as simple as:

1. Generate new secret

2. Encrypt with KMS

3. Update tfvars

4. Deploy through CI/CD

Handling Different Secret Types

The system handles various types of secrets elegantly:

Simple Credentials

hcl
DB_USER = "app"  # Can be plaintext
DB_PASS = "CiQA0TkZJabdTVyzUvmU..."  # Encrypted

Complex JSON Structures

ADMINS = "['admin@company.com']"  # Python list as string

Multi-line Keys (like RSA Private Keys)

DOCUSIGN_JWT_PRIVATE_KEY = "CiQA0TkZJevqtcXkUTOY3Ddo..."  # Base64 encoded

The system handles these by:

1. Encrypting the entire content (including newlines)

2. Base64 encoding for storage

3. Proper decryption maintaining format

Kubernetes Integration

The final piece is getting secrets to applications. External Secrets Operator handles this:

1. Watches Secret Manager for changes

2. Creates Kubernetes Secrets automatically

3. Mounts to pods as environment variables or files

4. Handles rotation without pod restarts

Example pod configuration:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    envFrom:
    - secretRef:
        name: secrets

Lessons Learned

1. Start with Encryption Early

Retrofitting encryption is harder than starting with it. Design your secret management from day one.

2. Environment Separation is Critical

Using separate KMS keys for staging and production prevents accidental cross-environment secret usage.

3. Automate Everything

Manual secret management doesn't scale. Automation reduces errors and improves security.

4. Monitor and Audit

Set up alerts for:

- Failed decryption attempts

- Unusual secret access patterns

- Secret rotation reminders

5. Documentation is Security

Well-documented processes mean fewer mistakes. Our README provides clear instructions for the entire team.

Performance Considerations

Caching

Terraform caches decrypted values during runs, avoiding repeated KMS calls.

Batch Operations

The system processes all secrets in parallel, reducing deployment time.

Secret Manager Quotas

Be aware of GCP quotas:

- 60,000 secret versions per project

- 90,000 access requests per minute

Cost Optimization

The solution is cost-effective:

- KMS: $0.06 per key per month + $0.03 per 10,000 operations

- Secret Manager: $0.06 per secret per month + $0.03 per 10,000 operations

- Total monthly cost: ~$50 for 100 secrets with moderate usage

Future Improvements

Looking ahead, potential enhancements include:

1. Secret Rotation Automation: Implementing automatic rotation for database passwords and API keys

2. Break-glass Procedures: Emergency access workflows for critical situations

3. Multi-region Replication: For disaster recovery

4. Secret Usage Analytics: Track which services use which secrets

Conclusion

Building a secure, scalable secrets management system requires careful planning and the right tools. By combining GCP KMS for encryption, Terraform for infrastructure as code, and automated CI/CD pipelines, we've created a solution that:

- Keeps secrets secure at every stage

- Scales with our infrastructure

- Maintains compliance requirements

- Reduces operational overhead

The key takeaway? Security doesn't have to be complicated. With the right architecture and automation, you can build a secrets management system that's both secure and developer-friendly.

Remember: Your secrets are only as secure as your weakest link. Invest in proper secrets management early, and your future self (and security team) will thank you.

---

Have questions or suggestions? Feel free to reach out. Security is a community effort, and sharing knowledge makes us all stronger. at juliusoh@gmail.com

Resources

- [GCP KMS Documentation](https://cloud.google.com/kms/docs)

- [GCP Secret Manager](https://cloud.google.com/secret-manager/docs)

- [Terraform Google Provider](https://registry.terraform.io/providers/hashicorp/google/latest)

- [External Secrets Operator](https://external-secrets.io/)