Building a Secure DevOps Pipeline: Managing Secrets with GCP KMS and Terraform

Introduction

In modern DevOps practices, managing secrets securely is one of the most critical aspects of infrastructure management. Whether it's API keys, database credentials, or JWT private keys, these sensitive pieces of information need to be handled with care. In this article, I'll walk through how I implemented a comprehensive secrets management solution using Google Cloud Platform's Key Management Service (KMS) and Secret Manager, automated with Terraform and integrated into a CI/CD pipeline.

The Challenge

When managing multiple services in a cloud environment, you face several challenges:

- Storing hundreds of secrets securely

- Maintaining different configurations for staging and production environments

- Ensuring secrets are encrypted at rest and in transit

- Automating secret rotation and deployment

- Providing secure access to applications without exposing credentials

The Architecture

My solution leverages several GCP services and tools:

┌─────────────────┐
│   Developer     │
│   Encrypts      │
│   Secrets       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   GCP KMS       │
│  (Encryption)   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Terraform     │
│   Variables     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Terraform     │
│   Decrypts &    │
│   Provisions    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  GCP Secret     │
│   Manager       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│External Secrets │
│   Operator      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Kubernetes    │
│     Pods        │
└─────────────────┘

How Encryption Works

Step 1: Encrypting Secrets with GCP KMS

The first layer of security comes from Google Cloud KMS. Before any secret enters our repository, it's encrypted using environment-specific KMS keys:

echo -n "your-secret-value" | gcloud kms encrypt \
  --location=global \
  --keyring=stage-keyring \
  --key=key-name \
  --plaintext-file=- \
  --ciphertext-file=- \
  --project=your-project-id \
  | base64

This command:

1. Takes your plaintext secret

2. Encrypts it using a specific KMS key ring and key

3. Outputs the ciphertext in base64 format

The resulting encrypted value looks something like:

CiQA0TkZJRZqLSLT2w+OLxu1imQozdof+xBGTjepJbwrUW/bynESPQBCuzVB...

Step 2: Storing Encrypted Values in Terraform

The encrypted values are stored in Terraform variable files (`*.tfvars`). Here's the structure:

services_config = {

  "hge-django" = {

    secrets = {

      AWS_ACCESS_KEY_ID     = "CiQA0TkZJRZqLSLT2w+OLxu1imQo..."

      AWS_SECRET_ACCESS_KEY = "CiQA0TkZJUeuYXTwfli7eaNdDg9s..."

      DATABASE_URL          = "CiQA0TkZJRJ/hBTBsbb4UFp+O3JG..."

      API_HOST              = "https://api.com/api"  # Plaintext

    }

    encrypted_keys = [

      "AWS_ACCESS_KEY_ID",

      "AWS_SECRET_ACCESS_KEY",

      "DATABASE_URL"

    ]

  }

}

Notice how:

- Sensitive values are encrypted (AWS keys, database URLs)

- Non-sensitive values can remain in plaintext (API endpoints)

- The encrypted_keys array tells Terraform which values need decryption

The Terraform Magic

Dynamic Decryption

The heart of the system is in the Terraform configuration. Here's how it works:

locals {

  # Flatten and decrypt secrets for all services

  decrypted_secrets = {
    for service_name, config in var.services_config : service_name => {
      for key, value in config.secrets : key => (
        contains(config.encrypted_keys, key) ?
        data.google_kms_secret.secrets["${service_name}-${key}"].plaintext :
        value
      )
    }
  }

  # Prepare encrypted secrets for KMS decryption

  encrypted_secrets = merge([
    for service_name, config in var.services_config : {
      for key in config.encrypted_keys :
      "${service_name}-${key}" => {
        service = service_name
        key     = key
        value   = config.secrets[key]
      }
    }
  ]...)
}

# Decrypt using KMS

data "google_kms_secret" "secrets" {
  for_each = local.encrypted_secrets
  crypto_key = "projects/${var.project_id}/locations/global/keyRings/${var.environment}-keyring/cryptoKeys/${var.environment}-key"
  ciphertext = each.value.value

}

This Terraform code:

1. Iterates through all service configurations

2. Identifies which secrets are encrypted

3. Decrypts them using GCP KMS

4. Combines decrypted and plaintext values

Provisioning to Secret Manager

Once decrypted, the secrets are provisioned to GCP Secret Manager:


module "secret-manager" {
  source     = "GoogleCloudPlatform/secret-manager/google"
  version    = "~> 0.5"
  project_id = var.project_id
  secrets = nonsensitive(flatten([
    for service_name, secrets in local.decrypted_secrets : [
      for key, value in secrets : {
        name        = "${service_name}-${lower(key)}"
        secret_data = value
      }
      if value != ""
    ]
  ]))
  secret_accessors_list = [
    "group:gcp-organization-admins@company.com"
  ]
}

This creates secrets in GCP Secret Manager with:

- Consistent naming: service-name-secret-key

- Proper access controls

- Automatic versioning

The Workflow

Here's the complete workflow for adding a new secret:

1. Developer encrypts the secret using GCP KMS

2. Updates the Terraform variables file with the encrypted value

3. Creates a pull request for review

4. CI runs Terraform plan to validate changes

5. After approval and merge, CI/CD deploys to staging

6. Manual approval triggers production deployment

7. External Secrets Operator syncs to Kubernetes

8. Applications access secrets from Kubernetes secrets

Security Best Practices

1. Encryption at Rest

All sensitive values are encrypted before entering version control. Even if someone gains access to the repository, they can't decrypt without KMS access.

2. Principle of Least Privilege

- KMS keys are environment-specific

- Secret Manager access is controlled via IAM groups

- Applications only access secrets they need

3. Audit Trail

- GCP Cloud Audit Logs track all KMS operations

- Secret Manager maintains access logs

- Terraform state is stored securely in GCS

4. Separation of Concerns

- Developers can add secrets without accessing production systems

- DevOps team controls deployment

- Applications never see encryption keys

5. Automated Rotation

With everything in Terraform, rotating secrets is as simple as:

1. Generate new secret

2. Encrypt with KMS

3. Update tfvars

4. Deploy through CI/CD

Handling Different Secret Types

The system handles various types of secrets elegantly:

Simple Credentials

hcl
DB_USER = "app"  # Can be plaintext
DB_PASS = "CiQA0TkZJabdTVyzUvmU..."  # Encrypted

Complex JSON Structures

ADMINS = "['admin@company.com']"  # Python list as string

Multi-line Keys (like RSA Private Keys)

DOCUSIGN_JWT_PRIVATE_KEY = "CiQA0TkZJevqtcXkUTOY3Ddo..."  # Base64 encoded

The system handles these by:

1. Encrypting the entire content (including newlines)

2. Base64 encoding for storage

3. Proper decryption maintaining format

Kubernetes Integration

The final piece is getting secrets to applications. External Secrets Operator handles this:

1. Watches Secret Manager for changes

2. Creates Kubernetes Secrets automatically

3. Mounts to pods as environment variables or files

4. Handles rotation without pod restarts

Example pod configuration:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: app
    envFrom:
    - secretRef:
        name: secrets

Lessons Learned

1. Start with Encryption Early

Retrofitting encryption is harder than starting with it. Design your secret management from day one.

2. Environment Separation is Critical

Using separate KMS keys for staging and production prevents accidental cross-environment secret usage.

3. Automate Everything

Manual secret management doesn't scale. Automation reduces errors and improves security.

4. Monitor and Audit

Set up alerts for:

- Failed decryption attempts

- Unusual secret access patterns

- Secret rotation reminders

5. Documentation is Security

Well-documented processes mean fewer mistakes. Our README provides clear instructions for the entire team.

Performance Considerations

Caching

Terraform caches decrypted values during runs, avoiding repeated KMS calls.

Batch Operations

The system processes all secrets in parallel, reducing deployment time.

Secret Manager Quotas

Be aware of GCP quotas:

- 60,000 secret versions per project

- 90,000 access requests per minute

Cost Optimization

The solution is cost-effective:

- KMS: $0.06 per key per month + $0.03 per 10,000 operations

- Secret Manager: $0.06 per secret per month + $0.03 per 10,000 operations

- Total monthly cost: ~$50 for 100 secrets with moderate usage

Future Improvements

Looking ahead, potential enhancements include:

1. Secret Rotation Automation: Implementing automatic rotation for database passwords and API keys

2. Break-glass Procedures: Emergency access workflows for critical situations

3. Multi-region Replication: For disaster recovery

4. Secret Usage Analytics: Track which services use which secrets

Conclusion

Building a secure, scalable secrets management system requires careful planning and the right tools. By combining GCP KMS for encryption, Terraform for infrastructure as code, and automated CI/CD pipelines, we've created a solution that:

- Keeps secrets secure at every stage

- Scales with our infrastructure

- Maintains compliance requirements

- Reduces operational overhead

The key takeaway? Security doesn't have to be complicated. With the right architecture and automation, you can build a secrets management system that's both secure and developer-friendly.

Remember: Your secrets are only as secure as your weakest link. Invest in proper secrets management early, and your future self (and security team) will thank you.

---

Have questions or suggestions? Feel free to reach out. Security is a community effort, and sharing knowledge makes us all stronger. at juliusoh@gmail.com

Resources

- [GCP KMS Documentation](https://cloud.google.com/kms/docs)

- [GCP Secret Manager](https://cloud.google.com/secret-manager/docs)

- [Terraform Google Provider](https://registry.terraform.io/providers/hashicorp/google/latest)

- [External Secrets Operator](https://external-secrets.io/)

Building a Secure DevOps Pipeline: Managing Secrets with GCP KMS and Terraform

Introduction

The Challenge

The Architecture

How Encryption Works

Step 1: Encrypting Secrets with GCP KMS

Step 2: Storing Encrypted Values in Terraform

The Terraform Magic

Provisioning to Secret Manager

The Workflow

Security Best Practices

1. Encryption at Rest

2. Principle of Least Privilege

3. Audit Trail

4. Separation of Concerns

5. Automated Rotation

Handling Different Secret Types

Lessons Learned

Performance Considerations

Cost Optimization

Future Improvements

Conclusion

Resources

Comments

More from this blog

Deploying Applications to AWS ECS with Terraform: Infrastructure as Code Guide

Command Palette

Introduction

The Challenge

The Architecture

How Encryption Works

Step 1: Encrypting Secrets with GCP KMS

Step 2: Storing Encrypted Values in Terraform

The Terraform Magic

Provisioning to Secret Manager

The Workflow

Security Best Practices

1. Encryption at Rest

2. Principle of Least Privilege

3. Audit Trail

4. Separation of Concerns

5. Automated Rotation

Handling Different Secret Types

Lessons Learned

Performance Considerations

Cost Optimization

Future Improvements

Conclusion

Resources

Comments

More from this blog