Command Palette
Search for a command to run...
Comments
Join the discussionNo comments yet. Be the first to comment.
Search for a command to run...
No comments yet. Be the first to comment.
In modern DevOps practices, managing secrets securely is one of the most critical aspects of infrastructure management. Whether it's API keys, database credentials, or JWT private keys, these sensitive pieces of information need to be handled with care. In this article, I'll walk through how I implemented a comprehensive secrets management solution using Google Cloud Platform's Key Management Service (KMS) and Secret Manager, automated with Terraform and integrated into a CI/CD pipeline.
When managing multiple services in a cloud environment, you face several challenges:
- Storing hundreds of secrets securely
- Maintaining different configurations for staging and production environments
- Ensuring secrets are encrypted at rest and in transit
- Automating secret rotation and deployment
- Providing secure access to applications without exposing credentials
My solution leverages several GCP services and tools:
┌─────────────────┐
│ Developer │
│ Encrypts │
│ Secrets │
└────────┬────────┘
│
▼
┌─────────────────┐
│ GCP KMS │
│ (Encryption) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Terraform │
│ Variables │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Terraform │
│ Decrypts & │
│ Provisions │
└────────┬────────┘
│
▼
┌─────────────────┐
│ GCP Secret │
│ Manager │
└────────┬────────┘
│
▼
┌─────────────────┐
│External Secrets │
│ Operator │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Kubernetes │
│ Pods │
└─────────────────┘
The first layer of security comes from Google Cloud KMS. Before any secret enters our repository, it's encrypted using environment-specific KMS keys:
echo -n "your-secret-value" | gcloud kms encrypt \
--location=global \
--keyring=stage-keyring \
--key=key-name \
--plaintext-file=- \
--ciphertext-file=- \
--project=your-project-id \
| base64
This command:
1. Takes your plaintext secret
2. Encrypts it using a specific KMS key ring and key
3. Outputs the ciphertext in base64 format
The resulting encrypted value looks something like:
CiQA0TkZJRZqLSLT2w+OLxu1imQozdof+xBGTjepJbwrUW/bynESPQBCuzVB...
The encrypted values are stored in Terraform variable files (`*.tfvars`). Here's the structure:
services_config = {
"hge-django" = {
secrets = {
AWS_ACCESS_KEY_ID = "CiQA0TkZJRZqLSLT2w+OLxu1imQo..."
AWS_SECRET_ACCESS_KEY = "CiQA0TkZJUeuYXTwfli7eaNdDg9s..."
DATABASE_URL = "CiQA0TkZJRJ/hBTBsbb4UFp+O3JG..."
API_HOST = "https://api.com/api" # Plaintext
}
encrypted_keys = [
"AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY",
"DATABASE_URL"
]
}
}
Notice how:
- Sensitive values are encrypted (AWS keys, database URLs)
- Non-sensitive values can remain in plaintext (API endpoints)
- The encrypted_keys array tells Terraform which values need decryption
Dynamic Decryption
The heart of the system is in the Terraform configuration. Here's how it works:
locals {
# Flatten and decrypt secrets for all services
decrypted_secrets = {
for service_name, config in var.services_config : service_name => {
for key, value in config.secrets : key => (
contains(config.encrypted_keys, key) ?
data.google_kms_secret.secrets["${service_name}-${key}"].plaintext :
value
)
}
}
# Prepare encrypted secrets for KMS decryption
encrypted_secrets = merge([
for service_name, config in var.services_config : {
for key in config.encrypted_keys :
"${service_name}-${key}" => {
service = service_name
key = key
value = config.secrets[key]
}
}
]...)
}
# Decrypt using KMS
data "google_kms_secret" "secrets" {
for_each = local.encrypted_secrets
crypto_key = "projects/${var.project_id}/locations/global/keyRings/${var.environment}-keyring/cryptoKeys/${var.environment}-key"
ciphertext = each.value.value
}
This Terraform code:
1. Iterates through all service configurations
2. Identifies which secrets are encrypted
3. Decrypts them using GCP KMS
4. Combines decrypted and plaintext values
Once decrypted, the secrets are provisioned to GCP Secret Manager:
module "secret-manager" {
source = "GoogleCloudPlatform/secret-manager/google"
version = "~> 0.5"
project_id = var.project_id
secrets = nonsensitive(flatten([
for service_name, secrets in local.decrypted_secrets : [
for key, value in secrets : {
name = "${service_name}-${lower(key)}"
secret_data = value
}
if value != ""
]
]))
secret_accessors_list = [
"group:gcp-organization-admins@company.com"
]
}
This creates secrets in GCP Secret Manager with:
- Consistent naming: service-name-secret-key
- Proper access controls
- Automatic versioning
Here's the complete workflow for adding a new secret:
1. Developer encrypts the secret using GCP KMS
2. Updates the Terraform variables file with the encrypted value
3. Creates a pull request for review
4. CI runs Terraform plan to validate changes
5. After approval and merge, CI/CD deploys to staging
6. Manual approval triggers production deployment
7. External Secrets Operator syncs to Kubernetes
8. Applications access secrets from Kubernetes secrets
All sensitive values are encrypted before entering version control. Even if someone gains access to the repository, they can't decrypt without KMS access.
- KMS keys are environment-specific
- Secret Manager access is controlled via IAM groups
- Applications only access secrets they need
- GCP Cloud Audit Logs track all KMS operations
- Secret Manager maintains access logs
- Terraform state is stored securely in GCS
- Developers can add secrets without accessing production systems
- DevOps team controls deployment
- Applications never see encryption keys
With everything in Terraform, rotating secrets is as simple as:
1. Generate new secret
2. Encrypt with KMS
3. Update tfvars
4. Deploy through CI/CD
The system handles various types of secrets elegantly:
Simple Credentials
hcl
DB_USER = "app" # Can be plaintext
DB_PASS = "CiQA0TkZJabdTVyzUvmU..." # Encrypted
Complex JSON Structures
ADMINS = "['admin@company.com']" # Python list as string
Multi-line Keys (like RSA Private Keys)
DOCUSIGN_JWT_PRIVATE_KEY = "CiQA0TkZJevqtcXkUTOY3Ddo..." # Base64 encoded
The system handles these by:
1. Encrypting the entire content (including newlines)
2. Base64 encoding for storage
3. Proper decryption maintaining format
Kubernetes Integration
The final piece is getting secrets to applications. External Secrets Operator handles this:
1. Watches Secret Manager for changes
2. Creates Kubernetes Secrets automatically
3. Mounts to pods as environment variables or files
4. Handles rotation without pod restarts
Example pod configuration:
apiVersion: v1
kind: Pod
spec:
containers:
- name: app
envFrom:
- secretRef:
name: secrets
1. Start with Encryption Early
Retrofitting encryption is harder than starting with it. Design your secret management from day one.
2. Environment Separation is Critical
Using separate KMS keys for staging and production prevents accidental cross-environment secret usage.
3. Automate Everything
Manual secret management doesn't scale. Automation reduces errors and improves security.
4. Monitor and Audit
Set up alerts for:
- Failed decryption attempts
- Unusual secret access patterns
- Secret rotation reminders
5. Documentation is Security
Well-documented processes mean fewer mistakes. Our README provides clear instructions for the entire team.
Caching
Terraform caches decrypted values during runs, avoiding repeated KMS calls.
Batch Operations
The system processes all secrets in parallel, reducing deployment time.
Secret Manager Quotas
Be aware of GCP quotas:
- 60,000 secret versions per project
- 90,000 access requests per minute
The solution is cost-effective:
- KMS: $0.06 per key per month + $0.03 per 10,000 operations
- Secret Manager: $0.06 per secret per month + $0.03 per 10,000 operations
- Total monthly cost: ~$50 for 100 secrets with moderate usage
Looking ahead, potential enhancements include:
1. Secret Rotation Automation: Implementing automatic rotation for database passwords and API keys
2. Break-glass Procedures: Emergency access workflows for critical situations
3. Multi-region Replication: For disaster recovery
4. Secret Usage Analytics: Track which services use which secrets
Building a secure, scalable secrets management system requires careful planning and the right tools. By combining GCP KMS for encryption, Terraform for infrastructure as code, and automated CI/CD pipelines, we've created a solution that:
- Keeps secrets secure at every stage
- Scales with our infrastructure
- Maintains compliance requirements
- Reduces operational overhead
The key takeaway? Security doesn't have to be complicated. With the right architecture and automation, you can build a secrets management system that's both secure and developer-friendly.
Remember: Your secrets are only as secure as your weakest link. Invest in proper secrets management early, and your future self (and security team) will thank you.
---
Have questions or suggestions? Feel free to reach out. Security is a community effort, and sharing knowledge makes us all stronger. at juliusoh@gmail.com
- [GCP KMS Documentation](https://cloud.google.com/kms/docs)
- [GCP Secret Manager](https://cloud.google.com/secret-manager/docs)
- [Terraform Google Provider](https://registry.terraform.io/providers/hashicorp/google/latest)
- [External Secrets Operator](https://external-secrets.io/)