DEV Community

Cover image for AWS Secrets Manager Patterns for Python Microservices
Muhammad Umair Virk
Muhammad Umair Virk

Posted on Edited on

AWS Secrets Manager Patterns for Python Microservices

#aws #security #devops #python

Hardcoding secrets is the fastest way to get paged at 3 AM — or worse, flagged in a security audit.

Centralized secret management is not optional once you run more than one microservice. I have implemented shared retrieval layers in Python (Flask, FastAPI, Lambda) using AWS Secrets Manager. These are the patterns that survive production — caching, least privilege, rotation, and what breaks when you skip any of them.

Pattern 1: Never bake secrets into images 

# ❌ NEVER
DATABASE_URL = "postgresql://user:password123@host/db"

# ✅ Runtime injection
import os
DATABASE_URL = os.environ["DATABASE_URL"]
Enter fullscreen mode Exit fullscreen mode

In Docker, use .env files locally and AWS-injected env vars in production. Add .env to .dockerignore and .gitignore.

Pattern 2: Centralized retrieval helper 

import boto3
import json
from functools import lru_cache

client = boto3.client("secretsmanager", region_name="me-south-1")


@lru_cache(maxsize=32)
def get_secret(secret_name: str) -> dict:
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])


def get_database_url() -> str:
    secrets = get_secret("prod/taskflow/database")
    return (
        f"postgresql://{secrets['username']}:{secrets['password']}"
        f"@{secrets['host']}:{secrets['port']}/{secrets['dbname']}"
    )
Enter fullscreen mode Exit fullscreen mode

Why @lru_cache? Secrets Manager charges per API call. Cache in-process for Lambda warm instances and long-running containers. Invalidate on rotation events if needed.

Pattern 3: IAM least privilege

Each service gets its own IAM role with access to only its secrets:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:me-south-1:123456789:secret:prod/taskflow/*"
}
Enter fullscreen mode Exit fullscreen mode

Never use one mega-role for all secrets across all services.

Pattern 4: Lambda vs. long-running services

Runtime Strategy
Lambda Fetch on cold start, cache in module scope
ECS / Docker Fetch on startup, cache in app singleton
Local dev .env file or LocalStack 
# Lambda — module-level cache survives warm invocations
_db_url = None

def get_db_url():
    global _db_url
    if _db_url is None:
        _db_url = get_database_url()
    return _db_url
Enter fullscreen mode Exit fullscreen mode

Pattern 5: Secret rotation

Enable automatic rotation in Secrets Manager for database credentials. Your app must handle connection drops gracefully:

from sqlalchemy import create_engine
from sqlalchemy.pool import NullPool

engine = create_engine(get_database_url(), pool_pre_ping=True)
Enter fullscreen mode Exit fullscreen mode

pool_pre_ping=True verifies connections before use — critical after rotation.

Pattern 6: Structured logging — never log secrets 

import logging
logger = logging.getLogger(__name__)

# ❌ logger.info(f"Connecting to {DATABASE_URL}")
# ✅
logger.info("database_connection_initialized", extra={"host": secrets["host"]})
Enter fullscreen mode Exit fullscreen mode

Redact secrets in error handlers and APM tools.

Pattern 7: Compose locally, Secrets Manager in prod 

# docker-compose.yml (local)
services:
  api:
    env_file:
      - .env
Enter fullscreen mode Exit fullscreen mode 
# ECS task definition (prod)
secrets:
  - name: DATABASE_URL
    valueFrom: arn:aws:secretsmanager:...:secret:prod/taskflow/database-url
Enter fullscreen mode Exit fullscreen mode

Same app code. Different secret source per environment.

What I built

At work, I implemented a shared secret retrieval layer used across Python and Go services — standardizing how services access database credentials, API keys, and third-party tokens. Result: zero hardcoded secrets in repos, audit-friendly access logs, and rotation without redeploying application code.

Checklist for your next project

  • [ ] All secrets in Secrets Manager (or Parameter Store for non-sensitive config)
  • [ ] .env in .gitignore and .dockerignore
  • [ ] IAM role per service, least privilege
  • [ ] In-process caching with rotation awareness
  • [ ] pool_pre_ping on database connections
  • [ ] No secrets in logs, CI output, or error messages

Further reading

Muhammad Umair Virk — Backend Engineer, UAE. Python · AWS · microservices · payments.

Top comments (0)