How do you manage configuration in microservices?
Here’s a practical playbook for config management in microservices—what to store, where to store it, and how to ship it safely.
1) Principles (what “good” looks like)
- Externalize config: never hard‑code; keep it out of images/binaries (12‑factor).
- Per‑environment overrides: dev/stage/prod differ via config, not code.
- Immutable deploys, mutable config: change behavior without rebuilds.
- Separation of concerns: config ≠ secrets (manage them differently).
- Least privilege & audit: who changed what, when.
2) Types of configuration
- Non‑secret app settings: feature flags, URLs, timeouts, limits.
- Environment & infra: DB hosts, cache endpoints, topic names.
- Secrets: passwords, API keys, tokens, certs (rotate!).
- Dynamic runtime config: feature gates, kill‑switches, traffic %.
3) Storage & delivery options
A) Simple & reliable (good defaults)
- Environment variables for per‑service runtime.
- Kubernetes ConfigMaps for non‑secrets; Secrets for secrets.
- Docker/Helm values per environment.
B) Centralized config service (for fleets)
- Spring Cloud Config / Consul KV / etcd / AWS AppConfig: versioned config from a central store.
- Patterns: clients fetch on startup + watch/TTL to refresh at runtime.
-
Pros: consistency, dynamic changes, audit trails.
Cons: extra HA component; must cache/fallback.
C) Secrets managers (never treat like plain config)
- HashiCorp Vault, AWS Secrets Manager, AWS SSM Parameter Store, GCP Secret Manager, Azure Key Vault.
- Use short‑lived creds, automatic rotation, envelope encryption (KMS).
- Deliver via sidecar/agent, init containers, or native SDK with caching.
D) Feature flags
- LaunchDarkly/Unleash/Flagsmith or homegrown in config store.
- Target by user/tenant/percentage; enable canary & kill‑switches without redeploy.
4) Refresh & safety patterns
- Hot reload: watch config keys; apply atomically with validation.
- Graceful fallback: on fetch failure, use last known good (LKG) from local cache.
- Boot sequence: minimal bootstrap (registry/secret endpoints) shipped with image; everything else fetched.
- Circuit breakers/timeouts around config/secret lookups to avoid startup hangs.
-
Config schema & validation: fail fast on bad types/ranges; add constraints (e.g.,
0 < TIMEOUT_MS < 30000
).
5) Versioning, change control, and rollout
- Version every config (Git commit/param version). Tag with app version if needed.
- GitOps: config as code; PR reviews, CI validation, audit trails.
- Progressive delivery: apply to 1% → 10% → 100% via flags or label selectors.
-
Blue‑green config: keep
/v1
and/v2
keys side‑by‑side; switch pointers. - Rollback: one click to LKG; keep N previous versions.
6) Security hygiene
- Never log secrets; mask in UIs/logs.
- Use mTLS between apps and config/secret stores.
- RBAC: per‑service identities (SPIFFE/SPIRE, IAM roles, K8s service accounts).
- Encrypt at rest + in transit; restrict egress.
- Rotate: credentials, tokens, and certificates on a schedule or automatically.
7) Observability for config
- Emit config fingerprints (hash/version) on startup and on reload.
- Metrics: config load successes/failures, reload counts, LKG usage.
- Tracing: add config version to spans for incident correlation.
8) Anti‑patterns (avoid)
- Baking secrets into images/env files in Git.
- Shared “global” config that couples unrelated services.
- Runtime fetch with no caching/fallback (single point of failure).
- Massive “god” JSON with all flags for all services.
- Unbounded dynamic changes (no validation/rollout plan).
9) Minimal, copy‑pasteable examples
Kubernetes (non‑secret & secret)
apiVersion: v1
kind: ConfigMap
metadata: { name: orders-config }
data:
APP_TIMEOUT_MS: "800"
FEATURE_NEW_CHECKOUT: "false"
---
apiVersion: v1
kind: Secret
metadata: { name: orders-secrets }
type: Opaque
stringData:
DB_URL: "postgres://orders:***@db:5432/orders"
# Deployment excerpt
env:
- name: APP_TIMEOUT_MS
valueFrom: { configMapKeyRef: { name: orders-config, key: APP_TIMEOUT_MS } }
- name: FEATURE_NEW_CHECKOUT
valueFrom: { configMapKeyRef: { name: orders-config, key: FEATURE_NEW_CHECKOUT } }
- name: DB_URL
valueFrom: { secretKeyRef: { name: orders-secrets, key: DB_URL } }
Spring Boot (pull from Spring Cloud Config + Vault)
spring.cloud.config.uri=https://config:8888
spring.cloud.vault.uri=https://vault:8200
spring.cloud.vault.authentication=kubernetes
management.endpoint.refresh.enabled=true
At runtime: POST /actuator/refresh
(or bus refresh) to hot‑reload after validation.
Consul KV watch (pseudo)
consul kv get app/orders/config # bootstrap
consul watch -type=key -key app/orders/config ./on-change.sh
Feature flag check (pseudo)
if flags.is_enabled("new_checkout", user_id):
enable_new_flow()
else:
use_legacy()
10) Checklist
- Config outside code/image ✅
- Secrets in a secrets manager, rotated, never logged ✅
- Central store (optional) with caching + LKG ✅
- Hot reload with schema validation & safe defaults ✅
- Versioned, reviewed, auditable changes (GitOps) ✅
- Progressive rollout & instant rollback ✅
- mTLS, RBAC, least privilege ✅
- Telemetry: config version/fingerprint, reload metrics ✅