Secrets Management

Two mechanisms, split by who owns the value.

Mechanism	Owns	Committed?
Terraform-generated	database-credentials, spaces-credentials	No — created by terraform apply
Sealed Secrets	Everything else	Yes — encrypted, safe in git

Why Sealed Secrets

A plain Kubernetes Secret is base64, not encryption. Committing one publishes it.

Bitnami Sealed Secrets fixes this with asymmetric encryption: you encrypt with a public certificate, and only the controller running in the cluster holds the private key. The resulting SealedSecret is safe to commit, review, and diff — and the controller decrypts it into a real Secret in-cluster.

Rendering diagram…

Encryption is scoped to a namespace and secret name, so a sealed secret cannot be moved to another namespace or renamed — which is why staging needs its own re-sealed copies.

Inventory

Secret	Keys	Consumed by	Managed by
database-credentials	DATABASE_URL, DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, DB_NAME	api, anansi, ogun, igdb-heartbeat	Terraform
spaces-credentials	AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, S3_ENDPOINT, S3_URL, S3_BUCKET, CDN_ENDPOINT	api, ogun, igdb-heartbeat	Terraform
redis-secrets	REDIS_HOST, REDIS_PORT, REDIS_PASSWORD, REDIS_URI	api, esu, anansi, ogun	Sealed
kyc-storage	KYC_SPACES_BUCKET, KYC_SPACES_REGION, KYC_SPACES_ENDPOINT, KYC_SPACES_ACCESS_KEY, KYC_SPACES_SECRET_KEY	api	Terraform
api-secrets	JWT_SECRET, RESET_SECRET (both ≥ 32 characters or the API won't start), RESEND_API_KEY (to add), REVENUECAT_WEBHOOK_SECRET, KYC_ENCRYPTION_KEY, optional REVENUECAT_V2_SECRET_KEY and REVENUECAT_PROJECT_ID	api	Sealed
esu-secrets	API_SERVICE_TOKEN, REALTIME_ADMIN_TOKEN	api, esu	Sealed
stripe-secrets	STRIPE_PUBLISHABLE_KEY, STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, optional STRIPE_TRANSFER_WEBHOOK_SECRET	api, anansi	Sealed
mux-secrets	MUX_ACCESS_TOKEN, MUX_SECRET_KEY, MUX_WEBHOOK_SECRET	api, ogun	Sealed
livekit-secrets	LIVEKIT_HOST, LIVEKIT_WS_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET	api	Sealed
igdb-secrets	IGDB_CLIENT_ID, IGDB_CLIENT_SECRET	igdb-heartbeat	Sealed
grafana-secrets	Grafana admin credentials	Grafana	Sealed

CDN_ENDPOINT is the bare CDN host (playpalz-media-production.nyc3.cdn.digitaloceanspaces.com, no scheme), straight from digitalocean_cdn.endpoint. Anything that builds URLs from it must add https://; S3_URL already includes it.

KYC_ENCRYPTION_KEY and STRIPE_WEBHOOK_SECRET are referenced with optional: true in api-deployment.yaml. The API starts without them, and creator onboarding and the Stripe webhook answer 503 until they are sealed in. KYC_ENCRYPTION_KEY is the one secret here whose loss destroys data: every stored ID becomes unreadable. Keep a copy in the team password manager.

Terraform keeps database-credentials, spaces-credentials and kyc-storage because it creates the underlying resources and already knows the values — sealing them would mean copying secrets by hand for no benefit.

Setting up the controller

Once per cluster:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update

helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace sealed-secrets --create-namespace

kubectl get pods -n sealed-secrets

Then fetch the public certificate, which is committed so anyone can seal a secret without cluster access:

kubeseal --fetch-cert \
  --controller-name=sealed-secrets \
  --controller-namespace=sealed-secrets \
  > infra/k8s/sealed-secrets/public-cert.pem

Creating a sealed secret

infra/k8s/sealed-secrets/seal.sh wraps kubeseal with the right certificate and namespace:

kubectl create secret generic livekit-secrets \
  --namespace default \
  --from-literal=LIVEKIT_HOST='ws.playpalz.gg' \
  --from-literal=LIVEKIT_WS_URL='wss://ws.playpalz.gg' \
  --from-literal=LIVEKIT_API_KEY='…' \
  --from-literal=LIVEKIT_API_SECRET='…' \
  --dry-run=client -o yaml \
| ./infra/k8s/sealed-secrets/seal.sh livekit-secrets

Or from an env file:

kubectl create secret generic stripe-secrets \
  --namespace default \
  --from-env-file=./apps/api/.env.stripe \
  --dry-run=client -o yaml \
| ./infra/k8s/sealed-secrets/seal.sh stripe-secrets

--dry-run=client is what keeps the plaintext secret out of the cluster and off disk — it is generated, piped, and discarded.

Staging has its own copy of every secret, sealed for the staging namespace. A SealedSecret only decrypts in the namespace it was sealed for. Pass the namespace as the second argument and the file lands in infra/k8s/sealed-secrets/staging/:

kubectl create secret generic mux-secrets --namespace staging \
  --from-literal=MUX_ACCESS_TOKEN='…' --from-literal=MUX_SECRET_KEY='…' --from-literal=MUX_WEBHOOK_SECRET='…' \
  --dry-run=client -o yaml \
| ./infra/k8s/sealed-secrets/seal.sh mux-secrets staging
kubectl apply -f infra/k8s/sealed-secrets/staging/mux-secrets.yaml

Staging's values:

• Generated for staging: JWT, reset and KYC keys, the service tokens, and the Postgres and Redis passwords.
• Test mode: Stripe.
• Copied from production: LiveKit and IGDB.
• Placeholders until set: Mux, and the Stripe webhook secrets.
• From Terraform (staging.tf): spaces-credentials and kyc-storage, with keys scoped to the staging buckets, plus the registry pull secret.

dev-notes/staging-credentials.md (gitignored) holds the one value someone has to paste elsewhere: RevenueCat's staging webhook header.

Then apply and commit:

kubectl apply -f infra/k8s/sealed-secrets/livekit-secrets.yaml
git add infra/k8s/sealed-secrets/livekit-secrets.yaml

Consuming a secret

env:
  - name: JWT_SECRET
    valueFrom:
      secretKeyRef:
        name: api-secrets
        key: JWT_SECRET

Pods do not pick up a changed secret automatically:

kubectl rollout restart deployment/playpalz-api

Rotating

1. Create and seal a new version of the secret.
2. kubectl apply the sealed secret.
3. kubectl rollout restart every deployment that consumes it.
4. Commit the updated file.

Rotating JWT_SECRET logs every user out — every issued token becomes unverifiable. Rotating REALTIME_ADMIN_TOKEN requires restarting api, esu, and ogun together, or realtime fan-out breaks in the gap.

Local secrets

Local development uses .env files, one per service, each with a committed .env.example. Never commit a real .env — the root .gitignore covers them.

Housekeeping

Plaintext credentials on disk in dev-notes/

dev-notes/secrets-commands.md contains live production credentials in plaintext — the Spaces access key and secret, the production PostgreSQL connection string and password, and the LiveKit API secret. dev-notes/info.md contains an account password.

dev-notes/ is listed in .gitignore, so none of this is in git history — the exposure is limited to developer machines and anything that syncs them (cloud drives, backups, AI tooling with filesystem access).

Treat those files as a credential leak waiting to happen: move the values to a password manager, replace them in the file with placeholders, and rotate anything that has been shared or synced.

dev-notes/ is not in the repository

Because the whole directory is git-ignored, a fresh clone does not include it. That takes the design system specification (dev-notes/playpalz-design-system.html), the sealed-secrets guide, and every implementation plan with it.

The credential files should stay out of git. The design system and the operational guides should not — consider moving the non-sensitive ones into apps/docs/ so they reach the next developer.

Disaster recovery

The sealed-secrets controller's private key is the only thing that can decrypt the committed sealed secrets. If you lose the cluster, you lose the ability to decrypt them.

Back up the key:

kubectl get secret -n sealed-secrets \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-key-backup.yaml

Store that file somewhere genuinely secure — it is equivalent to every secret it protects. Restore by applying it to a new cluster before the controller generates a fresh key.

Without the backup, recovery means re-creating and re-sealing every secret from their original sources.