Backups

What is backed up, where, for how long, and what isn't yet. Restoring is in Restoring Data.

Today, only the database has a backup outside one place

Postgres has DigitalOcean's daily backups and 7 days of point-in-time recovery. Everything else listed as a gap below is lost if its single copy goes. Until CI/CD Phase 5 ships, do the manual steps on this page.

Inventory

Data	Lives in	Backup today	Planned (CI/CD Phase 5)
Postgres (users, posts, coins, earnings, payouts)	Managed cluster playpalz-production-db (PG 15, private VPC)	Daily DO backups (~21:45 UTC), kept 7 days; point-in-time recovery to any second in the last 7 days	Pre-deploy pg_dump (30 days), nightly dump (35 days), monthly (400 days), all age-encrypted in playpalz-backups; weekly restore test
Redis (BullMQ queues, caches, presence)	StatefulSet playpalz-redis, PVC playpalz-redis-pvc (5Gi)	On-volume only AOF (fsync every second) + RDB snapshots survive restarts, not the loss of the volume	Daily redis-cli --rdb to playpalz-backups/redis/ (14 days)
Media (photos, video originals, thumbnails)	Spaces bucket playpalz-media-production + CDN	None No versioning: a deleted or overwritten object is gone	Bucket versioning; noncurrent versions expire after 30 days
KYC documents	Spaces bucket playpalz-kyc	Deliberately none. Encrypted, unversioned, so deletion under the retention policy is real	Unchanged
Sealed-secrets key (decrypts every SealedSecret in git)	Secret in namespace sealed-secrets	None Lose the cluster and every sealed secret in git is unreadable	Weekly encrypted export to playpalz-backups/cluster/ (90 days)
Terraform state	infra/terraform/terraform.tfstate on one laptop (gitignored; the S3 backend is commented out)	None	Spaces backend in versioned bucket playpalz-tfstate
Code, manifests, sealed secrets, migrations	GitHub cryptixcoder/playpals	Git	Unchanged
Container images	DOCR playpalzproduction	Rebuildable from git at any commit	Digests from the last 20 Helm revisions are never garbage-collected
Mobile builds and updates	EAS	EAS keeps every build and update	Unchanged
Third-party state (Stripe, RevenueCat, Mux, App Store, Play)	The providers	Their systems of record; Postgres keeps the IDs to reconcile against	Unchanged

Restore order after a disaster: Terraform state → cluster → sealed-secrets key → secrets → Postgres → Redis → services. See Restoring Data → Full recovery.

Until Phase 5: manual backups Today

Do these now, and again after any change to what they cover.

Sealed-secrets key: once, and after every key renewal (30 days)

kubectl -n sealed-secrets get secret \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-keys-$(date +%F).yaml

Store the file in the team password manager, not in git and not on a laptop, then delete the local copy. Anyone holding it can decrypt every secret in infra/k8s/sealed-secrets/.

Terraform state: after every terraform apply

Copy infra/terraform/terraform.tfstate to the password manager or an encrypted, access-controlled store. It contains resolved secrets: the database password and the Spaces keys. Losing it doesn't take production down, but every resource would have to be imported back by hand.

A Postgres dump: before a risky migration or a large data change

The database only accepts connections from inside the VPC, so dump from a pod in the cluster:

# A throwaway pod with the database credentials
kubectl run pg-dump --restart=Never --image=postgres:15-alpine \
  --overrides='{"spec":{"containers":[{"name":"pg-dump","image":"postgres:15-alpine","command":["sleep","3600"],"envFrom":[{"secretRef":{"name":"database-credentials"}}]}]}}'
kubectl wait --for=condition=Ready pod/pg-dump

# Stream a custom-format dump to your machine, encrypted on the way
kubectl exec pg-dump -- sh -c 'PGPASSWORD="$DB_PASSWORD" pg_dump -Fc -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" "$DB_NAME"' \
  | age -r <team-age-public-key> > playpalz-$(date +%F-%H%M).dump.age

kubectl delete pod pg-dump

The dump holds personal data. Keep it encrypted (age), keep it off shared drives, and delete it once the change has settled. Check that it's usable with age -d -i key.txt playpalz-….dump.age | pg_restore --list | head.

Redis: before maintenance that touches the volume

A CSI snapshot of the volume. The cluster has the do-block-storage snapshot class:

kubectl apply -f - <<'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: redis-manual-YYYYMMDD
  namespace: default
spec:
  volumeSnapshotClassName: do-block-storage
  source:
    persistentVolumeClaimName: playpalz-redis-pvc
EOF
kubectl get volumesnapshot redis-manual-YYYYMMDD   # READYTOUSE: true

The class's deletion policy is Delete: deleting the VolumeSnapshot object deletes the snapshot. Snapshots cost $0.06/GiB-month; delete old ones.

Redis holds queues and caches, not records. Losing it loses queued media jobs and makes everyone sign in again on realtime. It doesn't lose money or content. Back it up before volume work; don't depend on it.

How Phase 5 works Planned

• Bucket playpalz-backups:

  • Private Spaces bucket with versioning and prevent_destroy.

  • Its own bucket-scoped key, the same pattern as playpalz-kyc.

  • Lifecycle expiry per prefix:

    Prefix	Retention
    predeploy/	30 days
    nightly/	35 days
    monthly/	400 days
    redis/	14 days
    cluster/	90 days

• Encryption: every object is encrypted with age to the team's public key. The private key lives only in the password manager, so the cluster can write backups but can't read them.

• Jobs: each one pings a dead-man's-switch monitor on success, so a backup that silently stops running raises an alert.

  Job	When	Writes
  Pre-deploy dump (Helm hook, api release)	Before any production deploy whose commit range touches prisma/migrations. A failed dump stops the deploy.	predeploy/<sha>-<ts>.dump.age
  Nightly dump (CronJob)	03:00 UTC	nightly/, and on the 1st also monthly/
  Redis (CronJob)	Daily	redis/ (RDB)
  Cluster state (CronJob)	Weekly	cluster/ (sealed-secrets keys)
  Restore test (CronJob)	Weekly	Nothing kept. Restores the latest nightly into a throwaway Postgres and row-counts key tables.

• Terraform state moves to a versioned Spaces bucket (terraform init -migrate-state). Versioning is its backup.

• Media bucket versioning is turned on; noncurrent versions expire after 30 days.

Restore drills

A backup that has never been restored isn't a backup.

Drill	How often	How
Postgres PITR into a new cluster	Quarterly	Restore → PITR, against a throwaway cluster; delete it afterwards
Dump restore	Weekly (automated from Phase 5), monthly by hand before that	Restore → from a dump into a local Postgres
Sealed-secrets key	Once, then yearly	Restore the key into a local kind cluster with the controller, and unseal one secret
Media object	After versioning ships, quarterly	Delete a test object, restore its previous version

Record each drill (date, duration, problems) in the incident log. The time a drill takes is your real recovery time.