Restoring Data

How to get data back. What exists to restore from is in Backups. If a deploy broke something but the data is fine, you want Rollback, not this page.

Before any restore

A restore rewinds data, and everything written after the restore point is lost. That means signups, posts, coin purchases, tips, bookings and payouts.

1. Be certain.
   • Is the data really gone, or is a bug hiding it?
   • Can a forward fix (a migration or a script) repair it instead?
   • Can you restore only the affected rows (option B) rather than the whole database?
2. Pick the restore point. Find the last moment the data was good (logs, Grafana, the migration's deploy time) and write it down in UTC. Choose a point just before the damage.
3. Stop the damage. Roll back the bad release, and pause anything still writing bad data: scale the service to 0, or switch off the job.
4. Announce it. Whoever is restoring says so in the team channel, with the restore point. Nobody deploys until the restore is finished.

Postgres

Choose the smallest restore that fixes the problem:

Situation	Use
A few rows or one table damaged (bad script, bug, accidental delete)	B: from a dump into a scratch database, then copy back what's needed
Widespread damage within the last 7 days	A: point-in-time recovery
Widespread damage older than 7 days	B with a full restore from a nightly or monthly dump (Phase 5)
The cluster itself is gone	A from its last backup, or B from the latest dump

A. Point-in-time recovery into a new cluster

DigitalOcean restores into a new cluster. The damaged one keeps running until you switch over, which also means you can compare the two.

OLD=playpalz-production-db
NEW=playpalz-production-db-restore-$(date +%Y%m%d)
VPC=$(doctl vpcs list --format ID,Name --no-header | awk '/playpalz/{print $1}')

doctl databases create "$NEW" \
  --restore-from-cluster-name "$OLD" \
  --restore-from-timestamp "2026-10-30 14:05:00 +0000 UTC" \
  --engine pg --version 15 --size db-s-2vcpu-4gb --num-nodes 1 \
  --region nyc3 --private-network-uuid "$VPC" \
  --wait

Leave out --restore-from-timestamp to use the latest daily backup. Restoring takes from minutes to an hour, depending on size.

Let the cluster in. The new cluster has no firewall rules. Copy the old cluster's rules, which are the Kubernetes cluster and the LiveKit droplet:

NEW_ID=$(doctl databases list --format ID,Name --no-header | awk -v n="$NEW" '$2==n{print $1}')
K8S_ID=$(doctl kubernetes cluster list --format ID,Name --no-header | awk '/playpalz/{print $1}')
OLD_ID=$(doctl databases list --format ID,Name --no-header | awk -v n="$OLD" '$2==n{print $1}')
doctl databases firewalls list "$OLD_ID"     # the rules to copy
doctl databases firewalls append "$NEW_ID" --rule k8s:"$K8S_ID"
doctl databases firewalls append "$NEW_ID" --rule droplet:<livekit-droplet-id>
doctl databases user list "$NEW_ID"          # the app user should be there
doctl databases connection "$NEW_ID" --private --format Host,Port

Check it before switching. From a pod in the cluster (see the throwaway pg-dump pod in Backups), connect to the new host and confirm the data looks right at the restore point: row counts, the newest rows in the damaged tables, and _prisma_migrations.

Switch over. database-credentials is managed by Terraform, so edit it in place now and reconcile Terraform afterwards:

kubectl edit secret database-credentials      # DB_HOST, DB_PORT and the host in DATABASE_URL (base64)
kubectl rollout restart deployment/playpalz-api deployment/playpalz-anansi \
  deployment/playpalz-ogun deployment/playpalz-igdb-heartbeat
kubectl rollout status deployment/playpalz-api

Afterwards:

• Reconcile Terraform before the next apply. It still points at the old cluster and would rewrite the secret back to it. Move the resource to the new cluster with terraform state rm / terraform import, and update the name. Read the plan: any must be replaced on a database resource is a stop.
• Reconcile money for the lost window. Stripe, RevenueCat and the stores are the systems of record for payments made after the restore point:
  • Run reconciliation from Jobs in the admin panel.
  • Compare purchases and transfers in the Stripe and RevenueCat dashboards for that window.
  • Re-apply anything missing with the admin tools, not raw SQL.
• Keep the old cluster for a few days, powered on, in case something needs copying from it. Then destroy it.

B. From a dump: tables or rows

Restore a dump into a scratch database, never into production, and copy back only what's needed.

# Local Postgres 15 (docker compose up -d), a scratch database
createdb -h localhost -U postgres playpalz_restore
age -d -i key.txt playpalz-2026-10-30-1400.dump.age \
  | pg_restore -h localhost -U postgres -d playpalz_restore --no-owner --no-privileges

# Inspect, then extract exactly what's needed
psql -h localhost -U postgres playpalz_restore -c 'select count(*) from "Post" where "userId" = ...'
pg_dump -h localhost -U postgres playpalz_restore --data-only -t '"Post"' --where ... > rows.sql

Copying back into production:

• Write it as a reviewed SQL script. Two people read it. Run it in a transaction from a pod in the cluster, and check the row counts before committing.
• Never hand-insert into money tables: the coin ledger, earnings, payouts or purchases. They're double-entry, with triggers and constraints. Use the admin panel's corrections and reconciliation, so the books still balance.
• Mind foreign keys and IDs. Restore parents before children, and keep the original IDs. Mobile clients and Stripe metadata refer to them.

A full restore from a dump (the whole database is older than 7 days, or the cluster is gone): create an empty cluster (doctl databases create … --version 15), let the cluster in, and pg_restore into it from a pod. Then switch over exactly as in option A.

Redis

Redis holds queues and caches. Before restoring anything, ask whether starting empty is fine. Usually it is:

• queued media jobs are reprocessed by the media sweep
• people sign in to realtime again

From a volume snapshot:

kubectl scale statefulset/playpalz-redis --replicas=0
kubectl delete pvc playpalz-redis-pvc                 # the snapshot is independent of it
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: playpalz-redis-pvc
  namespace: default
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: do-block-storage
  resources:
    requests:
      storage: 5Gi
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: redis-manual-YYYYMMDD
EOF
kubectl scale statefulset/playpalz-redis --replicas=1
kubectl rollout restart deployment/playpalz-api deployment/playpalz-esu deployment/playpalz-ogun deployment/playpalz-anansi

Phase 5 From an RDB file in playpalz-backups/redis/:

1. Scale Redis to 0.
2. Mount the volume in a helper pod, put the file at /data/dump.rdb, and remove /data/appendonlydir. With AOF on, Redis loads the AOF instead of the RDB.
3. Start Redis.
4. Run redis-cli BGREWRITEAOF to rebuild the AOF from the loaded data.

Media (Spaces)

After versioning ships (Phase 5) Until then, a deleted or overwritten media object can't be recovered.

EP=https://nyc3.digitaloceanspaces.com
B=playpalz-media-production
aws s3api list-object-versions --endpoint-url $EP --bucket $B --prefix "path/to/object"
aws s3api copy-object --endpoint-url $EP --bucket $B --key "path/to/object" \
  --copy-source "$B/path/to/object?versionId=<version-id>"
doctl compute cdn flush <cdn-id> --files "/path/to/object"     # so the CDN serves it again

Use the media Spaces key (spaces-credentials) as the AWS credentials. The KYC bucket is unversioned on purpose, so it can't be restored.

Sealed-secrets key

You need this when the cluster is rebuilt. Without the old key, the controller can't decrypt the SealedSecrets in git.

# New cluster, sealed-secrets controller installed (Terraform / Helm), before applying any SealedSecret
kubectl apply -f sealed-secrets-keys-YYYY-MM-DD.yaml          # from the password manager
kubectl -n sealed-secrets delete pod -l app.kubernetes.io/name=sealed-secrets
kubectl apply -f infra/k8s/sealed-secrets/
kubectl get secrets                                          # unsealed secrets appear

If the key is lost, every secret has to be re-created from its source (Stripe, Mux, RevenueCat and so on) and re-sealed with the new controller's key. Rotating the ones you can is wise anyway.

Terraform state

• Today: copy the latest saved terraform.tfstate back into infra/terraform/ from the password manager, then run make plan. The plan should show no changes. Anything it wants to create already exists, so terraform import it rather than applying.
• Phase 5 The state lives in the versioned playpalz-tfstate bucket. Restore an earlier object version with aws s3api copy-object … ?versionId=, as for media.

Full recovery

This is the order for rebuilding everything, for example after losing the cluster.

1. Terraform state (above). Then make plan and make apply to recreate the cluster, the registry credentials and the managed resources that are missing.
2. Sealed-secrets key, then kubectl apply -f infra/k8s/sealed-secrets/ (above).
3. Postgres. The managed cluster survives a Kubernetes rebuild. If it's gone too, use A or a full restore from a dump.
4. Redis. Start empty, or restore from a snapshot or RDB.
5. Services. Install the releases on the new cluster at the last good commit's tags:

   cd infra/helm
   helm upgrade --install playpalz-platform platform -n default -f values/production/platform.yaml --wait
   for svc in api web admin docs esu ogun anansi igdb-heartbeat; do
     helm upgrade --install playpalz-$svc service -n default -f values/production/$svc.yaml \
       --set image.tag=<commit> --history-max 20 --wait --timeout 10m
   done

   deploy.sh only upgrades existing releases, so a fresh cluster uses helm upgrade --install directly, without --atomic. The api release's hook migrates the restored database first.
6. DNS and TLS. If the load balancer IP changed, update the A records (Terraform dns.tf). cert-manager re-issues the certificates.
7. Verify:
   • Every health check passes.
   • Sign in, the feed loads, a post with media works.
   • A test purchase works in sandbox.
   • The admin Jobs page shows the jobs running.