Rollback

A release broke production. This page gets you back to the last good state, by what broke.

Roll back first, debug second. If a deploy in the last hour lines up with the problem, revert it, confirm it's fixed, then investigate on staging or locally.

Pick the procedure

What broke	Roll back by	Data at risk
One service's code	that service's previous revision	none
Several services / a shared package / the api–esu contract	every affected service to the same good commit	none
Config or environment variables	the previous revision (env is part of it)	none
A secret's value	the previous SealedSecret from git	none
A deploy that included an additive migration	the code only; leave the schema	none
A destructive or data-corrupting migration	a forward fix; restore only as a last resort	everything since the restore point
Mobile JavaScript (EAS Update)	eas update:rollback	none
A mobile binary	halt the store rollout, ship a fix	none
Infrastructure (Terraform)	revert the commit, terraform apply	depends on the resource

Then, always: after any rollback.

Find the last good version

helm history playpalz-api -n default      # revision, date, status, description ("api 1a2b3c4")
helm get values playpalz-api -n default --revision 12 | grep -A3 '^image:'   # the image a revision ran

Time the problem started (Grafana, kubectl get events), then pick the revision deployed before it.

One service's code

Today With the production kubeconfig:

helm rollback playpalz-api -n default --wait --timeout 10m        # back one revision
helm rollback playpalz-api 12 -n default --wait --timeout 10m     # or to a specific one

The previous revision's image is pinned by digest, so this restores exactly what ran before. Values (env vars, replicas, resources) come back too. A rollback doesn't run the migrate hook.

Revision 1 of each release is the adoption, on :latest

The first revision of each release ("Adopted from infra/k8s") still names :latest with imagePullPolicy: Always. Rolling back to it pulls whatever latest points at now. Build scripts no longer push latest (they tag with the commit), so it stays on the image that was running at adoption. Don't push latest by hand.

With CI/CD Actions → Rollback → Run workflow, with:

Input
environment	production or staging
services	api, or several: api,esu
revision	blank for the previous one, or a number from helm history

It runs helm rollback for each service, then the smoke tests. The registry keeps every digest from the last 20 revisions, so the old image is always still there. The local helm rollback above remains the break-glass path when GitHub is unavailable.

Several services at once

Use this when the bad release changed a shared package (@playpals/db, @playpals/types, @playpals/socket-client), or the socket contract between api and esu, or when you can't tell which service is at fault. Rolling back only one of them would leave services disagreeing about a contract.

Today Roll each affected service back to the revision that ran the same good commit. The tag is in each revision's description:

for svc in api esu; do helm history playpalz-$svc -n default --max 5; done
helm rollback playpalz-api <rev> -n default --wait
helm rollback playpalz-esu <rev> -n default --wait

With CI/CD Actions → Deploy → Run workflow with sha set to the last good commit and environment: production. That moves every service to that commit's images. Services already on it don't restart. This is the whole-system rollback.

Config or environment variables

Env vars, replicas and resources are part of the Helm revision, so the code procedure above reverts them too. Fix the values file in git afterwards, so the next deploy doesn't bring the bad value back.

A secret

SealedSecrets aren't part of a deploy's revision, so rolling back a service does not restore a secret. Restore the previous version from git:

git log --oneline -- infra/k8s/sealed-secrets/stripe-secrets.yaml
git show <good-commit>:infra/k8s/sealed-secrets/stripe-secrets.yaml | kubectl apply -f -
kubectl rollout restart deployment/playpalz-api deployment/playpalz-anansi   # every consumer

The controller keeps all its old sealing keys, so an older SealedSecret still decrypts. Consumers of each secret are listed in Secrets Management.

A release with a migration

Migrations don't roll back. Prisma has no down-migrations here, and Helm rollback doesn't run them.

Additive migration (new table, nullable column, index): roll back the code only. The previous release ignores what it doesn't know about. Leave the schema; the next release will use it. This is the normal case, and why migrations must be additive.

Destructive or wrong migration (data dropped, mangled or wrongly backfilled):

1. Roll back the code if the old code can still run against the new schema. If it can't, keep the new code and put the app into a degraded state rather than serve errors.
2. Prefer a forward fix: a new migration that repairs the schema or data, deployed normally.
3. Only if data is lost and a forward fix can't recover it: restore. Use the pre-deploy dump to recover specific tables or rows, and point-in-time recovery to rewind everything. Both lose every write since the restore point, including purchases, tips and payouts. That needs reconciling afterwards.

Mobile

• A bad OTA update: eas update:rollback (interactive), or republish the previous update. Installed apps pick it up at their next check. See Builds & Releases → Rolling back.
• A bad binary: stop it spreading. App Store Connect: Pause Phased Release. Play Console: Halt rollout. Then ship a fixed build, or an OTA update if the fix is JS-only. Users who already installed it can't be downgraded.

Infrastructure

git revert <commit>        # the Terraform change
cd infra/terraform && make plan && make apply

Read the plan. A revert can itself destroy and recreate a resource. Anything holding data (database cluster, buckets, volumes) needs a backup checked first.

Service-specific cautions

Service	Before rolling back
anansi	Check Jobs in the admin panel. Don't roll back mid payout run; let it finish or fail. A failed run is retried from the Payouts page afterwards.
ogun	In-flight BullMQ jobs are retried by the replacement pods. A job shape changed by the bad release may fail on the old code; check the media queue afterwards.
api	Rolling back doesn't undo migrations (above). Mobile clients keep working as long as the API stays backward compatible.
esu	Clients reconnect automatically. Roll back with api if the socket contract changed.
web, admin, docs	Stateless; always safe.
igdb-heartbeat	Singleton; safe. Catches up on the next cycle.
redis	Not deployed per release. Data problems: see Restore → Redis.

After any rollback

1. Verify:
   • Health endpoints, the error rate in Grafana, kubectl get pods.
   • The symptom that triggered the rollback is gone.
2. Stop it coming back:
   • With CI/CD, the next merge to master redeploys everything at that commit, bad change included. git revert the bad commit on master (through a PR) before anything else merges.
3. Write it up: what broke, how it was detected, when it was rolled back, and why the checks didn't catch it. See Runbooks → After an incident.