Building & Deploying

How a merged commit becomes running containers. Before this: Developing & Committing. If a release goes wrong: Rollback.

	Status
Manual build from a laptop, deployed per service with Helm (infra/helm/deploy.sh)	Today
Migrations as a Helm hook on the api release	CI/CD Phase 2
Staging environment (--env staging)	CI/CD Phase 3
Images built by GitHub Actions and deployed to staging on merge	CI/CD Phase 4
Production promoted with a "Run workflow" button	CI/CD Phase 4
A database dump before any deploy with migrations	CI/CD Phase 5

Each service is a Helm release, playpalz-<service>, in the default namespace (see Kubernetes):

Service	Build script	Strategy
api	scripts/build-api.sh	Rolling, 3 replicas
web	scripts/build-web.sh	Rolling, 3 replicas
admin	scripts/build-admin.sh	Rolling, 2 replicas
docs	scripts/build-docs.sh	Rolling, 2 replicas
esu	scripts/build-esu.sh	Rolling, 2 replicas
ogun	scripts/build-ogun.sh	Rolling, HPA 2–8
anansi	scripts/build-anansi.sh	Recreate, singleton
igdb-heartbeat	scripts/build-igdb-heartbeat.sh	Recreate, singleton

Registry: registry.digitalocean.com/playpalzproduction/<service>.

Today: manual deploy Today

Nothing but this checklist stands between your working tree and production, so follow it exactly.

Steps

# 1. Be on the commit you mean to ship
git switch master && git pull
git status                                  # must be clean
SVC=api
TAG=$(git rev-parse --short HEAD)

# 2. CI is green for this commit (Actions tab), or run the checks locally

# 3. Build and push the commit-tagged image
doctl registry login
./scripts/build-$SVC.sh "$TAG"

# 4. Deploy it: resolves the tag to its digest, asks for confirmation, then
#    helm upgrade --atomic (api: the migrate hook runs first)
infra/helm/deploy.sh "$SVC" "$TAG"

# 5. Verify
curl -sS https://api.playpalz.gg/health
kubectl logs -l app=playpalz-$SVC --tail=50

Try it on staging first. The same steps with --env staging deploy to *.staging.playpalz.gg. The images are shared, except web: build ./scripts/build-web.sh "$TAG" staging as well. See Environments → Staging.

What deploy.sh does:

• Pins the image by digest. It resolves the tag in the registry and deploys repository@sha256:…, so helm rollback returns to exactly the image that ran before, even if the tag moves.
• Undoes a failed rollout by itself. --atomic --wait: if the new pods don't become ready within 10 minutes, Helm rolls the release back.
• Records the deploy as a revision in helm history playpalz-<service>, with the tag in its description, and keeps the last 20 revisions.
• Only upgrades existing releases. It refuses to install one, because a failed first install under --atomic would delete the objects.

Don't use kubectl set image, kubectl apply or kubectl edit on these Deployments. The release's next deploy reverts them, and they don't appear in helm history. Change the values file instead.

Every build script builds with --platform linux/amd64 from the repo root. Never run docker build by hand without it: an arm64 image from an Apple Silicon Mac fails on the cluster with exec format error.

Migrations

Migrations run in a Helm hook on the api release. On every deploy.sh api …:

1. Before any pod changes, a Job (playpalz-api-migrate) runs prisma migrate deploy with the new image.
2. If it fails, the upgrade stops and the running API is untouched.
3. With nothing new, it applies nothing.

kubectl logs job/playpalz-api-migrate      # the last run's output (kept until the next deploy)

• If the hook fails, fix the migration in a new commit. A half-applied migration is marked failed in _prisma_migrations; see Migrations.
• Take a backup first if the migration does anything beyond adding (it shouldn't — see the migration rules):
  • doctl databases backups <cluster-id> lists the daily backups.
  • Point-in-time recovery covers the last 7 days.
  • For a contract migration, take a dump.
• helm rollback doesn't undo a migration. That's why migrations must stay compatible with the previous release.

Deploy order

Migration involved: deploy api first (its hook migrates), then the other services that use the new shape. A contract migration ships only after every service has stopped using the old shape.

Shared package change (@playpals/db, @playpals/types, @playpals/socket-client): rebuild and deploy every service that imports it, from the same commit. The package is compiled into each image, so a partial deploy leaves services disagreeing about a contract.

Realtime contract change: deploy esu and api together, and roll them back together.

anansi: don't deploy while a payout run is in progress. Check Jobs in the admin panel (the payout-calculate and payout-transfer rows). Recreate stops the old pod before the new one starts.

Secrets and config

Changing a secret doesn't restart anything:

kubectl apply -f infra/k8s/sealed-secrets/<name>.yaml
kubectl rollout restart deployment/<every-consumer>

Add new secret keys before the code that reads them, and remove old keys only after the code that read them is gone. A secret isn't part of a deploy's revision, so a rollback doesn't bring back a key you removed. See Secrets Management.

Infrastructure changes

cd infra/terraform
make plan            # read it — watch for "destroy and then create replacement"
make apply

See Terraform.

Checklist

• [ ] On master, tree clean, on the commit you mean to ship; CI green
• [ ] Migrations are additive; a backup exists if not
• [ ] New env vars and secret keys are in place before the deploy (env vars go in infra/helm/values/production/<service>.yaml)
• [ ] No payout run in progress (if deploying anansi)
• [ ] Commit-tagged image built and pushed
• [ ] infra/helm/deploy.sh <service> <tag> succeeded (for api, including the migrate hook)
• [ ] Health endpoint OK; logs show no new errors for 5 minutes
• [ ] You know the previous revision (helm history playpalz-<service> -n default) in case you need to roll back

With CI/CD Planned

The plan (CI/CD Phases 1–5) replaces the laptop with GitHub Actions and the raw manifests with one Helm release per service, per namespace (staging, default for production).

PR ──► checks (types, lint, tests, migration safety, docker build, helm lint)
merge to master
  └─► build changed services, tag sha-<commit>; retag unchanged ones (same digest)
      └─► deploy to staging (api first: its migrate hook runs) ──► smoke tests
            └─► "Run workflow: deploy production" (manual, GitHub Free)
                  ├─ pre-deploy pg_dump, if this release has migrations
                  ├─ migrate Job
                  ├─ helm upgrade --atomic per service ──► smoke tests
                  └─ smoke test fails ──► automatic helm rollback of what changed

What changes for you:

• You don't build or deploy by hand. Merging to master puts the commit on staging. Production runs the same images (by digest) when someone runs deploy production in the Actions tab.
• Every release is one commit. All eight services are always at the same commit's images. Unchanged services get a new tag on the same digest, so Helm sees no change and doesn't restart them.
• Images are referenced by digest, so a rollback restores exactly the bytes that ran before.
• Migrations run once, in the api release's hook Job (as today), after a pg_dump to the backup bucket (Phase 5). A failed dump or migration stops the deploy before any pod changes.
• A failed rollout undoes itself. --atomic reverts a release whose pods don't become ready. If the smoke tests fail afterwards, the workflow rolls back every service it upgraded.
• Rollback is a workflow too. See Rollback.

The manual procedure above stays valid as the break-glass path until the pipeline is in place, and afterwards whenever GitHub Actions is unavailable. Both use the same releases, so either can deploy or roll back what the other did.

Mobile releases

A separate pipeline: EAS Build, EAS Submit and EAS Update. See Builds & Releases.

The coupling: installed apps run old code for weeks, so backend changes must stay compatible with the app versions in the stores. Never ship a breaking API change together with the app that needs it.