Runbooks

Production incident procedures. For local problems see Troubleshooting.

First response

kubectl get pods                                     # anything not Running?
kubectl get events --sort-by=.lastTimestamp | tail -30
kubectl top nodes && kubectl top pods

curl -sS -o /dev/null -w "%{http_code}\n" https://api.playpalz.gg/health
curl -sS -o /dev/null -w "%{http_code}\n" https://socket.playpalz.gg/health
curl -sS -o /dev/null -w "%{http_code}\n" https://www.playpalz.gg

If a deploy in the last hour lines up with the problem, roll it back first: Rollback (kubectl rollout history deployment/playpalz-<service>).

Then Grafana at grafana.playpalz.gg:

{app=~"playpalz-.*"} | json | level >= 50

The API is down

Symptom: api.playpalz.gg returns 5xx or does not respond. Everything in the app breaks.

kubectl get pods -l app=playpalz-api
kubectl describe pod <pod>
kubectl logs -l app=playpalz-api --tail=200

What you see	Cause	Action
CrashLoopBackOff	Bad config or a startup exception	Read the logs; check for a missing env var after a recent deploy
ImagePullBackOff	Registry auth or a bad tag	Check the registry-playpalzproduction pull secret
OOMKilled in describe	Memory limit hit	Raise limits, or find the leak
Pending	No schedulable node	kubectl top nodes; the pool autoscales 2–10
Pods Running, requests failing	Database or Redis unreachable	See below

Immediate mitigation: if a recent deploy caused it, roll back with helm rollback playpalz-api -n default --wait. See Rollback.

The database is unreachable

Symptom: every service logs Prisma connection errors.

doctl databases list
doctl databases get <database-id>
kubectl exec -it <api-pod> -- node -e "require('net').connect(25060,'<db-host>').on('connect',()=>{console.log('ok');process.exit(0)}).on('error',e=>{console.log(e.message);process.exit(1)})"

Check in order:

1. Database status in the DigitalOcean console — maintenance, failover, or an incident.
2. Firewall — digitalocean_database_firewall restricts access to the cluster. A new node pool or a recreated cluster can fall outside it.
3. Connection exhaustion — db-s-2vcpu-4gb has a bounded connection limit, and three API replicas plus four other services each hold a pool. This is the most likely cause under load.

SELECT count(*), state FROM pg_stat_activity GROUP BY state;
SELECT pid, state, query_start, left(query, 80) FROM pg_stat_activity
  WHERE state = 'active' ORDER BY query_start;

Scaling replicas down temporarily relieves connection pressure. The durable fix is a connection pooler (PgBouncer, or DigitalOcean's managed pooler).

Redis is down

Symptom: realtime stops delivering, media stops processing, feed latency rises.

kubectl get pods -l app=playpalz-redis
kubectl logs -l app=playpalz-redis --tail=100
kubectl exec -it <redis-pod> -- redis-cli -a <password> PING

Single replica with a 5 Gi PVC, so a restart loses in-flight state but not persisted data.

Impact	Recovery
BullMQ queue	Jobs persisted in Redis survive; in-flight jobs retry
Socket.IO adapter	Cross-pod fan-out stops entirely until it returns
Feed / trending cache	Cache misses fall through to Postgres — slower, still correct
anansi payout lock	A run cannot acquire the lock; it fails safe

After recovery, restart esu so its adapter reconnects cleanly:

kubectl rollout restart deployment/playpalz-esu

Realtime is broken

Symptom: messages arrive only on refresh; presence and typing do nothing.

kubectl get pods -l app=playpalz-esu
kubectl logs -l app=playpalz-esu --tail=200
curl https://socket.playpalz.gg/health

In Grafana:

{app="playpalz-esu"} | json | event = "socket:auth_failed"
{app="playpalz-esu"} |= "redis"

Symptom	Cause
Connect then immediate disconnect	JWT_SECRET mismatch between api-secrets and what esu reads
Every join AUTHZ_DENIED	API_BASE_URL wrong, or the API is failing the authz endpoints
Connects fine, no events	Redis adapter down, or the API cannot reach /admin/emit/*
Sporadic drops	Ingress timeout, or the sticky-session cookie not being honoured

Check the API side too — it is the producer:

{app="playpalz-api"} |= "REALTIME_SERVICE_URL"

Media is not processing

Symptom: uploads succeed but never get thumbnails; posts stay pending.

kubectl get pods -l app=playpalz-ogun
kubectl logs -l app=playpalz-ogun --tail=200

kubectl exec -it <redis-pod> -- redis-cli -a <pw> LLEN 'bull:media-processing:wait'
kubectl exec -it <redis-pod> -- redis-cli -a <pw> LRANGE 'bull:media-processing:failed' 0 -1

Queue state	Meaning
wait growing, worker logs silent	Worker is not consuming — restart ogun
failed growing	Processing is erroring — read the error and Media.errorMessage
Queue empty, media still pending	The API is not enqueuing — check API logs

Video specifically: Media.status only becomes ready when Mux calls the webhook. Confirm the webhook is registered and reaching api.playpalz.gg/api/v1/webhooks/mux.

Remember ogun has no health probe, so Kubernetes will not restart a wedged worker. Restart it yourself:

kubectl rollout restart deployment/playpalz-ogun

A payout run failed

Highest-severity class of incident — real money.

Open the admin panel. Jobs shows each anansi run (scheduled or manual) with its result or error; Payouts shows the month's payouts, their status, and the Stripe balance. Every action there is audited. Fall back to anansi's admin API (anansi) only if the panel or the API is down.

kubectl logs -l app=playpalz-anansi --since=48h

The 25th run failed

1. Read the run's error on Jobs (Payouts — calculate and top up) and the logs: a Stripe error, a database error, or a lock conflict.
2. On Payouts, pick the month and Preview. It writes nothing.
3. Calculate + top up for that month. It's safe to repeat: anything already in a payout isn't added again.

The 1st run failed

1. On Payouts, pick last month. If the Stripe balance doesn't cover the pending total, the run stopped without paying anyone: wait for the top-up to clear, then Send transfers.
2. Retry failed payouts (one row, or all). anansi checks Stripe for an earlier transfer first, so a retry never pays twice.
3. Persistent failures are usually creator-side: incomplete Stripe Connect onboarding, or a restricted account. Check the Stripe dashboard for that connected account.

A PlayPal's cash-out looks stuck

Find their CoinCashout rows. requested: waiting for the next 25th. scheduled: attached to a Payout; check that payout's status on Payouts (a failed transfer is retried there, and the cash-out is paid when it succeeds). paid but no money: check the Stripe transfer on the payout. returned: the transfer was reversed, and the coins are back in their available earnings. A cash-out still scheduled on a completed payout means settling failed after the transfer; the anansi logs say so, and reconciliation flags it.

A creator was paid twice

Stop and investigate before doing anything else.

1. Find the duplicate Ledger entries and Payout rows.
2. Check the idempotencyKey values — if they differ, the guard was bypassed and that is the bug.
3. Reverse in Stripe, and record an ADJUSTMENT ledger entry. Never edit or delete an existing ledger row.
4. Verify anansi is still at replicas: 1.

Coin balances look wrong

Balances come only from the coin ledger. Never edit Wallet.coinBalance to "fix" one; nothing reads it any more.

1. Pull the user's coin history: GET /wallet/transactions as the user, or the CoinEntry rows for their CoinAccounts.
2. Check the ledger-wide invariants: every transaction sums to zero, and each user account's balance equals the sum of its entries. The expectLedgerInvariants query in apps/api/__tests__/db/helpers.ts checks both.
3. Check the latest run on Coins → Reconciliation in the admin panel, or run it now. A failing check names the accounts, users or cash-outs involved.
4. Corrections are new transactions, never edits: Adjust balance, Write off debt or Reverse on a tip, all on the user's admin page. The database refuses an UPDATE or DELETE on the ledger.

Reconciliation reported drift

1. Open the run on Coins → Reconciliation. Each failing check lists what it found.
2. stripe_transfers_match: compare the payout with its transfer in the Stripe dashboard. A reversal we missed should arrive as transfer.reversed; check the transfer webhook is delivering.
3. no_unsettled_cashouts_on_completed_payouts: the transfer went out but settling the cash-outs failed. Re-running the transfer job settles them (the Stripe idempotency key returns the same transfer; nobody is paid twice).
4. Any ledger check (ledger_balanced, balances_match_entries, buckets_in_range): the database triggers should make these impossible. Freeze affected users' coins and escalate; don't correct until the cause is known.
5. Run the checks again from the admin panel after fixing.

Suspected tip fraud

Fraud flags hold a PlayPal's newly earned coins longer, so there's time to look.

1. Coins → Fraud flags lists open flags with sender, recipient and what triggered them.
2. Open both users. Look for one sender funding one PlayPal, new accounts, refunded purchases.
3. To act: Freeze coins on either user, Reverse specific tips, or Revoke PlayPal with forfeit. Then resolve the flag as actioned with a note of what you did.
4. Nothing wrong: resolve it as cleared.

A user paid but got no coins

1. Find the purchase: CoinPurchase by userId, or by storeTransactionId (the Apple transaction id or Play order id from their receipt). If it is there, it was credited; check their /wallet/transactions.
2. If not, find the event: RevenueCatEvent rows of type NON_RENEWING_PURCHASE whose payload has that transaction_id, and the API logs for [coins] lines about it. Common causes:
   • sandbox purchase refused: a TestFlight or test-track purchase against production. Expected.
   • product with no coin package: add the CoinPackage row, then have the user open the store screen (it re-syncs recent purchases) or redeliver the event from RevenueCat's dashboard.
   • unknown RevenueCat user: RevenueCat retried and gave up. Fix the user mapping, then redeliver.
3. Never credit by hand with SQL. Redeliver the event. If the store can't redeliver, credit PURCHASED with Adjust balance on the user's admin page, quoting the store transaction id in the reason.

A user asks about a coin debt

Debt comes only from refunds. Find the CoinRefund rows for their userId: each shows the purchase, how much came back from their own coins and from tips' recipients, and what was left owing (toDebt). If the store later reversed the refund, the row says reversed and the debt was cancelled automatically. Don't edit balances. If the platform decides to forgive it, use Write off debt on the user's admin page.

Refunds received before clawbacks (one time)

The release before clawbacks stored refund events without acting on them. After the clawback release is live:

cd apps/api
pnpm coins:reprocess-refunds            # dry run: lists them
pnpm coins:reprocess-refunds --apply

Each refund applies once, so a second run changes nothing. It exits non-zero if any event failed.

Coin ledger cutover (one time)

The release that moved coins onto the ledger stops writing Wallet.coinBalance. Existing balances are carried over by a backfill, run after that release has fully rolled out:

cd packages/database
pnpm build
pnpm coins:backfill            # dry run: reports what it would carry over
pnpm coins:backfill --apply

It posts only what is missing, so run it again if an old pod may have written a balance during the rollout. It exits non-zero and lists any wallet whose balance went down after it was carried over; review those by hand.

Creator onboarding is failing

Every creator onboarding endpoint returns 503 when its dependencies aren't configured. It never falls back.

kubectl logs -l app=playpalz-api --since=1h | grep -iE "creator|kyc|stripe"
kubectl exec deploy/playpalz-api -- printenv | grep -E "^(KYC_SPACES_BUCKET|STRIPE_WEBHOOK_SECRET)=" | cut -d= -f1

Symptom	Cause	Fix
ID upload 503	KYC_SPACES_* or KYC_ENCRYPTION_KEY unset	Check the kyc-storage secret (Terraform) and the KYC_ENCRYPTION_KEY key in api-secrets
Payout or bank step 503	STRIPE_SECRET_KEY unset	Check stripe-secrets
Payout step 400 for everyone	The restricted key lacks Connect permissions, or EXPO_PUBLIC_STRIPE_PUBLISHABLE_KEY is from a different Stripe account than the secret key	Logs show the Stripe code and requestId; look the request up in the Stripe dashboard
Creator stuck at payoutsEnabled = false	Stripe still needs something, or the webhook isn't arriving	Check CreatorProfile.stripeRequirementsDue; check the event destination's delivery log in Stripe
Webhook answers 400	Signing secret mismatch	Copy the destination's secret into stripe-secrets, reseal, restart
Webhook answers 503	STRIPE_WEBHOOK_SECRET unset	Seal it into stripe-secrets

Rotating the KYC encryption key

Follow Key rotation. Never remove a previous key (KYC_ENCRYPTION_KEY_V<n>) while rows with that keyVersion exist:

SELECT "keyVersion", count(*) FROM "VerificationAsset" GROUP BY 1;

If KYC_ENCRYPTION_KEY is lost

Every stored ID image becomes permanently unreadable, and there is no recovery. Mark affected SUBMITTED creators as REJECTED with a reason asking them to re-capture their ID.

TLS certificate expired or failing

kubectl get certificate
kubectl describe certificate playpalz-tls
kubectl logs -n cert-manager -l app=cert-manager --tail=100

Usual causes: DNS changed and the HTTP-01 challenge can no longer reach the ingress; port 80 blocked; or Let's Encrypt rate limiting after repeated failures. See DNS & TLS.

Everything is unreachable

kubectl cluster-info
kubectl get nodes
doctl kubernetes cluster list
kubectl get service -n ingress-nginx
dig api.playpalz.gg +short

Check, in order: the DigitalOcean status page, node health, the LoadBalancer's external IP, and whether DNS still resolves to it.

Restoring the database

Managed Postgres has daily backups and point-in-time recovery for the last 7 days. A restore creates a new cluster, which you then switch database-credentials over to. Everything written after the restore point is lost, so first check whether a forward fix or a partial restore will do.

The full procedure, including the firewall, the switch-over, the Terraform reconciliation and the money reconciliation, is in Restoring Data → Postgres.

App Review demo account

App Review signs in with the account set in EAS_METADATA_REVIEW_DEMO_USERNAME / EAS_METADATA_REVIEW_DEMO_PASSWORD (apps/mobile/store.config.js, pushed with eas metadata:push). Before every submission, in production:

1. Log in with it on a production build: it must exist, be active, and be fully onboarded.
2. It must be an adult (birthdate 18+) so messaging isn't restricted, and follow a few creators with recent posts, so the feed, Explore and DMs aren't empty.
3. A sandbox purchase (coins, a VIP pass) must succeed with an App Store sandbox tester.
4. Don't change its password or "log out all devices" after the submission is sent; both revoke the reviewer's session.
5. Keep a second, already-verified PlayPal account ready in case review asks to see creator flows (going live, Creator Studio, payouts).

Escalation

Situation	Do
Money moved incorrectly	Stop anansi, preserve the ledger, involve whoever owns finance
Suspected data breach	Rotate JWT_SECRET (logs everyone out), rotate all sealed secrets, preserve logs
Total outage > 15 min	Check the DigitalOcean status page; open a support ticket
Data loss	Stop writes, assess the blast radius, restore from a snapshot

After an incident

1. Write down the timeline and the root cause.
2. Add the failure mode to this page if it is not here.
3. Add the monitoring that would have caught it earlier — see Observability.
4. Fix the underlying cause, not just the symptom.