Kubernetes

Workloads are Helm releases, one per service plus one for shared infrastructure, in two namespaces: default (production) and staging (same releases, values in infra/helm/values/staging/, plus an in-cluster Postgres in its platform release). Charts and values are in infra/helm/; sealed secrets stay in infra/k8s/sealed-secrets/, applied with kubectl.

Releases

Release	Chart	Values	Owns
playpalz-api	service	values/production/api.yaml	Deployment, Service, PDB, migrate hook Job
playpalz-web	service	values/production/web.yaml	Deployment, Service, PDB
playpalz-admin	service	values/production/admin.yaml	Deployment, Service, PDB
playpalz-docs	service	values/production/docs.yaml	Deployment, Service, PDB
playpalz-esu	service	values/production/esu.yaml	Deployment, Service, PDB
playpalz-ogun	service	values/production/ogun.yaml	Deployment, HPA, PDB
playpalz-anansi	service	values/production/anansi.yaml	Deployment, Service
playpalz-igdb-heartbeat	service	values/production/igdb-heartbeat.yaml	Deployment, Service
playpalz-platform	platform	values/production/platform.yaml	Redis (StatefulSet, PVC, Service), the four public Ingresses

Outside Helm:

• infra/k8s/sealed-secrets/ holds the encrypted secrets.
• infra/k8s/monitoring/grafana-ingress.yaml is in the monitoring namespace.
• The cluster add-ons (cert-manager, ingress-nginx, sealed-secrets, loki-stack, metrics-server) are separate releases of their own.

helm list -n default                       # every release, revision and status
helm list -n staging                       # the same for staging
helm history playpalz-api -n default       # revisions: what was deployed when
helm get values playpalz-api -n default    # the values (and image) a release runs with

Why one release per service: each service rolls back on its own (helm rollback playpalz-esu), and a release's history is that service's deploy history.

Why a platform release: playpalz-ingress routes both web and api, so it can't belong to either service. The Redis PVC is annotated helm.sh/resource-policy: keep: uninstalling the release never deletes Redis's data.

Deployment shape

Service	Replicas	Strategy	Probes	Ingress
web	3	RollingUpdate, maxUnavailable: 0	GET /	✔
api	3	RollingUpdate, maxUnavailable: 0	GET /health	✔
esu	2	RollingUpdate	GET /health	✔
ogun	2 (HPA 2–8)	RollingUpdate	heartbeat file (/tmp/ogun-alive, once its new image ships)	—
anansi	1	Recreate	GET /health	—
igdb-heartbeat	1	Recreate	GET /health	—
redis	1 (StatefulSet)	—	✔	—

maxUnavailable: 0 with maxSurge: 1 means a new pod becomes ready before an old one is removed — zero-downtime rollouts at the cost of briefly running one extra pod.

The two singletons use Recreate on purpose. Both are cron-driven, and overlapping pods during a rollout could mean a duplicate payout run or a duplicate catalog sync.

Autoscaling

Only ogun has an HPA:

minReplicas: 2
maxReplicas: 8
metrics:
  - type: Resource
    resource: { name: cpu,    target: { type: Utilization, averageUtilization: 70 } }
  - type: Resource
    resource: { name: memory, target: { … } }

That is the right service to autoscale — image processing is bursty and CPU-bound. The node pool itself autoscales 2–10 nodes, so a burst can add capacity underneath.

The HPA reads CPU and memory from metrics-server (release metrics-server in kube-system, values in infra/helm/metrics-server-values.yaml), which also powers kubectl top. It was installed on 2026-10-30. Before that the HPA reported ScalingActive False and ogun never scaled. If kubectl top says Metrics API not available, check that release first:

kubectl -n kube-system get deploy metrics-server
kubectl describe hpa playpalz-ogun-hpa | grep -A4 Conditions    # ScalingActive should be True

api is fixed at 3 replicas. An HPA on it would be a sensible addition, bounded by the Postgres connection limit.

Redis in-cluster

A StatefulSet with a 5 Gi PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: playpalz-redis-pvc }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 5Gi } }

A StatefulSet gives stable network identity, and the PVC persists AOF/RDB snapshots across restarts.

Single replica, no failover

Redis backs the media queue, the Socket.IO adapter, the feed and trending caches, and anansi's payout lock. If the pod is rescheduled, realtime fan-out stops and queue operations fail until it comes back. The PVC means data survives, but availability does not.

For a queue and a cache this is a defensible cost trade. Revisit before Redis holds anything that cannot be rebuilt.

The manifest's header comment mentions apps/hermes — a service that no longer exists in this repository. Harmless, but stale.

Security context

Every service container:

securityContext:
  allowPrivilegeEscalation: false
  capabilities: { drop: [ALL] }

# pod level
securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  fsGroup: 1001

This is consistently applied and worth preserving in anything new.

Placement

api and web use soft pod anti-affinity:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector: { matchExpressions: [{ key: app, operator: In, values: [playpalz-api] }] }
          topologyKey: kubernetes.io/hostname

preferred rather than required, so replicas spread across nodes when possible but a busy cluster never leaves pods unschedulable.

Images

image: registry.digitalocean.com/playpalzproduction/api@sha256:…   # pinned by digest
imagePullSecrets:
  - name: registry-playpalzproduction

infra/helm/deploy.sh <service> <tag> resolves the tag to its digest and deploys that, so every Helm revision names exactly the image it ran, and helm rollback restores those bytes even if the tag is later moved. The tag is recorded on the Deployment as the playpalz.gg/image-tag annotation and in helm history.

Ingress

Four Ingress objects in the playpalz-platform release (infra/helm/values/production/platform.yaml), plus Grafana's in infra/k8s/monitoring/:

Main — playpalz.gg, www.playpalz.gg, api.playpalz.gg:

cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.2 TLSv1.3"
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"

Socket — socket.playpalz.gg, deliberately separate:

nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/affinity-mode: "persistent"
nginx.ingress.kubernetes.io/session-cookie-name: "ESUSESSION"

The file's own comments explain both choices: HTTP/1.1 and hour-long timeouts must not apply to REST routes, and Engine.IO long-polling issues several short-lived requests per connection that must land on the same pod, since session state is in that pod's memory.

Admin — admin.playpalz.gg → playpalz-admin:3000, separate so it can be locked down without touching customer routes. 20 req/s per IP, 1 MB bodies, and a commented-out whitelist-source-range for restricting it to office/VPN IPs. See admin.

Grafana — grafana.playpalz.gg.

A staging Ingress for staging.playpalz.gg / api-staging.playpalz.gg is present but commented out.

Operating

kubectl get pods
kubectl get deployments
kubectl get ingress
kubectl get hpa

kubectl logs -l app=playpalz-api --tail=100 -f
kubectl describe pod <pod>
kubectl exec -it <pod> -- sh

kubectl rollout restart deployment/playpalz-api      # same image, new pods (e.g. after a secret change)
kubectl rollout status  deployment/playpalz-api

helm history  playpalz-api -n default                # deploys, with image tag and status
helm rollback playpalz-api -n default --wait         # back one revision; see Rollback

kubectl top nodes
kubectl top pods

Applying changes

Change the values file (or the chart) in a PR, then deploy the release:

infra/helm/deploy.sh api <tag>      # a service: values + image
infra/helm/deploy.sh platform       # Redis and the ingresses

The values files are the source of truth. A kubectl edit, kubectl scale or kubectl set image on a Helm-managed object is reverted by that release's next deploy; put the change in the values file. scripts/ci/check-helm.sh (run by CI) lints, renders and schema-validates every release.

Adopting production (one time)

Production was created with kubectl apply from infra/k8s/*-deployment.yaml. Helm took it over with infra/helm/adopt-production.sh:

• It labels the existing objects as belonging to each release, then installs the releases.
• The charts render the pod templates exactly as they ran, so no pod restarted.
• Run it without --apply to see the diff first.
• Once adopted, the old manifests were removed.

The default namespace

Everything is in default, which is workable at this size but has costs: no namespace-level resource quotas, no network policies between tiers, and no clean separation when staging arrives. Staging (CI/CD Phase 3) gets its own namespace, with the same charts and its own values files. See Environments.