Observability

Centralised logging with Grafana Loki. There is no metrics stack and no alerting.

Capability	Status
Structured logs	Loki + Promtail + Grafana
Mobile crash reporting	PostHog JS errors via reportFatal; native crashes and source maps not yet
Socket event tracing	esu, opt-out
Socket.IO live dashboard	Available, disabled by default
Metrics (Prometheus)	None
Alerting	None
Backend error tracking	None
Distributed tracing	None

Logging stack

Deployed via the grafana/loki-stack Helm chart into the monitoring namespace:

Rendering diagram…

Configuration in infra/helm/loki-stack-values.yaml:

loki:
  persistence: { enabled: true, storageClassName: do-block-storage, size: 20Gi }
  config: { auth_enabled: false }

promtail:
  config: { snippets: { pipelineStages: [{ cri: {} }] } }

grafana:
  admin: { existingSecret: grafana-secrets, userKey: admin-user, passwordKey: admin-password }
  ingress: { enabled: false }          # managed in infra/k8s/monitoring/grafana-ingress.yaml instead
  datasources: [{ name: Loki, type: loki, url: http://loki-stack:3100, isDefault: true }]

auth_enabled: false is acceptable only because Loki is not exposed outside the cluster — access is through Grafana, which is behind its own credentials.

Full install steps: dev-notes/loki-setup.md.

Structured logging with pino

Every backend service uses pino. The object comes first, the message second:

logger.info({ port: config.port }, "Server started");
logger.error({ err: error, mediaId }, "image processing failed");

That ordering is what makes the fields queryable in Loki. String interpolation (logger.info(`port ${port}`)) produces a flat message you cannot filter on.

ogun goes further and namespaces by module:

const log = logger.child({ module: "processImage" });

so {app="playpalz-ogun"} | json | module = "processImage" isolates one stage of the pipeline, and mediaId traces one item end to end.

In development, pino-pretty renders logs readably. In production they stay JSON for Loki.

Useful queries

{app="playpalz-api"}                                    # everything from the API
{app="playpalz-api"} | json | level >= 50               # errors and above
{app="playpalz-ogun"} | json | module = "processImage"
{app="playpalz-ogun"} | json | mediaId = "clx…"         # one media item's history
{app="playpalz-esu"} | json | event = "socket:in"       # inbound socket events
{app="playpalz-esu"} | json | userId = "clx…"           # one user's realtime session
{app="playpalz-anansi"} |= "cron"                       # payout runs

pino levels: 10 trace, 20 debug, 30 info, 40 warn, 50 error, 60 fatal.

Socket event logging

esu logs every inbound and outbound socket event when SOCKET_EVENT_LOGGING=true (the default):

socket.onAny((event, ...args) => {
  logger.info({ event: "socket:in", name: event, userId, socketId, args: summarizeSocketArgs(args) },
    `[esu-realtime] <- ${event}`);
});

summarizeSocketArgs truncates payloads so message bodies do not end up in the log store. Set the variable to false if volume becomes a problem.

Socket.IO Admin UI

A live view of connections, rooms, and events. Disabled by default:

pnpm --filter @playpals/esu admin-ui:creds     # generates a bcrypt hash

ADMIN_UI_ENABLED=true
ADMIN_UI_USERNAME=admin
ADMIN_UI_PASSWORD_HASH=<hash>

Then open admin.socket.io and connect to socket.playpalz.gg. It is instrumented in production mode, so payloads are not exposed, and gated behind basic auth.

Enabling it adds https://admin.socket.io to the CORS allowlist and requires the polling transport — which is why the socket ingress pins clients with the ESUSESSION cookie.

Health endpoints

Service	Endpoint
api	GET /health → {"status":"ok"}
esu	GET /health → {"ok":true}
anansi	GET /health
igdb-heartbeat	GET /health → {"status":"ok","service":"igdb-heartbeat"}
web	GET /
ogun	none

Kubernetes liveness and readiness probes use these. ogun has neither, so a wedged worker looks healthy to the cluster.

Mobile crash reporting

PostHog Error Tracking (posthog-react-native). apps/mobile/lib/posthog.ts creates the client outside development when EXPO_PUBLIC_POSTHOG_KEY is set; without the key everything below is a no-op.

apps/mobile/lib/error-reporting.ts exposes reportFatal(error, context). It logs in development and calls posthog.captureException in production, with the source, whether it was fatal, the app version, environment and platform. Every path that can catch a fatal calls it:

Layer	File	Catches
Root error boundary	apps/mobile/app/_layout.tsx (ErrorBoundary export)	Render errors anywhere, including inside the providers
Screen error boundary	apps/mobile/app/(app)/_layout.tsx (ErrorBoundary export)	Render errors in any (app) screen or modal, with providers still mounted
Global JS handler	apps/mobile/lib/global-error-handler.ts	Throws outside React — event handlers, timers, async callbacks

Unhandled promise rejections have no app-level hook, so PostHog's own autocapture covers those (unhandledRejections: true). Its uncaughtExceptions autocapture is off, because the global handler above already reports them and they would arrive twice. The signed-in user's id (never email or name) is attached via posthog.identify in AuthContext and cleared on logout.

Still open:

• Native crashes (bad native module, OOM, a fatal in a native view) kill the process before any JS runs. They need @posthog/react-native-plugin and nativeCrashes: true, plus native debug symbols.
• Source maps. Stack traces are from the Hermes bundle until source maps are uploaded on EAS builds (PostHog's React Native source-map guide; the CLI token belongs in EAS secrets).

Mobile product analytics

trackEvent in apps/mobile/lib/revenueCat/analytics.ts sends the paywall and purchase events (PAYWALL_VIEWED, PAYWALL_DISMISSED, SUBSCRIPTION_PURCHASED, SUBSCRIPTION_RENEWED, COIN_BUNDLE_PURCHASED, VIP_SUBSCRIPTION_PURCHASED, RESTORE_PURCHASES) to the same PostHog client with posthog.capture. Like crash reports, they are off in development and without a key, and are linked to the signed-in user's id. The App Store privacy label and the Play Data Safety form must list them: Usage Data (product interaction) and Purchases, linked to the user, not used for tracking.

Quick triage

kubectl get pods                                  # anything not Running?
kubectl top nodes && kubectl top pods             # resource pressure
kubectl logs -l app=playpalz-api --tail=100 -f
kubectl describe pod <pod>                        # events — OOMKilled, ImagePullBackOff
kubectl get events --sort-by=.lastTimestamp | tail -30

The gaps that matter

No metrics and no alerting

Nothing measures request latency, error rate, queue depth, or database connection usage, and nothing pages anyone when something breaks. Every incident is discovered by a user reporting it or by someone happening to look at Grafana.

The highest-value additions, roughly in order:

1. Alerting on what already exists. Grafana can alert on Loki queries — error rate above a threshold, anansi cron failing to log a run, a spike in socket auth failures. This needs no new infrastructure.
2. Error tracking, mobile and backend. Nothing groups exceptions today. A single tool across the mobile app and the Node services would give deduplicated, symbolicated stack traces.
3. Prometheus + metrics. prom-client in the services, kube-state-metrics for the cluster, dashboards for latency and queue depth.
4. Uptime monitoring. An external check against api.playpalz.gg/health catches the failure mode where the whole cluster is unreachable and therefore not logging anything.

Retention

Loki has 20 Gi of block storage and no explicit retention policy in the values file, so it will fill and start dropping the oldest data. Set limits_config.retention_period (30 days is a reasonable default) rather than discovering the limit during an incident.