Media Pipeline

Uploads never block a request. The API takes the original bytes and returns immediately; the ogun worker does the expensive work off the request path.

Two upload paths

There are two ways media reaches the platform, and both converge on the same queue.

Direct-to-storage (preferred)

The client uploads straight to DigitalOcean Spaces with a presigned URL. The API never touches the bytes.

Rendering diagram…

The storage key is namespaced per user and year:

uploads/<userId>/<images|videos>/<year>/<uuid>.<ext>

Presigned URLs expire after 5 minutes.

/media/upload/complete (apps/api/src/services/media.ts) only accepts a key under the caller's own uploads/<userId>/ prefix, for the caller's own post while it is still pending, and enforces the post shape: at most 5 photos, or exactly one video, never both. Registering the same key twice is a no-op (client retries). order is the file's position in the composer — uploads run in parallel, so arrival order is not it. width/height are the picker's and only lay out the author's optimistic post; ogun replaces them.

The composer re-encodes HEIC/HEIF to JPEG before upload (expo-image-manipulator): the prebuilt libvips in Sharp cannot decode HEVC-coded HEIC, and such uploads fail permanently.

Multipart through the API

POST /api/v1/posts accepts multipart/form-data and streams files to Spaces with multer-s3. It is simpler for the client, but the bytes pass through the API pod. Use the presigned path for anything large.

Both paths end the same way: a Media row with status: "pending" and a job on the media-processing queue.

The queue

Defined once in packages/queue/src/index.ts so producer and consumer cannot disagree:

export const PROCESS_MEDIA_QUEUE = "media-processing";

export const ProcessMediaQueue = new Queue(PROCESS_MEDIA_QUEUE, {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: "exponential", delay: 5000 },
    removeOnComplete: { age: 60 * 60, count: 1000 },
    removeOnFail: { age: 24 * 60 * 60 },
  },
});

Five attempts with exponential backoff starting at 5 s (≈5, 10, 20, 40 s between attempts). Failed jobs stay in Redis for 24 h for inspection. The Redis client sets maxRetriesPerRequest: null, which BullMQ requires.

ogun consumes with concurrency: 5 per worker process.

Image processing

apps/ogun/src/jobs/processImage.ts, with the Sharp work in apps/ogun/src/lib/imagePipeline.ts:

1. GetObject the original from Spaces.
2. Analyse it: display width/height after EXIF orientation (metadata().autoOrient), the orientation's rotation, a thumbhash placeholder, and a focal point from sharp.strategy.attention.
3. Render the variants (below) and PutObject each to variants/<mediaId>/v3/<name>.<ext>.
4. Upsert a Variant row per output on (mediaId, type, format), and delete rows for types this run did not produce (the cropped v1/v2 set).
5. Set Media.status = "ready" with width, height, aspectRatio, rotation, placeholder, focalX, focalY and metaVersion: 3.
6. Emit media:processing:update to the author.

Variants

Nothing is ever cropped server-side, and nothing is upscaled. Variants are width-based and keep the photo's aspect ratio; the client crops (toward the focal point) at draw time.

Variant	Box	Format
w320, w640, w1080, w1440	that width, height capped at 2048	WebP q80
full	2048 on the long edge	WebP q80
og	1200 on the long edge	JPEG (mozjpeg q80), alpha flattened on white

og is for link previews, email and push — the app never draws it. A width variant is skipped when it would come out the same size as one already planned (small originals get w320 + full only; a 1×1 gets full + og). See planVariants().

Every output is auto-oriented, has no EXIF (Sharp drops metadata unless withMetadata() is called), and is uploaded public-read with Cache-Control: public, max-age=31536000, immutable. The immutable header is safe because the key contains the media id and VARIANT_VERSION (v3, in apps/ogun/src/lib/variantStore.ts). Changing variant output means bumping it and backfilling — overwriting an existing key would leave the old bytes in CDN and client caches.

Variant.width/height store the actual output size.

Metadata every media row carries

Column	Meaning
width, height	Display size: EXIF orientation / video rotation applied
aspectRatio	width / height
placeholder	Base64 thumbhash, drawn while the media loads
focalX, focalY	0..1; the client biases cover crops toward it
rotation	Source rotation that was applied (0/90/180/270), diagnostic
posterUrl	Video: upright poster frame (largest poster variant)
altText	Optional description, used as the accessibility label
metaVersion	3 once the v3 pipeline has processed the row

The client never reads these raw: toMediaItem() in @playpals/types normalizes a row from any source (feed API, realtime event, optimistic upload) — legacy rows without dimensions, cropped v1 variants, seconds stored as ms — into the one shape the renderers use.

Video processing

apps/ogun/src/jobs/processVideo.ts describes the video, then hands it to Mux:

1. Probe the original with ffprobe (apps/ogun/src/lib/ffprobe.ts). Phones store most video landscape-coded with a display matrix — an iPhone portrait clip is 1920×1080 pixels rotated −90°. The probe reads that rotation (display matrix, or the legacy rotate tag), swaps width/height for 90/270, applies non-square pixels, and reads the duration. This is the fix for landscape videos rendering as portrait: dimensions used to come from the picker or Mux, either of which can be the stored (pre-rotation) size.
2. Poster: ffmpeg grabs one frame (it auto-rotates), rendered as upright poster-w320/640/1080 WebP variants plus a thumbhash.
3. Write width, height, aspectRatio, rotation, durationMs, placeholder, posterUrl, metaVersion: 3. Status stays pending.
4. Create the Mux asset. There is no ogun-side transcode — Mux applies the rotation on ingest, and the client sizes the box from the probed dimensions.

A probe or poster failure is logged and not fatal: the video still goes to Mux, and the webhook's track size becomes the fallback. The ogun image installs ffmpeg for this.

const asset = await mux.video.assets.create({
  inputs: [{ url: media.originalUrl }],
  playback_policies: ["public"],
  video_quality: "basic",
});

The job completes as soon as Mux accepts the asset. Completion arrives later as a webhook:

Rendering diagram…

The webhook (videoReadyUpdate() in apps/api/src/services/media.ts) converts Mux's duration from seconds to ms (rows written before this fix hold seconds; toMediaItem corrects them), and only writes Mux's track size for rows ogun could not probe (metaVersion < 3), never as 0.

Video status depends on a reachable webhook

If Mux cannot call POST /api/v1/webhooks/mux, video media stays pending forever even though transcoding succeeded. Locally, tunnel it (ngrok http 4000) and register the tunnel URL in the Mux dashboard.

Media states

Status	Meaning
pending	Row created, job queued or in flight
ready	Variants exist (image) or Mux reports the asset ready (video)
failed	Processing threw; errorMessage holds the reason

Retries

Both processors route errors through handleProcessingFailure() in apps/ogun/src/jobs/handleFailure.ts. The worker passes isFinalAttempt from the BullMQ job's attemptsMade / opts.attempts:

Error	Attempt	What happens
Transient (network, TLS, Mux 5xx…)	not last	Row left pending, error re-thrown, BullMQ reschedules
Transient	last	Row set failed, syncPostStatus, realtime emit, re-thrown
Permanent (NoSuchKey/404 original, undecodable image incl. HEVC HEIC)	any	Row set failed at once, UnrecoverableError skips remaining attempts

Leaving the row pending between attempts is deliberate: the author keeps seeing a spinner instead of a failure the next attempt may undo.

Retries are idempotent — variants upsert on the unique (mediaId, type, format), so a retry after a partial upload updates the rows it already wrote rather than inserting duplicates.

Jobs enqueued with mode: "backfill" reprocess rows that are already published: a failure there is logged and re-thrown for BullMQ, but never marks the row failed, recomputes the post status, or notifies anyone.

Re-driving failed media

Media that exhausted its retries (e.g. during an outage) is not picked up again automatically. Reset and re-enqueue it with:

# locally
cd apps/ogun && pnpm retry-failed --dry-run --since 2026-09-01 [--type image|video] [--limit 500]

# in the cluster
kubectl exec deploy/playpalz-ogun -- node dist/scripts/retry-failed-media.js --since 2026-09-01

The script sets each row back to pending, recomputes its post status, and enqueues a fresh job.

Backfilling to v3

Rows processed before v3 have no placeholder, focal point or poster, may have pre-rotation video dimensions, and may have cropped variants. Bring them up to date with:

cd apps/ogun && pnpm backfill-media-v3 --dry-run [--type image|video] [--since <date>] [--limit 5000] [--batch 50] [--pause-ms 5000]

# in the cluster
kubectl exec deploy/playpalz-ogun -- node dist/scripts/backfill-media-v3.js --dry-run

It enqueues metaVersion < 3, status: "ready" rows newest first, in paced batches so fresh uploads are not starved. Images are fully reprocessed; videos are probed (from the original, else from the Mux HLS rendition) and get a poster, but are never re-sent to Mux. Status never changes. It is done when --dry-run reports 0. The migration that added the v3 columns also renumbered Media.order for posts whose rows all sat at 0 (by upload time — the composer's order was never recorded).

Publish-when-ready

A post is withheld from everyone but its author until its media is ready, the way Instagram behaves. Post.status is the gate, recomputed from all of a post's media rows by syncPostStatus() in packages/database/src/queries/post.ts after every media status write:

Media rows	Resulting Post.status
none (text post)	ready
all ready	ready
any failed	failed
otherwise	pending

The feed applies onlyPublishedPosts(viewerId) from the same module, which admits status: "ready" plus the viewer's own posts regardless of status. So the author watches their own upload process and nobody else ever sees a processing placeholder.

This is what keeps the fan-out at one socket emit per completed upload. Because no other viewer is looking at a pending post, there is nothing to broadcast to — no per-post rooms, no follower cache invalidation. A newly-ready post reaches followers on their next feed fetch, within the 60s feed cache TTL or immediately on pull-to-refresh.

How the client learns it is done

Two mechanisms, belt and braces:

1. Realtime — media:processing:update on the user:<id> room, consumed by useRealtimeMediaProcessing. This is the author's path and the one that matters.
2. Polling — useMediaProcessingPoll re-fetches the post until media reports ready. Now a fallback only: it covers a dropped socket event, and a viewer who deep-links to a post that is still processing.

While processing, the author sees their own picked file rather than a grey box — apps/mobile/lib/local-media-preview.ts holds the local URIs, keyed by post id so they survive the feed invalidation that follows a successful post.

Storage layout

Prefix	Contents
uploads/<userId>/images/<year>/<uuid>.<ext>	Originals (image)
uploads/<userId>/videos/<year>/<uuid>.<ext>	Originals (video)
variants/<mediaId>/v3/<name>.webp, og.jpg	Generated variants, including poster-* for video
variants/<mediaId>/v2/…, variants/<mediaId>/<type>.jpg	Pre-v3 output, no longer referenced once backfilled

Originals are retained after processing. There is no lifecycle rule expiring them, so storage grows monotonically — worth revisiting as volume increases.

Rendering in the app

One component draws post media everywhere — PostMedia in apps/mobile/components/media/ — and every box is computed from metadata before anything loads (layout math in packages/types/src/media-layout.ts, unit-tested):

• 1 photo or video: its own aspect, clamped to 4:5…1.91:1 (MIN_ASPECT/MAX_ASPECT); outside that it is cover-cropped toward the focal point.
• 2–5 photos: a 16:9 grid with a 2px gutter, rounded outer corners only — 2: two 8:9 tiles; 3: an 8:9 tile beside two stacked 16:9; 4 and 5: 2×2 of 16:9, with "+1" on the fourth for 5.
• Tiles pick the smallest variant covering their size × pixel ratio and show the thumbhash meanwhile.

Tapping a photo opens the image viewer at that index; tapping a video opens the video viewer; the card body opens Post Detail. The viewers (apps/mobile/components/media-viewer/) are an overlay mounted in app/(app)/_layout.tsx, animating from the tapped tile's on-screen frame. The video viewer reuses the feed's player (lib/post-media/video-players.ts), so playback continues. Native video controls and native fullscreen are disabled on every VideoView. In lists, at most one video plays — muted, the topmost one at least 70% visible (lib/post-media/active-video.ts).

Operating the pipeline

# Is anything queued?
redis-cli -a <password> KEYS 'bull:media-processing:*'

# What failed?
redis-cli -a <password> LRANGE 'bull:media-processing:failed' 0 -1

# Worker logs
kubectl logs -l app=playpalz-ogun --tail=100 -f

Scaling throughput means adding ogun replicas — the work is CPU-bound in Sharp (and ffmpeg for posters). Watch Mux rate limits before scaling video volume aggressively.