Ansible & the LiveKit Node

Everything in the platform runs on Kubernetes except one thing: the LiveKit media server. It lives on a dedicated DigitalOcean droplet, and Ansible is what turns that bare Ubuntu box into a working server. Terraform creates the droplet and its firewall; it installs nothing.

That division matters when something breaks. If the droplet exists but nothing answers on ws.playpalz.gg, Terraform did its job and Ansible has not run.

Why LiveKit is not in the cluster

WebRTC needs a wide UDP port range (50000–60000) and direct host networking to publish ICE candidates that clients can actually reach. Expressing that through an ingress controller and a shared load balancer is awkward and fragile, so LiveKit gets its own host. See Livestreaming & Voice for how the API issues tokens against it.

Layout

Path	Contains
ansible.cfg	Inventory path, root remote user, SSH key, host-key checking disabled
inventory/hosts.yml	The single livekit host and its public IP
inventory/group_vars/livekit/vars.yml	Ports, versions, paths, room defaults
inventory/group_vars/livekit/vault.yml	Encrypted — livekit_api_key, livekit_api_secret
playbooks/site.yml	Full provision: common → caddy → livekit
playbooks/livekit.yml	Targeted config redeploy or binary upgrade
roles/common/	Packages, UTC, hostname, SSH hardening, UFW, fail2ban, logrotate
roles/caddy/	Caddy install, Caddyfile, TLS termination
roles/livekit/	System user, binary, config, systemd unit

Provisioning the node

cd infra/ansible
ansible-playbook playbooks/site.yml --ask-vault-pass

The playbook is idempotent — re-running it is the normal way to reconcile drift.

The inventory IP is not automatic

inventory/hosts.yml and vars.yml both hardcode the droplet's public IP. Terraform does not write them. If the droplet is ever destroyed and recreated it comes back with a new address, and the playbook will silently target the old, dead one.

After any rebuild, take livekit_ip from terraform output and update both files.

Verifying it worked

terraform -chdir=../terraform output -raw livekit_ip   # expected address
dig +short ws.playpalz.gg                              # must match
curl -sI https://ws.playpalz.gg                        # Caddy + TLS answering

On the host:

systemctl status livekit caddy
journalctl -u livekit -f
tail -f /var/log/livekit/livekit.log

A useful quick check from your laptop: port 22 open but 7880 closed means the droplet is up and Ansible has not run.

What each role does

common — apt upgrade and essential packages, UTC timezone, hostname set to livekit_domain, PasswordAuthentication no in sshd, then UFW allowing 22, 80, 443, 7880, 7881, 3478/udp and the 50000–60000/udp ICE range with a default-deny incoming policy. Also enables fail2ban and installs a logrotate rule keeping 14 days of LiveKit logs.

UFW duplicates the DigitalOcean firewall in livekit.tf on purpose — the cloud firewall is the outer gate, UFW the host-level one. Open a new port in both or traffic will not reach LiveKit.

caddy — installs Caddy from its apt repository and renders /etc/caddy/Caddyfile. Caddy owns 443, terminates TLS with an automatic Let's Encrypt certificate, and reverse-proxies to localhost:7880. This is why clients connect to wss://ws.playpalz.gg on the standard HTTPS port rather than to 7880 directly.

livekit — creates the livekit system user (nologin), downloads the pinned release binary from GitHub, symlinks it to /usr/local/bin/livekit-server, renders /etc/livekit/livekit.yaml at mode 0640, and installs a hardened systemd unit (NoNewPrivileges, ProtectSystem=strict, PrivateTmp, LimitNOFILE=65536).

TURN is deliberately disabled

turn.enabled: false in the config template. Caddy owns 443 and would conflict with LiveKit's TURN/TLS port. UDP ICE covers the large majority of clients; symmetric-NAT users would need TURN re-enabled on a dedicated port.

API credentials

livekit_api_key and livekit_api_secret live in the Ansible vault and are rendered into the keys: block of /etc/livekit/livekit.yaml. They are the same pair the API uses to mint access tokens, so the two must stay in sync — rotating one without the other rejects every token.

ansible-vault view inventory/group_vars/livekit/vault.yml
ansible-vault edit inventory/group_vars/livekit/vault.yml   # then re-run the playbook

The API consumes four variables, all pointing at the same host:

Variable	Value	Used for
LIVEKIT_HOST	https://ws.playpalz.gg	Server-side REST (RoomServiceClient)
LIVEKIT_WS_URL	wss://ws.playpalz.gg	Handed to clients to join rooms
LIVEKIT_API_KEY	from vault	Signing access tokens
LIVEKIT_API_SECRET	from vault	Signing access tokens

LIVEKIT_HOST is https and LIVEKIT_WS_URL is wss — they are not interchangeable. See Secrets Management for sealing these into the cluster.

Common changes

Config only (ports, room defaults, log level) — edit vars.yml, then:

ansible-playbook playbooks/livekit.yml --tags config --ask-vault-pass

Upgrade LiveKit — bump livekit_version in vars.yml, then:

ansible-playbook playbooks/livekit.yml --tags upgrade --ask-vault-pass

Both notify the Restart livekit handler, which drops active calls. Treat an upgrade as a brief outage for anyone in a voice room.

Rotate credentials — ansible-vault edit, re-run playbooks/livekit.yml --tags config, then re-seal livekit-secrets and restart the API so both sides move together.

Troubleshooting

Symptom	Likely cause
Playbook cannot reach the host	Inventory IP still points at a destroyed droplet
UNREACHABLE, permission denied	Missing ~/.ssh/playpalz_livekit, or key not on the droplet
Decryption failed	Wrong vault password
Port 22 open, 7880 closed	Droplet provisioned, Ansible never ran
TLS errors on ws.playpalz.gg	DNS not pointing at the droplet yet, so Caddy cannot complete ACME
Tokens rejected with a valid URL	Vault key pair differs from what the API has
Connects then no audio	UDP 50000–60000 blocked in the DO firewall or UFW

ansible.cfg sets host_key_checking = False, so a rebuilt droplet reusing an address will not block the playbook — but a stale ~/.ssh/known_hosts entry can still trip up a manual ssh.