Skip to content

Load testing the render pipeline

backend/tests/test_render.py assembles real video with real FFmpeg, no mocks — correct for checking the pipeline's behaviour. It only ever renders one clip set at a time, though, and production does not: several members of different circles can finish a turn within the same few minutes, and each one triggers a render-fallback that ties up a thread and a full FFmpeg encode. scripts/load_test_render.py (issue #98) exists to answer the question test_render.py cannot: what happens when N of those land on the API at once — response times, failures, and whether the process runs out of memory before it runs out of CPU.

This is not a CI check. Per the issue, it is a manual/nightly tool: run it by hand before a change to the render pipeline, or occasionally against a production-shaped instance to see whether the numbers have moved.

What it needs

  • The server-side fallback render, which is off by default: SERVER_SIDE_RENDER_ENABLED=true. Without it every render in the batch comes back 501 immediately (see architecture.md).
  • ffmpeg and ffprobe on the machine running the script — it synthesises its own test clips with ffmpeg, the same way test_render.py and scripts/demo_loop.py do.
  • httpx, already a backend dependency (backend/pyproject.toml); nothing else. No locust, no k6 — plain asyncio/httpx is enough to fire N requests concurrently and time them, and it stays in the same language and toolchain as the rest of the test suite.

Running it

Against docker-compose, which is the shape production actually runs in:

SERVER_SIDE_RENDER_ENABLED=true docker compose up -d --build
python scripts/load_test_render.py --renders 5 --clips 2

Useful flags:

  • --api URL — the API to drive (default http://localhost:8000).
  • --renders N — how many renders to fire at once (default 3).
  • --clips N — how many clips staged per render (default 2).
  • --clip-seconds N — length of each synthesised clip (default 6s).
  • --email / --password — reuse an existing account instead of registering a new one. Registration is capped at 5/hour per IP (limits.md); the script registers a throwaway account by default, which is fine for a sanity check but will run out of headroom before you reach an interesting N. Register one account by hand first for anything beyond a handful of renders.
  • --docker-container NAME — sample the API container's memory (docker stats) once a second while the batch is in flight, e.g. --docker-container vlogroulette-api-1 (check the actual name with docker compose ps).
  • --timeout SECONDS — how long each HTTP request may take before the script counts it as failed (default 180).
  • --json PATH — dump the raw per-render results (status, latency, error) so two runs can be diffed later.

Raise --timeout for N≥3. Renders now queue one at a time (see below), so the last request in a batch waits for every render ahead of it before its own starts. The server lets a request wait 300 s for a slot and then allows the render itself to run; the script's 180 s gives up first, so a slow render or two ahead of it turns a queue the server is handling fine into a client-side "failure". Something like --timeout 900 measures the queue rather than the script's patience.

Staging the clips (creating a circle, drawing the turn, uploading each take) happens sequentially first and is not timed — only the concurrent POST /turns/{id}/render-fallback calls are. The script's own summary reports count of ok/failed, wall time for the whole batch, and latency min/p50/p95/max/stdev.

Watch the /turns/... request budgets too: staging takes and asking for renders each have their own per-IP hourly limit (200 takes, 10 renders; see limits.md). The script warns if a run would exceed either -- past that point you are measuring 429s, not the render pipeline, unless you raise or disable the limits on the instance under test.

Sanity-checking the script itself

You do not need docker-compose or Postgres just to confirm the script drives the API correctly. The backend defaults to SQLite when DATABASE_URL is unset, so a plain uvicorn process is enough for a small run:

cd backend
SERVER_SIDE_RENDER_ENABLED=true SCHEDULER_ENABLED=false \
  .venv/bin/uvicorn app.main:app --port 8000 &

cd ..
python scripts/load_test_render.py --renders 2 --clips 2 --clip-seconds 5

This is what the script's own verification used — it is a check that the harness works end to end (setup, concurrent firing, reporting), not a measurement of anything resembling production load: SQLite, no resource caps, and N=2–3 tell you nothing about where the fallback render actually degrades.

What the SLO turned out to be

That run has now been done, against what was then the real target instance (a container capped at --cpus=1 --memory=1g --memory-swap=1g, matching fly.io's shared-cpu-1x, 1 GB). Production has since dropped to 512 MB to save money (see limits.md); nothing below has been re-measured there. It didn't converge on "N before it degrades" the way a capacity sweep usually does — it hit the wall at the very first step:

  • N=1 — fine, comfortably.
  • N=2, smallest possible clips — both die to OOM. Every N above that failed too. There was no gradient to sweep; the ceiling is the floor.

The reason a sweep found a cliff instead of a curve: render_fallback used to let every concurrent request shell out to its own ffmpeg process with nothing to stop it, on a machine where a single render already used most of the 1 GB budget. Two at once was never going to fit, regardless of clip count.

Backpressure exists now. app/services/render_gate.py serialises the actual render() call behind an in-process semaphore (settings.max_concurrent_renders, default 1, matching the measurement above) — a second request queues instead of racing the first for memory, and gives up with a 503 rather than hanging if the queue runs longer than settings.render_queue_timeout_seconds (default 300s). Re-run afterward at N=2/3/5, every render now succeeds, with latencies stacking in multiples of one render's own time — queued, not failing. See architecture.md for the full account.

"Degraded" is now "queued," not "one render's failure taking others down with it" (point 3 from the original version of this section) — which is the threshold that mattered. Raising max_concurrent_renders above 1 is only a real option if the machine's memory grows to match; nothing here assumes that will happen, and this script is exactly what to re-run if it does.