Testing¶

Four layers, each catching a different class of bug — none of them a substitute for the others. The backend suite proves the logic is right; nothing in it ever starts Postgres, writes a real object, or drives the app the way a phone does. The full-stack smoke test proves the opposite: that the pieces boot and talk to each other for real, without re-testing every edge case pytest already owns. Mobile's suite covers the client the same way — pure logic in ts-jest, real component trees in jest-expo — and stops exactly where a real device begins.

flowchart TD
    subgraph unit["Logic — fast, every PR and every push to main"]
        PT["pytest<br/>backend/tests/"]
        JT["Jest (ts-jest)<br/>mobile/src/**/__tests__"]
        SC["Jest (jest-expo)<br/>mobile/src/__tests__/screens"]
    end
    subgraph real["Real infra — its own job"]
        SM["Full-stack smoke test<br/>scripts/smoke_test.py"]
        NA["native.yml<br/>compile + API-level lint<br/>(native paths only)"]
    end
    subgraph manual["Real device / real load — manual or scheduled"]
        LT["Load test<br/>scripts/load_test_render.py"]
        MA["Maestro E2E<br/>.maestro/"]
        AT["androidTest<br/>on-device instrumentation"]
        KT["Kotlin JVM tests<br/>android/src/test"]
        PC["Nightly push check<br/>scripts/push_check.py"]
    end

    PT -.->|"needs it running for real"| SM
    JT -.-> SC
    NA -.->|"same Kotlin module"| AT
    NA -.-> KT
    SM -.->|"same render pipeline,<br/>at scale"| LT

    style unit fill:#1f6f43,color:#fff
    style real fill:#8a6d1f,color:#fff
    style manual fill:#5a3d99,color:#fff

What CI runs, and when¶

ci.yml runs on every pull request and on every push to main — not on a push to any other branch. The rest only run when what they check could have changed, or on a clock:

Workflow	Runs	What it does
`ci.yml` → `backend`	PRs, pushes to `main`	`ruff check app tests ../scripts`, then `pytest --cov` (with real FFmpeg installed, for `test_render.py`)
`ci.yml` → `mobile`	PRs, pushes to `main`	`npx expo-doctor`, `tsc --noEmit`, `npm run lint`, `npm run test:coverage`, `npm run check:contract`
`ci.yml` → `smoke`	PRs, pushes to `main`	The full-stack smoke test against `docker compose`
`ci.yml` → `audit`	PRs, pushes to `main`	`npm audit` (full report, fails only on critical) and `pip-audit` on `requirements.lock` (fails on any finding)
`native.yml`	PRs and pushes to `main` that touch `mobile/modules/**`, the app config or `package.json`; by hand	Compile and lint the Kotlin module; the instrumented tests only by hand — see Native
`docs.yml`	PRs and pushes to `main` that touch `docs/` or `mkdocs.yml`	`mkdocs build --strict` — a broken internal link fails it — and, from `main`, publishes the site
`e2e-mobile.yml`	By hand only	The Maestro flows on an emulator
`push-check.yml`	06:00 UTC, and by hand	The nightly push check
`backup-restore-drill.yml`	04:00 UTC, and by hand	`scripts/restore_drill.py` against a freshly seeded `docker compose` stack: dump, restore into a throwaway Postgres, check row counts, Alembic revision, foreign keys and media

The contract check regenerates src/api/generated-types.ts from the backend's OpenAPI schema and fails if it differs from what is committed — the app's src/api and the FastAPI routers change in separate PRs, and nothing else notices when they drift apart.

The dependency audit is not advisory. npm audit only stops the build on a critical advisory, because every high one at the time was transitive through expo's own build tooling and nobody's to bump — a check that can never go green teaches people to scroll past red. pip-audit has no severity threshold to set, so any finding it reports fails the job; the one exception is --ignore-vuln PYSEC-2026-1325, a timing attack in ecdsa (pulled in by python-jose) with no upstream fix, ignored deliberately with the reason next to it in ci.yml. A new unfixable advisory gets the same treatment, not a looser check.

Backend (`backend/tests/`)¶

pytest --cov, run on every PR and every push to main, after ruff check. Every test here talks to SQLite and every *_backend setting (storage, push, email) is forced to a no-op — see the _no_backend_talks_to_the_outside_world fixture in conftest.py. That is deliberate: it makes the suite fast and hermetic, and it is exactly why it cannot be the whole story (see Full-stack smoke test below).

One real exception: test_render.py shells out to actual FFmpeg rather than mocking it, because the render pipeline's failures are all in what FFmpeg actually does with the bytes — a pixel format ffmpeg silently renegotiates, a timebase two encodes disagree on. A mock would never catch either. This is also where the >2-clip cases live that caught the transition-assembly memory bug — see Known failure modes and safety nets.

Running it locally needs Python 3.11 or newer (requires-python in pyproject.toml; CI uses 3.12) and ffmpeg and ffprobe on PATH. Without either binary, test_render.py skips as a whole module — the run still goes green, with every render test reported as s, so a green local run without FFmpeg says nothing about the render pipeline.

test_docs_stay_true.py is the documentation's own check: every setting in Settings has to appear in .env.example (or be excluded on purpose), the example's values have to agree with the code's defaults, and every relative link in docs/*.md has to point at a file that exists. It only covers facts with one unambiguous source; the rest of this page is checked by people.

Mobile (`mobile/src/**/tests`)¶

npm test, two Jest projects (mobile/jest.config.js):

unit (ts-jest, Node environment) — pure logic: the EDL, the write queue, storage, the API client. test:coverage's threshold is not measured over all of it but over an explicit list of 18 pure modules (collectCoverageFrom in jest.config.js) — the EDL and engine choice, the write queue, storage, api/retry.ts and api/multipart.ts, formatting and language. src/api/client.ts has tests (below) but is not in that list, and neither are screens: see the comment at the top of jest.config.js for why a coverage figure over them would be a facade, not a signal.
screens (jest-expo) — real component trees via @testing-library/react-native, under src/__tests__/screens/ rather than app/ (a *.test.tsx there would become a bogus route — expo-router turns every file under app/ into one). This is where the reachability regression tests live: assertions that a screen's critical content sits inside a ScrollView/FlatList with a real scroll viewport, not just present somewhere in the tree. jest-expo does no real layout, so these check structure, not pixels — a screen can still clip content in a way only a real device shows. Added after eight screens shipped with content stuck outside any scroll container; see the ScrollView entry in gotchas.md.

Both projects map expo-localization to mobile/jest/expo-localization.js, a phone set to Spanish, so existing assertions keep describing what a Spanish phone shows. English is covered where the words change order (src/ui/__tests__/format.en.test.ts, which mocks the module to English) and by src/i18n/__tests__/language.test.ts, which pins which phone gets which language and that neither dictionary has a blank entry.

Network-failure behaviour of ApiClient and the upload retry path (stageWithRetry) is pinned down in src/api/__tests__/client.test.ts and src/edit/__tests__/serverRender.test.ts: a dropped connection, a request that times out (ApiClient.request() carries its own 30 s deadline via AbortController), and a transient 5xx during upload getting the same backoff a dropped connection does. A 4xx, and the specific "server render disabled" 501, are still treated as final — retrying either would not change the server's mind.

Accessibility lint — eslint-plugin-react-native-a11y's rules are wired into mobile/eslint.config.js, catching missing labels, invalid roles, and nested touchables. They run at warn, not error: the first pass across existing screens surfaced dozens of pre-existing findings, too many to fix as part of adding the lint. npm run lint runs as its own step in ci.yml's mobile job, alongside expo-doctor, tsc, the test suite and the contract check — so it fails the build on a real lint error (the one that existed, app.config.js using __dirname without a Node-aware ESLint environment, is fixed), but the warn-level a11y findings stay visible-but-non-blocking on purpose. Deciding whether those should block on new violations (rather than the pre-existing backlog) is still unstarted work.

Full-stack smoke test¶

scripts/smoke_test.py, wired into ci.yml as its own smoke job — on every PR and every push to main, no secrets required (every setting in docker-compose.yml has a workable default). It drives one complete user journey over plain HTTP against a real stack: real Postgres, real (local) storage, a real built Docker image — register → login → create a circle → a turn is drawn → recording/rendering heartbeats → upload → premiere → the audience downloads and byte-matches the vlog → a push round-trip. One happy path, no variants: every edge case here already has dedicated pytest coverage, and duplicating them against a slower, real-infra harness would only make CI slower without adding a signal.

It deliberately never shells out to FFmpeg — the app's own architecture is "the edit happens on the client," so the server-visible trace of a render is just the turn's status walking recording → rendering. Real FFmpeg is exercised elsewhere (test_render.py, and the load test below).

Also runnable by hand against any live deployment (docs/deployment.md → "Checking it") as a pre/post-deploy check.

Load testing the render pipeline¶

scripts/load_test_render.py fires N concurrent render-fallback requests at a running backend and reports latency, failures, and (optionally, via --docker-container) the API container's memory footprint while they run. It exists because render-fallback is a synchronous endpoint that ties up one of Starlette's threadpool workers for the whole encode, and FFmpeg is exactly as heavy as limits.md says: one shared-cpu-1x machine, and "the memory is for FFmpeg, not the API."

Not wired into CI — it needs SERVER_SIDE_RENDER_ENABLED=true and a machine sized like production to mean anything; see docs/load-testing.md for how to run it that way.

Run against what was then the production spec (docker update --cpus=1 --memory=1g --memory-swap=1g, matching Fly's shared-cpu-1x, 1 GB; production is now 512 MB, so re-run it with --memory=512m --memory-swap=512m to see the current machine), this is what turned up two real production bugs and settled the SLO question the issue originally asked. A single render already sits close to the memory ceiling even after the transition-assembly fix: two concurrent renders, of any size, reliably OOM-killed both. The answer wasn't a number to tune — it was that this machine cannot safely run more than one real assembly at a time, so render_fallback now enforces exactly that (RenderGate in app/services/render_gate.py, settings.max_concurrent_renders, default 1): a second request queues instead of racing the first for memory, and gives up with a 503 rather than blocking forever if the queue is too long (settings.render_queue_timeout_seconds). Verified with this script at N=2/3/5 concurrent: all now succeed, latencies stacking in multiples of one render's time — queued, not failing. Raising the limit above 1 is only a real option if the machine's memory grows to match; nothing here assumes that will happen.

Native (Android)¶

Three layers, of which CI runs one on its own.

native.yml's compile job runs :vlog-editor:compileDebugKotlin and :vlog-editor:lintDebug on every PR, and every push to main, that touches native paths (mobile/modules/**, the app config, package.json), and on demand. Editing a screen does not pay for a Gradle run. The API-level lint (NewApi — a call above the declared minSdk that compiles fine and crashes only on an older phone) blocks the job; it used to be continue-on-error: true (#105); see the minSdk entry in gotchas.md for the incident that made this matter.
JVM unit tests in mobile/modules/vlog-editor/android/src/test/ — StallGuardTest, DecoderDrainTest, EngineCapabilitiesTest and VideoPipelineSampleSizeTest. They cover the parts of the engine that were written against small interfaces precisely so they could run without a device: the stall deadline, the shared decoder feed/drain loop, the capability check's decision over a made-up codec list, and how far a large photo is downsampled before it becomes a texture. No workflow runs them yet; adding them to native.yml's compile job is #169. Until then, run them by hand (JDK 17 and an Android SDK, the same as a local build):
```
cd mobile
npx expo prebuild --platform android --no-install
cd android && ./gradlew :vlog-editor:testDebugUnitTest
```
On-device androidTest instrumentation (#81, extended in #134) — 14 tests across TranscoderInstrumentedTest, VideoPipelineInstrumentedTest and AudioPipelineInstrumentedTest, with real fixture clips (rotations, mixed resolutions and frame rates, a silence gap, a truncated file). It is what actually exercises MediaCodec/OpenGL behaviour, which neither the compile job nor the JVM tests can. native.yml has an instrumented-tests job for it — workflow_dispatch only, connectedDebugAndroidTest on an API 34 google_apis/x86_64 emulator, with the report uploaded as an artifact — kept off push/PR for the same reason as the Maestro job: an emulator boot is slow and the runner's KVM is not guaranteed fast. Three of the tests failed the first time anyone ran the suite (#135); #137 fixed them — swapped rotation fixtures and a real gap in how VideoPipeline noticed a truncated source — and all 14 pass on the maestro_e2e emulator. That run was local: the instrumented-tests job has not been dispatched in GitHub Actions yet, and its header comment still describes the three failures as expected (being corrected in #173).

Maestro E2E¶

.maestro/ holds five YAML flows (register/login, create-circle-and-invite, turn-draw smoke, record-edit-publish, react-and-comment) and a workflow_dispatch-only, non-blocking CI job (.github/workflows/e2e-mobile.yml) that builds the app and drives them against a real emulator.

All five flows have been run for real against a live emulator and pass, first locally and since then in GitHub Actions — a maestro_e2e AVD (Pixel 5, API 34, google_apis/x86_64), a release APK with the native vlog-editor module built in, against a real docker compose backend. Getting there found and fixed two real bugs: PUBLIC_BASE_URL pointing at localhost broke publishing from the emulator (it has to be the emulator's host alias, 10.0.2.2, for an uploaded file to actually be reachable), and a racy "stop recording" tap that could double a take. There's no in-app action to force a draw on demand, so 03–05 seed that state directly against the API first (scripts/seed_maestro_turn.sh) rather than waiting on the real scheduler. The CI workflow (e2e-mobile.yml, workflow_dispatch-only, not wired into push/PR — an emulator boot plus a real Gradle build is 15–20+ minutes) mirrors this local run as closely as a CI emulator allows, and has gone green with all five flows: runs 36131208013 (2026-09-25) and 36574008095 (2026-09-29). It is still manual, and still not a merge blocker. See .maestro/README.md for the full account, including every selector quirk and timing assumption — that file is the source of truth here, not this page, since it changes as the flows are actually run again. The flows select Spanish labels, so the app they run against has to be built with EXPO_PUBLIC_LANGUAGE=es; see "The Maestro flows select buttons by their Spanish label" in gotchas.md.

Nightly push check¶

scripts/push_check.py walks the one part of the system that cannot be verified by reading code: push is fire-and-forget, so a broken setup and a working one look identical until nobody hears about their turn. .github/workflows/push-check.yml gives it a clock (06:00 UTC, workflow_dispatch too) instead of a human remembering to run it.

Active. PUSH_CHECK_EMAIL/PUSH_CHECK_PASSWORD repo secrets point at the project owner's own account — the only one, of the handful in the production database, with a device token on file (the app has been opened on a real device with notification permission granted). A manual workflow_dispatch run confirmed the whole chain for real: Expo accepted the send and the notification landed on the phone.

What to run, and when¶

Change	Run this
Backend logic	`pytest` (fast, and CI runs it on every PR already)
Anything touching storage, migrations, or the deploy shape	`scripts/smoke_test.py` against `docker compose up`
The render pipeline specifically	`test_render.py`, and `scripts/load_test_render.py` if the change could affect memory or latency under concurrency
A mobile screen	`npm test`, and add a `screens` project test if it renders a scrollable area
The API client / retry logic	`npm test` — see `client.test.ts` for the existing failure-mode coverage to extend
The native Kotlin module	Let `native.yml` compile and lint it; run the JVM tests by hand (`testDebugUnitTest`, until #169), and `androidTest` for behaviour changes
Anything user-facing across the whole app	Maestro (`.maestro/`) — see `.maestro/README.md` for how to run it locally