Architecture

Guiding principle: heavy client, light server

What drives the cost of a video product is whoever processes the frames. Here the phone assembles the vlog and the server never sees a raw frame: it only signs URLs, keeps rows in PostgreSQL and decides whose turn it is.

The premiere spike — the whole group opening the vlog at once — is soaked up by the CDN. The origin never notices. The topology is drawn below, and the exchange that makes it work in Storage and network.

The pieces

flowchart LR
    subgraph phone["Phone (does the work)"]
        CAM[Camera] --> MAN[(Manifest<br/>on disk)]
        MAN --> EDL[EDL]
        EDL --> NAT[Native engine<br/>MediaCodec + OpenGL]
    end

    subgraph fly["Fly.io - one machine, Amsterdam"]
        API[FastAPI]
        SCH[Scheduler<br/>every 15 min]
    end

    NEON[(Neon Postgres<br/>Frankfurt)]
    R2[(Cloudflare R2)]
    CDN{{Public bucket domain}}
    EXPO[Expo Push] --> FCM[FCM] --> phone

    NAT -->|"presigned PUT"| R2
    phone <-->|"JSON only"| API
    API --- NEON
    SCH --- NEON
    API -->|"signs, never carries"| R2
    R2 --- CDN
    CDN -->|"playback"| phone
    API --> EXPO

    style R2 fill:#f38020,color:#fff
    style NEON fill:#00e599,color:#000
    style fly fill:#7b3fe4,color:#fff

The arrow that is missing is the point: no video ever passes through the API.

Backend (backend/)

Module	Responsibility
app/api/	HTTP routers: auth, circles, turns, media (dev only)
app/services/turns.py	Turn life cycle and the re-roll vote. The heart of the loop
app/services/draw.py	The draw: weighted RNG on top of a CSPRNG
app/services/storage.py	Object abstraction: LocalStorage (dev) / S3Storage (R2)
app/services/push.py	Expo Push Service, with a null backend for development
app/services/edl.py	The edit algorithm (mirror of the client's)
app/services/render.py	Fallback FFmpeg render (optional, off)
app/workers/scheduler.py	APScheduler: one tick() per interval (900 s live)

Client (mobile/)

Module	Responsibility
src/edit/edl.ts	Edit Decision List: cuts and transitions. Pure
src/edit/vad.ts	Silence detection (FFmpeg's silencedetect). Unreachable -- see below
src/edit/commands.ts	FFmpeg command construction. Pure
src/edit/renderer.ts	Orchestrates VAD -> EDL -> normalisation -> assembly (FFmpeg). Unreachable -- see below
src/edit/nativeRenderer.ts	The path a device actually takes: the same job through the native engine
src/edit/engine.ts	Picks between ffmpeg, native and server, and says what each gives up
modules/vlog-editor/	The native engine: MediaCodec + GL on Android
src/edit/serverRender.ts	Compatibility path: assemble on the server
src/edit/publish.ts	Upload with retries, confirmation and purge of raw clips
src/storage/clipVault.ts	Clip vault in the sandbox plus a persistent manifest
src/state/	Session, circles, recording, recovery after a power cut
src/state/notifications.ts	Push registration, a no-op inside Expo Go
src/ui/Splash.tsx	Boot screen shown while the session and any pending render resume
src/ui/Sheet.tsx	Bottom sheet behind the circle's overflow menu
src/ui/Roulette.tsx	The draw reveal: names spin past and land on the vlogger
src/storage/seenDraws.ts	Which reveals this device has already played
app/	Screens (expo-router)

The draw

app/services/draw.py. Not a flat random.choice, because in a group of four that produces streaks that kill the fun:

1. Active members only.
2. The previous vlogger is excluded (unless they are the only option).
3. Each candidate's weight is (group_max_turns - their_turns) + 1. Whoever has come up least gets more tickets.
4. The tie-break uses secrets.SystemRandom, not random.

The tests (backend/tests/test_draw.py) check that across 60 rounds with five people everybody comes up, and that nobody lands outside 5..19 turns against a fair share of 12. The bounds are deliberately loose: a tight assertion on a random draw is a flaky test, and an earlier version asserting a spread of 2 failed about one run in six. See decisions.md.

The re-roll vote

A turn can be re-drawn on demand if the circle agrees. POST /turns/{id}/reroll-vote records a vote; the moment a strict majority is in, the re-roll happens inside that same request and the response is already the new turn. Nobody waits for the scheduler.

Design decisions worth knowing:

• Strict majority, n // 2 + 1. With 2 members that means both of them — a majority of two people is two people. With 1 it means 1, so a solo circle can re-roll itself.
• During a premiere the same vote needs everyone. Re-rolling an open turn costs nothing; ending a premiere early takes the vlog away from whoever has not watched it, so that one is unanimous. The turn goes to archived, exactly where it would have landed after the full 24 h.
• No strike. Unlike a missed deadline, this is the group deciding, not the vlogger failing. The turn ends as skipped, not missed.
• The vlogger may vote too. Often the person who most wants out is the one holding the turn.
• Votes from people who left stop counting. Otherwise someone could leave the circle and leave behind a vote they can no longer withdraw, holding the turn hostage.
• Votes belong to the turn, so they vanish with it — there is no counter to reset when the next turn opens.

What the phone keeps, and for how long

Raw takes are the heaviest thing the app touches, so the sandbox is swept from three directions:

• On publish. purgeRawClips deletes the takes and purgeTurn removes the turn's directory outright, finished mp4 included. Same on both render paths, and the server clears its staging area too.
• On every render. The normalised segments live in a render/ working directory cleared in a finally. They are as heavy as the takes themselves, and a failed render used to leave them behind until a publish that was never going to happen.
• At startup. Takes for a turn that has since closed — someone else was drawn, the circle re-rolled, the deadline passed — are purged by state/recovery.ts. Nothing revisits an idle manifest during normal use, so before this they simply accumulated. It only deletes on a definitive answer from the server: a network error means "unknown", and footage is not worth deleting on a guess.

Clips the edit drops are not deleted early, and cannot be: which takes get dropped is only known once the EDL is computed — a take can fall below the minimum length, or land past the total the vlog can hold — and until then they are the user's only copy.

Circles with a single member

A one-person circle is a real, supported case, not a degenerate one: it is how most people will try the app before inviting anyone.

The draw excludes the previous vlogger unless they are the only option, so the roulette simply keeps landing on the same person. The premiere notification goes to "everyone except the vlogger", which is nobody, so no push is sent and nothing errors. You can watch your own premiere and it counts as a view. test_reroll.py covers the full solo loop end to end.

The auto-edit engine

Which engine renders a given vlog is decided by src/edit/engine.ts, and it is the one place that knows what each one cannot do:

flowchart TD
    START([Assemble this vlog]) --> FF{Native FFmpeg<br/>linked into the build?}
    FF -->|yes| FFE[["ffmpeg: everything,<br/>on the device"]]
    FF -->|no| NAT{Native module present<br/>and reporting available?}
    NAT -->|"no: Expo Go, or a<br/>build older than it"| SRVQ
    NAT -->|yes| MAT{Does it declare it can<br/>carry this material?}
    MAT -->|yes| DEV[["native: on the device"]]
    MAT -->|"no: e.g. stills on an<br/>engine without an encoder"| SRVQ{Is the server<br/>fallback switched on?}
    SRVQ -->|yes| SRV[["server: FFmpeg, same EDL"]]
    SRVQ -->|no| NONE[["none: nothing can render it"]]

    style DEV fill:#2d7d46,color:#fff
    style FFE fill:#2d7d46,color:#fff
    style SRV fill:#8a6d1f,color:#fff
    style NONE fill:#8a2f2f,color:#fff

Two things the shape does not show.

The FFmpeg branch is unreachable today. It is checked first because it would be the best of both — on the device and the full treatment — but ffmpeg-kit-react-native is deprecated and its binaries are gone from Maven Central, so nothing sets that flag.

"Can it carry this material" is capability-driven, not a fixed rule. It asks the engine, and the Android engine has answered yes to stills since it gained a real encoder. It is written this way so a device or a build that cannot still produces a vlog rather than a failure.

The chooser reports what it gave up rather than silently degrading, and distinguishes "this phone has no engine" from "this material needs one", so nobody goes hunting for a fault in a device that is working fine: describeCompromise() produces the text that ends up in the vlog's render_notes, so after the fact you can tell which path a vlog took and why.

The edit is split in two halves so the hard part is testable:

1. The EDL (the decision). Given each clip and its silence spans, it computes the exact cuts. It is a pure function — no files, no FFmpeg — and it exists twice, in TypeScript (mobile/src/edit/edl.ts) and Python (backend/app/services/edl.py), with the same semantics and the same tests. Rules:

• Silence trimming is off (src/edit/trimming.ts). The spec called for it and it is built, tested and working on both the device and the server -- but on real footage it reads as the app editing over you rather than for you. A pause someone left before speaking is not necessarily dead air, and nothing here can tell the difference. Kept behind one constant, because it is a judgement about feel rather than a defect.
• When it is on, leading and trailing silence goes with a 120 ms cushion, so the cut does not bite into the first syllable.
• Internal silences are left alone: a pause in the middle of a sentence is information, not noise.
• A clip that drops below 0.6 s after trimming is discarded.
• A clip that is entirely silent (a cutaway, a landscape) is kept whole: the safe failure is not to cut.
• Transitions overlap, so they add nothing to the total duration.

The cap belongs to the finished vlog, and this is where it is enforced: the last take is trimmed to fit and anything past it is dropped. The camera works against a slightly larger recording allowance (recording_allowance_factor, 1.05x), so that the cap can actually be reached: transitions overlap, taking a quarter of a second off at every cut, so 120s of footage makes a vlog a few seconds short of 120s.

It was 1.5x while the edit trimmed silences, which reclaimed roughly that much on ordinary speech. With trimming off, that headroom became a minute of footage somebody shoots and then loses to the cap, so it is now sized to what the edit actually removes — under five seconds even at twenty takes. A test derives the figure from the transition length rather than asserting it, so changing one and not the other is caught.

The two numbers are shown side by side on the camera screen and the edit screen says how many takes did not fit, so nothing is ever lost silently.

2. The execution. Two implementations run the same EDL: the native engine on the device, and FFmpeg on the server. What follows is the FFmpeg one, which the tests below cover; the native engine does the equivalent through MediaCodec and OpenGL. Every cut is normalised to the same profile (1080x1920, 30 fps, 48 kHz stereo) before chaining. Skip that step and mixing a portrait 30 fps take with a landscape 60 fps one drifts the result's audio out of sync. Then a single filter_complex chains the segments with xfade/acrossfade, mixes the music bed at -18 dB and applies loudnorm.

Two details that break quietly, both covered by tests on either side (commands.test.ts, test_render.py):

• xfade measures its offset against the already-chained stream, not the standalone clip. You have to accumulate the duration while subtracting each fade's overlap.
• The pixel format has to be pinned. Left alone, xfade negotiates yuv444p and libx264 encodes "High 4:4:4 Predictive". Desktop players decode that without complaint, which is exactly why it survived review — on a phone, whose hardware decoder only does 4:2:0, it plays as perfect audio over a garbled picture. Every encode ends in format=yuv420p with an explicit -pix_fmt, -profile:v high and -level:v 4.0. loudnorm needs the same treatment on the audio side: it works at 192 kHz internally and leaves the output there unless a trailing aresample pulls it back to 48 kHz.

Storage and network (zero egress)

Two jobs live here that look like one, and separating them is most of the design. The server is a small program that has to be awake at all times -- it holds who is in which circle, draws turns, and counts down the 24 hours of a premiere -- but it only ever moves text. The video is 20-50 MB written once and read by everyone in the circle at roughly the same moment, which is transport, not computation.

Putting both on one machine breaks at exactly the wrong time: every viewer would be taking bandwidth and CPU from the process that also has to answer the API, during the premiere, which is when they all arrive at once. So the bytes never touch the server:

sequenceDiagram
    participant P as Phone
    participant A as API (Fly)
    participant R as R2
    participant O as Other phones

    P->>A: vlog is ready, where do I put it?
    A-->>P: presigned URL (signed, expiring)
    P->>R: PUT the file, straight past the server
    P->>A: uploaded
    A->>R: does it really exist?
    A-->>O: push: something premiered
    O->>A: anything new?
    A-->>O: playback URL
    O->>R: watch it from Cloudflare's edge

Step 3 is the one that matters. The presigned URL is a signed, expiring permit: the phone hands it to R2 and R2 accepts the upload on its own. The server authorises the transfer without carrying it, which is what lets the machine stay small enough to cost a few dollars a month.

R2 rather than S3 for one reason: egress. Most object stores charge per gigabyte downloaded, and this app's whole shape is one upload read by every member, every week, forever. R2 charges nothing for it.

STORAGE_BACKEND=s3 points at any S3-compatible service; the configuration is written for Cloudflare R2:

S3_ENDPOINT_URL=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
S3_BUCKET=vlogroulette-media
CDN_BASE_URL=https://media.yourdomain.com

With CDN_BASE_URL set, playback URLs point at the CDN domain and are not signed: the object caches at the edge. Without it, a read URL is signed against the bucket (handy for testing without your own domain).

STORAGE_BACKEND=local writes to disk and serves under /media/.... That router is only mounted in local mode: in production the API has no route that moves video bytes at all.

What happens to a vlog afterwards

A premiere lasts 24 hours and then the turn is archived — but the object stays in the bucket, and archived turns still serialise a playback URL. So the history was always there in the API; for a long time nothing in the app asked for it, and a vlog simply vanished from view while continuing to cost storage forever.

app/history/[circleId].tsx is what closes that. The rules about what counts as history are pure and tested (src/storage/history.ts):

• Only turns that produced a playable vlog. A missed or re-rolled turn is not history; a row whose only content is that somebody let the group down is worse than no row.
• Not the current premiere, which the circle screen already shows — twice on screen, saying different things about the time left, reads as a bug.
• Grouped by month. A flat list of dates reads like a log; months are how people remember when something happened.

This also changes an open question rather than answering it: keeping every vlog forever is defensible now that the group can watch them. It was not, while they were being stored and never shown.

The system clock

app/workers/scheduler.py runs turns.tick() on an interval: SCHEDULER_INTERVAL_SECONDS, 60 s by default for development and 900 s in production (backend/fly.toml). The difference is Neon: its free tier suspends the compute after five idle minutes, and a query a minute would keep it awake around the clock -- see limits.md. One tick:

1. Closes turns whose deadline passed -> missed, a strike for the vlogger and an immediate draw for the next one.
2. Closes expired premieres -> archived plus a fresh draw.
3. Starts the first turn of circles that do not have one yet.

tick() is idempotent: you can call it a thousand times without duplicating turns or notifications. With several API instances behind a load balancer, keep SCHEDULER_ENABLED=true on exactly one of them (or move the tick to an external job that calls an internal endpoint).

Turn states

The whole product is a state machine over one row, and it lives in data-model.md with a diagram, so there is one copy of it rather than two that drift apart.

The distinction worth carrying here: missed and skipped both hand the turn on immediately, and only missed carries a strike.

Server-side fallback render

SERVER_SIDE_RENDER_ENABLED=true enables POST /turns/{id}/render-fallback, which takes the raw clips and assembles them on the server with the same algorithm.

It exists for two specific reasons, and it is off by default because it cuts against the project's economics:

1. A build with no native module in it -- Expo Go, or one older than the module -- has no engine on the device at all. Without this path the product cannot be tried without compiling a binary.
2. It lets the assembly pipeline be tested in CI without a device.

The client only uses it as a reserve: engine.ts returns server when the device has no usable engine, or when the engine it has says it cannot carry the material, and the edit screen goes through serverRender.ts. If the backend has it off it answers 501 and the app explains that it must be enabled, or a development build used.

Takes go up one request each, streamed off disk by expo-file-system and staged server-side (app/services/staging.py) until the assembly runs. The first version sent every take in a single multipart body; that body has to be built in JS memory on the phone, and a mobile connection drops the request long before fifty megabytes have made it across — which showed up as a bare "network connection failed" with nothing to act on. Each upload retries with backoff, except against a server rejection, which retrying cannot fix. DELETE /turns/{id}/clips clears the staging area so a retry does not assemble the same take twice.

Turning it on in production means sizing CPU and inbound bandwidth for it: the light server becomes a video server.

Open decisions

• Playback used to be expo-av. Its Video component is deprecated and rendered 1080x1920 H.264 as colour smears on Android — the server render was provably clean, the player was mangling it. The app now uses expo-video.
• Stills are letterboxed, video is cropped. A video take can be reframed to fill the screen and lose little; cropping a landscape photo into a vertical frame throws most of the picture away, and unlike a take there is no shooting it again. So stills are shown whole with black bars, video covers the frame.
• Stills from the gallery. A photo becomes a segment through -loop 1 -t, held for a duration the client chooses — nothing in the file can say how long a still should stay on screen, so it rides along as a photo_seconds field on the upload. JPEG colour is full-range and camera footage is TV-range, so the range is converted explicitly; leaving it to ffmpeg produced a brightness jump at the cut.
• Music bed. The pipeline supports one (musicUri, -18 dB, mixed with amix), but no file ships with the project: licensing has to be settled before bundling music into the app.
• Scheduler across instances. Not a preference any more but a hard constraint: the scheduler and the rate limiter both count in memory, so the deployment is pinned to exactly one machine and a second one would draw turns twice. Lifting it means the schedule in the database behind a lock and the limiter in something shared. See decisions.md.
• Deleting the vlog after archiving. Nothing in the app ever deletes one, so R2 holds a circle's whole history and only grows. Trivial at today's size and a real product decision eventually: keep forever, expire, or archive. Whatever is decided also has to reach scripts/backup.py, which deliberately keeps local copies of objects that have gone from the bucket.
• The FFmpeg client pipeline is dead code. renderer.ts, commands.ts and vad.ts are written against ffmpeg-kit-react-native, which is deprecated on npm and whose native binaries have been withdrawn: every version of com.arthenica:ffmpeg-kit-* now 404s from Maven Central. It was never added as a dependency, so engine.ts never reaches that branch. The architecture's central claim is not waiting on it — modules/vlog-editor answered it, on real hardware — but the modules are kept, because they are the reference the server render mirrors and because an in-process FFmpeg would use them unchanged. roadmap.md has the history.
• Remote push in Expo Go. SDK 53 removed it, so notifications.ts detects Expo Go and skips registration entirely rather than logging an error nobody can act on. Real notifications need a development build.
• expo-file-system legacy API. SDK 54 introduced a new File/Directory API and moved the old one to expo-file-system/legacy, which is what the client still imports — it is the only one that exposes uploadAsync, the streaming upload. Porting is separate work.