Deployment: Fly.io, Neon and Cloudflare R2

One machine running the API, a managed Postgres on Neon, and the video in Cloudflare R2 behind a public bucket domain. The phone does the editing, so the server stays small on purpose: it draws turns, hands out upload URLs and tells people when something premiered.

What this shape means before you touch anything

Exactly one machine. Not two. Two things in this app count in memory and would go quietly wrong if it ever scaled out:

• The turn scheduler runs inside the process. Two machines means two schedulers, and a turn drawn twice.
• Rate limiting counts per address in memory. Two machines means each counts roughly half the requests, so every limit is silently double what it says.

fly.toml pins this (min_machines_running = 1, auto_stop_machines = false) and the comment there says the same thing. Making the app scale out is real work -- the schedule has to move into the database with a lock, and the limiter into something shared -- not a config change.

Auto-stop stays off. A stopped machine runs no scheduler, so nobody would learn their turn had started until somebody happened to open the app.

The origin is reachable. This is the difference from running behind a tunnel, and it changes who the rate limiter believes. Fly's proxy sets Fly-Client-IP and overwrites it, so it cannot be forged; X-Forwarded-For is appended to and its first entry is whatever the caller sent. CLIENT_IP_HEADER names the one to read, and reading nothing at all is safe -- everyone then shares one budget. Do not point it at X-Forwarded-For.

Server-side rendering is the fallback, and it is expensive here. Two things send an edit to it: a phone with no native module at all, or an engine that says it cannot carry this material. The Android engine now asks the codec list (EngineCapabilities.kt, #161) and reports itself unavailable on a phone with no AVC encoder for the output size or no AAC encoder -- that phone is sent here too. When it is available it is available for everything, so on Android the material never decides it; the iOS module reports no stills, no fades and no loudness, so that second case becomes real the moment iOS compiles. FFmpeg on 1080p on a shared CPU takes minutes and slows the API while it does. It is on because the alternative is those vlogs failing outright, and it should stay rare: if it stops being rare, find out why the device engine is being skipped rather than buying a bigger machine.

Once, on your side

You need three accounts: Fly.io with a card on it (this costs a few dollars a month, not nothing -- about $3.20 for the one 512 MB machine, see limits.md; Fly does not charge invoices under $5, which is why a short first month may bill nothing), Neon for the database, and Cloudflare for R2. Only the Fly one is paid.

export PATH="$HOME/.fly/bin:$PATH"
flyctl auth login

The database

Neon, not Fly's own Managed Postgres. The smallest Fly plan is $38 a month against a free Neon project, and at this size the difference buys reliability this does not need yet. It cost one change: DATABASE_URL. See decisions.md.

At https://console.neon.tech, create a project on the Free plan in Frankfurt (eu-central-1), which is the region nearest the Fly machine in Amsterdam.

Take the direct connection string, not the pooled one. psycopg 3 prepares statements on its own and PgBouncer in transaction mode does not support that; the failure arrives later and looks unrelated. See gotchas.

It will start postgres://, which SQLAlchemy does not accept -- the app rewrites that itself (app/core/config.py), so paste it exactly as given.

What the free plan gives you also shapes the deployment: the compute suspends after five idle minutes, which is why SCHEDULER_INTERVAL_SECONDS is 900 in fly.toml rather than the 60 used in development. The arithmetic is in limits.md.

The bucket

In the Cloudflare dashboard, R2 → Create bucket, named vlogroulette-media. Then Manage R2 API tokens → Create API token, with Object Read & Write scoped to that bucket. Keep three things: the access key id, the secret, and the account id, which is the <account-id> in the endpoint https://<account-id>.r2.cloudflarestorage.com.

Then turn on a public URL for the bucket: Settings → Public access. Either the r2.dev domain or a custom one works. This is what CDN_BASE_URL points at, and it is what makes a premiere cheap -- the video is served from Cloudflare's edge and never touches the API machine. Without it, playback URLs are signed and every viewer streams through R2 directly, which still works and still has no egress fee, but loses the caching.

The app

flyctl apps create vlogroulette-api

flyctl secrets set \
  SECRET_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')" \
  DATABASE_URL="postgres://..." \
  S3_ENDPOINT_URL="https://<account-id>.r2.cloudflarestorage.com" \
  S3_BUCKET="vlogroulette-media" \
  S3_ACCESS_KEY_ID="..." \
  S3_SECRET_ACCESS_KEY="..." \
  CDN_BASE_URL="https://<your-public-bucket-domain>" \
  PUBLIC_BASE_URL="https://vlogroulette-api.mintos.space" \
  CLIENT_IP_HEADER="fly-client-ip" \
  SERVER_SIDE_RENDER_ENABLED="true"

flyctl deploy --ha=false

--ha=false is not optional. Fly creates two machines for a new app by default, and nothing in fly.toml prevents it -- min_machines_running is a floor, not a ceiling. Two machines means two schedulers, which means a turn drawn twice. Check it afterwards:

flyctl status --app vlogroulette-api    # exactly one machine under Machines

SECRET_KEY signs every session token. Changing it later logs everyone out.

The app refuses to start on three things outside development (check_production_settings in app/main.py):

• a weak SECRET_KEY -- not only the repository default, but any value in WEAK_SECRETS (every placeholder this repo has ever shipped, .env.example's included) or shorter than 32 characters;
• DEBUG on, which is also its default -- fly.toml sets it to false;
• STORAGE_BACKEND=local, whose /media route has no authentication.

The check is skipped only when ENVIRONMENT is exactly development or test. Anything else, a typo included, gets the strict treatment.

PUBLIC_BASE_URL in the command above does nothing on this deployment: only the local storage backend reads it, to build /media URLs. It is harmless and can be left out.

Migrations

fly.toml runs alembic upgrade head as the release_command, once per deploy in a machine of its own, and sets RUN_MIGRATIONS=false so the container's entrypoint does not try again. The entrypoint (backend/docker-entrypoint.sh) migrates on boot when RUN_MIGRATIONS is unset or true, which is what docker-compose relies on. With SQLite -- development and tests -- neither runs: the app creates the tables itself at startup.

Error reporting

flyctl secrets set SENTRY_DSN="https://...@....ingest.sentry.io/..."

Without it nothing is reported anywhere, and with ENVIRONMENT=test nothing is reported even when it is set. SENTRY_TRACES_SAMPLE_RATE stays at 0.0: see limits.md. RELEASE is stamped on every event so a stack trace can be tied to a deployment; whether it falls back to a value Fly provides on its own when unset is being settled in #171 -- check app/core/config.py before relying on it.

Settings worth knowing about

The rest have defaults that suit this deployment, and are documented one by one in backend/.env.example. The ones somebody may actually need to touch:

Setting	Default	What it does
CORS_ALLOWED_ORIGINS	[]	Origins allowed credentialed cross-origin requests. Empty mounts no CORS at all, which is right for native apps; with DEBUG on, empty allows * without credentials. A literal "*" is refused.
UPLOAD_URL_TTL_SECONDS	3600	How long a presigned upload URL lasts (1 h).
PLAYBACK_URL_TTL_SECONDS	86400	How long a signed playback URL lasts (24 h). Irrelevant while CDN_BASE_URL is set.
FFMPEG_TIMEOUT_SECONDS	600	Ceiling for one ffmpeg/ffprobe process in the server-side render.
STAGING_DIR	.staging	Where raw takes wait for a server-side render, on the machine's own disk.
RESET_CODE_TTL_MINUTES	15	Lifetime of a password-reset code.
REMINDER_HOURS_BEFORE	24	How long before a deadline the vlogger is reminded.
EXPO_PUSH_URL	Expo's send endpoint	Only worth changing to point at a test double.

backend/.env.example also carries DEFAULT_CADENCE_DAYS, DEFAULT_MAX_TOTAL_SECONDS and DEFAULT_CLIP_MAX_SECONDS; nothing reads them -- the per-circle defaults live in the request schema -- and #170 is deciding whether they go. See limits.md.

The domains

All three services answer on mintos.space, and nothing was moved to do it — Fly still runs the container, R2 still holds the video, Pages still serves the site.

Name	Points at	Proxied?
vlogroulette-api.mintos.space	Fly, A + AAAA	No
vlogroulette-media.mintos.space	R2 bucket	Yes
vlogroulette-docs.mintos.space	Pages project	Yes

The API record must stay DNS-only. Fly issues and terminates its own certificate for that hostname; putting Cloudflare's proxy in front would need an edge certificate covering the name, and would break it. R2 and Pages are the opposite: they are only served through the proxy.

Why hyphens and not api.vlogroulette.mintos.space. Cloudflare's free Universal SSL covers the apex and one level of subdomain on a full-setup zone. A second level needs Advanced Certificate Manager, around ten dollars a month, to achieve the same thing with a dot instead of a hyphen.

Email, for password resets

Without this a forgotten password is a lost account: there is no other way back in. It stays off until configured, and while off a reset still works in development — the code goes to the server log, which is where you would look.

Currently Resend over SMTP, sending as no-reply@mintos.space.

flyctl secrets set \
  EMAIL_BACKEND=smtp \
  EMAIL_FROM="VlogRoulette <no-reply@mintos.space>" \
  SMTP_HOST=smtp.resend.com SMTP_PORT=587 \
  SMTP_USERNAME=resend \
  SMTP_PASSWORD="<the Resend API key>"

The username is literally resend; the password is the API key. Because it speaks SMTP rather than a provider's API, moving to a different service later is four settings and no code.

Setting up a sending domain

Resend needs DNS records before it will send as your domain. Use its Auto configure button rather than copying them by hand — the DKIM value is a long public key, and one wrong character means mail silently stops leaving.

The records land on a send. subdomain, deliberately, and this is worth understanding before touching anything: mintos.space already carried MX records and an SPF from Namecheap's default forwarding, left behind when the domain moved to Cloudflare. Had Resend put its SPF on the apex it would have collided with that one, and two SPF records break both. Always check for existing mail configuration on a domain before adding to it.

A consequence worth knowing: those inherited MX records point at a registrar service that no longer serves this domain, so replies to no-reply@ go nowhere. Normal for a no-reply address, and fixable with Cloudflare Email Routing if it ever matters.

DMARC is offered by Resend and not configured here. v=DMARC1; p=none; is monitoring only and harmless, but it applies to the whole domain rather than this project, so it is the domain owner's call.

Checking it, in the order things break

curl https://vlogroulette-api.mintos.space/health

storage should read s3, environment production and scheduler true; server_render says whether the fallback render is on, and is what the app reads to decide whether to offer it. scheduler is the SCHEDULER_ENABLED setting echoed back, not a sign the scheduler is alive -- a crashed scheduler still reads true here. Then, from a checkout pointed at the same settings:

backend/.venv/bin/python scripts/smoke_test.py --api https://vlogroulette-api.mintos.space

Walks one full user journey over real HTTP -- register, login, create a circle, draw a turn, record and render on the "device", upload, premiere, have the audience actually download the bytes, and round-trip a push notification -- against whatever is actually running, not a test double. Run it before pointing traffic at a new deploy and again right after; see scripts/smoke_test.py for what it does and does not verify (PUSH_BACKEND below decides whether the push step is a real send or a wired-but-silent no-op).

backend/.venv/bin/python scripts/r2_check.py

This is the one that matters. It presigns an upload, PUTs it with a plain HTTP request the way the phone does, downloads it through the playback URL, compares the bytes and deletes it. Everything that goes wrong with R2 goes wrong in the gap between signing and uploading, and this is the only thing that crosses it.

backend/.venv/bin/python scripts/push_check.py --api https://vlogroulette-api.mintos.space --email ...

That last one also runs by itself: .github/workflows/push-check.yml, daily at 06:00 UTC (and by hand from the Actions tab), with the repository secrets PUSH_CHECK_API_URL (optional; production by default), PUSH_CHECK_EMAIL and PUSH_CHECK_PASSWORD. It sends a real notification to whichever phone has that account signed in, every morning -- that is the point, and also why the account should be one whose owner expects it. A green run means Expo accepted the push, not that anybody saw it. See push-setup.md.

Pointing the app at it

mobile/eas.json carries EXPO_PUBLIC_API_URL per build profile, and preview and production already point at https://vlogroulette-api.mintos.space. Both that and the Sentry DSN beside it are baked in at build time, not read at runtime, so changing either means a rebuild. The development profile carries neither: the dev client takes the API URL from the Metro command line instead.

Backups

Two things cannot be rebuilt, and they fail differently. The vlogs: nothing in the app ever deletes one, so R2 holds a group's entire history and holds the only copy of it. The database: Neon keeps a recovery window, which covers a mistake and does not cover the project or the account going away.

backend/.venv/bin/python scripts/backup.py --out ~/vlogroulette-backups

It dumps the database and mirrors the bucket, skipping objects it already has, so the second run onwards is quick. It only ever adds -- a local file whose object has vanished is kept and counted, which is the point of a copy. The dump is taken with a pg_dump of the server's major version, through Docker, because a dump from a newer one produces a file the server cannot read back.

It reads the same backend/.env as the app, and refuses to run unless STORAGE_BACKEND=s3 -- with local storage there is no bucket to copy. Two more things land in the destination:

• The Android signing key, if ANDROID_KEYSTORE_PATH points at it, copied to <out>/keys with permissions 0600. Lose that file and no installed copy of the app can ever be updated again. Its passwords are deliberately not copied -- they stay in backend/.env, so one lost disk is not the whole set.
• manifest.json, one entry appended per successful run: when, which dump, and how many objects and bytes. The history of what the backups held.

--verify-only checks what is already there without copying: every object present, and the newest dump readable by pg_restore --list, which a truncated file is not.

On a schedule

sh deploy/systemd/install.sh

A systemd user timer, not a cron entry, and the difference is the whole point on a laptop: cron silently skips a run that falls while the machine is off, which is most nights, and that is how somebody ends up believing in backups they do not have. Persistent=true records when the job last ran and catches up on the next boot.

Daily at 22:00 with up to fifteen minutes of jitter. To change it, edit the timer and systemctl --user daemon-reload.

systemctl --user list-timers vlogroulette-backup.timer   # when it next runs
journalctl --user -u vlogroulette-backup -n 50           # what it did
systemctl --user start vlogroulette-backup.service       # run it now
loginctl enable-linger $USER                             # run with the session closed

A failed run reports to Sentry, tagged environment: backup. Nobody reads the output of something on a timer, and a backup that has been failing quietly for three months is worse than no backup at all -- you believe you are covered. This path was tested by pointing the script at a bucket that does not exist and watching the issue arrive.

Why this is on your machine rather than in the cloud

Everything else here is rented, and a backup is the one thing that should not be. Its value comes from being independent of what it is copying: a copy that production's own credentials can reach is not protection against those credentials being misused, or against a bug with a delete in it. Nothing in production knows this copy exists, and most of the time the machine holding it is switched off.

The gap that leaves is real and worth naming: a laptop that stays off for a fortnight backs up nothing for a fortnight, and a laptop that dies takes the copy with it. The cheap answer to that, if it ever matters, is a scheduled job that copies only the database somewhere else -- kilobytes, unlike the video.

Restoring

Restoring the database, into a scratch copy first rather than over anything:

psql "$DATABASE_URL" -c 'create database restore_check'
pg_restore --no-owner --no-privileges -d "<url of restore_check>" the.dump

This has actually been run, on 2026-08-21, and came back with every table, the right Alembic revision and the rows that were in production. That is the only reason this section could originally say restoring works -- a human doing it once by hand, and never repeating it.

That is now automated: scripts/restore_drill.py takes a real pg_dump (or checks an existing one with --dump), restores it into a throwaway Postgres container, and confirms row counts match table-by-table, the Alembic revision is current, foreign keys resolve, and -- given --media-dir -- sampled vlogs from the backup are present, non-empty and valid MP4s. It runs on a schedule in .github/workflows/backup-restore-drill.yml, and can be run by hand:

docker compose up -d --build
backend/.venv/bin/python scripts/smoke_test.py
backend/.venv/bin/python scripts/restore_drill.py

Media goes back with any S3 client -- the mirror is laid out under media/ with the object keys as paths, so uploading that tree to the bucket restores it unchanged.

Sweeping up what never became a vlog

backend/.venv/bin/python scripts/sweep_orphans.py --dry-run   # report only
backend/.venv/bin/python scripts/sweep_orphans.py             # remove

Two things are left behind when a turn ends some way other than the one that cleans up after itself (#79): objects in the bucket with no Vlog row -- an upload nothing ever confirmed -- and staging directories of raw takes for turns that are no longer open. The script removes both and never touches a key a vlog points at. Anything younger than ORPHAN_SWEEP_GRACE_MINUTES (default 60) is left alone whatever the database says, because an object can land in storage a moment before its row commits.

Nothing runs it. There is no timer, no workflow and no scheduler job: it happens when somebody types it. Always --dry-run first.

It reads backend/.env, so from a checkout pointed at production it sweeps the real bucket -- but the staging directory it sweeps is the one on the machine running it. scripts/ is not in the container image, so the Fly machine's own STAGING_DIR is out of its reach; that disk is not a volume, so a deploy or a restart empties it anyway.

The documentation site

The Markdown in docs/ is the source; the site is generated from it and never edited directly.

.venv-docs/bin/mkdocs serve          # local, with live reload
.venv-docs/bin/mkdocs build --strict # what CI runs

--strict fails on a broken internal link, which is the failure mode that quietly makes documentation untrustworthy.

Published to https://vlogroulette-docs.mintos.space by .github/workflows/docs.yml on every push to main that touches docs/ or mkdocs.yml. Pull requests build the site to check it and publish nothing.

The Cloudflare token behind it is an account token (cfat_ prefix) with a single permission, Cloudflare Pages → Edit. It cannot reach R2, DNS or billing. Note that account tokens verify at /accounts/{id}/tokens/verify, not the /user/tokens/verify endpoint most examples show.

What to watch afterwards

• Errors go to Sentry when SENTRY_DSN is set, and nowhere when it is not. flyctl logs still has everything the machine printed, but only until it is replaced.
• The backup runs on your laptop, and only while it is on. The systemd timer catches up a run missed overnight, but a machine switched off for a fortnight backs up nothing for a fortnight. systemctl --user list-timers says when it last managed one.
• Storage only grows. No vlog is ever deleted, by design -- worth knowing before it is a surprise on a bill. The one exception is leftovers that never became a vlog, below, and only when somebody runs the sweep.
• The machine count. flyctl status should always show one. If something ever scales it, re-read the top of this page.