Backups, phase 1 (contract between workers)

The background is in docs/architecture.md §8.2–8.4. Phase 1 makes complete, verifiable backups locally: a directory that an external tool (restic, rclone, or borg with append-only keys) ships off-site. Shipping it, standby replication and push mirrors come later.

The global rules apply:

No unwrap in non-test code.
Tests use tempfile::TempDir.
Never put cargo target dirs or big files in /tmp.

Backup set layout

<out>/forge-backup-<UTC yyyymmddThhmmssZ>/
  manifest.json          # format version, forge version, created_at, instance base_url, files [{path, bytes, sha256}], repos [{id, owner, name, refs_sha256, bundle | null (empty repo)}]
  db/forge.db            # consistent snapshot
  repos/<repo id>.bundle # `git bundle create --all`, then `git bundle verify` before the manifest is written
  keys/forge.key         # only with --include-key; otherwise the manifest records "key_included": false

refs_sha256 is the sha256 of sorted git for-each-ref --format='%(objectname) %(refname)' (the reconcile checksum from architecture §8.2). It's computed from the bundle-time ref list, so a restore can compare against it.
The directory is written as .partial-… and renamed atomically when complete. A crash never leaves something that looks like a finished backup.
Files are 0600 and directories 0700.

DB snapshot

Use VACUUM INTO '<path>' if Turso 0.8.x supports it; the worker must verify this with a test.
Fallback: hold the store's connection mutex so there are no concurrent writers, run PRAGMA wal_checkpoint(TRUNCATE) (or the Turso equivalent), copy the DB file (and the WAL, if any is left), then release.
Verify the result by opening it read-only with a fresh connection and running PRAGMA integrity_check (or quick_check), with the migrations version check.

Commands (`forge-web admin …`)

backup --out DIR [--include-key] [--repos-only | --db-only]: prints the path and a summary, and exits non-zero on any failure.
backup-verify PATH: recomputes every sha256, runs git bundle verify on each bundle (in a scratch repo under PATH's parent, not /tmp), opens the DB snapshot read-only and runs integrity_check, and checks the manifest's repo list against the DB snapshot's repos table.
restore --from PATH --data-dir NEW_DIR: refuses a non-empty NEW_DIR. Copies the DB, then for each repo runs git clone --mirror <bundle> into repo_path(NEW_DIR, id) (or init --bare + fetch from the bundle with +refs/*:refs/*) and removes the remote. Then writes the managed gitconfig and hooks the same way boot does, compares refs_sha256 per repo, and writes forge.key if the set has one (otherwise warns loudly that the key must be restored by hand).
fsck [--full]: git fsck (--full if asked) on every repo plus the DB integrity_check. Reports per repo, and exits non-zero on any problem.

Scheduler (config `[backup]`)

[backup]
dir = ""                  # empty = disabled
interval_hours = 24       # full backup set
keep = 7                  # keep the newest N complete sets in dir (delete older complete sets only; never touch other files)
healthcheck_url = ""      # optional; GET <url>/start, then <url> on success or <url>/fail with the error (healthchecks.io style)
fsck_interval_days = 7

A tokio task started at boot runs the same code as the CLI. Healthcheck pings use a plain reqwest client with a 10s timeout; the URL is admin-configured and trusted.
GET /admin/backups (instance admins): the last 20 runs (time, duration, size, ok or error), backup and fsck statuses, and a "Run backup now" POST with CSRF. Runs are recorded in a table backup_runs(id, kind 'backup'|'fsck', started_at, finished_at, ok, path, bytes, error), migration 0014_backup_runs.sql.

Docs

docs/backup.md covers:

what is in a set and what isn't (the key);
how to ship it off-site with restic append-only or rclone, plus a sample systemd timer;
how to restore, step by step;
a quarterly restore drill;
the fencing reminder from architecture §8.5.

Tahinli/forge