Tahinli/forge
public0 watching
★ 0
| 1 | # Wave B: releases, LFS, signed commits, standby replication (contracts between workers) |
| 2 | |
| 3 | The global rules apply: |
| 4 | - No JS and no inline styles. |
| 5 | - No unwrap in non-test code. |
| 6 | - Tests use `tempfile::TempDir`. |
| 7 | - Never put cargo target dirs or big files in /tmp. |
| 8 | - New crates go in at their latest version via `cargo add`, recorded in the root `[workspace.dependencies]`. |
| 9 | - User values come after `--end-of-options`. |
| 10 | |
| 11 | Migration numbers are fixed: `0020_releases`, `0021_lfs`, `0022_signatures`, `0023_replication`. Keep `store/migrate.rs` sorted. |
| 12 | |
| 13 | ## 1. Releases (migration 0020) |
| 14 | ```sql |
| 15 | CREATE TABLE releases (id TEXT PRIMARY KEY, repo_id TEXT NOT NULL REFERENCES repos(id), tag TEXT NOT NULL, |
| 16 | target_oid TEXT NOT NULL, title TEXT NOT NULL, body TEXT NOT NULL DEFAULT '', draft INTEGER NOT NULL DEFAULT 0, |
| 17 | prerelease INTEGER NOT NULL DEFAULT 0, author_id TEXT NOT NULL REFERENCES users(id), created_at TEXT NOT NULL, |
| 18 | published_at TEXT, UNIQUE (repo_id, tag)); |
| 19 | CREATE TABLE release_assets (id TEXT PRIMARY KEY, release_id TEXT NOT NULL REFERENCES releases(id) ON DELETE CASCADE, |
| 20 | name TEXT NOT NULL, size INTEGER NOT NULL, sha256 TEXT NOT NULL, content_type TEXT NOT NULL, |
| 21 | download_count INTEGER NOT NULL DEFAULT 0, uploader_id TEXT NOT NULL REFERENCES users(id), created_at TEXT NOT NULL, |
| 22 | UNIQUE (release_id, name)); |
| 23 | ``` |
| 24 | - **Storage:** |
| 25 | - Use the `object_store` crate (latest) with `LocalFileSystem` rooted at `<data_dir>/blobs`, behind a small `BlobStore` wrapper in forge-web (`put_stream`, `get_stream`, `delete`), so S3 can be added later. |
| 26 | - Keys are `assets/<asset id>`. |
| 27 | - Uploads stream to a temp object while computing sha256, capped at `[releases] max_asset_mb = 512`. |
| 28 | - **Tags:** |
| 29 | - Creating a release for a tag that doesn't exist creates it as a lightweight tag at the chosen branch's head, through the existing ref update path (`after_ref_update`). |
| 30 | - Deleting a release keeps the tag. |
| 31 | - **Pages** (Write to create, edit or delete; Read to view, and drafts are visible only to Write): |
| 32 | - `/{o}/{r}/releases`: list, newest first, with a "Latest" badge. |
| 33 | - `/releases/new`, `/releases/tag/{tag}`, `/releases/{id}/edit`, `/releases/{id}/delete`. |
| 34 | - Asset upload is a `multipart/form-data` POST, streamed. Check what Topcoat 0.10 offers for multipart, or use the `multer` crate on the raw body. |
| 35 | - Downloads: `/{o}/{r}/releases/download/{tag}/{name}`. Stream with `Content-Disposition: attachment` and `X-Content-Type-Options: nosniff`, and increment the counter. |
| 36 | - **Other surfaces:** |
| 37 | - Source archives: link to the existing archive route if there is one. |
| 38 | - Webhook event `release` (`published`, `updated`, `deleted`). |
| 39 | - Repo home sidebar: "Latest release". |
| 40 | - **Cleanup:** |
| 41 | - `Store::delete_repo` deletes the release rows. |
| 42 | - The web layer deletes the blobs; the repo purge path must call it. |
| 43 | - Backups: the blobs dir is included in a backup set as `blobs/` (extend `backup/create.rs` and restore) **only if** it is simple. Otherwise document it in `docs/backup.md` as an rsync-safe directory (blobs are immutable; write temp, fsync, rename). |
| 44 | |
| 45 | ## 2. Git LFS (migration 0021) |
| 46 | ```sql |
| 47 | CREATE TABLE lfs_objects (repo_id TEXT NOT NULL REFERENCES repos(id), oid TEXT NOT NULL, size INTEGER NOT NULL, |
| 48 | created_at TEXT NOT NULL, PRIMARY KEY (repo_id, oid)); |
| 49 | ``` |
| 50 | - **Storage:** content-addressed at `<data_dir>/lfs/<oid[0..2]>/<oid[2..4]>/<oid>`, shared across repos. Access is per repo through `lfs_objects`. Write temp, verify sha256 and size, fsync, rename. |
| 51 | - **Batch API:** `POST /{o}/{r}.git/info/lfs/objects/batch`, plus the path without `.git`. |
| 52 | - Content type `application/vnd.git-lfs+json`. |
| 53 | - Basic transfer only. Operation `download` needs Read; `upload` needs Write, and the repo must not be archived or a pull mirror. |
| 54 | - **Auth:** HTTP Basic with a personal access token, exactly like smart-HTTP (reuse its auth code), or anonymous for public reads. 401 sends `LFS-Authenticate: Basic realm="forge"`. |
| 55 | - **Response actions:** `href` points at `/{o}/{r}.git/info/lfs/objects/{oid}` (GET and PUT) with header `Authorization: RemoteAuth <token>`. |
| 56 | - The token is a short-lived (1h) HMAC derived from the instance key over (repo id, oid, op, expiry), so action requests don't need the PAT again. |
| 57 | - Uploads also get a `verify` action. |
| 58 | - **Validation:** oid is 64 lowercase hex; size is at most `[lfs] max_object_mb = 2048`. |
| 59 | - **Transfers and verify:** |
| 60 | - GET streams the file. |
| 61 | - PUT streams to temp with a hash check, then inserts the `lfs_objects` row. |
| 62 | - `POST .../objects/verify` checks that the row exists and the size matches. |
| 63 | - **Locks API:** not implemented. Return 404 for `/locks*`, which the client handles as "locking unsupported". |
| 64 | - **SSH:** the `git-lfs-authenticate <path> <download|upload>` exec in forge-ssh returns JSON `{href, header: {Authorization: "RemoteAuth <token>"}, expires_in}` with a repo-scoped token. The token is the same HMAC with op `any`, valid for both ops subject to the role check, so add a `lfs_token` helper in forge-core. |
| 65 | - **UI:** blob view detects LFS pointer files (`version https://git-lfs.github.com/spec/v1`) and shows "Stored with Git LFS (size)" with a download link to the media route `/{o}/{r}/media/{ref}/{path}`, which serves the LFS object, on the user-content domain if the raw route is served there. |
| 66 | - **Cleanup:** `delete_repo` removes rows. A GC of unreferenced files is an admin CLI, `admin lfs-gc`: a file with no row in any repo is deleted if it is older than 1 day. |
| 67 | - **Test:** the e2e uses the real `git lfs` binary only if it is installed; otherwise skip with a message. Unit and route tests cover the batch, PUT, GET and verify flows directly. |
| 68 | |
| 69 | ## 3. Signed commit badges (migration 0022) |
| 70 | ```sql |
| 71 | CREATE TABLE gpg_keys (id TEXT PRIMARY KEY, user_id TEXT NOT NULL REFERENCES users(id), fingerprint TEXT NOT NULL UNIQUE, |
| 72 | armored TEXT NOT NULL, emails TEXT NOT NULL /* json array from the key's uids */, created_at TEXT NOT NULL); |
| 73 | CREATE TABLE commit_signatures (repo_id TEXT NOT NULL, oid TEXT NOT NULL, status TEXT NOT NULL /* verified|unverified|unsigned|unknown_key|bad */, |
| 74 | kind TEXT /* ssh|gpg */, signer_user_id TEXT, key_fingerprint TEXT, checked_at TEXT NOT NULL, PRIMARY KEY (repo_id, oid)); |
| 75 | ``` |
| 76 | - **SSH signatures:** |
| 77 | - Build an allowed-signers file under `<data_dir>/run/allowed_signers`, rewritten atomically whenever any SSH key is added or removed. Each line is `<user email> namespaces="git" <key type> <base64>`, for every SSH key of every user. Users' emails are on the users table. |
| 78 | - Verify with `git -c gpg.ssh.allowedSignersFile=<file> verify-commit --raw <oid>`. Its output gives the principal and the fingerprint; map the fingerprint back to `ssh_keys`. |
| 79 | - **Verified** means a good signature, the key belongs to user U, and the committer email equals U's email. |
| 80 | - **GPG signatures:** |
| 81 | - Use a forge-managed `GNUPGHOME=<data_dir>/gnupg` (0700). Import every `gpg_keys.armored` on add and remove the key on delete, via the `gpg` binary. |
| 82 | - If `gpg` is missing, the GPG status is `unknown_key` and the settings page says GPG verification is unavailable. |
| 83 | - Run `verify-commit` with `gpg.program` defaulted. |
| 84 | - **Verified** means a good signature by a key whose fingerprint is in `gpg_keys` for a user whose key emails include the committer email. |
| 85 | - **Cache:** results go in `commit_signatures`. When a user's SSH or GPG keys change, delete that user's cache rows (`signer_user_id`) **and** every row with status `unknown_key`. |
| 86 | - **UI:** |
| 87 | - Settings → "GPG keys" page: add (paste armored), list with fingerprint and emails, delete. |
| 88 | - Commit list and commit page show a badge: "Verified" (green, title shows signer and key) or "Unverified" (signed but unmatched or bad). Unsigned commits show nothing. |
| 89 | - Verification is per page, for at most the 50 commits shown, using the cache. Uncached commits are verified inline with a 5s total budget; anything left unchecked shows no badge and gets checked next time. |
| 90 | |
| 91 | ## 4. Standby replication (migration 0023) |
| 92 | The standby is a **passive SSH host**: a dedicated unix account on another provider that holds bare repos and DB snapshots. It is not a running forge. At failover, `admin restore` is run from its data (see `docs/backup.md` and architecture §8.5). |
| 93 | |
| 94 | - **Config:** |
| 95 | ```toml |
| 96 | [replication] |
| 97 | target = "" # "user@host:/srv/forge-standby" ; empty = disabled |
| 98 | ssh_port = 22 |
| 99 | identity_file = "" # private key for the standby account (0600) |
| 100 | known_hosts_file = "" # pinned host key; StrictHostKeyChecking=yes always |
| 101 | db_snapshot_minutes = 15 |
| 102 | reconcile_minutes = 60 |
| 103 | ``` |
| 104 | - **Transport:** |
| 105 | - Use the system `ssh` binary with `-i identity -o IdentitiesOnly=yes -o UserKnownHostsFile=<file> -o StrictHostKeyChecking=yes -o BatchMode=yes -p port`. Git gets it through `GIT_SSH_COMMAND` built the same way. |
| 106 | - The remote base path is validated against `^/[A-Za-z0-9._/-]+$` with no `..`. |
| 107 | - Remote shell commands are built only from validated paths and fixed words; still quote each arg with single quotes. |
| 108 | - **Remote layout:** `<base>/repos/<repo id>.git`, `<base>/db/forge-<UTC ts>.db` (keep the newest 24), `<base>/db/latest -> forge-<ts>.db` (symlink, updated atomically with `ln -sfn` and `mv -T`), and `<base>/keys/` (never written by forge; the operator escrows the key). |
| 109 | - **Repo replication:** |
| 110 | - After every `after_ref_update` (debounced 5s per repo), and for every repo at each reconcile: |
| 111 | 1. `ssh … "git init --bare --quiet '<base>/repos/<id>.git'"` (idempotent); |
| 112 | 2. `git push --mirror --force ssh://user@host:port<base>/repos/<id>.git`; |
| 113 | 3. compute `refs_sha256` locally (`forge_git::backup::refs_checksum`) and remotely (`ssh … "git -C '<path>' for-each-ref --format='%(objectname) %(refname)'"`, then hash it the same way locally); a mismatch triggers a re-push, and a second mismatch records an error. |
| 114 | - Deleted repos: delete the remote repo dir at reconcile (`rm -rf` of exactly `<base>/repos/<id>.git` after validation). |
| 115 | - **DB:** every `db_snapshot_minutes`, `Store::snapshot_to` into `<data_dir>/run/replica-snap.db`, `verify_snapshot`, then `scp`-equivalent via `ssh … "cat > '<base>/db/.tmp-<ts>' && mv '<base>/db/.tmp-<ts>' '<base>/db/forge-<ts>.db'"`, streaming the file to stdin. Then update `latest` and prune to 24. |
| 116 | - **Status:** |
| 117 | - Table `replication_status(subject TEXT PRIMARY KEY /* 'db' or repo id */, last_ok_at TEXT, last_error TEXT, last_attempt_at TEXT, refs_sha256 TEXT)`. |
| 118 | - `/admin/replication` (admin): target, last DB snapshot shipped, a repos table (ok, lagging, error), a "Reconcile now" button (POST, CSRF), and audit `replication_reconcile_started`. |
| 119 | - The healthcheck reuses `[backup] healthcheck_url`? No: add an optional `[replication] healthcheck_url` pinged after each reconcile. |
| 120 | - **Test:** no real SSH in unit tests. Abstract the transport behind a trait `StandbyTransport { run(cmd) -> output, push_mirror(repo_path, remote) , upload(local, remote_path) }`, with a `LocalTransport` used in tests that maps the remote base to a local temp dir (it runs the same commands without ssh). The real `SshTransport` is unit-tested only for argument construction. |
| 121 | - **Docs:** a new section in `docs/backup.md`, "Standby replication", covering setting up the standby account (restricted key with `restrict` in authorized_keys), the known_hosts pin, and a failover runbook pointing at architecture §8.5. |