forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
11.1 KiB

Wave B: releases, LFS, signed commits, standby replication (contracts between workers)

The global rules apply:

  • No JS and no inline styles.
  • No unwrap in non-test code.
  • Tests use tempfile::TempDir.
  • Never put cargo target dirs or big files in /tmp.
  • New crates go in at their latest version via cargo add, recorded in the root [workspace.dependencies].
  • User values come after --end-of-options.

Migration numbers are fixed: 0020_releases, 0021_lfs, 0022_signatures, 0023_replication. Keep store/migrate.rs sorted.

1. Releases (migration 0020)

CREATE TABLE releases (id TEXT PRIMARY KEY, repo_id TEXT NOT NULL REFERENCES repos(id), tag TEXT NOT NULL,
  target_oid TEXT NOT NULL, title TEXT NOT NULL, body TEXT NOT NULL DEFAULT '', draft INTEGER NOT NULL DEFAULT 0,
  prerelease INTEGER NOT NULL DEFAULT 0, author_id TEXT NOT NULL REFERENCES users(id), created_at TEXT NOT NULL,
  published_at TEXT, UNIQUE (repo_id, tag));
CREATE TABLE release_assets (id TEXT PRIMARY KEY, release_id TEXT NOT NULL REFERENCES releases(id) ON DELETE CASCADE,
  name TEXT NOT NULL, size INTEGER NOT NULL, sha256 TEXT NOT NULL, content_type TEXT NOT NULL,
  download_count INTEGER NOT NULL DEFAULT 0, uploader_id TEXT NOT NULL REFERENCES users(id), created_at TEXT NOT NULL,
  UNIQUE (release_id, name));
  • Storage:
    • Use the object_store crate (latest) with LocalFileSystem rooted at <data_dir>/blobs, behind a small BlobStore wrapper in forge-web (put_stream, get_stream, delete), so S3 can be added later.
    • Keys are assets/<asset id>.
    • Uploads stream to a temp object while computing sha256, capped at [releases] max_asset_mb = 512.
  • Tags:
    • Creating a release for a tag that doesn't exist creates it as a lightweight tag at the chosen branch's head, through the existing ref update path (after_ref_update).
    • Deleting a release keeps the tag.
  • Pages (Write to create, edit or delete; Read to view, and drafts are visible only to Write):
    • /{o}/{r}/releases: list, newest first, with a "Latest" badge.
    • /releases/new, /releases/tag/{tag}, /releases/{id}/edit, /releases/{id}/delete.
    • Asset upload is a multipart/form-data POST, streamed. Check what Topcoat 0.10 offers for multipart, or use the multer crate on the raw body.
    • Downloads: /{o}/{r}/releases/download/{tag}/{name}. Stream with Content-Disposition: attachment and X-Content-Type-Options: nosniff, and increment the counter.
  • Other surfaces:
    • Source archives: link to the existing archive route if there is one.
    • Webhook event release (published, updated, deleted).
    • Repo home sidebar: "Latest release".
  • Cleanup:
    • Store::delete_repo deletes the release rows.
    • The web layer deletes the blobs; the repo purge path must call it.
    • Backups: the blobs dir is included in a backup set as blobs/ (extend backup/create.rs and restore) only if it is simple. Otherwise document it in docs/backup.md as an rsync-safe directory (blobs are immutable; write temp, fsync, rename).

2. Git LFS (migration 0021)

CREATE TABLE lfs_objects (repo_id TEXT NOT NULL REFERENCES repos(id), oid TEXT NOT NULL, size INTEGER NOT NULL,
  created_at TEXT NOT NULL, PRIMARY KEY (repo_id, oid));
  • Storage: content-addressed at <data_dir>/lfs/<oid[0..2]>/<oid[2..4]>/<oid>, shared across repos. Access is per repo through lfs_objects. Write temp, verify sha256 and size, fsync, rename.
  • Batch API: POST /{o}/{r}.git/info/lfs/objects/batch, plus the path without .git.
    • Content type application/vnd.git-lfs+json.
    • Basic transfer only. Operation download needs Read; upload needs Write, and the repo must not be archived or a pull mirror.
    • Auth: HTTP Basic with a personal access token, exactly like smart-HTTP (reuse its auth code), or anonymous for public reads. 401 sends LFS-Authenticate: Basic realm="forge".
    • Response actions: href points at /{o}/{r}.git/info/lfs/objects/{oid} (GET and PUT) with header Authorization: RemoteAuth <token>.
      • The token is a short-lived (1h) HMAC derived from the instance key over (repo id, oid, op, expiry), so action requests don't need the PAT again.
      • Uploads also get a verify action.
    • Validation: oid is 64 lowercase hex; size is at most [lfs] max_object_mb = 2048.
  • Transfers and verify:
    • GET streams the file.
    • PUT streams to temp with a hash check, then inserts the lfs_objects row.
    • POST .../objects/verify checks that the row exists and the size matches.
  • Locks API: not implemented. Return 404 for /locks*, which the client handles as "locking unsupported".
  • SSH: the git-lfs-authenticate <path> <download|upload> exec in forge-ssh returns JSON {href, header: {Authorization: "RemoteAuth <token>"}, expires_in} with a repo-scoped token. The token is the same HMAC with op any, valid for both ops subject to the role check, so add a lfs_token helper in forge-core.
  • UI: blob view detects LFS pointer files (version https://git-lfs.github.com/spec/v1) and shows "Stored with Git LFS (size)" with a download link to the media route /{o}/{r}/media/{ref}/{path}, which serves the LFS object, on the user-content domain if the raw route is served there.
  • Cleanup: delete_repo removes rows. A GC of unreferenced files is an admin CLI, admin lfs-gc: a file with no row in any repo is deleted if it is older than 1 day.
  • Test: the e2e uses the real git lfs binary only if it is installed; otherwise skip with a message. Unit and route tests cover the batch, PUT, GET and verify flows directly.

3. Signed commit badges (migration 0022)

CREATE TABLE gpg_keys (id TEXT PRIMARY KEY, user_id TEXT NOT NULL REFERENCES users(id), fingerprint TEXT NOT NULL UNIQUE,
  armored TEXT NOT NULL, emails TEXT NOT NULL /* json array from the key's uids */, created_at TEXT NOT NULL);
CREATE TABLE commit_signatures (repo_id TEXT NOT NULL, oid TEXT NOT NULL, status TEXT NOT NULL /* verified|unverified|unsigned|unknown_key|bad */,
  kind TEXT /* ssh|gpg */, signer_user_id TEXT, key_fingerprint TEXT, checked_at TEXT NOT NULL, PRIMARY KEY (repo_id, oid));
  • SSH signatures:
    • Build an allowed-signers file under <data_dir>/run/allowed_signers, rewritten atomically whenever any SSH key is added or removed. Each line is <user email> namespaces="git" <key type> <base64>, for every SSH key of every user. Users' emails are on the users table.
    • Verify with git -c gpg.ssh.allowedSignersFile=<file> verify-commit --raw <oid>. Its output gives the principal and the fingerprint; map the fingerprint back to ssh_keys.
    • Verified means a good signature, the key belongs to user U, and the committer email equals U's email.
  • GPG signatures:
    • Use a forge-managed GNUPGHOME=<data_dir>/gnupg (0700). Import every gpg_keys.armored on add and remove the key on delete, via the gpg binary.
    • If gpg is missing, the GPG status is unknown_key and the settings page says GPG verification is unavailable.
    • Run verify-commit with gpg.program defaulted.
    • Verified means a good signature by a key whose fingerprint is in gpg_keys for a user whose key emails include the committer email.
  • Cache: results go in commit_signatures. When a user's SSH or GPG keys change, delete that user's cache rows (signer_user_id) and every row with status unknown_key.
  • UI:
    • Settings → "GPG keys" page: add (paste armored), list with fingerprint and emails, delete.
    • Commit list and commit page show a badge: "Verified" (green, title shows signer and key) or "Unverified" (signed but unmatched or bad). Unsigned commits show nothing.
    • Verification is per page, for at most the 50 commits shown, using the cache. Uncached commits are verified inline with a 5s total budget; anything left unchecked shows no badge and gets checked next time.

4. Standby replication (migration 0023)

The standby is a passive SSH host: a dedicated unix account on another provider that holds bare repos and DB snapshots. It is not a running forge. At failover, admin restore is run from its data (see docs/backup.md and architecture §8.5).

  • Config:
[replication]
target = ""                 # "user@host:/srv/forge-standby" ; empty = disabled
ssh_port = 22
identity_file = ""          # private key for the standby account (0600)
known_hosts_file = ""       # pinned host key; StrictHostKeyChecking=yes always
db_snapshot_minutes = 15
reconcile_minutes = 60
  • Transport:
    • Use the system ssh binary with -i identity -o IdentitiesOnly=yes -o UserKnownHostsFile=<file> -o StrictHostKeyChecking=yes -o BatchMode=yes -p port. Git gets it through GIT_SSH_COMMAND built the same way.
    • The remote base path is validated against ^/[A-Za-z0-9._/-]+$ with no ...
    • Remote shell commands are built only from validated paths and fixed words; still quote each arg with single quotes.
  • Remote layout: <base>/repos/<repo id>.git, <base>/db/forge-<UTC ts>.db (keep the newest 24), <base>/db/latest -> forge-<ts>.db (symlink, updated atomically with ln -sfn and mv -T), and <base>/keys/ (never written by forge; the operator escrows the key).
  • Repo replication:
    • After every after_ref_update (debounced 5s per repo), and for every repo at each reconcile:
      1. ssh … "git init --bare --quiet '<base>/repos/<id>.git'" (idempotent);
      2. git push --mirror --force ssh://user@host:port<base>/repos/<id>.git;
      3. compute refs_sha256 locally (forge_git::backup::refs_checksum) and remotely (ssh … "git -C '<path>' for-each-ref --format='%(objectname) %(refname)'", then hash it the same way locally); a mismatch triggers a re-push, and a second mismatch records an error.
    • Deleted repos: delete the remote repo dir at reconcile (rm -rf of exactly <base>/repos/<id>.git after validation).
  • DB: every db_snapshot_minutes, Store::snapshot_to into <data_dir>/run/replica-snap.db, verify_snapshot, then scp-equivalent via ssh … "cat > '<base>/db/.tmp-<ts>' && mv '<base>/db/.tmp-<ts>' '<base>/db/forge-<ts>.db'", streaming the file to stdin. Then update latest and prune to 24.
  • Status:
    • Table replication_status(subject TEXT PRIMARY KEY /* 'db' or repo id */, last_ok_at TEXT, last_error TEXT, last_attempt_at TEXT, refs_sha256 TEXT).
    • /admin/replication (admin): target, last DB snapshot shipped, a repos table (ok, lagging, error), a "Reconcile now" button (POST, CSRF), and audit replication_reconcile_started.
    • The healthcheck reuses [backup] healthcheck_url? No: add an optional [replication] healthcheck_url pinged after each reconcile.
  • Test: no real SSH in unit tests. Abstract the transport behind a trait StandbyTransport { run(cmd) -> output, push_mirror(repo_path, remote) , upload(local, remote_path) }, with a LocalTransport used in tests that maps the remote base to a local temp dir (it runs the same commands without ssh). The real SshTransport is unit-tested only for argument construction.
  • Docs: a new section in docs/backup.md, "Standby replication", covering setting up the standby account (restricted key with restrict in authorized_keys), the known_hosts pin, and a failover runbook pointing at architecture §8.5.