forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
32.8 KiB
1# Forge Architecture
2
3Status: milestones 1 and 2 are implemented. Later milestones are outlined here and not built yet.
4"Forge" is a placeholder name. Renaming is a find-and-replace.
5
6Sources: the M1 plan (`~/.claude/plans/purring-prancing-wirth.md`), the interface contract (`docs/m1-interfaces.md`), and the research notes (`docs/research.md`, sections 1-16). Items marked *(unverified)* in the research are not confirmed.
7
8---
9
10## 1. Goals and principles
11
12- **Independence.** Forge is a standalone app. It has no dependency on other projects, no shared code, and no hosted forge in the loop. The owner was banned from GitHub and lost five years of work, so nothing critical may depend on a third party's account decisions.
13- **Data safety first.** Metadata and repositories both have to survive disk loss, a compromised server, and a provider account termination. Section 8 covers this.
14- **Single writer.** One primary instance accepts writes. Replicas and backups are copies. Failover is manual and fenced (section 8.5). There is no consensus layer.
15- **The DB holds metadata.** Users, sessions, keys, tokens, repos, and later issues, PRs, labels, and milestones live in Turso. Issues and PRs are not stored in git.
16- **Git is the source of truth for code.** Objects and refs live in bare repositories on disk. The DB never holds file contents or ref values that git can answer.
17- **Latest dependencies.** Every crate is added with `cargo add` at its latest release and recorded in `[workspace.dependencies]`. Versions are not copied from other projects.
18- **Security by default.** Deny unless allowed. One authorization function. No shell. Private repos return 404. Secrets are hashed or sealed at rest. Section 9 has the checklist.
19
20---
21
22## 2. Stack and why
23
24| Layer | Choice | Why |
25|---|---|---|
26| Web and HTTP | Topcoat 0.10 on hyper | Server-rendered UI and git smart-HTTP in the same app. Topcoat routes accept streaming bodies with no size limit and no default request timeout, which git needs. |
27| Git smart-HTTP | Routes in Topcoat, backed by the `git` binary | No axum. The git child process streams request and response bodies directly. |
28| Metadata DB | Turso | Embedded SQLite-compatible store. Holds all metadata. WAL with `synchronous=FULL` (section 8.3). |
29| Git serving | `git` binary (`upload-pack`, `receive-pack`, merges, blame, verify, maintenance) | Git's own implementation is the only trustworthy server side. |
30| Git reads | `gix` in-process | Object, tree, and blob reads, rev-walk, diffs, rename detection, archives (research §7). |
31| SSH | russh 0.64.x | Built-in SSH server for git over SSH. No OpenSSH dependency. |
32| Hooks | `forge hook` subcommand over a unix socket | Hook policy lives in the app, not in files users can edit. |
33
34Not used: `git2` (adds a C dependency, research §7). Dumb HTTP (disabled, research §1).
35
36---
37
38## 3. Crates and responsibilities
39
40Cargo workspace at the repo root. Edition 2024, resolver 3.
41
42```
43project_git/
44 Cargo.toml .gitignore config/forge.toml deploy/forge.service style/main.scss
45 docs/{research.md, architecture.md, m1-interfaces.md}
46 crates/
47 forge-core/ domain types, store, migrations, config, secret key, authz, names, passwords, tokens, sessions
48 forge-git/ safe git command builder, process spawn and streaming, pkt-line, repo init, managed gitconfig, version check
49 forge-ssh/ russh server: pubkey auth, exec allow-list and parse, GIT_PROTOCOL env, piping, timeouts
50 forge-web/ crate, builds the `forge` binary: Topcoat UI, auth, smart-HTTP routes, internal hook API, `hook` subcommand, admin CLI
51```
52
53Module ownership (from the contract):
54
55- **forge-core**
56 - `error.rs`, `time.rs` (`now_stamp()`, `stamp_after()`), `model.rs` (domain types)
57 - `config.rs` (`Config::load`), `secret.rs` (`load_or_create_key`, `seal`, `open`)
58 - `names.rs` (`normalize_name`, `validate_name`, `RESERVED`), `password.rs` (argon2id)
59 - `token.rs` (`generate_token`, `hash_token`, `constant_time_eq`), `session.rs`
60 - `authz.rs` (`Action`, `Denied`, `authorize`)
61 - `store/mod.rs` (`Store`), `store/migrate.rs` (migration runner)
62 - `migrations/0001_init.sql`
63- **forge-git**
64 - `command.rs` (`GitCommand`), `version.rs`, `pktline.rs`, `repo.rs` (`repo_path`, `init_bare`, `list_branches`)
65 - `setup.rs` (`write_managed_gitconfig`, `write_hook_shims`), `service.rs` (`GitService`, `service_command`)
66- **forge-ssh**
67 - `backend.rs` (`GitBackend` trait), `parse.rs` (`parse_exec`), `server.rs` (`run`, host key load or create, russh `Handler`)
68 - The crate does not know about the DB. It calls `GitBackend` for auth and for preparing the git process.
69- **forge-web**
70 - `main.rs` (arg dispatch: `serve` default, `hook <name>`, `admin create-user`), `boot.rs`, `state.rs`
71 - `auth.rs` (session cookie, `current_user()`, CSRF check, Basic-auth parse, token auth)
72 - `git_http.rs`, `internal.rs` (hook API on unix socket), `hook_cmd.rs`, `ssh_backend.rs` (impl `GitBackend`), `admin_cli.rs`
73 - `views/mod.rs`, `pages/{setup, login, dashboard, repo, new_repo, settings_keys, settings_tokens, settings_sessions, admin_users}.rs`
74
75Domain types (`forge-core::model`): `Id = String` (ULID), `Visibility {Public, Private}`, `Role {Read, Triage, Write, Maintain, Admin}`, plus `User`, `Repo`, `SshKey`, `AccessToken`, `Session`.
76
77Store API (`Store`: async, `Clone`, wraps the Turso connection). `open()` sets WAL, `synchronous=FULL`, `foreign_keys=ON`, and a busy timeout, then runs migrations. It exposes user, session, SSH key, token, repo, collaborator, and push-event methods (the full list is in the contract).
78
79---
80
81## 4. Data model
82
83### 4.1 Milestone 1 tables (`0001_init.sql`)
84
85Conventions: ids are ULID text. Timestamps are UTC RFC 3339 with 9 fractional digits and a `Z` suffix. Booleans are integers 0/1. Foreign keys are enforced.
86
87**`schema_migrations`**
88- `version` (PK). Records applied migrations.
89
90**`users`**
91- `id` TEXT PK (ULID)
92- `username` TEXT NOT NULL (display form)
93- `username_folded` TEXT NOT NULL UNIQUE (NFKC + case-fold)
94- `email` TEXT NULL
95- `password_hash` TEXT NOT NULL (argon2id)
96- `is_admin` INTEGER NOT NULL DEFAULT 0
97- `created_at` TEXT NOT NULL
98
99**`sessions`**
100- `id` TEXT PK
101- `user_id` TEXT NOT NULL REFERENCES `users(id)` ON DELETE CASCADE
102- `token_hash` TEXT NOT NULL UNIQUE (sha256 of the random session token; the raw token is only in the cookie)
103- `csrf_token` TEXT NOT NULL
104- `created_at`, `expires_at`, `last_seen` TEXT NOT NULL
105- `user_agent` TEXT NULL
106- Queries return only unexpired rows.
107
108**`ssh_keys`**
109- `id` TEXT PK
110- `user_id` TEXT NOT NULL REFERENCES `users(id)` ON DELETE CASCADE
111- `title` TEXT NOT NULL
112- `algorithm` TEXT NOT NULL
113- `key_blob` BLOB NOT NULL UNIQUE (raw SSH wire-format public key; exact-match lookup)
114- `fingerprint` TEXT NOT NULL
115- `created_at` TEXT NOT NULL, `last_used_at` TEXT NULL
116
117**`access_tokens`**
118- `id` TEXT PK
119- `user_id` TEXT NOT NULL REFERENCES `users(id)` ON DELETE CASCADE
120- `name` TEXT NOT NULL
121- `token_hash` TEXT NOT NULL UNIQUE (hex sha256; the token is shown once)
122- `scopes` TEXT NOT NULL (serialized list)
123- `expires_at` TEXT NULL (default expiry is 30 days, research §10)
124- `created_at` TEXT NOT NULL, `last_used_at` TEXT NULL
125
126**`repos`**
127- `id` TEXT PK (ULID; determines the on-disk path)
128- `owner_id` TEXT NOT NULL REFERENCES `users(id)`
129- `name` TEXT NOT NULL (display form)
130- `name_folded` TEXT NOT NULL, with `UNIQUE(owner_id, name_folded)`
131- `description` TEXT NOT NULL DEFAULT ''
132- `visibility` TEXT NOT NULL CHECK (`public` or `private`)
133- `default_branch` TEXT NOT NULL
134- `archived` INTEGER NOT NULL DEFAULT 0
135- `created_at` TEXT NOT NULL
136
137**`repo_redirects`**
138- Old owner and name mapped to a repo id, so old web and git URLs keep working after a rename. Exact columns are set in `0001_init.sql`; the contract does not fix them.
139
140**`collaborators`**
141- `repo_id` TEXT NOT NULL REFERENCES `repos(id)` ON DELETE CASCADE
142- `user_id` TEXT NOT NULL REFERENCES `users(id)` ON DELETE CASCADE
143- `role` TEXT NOT NULL CHECK (`read`, `triage`, `write`, `maintain`, `admin`)
144- PK `(repo_id, user_id)`
145
146**`push_events`**
147- `id` TEXT PK
148- `repo_id`, `actor_id` TEXT NOT NULL (foreign keys)
149- `ref_name`, `old_oid`, `new_oid` TEXT NOT NULL
150- `protocol` TEXT NOT NULL CHECK (`http` or `ssh`)
151- `created_at` TEXT NOT NULL
152
153Secrets (`forge.key` is not in the DB). Values sealed with XChaCha20-Poly1305 are stored as `v1:` plus base64(nonce || ciphertext). M1 has no sealed column yet. Mirror credentials and TOTP secrets are the first candidates (M5 and later).
154
155### 4.2 Later tables (outline)
156
157| Table | Milestone | Purpose |
158|---|---|---|
159| `issues` | M3 | Number per repo, title, body, state, author, timestamps |
160| `comments` | M3 | Issue and PR comments, polymorphic by parent, edit history |
161| `labels` | M3 | Per-repo labels, plus an `issue_labels` join |
162| `milestones` | M3 | Per-repo milestones, due date, state |
163| `pulls` | M4 | PR number, base and head refs, head oid, merge state, outdated flag |
164| `reviews` | M4 | Reviewer, state (approve, request changes, comment), commit oid |
165| `review_comments` | M4 | Commit oid, path, line, side, diff hunk, outdated flag (kept, never deleted) |
166| `webhooks` | M5 | Target URL, events, encrypted secret (sealed) |
167| `deliveries` | M5 | Delivery id, attempt, status, request and response, retry state |
168| `mirrors` | M5 | Pull and push mirrors, interval, backoff, sealed credentials |
169| `commit_statuses` | M6 | Commit oid, context, state, target URL, creator |
170| `replication_state` | M5 | Per-repo, per-replica last ref checksum, last success and failure |
171
172---
173
174## 5. Request flows
175
176### 5.1 Web page with session and CSRF
177
1781. The browser sends a request with cookie `__Host-forge_session` (`Secure`, `HttpOnly`, `SameSite=Lax`, `Path=/`).
1792. `auth::current_user()` hashes the cookie value and calls `session_by_token_hash`. No row, or an expired row, means anonymous.
1803. For a valid session, `touch_session` updates `last_seen`. The user is loaded with `user_by_id`.
1814. The page handler builds its view. Repo pages call `authorize(actor, repo, role, Action::Read)` first (section 6).
1825. A POST form includes the session's `csrf_token` as a hidden field. The handler compares it in constant time. A missing or wrong token is rejected before any side effect.
1836. On success, the handler performs the write through `Store` and redirects (PRG).
1847. First run: while `user_count() == 0`, `/setup` creates the first admin. After that the route is closed.
185
186### 5.2 Smart-HTTP fetch and push
187
1881. Client sends `GET /{owner}/{repo}/info/refs?service=git-upload-pack` (or `git-receive-pack`). A trailing `.git` on the repo segment is stripped.
1892. Auth: for upload-pack on a public repo, anonymous is allowed. Otherwise the request needs `Authorization: Basic`. The username is ignored. The password is a `forge_` token. The token is hashed and checked with `token_by_hash`. Failure gives 401 with `WWW-Authenticate: Basic`.
1903. `authorize()` runs for Read (fetch) or Write (push). A private repo the caller cannot see gives 404.
1914. The handler spawns `git <svc> --stateless-rpc --advertise-refs` through `service_command`, with `GIT_PROTOCOL` set from the `Git-Protocol` header. It writes the pkt-line header `# service=git-<svc>`, then `0000`, then streams git's stdout. Response has `Cache-Control: no-cache`.
1925. Client sends `POST /{owner}/{repo}/git-upload-pack` or `/git-receive-pack`. If `Content-Encoding: gzip`, the body is inflated first.
1936. Request body streams to git's stdin and git's stdout streams back. The route has no timeout and no body-size middleware. Nothing buffers a whole push.
1947. For push, receive-pack runs the pre-receive and post-receive hooks (section 5.4). Objects land in quarantine until pre-receive accepts them.
1958. Protocol v2 clients go straight to POST after the GET.
196
197### 5.3 SSH fetch and push
198
1991. Client connects to `ssh_listen` (default `0.0.0.0:2222`). russh runs the handshake under `auth_timeout` and `inactivity_timeout`.
2002. `auth_publickey`: `GitBackend::authenticate_key(algorithm, key_blob)` does an exact byte match on `ssh_keys.key_blob`. `ssh-rsa` with SHA-1 is disabled. On success, `last_used_at` is touched.
2013. Client sends an `env` request. Only `GIT_PROTOCOL` is accepted. Other names are ignored or refused.
2024. Client sends `exec`. `parse_exec` accepts only `git-upload-pack` or `git-receive-pack` with a single-quoted path. Quotes and newlines inside the path are rejected. No shell is used.
2035. `GitBackend::prepare(user_id, service, repo_path, git_protocol)` resolves the repo, runs `authorize()`, and builds the git `Command` through `forge-git`. On `Err(msg)`, the message goes to stderr and the exit status is 1.
2046. The child's stdin and stdout are piped to the SSH channel. For receive-pack, `FORGE_*` env vars are set (section 5.4).
2057. When git exits, the server sends `exit_status` and closes the channel.
206
207### 5.4 Pre-receive and post-receive hooks over the unix socket
208
209Setup at boot: `write_hook_shims()` writes `<data_dir>/hooks/pre-receive` and `post-receive` (mode 0755) as shims that run `exec <forge> hook <name>`. `core.hooksPath` in the managed gitconfig points there. Users cannot write hooks.
210
211Push:
212
2131. The `forge` process spawns receive-pack with `FORGE_INTERNAL_SOCKET`, `FORGE_INTERNAL_TOKEN` (random per boot), `FORGE_ACTOR_ID`, `FORGE_REPO_ID`, and `FORGE_PROTOCOL` (`http` or `ssh`).
2142. Git runs `pre-receive` with `<old> <new> <ref>` lines on stdin. Git has placed the new objects in quarantine (`GIT_QUARANTINE_PATH`, `GIT_OBJECT_DIRECTORY`, `GIT_ALTERNATE_OBJECT_DIRECTORIES`). They are invisible to other processes until the hook accepts.
2153. The `hook pre-receive` client reads stdin and POSTs JSON to `/hook/pre-receive` on `<data_dir>/internal.sock`. Body: `{repo_id, actor_id, protocol, updates:[{old,new,ref}], quarantine_env:{...}}`. Header: `X-Forge-Internal-Token`.
2164. The internal router checks the token in constant time. It then applies policy:
217 - reject pushes to archived repos
218 - reject deleting the default branch
219 - reject force-pushes to the default branch, using `rev-list --max-count=1 <old> ^<new>` with the `quarantine_env` vars set on the command. Any output means a force push.
220 - reject ref names with a leading `-` or containing `..`
2215. The response is `{allow, message}`. The hook prints `message` to stderr and exits 0 if allowed, 1 if not. A non-zero exit rejects the whole push.
2226. On accept, git moves the objects out of quarantine and updates refs. Git updates refs in one transaction.
2237. Post-receive: `hook post-receive` POSTs the same updates to `/hook/post-receive`. The handler calls `record_push_event` for each ref. Replication (M5), webhooks (M5), PR ref updates (M4), and cache invalidation (M2) attach here. Post-receive cannot reject.
2248. Hook failures are fail-closed for pre-receive. If the socket is unreachable, the push is rejected.
225
226### 5.5 User-content domain for raw files (M2)
227
228Raw and user-uploaded files are served from `user_content_base_url` (for example `raw.<domain>`), a separate origin from the app.
229
2301. Request arrives on the user-content host. The path is parsed as owner, repo, ref, and file path.
2312. Repo lookup, then `authorize(actor, repo, role, Action::Read)`. Anonymous is allowed for public repos. A private repo the caller cannot see gives 404.
2323. Every path component is checked for symlinks before reading, not only the last one (research §4).
2334. The blob is read through `gix` and streamed. HTML and SVG are served with `Content-Disposition: attachment`.
2345. Every response carries `X-Content-Type-Options: nosniff` and a strict CSP.
2356. Cookies from the app origin are never sent to this origin. Session auth is not accepted on this host.
236
237---
238
239### 5.6 Response compression
240
241Topcoat's builder-wide compression is off on the public router, so it cannot compress git responses by mistake. `crates/forge-web/src/compression.rs` (`ResponseCompression`) runs instead. It gzips or brotlis a response only when all of these hold:
242
243- the content type is text (HTML, CSS, plain text, JSON, SVG, JavaScript);
244- the body size is known and between 32 bytes and 4 MiB (streamed bodies are never compressed);
245- there is no `Content-Encoding` or `Content-Range` header;
246- the request path is not a git smart-HTTP, LFS, raw, archive or download route (`/info/refs`, `/git-upload-pack`, `/git-receive-pack`, `/info/lfs`, and `raw`, `archive`, `download` segments).
247
248Content types outside the list (`application/x-git-*`, `application/octet-stream`, archives, images) are never compressed. The raw user-content listener has compression off because its ETag and Content-Length must match the stored bytes.
249
250## 6. Authorization model
251
252One function: `forge_core::authz::authorize(actor: Option<&User>, repo: &Repo, collaborator_role: Option<Role>, action: Action) -> Result<(), Denied>`.
253
254- `Action`: `Read`, `Write`, `Admin`.
255- `Denied`: `NotFound` or `Forbidden`.
256- Callers: web pages, smart-HTTP, SSH `prepare`, the user-content domain, and hooks. Nothing else decides access.
257
258Rules (in order):
259
2601. The owner or an instance admin has all actions.
2612. A public repo allows `Read` to anyone, including anonymous.
2623. A private repo with no role for the actor returns `NotFound`.
2634. Role `Read` or `Triage` allows `Read`.
2645. Role `Write` or `Maintain` allows `Read` and `Write`.
2656. Role `Admin` allows all actions.
2667. The actor can see the repo but lacks the right: `Forbidden`.
2678. Archived repos deny `Write` with `Forbidden`.
268
269Status mapping: `NotFound` becomes 404 everywhere, including git and SSH. `Forbidden` becomes 403 on web and after authentication. For git over HTTP, an unauthenticated request that needs auth gets 401 with the Basic challenge. A private repo is never revealed by a 403. Any future go-get-style endpoints follow the same rule.
270
271Roles: `Read`, `Triage`, `Write`, `Maintain`, `Admin` (per repo, from `collaborators`), plus instance admin. Org teams come later (research §10).
272
273---
274
275## 7. Git layer
276
277### 7.1 Command builder (`forge-git::GitCommand`)
278
279Rules:
280
281- No shell. Ever. `tokio::process::Command` only.
282- `env_clear()`, then `PATH=/usr/bin:/bin`, `HOME=<data_dir>`, `GIT_CONFIG_NOSYSTEM=1`, `GIT_CONFIG_GLOBAL=<data_dir>/gitconfig`.
283- `arg()` takes only `&'static str` flags, which are trusted.
284- `git_dir()` places `--git-dir=<path>` before the subcommand.
285- `user_revs()` appends `--end-of-options`, then the revisions. It rejects any revision starting with `-`.
286- `paths()` appends `--` then the paths.
287- `env()` takes only `&'static str` keys.
288- User input reaches git only through `GitCommand`.
289- Repo-local config is not trusted (research §4). Submodules are never recursed.
290
291### 7.2 Managed gitconfig (`<data_dir>/gitconfig`)
292
293Written at boot by `write_managed_gitconfig()`. Keys:
294
295- `core.hooksPath=<data_dir>/hooks`
296- `receive.fsckObjects=true`, `transfer.fsckObjects=true`
297- `core.fsync` and `core.fsyncMethod` set explicitly. The default `committed,-loose-object` risks losing recent work (research §16).
298- `uploadpack.allowFilter=true`, `uploadpack.allowRefInWant=true`
299- `receive.advertisePushOptions=true`
300- `protocol.file.allow=user` (research §4)
301- Later: `uploadpack.hideRefs` and `transfer.hideRefs` for internal refs (M4 `refs/pull/*` as needed)
302
303### 7.3 Repo path layout
304
305`<data_dir>/repos/<id[0..2]>/<id[2..4]>/<id>.git`, where `<id>` is the lowercased ULID. Created with `git init --bare` through the builder. Renames are DB-only, because the path depends on the id, not the name.
306
307### 7.4 Maintenance and caches (M2 and later)
308
309- `git maintenance` with incremental repack and loose-objects, or `git repack --geometric=2 --write-midx`.
310- Commit-graph with changed-path Bloom filters (makes `git log -- path` fast).
311- Bitmaps for clone speed.
312- A pool of long-running `git cat-file --batch-command` processes, one per repo, for hot reads (research §7).
313- "Last commit per file" cache keyed by `(commit, path)` for tree views (research §5).
314- Archive cache, with no promise of stable checksums across git versions.
315- Forks: full copies first. Alternates or object pools are deferred, and pools must never be pruned (research §5).
316- Ref updates use `git update-ref --stdin` for atomicity.
317
318### 7.5 Minimum git version
319
320- Boot calls `require_min_version(env, (2,38,0))`. Boot fails if git is older.
321- 2.38 is required for `merge-tree --write-tree` (M4).
322- In practice, track the latest git release for security fixes. Keep it patched (CVE-2024-32002 and CVE-2025-48384 are submodule hook-execution bugs, research §4).
323
324---
325
326## 8. Data safety and reliability
327
328### 8.1 Topology
329
330- **Primary** (writes, serves users).
331- **Standby** at a different provider. It receives git mirrors, DB snapshots, and bundles.
332- **Immutable backups** at a third location and provider.
333- Provider diversity covers provider, payment method, contact email, and jurisdiction. Research §11 notes that account termination can happen without warning. Lesson from SourceHut (2024): the standby was under 30s behind, but backups were up to 12h old with no off-site copy. Lesson from OVH Strasbourg (2021): backups at the same site burned with the primary.
334
335### 8.2 Git replication (single writer)
336
3371. After post-receive, a replication job takes a per-repo lock and runs `git push --mirror` to each replica over SSH. Failures are recorded and retried with backoff. Push mirrors run after each push (M5).
3382. Hourly reconcile: compare a ref checksum (sha256 of sorted `git for-each-ref --format='%(objectname) %(refname)'`) between primary and replica. On mismatch, re-mirror and run `git fsck`.
3393. Daily `git bundle create --all` per repo to off-provider storage, then `git bundle verify`.
3404. Weekly `git fsck --full` on replicas from cron. `git maintenance` has no fsck task.
3415. Never `rsync` or `cp` a live repo or a live DB.
342
343### 8.3 Metadata DB (Turso)
344
345- Turso supports only `synchronous` OFF or FULL. Use WAL with FULL.
346- Litestream is not verified on Turso files, and Turso says mixed SQLite and Turso access across processes is unsupported. Spike before relying on it (open question 2).
347- Turso Sync and CDC are documented only for Turso Cloud. Do not depend on them for self-hosting.
348- Verified path: `VACUUM INTO` snapshots every 5-15 minutes. Ship each snapshot to the standby and to backups. Keep a JSON or F3 export as a second format (F3: friendly forge format, JSON plus attachments, research §9).
349- Integrity: `PRAGMA quick_check` daily. `integrity_check` plus `foreign_key_check` weekly. Alert on any result other than `ok`.
350
351### 8.4 Integrity, backups, and drills
352
353- Backups follow 3-2-1-1-0: 3 copies, 2 media, 1 offsite, 1 immutable, 0 verify errors.
354- Immutable storage: Backblaze B2 Object Lock in compliance mode, Cloudflare R2 bucket locks, or Wasabi compliance mode.
355- The server holds no delete credentials. Either a separate machine pulls the backups, or the server pushes with append-only keys (rest-server `--append-only`, or BorgBase append-only).
356- Tools: restic (mature, `check --read-data-subset`), rustic (Rust, restic-compatible, newer), borg 1.4 (stable).
357- Keys: add a second restic key and store keys offline. Lost key means lost data.
358- Instance key `forge.key` (encrypts sealed values) is backed up separately and escrowed offline. Without it, sealed secrets are gone.
359- Restore drill monthly: automated restore to a fresh VPS. Time it. That time is the real RTO.
360- Dead-man switch: healthchecks.io pings with `/start` and `/fail` for each backup and integrity job. A missed ping alerts.
361- Filesystem: btrfs (or ZFS with at least 4 GB RAM) for checksums, scrub, and snapshots. ext4 or XFS with tested backups is acceptable on small VPSes.
362- App files: write temp file, fsync it, rename it, fsync the directory.
363- After any unclean shutdown, run `git fsck` (GitLab runbook, research §11).
364
365### 8.5 Manual failover with fencing
366
3671. Stop writes on the primary (block the HTTP and SSH listeners, or put the instance in read-only mode).
3682. Fence the old primary: revoke its replication and backup credentials and make sure it cannot accept writes, so it cannot come back as a second writer. Do this before promotion.
3693. Confirm the standby's last `VACUUM INTO` snapshot and last ref reconcile. Record the data-loss window.
3704. Restore the DB snapshot and repos on the standby. Run `quick_check` and `fsck`.
3715. Promote the standby. Switch DNS. DNS and health checks alone do not enforce a single writer, so fencing in step 2 is required.
3726. Record the incident and the actual RTO.
373
374---
375
376## 9. Security checklist
377
378Do:
379
380- Put `--end-of-options` before user revisions and `--` before paths. Reject names and refs that start with `-`.
381- Route all git calls through `GitCommand`.
382- Check every path component for symlinks when rendering, archiving, or writing.
383- Use exact, case-sensitive key lookup (Gitea auth bypass, research §2).
384- Disable `ssh-rsa` with SHA-1. Set handshake and auth timeouts (Gogs DoS, research §2).
385- Accept only `GIT_PROTOCOL` from SSH env requests (Gogs CVE-2024-39930, research §2).
386- Store tokens and session tokens as sha256 hashes. Compare in constant time. Show tokens once.
387- Keep tokens out of URLs and logs.
388- Return 404 for both missing and forbidden private repos.
389- Normalize names with NFKC and case-fold. Keep a unique index on the folded form. Keep a reserved-names list. Reject names ending in `.git` or `.atom`.
390- Serve raw and user content from a separate origin with `nosniff` and a strict CSP. Use `Content-Disposition: attachment` for HTML and SVG.
391- Sanitize rendered markdown with comrak then ammonia. Allow only http, https, and mailto links.
392- Use linear-time regex only (Rust `regex` crate).
393- Cap request bodies on non-git routes, diff sizes, blame and log time, and archive sizes. Git routes are the exception, with streaming and no cap.
394- For SSRF (mirrors, webhooks, migrations): resolve DNS, pin the IP, block private and link-local ranges, re-check after every redirect, and block `file://`.
395- Run as a dedicated service user. Keep the git binary patched.
396- Set `receive.fsckObjects`.
397- Rate limit per user and per IP, with writes weighted heavier. Gate open registration behind email verification.
398
399Don't:
400
401- Don't use a shell, `sh -c`, or string-built commands.
402- Don't use dumb HTTP or `http.getanyfile`.
403- Don't recurse into submodules or trust repo-local config.
404- Don't let users write hooks or config.
405- Don't hard-delete users. Reassign their issues and comments to a ghost user (M3).
406- Don't let the browser use the app origin for raw content.
407- Don't bypass branch rules through a non-PR path (the Gitea gap: required checks only applied to PR merges, research §3).
408- Don't rely on Turso Sync or Litestream until spiked.
409
410---
411
412## 10. Milestones
413
414### M1: clone and push work (this plan)
415
416- Scaffold the workspace (`cargo add`, latest versions), config template, `.gitignore`, systemd unit, SCSS build.
417- Local accounts with argon2id. First user becomes admin, via `admin create-user` or `/setup` while zero users exist. Registration is `closed` by default.
418- Sessions, CSRF on every POST, `__Host-` cookie. Access tokens (`forge_`) and SSH keys with exact blob match.
419- Repo create with the ID-based path. Visibility, collaborators, archive flag.
420- Smart HTTP (`info/refs`, upload-pack, receive-pack, gzip, streaming) and SSH (exec allow-list, `GIT_PROTOCOL`, piping, exit status).
421- Pre-receive policy (archived, default-branch delete and force-push, bad ref names) and post-receive `push_events`. Minimal UI: setup, login, dashboard, new repo, repo page, settings, admin users, `/healthz`.
422- Tests: unit tests in core and git; end-to-end `git_e2e.rs` over HTTPS and SSH, with negative cases (bad token, cross-user private 404, forced default-branch push rejected, unknown SSH command rejected, traversal rejected, POST without CSRF rejected).
423
424### M2: repo browsing and raw domain (implemented)
425
426Built:
427
428- **forge-git** (`browse.rs`, gix, blocking, called through `spawn_blocking`): ref listing, revision resolution to a full OID, splitting `<ref>/<path>`, tree and blob reads from the object database, commit reads, and path validation. Symlinks are shown as their target text and never resolved.
429- **forge-git** (`history.rs`, git CLI, OIDs only): paginated `log`, `commit_diff` against the first parent with rename detection and the limits in section 10 below, one-walk `last_commits`, and `archive_command`.
430- **forge-core** (`raw_token.rs`): `sign_raw` / `verify_raw` for private raw links (10 minute TTL).
431- **forge-web** `render/`: `markdown.rs` (comrak, then ammonia, relative links rewritten to the repo tree, blob, and raw bases) and `highlight.rs` (syntect, class-based output, served at `/assets/highlight.css`).
432- **forge-web** `raw.rs`: the user-content listener, bound by `boot.rs` on `user_content_listen`, with explicit routes only.
433- **forge-web** pages: `repo` (home and README), `tree`, `blob`, `refs` (branches and tags), `commits`, `commit`, `archive`.
434
435Routes on the app origin:
436
437- `/{owner}/{repo}`, `/{owner}/{repo}/tree/{*rest}`, `/{owner}/{repo}/blob/{*rest}`
438- `/{owner}/{repo}/commits/{*rest}` (`?page=`, 50 per page), `/{owner}/{repo}/commit/{oid}`
439- `/{owner}/{repo}/archive/{ref}.tar.gz` and `.zip`
440- `/{owner}/{repo}/branches`, `/{owner}/{repo}/tags`
441
442Routes on the user-content origin: `GET /{owner}/{repo}/{*rest}`, where `rest` is `<ref>/<path>`.
443
444Limits: blob view 1 MiB; highlighting skipped above 512 KiB or 20,000 lines; raw files up to 8 MiB held in memory, larger ones streamed; diffs 300 files, 2,000 lines per file, 20,000 lines total; last-commit lookup is one bounded `git log` walk (2,000 commits, 3 second timeout), with no cache.
445
446Known gaps:
447
448- Footnote ids are not prefixed `user-content-` (heading ids are).
449- No Dockerfile grammar in syntax highlighting.
450- A background commit-graph and maintenance run after push is being added. It is not in place yet.
451- The app CSP header is computed once per process (`OnceLock`), so a change to `user_content_base_url` needs a restart.
452
453- Tree, blob, commits, and diff views through `gix`, with `git log` and `git diff` as fallback where needed.
454- README rendering with comrak, then ammonia. Relative links and images rewritten to repo paths.
455- Syntax highlighting (syntect, or tree-sitter with matching core versions).
456- Raw file serving on `user_content_base_url`, with the headers in section 5.5.
457- Symlink-safe path resolution, and `cat-file --batch-command` pool.
458- Caches: last commit per file, archives. Maintenance schedule and commit-graph.
459
460### M3: issues
461
462- Issues with per-repo numbers, state, author, timestamps, and edit history.
463- Comments on issues (polymorphic `comments` table), with ghost-user reassignment on account deletion.
464- Labels and milestones, with per-repo scope.
465- Issue list and detail pages, filters, and permissions through `authorize()`.
466- Markdown bodies sanitized as in M2.
467
468### M4: pull requests
469
470- PR head stored as `refs/pull/N/head` in the base repo, updated on every push to the head branch in post-receive.
471- Mergeability checked with `git merge-tree --write-tree`, recomputed when base or head moves. Optional `refs/pull/N/merge`.
472- Three-dot diffs from the merge base, rename detection (`-M`), binary detection, lazy-loaded large files.
473- Reviews (approve, request changes, comment) and review comments anchored to commit, path, line, side, and hunk. Comments are marked outdated, not deleted.
474- Merge strategies: merge commit (`commit-tree` from merge-tree output), squash, rebase, fast-forward only.
475- `range-diff` for "what changed since the last review." AGit (`refs/for/<branch>`) is considered later.
476
477### M5: webhooks, push mirrors, replication, backups, export
478
479- Webhooks: HMAC-SHA256 signature over the raw body, delivery id header, 10s timeout, retry queue, delivery log.
480- Pull mirrors on an interval with a minimum; push mirrors with sealed credentials, run after each push with backoff. SSRF rules from section 9 apply.
481- Git replication per section 8.2: per-repo lock, `push --mirror`, hourly reconcile, daily bundles, weekly `fsck --full` on replicas.
482- DB: `VACUUM INTO` snapshots every 5-15 minutes, integrity checks, JSON or F3 export.
483- Backups per section 8.4: immutable tier, append-only push, restore drills, dead-man alerts.
484- Manual failover runbook per section 8.5, tested once end to end.
485
486### M6: custom CI runner and commit-status API
487
488- Commit-status API (`commit_statuses`: commit oid, context, state, target URL, creator). Required checks enforced on push and PR merge, not only on PR merge (research §3).
489- Webhooks from M5 drive CI job dispatch.
490- Custom runner: a separate process with a narrow protocol, that reports status back through the API. Runner credentials are scoped tokens.
491- Consider a Gitea-compatible API subset later, which would let Woodpecker, Renovate, and `tea` work (research §9). Decide when the API surface is known.
492- Log retention and secret masking for job output.
493
494---
495
496## 11. Open questions
497
4981. **Project name.** "forge" is a placeholder. Renaming is a find-and-replace, but pick it before public URLs and crate names are used.
4992. **Does Litestream work with Turso files?** Unverified. Turso says mixed SQLite and Turso access across processes is unsupported. Run a spike before relying on it for replication. Fallback is the `VACUUM INTO` path in section 8.3.
5003. **Hosting providers.** Primary, standby, and backup providers must differ in provider, payment method, contact email, and jurisdiction (section 8.1). Candidates are not chosen. Confirm after reviewing the provider risk notes in research §11 (some are unverified).