Tahinli/forge
public0 watching
★ 0
| 1 | # Git forge research notes |
| 2 | |
| 3 | Compiled 2026-10-15 from 10 parallel research passes. Sources are linked inline. |
| 4 | Items marked *(unverified)* came from secondary sources or aggregators. Many 2026 CVE IDs below came from aggregator sites, so check them against GHSA/NVD before citing them. |
| 5 | |
| 6 | --- |
| 7 | |
| 8 | ## 1. Smart HTTP |
| 9 | |
| 10 | - **Endpoints:** |
| 11 | - `GET /info/refs?service=git-{upload,receive}-pack` |
| 12 | - `POST /git-upload-pack` |
| 13 | - `POST /git-receive-pack` |
| 14 | - **Content types:** |
| 15 | - advertisement: `application/x-git-<svc>-advertisement` |
| 16 | - request: `application/x-git-<svc>-request` |
| 17 | - result: `application/x-git-<svc>-result` |
| 18 | - Always send `Cache-Control: no-cache`. |
| 19 | - <https://git-scm.com/docs/gitprotocol-http> |
| 20 | - **Advertisement body:** a pkt-line `# service=git-upload-pack`, then a flush `0000`, then the output of `git upload-pack --stateless-rpc --advertise-refs`. |
| 21 | - **Protocol v2:** pass the `Git-Protocol` request header through to the `GIT_PROTOCOL` env var. With v2, clients go straight to POST. <https://git-scm.com/docs/gitprotocol-v2> |
| 22 | - **Gzip:** request bodies may arrive with `Content-Encoding: gzip`. Inflate them before piping to git. |
| 23 | - **Streaming:** stream request and response bodies to and from the git child process. Never buffer a whole push. |
| 24 | - Clients use chunked POSTs above `http.postBuffer` (1 MiB). |
| 25 | - GitLab moved git traffic out of Rails into Workhorse because the app server's 60s timeout killed long requests. **Git routes must have no request timeout and no body-size middleware.** <https://docs.gitlab.com/development/workhorse/handlers/> |
| 26 | - **Keepalive:** `uploadpack.keepAlive` defaults to 5s, so proxies must not apply idle timeouts to git routes. |
| 27 | - **Auth:** answer 401 with `WWW-Authenticate: Basic`. The username is ignored or must match, and the password is a personal access token. Anonymous read is allowed only for public repos. |
| 28 | - **Dumb protocol:** skip it, and set `http.getanyfile=false`. |
| 29 | - **Partial clone:** enable `uploadpack.allowFilter` and `uploadpack.allowRefInWant`. |
| 30 | |
| 31 | ## 2. SSH |
| 32 | |
| 33 | - The client runs `git-upload-pack '/path'`, `git-receive-pack '/path'` or `git-upload-archive '/path'`. |
| 34 | - **Allow-list these exact command names.** |
| 35 | - Parse the single-quoted argument strictly: reject quotes and newlines inside it. |
| 36 | - Never use a shell. |
| 37 | - <https://git-scm.com/docs/gitprotocol-pack> |
| 38 | - **Environment:** accept only `GIT_PROTOCOL` from env requests. Gogs CVE-2024-39930 was an RCE through env handling in its built-in SSH server. <https://github.com/gogs/gogs/security/advisories/GHSA-vm62-9jw3-c8w3> |
| 39 | - **Key lookup** must be an exact, case-sensitive match on the key blob. Gitea had an auth bypass from a case-insensitive SQLite `LIKE`. <https://github.com/go-gitea/gitea/security/advisories/GHSA-8pwg-hcgm-54cp> |
| 40 | - **Key types:** disable `ssh-rsa` with SHA-1 signatures (OpenSSH 8.8 default). russh still lists it as supported, so turn it off explicitly. |
| 41 | - **Timeouts:** set handshake and auth timeouts. Gogs had a DoS from stalled SSH handshakes. |
| 42 | - **russh:** version 0.64.x. `server::Handler` provides `auth_publickey`, `env_request`, `exec_request` and `data`; `Session` provides `data`, `eof`, `exit_status_request` and `close`. |
| 43 | - **Alternative:** OpenSSH with `AuthorizedKeysCommand` plus `AcceptEnv GIT_PROTOCOL`. |
| 44 | |
| 45 | ## 3. Push handling and hooks |
| 46 | |
| 47 | - **pre-receive:** |
| 48 | - Runs once per push and gets `<old> <new> <ref>` lines on stdin. |
| 49 | - A non-zero exit rejects the **whole** push. **All policy goes here.** |
| 50 | - New objects sit in a **quarantine** directory (`GIT_QUARANTINE_PATH`, `GIT_OBJECT_DIRECTORY`, `GIT_ALTERNATE_OBJECT_DIRECTORIES`). They aren't visible to other processes until the hook accepts, so any git command the hook runs must inherit those env vars. |
| 51 | - **update:** runs per ref and can reject a single ref. |
| 52 | - **post-receive:** runs after the push and cannot reject. Use it to enqueue events: webhooks, PR updates, cache invalidation, mirror pushes. |
| 53 | - **reference-transaction** (git 2.28+) can veto any ref update, including ones that don't come from a push. |
| 54 | - **Hook design (Gitea pattern):** a tiny hook binary calls back into the app's internal API. |
| 55 | - Use a unix socket plus a shared secret. |
| 56 | - Pass user ID, repo ID and protocol through env vars set when spawning receive-pack. |
| 57 | - <https://github.com/go-gitea/gitea/blob/main/cmd/hook.go> |
| 58 | - **Force-push detection:** `git rev-list --max-count=1 <old> ^<new>`; any output means a force push. |
| 59 | - **Deletion:** `new == 0000…`. |
| 60 | - **Server config:** |
| 61 | - `receive.fsckObjects=true` |
| 62 | - `receive.maxInputSize` |
| 63 | - `receive.advertisePushOptions=true` |
| 64 | - `core.hooksPath` pointing to our own read-only directory. Users never write hooks or config. |
| 65 | - **Gitea gap to avoid:** its branch protection with required checks only applies to PR merges, so direct pushes bypass it. |
| 66 | - **AGit flow:** `git push origin HEAD:refs/for/main -o title=…` creates a PR without a fork. It goes through the `proc-receive` hook with `receive.procReceiveRefs`. |
| 67 | |
| 68 | ## 4. Security checklist |
| 69 | |
| 70 | - **Argument injection:** |
| 71 | - Always put `--end-of-options` before user-supplied revisions and `--` before paths (git 2.24+). |
| 72 | - Reject refs and names starting with `-`. |
| 73 | - Build every git command through one builder that enforces this. |
| 74 | - Examples: Gitea CVE-2022-42968; go-git CVE-2025-21613. |
| 75 | - **Symlinks and paths:** check **every** path component for symlinks when rendering, archiving or writing, not just the last one. Gogs CVE-2025-8110 is in the CISA KEV catalog of actively exploited bugs. |
| 76 | - **Untrusted repos (imports and mirrors):** |
| 77 | - Never recurse into submodules. |
| 78 | - Never trust repo-local config. |
| 79 | - Keep `protocol.file.allow=user`. |
| 80 | - Run as a dedicated service user. |
| 81 | - Keep the git binary patched: CVE-2024-32002 and CVE-2025-48384 are hook-execution bugs via submodules. |
| 82 | - **SSRF** (migrations, mirrors, webhooks): resolve DNS, pin the resulting IP, block private and link-local ranges, and **re-check after every redirect**. Block `file://`. Gitea and Gogs had several 2026 CVEs here *(unverified IDs)*. This matters most for us, because mirrors are a core feature. |
| 83 | - **XSS:** |
| 84 | - Serve raw and user-uploaded content from a **separate origin** (e.g. `raw.<domain>`) with `X-Content-Type-Options: nosniff` and a strict CSP. |
| 85 | - Use `Content-Disposition: attachment` for HTML and SVG. |
| 86 | - Sanitize rendered markdown (ammonia) and allow only http(s) and mailto links. |
| 87 | - **DoS:** |
| 88 | - Cap request bodies, diff sizes, blame and log time, and archive sizes. |
| 89 | - Use only linear-time regex (Rust's `regex` crate is fine). Gitea had a ReDoS in CODEOWNERS matching. |
| 90 | - **Private repos:** return **404 for both missing and forbidden**, everywhere, including any go-get style endpoints. |
| 91 | - **Tokens:** store hashes and compare in constant time. Never put tokens in URLs or logs. |
| 92 | - **Names:** |
| 93 | - NFKC-normalize and case-fold, with a unique index on the folded form. |
| 94 | - Keep a reserved-names list. |
| 95 | - Don't allow names to end in `.git` or `.atom`. |
| 96 | |
| 97 | ## 5. Storage and maintenance |
| 98 | |
| 99 | - **Layout by ID**, e.g. `repos/ab/cd/<id>.git` (GitLab hashed storage). Renames are then DB-only, with a redirect table for old web and git URLs. GitHub redirects git remotes after a rename. |
| 100 | - **Maintenance:** |
| 101 | - Schedule `git maintenance` (incremental-repack plus loose-objects), or `git repack --geometric=2 --write-midx`. |
| 102 | - Write a commit-graph with changed-path Bloom filters; these make `git log -- path` fast. |
| 103 | - Write bitmaps for clone speed. |
| 104 | - **Hidden refs:** set `uploadpack.hideRefs` / `transfer.hideRefs` for internal refs. |
| 105 | - **Atomic ref updates:** use `git update-ref --stdin`, which updates all refs in one transaction. |
| 106 | - **Tree view "last commit per file":** this is expensive and needs a cache keyed by (commit, path). GitLab measured a ~22× gain from caching it. |
| 107 | - **Archives:** cache them, but don't promise stable checksums. Bytes change across git versions. |
| 108 | - **Forks:** dedupe with alternates or object pools, but pools must never be pruned. Leaking objects between private forks is a risk. **Defer dedup and do full copies at first.** |
| 109 | - **Backups:** snapshot the refs first, then the objects, or briefly pause writes. Back up the **instance secret key** separately; Gitea loses 2FA and mirror credentials without it. |
| 110 | |
| 111 | ## 6. Pull requests |
| 112 | |
| 113 | - **PR refs:** store the head as `refs/pull/N/head` in the base repo and update it on every push to the head branch. |
| 114 | - **Mergeability:** |
| 115 | - Check with `git merge-tree --write-tree base head`, which needs no worktree and reports conflicts. |
| 116 | - Recompute whenever the base or head moves. |
| 117 | - A test-merge ref `refs/pull/N/merge` is optional. |
| 118 | - **Diffs:** show **three-dot** diffs (from the merge base). |
| 119 | - **Merge strategies:** |
| 120 | - merge commit, built with `commit-tree` from the merge-tree result |
| 121 | - squash |
| 122 | - rebase |
| 123 | - fast-forward only |
| 124 | - **Review comments:** store the commit SHA, path, line, side and the diff hunk context. When the anchored code changes, mark the comment **outdated**; don't delete it. |
| 125 | - **Large diffs:** collapse or lazy-load big files, detect binary files, and use rename detection (`-M`). |
| 126 | - **Range-diff:** `git range-diff` gives "what changed since the last review". |
| 127 | - **Gerrit/AGit:** an alternative workflow that pushes to `refs/for/<branch>`. |
| 128 | |
| 129 | ## 7. Git library split |
| 130 | |
| 131 | - gix 0.89 *(unverified version)*: |
| 132 | - **Use it in-process for:** |
| 133 | - object, tree and blob reads |
| 134 | - rev-parse and rev-walk |
| 135 | - tree and blob diff |
| 136 | - rename detection |
| 137 | - tar and zip archives |
| 138 | - pack encode and decode |
| 139 | - basic ref transactions |
| 140 | - **Missing in gix:** |
| 141 | - the upload-pack/receive-pack server side |
| 142 | - signature verification |
| 143 | - commit-graph writing and bitmaps |
| 144 | - **Partial in gix:** |
| 145 | - blame (no rename tracking; "not competitive with git") |
| 146 | - merge workflow |
| 147 | - **Shell out to `git` for:** serving fetch and push, merges and conflicts, blame, `verify-commit`/`verify-tag`, and maintenance. |
| 148 | - **Hot-path fallback:** a pool of long-running `git cat-file --batch-command` processes, one per repo. Gitea and Gitaly both use this. |
| 149 | - **Skip git2**, since it adds a C dependency. |
| 150 | - **Minimum git version:** 2.38+ for `merge-tree --write-tree`. In practice, track the latest release for security fixes. |
| 151 | |
| 152 | ## 8. Features and crates |
| 153 | |
| 154 | - **Markdown:** comrak (GFM, alerts, footnotes, math) followed by ammonia for sanitization. Rewrite relative links and images to repo paths. |
| 155 | - **Syntax highlighting:** syntect (mature), or arborium/tree-sitter. Tree-sitter grammars must match the core version. |
| 156 | - **Language stats:** tokei (not equivalent to Linguist). |
| 157 | - **Code search:** a trigram index (zoekt-style) or tantivy. Defer it. |
| 158 | - **Commit signatures:** `git verify-commit` with `gpg.format=ssh` and a generated `allowedSignersFile` built from users' SSH signing keys. |
| 159 | - **Webhooks:** |
| 160 | - Sign with an `X-Hub-Signature-256`-style HMAC-SHA256 over the raw body. |
| 161 | - Send a delivery ID header and use a 10s timeout. |
| 162 | - Keep a retry queue and a delivery log. |
| 163 | - **LFS:** the batch API (`POST /objects/batch`) plus the locks API. Storage can go to `in` or `object_store`. |
| 164 | - **CODEOWNERS:** the last matching pattern wins. |
| 165 | - **Mailmap:** apply `.mailmap` in display. |
| 166 | - **Avatars:** Gravatar uses SHA-256 of the email. |
| 167 | - **Feeds:** atom_syndication. |
| 168 | |
| 169 | ## 9. Ecosystem and portability |
| 170 | |
| 171 | - **Woodpecker CI** supports Gitea and Forgejo forges, and so do Renovate and `tea`. **A Gitea-compatible API subset would make them all work for us almost for free.** |
| 172 | - **Forgejo Actions:** its runner is a fork of `act` with a Connect/JSON runner protocol *(unverified)*. It's explicitly not identical to GitHub Actions. |
| 173 | - **Minimum CI surface:** webhooks plus a commit-status API. |
| 174 | - **F3 (Friendly Forge Format):** JSON plus attachments for issues, PRs, labels and milestones, with a Forgejo driver. A good export target. <https://f3.forgefriends.org/> |
| 175 | - **Radicle and git-bug** store issues and patches **inside git**, so they travel with every clone and mirror. |
| 176 | - **Software Heritage** has a Gitea/Forgejo lister ("Add Forge Now"). Being Gitea-API-compatible would also let it archive us automatically. |
| 177 | - **Forgejo federation:** ForgeFed; v16 (2026-07) shipped federated stars as a first step. |
| 178 | |
| 179 | ## 10. Operations and product |
| 180 | |
| 181 | - **Permission model:** Read, Triage, Write, Maintain and Admin, plus collaborators. Org teams come later. |
| 182 | - **One authorization function** must serve web, API, smart HTTP, SSH and hooks. |
| 183 | - **Ghost user:** on account deletion, reassign issues and comments to a ghost user; never hard-delete them. |
| 184 | - **Personal access tokens:** expiry (default 30 days), scopes, stored hashed. |
| 185 | - **Deploy keys:** read-only by default, with write as an opt-in. |
| 186 | - **Archived repos:** fully read-only. |
| 187 | - **Mirrors:** |
| 188 | - Pull mirrors run on an interval, with a minimum interval. |
| 189 | - Push mirrors keep encrypted credentials. |
| 190 | - Run them after each push with a backoff. |
| 191 | - **Rate limiting:** per user and per IP, with writes weighted heavier. |
| 192 | - **Open registration** (if ever): require email verification and rate-limit new accounts. Codeberg sees bot accounts opening hundreds of issues per hour. |
| 193 | - **Takedowns:** only relevant if the instance becomes public. |
| 194 | |
| 195 | --- |
| 196 | |
| 197 | # Part 2: Data safety and reliability |
| 198 | |
| 199 | ## 11. Lessons from real incidents |
| 200 | |
| 201 | - **SourceHut (Jan 2024):** a DDoS plus upstream null-routing took down the primary datacenter. They migrated to Amsterdam over about a week. The standby DB was under 30s behind, but **storage backups were up to 12h old and object storage had no off-site copy**. <https://sourcehut.org/blog/2024-01-19-outage-post-mortem/> |
| 202 | - **OVH Strasbourg fire (2021):** backups kept **at the same site** burned with the primary. Courts awarded customers damages. |
| 203 | - **Provider account risk:** Hetzner can terminate with 30 days' notice, and customers have reported notices that never arrived. OVH has reports of suspensions without warning *(unverified reviews)*. The protection is diversity: different provider, **payment method, contact email** and jurisdiction. |
| 204 | - **Git corruption** happens on power loss or storage hiccups, typically as zero-length loose objects. GitLab's runbook says to run `git fsck` after any unclean shutdown. |
| 205 | |
| 206 | ## 12. How big hosts replicate, and what fits us |
| 207 | |
| 208 | - **GitHub Spokes:** 3 replicas, with each write committed by a majority under a lock. Repos re-replicate after a failure. This needs consensus and is overkill for us. |
| 209 | - **GitLab Gitaly Cluster (Praefect):** has documented known issues, such as snapshot restores that can't be reconciled and hook races. It's a cautionary example of complexity. |
| 210 | - **Raft (openraft):** pre-1.0. Overkill here. |
| 211 | - **Split-brain** is the main danger of more than one writer. **Fence the old primary before promoting the standby.** DNS and health checks alone don't enforce a single writer. |
| 212 | - **Never rsync or `cp` a live repo or a live DB.** |
| 213 | |
| 214 | ## 13. Git replication approach (single writer) |
| 215 | |
| 216 | 1. After post-receive, a replication job takes a **per-repo lock** and runs `git push --mirror` to each replica over SSH. It records failures and retries with backoff. |
| 217 | 2. **Hourly reconcile:** compare a ref checksum (sha256 of sorted `git for-each-ref --format='%(objectname) %(refname)'`) between primary and replica. On a mismatch, re-mirror and run `git fsck`. |
| 218 | 3. **Daily `git bundle create --all`** for each repo to off-provider storage, then `git bundle verify`. |
| 219 | 4. Replicas get a weekly `git fsck --full` from cron. `git maintenance` has no fsck task. |
| 220 | |
| 221 | ## 14. Metadata (Turso) safety |
| 222 | |
| 223 | - Turso supports only `synchronous` OFF or FULL. **Use WAL with FULL.** |
| 224 | - **Litestream is not verified on Turso files.** Turso says mixed SQLite/Turso access across processes is unsupported. Run a spike before relying on it. |
| 225 | - **Turso Sync/CDC** is documented only against Turso Cloud, and self-hosted sync is unconfirmed. Don't depend on it. |
| 226 | - **Verified path:** `VACUUM INTO` (supported by Turso) produces consistent snapshots on a schedule, which are then shipped to the replica and to backups. |
| 227 | - **Integrity checks:** `quick_check` daily, `integrity_check` plus `foreign_key_check` weekly. Alert on any result other than `ok`. |
| 228 | - **Instance secret key** (it encrypts mirror credentials, 2FA secrets and similar): back it up separately and escrow it offline. |
| 229 | |
| 230 | ## 15. Backups (3-2-1-1-0) |
| 231 | |
| 232 | - **3-2-1-1-0:** 3 copies, 2 media, 1 offsite, 1 **immutable**, 0 verify errors. |
| 233 | - **Immutable storage:** Backblaze B2 Object Lock in **compliance mode** (nobody can shorten it), Cloudflare R2 bucket locks, or Wasabi compliance mode. |
| 234 | - **Survive a compromised server:** the server holds **no delete credentials**. Either a separate machine pulls the backups, or the server pushes with append-only keys (rest-server `--append-only`, or BorgBase append-only). |
| 235 | - **Tools:** |
| 236 | - restic: mature, with `check --read-data-subset` for verification. |
| 237 | - rustic: Rust and restic-compatible, but newer. |
| 238 | - borg: 1.4 is stable. |
| 239 | - **Encryption keys:** add a second restic key and store the keys offline. A lost key means lost data. |
| 240 | - **Monitoring:** a dead-man switch via healthchecks.io pings with `/start` and `/fail`. |
| 241 | - **Restore drill:** monthly, automated, restoring to a fresh VPS. Time it; that time is the real recovery time (RTO). |
| 242 | |
| 243 | ## 16. Storage-level integrity |
| 244 | |
| 245 | - **Filesystem:** |
| 246 | - btrfs (or ZFS with ≥4GB RAM) gives checksums, scrub, snapshots and `send`/`receive`. |
| 247 | - On tiny VPSes, ext4/XFS with tested backups is fine. |
| 248 | - **Git fsync:** set `core.fsync` explicitly to cover loose objects and refs, plus a `core.fsyncMethod`. The default `committed,-loose-object` "risks losing recent work". <https://git-scm.com/docs/git-config> |
| 249 | - **App files:** write a temp file, fsync it, rename it, then fsync the directory. |