forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
17.2 KiB

Git forge research notes

Compiled 2026-10-15 from 10 parallel research passes. Sources are linked inline. Items marked (unverified) came from secondary sources or aggregators. Many 2026 CVE IDs below came from aggregator sites, so check them against GHSA/NVD before citing them.

1. Smart HTTP

  • Endpoints:
    • GET /info/refs?service=git-{upload,receive}-pack
    • POST /git-upload-pack
    • POST /git-receive-pack
  • Content types:
    • advertisement: application/x-git-<svc>-advertisement
    • request: application/x-git-<svc>-request
    • result: application/x-git-<svc>-result
    • Always send Cache-Control: no-cache.
    • https://git-scm.com/docs/gitprotocol-http
  • Advertisement body: a pkt-line # service=git-upload-pack, then a flush 0000, then the output of git upload-pack --stateless-rpc --advertise-refs.
  • Protocol v2: pass the Git-Protocol request header through to the GIT_PROTOCOL env var. With v2, clients go straight to POST. https://git-scm.com/docs/gitprotocol-v2
  • Gzip: request bodies may arrive with Content-Encoding: gzip. Inflate them before piping to git.
  • Streaming: stream request and response bodies to and from the git child process. Never buffer a whole push.
    • Clients use chunked POSTs above http.postBuffer (1 MiB).
    • GitLab moved git traffic out of Rails into Workhorse because the app server's 60s timeout killed long requests. Git routes must have no request timeout and no body-size middleware. https://docs.gitlab.com/development/workhorse/handlers/
  • Keepalive: uploadpack.keepAlive defaults to 5s, so proxies must not apply idle timeouts to git routes.
  • Auth: answer 401 with WWW-Authenticate: Basic. The username is ignored or must match, and the password is a personal access token. Anonymous read is allowed only for public repos.
  • Dumb protocol: skip it, and set http.getanyfile=false.
  • Partial clone: enable uploadpack.allowFilter and uploadpack.allowRefInWant.

2. SSH

  • The client runs git-upload-pack '/path', git-receive-pack '/path' or git-upload-archive '/path'.
  • Environment: accept only GIT_PROTOCOL from env requests. Gogs CVE-2024-39930 was an RCE through env handling in its built-in SSH server. https://github.com/gogs/gogs/security/advisories/GHSA-vm62-9jw3-c8w3
  • Key lookup must be an exact, case-sensitive match on the key blob. Gitea had an auth bypass from a case-insensitive SQLite LIKE. https://github.com/go-gitea/gitea/security/advisories/GHSA-8pwg-hcgm-54cp
  • Key types: disable ssh-rsa with SHA-1 signatures (OpenSSH 8.8 default). russh still lists it as supported, so turn it off explicitly.
  • Timeouts: set handshake and auth timeouts. Gogs had a DoS from stalled SSH handshakes.
  • russh: version 0.64.x. server::Handler provides auth_publickey, env_request, exec_request and data; Session provides data, eof, exit_status_request and close.
  • Alternative: OpenSSH with AuthorizedKeysCommand plus AcceptEnv GIT_PROTOCOL.

3. Push handling and hooks

  • pre-receive:
    • Runs once per push and gets <old> <new> <ref> lines on stdin.
    • A non-zero exit rejects the whole push. All policy goes here.
    • New objects sit in a quarantine directory (GIT_QUARANTINE_PATH, GIT_OBJECT_DIRECTORY, GIT_ALTERNATE_OBJECT_DIRECTORIES). They aren't visible to other processes until the hook accepts, so any git command the hook runs must inherit those env vars.
  • update: runs per ref and can reject a single ref.
  • post-receive: runs after the push and cannot reject. Use it to enqueue events: webhooks, PR updates, cache invalidation, mirror pushes.
  • reference-transaction (git 2.28+) can veto any ref update, including ones that don't come from a push.
  • Hook design (Gitea pattern): a tiny hook binary calls back into the app's internal API.
  • Force-push detection: git rev-list --max-count=1 <old> ^<new>; any output means a force push.
  • Deletion: new == 0000….
  • Server config:
    • receive.fsckObjects=true
    • receive.maxInputSize
    • receive.advertisePushOptions=true
    • core.hooksPath pointing to our own read-only directory. Users never write hooks or config.
  • Gitea gap to avoid: its branch protection with required checks only applies to PR merges, so direct pushes bypass it.
  • AGit flow: git push origin HEAD:refs/for/main -o title=… creates a PR without a fork. It goes through the proc-receive hook with receive.procReceiveRefs.

4. Security checklist

  • Argument injection:
    • Always put --end-of-options before user-supplied revisions and -- before paths (git 2.24+).
    • Reject refs and names starting with -.
    • Build every git command through one builder that enforces this.
    • Examples: Gitea CVE-2022-42968; go-git CVE-2025-21613.
  • Symlinks and paths: check every path component for symlinks when rendering, archiving or writing, not just the last one. Gogs CVE-2025-8110 is in the CISA KEV catalog of actively exploited bugs.
  • Untrusted repos (imports and mirrors):
    • Never recurse into submodules.
    • Never trust repo-local config.
    • Keep protocol.file.allow=user.
    • Run as a dedicated service user.
    • Keep the git binary patched: CVE-2024-32002 and CVE-2025-48384 are hook-execution bugs via submodules.
  • SSRF (migrations, mirrors, webhooks): resolve DNS, pin the resulting IP, block private and link-local ranges, and re-check after every redirect. Block file://. Gitea and Gogs had several 2026 CVEs here (unverified IDs). This matters most for us, because mirrors are a core feature.
  • XSS:
    • Serve raw and user-uploaded content from a separate origin (e.g. raw.<domain>) with X-Content-Type-Options: nosniff and a strict CSP.
    • Use Content-Disposition: attachment for HTML and SVG.
    • Sanitize rendered markdown (ammonia) and allow only http(s) and mailto links.
  • DoS:
    • Cap request bodies, diff sizes, blame and log time, and archive sizes.
    • Use only linear-time regex (Rust's regex crate is fine). Gitea had a ReDoS in CODEOWNERS matching.
  • Private repos: return 404 for both missing and forbidden, everywhere, including any go-get style endpoints.
  • Tokens: store hashes and compare in constant time. Never put tokens in URLs or logs.
  • Names:
    • NFKC-normalize and case-fold, with a unique index on the folded form.
    • Keep a reserved-names list.
    • Don't allow names to end in .git or .atom.

5. Storage and maintenance

  • Layout by ID, e.g. repos/ab/cd/<id>.git (GitLab hashed storage). Renames are then DB-only, with a redirect table for old web and git URLs. GitHub redirects git remotes after a rename.
  • Maintenance:
    • Schedule git maintenance (incremental-repack plus loose-objects), or git repack --geometric=2 --write-midx.
    • Write a commit-graph with changed-path Bloom filters; these make git log -- path fast.
    • Write bitmaps for clone speed.
  • Hidden refs: set uploadpack.hideRefs / transfer.hideRefs for internal refs.
  • Atomic ref updates: use git update-ref --stdin, which updates all refs in one transaction.
  • Tree view "last commit per file": this is expensive and needs a cache keyed by (commit, path). GitLab measured a ~22× gain from caching it.
  • Archives: cache them, but don't promise stable checksums. Bytes change across git versions.
  • Forks: dedupe with alternates or object pools, but pools must never be pruned. Leaking objects between private forks is a risk. Defer dedup and do full copies at first.
  • Backups: snapshot the refs first, then the objects, or briefly pause writes. Back up the instance secret key separately; Gitea loses 2FA and mirror credentials without it.

6. Pull requests

  • PR refs: store the head as refs/pull/N/head in the base repo and update it on every push to the head branch.
  • Mergeability:
    • Check with git merge-tree --write-tree base head, which needs no worktree and reports conflicts.
    • Recompute whenever the base or head moves.
    • A test-merge ref refs/pull/N/merge is optional.
  • Diffs: show three-dot diffs (from the merge base).
  • Merge strategies:
    • merge commit, built with commit-tree from the merge-tree result
    • squash
    • rebase
    • fast-forward only
  • Review comments: store the commit SHA, path, line, side and the diff hunk context. When the anchored code changes, mark the comment outdated; don't delete it.
  • Large diffs: collapse or lazy-load big files, detect binary files, and use rename detection (-M).
  • Range-diff: git range-diff gives "what changed since the last review".
  • Gerrit/AGit: an alternative workflow that pushes to refs/for/<branch>.

7. Git library split

  • gix 0.89 (unverified version):
    • Use it in-process for:
      • object, tree and blob reads
      • rev-parse and rev-walk
      • tree and blob diff
      • rename detection
      • tar and zip archives
      • pack encode and decode
      • basic ref transactions
    • Missing in gix:
      • the upload-pack/receive-pack server side
      • signature verification
      • commit-graph writing and bitmaps
    • Partial in gix:
      • blame (no rename tracking; "not competitive with git")
      • merge workflow
  • Shell out to git for: serving fetch and push, merges and conflicts, blame, verify-commit/verify-tag, and maintenance.
  • Hot-path fallback: a pool of long-running git cat-file --batch-command processes, one per repo. Gitea and Gitaly both use this.
  • Skip git2, since it adds a C dependency.
  • Minimum git version: 2.38+ for merge-tree --write-tree. In practice, track the latest release for security fixes.

8. Features and crates

  • Markdown: comrak (GFM, alerts, footnotes, math) followed by ammonia for sanitization. Rewrite relative links and images to repo paths.
  • Syntax highlighting: syntect (mature), or arborium/tree-sitter. Tree-sitter grammars must match the core version.
  • Language stats: tokei (not equivalent to Linguist).
  • Code search: a trigram index (zoekt-style) or tantivy. Defer it.
  • Commit signatures: git verify-commit with gpg.format=ssh and a generated allowedSignersFile built from users' SSH signing keys.
  • Webhooks:
    • Sign with an X-Hub-Signature-256-style HMAC-SHA256 over the raw body.
    • Send a delivery ID header and use a 10s timeout.
    • Keep a retry queue and a delivery log.
  • LFS: the batch API (POST /objects/batch) plus the locks API. Storage can go to in or object_store.
  • CODEOWNERS: the last matching pattern wins.
  • Mailmap: apply .mailmap in display.
  • Avatars: Gravatar uses SHA-256 of the email.
  • Feeds: atom_syndication.

9. Ecosystem and portability

  • Woodpecker CI supports Gitea and Forgejo forges, and so do Renovate and tea. A Gitea-compatible API subset would make them all work for us almost for free.
  • Forgejo Actions: its runner is a fork of act with a Connect/JSON runner protocol (unverified). It's explicitly not identical to GitHub Actions.
  • Minimum CI surface: webhooks plus a commit-status API.
  • F3 (Friendly Forge Format): JSON plus attachments for issues, PRs, labels and milestones, with a Forgejo driver. A good export target. https://f3.forgefriends.org/
  • Radicle and git-bug store issues and patches inside git, so they travel with every clone and mirror.
  • Software Heritage has a Gitea/Forgejo lister ("Add Forge Now"). Being Gitea-API-compatible would also let it archive us automatically.
  • Forgejo federation: ForgeFed; v16 (2026-07) shipped federated stars as a first step.

10. Operations and product

  • Permission model: Read, Triage, Write, Maintain and Admin, plus collaborators. Org teams come later.
  • One authorization function must serve web, API, smart HTTP, SSH and hooks.
  • Ghost user: on account deletion, reassign issues and comments to a ghost user; never hard-delete them.
  • Personal access tokens: expiry (default 30 days), scopes, stored hashed.
  • Deploy keys: read-only by default, with write as an opt-in.
  • Archived repos: fully read-only.
  • Mirrors:
    • Pull mirrors run on an interval, with a minimum interval.
    • Push mirrors keep encrypted credentials.
    • Run them after each push with a backoff.
  • Rate limiting: per user and per IP, with writes weighted heavier.
  • Open registration (if ever): require email verification and rate-limit new accounts. Codeberg sees bot accounts opening hundreds of issues per hour.
  • Takedowns: only relevant if the instance becomes public.

Part 2: Data safety and reliability

11. Lessons from real incidents

  • SourceHut (Jan 2024): a DDoS plus upstream null-routing took down the primary datacenter. They migrated to Amsterdam over about a week. The standby DB was under 30s behind, but storage backups were up to 12h old and object storage had no off-site copy. https://sourcehut.org/blog/2024-01-19-outage-post-mortem/
  • OVH Strasbourg fire (2021): backups kept at the same site burned with the primary. Courts awarded customers damages.
  • Provider account risk: Hetzner can terminate with 30 days' notice, and customers have reported notices that never arrived. OVH has reports of suspensions without warning (unverified reviews). The protection is diversity: different provider, payment method, contact email and jurisdiction.
  • Git corruption happens on power loss or storage hiccups, typically as zero-length loose objects. GitLab's runbook says to run git fsck after any unclean shutdown.

12. How big hosts replicate, and what fits us

  • GitHub Spokes: 3 replicas, with each write committed by a majority under a lock. Repos re-replicate after a failure. This needs consensus and is overkill for us.
  • GitLab Gitaly Cluster (Praefect): has documented known issues, such as snapshot restores that can't be reconciled and hook races. It's a cautionary example of complexity.
  • Raft (openraft): pre-1.0. Overkill here.
  • Split-brain is the main danger of more than one writer. Fence the old primary before promoting the standby. DNS and health checks alone don't enforce a single writer.
  • Never rsync or cp a live repo or a live DB.

13. Git replication approach (single writer)

  1. After post-receive, a replication job takes a per-repo lock and runs git push --mirror to each replica over SSH. It records failures and retries with backoff.
  2. Hourly reconcile: compare a ref checksum (sha256 of sorted git for-each-ref --format='%(objectname) %(refname)') between primary and replica. On a mismatch, re-mirror and run git fsck.
  3. Daily git bundle create --all for each repo to off-provider storage, then git bundle verify.
  4. Replicas get a weekly git fsck --full from cron. git maintenance has no fsck task.

14. Metadata (Turso) safety

  • Turso supports only synchronous OFF or FULL. Use WAL with FULL.
  • Litestream is not verified on Turso files. Turso says mixed SQLite/Turso access across processes is unsupported. Run a spike before relying on it.
  • Turso Sync/CDC is documented only against Turso Cloud, and self-hosted sync is unconfirmed. Don't depend on it.
  • Verified path: VACUUM INTO (supported by Turso) produces consistent snapshots on a schedule, which are then shipped to the replica and to backups.
  • Integrity checks: quick_check daily, integrity_check plus foreign_key_check weekly. Alert on any result other than ok.
  • Instance secret key (it encrypts mirror credentials, 2FA secrets and similar): back it up separately and escrow it offline.

15. Backups (3-2-1-1-0)

  • 3-2-1-1-0: 3 copies, 2 media, 1 offsite, 1 immutable, 0 verify errors.
  • Immutable storage: Backblaze B2 Object Lock in compliance mode (nobody can shorten it), Cloudflare R2 bucket locks, or Wasabi compliance mode.
  • Survive a compromised server: the server holds no delete credentials. Either a separate machine pulls the backups, or the server pushes with append-only keys (rest-server --append-only, or BorgBase append-only).
  • Tools:
    • restic: mature, with check --read-data-subset for verification.
    • rustic: Rust and restic-compatible, but newer.
    • borg: 1.4 is stable.
  • Encryption keys: add a second restic key and store the keys offline. A lost key means lost data.
  • Monitoring: a dead-man switch via healthchecks.io pings with /start and /fail.
  • Restore drill: monthly, automated, restoring to a fresh VPS. Time it; that time is the real recovery time (RTO).

16. Storage-level integrity

  • Filesystem:
    • btrfs (or ZFS with ≥4GB RAM) gives checksums, scrub, snapshots and send/receive.
    • On tiny VPSes, ext4/XFS with tested backups is fine.
  • Git fsync: set core.fsync explicitly to cover loose objects and refs, plus a core.fsyncMethod. The default committed,-loose-object "risks losing recent work". https://git-scm.com/docs/git-config
  • App files: write a temp file, fsync it, rename it, then fsync the directory.