Push and pull mirrors (contract between workers)

The global rules apply:

• No JS and no inline styles.
• No unwrap in non-test code.
• Tests use tempfile::TempDir.
• Never put cargo target dirs or big files in /tmp.
• New crates are added at their latest version with cargo add.

Model

• Pull mirror: a repo whose refs come from a remote URL. The forge fetches on an interval and on "Sync now".
  • Pushes to it are rejected in pre-receive with "this repository is a pull mirror".
  • Issues and PRs still work.
  • At most one pull mirror per repo. It is created through the new-repo form ("Mirror an existing repository") or added in settings while the repo is empty (no refs at all). Settings refuses the add on a repo with refs, and the form is not offered when a pull mirror already exists.
• Push mirror: a remote that receives our refs after every push (debounced 10s per mirror) and on its interval. A repo can have up to 10.
• Wikis are not mirrored. Pull and push mirrors only sync the code repo. A repo's wiki (<repo>.wiki.git) stays on this forge and is not sent to push-mirror remotes or fetched from pull-mirror remotes. Wiki pushes are not refused by the pull-mirror rule.
• Refs synced (both directions): refs/heads/* and refs/tags/* only, with prune. Pull: +refs/heads/*:refs/heads/* and +refs/tags/*:refs/tags/*. Push: the same refspecs plus --prune --force. Never refs/pull/*.
• URLs:
  • Validate with the webhook policy: webhooks::client::validate_url and is_forbidden_ip on every resolved address before each run, so private and loopback targets are rejected unless [webhooks] allowed_hosts allows them.
  • DNS pinning: the worker resolves the host once per run (check_remote), checks every address (one forbidden address refuses the host), and pins the first one into git with -c http.curloptResolve=<host>:<port>:<ip> (IPv6 in brackets). Git then connects to the checked address and never resolves the name again, which closes the DNS rebinding window. No pin applies to an address literal or an allowed_hosts host.
  • https only, unless [webhooks] allow_http is on.
  • No userinfo in the URL.
  • Never file://, ssh:// or ext::.
• Credentials: an optional username and password/token. The secret is sealed with the instance key, the same way webhook secrets are.
  • They are passed to git only through environment config: GIT_CONFIG_COUNT, GIT_CONFIG_KEY_0=http.extraHeader, GIT_CONFIG_VALUE_0=Authorization: Basic <b64>.
  • Never put them in argv, the stored URL, logs, errors or audit entries. Redact the URL to host and path in the UI and in errors.
• git safety for remote operations:
  • -c protocol.allow=never -c protocol.https.allow=always (plus protocol.http.allow=always only when allow_http is on);
  • -c http.followRedirects=false -c credential.helper=;
  • GIT_TERMINAL_PROMPT=0;
  • a timeout (default 10 min, killing the process).
  • Use the existing GitCommand builder (managed gitconfig, cleared env) and --end-of-options before the URL.
• Intervals: default 8 hours, minimum 10 minutes, 0 = manual only.

Schema (migration 0017_mirrors.sql)

CREATE TABLE repo_mirrors (
  id TEXT PRIMARY KEY,
  repo_id TEXT NOT NULL REFERENCES repos(id),
  direction TEXT NOT NULL,             -- 'pull' | 'push'
  url TEXT NOT NULL,
  username TEXT NOT NULL DEFAULT '',
  secret_sealed BLOB,                  -- NULL = no password
  interval_minutes INTEGER NOT NULL DEFAULT 480,
  enabled INTEGER NOT NULL DEFAULT 1,
  next_sync_at TEXT NOT NULL,
  last_sync_at TEXT, last_ok INTEGER, last_error TEXT,
  created_at TEXT NOT NULL);
CREATE INDEX repo_mirrors_due ON repo_mirrors(enabled, next_sync_at);
CREATE UNIQUE INDEX repo_mirrors_one_pull ON repo_mirrors(repo_id) WHERE direction = 'pull';

• Store::delete_repo (and the repo trash purge) must delete the repo's mirror rows.

forge-core (src/mirrors.rs model plus store/mirrors.rs), worker C1

pub enum MirrorDirection { Pull, Push }      // as_str / FromStr
pub struct Mirror { pub id: Id, pub repo_id: Id, pub direction: MirrorDirection, pub url: String, pub username: String,
  pub has_secret: bool, pub interval_minutes: u32, pub enabled: bool, pub next_sync_at: String,
  pub last_sync_at: Option<String>, pub last_ok: Option<bool>, pub last_error: Option<String>, pub created_at: String }
pub struct NewMirror<'a> { pub repo_id: &'a str, pub direction: MirrorDirection, pub url: &'a str, pub username: &'a str,
  pub secret: Option<&'a [u8]> /* plaintext; the store seals it */, pub interval_minutes: u32 }

create_mirror(&NewMirror) -> Result<Mirror>          // Conflict if a second pull mirror; Invalid if over 10 push mirrors
list_mirrors(repo_id) -> Result<Vec<Mirror>>
mirror(id) -> Result<Option<Mirror>>
pull_mirror(repo_id) -> Result<Option<Mirror>>       // used by pre-receive and the UI
update_mirror(id, url, username, secret: SecretUpdate /* Keep | Clear | Set(&[u8]) */, interval_minutes, enabled) -> Result<()>
delete_mirror(id) -> Result<()>
mirror_secret(id) -> Result<Option<Vec<u8>>>         // unsealed, only for the sync worker
due_mirrors(now, limit) -> Result<Vec<Mirror>>
schedule_mirror(id, next_sync_at) -> Result<()>      // "sync now" sets it to now
record_mirror_result(id, ok: bool, error: Option<&str>, next_sync_at) -> Result<()>

forge-git (src/mirror.rs), worker C1

pub struct RemoteAuth { pub username: String, pub secret: Vec<u8> }   // Debug impl redacts
pub struct PinnedHost { pub host: String, pub port: u16, pub ip: IpAddr }  // config_pair() -> "http.curloptResolve=host:port:ip"
pub async fn mirror_fetch(env: &GitEnv, repo: &Path, url: &str, auth: Option<&RemoteAuth>, allow_http: bool, pin: Option<&PinnedHost>, timeout: Duration) -> Result<()>;
pub async fn mirror_push(env: &GitEnv, repo: &Path, url: &str, auth: Option<&RemoteAuth>, allow_http: bool, pin: Option<&PinnedHost>, timeout: Duration) -> Result<()>;

• pin is the address the worker checked (see DNS pinning above). It is passed as a -c pair built by PinnedHost::config_pair, which rejects a host that is not a plain DNS name. Its port must match the URL's port (443 or 80 by scheme).
• Errors carry git's stderr (truncated to 2 KiB) with the URL's userinfo and any Authorization text scrubbed.
• After a pull fetch, set HEAD to the remote's default branch if ours doesn't exist. git ls-remote --symref <url> HEAD is part of the same fetch call.

forge-web, worker C1

• crate::mirrors::worker: a tokio task started at boot, polling every 60s and woken by a Notify on "sync now" or a push. It claims due mirrors, with at most 2 running concurrently and never two on the same repo.
  • Pull: snapshot the refs (git for-each-ref) before and after, then call post_update::after_ref_update for the changed refs with the ghost or system actor (sender null). Webhooks and PR sync then work as for a push.
  • Push: runs mirror_push.
  • Each run records the result, sets next_sync_at = now + interval (or far future when the interval is 0), and is audited only on failure transitions (ok → failed).
• crate::mirrors::enqueue_push_mirrors(state, repo_id) is called from after_ref_update. It sets every enabled push mirror's next_sync_at to now+10s (debounce) and notifies.
• check_remote(url, policy) -> Result<Option<PinnedHost>> does the URL and address checks and returns the pin; pin_checked(host, port, resolved, policy) is the pure address check that picks the first address.
• Pre-receive: reject pushes to a repo with a pull mirror.

forge-web UI, worker C2 (against the API above; while C1 is unfinished, stub nothing and wait or retry the compile)

• Add pull mirror to an empty repo (repo settings → Mirrors, Admin): shown only when the repo has no pull mirror and load_refs returns no refs. Its form (URL, username, password/token, interval) posts to the same /settings/mirrors route with a hidden direction=pull. The handler refuses a repo with refs (redirect with notice=pull-not-empty), validates the URL with check_form, calls store.create_mirror with Pull, schedules it now and notifies mirror_waker (notice=pull-created). It is audited as mirror_created, like the push form. Non-admins get 404; archived repos get 403.
• New repo form: a "Mirror an existing repository" section with URL, username, password/token, and interval. On submit, create the repo (empty), create the pull mirror, and schedule it now. The repo page shows a "Mirror of , last synced …" line under the header (all repo pages, via RepoHead).
• Repo settings → "Mirrors" page (/{o}/{r}/settings/mirrors, Admin):
  • List the mirrors with direction, redacted URL, interval, enabled flag, last sync, ok/error (error in <details>), and "Sync now" (POST, CSRF).
  • An add-push-mirror form; edit form (the secret is write-only: blank keeps it, plus a "clear password" checkbox); delete.
  • Converting a pull mirror into a normal repo is "Delete" on the pull mirror row. Confirm text: "Stop mirroring; the repository becomes a normal repository".
• Audit variants: mirror_created, mirror_updated, mirror_deleted, mirror_sync_failed (with the redacted URL).
• Tests:
  • page permissions (including the add-pull POST as non-admin → 404);
  • add pull mirror allowed on an empty repo, refused on a repo with refs, form shown only when allowed;
  • pin: config pair for IPv4 and IPv6, the first checked address is pinned, a forbidden address refuses the host, and a fetch to a pinned name that never resolves fails at connect, not at resolve;
  • a secret never rendered;
  • SSRF rejection of a loopback URL in the form;
  • pull mirror end to end with a file-less local HTTP smart server, or else allow-list 127.0.0.1 in the test config and use the forge's own HTTP git server as the remote;
  • the push to a pull mirror rejected.