forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
9.8 KiB
1# Push and pull mirrors (contract between workers)
2
3The global rules apply:
4- No JS and no inline styles.
5- No unwrap in non-test code.
6- Tests use `tempfile::TempDir`.
7- Never put cargo target dirs or big files in /tmp.
8- New crates are added at their latest version with `cargo add`.
9
10## Model
11- **Pull mirror:** a repo whose refs come from a remote URL. The forge fetches on an interval and on "Sync now".
12 - Pushes to it are rejected in pre-receive with "this repository is a pull mirror".
13 - Issues and PRs still work.
14 - At most one pull mirror per repo. It is created through the new-repo form ("Mirror an existing repository") or added in settings while the repo is empty (no refs at all). Settings refuses the add on a repo with refs, and the form is not offered when a pull mirror already exists.
15- **Push mirror:** a remote that receives our refs after every push (debounced 10s per mirror) and on its interval. A repo can have up to 10.
16- **Wikis are not mirrored.** Pull and push mirrors only sync the code repo. A repo's wiki (`<repo>.wiki.git`) stays on this forge and is not sent to push-mirror remotes or fetched from pull-mirror remotes. Wiki pushes are not refused by the pull-mirror rule.
17- **Refs synced (both directions):** `refs/heads/*` and `refs/tags/*` only, with prune. Pull: `+refs/heads/*:refs/heads/*` and `+refs/tags/*:refs/tags/*`. Push: the same refspecs plus `--prune --force`. **Never** `refs/pull/*`.
18- **URLs:**
19 - Validate with the webhook policy: `webhooks::client::validate_url` and `is_forbidden_ip` on every resolved address before each run, so private and loopback targets are rejected unless `[webhooks] allowed_hosts` allows them.
20 - **DNS pinning:** the worker resolves the host once per run (`check_remote`), checks every address (one forbidden address refuses the host), and pins the first one into git with `-c http.curloptResolve=<host>:<port>:<ip>` (IPv6 in brackets). Git then connects to the checked address and never resolves the name again, which closes the DNS rebinding window. No pin applies to an address literal or an `allowed_hosts` host.
21 - `https` only, unless `[webhooks] allow_http` is on.
22 - No userinfo in the URL.
23 - Never `file://`, `ssh://` or `ext::`.
24- **Credentials:** an optional username and password/token. The secret is sealed with the instance key, the same way webhook secrets are.
25 - They are passed to git **only through environment config**: `GIT_CONFIG_COUNT`, `GIT_CONFIG_KEY_0=http.extraHeader`, `GIT_CONFIG_VALUE_0=Authorization: Basic <b64>`.
26 - **Never** put them in argv, the stored URL, logs, errors or audit entries. Redact the URL to host and path in the UI and in errors.
27- **git safety for remote operations:**
28 - `-c protocol.allow=never -c protocol.https.allow=always` (plus `protocol.http.allow=always` only when `allow_http` is on);
29 - `-c http.followRedirects=false -c credential.helper=`;
30 - `GIT_TERMINAL_PROMPT=0`;
31 - a timeout (default 10 min, killing the process).
32 - Use the existing `GitCommand` builder (managed gitconfig, cleared env) and `--end-of-options` before the URL.
33- **Intervals:** default 8 hours, minimum 10 minutes, `0` = manual only.
34
35## Schema (migration `0017_mirrors.sql`)
36```sql
37CREATE TABLE repo_mirrors (
38 id TEXT PRIMARY KEY,
39 repo_id TEXT NOT NULL REFERENCES repos(id),
40 direction TEXT NOT NULL, -- 'pull' | 'push'
41 url TEXT NOT NULL,
42 username TEXT NOT NULL DEFAULT '',
43 secret_sealed BLOB, -- NULL = no password
44 interval_minutes INTEGER NOT NULL DEFAULT 480,
45 enabled INTEGER NOT NULL DEFAULT 1,
46 next_sync_at TEXT NOT NULL,
47 last_sync_at TEXT, last_ok INTEGER, last_error TEXT,
48 created_at TEXT NOT NULL);
49CREATE INDEX repo_mirrors_due ON repo_mirrors(enabled, next_sync_at);
50CREATE UNIQUE INDEX repo_mirrors_one_pull ON repo_mirrors(repo_id) WHERE direction = 'pull';
51```
52- `Store::delete_repo` (and the repo trash purge) must delete the repo's mirror rows.
53
54## forge-core (`src/mirrors.rs` model plus `store/mirrors.rs`), worker C1
55```rust
56pub enum MirrorDirection { Pull, Push } // as_str / FromStr
57pub struct Mirror { pub id: Id, pub repo_id: Id, pub direction: MirrorDirection, pub url: String, pub username: String,
58 pub has_secret: bool, pub interval_minutes: u32, pub enabled: bool, pub next_sync_at: String,
59 pub last_sync_at: Option<String>, pub last_ok: Option<bool>, pub last_error: Option<String>, pub created_at: String }
60pub struct NewMirror<'a> { pub repo_id: &'a str, pub direction: MirrorDirection, pub url: &'a str, pub username: &'a str,
61 pub secret: Option<&'a [u8]> /* plaintext; the store seals it */, pub interval_minutes: u32 }
62
63create_mirror(&NewMirror) -> Result<Mirror> // Conflict if a second pull mirror; Invalid if over 10 push mirrors
64list_mirrors(repo_id) -> Result<Vec<Mirror>>
65mirror(id) -> Result<Option<Mirror>>
66pull_mirror(repo_id) -> Result<Option<Mirror>> // used by pre-receive and the UI
67update_mirror(id, url, username, secret: SecretUpdate /* Keep | Clear | Set(&[u8]) */, interval_minutes, enabled) -> Result<()>
68delete_mirror(id) -> Result<()>
69mirror_secret(id) -> Result<Option<Vec<u8>>> // unsealed, only for the sync worker
70due_mirrors(now, limit) -> Result<Vec<Mirror>>
71schedule_mirror(id, next_sync_at) -> Result<()> // "sync now" sets it to now
72record_mirror_result(id, ok: bool, error: Option<&str>, next_sync_at) -> Result<()>
73```
74
75## forge-git (`src/mirror.rs`), worker C1
76```rust
77pub struct RemoteAuth { pub username: String, pub secret: Vec<u8> } // Debug impl redacts
78pub struct PinnedHost { pub host: String, pub port: u16, pub ip: IpAddr } // config_pair() -> "http.curloptResolve=host:port:ip"
79pub async fn mirror_fetch(env: &GitEnv, repo: &Path, url: &str, auth: Option<&RemoteAuth>, allow_http: bool, pin: Option<&PinnedHost>, timeout: Duration) -> Result<()>;
80pub async fn mirror_push(env: &GitEnv, repo: &Path, url: &str, auth: Option<&RemoteAuth>, allow_http: bool, pin: Option<&PinnedHost>, timeout: Duration) -> Result<()>;
81```
82- `pin` is the address the worker checked (see DNS pinning above). It is passed as a `-c` pair built by `PinnedHost::config_pair`, which rejects a host that is not a plain DNS name. Its port must match the URL's port (443 or 80 by scheme).
83- Errors carry git's stderr (truncated to 2 KiB) with the URL's userinfo and any `Authorization` text scrubbed.
84- After a pull fetch, set HEAD to the remote's default branch if ours doesn't exist. `git ls-remote --symref <url> HEAD` is part of the same fetch call.
85
86## forge-web, worker C1
87- `crate::mirrors::worker`: a tokio task started at boot, polling every 60s and woken by a `Notify` on "sync now" or a push. It claims due mirrors, with at most 2 running concurrently and never two on the same repo.
88 - **Pull:** snapshot the refs (`git for-each-ref`) before and after, then call `post_update::after_ref_update` for the changed refs with the ghost or system actor (sender null). Webhooks and PR sync then work as for a push.
89 - **Push:** runs `mirror_push`.
90 - Each run records the result, sets `next_sync_at = now + interval` (or far future when the interval is 0), and is audited only on failure transitions (ok → failed).
91- `crate::mirrors::enqueue_push_mirrors(state, repo_id)` is called from `after_ref_update`. It sets every enabled push mirror's `next_sync_at` to now+10s (debounce) and notifies.
92- `check_remote(url, policy) -> Result<Option<PinnedHost>>` does the URL and address checks and returns the pin; `pin_checked(host, port, resolved, policy)` is the pure address check that picks the first address.
93- **Pre-receive:** reject pushes to a repo with a pull mirror.
94
95## forge-web UI, worker C2 (against the API above; while C1 is unfinished, stub nothing and wait or retry the compile)
96- **Add pull mirror to an empty repo** (repo settings → Mirrors, Admin): shown only when the repo has no pull mirror and `load_refs` returns no refs. Its form (URL, username, password/token, interval) posts to the same `/settings/mirrors` route with a hidden `direction=pull`. The handler refuses a repo with refs (redirect with `notice=pull-not-empty`), validates the URL with `check_form`, calls `store.create_mirror` with `Pull`, schedules it now and notifies `mirror_waker` (`notice=pull-created`). It is audited as `mirror_created`, like the push form. Non-admins get 404; archived repos get 403.
97- **New repo form:** a "Mirror an existing repository" section with URL, username, password/token, and interval. On submit, create the repo (empty), create the pull mirror, and schedule it now. The repo page shows a "Mirror of <redacted url>, last synced …" line under the header (all repo pages, via `RepoHead`).
98- **Repo settings → "Mirrors" page** (`/{o}/{r}/settings/mirrors`, Admin):
99 - List the mirrors with direction, redacted URL, interval, enabled flag, last sync, ok/error (error in `<details>`), and "Sync now" (POST, CSRF).
100 - An add-push-mirror form; edit form (the secret is write-only: blank keeps it, plus a "clear password" checkbox); delete.
101 - Converting a pull mirror into a normal repo is "Delete" on the pull mirror row. Confirm text: "Stop mirroring; the repository becomes a normal repository".
102- **Audit variants:** `mirror_created`, `mirror_updated`, `mirror_deleted`, `mirror_sync_failed` (with the redacted URL).
103- **Tests:**
104 - page permissions (including the add-pull POST as non-admin → 404);
105 - add pull mirror allowed on an empty repo, refused on a repo with refs, form shown only when allowed;
106 - pin: config pair for IPv4 and IPv6, the first checked address is pinned, a forbidden address refuses the host, and a fetch to a pinned name that never resolves fails at connect, not at resolve;
107 - a secret never rendered;
108 - SSRF rejection of a loopback URL in the form;
109 - pull mirror end to end with a `file`-less local HTTP smart server, or else allow-list `127.0.0.1` in the test config and use the forge's own HTTP git server as the remote;
110 - the push to a pull mirror rejected.