forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
19.9 KiB
1# M6 CI (contract between workers)
2
3**Decided with the user (don't re-open):**
4- Workflow file `.forge/ci.toml`. **No YAML anywhere.**
5- Jobs run in **rootless podman**.
6- The runner is a **separate `forge-runner` binary that pulls jobs** over HTTPS. The forge never connects to a runner.
7- v1 has **secrets, artifacts and live logs**. No cache.
8
9**Global rules:**
10- No JS and no inline styles.
11- No unwrap in non-test code.
12- Tests use `tempfile::TempDir`.
13- Never put cargo target dirs or big files in /tmp.
14- New crates go in at their latest version with `cargo add`, recorded in the root `[workspace.dependencies]`.
15- User values come after `--end-of-options` when passed to git.
16
17## 0. Crates
18- **New `crates/forge-ci`** (library; deps: serde, toml, serde_json, globset, thiserror). It is shared by forge-web and forge-runner, and has **no** turso, topcoat or tokio dependency.
19- **New `crates/forge-runner`** (binary). Depends on forge-ci, reqwest (rustls, stream), tokio, clap, serde, toml, tempfile, sha2 and tracing. **Not** forge-core or forge-web.
20- Add both to the root `members`.
21
22## 1. `forge-ci` (worker W1)
23```rust
24// config.rs : parse .forge/ci.toml
25pub struct CiConfig { pub on: Triggers, pub jobs: BTreeMap<String, JobSpec> } // job names: ^[a-z0-9][a-z0-9_-]{0,39}$, max 20 jobs
26pub struct Triggers { pub push: Vec<String> /* branch globs, e.g. "main", "release/*" */, pub tags: Vec<String> /* tag globs */,
27 pub pull_request: bool, pub manual: bool /* default true */ }
28pub struct JobSpec { pub image: String, pub run: String, pub needs: Vec<String>, pub secrets: Vec<String> /* ^[A-Z_][A-Z0-9_]{0,63}$ */,
29 pub env: BTreeMap<String, String>, pub artifacts: Vec<String> /* relative globs, no "..", no leading "/" */,
30 pub runs_on: Vec<String> /* runner labels, all must match; default [] */, pub timeout_minutes: u32 /* default 60, 1..=360 */,
31 pub network: bool /* default true */ }
32pub fn parse_config(text: &str) -> Result<CiConfig, ConfigError> // deny_unknown_fields; validates names, needs refer to existing jobs, no cycles, image non-empty without whitespace, run non-empty, ≤ 64 KiB file
33pub enum Event<'a> { Push { branch: &'a str }, Tag { tag: &'a str }, PullRequest, Manual }
34impl CiConfig { pub fn triggered_by(&self, ev: &Event) -> bool; pub fn job_order(&self) -> Vec<&str> /* topological, stable by name */ }
35
36// proto.rs : runner <-> forge JSON (serde, snake_case)
37pub struct RegisterRequest { pub token: String, pub name: String, pub labels: Vec<String> }
38pub struct RegisterResponse { pub runner_id: String, pub runner_secret: String }
39pub struct ClaimRequest { pub labels: Vec<String> } // POST, long-poll ≤ 30s; 204 = nothing
40pub struct JobAssignment { pub job_id: String, pub job_token: String, pub repo: String /* owner/name */, pub clone_url: String,
41 pub commit: String, pub git_ref: String, pub job_name: String, pub image: String, pub run: String,
42 pub env: BTreeMap<String,String>, pub secrets: BTreeMap<String,String>, pub artifacts: Vec<String>,
43 pub timeout_minutes: u32, pub network: bool }
44pub struct HeartbeatResponse { pub cancel: bool }
45pub struct FinishRequest { pub outcome: Outcome, pub exit_code: Option<i32>, pub message: Option<String> }
46pub enum Outcome { Success, Failure, Cancelled, Error }
47
48// mask.rs
49pub struct Masker // new(secrets: impl IntoIterator<Item=String>); ignores values shorter than 4 chars; mask(&str) -> String replaces every
50 // occurrence (and the base64 standard encoding of each value) with "***"; must handle a value split across two
51 // chunks: `MaskStream` with push(&[u8]) -> Vec<u8> that holds back max_len-1 bytes, and finish() -> Vec<u8>
52```
53Unit tests cover every validation error, glob triggers, topological order, cycle detection, and masking across chunk boundaries.
54
55## 2. Schema and store (migration `0024_ci`, worker W2)
56```sql
57CREATE TABLE ci_runners (id TEXT PRIMARY KEY, name TEXT NOT NULL, labels TEXT NOT NULL /* json */, secret_hash TEXT NOT NULL UNIQUE,
58 created_at TEXT NOT NULL, last_seen_at TEXT, disabled INTEGER NOT NULL DEFAULT 0);
59CREATE TABLE ci_registration_tokens (token_hash TEXT PRIMARY KEY, created_by TEXT NOT NULL REFERENCES users(id),
60 created_at TEXT NOT NULL, expires_at TEXT NOT NULL, used_at TEXT);
61CREATE TABLE ci_runs (id TEXT PRIMARY KEY, repo_id TEXT NOT NULL REFERENCES repos(id), number INTEGER NOT NULL,
62 trigger TEXT NOT NULL /* push|tag|pull_request|manual */, git_ref TEXT NOT NULL, commit_oid TEXT NOT NULL, pull_number INTEGER,
63 actor_id TEXT NOT NULL, state TEXT NOT NULL /* waiting_approval|queued|running|success|failure|cancelled|error */,
64 config_error TEXT, created_at TEXT NOT NULL, finished_at TEXT, UNIQUE (repo_id, number));
65CREATE TABLE ci_jobs (id TEXT PRIMARY KEY, run_id TEXT NOT NULL REFERENCES ci_runs(id) ON DELETE CASCADE, name TEXT NOT NULL,
66 spec TEXT NOT NULL /* JobSpec json */, state TEXT NOT NULL /* blocked|queued|running|success|failure|cancelled|error|skipped */,
67 runner_id TEXT, job_token_hash TEXT, exit_code INTEGER, message TEXT, queued_at TEXT, started_at TEXT, finished_at TEXT,
68 heartbeat_at TEXT, cancel_requested INTEGER NOT NULL DEFAULT 0, log_bytes INTEGER NOT NULL DEFAULT 0, UNIQUE (run_id, name));
69CREATE INDEX ci_jobs_queue ON ci_jobs(state, queued_at);
70CREATE TABLE ci_artifacts (id TEXT PRIMARY KEY, job_id TEXT NOT NULL REFERENCES ci_jobs(id) ON DELETE CASCADE, name TEXT NOT NULL,
71 size INTEGER NOT NULL, sha256 TEXT NOT NULL, created_at TEXT NOT NULL, UNIQUE (job_id, name));
72CREATE TABLE ci_secrets (repo_id TEXT NOT NULL REFERENCES repos(id), name TEXT NOT NULL, sealed BLOB NOT NULL,
73 created_at TEXT NOT NULL, updated_at TEXT NOT NULL, PRIMARY KEY (repo_id, name));
74```
75`crates/forge-core/src/ci.rs` holds the models. `crates/forge-core/src/store/ci.rs` holds the functions below; they follow the read-pool rules in `~/.cache/forge-orch/read-pool-rules.md`, with pure SELECTs on `read_conn()`.
76```rust
77pub enum RunState { WaitingApproval, Queued, Running, Success, Failure, Cancelled, Error } // as_str/FromStr
78pub enum JobState { Blocked, Queued, Running, Success, Failure, Cancelled, Error, Skipped } // as_str/FromStr, is_terminal()
79pub struct CiRunner { id, name, labels: Vec<String>, created_at, last_seen_at: Option<String>, disabled: bool }
80pub struct CiRun { id, repo_id, number: i64, trigger: String, git_ref, commit_oid, pull_number: Option<i64>, actor_id, state: RunState,
81 config_error: Option<String>, created_at, finished_at: Option<String> }
82pub struct CiJob { id, run_id, name, spec_json: String, state: JobState, runner_id: Option<String>, exit_code: Option<i64>,
83 message: Option<String>, queued_at, started_at, finished_at, heartbeat_at: Option<String>, cancel_requested: bool, log_bytes: i64 }
84pub struct CiArtifact { id, job_id, name, size: i64, sha256, created_at }
85pub struct NewJob<'a> { pub name: &'a str, pub spec_json: &'a str, pub needs: &'a [String] }
86
87// runners
88create_registration_token(created_by, ttl_hours: u32) -> Result<String /*plaintext, shown once*/> // stores sha256
89register_runner(token_plain, name, labels: &[String]) -> Result<(CiRunner, String /*runner secret plaintext*/)> // token single-use, not expired; Invalid otherwise
90runner_by_secret(secret_plain) -> Result<Option<CiRunner>> // constant-time compare via hash lookup; also updates last_seen_at
91list_runners() / set_runner_disabled(id, bool) / delete_runner(id)
92// runs and jobs
93create_run(repo_id, trigger, git_ref, commit_oid, pull_number: Option<i64>, actor_id, needs_approval: bool, jobs: &[NewJob], config_error: Option<&str>) -> Result<CiRun>
94 // number = max+1 per repo, in one transaction. Jobs with no needs start Queued (or Blocked when needs_approval), others Blocked.
95 // With config_error: no jobs, state Error.
96approve_run(run_id) -> Result<()> // WaitingApproval -> Queued; needless jobs -> Queued
97claim_job(runner: &CiRunner) -> Result<Option<(CiJob, CiRun, String /*job token plaintext*/)>>
98 // oldest Queued job whose spec.runs_on ⊆ runner.labels, in a transaction; sets Running, runner_id, started_at, heartbeat_at, token hash; run -> Running
99job_by_token(job_id, token_plain) -> Result<Option<CiJob>> // only while Running
100heartbeat(job_id) -> Result<bool /*cancel_requested*/>
101add_log_bytes(job_id, n: i64) -> Result<()>
102finish_job(job_id, state: JobState, exit_code: Option<i64>, message: Option<&str>) -> Result<CiRun>
103 // terminal; then unblock dependants (all needs Success -> Queued; any need not Success -> Skipped, cascading);
104 // run state recomputed: any running/queued/blocked -> Running; else Failure if any failure/error, Cancelled if any cancelled, else Success
105request_cancel(run_id) -> Result<()> // Queued/Blocked jobs -> Cancelled at once; Running jobs get cancel_requested = 1
106rerun(run_id, actor_id) -> Result<CiRun> // new run with the same commit, ref, trigger and job specs
107reap_lost_jobs(older_than: &str /*stamp*/) -> Result<Vec<CiJob>> // Running jobs with heartbeat_at < older_than -> Error "runner lost"
108run_by_number(repo_id, n) / run_by_id(id) / list_runs(repo_id, page, per_page) -> (Vec<CiRun>, total) / jobs_for_run(run_id) / job_by_id(id)
109add_artifact(job_id, name, size, sha256) / list_artifacts(job_id) / artifact(job_id, name)
110purge_old(logs_before: &str, artifacts_before: &str) -> Result<(Vec<String> /*job ids whose logs to delete*/, Vec<String> /*artifact ids*/)>
111// secrets (sealed with the instance key like mirror secrets)
112set_ci_secret(repo_id, name, value: &[u8]) / delete_ci_secret(repo_id, name) -> Result<bool> / list_ci_secret_names(repo_id) -> Result<Vec<(String, String /*updated_at*/)>>
113ci_secret_values(repo_id, names: &[String]) -> Result<BTreeMap<String, String>> // only for claim; missing names omitted
114```
115- `delete_repo` deletes the ci_runs (cascade) and ci_secrets rows.
116- User deletion reassigns `actor_id` and `created_by` to the ghost user.
117
118## 3. forge-web: scheduling (worker W3), `crates/forge-web/src/ci/{mod.rs, schedule.rs, reaper.rs}`
119- `pub async fn on_ref_update(state, repo, actor_id, refname, new_oid)` is called from `post_update::after_ref_update` for each updated `refs/heads/*` or `refs/tags/*` (not deletions; not pull mirrors pulling? **yes, pull mirrors also trigger**, with the system actor).
120 - It reads `.forge/ci.toml` at `new_oid` (forge-git blob read, capped at 64 KiB), parses it, and checks `triggered_by`.
121 - It creates the run with jobs in `job_order`, then sets a commit status `ci/<job>` = pending for each job, with `target_url` = the job page.
122 - A parse error creates a run in state Error with `config_error` and a status `ci/config` = error.
123 - No file means nothing happens.
124- `pub async fn on_pull_event(state, repo, pull, actor_id)` runs on PR open, synchronize and reopen (hook into the same places that emit the `pull_request` webhook).
125 - It reads the config at the PR **head** commit, with event `PullRequest`.
126 - **If the head repo is a fork, or the author doesn't have Write on the base repo:** `needs_approval = true`, the state is WaitingApproval, and the statuses are pending with the description "Waiting for approval".
127- `pub async fn manual_run(state, repo, branch, actor_id) -> Result<CiRun>` runs only when `manual` is true.
128- **Statuses:** whenever a job changes state, `crate::ci::sync_status(state, run, job)` sets `ci/<job>`:
129 - queued/blocked/running → pending;
130 - success → success;
131 - failure → failure;
132 - error/cancelled → error;
133 - skipped → error, "skipped".
134- **Reaper task, started at boot:**
135 - every 30s it runs `reap_lost_jobs(now - 120s)` and syncs the statuses;
136 - daily it runs `purge_old` with `[ci] log_retention_days` (default 90) and `artifact_retention_days` (default 30), and deletes the files.
137- **`AppState.ci_waker: Arc<Notify>`** is notified whenever a job becomes Queued, so long-polling claims wake.
138- **Config `[ci]`:**
139 - `enabled = true`
140 - `max_log_mb = 16`
141 - `max_artifact_mb = 512`
142 - `max_artifacts_per_job = 50`
143 - `log_retention_days = 90`
144 - `artifact_retention_days = 30`
145 - `registration_token_hours = 24`
146
147## 4. forge-web: runner API (worker W4), `crates/forge-web/src/ci/api.rs`
148All responses are JSON (`forge_ci::proto`). Runner auth is `Authorization: Bearer <runner_secret>`. Job auth is `Authorization: Bearer <job_token>` on `/jobs/{id}/…`, checked with `job_by_token`.
149- `POST /api/v1/ci/runners/register` takes RegisterRequest and returns RegisterResponse.
150- `POST /api/v1/ci/runners/claim` takes ClaimRequest. It loops `claim_job`; if nothing is found, it awaits `ci_waker` or 25s, then tries once more. It returns 200 with a JobAssignment, or 204.
151 - The labels in the request are ignored; the stored labels are used.
152 - Disabled runners get 403.
153 - Secrets come from `ci_secret_values(repo, spec.secrets)`, **except for runs whose trigger is pull_request from a fork** (always empty there).
154 - `clone_url` is the repo's HTTPS clone URL.
155- `POST /api/v1/ci/jobs/{id}/logs`: the body is a raw text chunk (≤ 1 MiB), with header `X-Forge-Log-Offset: <byte offset>`.
156 - It appends to `<data_dir>/ci/logs/<job id>.log` only if the offset equals the current size (idempotent retry: an equal end returns 200 without writing; a mismatch returns 409 with `X-Forge-Log-Size`).
157 - The server **masks again** with `Masker` built from the job's secret values before writing.
158 - The cap is `max_log_mb`. Past it, the server writes "[log truncated]" once and drops the rest.
159 - It calls `add_log_bytes`.
160- `POST /api/v1/ci/jobs/{id}/heartbeat` returns HeartbeatResponse (`cancel_requested`).
161- `PUT /api/v1/ci/jobs/{id}/artifacts/{name}` streams the raw body to the BlobStore (`crate::blobs`, key `ci/<artifact id>`), with a sha256 and the `max_artifact_mb` cap.
162 - The name must be `[A-Za-z0-9._+-]{1,128}` and must not start with ".".
163 - At most `max_artifacts_per_job`.
164- `POST /api/v1/ci/jobs/{id}/finish` takes FinishRequest and calls `finish_job`. It syncs statuses for this job and for every job that became skipped or queued, and notifies `ci_waker`.
165- **Git read with a job token:** smart-HTTP `authenticate` in `git_http.rs` also accepts Basic auth with username `ci-job` and password `<job_id>:<job_token>`. That gives read-only access to exactly that job's repo while the job is Running. Every write is refused.
166- Compression is excluded for `/api/v1/ci/` paths (add the path to `compression.rs`).
167
168## 5. forge-web UI: runs (worker W5), `crates/forge-web/src/pages/ci_runs.rs`, `ci_job.rs`
169- **"CI" repo tab.**
170 - `/{o}/{r}/ci` lists runs, newest first: number, state icon, trigger, ref, short commit, actor, and age.
171 - A "Run workflow" form (Write, `manual` allowed) offers a branch select.
172- **`/{o}/{r}/ci/runs/{n}`** shows the jobs table with state, duration and runner name, plus `config_error` in a `<pre>`.
173 - Buttons, each a POST with CSRF: Cancel (Write), Re-run (Write), and "Approve and run" (Write, only in WaitingApproval).
174- **`/{o}/{r}/ci/jobs/{job_id}`** shows the job log: the last 512 KiB of the log file in `<pre>`, with a link to the full log.
175 - **No JS:** while the job is not terminal, the page has `<meta http-equiv="refresh" content="3">`.
176 - Artifacts are listed with download links.
177- **`/{o}/{r}/ci/jobs/{job_id}/log.txt`** serves `text/plain; charset=utf-8`, `X-Content-Type-Options: nosniff`, and **streams live**. It sends the current file, then keeps following it (poll every 1s) until the job is terminal and the file is fully sent. This is the live log; a browser shows it progressively.
178- **`/{o}/{r}/ci/jobs/{job_id}/artifacts/{name}`** downloads as an attachment, with nosniff.
179- Read access follows repo Read. Logs of private repos are never on the user-content domain.
180- The commit list and PR page already show statuses through the status API, so check that the `ci/*` contexts render there.
181
182## 6. forge-web UI: secrets and runners (worker W6)
183- **`/{o}/{r}/settings/secrets`** (Admin):
184 - lists names and when each was updated, never the values;
185 - has an add/update form (name plus value textarea; the value is write-only);
186 - deletes with CSRF;
187 - records audit events `ci_secret_set` and `ci_secret_deleted` (name only).
188- **`/admin/runners`** (instance admin):
189 - lists runners with labels, last seen and disabled state;
190 - has disable/enable and delete buttons;
191 - has "Create registration token", which shows the token once along with the exact command `forge-runner register --url <base_url> --token <token> --name <name> --labels linux`;
192 - records audit events `ci_runner_registered`, `ci_runner_deleted` and `ci_registration_token_created`.
193
194## 7. `forge-runner` (worker W7)
195- **CLI:**
196 - `forge-runner register --url URL --token T --name N [--labels a,b] [--config PATH]` writes the config (TOML, 0600, default `~/.config/forge-runner/runner.toml`): url, runner_id, runner_secret, labels, `work_dir`, `concurrency = 1`, and the `[limits] memory = "4g"`, `cpus = "2"` and `pids = 4096` defaults.
197 - `forge-runner run [--config PATH]` loops claim → execute → finish, with `concurrency` jobs in parallel.
198- **Executor trait:**
199```rust
200#[async_trait-free: use Pin<Box<dyn Future>> or an enum]
201trait Executor { async fn run(&self, job: &JobAssignment, workspace: &Path, log: &mut LogSink, cancel: CancellationToken) -> Outcome+exit code }
202```
203 - **`PodmanExecutor`** runs `podman run --rm --name forge-job-<job_id> --pull=newer --network=<slirp4netns|none> --memory M --cpus C --pids-limit P --security-opt no-new-privileges --cap-drop ALL -v <ws>:/workspace:Z -w /workspace -e NAME... <image> sh -eu -c <run>`.
204 - Env and secret values are passed through the child's environment with `-e NAME` (name only), **never in argv**.
205 - On cancel or timeout it runs `podman kill forge-job-<id>`, then `podman rm -f`.
206 - The podman argv construction is unit-tested.
207 - **`HostExecutor`** exists only for tests, behind `#[cfg(any(test, feature = "test-host-executor"))]`. It runs `sh -eu -c` in the workspace with the env, and has no isolation.
208- **Workspace:** a fresh `work_dir/<job_id>`. Run `git init`, then `git fetch --depth=1 <clone_url> <commit>` with `GIT_CONFIG_*` env carrying `http.extraHeader: Authorization: Basic base64(ci-job:<job_id>:<token>)`, never in argv, with `GIT_TERMINAL_PROMPT=0`. Then `git checkout --detach <commit>`. The workspace is deleted afterwards.
209- **Logs:**
210 - stdout and stderr are merged line-wise and prefixed with nothing.
211 - They pass through `MaskStream` and are batched (≤ 64 KiB or 1s), then POSTed with the offset. On 409 the runner resyncs to `X-Forge-Log-Size`.
212 - The first lines are runner info: runner name, image, and commit.
213- **Heartbeat** every 10s; `cancel: true` triggers the cancellation token. The timeout is `timeout_minutes`.
214- **Artifacts:** after a successful run, expand the globs inside the workspace (the canonicalized path must stay inside it, symlinks must not escape, and only regular files count). Upload each file under its relative path flattened with `/` → `_`, respecting the limits.
215- **Finish:** report `Success` on exit 0, `Failure` on a non-zero exit, `Cancelled`, and `Error` for runner or podman failures, with a message.
216- **Tests:** a mock forge (a small hyper or tokio TCP server in tests) plus `HostExecutor` cover:
217 - claim → logs → artifacts → finish;
218 - masking;
219 - cancel;
220 - timeout;
221 - 409 resync.
222 - Real podman only in an `#[ignore]` test.
223
224## 8. End to end and docs (worker W8, after W1–W7)
225- **`crates/forge-web/tests/ci_e2e.rs`:**
226 1. Boot the forge and register a runner through the API.
227 2. Push a repo with `.forge/ci.toml` (two jobs, with needs, a secret and an artifact).
228 3. Drive the job with forge-runner's library API and `HostExecutor` (forge-runner exposes `lib.rs`).
229 4. Assert the commit statuses, the masked secret in the log, the artifact download, the skip on failure, and that a fork PR waits for approval.
230- **`docs/ci.md`:**
231 - the format reference with examples;
232 - runner install (rootless podman setup: subuid/subgid, `loginctl enable-linger`, a systemd user unit);
233 - the security model (fork PRs, secrets, masking limits);
234 - the runner protocol.