Tahinli/forge
public0 watching
★ 0
| 1 | # CI |
| 2 | |
| 3 | forge runs CI workflows defined in `.forge/ci.toml` at the root of a repository. Jobs run on |
| 4 | separate **runners**, which are `forge runner` on machines you control. The forge never |
| 5 | connects to a runner. Runners pull jobs over HTTPS. |
| 6 | |
| 7 | What v1 has: jobs with `needs`, secrets, artifacts, live logs, commit statuses, and manual, push, |
| 8 | tag and pull request triggers. What it does not have: a cache, or YAML. The workflow file is TOML. |
| 9 | |
| 10 | Jobs run in a rootless sandbox, with no container tool needed. The runner pulls and unpacks the |
| 11 | image itself, and runs each job in its own container built with libcontainer, the crate behind |
| 12 | youki. Each job gets its own container, its own workspace, and a read-only token. Podman is still |
| 13 | supported as an option (`executor = "podman"`). |
| 14 | |
| 15 | ## Format reference |
| 16 | |
| 17 | The file is limited to 64 KiB. Unknown keys are an error at every level (`deny_unknown_fields`). |
| 18 | A file with a parse error still creates a run, in state `error`, with the message shown on the run |
| 19 | page and a `ci/config` status set to `error`. |
| 20 | |
| 21 | ### `[on]` |
| 22 | |
| 23 | | Key | Type | Default | Meaning | |
| 24 | |----------------|-----------------|---------|---------| |
| 25 | | `push` | array of string | `[]` | Branch globs. A push to a matching `refs/heads/*` starts a run. | |
| 26 | | `tags` | array of string | `[]` | Tag globs. A push of a matching `refs/tags/*` starts a run. | |
| 27 | | `pull_request` | bool | `false` | Start a run when a pull request is opened, updated or reopened. The config is read at the **head** commit. | |
| 28 | | `manual` | bool | `true` | Writers can start a run on an existing branch from the CI tab. | |
| 29 | |
| 30 | ### `[jobs.<name>]` |
| 31 | |
| 32 | There must be at least one job, and at most 20. |
| 33 | |
| 34 | | Key | Type | Default | Rules | |
| 35 | |-------------------|----------------------|---------|-------| |
| 36 | | `image` | string | none | Required. Non-empty, no whitespace. Image reference, pulled by the runner. | |
| 37 | | `run` | string | none | Required. Non-empty. Runs as `sh -eu -c "<run>"` in `/workspace`. | |
| 38 | | `needs` | array of string | `[]` | Job names that must succeed first. Must exist. No cycles. | |
| 39 | | `secrets` | array of string | `[]` | Repo secret names, each `^[A-Z_][A-Z0-9_]{0,63}$`. Exported as env vars. | |
| 40 | | `env` | table of string | `{}` | Plain env vars. Names match the same pattern. A secret with the same name wins. | |
| 41 | | `artifacts` | array of string | `[]` | Relative globs in the workspace. No `..`, no leading `/`. | |
| 42 | | `runs_on` | array of string | `[]` | Runner labels. Every label must be present on the runner. | |
| 43 | | `timeout_minutes` | integer | `60` | Range 1 to 360. | |
| 44 | | `network` | bool | `true` | `false` gives the job an empty network namespace (loopback only). `true` shares the runner host's network namespace. Read [the caveats](#builtin-executor-caveats). | |
| 45 | |
| 46 | Job names match `^[a-z0-9][a-z0-9_-]{0,39}$`. |
| 47 | |
| 48 | The job's workspace is a fresh checkout of the exact commit, at `/workspace`. The job starts in that |
| 49 | directory. |
| 50 | |
| 51 | ### Example |
| 52 | |
| 53 | ```toml |
| 54 | [on] |
| 55 | push = ["main", "release/*"] |
| 56 | tags = ["v*"] |
| 57 | pull_request = true |
| 58 | manual = true |
| 59 | |
| 60 | [jobs.test] |
| 61 | image = "docker.io/library/rust:1" |
| 62 | run = "cargo test --workspace" |
| 63 | timeout_minutes = 30 |
| 64 | |
| 65 | [jobs.package] |
| 66 | image = "docker.io/library/rust:1" |
| 67 | needs = ["test"] |
| 68 | run = """ |
| 69 | cargo build --release |
| 70 | mkdir -p dist |
| 71 | cp target/release/forge dist/ |
| 72 | """ |
| 73 | artifacts = ["dist/*"] |
| 74 | |
| 75 | [jobs.deploy] |
| 76 | image = "docker.io/library/alpine:3" |
| 77 | needs = ["package"] |
| 78 | runs_on = ["linux", "x86_64"] |
| 79 | secrets = ["DEPLOY_KEY"] |
| 80 | env = { TARGET = "staging" } |
| 81 | network = true |
| 82 | run = "sh ./deploy.sh" |
| 83 | ``` |
| 84 | |
| 85 | Artifacts are stored under the flattened name: `dist/forge` is stored as `dist_forge`. |
| 86 | |
| 87 | ### Triggers and glob rules |
| 88 | |
| 89 | - Branch and tag globs use `*` and `**` with `/` as a separator. `*` matches within one path |
| 90 | segment and does **not** cross `/`. `**` matches across segments. |
| 91 | - `release/*` matches `release/1.2`, but not `release/1.2/hotfix`. |
| 92 | - `release/**` matches both. |
| 93 | - `main` matches only `main`. |
| 94 | - Artifact globs follow the same rule. `out/*.txt` matches `out/a.txt`, not `out/sub/a.txt`. |
| 95 | - Jobs whose `needs` fail or are skipped are marked `skipped`. A skipped job gets the status |
| 96 | `ci/<job>` = `error` with the description `skipped`. |
| 97 | |
| 98 | ### Commit statuses |
| 99 | |
| 100 | Every job reports one status on the commit it ran on. Its context is `ci/<job name>`. A run whose |
| 101 | config failed to parse reports `ci/config`. |
| 102 | |
| 103 | | Job state | Status state | Description | |
| 104 | |------------|--------------|-------------| |
| 105 | | queued, blocked, running | pending | | |
| 106 | | success | success | | |
| 107 | | failure | failure | | |
| 108 | | error, cancelled | error | | |
| 109 | | skipped | error | `skipped` | |
| 110 | |
| 111 | Pull request runs from forks, or from authors without write access, wait for approval. Their |
| 112 | statuses are pending with the description `Waiting for approval`. |
| 113 | |
| 114 | ## Security model |
| 115 | |
| 116 | ### Fork pull requests need approval and get no secrets |
| 117 | |
| 118 | A pull request run needs approval when its head repository is a fork, or when the author has no |
| 119 | write access to the base repository. Such a run stays in `waiting_approval`, and no job is queued |
| 120 | until a writer clicks **Approve and run** on the run page. |
| 121 | |
| 122 | Even after approval, a run from a fork gets **no secrets**. The runner receives an empty secret map |
| 123 | for that job, so `$MY_SECRET` is unset. A job from a fork still gets its plain `env` values, and it |
| 124 | runs on a runner like any other job, so review the workflow file in the pull request before you |
| 125 | approve. |
| 126 | |
| 127 | ### Masking limits |
| 128 | |
| 129 | The forge and the runner both mask secret values in logs, replacing each occurrence with `***`. |
| 130 | Read these limits before you rely on masking: |
| 131 | |
| 132 | - Values shorter than 4 characters are **not** masked. A secret of `abc` will appear in the log. |
| 133 | - Masking covers the raw value and its standard base64 encoding. It does **not** cover any other |
| 134 | transformation. A hex, URL-encoded, reversed or truncated copy of a secret is printed as is. |
| 135 | - A value is masked only where the whole value appears in the output. Printing the secret one |
| 136 | character at a time is not caught. |
| 137 | - Masking is applied while the stream is read, so a value split across two output chunks is still |
| 138 | masked. |
| 139 | |
| 140 | Treat job logs as sensitive. Never `echo` a secret, even though it is masked. |
| 141 | |
| 142 | ### Job tokens |
| 143 | |
| 144 | Each claimed job gets its own token. The token: |
| 145 | |
| 146 | - is **read-only**. It can fetch the job's repository, and its wiki (`<repo>.wiki`), through smart |
| 147 | HTTP with the user name `ci-job` and the password `<job id>:<token>`. Every write is refused; |
| 148 | - only works while the job is `running`. Once the job finishes, the token stops working for logs, |
| 149 | artifacts, heartbeat, finish and git reads; |
| 150 | - is masked in the log, along with the secrets. |
| 151 | |
| 152 | ### Secrets at rest |
| 153 | |
| 154 | Secret values are sealed with the instance key (`forge.key`), like mirror credentials. The secrets |
| 155 | page shows only names and the time each was last updated. Values are write-only, and writing a |
| 156 | value is audited by name. |
| 157 | |
| 158 | ## Runner install |
| 159 | |
| 160 | A runner is the `forge` binary, run as `forge runner`. The forge and the runner ship as one |
| 161 | binary, so an operator deploys one file. The default `builtin` executor needs nothing else. |
| 162 | |
| 163 | The `forge-runner` binary is a development and test convenience only. It has the same subcommands, |
| 164 | but operators should use `forge runner`, which the examples below show. |
| 165 | |
| 166 | ### 1. Check the host |
| 167 | |
| 168 | The builtin executor needs: |
| 169 | |
| 170 | - **Linux 5.11 or newer.** Overlayfs inside a user namespace needs it. On an older kernel, jobs |
| 171 | still run, but each one gets a full copy of the image instead of an overlay. |
| 172 | - **Unprivileged user namespaces.** Check that `/proc/sys/user/max_user_namespaces` is above 0. On |
| 173 | distributions that gate them, `kernel.unprivileged_userns_clone` must be 1. |
| 174 | - **cgroup v2 delegation for limits.** The `cpu`, `memory` and `pids` controllers must be delegated |
| 175 | to your user. systemd does this for user sessions on current distributions. Without it, jobs run |
| 176 | without `[limits]`, and each job log gets a `resource limits unavailable` line. |
| 177 | |
| 178 | Run the check: |
| 179 | |
| 180 | ```sh |
| 181 | forge runner check |
| 182 | ``` |
| 183 | |
| 184 | It runs one trivial job through the real sandbox. It prints `ok` or `FAIL` for each requirement, |
| 185 | with the reason, and exits non-zero when a check fails. Pass `--work-dir` to test the filesystem |
| 186 | that your jobs will use. |
| 187 | |
| 188 | ### Subordinate ids |
| 189 | |
| 190 | The builtin executor can give a job a range of uids and gids, so that package managers which |
| 191 | change file ownership (`apt`, `apk add`, `useradd`) work. Inside the job, uid `0` is your user, |
| 192 | and ids `1` to `n` are your subordinate range, where `n` is the smaller of the uid and gid range |
| 193 | sizes, at most 65536. |
| 194 | |
| 195 | It needs: |
| 196 | |
| 197 | - `newuidmap` and `newgidmap`, from the shadow-utils package. They are setuid helpers in |
| 198 | `/usr/bin`. |
| 199 | - An entry for your user in `/etc/subuid` and `/etc/subgid`. The podman command below adds one. |
| 200 | |
| 201 | Set the mode with `[sandbox] userns` in `runner.toml`: |
| 202 | |
| 203 | ```toml |
| 204 | [sandbox] |
| 205 | userns = "auto" # "auto" (the default), "subuid", or "single" |
| 206 | ``` |
| 207 | |
| 208 | - `auto` uses the subordinate range when the host has it. Otherwise it uses your own uid and gid |
| 209 | only (single mode), and logs `sandbox ids: single uid ...` at startup. |
| 210 | - `subuid` requires the range. A host without it fails at startup. |
| 211 | - `single` always uses your own uid and gid only. |
| 212 | |
| 213 | `forge runner check` prints the mode it resolved. In subuid mode it also checks that a job can |
| 214 | `chown` a file in `/tmp` to `1000:1000`. |
| 215 | |
| 216 | ### Optional: podman |
| 217 | |
| 218 | Set `executor = "podman"` in `runner.toml` to run jobs with rootless podman instead. Install podman |
| 219 | with your distribution's package manager, for example `sudo dnf install podman`. |
| 220 | |
| 221 | Rootless podman needs subordinate user and group ranges for your user: |
| 222 | |
| 223 | ```sh |
| 224 | grep "^$USER:" /etc/subuid /etc/subgid |
| 225 | ``` |
| 226 | |
| 227 | If either line is missing, add a range (it must not overlap any existing one): |
| 228 | |
| 229 | ```sh |
| 230 | sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 "$USER" |
| 231 | podman system migrate |
| 232 | ``` |
| 233 | |
| 234 | Check that podman runs without root: |
| 235 | |
| 236 | ```sh |
| 237 | podman run --rm docker.io/library/alpine:3 echo ok |
| 238 | ``` |
| 239 | |
| 240 | ### 2. Keep the runner alive after logout |
| 241 | |
| 242 | Systemd user services stop when the user logs out, unless lingering is enabled: |
| 243 | |
| 244 | ```sh |
| 245 | sudo loginctl enable-linger "$USER" |
| 246 | ``` |
| 247 | |
| 248 | ### 3. Register the runner |
| 249 | |
| 250 | Create a registration token on the forge admin page at `/admin/runners`. It is shown once, works |
| 251 | once, and expires after `registration_token_hours` (default 24). Then run on the runner machine: |
| 252 | |
| 253 | ```sh |
| 254 | forge runner register \ |
| 255 | --url https://forge.example \ |
| 256 | --token <token> \ |
| 257 | --name build-1 \ |
| 258 | --labels linux,x86_64 |
| 259 | ``` |
| 260 | |
| 261 | This writes `~/.config/forge-runner/runner.toml` with mode 0600, because it holds the runner secret. |
| 262 | Job workspaces go under `~/.cache/forge-runner/work` unless you pass `--work-dir`. |
| 263 | |
| 264 | `--url` must be the forge's public base URL, and the runner must reach it over HTTPS. |
| 265 | |
| 266 | The config file looks like this: |
| 267 | |
| 268 | ```toml |
| 269 | url = "https://forge.example" |
| 270 | runner_id = "..." |
| 271 | runner_secret = "..." # keep private |
| 272 | name = "build-1" |
| 273 | labels = ["linux", "x86_64"] |
| 274 | work_dir = "/home/ci/.cache/forge-runner/work" |
| 275 | concurrency = 1 |
| 276 | executor = "builtin" # or "podman" |
| 277 | auto_update = true # replace the runner binary with the forge's when idle |
| 278 | |
| 279 | [limits] |
| 280 | memory = "4g" # the job cgroup's memory.max, or podman --memory |
| 281 | cpus = "2" # the job cgroup's cpu.max, or podman --cpus |
| 282 | pids = 4096 # the job cgroup's pids.max, or podman --pids-limit |
| 283 | image_max_gb = 20 # cap on the unpacked size of one image (builtin) |
| 284 | |
| 285 | [images] |
| 286 | keep_days = 7 # images unused for this many days are deleted (builtin) |
| 287 | |
| 288 | [[registry]] # optional login for private images, matched by registry host |
| 289 | host = "ghcr.io" |
| 290 | username = "bot" |
| 291 | password = "..." # keep private, the file is mode 0600 |
| 292 | ``` |
| 293 | |
| 294 | `auto_update` (default `true`) lets the runner keep its binary in step with the forge. Claim |
| 295 | responses tell the runner which binary the forge serves. When it differs from the runner's, and no |
| 296 | jobs are running, the runner stops claiming, replaces its binary, and exits with status 75 so systemd |
| 297 | restarts it. It is skipped when the forge's binary is for a different OS or architecture. Set |
| 298 | `auto_update = false` to pin the binary by hand. |
| 299 | |
| 300 | `concurrency` is the number of jobs run in parallel. The default is 1. Each job runs with no |
| 301 | capabilities and `no_new_privileges`. The builtin executor also runs each job under a seccomp |
| 302 | profile that denies calls such as `mount`, `ptrace`, `bpf` and `unshare`. |
| 303 | |
| 304 | Image garbage collection runs at startup and then daily. Unpacked images are kept under |
| 305 | `work_dir/oci`. |
| 306 | |
| 307 | ### Builtin executor caveats |
| 308 | |
| 309 | Read these before you run untrusted or sensitive jobs on a runner: |
| 310 | |
| 311 | - **`network = true` shares the host's network namespace.** The builtin executor does not set up |
| 312 | a separate network, so a job with `network = true` can reach every service that the runner host |
| 313 | can reach, including services bound to the host's `localhost`. Use `network = false` for any job |
| 314 | that does not need the network. It gets only a loopback interface. |
| 315 | - **Subordinate ids are mapped in subuid mode.** Container ids `1` to `n` map to your subordinate |
| 316 | range, so a command can change file ownership to another user or group, as `apt` and `useradd` |
| 317 | do. This needs the setup in [Subordinate ids](#subordinate-ids). Without it, the runner falls |
| 318 | back to single mode, where only your own uid and gid exist in a job and such commands fail. |
| 319 | - **Image files on the host are owned by subordinate ids.** Unpacked images under `work_dir/oci` |
| 320 | have the ownership a job sees. A file that is owner `1000` inside a job is owned by |
| 321 | `start + 999` on the host, for example `525287` with the range `524288`. `ls -ln` shows the |
| 322 | numbers. The runner removes these files through a helper, so it can still clean them up. |
| 323 | - **With `network = true`, `/sys` is the host's `/sys`,** bind-mounted read-only. The job can read |
| 324 | host hardware and kernel details from it. |
| 325 | - **The job cgroup is placed in the parent slice.** It is named `forge-job-<job id>-<helper pid>` |
| 326 | and sits next to the runner's own cgroup, not under it. The runner's cgroup is never used for a |
| 327 | job, because stopping a job kills everything in its cgroup. |
| 328 | |
| 329 | ### 4. Run it under systemd |
| 330 | |
| 331 | Create `~/.config/systemd/user/forge-runner.service`, with the `forge` binary in |
| 332 | `~/.local/bin`: |
| 333 | |
| 334 | ```ini |
| 335 | [Unit] |
| 336 | Description=forge CI runner |
| 337 | After=network-online.target |
| 338 | Wants=network-online.target |
| 339 | |
| 340 | [Service] |
| 341 | ExecStart=%h/.local/bin/forge runner run --config %h/.config/forge-runner/runner.toml |
| 342 | Restart=on-failure |
| 343 | RestartSec=10 |
| 344 | TimeoutStopSec=120 |
| 345 | |
| 346 | [Install] |
| 347 | WantedBy=default.target |
| 348 | ``` |
| 349 | |
| 350 | Then enable and start it: |
| 351 | |
| 352 | ```sh |
| 353 | systemctl --user daemon-reload |
| 354 | systemctl --user enable --now forge-runner.service |
| 355 | journalctl --user -u forge-runner.service -f |
| 356 | ``` |
| 357 | |
| 358 | On shutdown the runner stops claiming, and it cancels and reports any job still running. A job |
| 359 | whose runner disappears without reporting is failed as `runner lost` after about two minutes, with |
| 360 | no heartbeat. |
| 361 | |
| 362 | ### Labels |
| 363 | |
| 364 | A job with `runs_on = ["linux", "x86_64"]` is only claimed by a runner that has both labels. Jobs |
| 365 | with no `runs_on` can go to any runner. |
| 366 | |
| 367 | ## Forge limits (`[ci]`) |
| 368 | |
| 369 | The `[ci]` section of `forge.toml` controls CI on the forge. These are the defaults: |
| 370 | |
| 371 | ```toml |
| 372 | [ci] |
| 373 | enabled = true |
| 374 | max_log_mb = 16 # a log past this is cut with "[log truncated]" |
| 375 | max_artifact_mb = 512 # per artifact |
| 376 | max_artifacts_per_job = 50 |
| 377 | log_retention_days = 90 |
| 378 | artifact_retention_days = 30 |
| 379 | registration_token_hours = 24 |
| 380 | ``` |
| 381 | |
| 382 | Retention runs once a day. Logs older than `log_retention_days` are deleted. The job rows stay, so |
| 383 | statuses and run pages still work. Artifacts older than `artifact_retention_days` are deleted from the |
| 384 | blob store. |
| 385 | |
| 386 | ## Runner protocol |
| 387 | |
| 388 | The runner talks to the forge only over HTTPS. JSON bodies are defined in `forge-ci::proto`. |
| 389 | |
| 390 | | Method and path | Auth | Purpose | |
| 391 | |-----------------|------|---------| |
| 392 | | `POST /api/v1/ci/runners/register` | registration token | Exchange a single-use token for a runner id and secret. | |
| 393 | | `POST /api/v1/ci/runners/claim` | `Bearer` runner secret | Long-poll (up to about 25 s) for a job. Returns the job assignment, or 204. Disabled runners get 403. | |
| 394 | | `POST /api/v1/ci/jobs/{id}/logs` | `Bearer` job token | Append a log chunk. The header `X-Forge-Log-Offset` must equal the current size. A mismatch returns 409 with `X-Forge-Log-Size`, and the runner resumes from there. | |
| 395 | | `POST /api/v1/ci/jobs/{id}/heartbeat` | `Bearer` job token | Sent every 10 s. The reply says whether the job was cancelled. | |
| 396 | | `PUT /api/v1/ci/jobs/{id}/artifacts/{name}` | `Bearer` job token | Upload one artifact as a raw body. | |
| 397 | | `POST /api/v1/ci/jobs/{id}/finish` | `Bearer` job token | Report `success`, `failure`, `cancelled` or `error`, with the exit code and an optional message. | |
| 398 | |
| 399 | The claim response (`JobAssignment`) carries the repository, the commit, the job's run script, |
| 400 | image, env, secrets, artifact globs, timeout and network setting. Secrets are empty for fork pull |
| 401 | request runs. |
| 402 | |
| 403 | For the workspace, the runner runs `git fetch --depth=1` on the clone URL with the job token, then |
| 404 | `git checkout --detach <commit>`. The token goes in an HTTP header, never on the command line. |
| 405 | |
| 406 | Browsers and other clients read job logs with `GET /{owner}/{repo}/ci/jobs/{id}/log.txt`. It |
| 407 | streams the file as it grows and ends when the job finishes. |