forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
11.3 KiB
1# Built-in executor for forge-runner (contract between workers)
2
3**Goal:** `forge-runner` runs jobs in containers with **no external container tool**. There's no podman: images are pulled and unpacked in-process, and the job runs in a rootless sandbox built with youki's `libcontainer` crate. `PodmanExecutor` stays as an option. Runner config gains `executor = "builtin"` (the default) or `"podman"`.
4
5The global rules apply:
6- No unwrap in non-test code.
7- Tests use `tempfile::TempDir`.
8- Never put cargo target dirs or big files in /tmp.
9- Crates go in at their latest version via `cargo add`, recorded in the root `[workspace.dependencies]`.
10- Only rustfmt your own files.
11
12Crates (latest): `oci-client`, `oci-spec`, `libcontainer` (youki; **check its 0.7 API in ~/.cargo/registry/src/*/libcontainer-*/ after adding it**), `tar`, `flate2`, `zstd` (if layers can be zstd), `sha2`, `nix` or `rustix`.
13
14## Design decisions (fixed)
15- **Self re-exec.** libcontainer forks and clones, which is unsafe inside a multi-threaded tokio process. So the runner never calls libcontainer in its own process. It spawns `/proc/self/exe __sandbox` (a hidden subcommand in main.rs) as a single-threaded child **with no tokio runtime**. The child reads a JSON `SandboxRequest` from **stdin** (the secrets travel there, never in argv or files), builds the container, runs it, forwards stdout and stderr, and exits with the job's exit code. Killing the helper (SIGTERM, then SIGKILL after a grace period) must tear the container down: the helper handles SIGTERM by killing and deleting the container.
16- **User namespace, v1: single-uid mapping.** Container uid 0 maps to the runner's uid, with no `newuidmap` needed. Layers are unpacked with ownership flattened to the runner's user, and setuid/setgid bits are stripped. Document that package managers which `chown` to other users may fail. (v2, later and not in this task: a subuid range through `newuidmap` when available.)
17- **Namespaces:** user, mount, pid, ipc, uts, cgroup. **Network:** `network = false` gets a new empty net namespace (loopback only). `network = true` **shares the host network namespace**, because rootless isolated networking needs pasta or slirp, which are external tools. Document this clearly in docs/ci.md: on `network = true` jobs can reach services on the runner host's localhost.
18- **Hardening:**
19 - no capabilities, in any of the sets;
20 - `no_new_privileges`;
21 - a seccomp profile: default allow, deny with EPERM for `keyctl`, `add_key`, `request_key`, `bpf`, `perf_event_open`, `ptrace`, `mount`, `umount2`, `pivot_root`, `swapon`, `swapoff`, `reboot`, `kexec_load`, `kexec_file_load`, `init_module`, `finit_module`, `delete_module`, `unshare`, `setns`, `open_by_handle_at`, `userfaultfd`, `acct`, `settimeofday`, `clock_settime`, `syslog`, `iopl`, `ioperm`, `quotactl`, `name_to_handle_at`, `lookup_dcookie`;
22 - masked paths and read-only paths as docker does (`/proc/kcore`, `/proc/keys`, `/proc/sys` read-only, and so on);
23 - rlimits: `RLIMIT_NPROC` set from `limits.pids`, and `RLIMIT_NOFILE` 65536.
24- **Limits:** use a cgroup v2 rootless cgroup through libcontainer (systemd or fs driver, whichever works rootless) for memory, cpus and pids from `[limits]`. **If cgroups can't be set up, run anyway**, and log one warning line into the job log ("resource limits unavailable: …"). The job timeout still applies.
25- **Filesystem:**
26 - The rootfs is an **overlay** mount with the cached image rootfs as `lowerdir`, plus `upperdir`/`workdir` in `<work_dir>/jobs/<job_id>/overlay/`. Rootless overlayfs inside the user namespace needs kernel ≥ 5.11, and this host has 7.2.
27 - If the overlay mount fails, fall back to a full copy of the rootfs into the job dir (try a reflink copy first) and log that.
28 - `/workspace` is a bind mount of the job workspace (rw); `/tmp` is a tmpfs; `/proc`, `/dev` (tmpfs plus the standard null, zero, full, random, urandom and tty device bind mounts from the host), `/dev/pts`, `/dev/shm`, `/sys` (ro).
29 - `/etc/resolv.conf` is bind-mounted from the host (ro) when `network = true`. `/etc/hosts` and `/etc/hostname` are generated.
30 - Everything under `jobs/<job_id>` is deleted after the job.
31- **Process:** `sh -eu -c <run>`, cwd `/workspace`.
32 - Env: the image config's `Env`, then `HOME=/root`, `CI=true`, `FORGE_CI=true`, then the job env, then the secrets.
33 - PATH comes from the image config, or the default `/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`.
34
35## Worker E1: images, `crates/forge-runner/src/oci/{mod.rs, pull.rs, unpack.rs, store.rs}`
36```rust
37pub struct ImageStore { root: PathBuf } // <work_dir>/oci
38impl ImageStore {
39 pub fn new(root: PathBuf) -> Self;
40 /// Resolves `image` ("docker.io/rust:1", "alpine", "ghcr.io/o/i@sha256:…"), pulls the manifest for linux/<host arch>,
41 /// downloads missing layer blobs (verified by sha256 digest, written temp+rename), unpacks once per manifest digest,
42 /// and returns the rootfs path + image config. Concurrent calls for the same image must not corrupt (lock file per digest).
43 pub async fn ensure(&self, image: &str, auth: Option<&RegistryAuth>, log: &mpsc::Sender<Vec<u8>>) -> Result<PreparedImage, ImageError>;
44 /// Deletes unpacked rootfs dirs and blobs not used for `keep_days` (by an mtime "last used" stamp file).
45 pub fn gc(&self, keep_days: u32) -> Result<GcStats, ImageError>;
46}
47pub struct PreparedImage { pub rootfs: PathBuf, pub digest: String, pub env: Vec<String>, pub working_dir: Option<String>, pub user: Option<String> }
48pub struct RegistryAuth { pub username: String, pub password: String } // Debug redacts
49```
50- **Image references:** `alpine` normalizes to `docker.io/library/alpine:latest`. Use oci-client's `Reference`.
51- **Pull policy:** always check the manifest digest (a HEAD or GET manifest), and reuse the unpacked rootfs when the digest matches. When the registry is unreachable and a rootfs for that tag was pulled before, use it and log a warning. To support this, keep a `tags/<sanitized ref>` file holding the last digest.
52- **Unpacking (security-critical):**
53 - Process the layers in order. Support gzip, zstd and uncompressed layers.
54 - **Never write outside the rootfs.** Reject entries whose normalized path is absolute or contains `..`.
55 - Resolve every parent component **without following symlinks that lead out**: create files with `openat`-style handling, or check each parent with `symlink_metadata`, and never follow a symlink in a parent path.
56 - Hardlinks must point inside the rootfs.
57 - Whiteouts: `.wh.<name>` deletes `<name>`; `.wh..wh..opq` clears the directory's earlier-layer contents.
58 - Skip device and fifo nodes, logging the count.
59 - Strip setuid/setgid. Keep the other permission bits, and set the owner to the current user.
60 - Cap total unpacked size at `[limits] image_max_gb` (default 20).
61- **Tests:**
62 - craft tar layers in the test (with the `tar` builder) to cover traversal (`../x`, `/abs`), a symlink-parent escape (`a -> /etc`, then `a/passwd`), a hardlink escape, whiteouts and opaque dirs, setuid stripping, and gzip and zstd;
63 - a reference-normalization test;
64 - one `#[ignore]` test that pulls `docker.io/library/alpine:latest` for real.
65
66## Worker E2: sandbox, `crates/forge-runner/src/sandbox/{mod.rs, spec.rs, helper.rs}`
67```rust
68#[derive(Serialize, Deserialize)]
69pub struct SandboxRequest { pub job_id: String, pub rootfs: PathBuf, pub job_dir: PathBuf /* overlay + container state */, pub workspace: PathBuf,
70 pub env: Vec<(String, String)>, pub run: String, pub network: bool, pub limits: Limits /* existing config::Limits */ }
71pub fn build_spec(req: &SandboxRequest, uid: u32, gid: u32, rootfs_mount: &Path) -> Result<oci_spec::runtime::Spec, SandboxError>; // pure, unit-tested
72pub fn helper_main() -> i32; // entry for `forge-runner __sandbox`: read SandboxRequest JSON from stdin, set up overlay (or copy fallback), run via libcontainer, forward output, return exit code (125 = sandbox setup failure, with the reason on stderr prefixed "forge-sandbox: ")
73```
74- Use libcontainer's rootless support. **Check what libcontainer 0.7 needs**: whether it requires a `bundle` dir with `config.json` (write it under `job_dir`; the env, including secrets, is inside it, so write it 0600 in a 0700 dir and delete it right after start if libcontainer allows, otherwise when the job ends), its state dir (`job_dir/state`), and its stdio handling (inherit the helper's stdout and stderr).
75- The helper must be **single-threaded with no tokio runtime**. In main.rs, intercept `__sandbox` before building the tokio runtime: switch `#[tokio::main]` to a manual runtime build after argument parsing.
76- **Tests:**
77 - `build_spec` unit tests: namespaces, the uid map, no capabilities, no_new_privileges, seccomp contents, mounts, net namespace on and off, env order, and secrets never in `process.args`;
78 - a **real sandbox test with no network or registry**:
79 - build a tiny rootfs in a TempDir by copying `/bin/sh` (or busybox if present) and the shared libraries `ldd` lists for it (parse the ldd output);
80 - run `echo hi; id -u; cat /proc/self/status | grep CapEff; touch /workspace/f` through the helper (via `cargo`'s built test binary? It's simpler to invoke `helper_main` logic through a spawned `env!("CARGO_BIN_EXE_forge-runner") __sandbox` from an integration test in `crates/forge-runner/tests/sandbox.rs`);
81 - assert that the output contains hi, uid 0, `CapEff: 0000000000000000`, and that the workspace file exists on the host;
82 - a second test with `network = false` asserts there's no non-loopback interface (`cat /proc/net/dev`);
83 - a third asserts that writing to `/proc/sys/kernel/hostname` fails;
84 - if user namespaces are unavailable on the machine (check `/proc/sys/kernel/unprivileged_userns_clone`, or a failing `unshare -U`), skip with a message.
85
86## Worker E3: wiring, after E1 and E2: `BuiltinExecutor` in `crates/forge-runner/src/executor.rs`, plus config, CLI, docs
87- `BuiltinExecutor { images: ImageStore, limits: Limits, helper: PathBuf /* std::env::current_exe() */ }` implements `Executor`:
88 1. `ensure` the image, logging "pulling …" lines into the job log;
89 2. spawn the helper with a piped stdin carrying `SandboxRequest`, and piped stdout and stderr, through the existing `forward_lines`;
90 3. on cancel or timeout send SIGTERM, wait `STOP_GRACE`, then SIGKILL;
91 4. map the outcome: exit 0 is Success; 125 with a "forge-sandbox:" stderr line is Error; anything else is Failure with the exit code.
92- **Runner config:**
93 - `executor = "builtin" | "podman"`, default builtin;
94 - `[limits] image_max_gb = 20`;
95 - `[images] keep_days = 7`, with optional `[[registry]] host, username, password`; the password is only in the 0600 runner.toml;
96 - GC runs at startup and daily.
97- **`forge-runner check`:** a new subcommand that reports whether user namespaces, overlayfs in a userns, cgroup delegation (which controllers) and seccomp work, so operators can diagnose a host.
98- **docs/ci.md:**
99 - replace "rootless podman" as the requirement with "Linux ≥ 5.11 with unprivileged user namespaces; cgroup v2 delegation for limits (systemd does this for user sessions)";
100 - podman becomes optional;
101 - document the `network = true` host-namespace caveat and the single-uid caveat.
102- **Tests:**
103 - a flow test using `BuiltinExecutor` with a locally prepared rootfs; inject a fake `ImageStore` result, or add an `ImageSource` trait so tests can supply a rootfs dir;
104 - the existing flow tests stay as they are.