Built-in executor for forge-runner (contract between workers)

Goal: forge-runner runs jobs in containers with no external container tool. There's no podman: images are pulled and unpacked in-process, and the job runs in a rootless sandbox built with youki's libcontainer crate. PodmanExecutor stays as an option. Runner config gains executor = "builtin" (the default) or "podman".

The global rules apply:

• No unwrap in non-test code.
• Tests use tempfile::TempDir.
• Never put cargo target dirs or big files in /tmp.
• Crates go in at their latest version via cargo add, recorded in the root [workspace.dependencies].
• Only rustfmt your own files.

Crates (latest): oci-client, oci-spec, libcontainer (youki; check its 0.7 API in ~/.cargo/registry/src//libcontainer-/ after adding it), tar, flate2, zstd (if layers can be zstd), sha2, nix or rustix.

Design decisions (fixed)

• Self re-exec. libcontainer forks and clones, which is unsafe inside a multi-threaded tokio process. So the runner never calls libcontainer in its own process. It spawns /proc/self/exe __sandbox (a hidden subcommand in main.rs) as a single-threaded child with no tokio runtime. The child reads a JSON SandboxRequest from stdin (the secrets travel there, never in argv or files), builds the container, runs it, forwards stdout and stderr, and exits with the job's exit code. Killing the helper (SIGTERM, then SIGKILL after a grace period) must tear the container down: the helper handles SIGTERM by killing and deleting the container.
• User namespace, v1: single-uid mapping. Container uid 0 maps to the runner's uid, with no newuidmap needed. Layers are unpacked with ownership flattened to the runner's user, and setuid/setgid bits are stripped. Document that package managers which chown to other users may fail. (v2, later and not in this task: a subuid range through newuidmap when available.)
• Namespaces: user, mount, pid, ipc, uts, cgroup. Network: network = false gets a new empty net namespace (loopback only). network = true shares the host network namespace, because rootless isolated networking needs pasta or slirp, which are external tools. Document this clearly in docs/ci.md: on network = true jobs can reach services on the runner host's localhost.
• Hardening:
  • no capabilities, in any of the sets;
  • no_new_privileges;
  • a seccomp profile: default allow, deny with EPERM for keyctl, add_key, request_key, bpf, perf_event_open, ptrace, mount, umount2, pivot_root, swapon, swapoff, reboot, kexec_load, kexec_file_load, init_module, finit_module, delete_module, unshare, setns, open_by_handle_at, userfaultfd, acct, settimeofday, clock_settime, syslog, iopl, ioperm, quotactl, name_to_handle_at, lookup_dcookie;
  • masked paths and read-only paths as docker does (/proc/kcore, /proc/keys, /proc/sys read-only, and so on);
  • rlimits: RLIMIT_NPROC set from limits.pids, and RLIMIT_NOFILE 65536.
• Limits: use a cgroup v2 rootless cgroup through libcontainer (systemd or fs driver, whichever works rootless) for memory, cpus and pids from [limits]. If cgroups can't be set up, run anyway, and log one warning line into the job log ("resource limits unavailable: …"). The job timeout still applies.
• Filesystem:
  • The rootfs is an overlay mount with the cached image rootfs as lowerdir, plus upperdir/workdir in <work_dir>/jobs/<job_id>/overlay/. Rootless overlayfs inside the user namespace needs kernel ≥ 5.11, and this host has 7.2.
  • If the overlay mount fails, fall back to a full copy of the rootfs into the job dir (try a reflink copy first) and log that.
  • /workspace is a bind mount of the job workspace (rw); /tmp is a tmpfs; /proc, /dev (tmpfs plus the standard null, zero, full, random, urandom and tty device bind mounts from the host), /dev/pts, /dev/shm, /sys (ro).
  • /etc/resolv.conf is bind-mounted from the host (ro) when network = true. /etc/hosts and /etc/hostname are generated.
  • Everything under jobs/<job_id> is deleted after the job.
• Process: sh -eu -c <run>, cwd /workspace.
  • Env: the image config's Env, then HOME=/root, CI=true, FORGE_CI=true, then the job env, then the secrets.
  • PATH comes from the image config, or the default /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin.

Worker E1: images, crates/forge-runner/src/oci/{mod.rs, pull.rs, unpack.rs, store.rs}

pub struct ImageStore { root: PathBuf }   // <work_dir>/oci
impl ImageStore {
  pub fn new(root: PathBuf) -> Self;
  /// Resolves `image` ("docker.io/rust:1", "alpine", "ghcr.io/o/i@sha256:…"), pulls the manifest for linux/<host arch>,
  /// downloads missing layer blobs (verified by sha256 digest, written temp+rename), unpacks once per manifest digest,
  /// and returns the rootfs path + image config. Concurrent calls for the same image must not corrupt (lock file per digest).
  pub async fn ensure(&self, image: &str, auth: Option<&RegistryAuth>, log: &mpsc::Sender<Vec<u8>>) -> Result<PreparedImage, ImageError>;
  /// Deletes unpacked rootfs dirs and blobs not used for `keep_days` (by an mtime "last used" stamp file).
  pub fn gc(&self, keep_days: u32) -> Result<GcStats, ImageError>;
}
pub struct PreparedImage { pub rootfs: PathBuf, pub digest: String, pub env: Vec<String>, pub working_dir: Option<String>, pub user: Option<String> }
pub struct RegistryAuth { pub username: String, pub password: String }   // Debug redacts

• Image references: alpine normalizes to docker.io/library/alpine:latest. Use oci-client's Reference.
• Pull policy: always check the manifest digest (a HEAD or GET manifest), and reuse the unpacked rootfs when the digest matches. When the registry is unreachable and a rootfs for that tag was pulled before, use it and log a warning. To support this, keep a tags/<sanitized ref> file holding the last digest.
• Unpacking (security-critical):
  • Process the layers in order. Support gzip, zstd and uncompressed layers.
  • Never write outside the rootfs. Reject entries whose normalized path is absolute or contains ...
  • Resolve every parent component without following symlinks that lead out: create files with openat-style handling, or check each parent with symlink_metadata, and never follow a symlink in a parent path.
  • Hardlinks must point inside the rootfs.
  • Whiteouts: .wh.<name> deletes <name>; .wh..wh..opq clears the directory's earlier-layer contents.
  • Skip device and fifo nodes, logging the count.
  • Strip setuid/setgid. Keep the other permission bits, and set the owner to the current user.
  • Cap total unpacked size at [limits] image_max_gb (default 20).
• Tests:
  • craft tar layers in the test (with the tar builder) to cover traversal (../x, /abs), a symlink-parent escape (a -> /etc, then a/passwd), a hardlink escape, whiteouts and opaque dirs, setuid stripping, and gzip and zstd;
  • a reference-normalization test;
  • one #[ignore] test that pulls docker.io/library/alpine:latest for real.

Worker E2: sandbox, crates/forge-runner/src/sandbox/{mod.rs, spec.rs, helper.rs}

#[derive(Serialize, Deserialize)]
pub struct SandboxRequest { pub job_id: String, pub rootfs: PathBuf, pub job_dir: PathBuf /* overlay + container state */, pub workspace: PathBuf,
  pub env: Vec<(String, String)>, pub run: String, pub network: bool, pub limits: Limits /* existing config::Limits */ }
pub fn build_spec(req: &SandboxRequest, uid: u32, gid: u32, rootfs_mount: &Path) -> Result<oci_spec::runtime::Spec, SandboxError>;  // pure, unit-tested
pub fn helper_main() -> i32;   // entry for `forge-runner __sandbox`: read SandboxRequest JSON from stdin, set up overlay (or copy fallback), run via libcontainer, forward output, return exit code (125 = sandbox setup failure, with the reason on stderr prefixed "forge-sandbox: ")

• Use libcontainer's rootless support. Check what libcontainer 0.7 needs: whether it requires a bundle dir with config.json (write it under job_dir; the env, including secrets, is inside it, so write it 0600 in a 0700 dir and delete it right after start if libcontainer allows, otherwise when the job ends), its state dir (job_dir/state), and its stdio handling (inherit the helper's stdout and stderr).
• The helper must be single-threaded with no tokio runtime. In main.rs, intercept __sandbox before building the tokio runtime: switch #[tokio::main] to a manual runtime build after argument parsing.
• Tests:
  • build_spec unit tests: namespaces, the uid map, no capabilities, no_new_privileges, seccomp contents, mounts, net namespace on and off, env order, and secrets never in process.args;
  • a real sandbox test with no network or registry:
    • build a tiny rootfs in a TempDir by copying /bin/sh (or busybox if present) and the shared libraries ldd lists for it (parse the ldd output);
    • run echo hi; id -u; cat /proc/self/status | grep CapEff; touch /workspace/f through the helper (via cargo's built test binary? It's simpler to invoke helper_main logic through a spawned env!("CARGO_BIN_EXE_forge-runner") __sandbox from an integration test in crates/forge-runner/tests/sandbox.rs);
    • assert that the output contains hi, uid 0, CapEff: 0000000000000000, and that the workspace file exists on the host;
    • a second test with network = false asserts there's no non-loopback interface (cat /proc/net/dev);
    • a third asserts that writing to /proc/sys/kernel/hostname fails;
    • if user namespaces are unavailable on the machine (check /proc/sys/kernel/unprivileged_userns_clone, or a failing unshare -U), skip with a message.

Worker E3: wiring, after E1 and E2: BuiltinExecutor in crates/forge-runner/src/executor.rs, plus config, CLI, docs

• BuiltinExecutor { images: ImageStore, limits: Limits, helper: PathBuf /* std::env::current_exe() */ } implements Executor:
  1. ensure the image, logging "pulling …" lines into the job log;
  2. spawn the helper with a piped stdin carrying SandboxRequest, and piped stdout and stderr, through the existing forward_lines;
  3. on cancel or timeout send SIGTERM, wait STOP_GRACE, then SIGKILL;
  4. map the outcome: exit 0 is Success; 125 with a "forge-sandbox:" stderr line is Error; anything else is Failure with the exit code.
• Runner config:
  • executor = "builtin" | "podman", default builtin;
  • [limits] image_max_gb = 20;
  • [images] keep_days = 7, with optional [[registry]] host, username, password; the password is only in the 0600 runner.toml;
  • GC runs at startup and daily.
• forge-runner check: a new subcommand that reports whether user namespaces, overlayfs in a userns, cgroup delegation (which controllers) and seccomp work, so operators can diagnose a host.
• docs/ci.md:
  • replace "rootless podman" as the requirement with "Linux ≥ 5.11 with unprivileged user namespaces; cgroup v2 delegation for limits (systemd does this for user sessions)";
  • podman becomes optional;
  • document the network = true host-namespace caveat and the single-uid caveat.
• Tests:
  • a flow test using BuiltinExecutor with a locally prepared rootfs; inject a fake ImageStore result, or add an ImageSource trait so tests can supply a rootfs dir;
  • the existing flow tests stay as they are.