forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
8.8 KiB
1# CI sandbox v2: subordinate id mapping (contract between workers)
2
3**Goal:** jobs get a real range of uids and gids, so package managers that `chown` to other users work (`apt`, `apk add`, `useradd`, …). Container uid 0 maps to the runner's uid, and container ids `1..=N` map to the runner's `/etc/subuid` and `/etc/subgid` range. This uses the setuid helpers `newuidmap` and `newgidmap` from shadow-utils, if present. Without them, the runner falls back to the v1 single-uid mapping.
4
5The global rules in `~/.cache/forge-orch/waveb-common.md` apply.
6
7The existing sandbox is described in `docs/ci-builtin-executor.md` and `~/.cache/forge-orch/libcontainer-notes.md`. Read both.
8
9## Host facts
10- `/usr/bin/newuidmap` and `/usr/bin/newgidmap` exist.
11- The subuid range is `tahinli:524288:65536`.
12- `unshare --map-auto --map-root-user chown 1000:1000 f` works and gives host owner `525287`. That is `start + 999`, because container 0 is the runner and container 1 is `start`.
13- Tests can use this command to create files with a subordinate owner. Skip the test when it fails.
14
15## Mapping
16- **subuid mode:** container `0` maps to the runner uid (count 1), and container `1` maps to `start` (count `n`), where `n = min(count, 65536)`. The same applies to gids. Container size is `1 + n`.
17- **single mode:** today's behaviour, `0 → uid`, count 1.
18
19## Worker V-A: `crates/forge-runner/src/sandbox/idmap.rs`, plus the hidden-command dispatch
20```rust
21#[derive(Clone, Copy, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
22#[serde(rename_all = "lowercase")]
23pub enum UsernsMode { #[default] Auto, Single, Subuid }
24
25#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
26pub struct IdMap { pub uid: u32, pub gid: u32, pub sub_uid: Option<(u32, u32)>, pub sub_gid: Option<(u32, u32)> } // (start, n)
27impl IdMap {
28 pub fn single() -> Self; // current uid/gid, no ranges
29 pub fn resolve(mode: UsernsMode) -> Result<Self, IdMapError>; // see below
30 pub fn is_subuid(&self) -> bool;
31 pub fn size(&self) -> u32; // 1 + n, or 1
32 pub fn describe(&self) -> String; // "subuid 524288+65536" | "single uid 1000"
33}
34impl Default for IdMap // = single()
35pub fn parse_subid(content: &str, user: &str, uid: u32) -> Vec<(u32, u32)>; // match name OR numeric uid, skip comments/bad lines
36/// SINGLE-THREADED processes only. unshare(NEWUSER|NEWNS), install the maps, make "/" rec-private.
37pub fn enter_mapped_userns(map: &IdMap) -> Result<(), IdMapError>;
38/// `<exe> __sandbox-rm <path>`, IdMap JSON on stdin: enter the userns, give dirs u+rwx, remove_dir_all. Exit 0, or 125 + "forge-sandbox: " line.
39pub fn rm_helper_main(args: &[OsString]) -> i32;
40/// Blocking. Single mode: remove_dir_all with a chmod-u+rwx retry, no spawn. Subuid mode: run `<exe> __sandbox-rm <path>` with the map on stdin.
41/// Missing path = Ok.
42pub fn remove_tree(map: &IdMap, path: &Path, exe: &Path) -> io::Result<()>;
43```
44
45**`resolve`**
46- **Auto:** use subuid when all of these hold:
47 - `newuidmap` and `newgidmap` exist in `/usr/bin`, `/bin`, `/usr/sbin` or `/sbin`;
48 - `/etc/subuid` and `/etc/subgid` each have an entry for the user (by name, from `getpwuid`, or by numeric uid);
49 - that entry has `count >= 65536`. Take the first such range. A smaller range is accepted with a warning only if it is ≥ 1000.
50
51 Otherwise use single.
52- **Subuid:** return an error instead of falling back.
53- **Single:** `single()`.
54
55**`enter_mapped_userns`, subuid mode**
561. Before unsharing, create a pipe and fork a child. The child blocks reading the pipe.
572. The parent runs `unshare(CLONE_NEWUSER | CLONE_NEWNS)`, then writes one byte.
583. The child runs these commands:
59 - `newuidmap <parent pid> 0 <uid> 1 1 <start> <n>`
60 - `newgidmap <parent pid> 0 <gid> 1 1 <start> <n>`
61
62 It runs them via `std::process::Command` (the child is single-threaded, so this is allowed), then `_exit`s with 0 or 1.
634. The parent waits with `waitpid` and checks the status. On failure it returns an error including newuidmap's stderr.
645. **Do NOT write `setgroups deny`**, because newgidmap needs `allow`. After mapping, call `setgroups(&[])`. Then make the mounts rec-private as today.
65
66**Single mode** keeps today's code: `setgroups deny` and direct map writes. Move it here from `helper.rs`; the helper then calls `enter_mapped_userns`.
67
68**Dispatch:**
69- `cli::is_sandbox_invocation()` becomes true for any argv[1] starting with `__sandbox`.
70- Add `pub fn hidden_main() -> i32` in `cli.rs`, which routes:
71 - `__sandbox` → `sandbox::helper_main()`
72 - `__sandbox-rm` → `idmap::rm_helper_main`
73 - `__sandbox-unpack` → `crate::oci::unpack_helper_main()`. This belongs to V-C; if it doesn't exist yet, add a stub returning 125 in `oci/mod.rs` that V-C replaces.
74- Use `hidden_main` from `cli::main_with_args`, `forge-runner`'s `main.rs` and forge-web's `main.rs`.
75
76**Tests**
77- `parse_subid` cases.
78- `size` and `describe`.
79- `resolve(Auto)` on this host yields subuid.
80- An integration test in `tests/sandbox.rs` (or a new `tests/idmap.rs`) that runs `forge-runner __sandbox-rm` on a TempDir holding a file chowned to 1000 by `unshare --map-auto --map-root-user`. The dir must be gone afterwards. Skip when `unshare --map-auto` fails.
81
82## Worker V-C: images with real ownership, in `crates/forge-runner/src/oci/`
83- **`unpack_layers` gains a `ownership: Ownership` parameter:**
84 - `pub enum Ownership { Flatten, Preserve { size: u32 } }`.
85 - `Flatten` is today's behaviour.
86 - `Preserve`:
87 - `lchown` every entry, including symlinks, to the tar header uid and gid.
88 - Ids `>= size` become 0 and are counted in `UnpackStats.remapped_ids`.
89 - setuid and setgid are still stripped.
90 - Directories get their exact mode at the end, deepest first, instead of `mode | 0o700`.
91 - Update all callers and tests.
92- **`ImageStore::with_idmap(map: IdMap, helper_exe: PathBuf)`:**
93 - When the map is subuid, the rootfs dir is `rootfs/<digest>-s<uid start>-<n>`, so a changed range re-unpacks. Single keeps `rootfs/<digest>`.
94 - Unpacking in subuid mode runs `<helper_exe> __sandbox-unpack` from `spawn_blocking`. The JSON `UnpackRequest { map, rootfs, layers, max_bytes }` goes on stdin and `UnpackStats` JSON comes back on stdout.
95 - `pub fn unpack_helper_main() -> i32` (in `oci/mod.rs` or a new `oci/helper.rs`):
96 - reads the request;
97 - calls `idmap::enter_mapped_userns`;
98 - calls `unpack_layers(.., Ownership::Preserve { size: map.size() })`;
99 - prints the stats and exits 0, or 125 with a "forge-sandbox:" line.
100- **GC and staging cleanup** remove trees with `idmap::remove_tree(&map, path, &helper_exe)`, because subuid-owned files can't be deleted by the runner directly.
101- **Tests:**
102 - `Preserve` unit tests that only check the remapping logic (a pure function, `fn map_id(id, size) -> (u32, bool)`).
103 - Path naming per mode.
104 - An integration test that unpacks a crafted layer with a file owned by 1000:1000 through `forge-runner __sandbox-unpack`, then checks that the host owner is `start + 999`. Skip when subuid is unavailable.
105
106## Worker V-B, after V-A: sandbox, executor, config, check, docs
107- **`SandboxRequest`** gains `#[serde(default)] pub idmap: IdMap`. The helper calls `idmap::enter_mapped_userns(&req.idmap)`. The container spec's uid and gid mappings become a single line each, `0 0 <size>`, which is relative to the helper's namespace.
108- **Workspace ownership:** after the job ends, the helper (still root in the userns) walks `req.workspace` without following symlinks and `lchown`s every entry whose uid or gid is not 0 to 0:0. The host runner then owns everything for artifacts and cleanup.
109- **Executor:** `reap_job` and any job-dir cleanup use `idmap::remove_tree`. `BuiltinExecutor` gets the map.
110- **Config:** `[sandbox] userns = "auto" | "single" | "subuid"` (default auto). It is resolved once at startup, logs `describe()`, and is passed to `ImageStore::with_idmap` and `BuiltinExecutor`. A failure to resolve in subuid mode is a startup error.
111- **`forge-runner check`:** prints the mode. In subuid mode it also checks that a probe job can `chown 1000:1000` a file in `/tmp`.
112- **docs/ci.md:**
113 - replace the single-uid caveat with the v2 behaviour;
114 - document the requirements: shadow-utils `newuidmap`/`newgidmap` and an `/etc/subuid` and `/etc/subgid` entry (`usermod --add-subuids 100000-165535 --add-subgids 100000-165535 <runner user>`);
115 - document the fallback;
116 - explain that cached image files on the host are owned by subordinate ids.
117- **Tests (`tests/sandbox.rs`, skip when subuid is unavailable):**
118 - a job that does `touch /tmp/f && chown 1000:1000 /tmp/f && stat -c '%u %g' /tmp/f` prints `1000 1000`;
119 - `cat /proc/self/uid_map` has size `1 + n`;
120 - a job that creates `/workspace/a` and chowns it to 1000 leaves `a` owned by the runner uid on the host;
121 - the job dir is gone afterwards;
122 - all existing single-mode tests still pass, with the map forced to single.