forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
8.8 KiB

CI sandbox v2: subordinate id mapping (contract between workers)

Goal: jobs get a real range of uids and gids, so package managers that chown to other users work (apt, apk add, useradd, …). Container uid 0 maps to the runner's uid, and container ids 1..=N map to the runner's /etc/subuid and /etc/subgid range. This uses the setuid helpers newuidmap and newgidmap from shadow-utils, if present. Without them, the runner falls back to the v1 single-uid mapping.

The global rules in ~/.cache/forge-orch/waveb-common.md apply.

The existing sandbox is described in docs/ci-builtin-executor.md and ~/.cache/forge-orch/libcontainer-notes.md. Read both.

Host facts

  • /usr/bin/newuidmap and /usr/bin/newgidmap exist.
  • The subuid range is tahinli:524288:65536.
  • unshare --map-auto --map-root-user chown 1000:1000 f works and gives host owner 525287. That is start + 999, because container 0 is the runner and container 1 is start.
  • Tests can use this command to create files with a subordinate owner. Skip the test when it fails.

Mapping

  • subuid mode: container 0 maps to the runner uid (count 1), and container 1 maps to start (count n), where n = min(count, 65536). The same applies to gids. Container size is 1 + n.
  • single mode: today's behaviour, 0 → uid, count 1.

Worker V-A: crates/forge-runner/src/sandbox/idmap.rs, plus the hidden-command dispatch

#[derive(Clone, Copy, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum UsernsMode { #[default] Auto, Single, Subuid }

#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
pub struct IdMap { pub uid: u32, pub gid: u32, pub sub_uid: Option<(u32, u32)>, pub sub_gid: Option<(u32, u32)> } // (start, n)
impl IdMap {
    pub fn single() -> Self;                                      // current uid/gid, no ranges
    pub fn resolve(mode: UsernsMode) -> Result<Self, IdMapError>; // see below
    pub fn is_subuid(&self) -> bool;
    pub fn size(&self) -> u32;                                    // 1 + n, or 1
    pub fn describe(&self) -> String;                             // "subuid 524288+65536" | "single uid 1000"
}
impl Default for IdMap  // = single()
pub fn parse_subid(content: &str, user: &str, uid: u32) -> Vec<(u32, u32)>; // match name OR numeric uid, skip comments/bad lines
/// SINGLE-THREADED processes only. unshare(NEWUSER|NEWNS), install the maps, make "/" rec-private.
pub fn enter_mapped_userns(map: &IdMap) -> Result<(), IdMapError>;
/// `<exe> __sandbox-rm <path>`, IdMap JSON on stdin: enter the userns, give dirs u+rwx, remove_dir_all. Exit 0, or 125 + "forge-sandbox: " line.
pub fn rm_helper_main(args: &[OsString]) -> i32;
/// Blocking. Single mode: remove_dir_all with a chmod-u+rwx retry, no spawn. Subuid mode: run `<exe> __sandbox-rm <path>` with the map on stdin.
/// Missing path = Ok.
pub fn remove_tree(map: &IdMap, path: &Path, exe: &Path) -> io::Result<()>;

resolve

  • Auto: use subuid when all of these hold:

    • newuidmap and newgidmap exist in /usr/bin, /bin, /usr/sbin or /sbin;
    • /etc/subuid and /etc/subgid each have an entry for the user (by name, from getpwuid, or by numeric uid);
    • that entry has count >= 65536. Take the first such range. A smaller range is accepted with a warning only if it is ≥ 1000.

    Otherwise use single.

  • Subuid: return an error instead of falling back.

  • Single: single().

enter_mapped_userns, subuid mode

  1. Before unsharing, create a pipe and fork a child. The child blocks reading the pipe.

  2. The parent runs unshare(CLONE_NEWUSER | CLONE_NEWNS), then writes one byte.

  3. The child runs these commands:

    • newuidmap <parent pid> 0 <uid> 1 1 <start> <n>
    • newgidmap <parent pid> 0 <gid> 1 1 <start> <n>

    It runs them via std::process::Command (the child is single-threaded, so this is allowed), then _exits with 0 or 1.

  4. The parent waits with waitpid and checks the status. On failure it returns an error including newuidmap's stderr.

  5. Do NOT write setgroups deny, because newgidmap needs allow. After mapping, call setgroups(&[]). Then make the mounts rec-private as today.

Single mode keeps today's code: setgroups deny and direct map writes. Move it here from helper.rs; the helper then calls enter_mapped_userns.

Dispatch:

  • cli::is_sandbox_invocation() becomes true for any argv[1] starting with __sandbox.
  • Add pub fn hidden_main() -> i32 in cli.rs, which routes:
    • __sandbox → sandbox::helper_main()
    • __sandbox-rm → idmap::rm_helper_main
    • __sandbox-unpack → crate::oci::unpack_helper_main(). This belongs to V-C; if it doesn't exist yet, add a stub returning 125 in oci/mod.rs that V-C replaces.
  • Use hidden_main from cli::main_with_args, forge-runner's main.rs and forge-web's main.rs.

Tests

  • parse_subid cases.
  • size and describe.
  • resolve(Auto) on this host yields subuid.
  • An integration test in tests/sandbox.rs (or a new tests/idmap.rs) that runs forge-runner __sandbox-rm on a TempDir holding a file chowned to 1000 by unshare --map-auto --map-root-user. The dir must be gone afterwards. Skip when unshare --map-auto fails.

Worker V-C: images with real ownership, in crates/forge-runner/src/oci/

  • unpack_layers gains a ownership: Ownership parameter:
    • pub enum Ownership { Flatten, Preserve { size: u32 } }.
    • Flatten is today's behaviour.
    • Preserve:
      • lchown every entry, including symlinks, to the tar header uid and gid.
      • Ids >= size become 0 and are counted in UnpackStats.remapped_ids.
      • setuid and setgid are still stripped.
      • Directories get their exact mode at the end, deepest first, instead of mode | 0o700.
    • Update all callers and tests.
  • ImageStore::with_idmap(map: IdMap, helper_exe: PathBuf):
    • When the map is subuid, the rootfs dir is rootfs/<digest>-s<uid start>-<n>, so a changed range re-unpacks. Single keeps rootfs/<digest>.
    • Unpacking in subuid mode runs <helper_exe> __sandbox-unpack from spawn_blocking. The JSON UnpackRequest { map, rootfs, layers, max_bytes } goes on stdin and UnpackStats JSON comes back on stdout.
    • pub fn unpack_helper_main() -> i32 (in oci/mod.rs or a new oci/helper.rs):
      • reads the request;
      • calls idmap::enter_mapped_userns;
      • calls unpack_layers(.., Ownership::Preserve { size: map.size() });
      • prints the stats and exits 0, or 125 with a "forge-sandbox:" line.
  • GC and staging cleanup remove trees with idmap::remove_tree(&map, path, &helper_exe), because subuid-owned files can't be deleted by the runner directly.
  • Tests:
    • Preserve unit tests that only check the remapping logic (a pure function, fn map_id(id, size) -> (u32, bool)).
    • Path naming per mode.
    • An integration test that unpacks a crafted layer with a file owned by 1000:1000 through forge-runner __sandbox-unpack, then checks that the host owner is start + 999. Skip when subuid is unavailable.

Worker V-B, after V-A: sandbox, executor, config, check, docs

  • SandboxRequest gains #[serde(default)] pub idmap: IdMap. The helper calls idmap::enter_mapped_userns(&req.idmap). The container spec's uid and gid mappings become a single line each, 0 0 <size>, which is relative to the helper's namespace.
  • Workspace ownership: after the job ends, the helper (still root in the userns) walks req.workspace without following symlinks and lchowns every entry whose uid or gid is not 0 to 0:0. The host runner then owns everything for artifacts and cleanup.
  • Executor: reap_job and any job-dir cleanup use idmap::remove_tree. BuiltinExecutor gets the map.
  • Config: [sandbox] userns = "auto" | "single" | "subuid" (default auto). It is resolved once at startup, logs describe(), and is passed to ImageStore::with_idmap and BuiltinExecutor. A failure to resolve in subuid mode is a startup error.
  • forge-runner check: prints the mode. In subuid mode it also checks that a probe job can chown 1000:1000 a file in /tmp.
  • docs/ci.md:
    • replace the single-uid caveat with the v2 behaviour;
    • document the requirements: shadow-utils newuidmap/newgidmap and an /etc/subuid and /etc/subgid entry (usermod --add-subuids 100000-165535 --add-subgids 100000-165535 <runner user>);
    • document the fallback;
    • explain that cached image files on the host are owned by subordinate ids.
  • Tests (tests/sandbox.rs, skip when subuid is unavailable):
    • a job that does touch /tmp/f && chown 1000:1000 /tmp/f && stat -c '%u %g' /tmp/f prints 1000 1000;
    • cat /proc/self/uid_map has size 1 + n;
    • a job that creates /workspace/a and chowns it to 1000 leaves a owned by the runner uid on the host;
    • the job dir is gone afterwards;
    • all existing single-mode tests still pass, with the map forced to single.