forge
ExploreLog in

Tahinli/forge

public
0 watching
★ 0
18.8 KiB

Wave C: orgs, email, wiki and polish (contract between workers)

The global rules in ~/.cache/forge-orch/waveb-common.md apply. Background research:

  • ~/.cache/forge-orch/orgs-study.md: every place where the owner is assumed to be a user.
  • ~/.cache/forge-orch/email-study.md: the im and iz mail designs.
  • ~/.cache/forge-orch/polish-audit.md: what exists already.

Shared-file rules (many workers in parallel)

  • Migrations: each worker owns exactly one number. Add one line to the array in crates/forge-core/src/store/migrate.rs, re-reading the file first.

    Migration Owner
    0025_orgs O1
    0026_mail M1
    0027_wiki W1
    0028_discussions D1
    0029_snippets S1
    0030_ui_prefs T1
    0031_advisories A1
    0032_repo_transfers P1
  • Store code: put it in a new crates/forge-core/src/store/<feature>.rs. Don't grow store/mod.rs, apart from the mod line and changes your section explicitly requires.

  • Shared files (pages/mod.rs route lists, views/mod.rs (shell and repo tabs), names.rs reserved words, lib.rs, style/main.scss): small hunks, and re-read the file right before every edit.

  • SCSS: each feature appends its own clearly commented block at the end of style/main.scss, or a partial if the file already uses partials.

  • Audit: every state-changing action writes an audit entry, as the existing features do. Use new action names, and grep the tests that assert audit counts. One of them asserts a total (48); if you add actions, update it.

  • No JS anywhere. Use forms and links only.

  • Tests: unit tests in the store module, plus one e2e file per feature in crates/forge-web/tests/<feature>_e2e.rs using tests/common/mod.rs.

O1: organizations core (wave 1)

Model: an org is a row in users with kind = 'org'. That keeps one namespace (/{name} works for both), keeps repos.owner_id → users(id), and leaves every owner_name join unchanged.

Migration 0025_orgs.sql:

ALTER TABLE users ADD COLUMN kind TEXT NOT NULL DEFAULT 'user' CHECK (kind IN ('user','org'));
ALTER TABLE users ADD COLUMN description TEXT;          -- only if no such column exists (check 0002_profile)
CREATE TABLE org_settings (org_id TEXT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
  default_repo_role TEXT NOT NULL DEFAULT 'read' CHECK (default_repo_role IN ('none','read','write')),
  members_can_create_repos INTEGER NOT NULL DEFAULT 1);
CREATE TABLE org_members (org_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  role TEXT NOT NULL CHECK (role IN ('owner','member')), created_at TEXT NOT NULL,
  PRIMARY KEY (org_id, user_id));
CREATE TABLE teams (id TEXT PRIMARY KEY, org_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  name TEXT NOT NULL, name_folded TEXT NOT NULL, description TEXT NOT NULL DEFAULT '',
  role TEXT NOT NULL CHECK (role IN ('read','triage','write','maintain','admin')),
  all_repos INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL, UNIQUE (org_id, name_folded));
CREATE TABLE team_members (team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
  user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, PRIMARY KEY (team_id, user_id));
CREATE TABLE team_repos (team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
  repo_id TEXT NOT NULL REFERENCES repos(id) ON DELETE CASCADE, PRIMARY KEY (team_id, repo_id));

Org rows:

  • They get an unusable password_hash ("!org").
  • Hard rule: an org can never authenticate. Every authentication path must reject kind = 'org' explicitly:
    • password login, sessions, tokens, SSH keys (including the add-key paths), LFS tokens, TOTP;
    • admin create-user/reset-password/set-username (refuse orgs);
    • the CI actor. Add a test for each.
  • The User struct gets kind: AccountKind { User, Org }, with a is_org() helper.

store/orgs.rs:

  • create_org(name, creator_id): the creator becomes owner, and the org gets the same name validation as a username, plus org_settings defaults.
  • org_members, add_org_member, set_org_member_role, and remove_org_member, which refuses to remove the last owner.
  • Teams CRUD, team members and team repos.
  • orgs_for_user.
  • delete_org, which refuses while the org owns repos.
  • org_repo_role(org_id, user_id, repo_id) -> Option<Role>: owner gives admin. Otherwise the max of default_repo_role (for members) and the roles of the user's teams that include the repo, or that have all_repos.

authz: extend the existing authorization input (see crates/forge-core/src/authz* and its callers in access.rs) with org_role: Option<Role>, and load it in the same place collaborators are loaded. The effective role is the max of the collaborator role and the org role. An owner check (owner_id == actor.id) never matches an org.

Audit every existing site that compares owner_id == user.id (orgs-study lists them: internal.rs:526/820, repo.rs:282, admin_users.rs, notify.rs) and route it through the effective role instead.

Notifications: wherever "the repo owner" is a recipient, an org owner expands to the org's owners.

User deletion: refuse to delete a user who is the last owner of any org. Show the message on the admin page and the CLI.

Tests:

  • the role math;
  • auth refusal for orgs;
  • the last-owner rules;
  • a store-level repo_by_owner_and_name for an org-owned repo.

O2: organizations UI (wave 2, after O1)

Routes: the prefix org is already reserved.

  • GET/POST /org/new. Allowed for everyone, unless the config [orgs] creation = "admin"; the default is "all".
  • /org/{name}/members: add by username (owners only), change role, remove, leave.
  • /org/{name}/teams, /org/{name}/teams/{team}: members, repos, role.
  • /org/{name}/settings: description, default role, members-can-create, delete.

Other pages:

  • /{name} (profile.rs): when the target is an org, show the repos visible to the viewer, the members list and the teams count.
  • New repo form: an owner <select> listing self plus the orgs where the viewer may create.
  • Transfer to an org is handled by P1, not here. O2 does not edit repo_rename.rs.
  • explore: orgs are listed separately or marked.
  • Admin users page: shows the kind.

Webhooks: the payload repository.owner stays the name string. Add "owner_kind": "user"|"org".

E2e: create an org, add a member, create a team with write on a repo, then:

  • that member can push over HTTPS;
  • a non-member can't see a private org repo (404);
  • the org can't log in.

M1: mail core (wave 1). Follow iz's outbox and im's token design (see the study).

  • Crate: lettre (latest) with smtp-transport, builder, tokio1-rustls, ring, webpki-roots, pool, and default-features = false.
  • 0026_mail.sql:
    • mail_settings, a singleton row: host, port, security ('starttls'|'tls'|'none'), username, password_sealed BLOB, from_addr, from_name, enabled;
    • mail_outbox (id, idem_key TEXT UNIQUE, to_addr, subject, body_text, created_at, attempts, next_attempt_at, lease_until, sent_at, last_error);
    • users.email_verified_at TEXT.
  • Sealing: the password is sealed with the existing instance key (forge.key, XChaCha20-Poly1305). Find the existing secret-sealing helper (used by repo secrets, CI and mirrors) and reuse it. Debug impls redact the password.
  • Transport: a Mailer trait (async fn send(&self, msg) -> Result). The implementations are:
    • SmtpMailer (lettre, 30 s timeout);
    • LogMailer, which writes to <data_dir>/mail-log/ and is enabled by [mail] transport = "log" in forge.toml, for dev and tests;
    • a test FakeMailer that collects messages.
  • enqueue(idem_key, to, subject, body): idempotent. The background worker leases rows and sends them, retrying up to 5 times with exponential backoff (1m, 5m, 30m, 2h, 12h), then gives up. Wire it into boot the same way the mirror worker is.
  • Admin page /admin/mail: the settings form, plus a "send test mail" button, and a list of the last 50 outbox rows (status and error, never the body).
  • Templates: plain text, built in one module crates/forge-web/src/mail/templates.rs as functions returning (subject, body). i18n will hook in later through that module only.
  • Tests: outbox idempotency, the retry schedule, the lease, sealing round-trip, and Debug redaction.

M2: mail flows (wave 2, after M1)

  • Password reset (/forgot, /reset/{token}):
    • Tokens are 32 random bytes, stored as sha256, with a 60-minute TTL.
    • Single use, enforced by an atomic UPDATE … SET used_at=? WHERE token_hash=? AND used_at IS NULL AND expires_at>? RETURNING user_id.
    • Mail goes only to verified addresses.
    • /forgot always shows the same response. The lookup and enqueue happen in a spawned task, so response timing doesn't reveal whether the account exists.
    • A successful reset revokes all sessions and is audited.
    • Rate-limit with the existing throttle.rs.
  • Email verification: setting or changing the email (profile and setup) sends a verify link (24 h TTL, single use) and clears email_verified_at. The settings page shows the status and a "resend" button.
  • Invites: registration = "invite". Admins invite by email from /admin/users, with a 7-day single-use token. /register?invite= creates the account with that email pre-verified.
  • Notification emails:
    • A per-user setting, on by default, that applies only to verified addresses.
    • Send for the existing notification kinds. Read notify.rs for the list.
    • Each mail has a link and the threading headers Message-ID, In-Reply-To and References, keyed per issue or PR.
    • The idempotency key is the notification id.
  • E2e with transport = "log" or FakeMailer: the reset flow end to end, token reuse refused, an expired token refused, an unverified address gets no mail, and the invite flow.

W1: wiki (wave 1)

  • Storage: a bare repo at <repo path minus .git>.wiki.git, created lazily on the first page save. Add a helper next to repo_path in forge-git/src/repo.rs.
  • 0027_wiki.sql: repos.wiki_enabled INTEGER NOT NULL DEFAULT 1.
  • Pages:
    • /{o}/{r}/wiki shows Home, or "create the first page";
    • /{o}/{r}/wiki/{page};
    • /{o}/{r}/wiki/{page}/edit;
    • /{o}/{r}/wiki/-/new;
    • /{o}/{r}/wiki/-/pages;
    • /{o}/{r}/wiki/{page}/history. Pages are <Title-With-Dashes>.md files on main. Markdown is rendered with the existing renderer, and wiki links [[Page]] resolve.
  • Web edits: commit via plumbing with the forge-git builder (hash-object -w --stdin, mktree or read-tree/update-index on a temp index, commit-tree, update-ref with the old value). The author is the user's name and email. A conflicting concurrent edit gets an error page, not a lost write.
  • Permissions: read needs repo Read. Edit and push need repo Write. Hide the tab when the wiki is disabled.
  • Git access:
    • Clone and push over HTTPS at /{o}/{r}.wiki.git (and /{o}/{r}.wiki), and over SSH at o/r.wiki.git.
    • Extend forge-ssh/src/parse.rs and git_http.rs resolution.
    • Repo names must not end in .wiki; add that to names.rs.
    • Pre-receive applies to wiki pushes, but skips the default-branch protections.
  • Coverage:
    • repo delete and trash move the wiki too;
    • backup bundles and verify include the wiki when present (verify.rs hard-codes repos/{id}.bundle, so add repos/{id}.wiki.bundle);
    • restore and admin fsck handle it too;
    • replication carries it, so read replication/ and extend it the same way as the main repo;
    • mirrors don't mirror wikis (document this).
  • E2e:
    • create a page on the web, clone the wiki over HTTPS and see it, push a change and see it on the web;
    • a Read-only user can't edit;
    • backup then verify includes the wiki.

D1: discussions (wave 1)

  • 0028_discussions.sql:
    • repos.discussions_enabled INTEGER NOT NULL DEFAULT 0;
    • discussions (id, repo_id, number, category ('general','ideas','qa','announcements'), title, body, author_id, answer_comment_id, locked, created_at, updated_at). Number them with the same per-repo counter as issues and PRs if one exists, or their own counter otherwise; check how issues number.
    • discussion_comments (id, discussion_id, parent_id NULL /* one level of replies */, author_id, body, created_at, updated_at, deleted_at).
  • Pages:
    • /{o}/{r}/discussions, with a category filter;
    • /{o}/{r}/discussions/new;
    • /{o}/{r}/discussions/{n}, showing replies, "mark as answer" in qa (author or Triage+), lock (Triage+), edit and delete own.
    • Only Maintain+ may post in announcements.
  • Reuse reactions and the markdown renderer. Mentions notify, the same way issues do.
  • A repo settings toggle and a repo tab.
  • E2e: create, reply, nested reply, answer, lock refusal, and the toggle hiding the tab.

S1: snippets (wave 1)

  • 0029_snippets.sql:
    • snippets (id, owner_id, title, description, visibility ('public','unlisted','private'), created_at, updated_at);
    • snippet_files (snippet_id, name, content, position);
    • snippet_revisions (id, snippet_id, author_id, created_at, files_json). Limits: 10 files, 1 MiB total.
  • Routes: use a reserved prefix, adding snippets to names.rs if needed.
    • /snippets: your snippets plus public ones;
    • /snippets/new;
    • /snippets/{id}, with syntax highlighting via the existing highlighter, and markdown rendered for .md;
    • /snippets/{id}/edit;
    • /snippets/{id}/revisions;
    • raw on the user-content origin at /-/snippets/{id}/{file}. Private snippets use the same token mechanism as private raw files.
  • unlisted means viewable by link, not listed.
  • E2e: create, edit (revision recorded), private snippet hidden from others (404), raw served with the right content type.

T1: theme and language preferences (wave 1)

  • 0030_ui_prefs.sql:
    • users.theme TEXT NOT NULL DEFAULT 'auto' CHECK (theme IN ('auto','light','dark'));
    • users.language TEXT NOT NULL DEFAULT 'auto'.
  • Theme: views/mod.rs shell emits <html data-theme="light|dark"> when the preference isn't auto.
    • SCSS: the existing dark palette under @media (prefers-color-scheme: dark) must apply only to :root:not([data-theme="light"]), and also under :root[data-theme="dark"]. The same goes for the highlight CSS.
    • The settings page /settings/appearance has theme and language selects. I18 provides the language list; until then, offer only auto and en.
    • Anonymous visitors get a footer form POSTing to /-/theme, which sets a forge_theme cookie (Lax, 1 year). The cookie is honoured when no user is signed in.
  • E2e: the setting persists, the html attribute is rendered, and the anonymous cookie works.

I18: i18n infrastructure (wave 1). Follow im's design (~/Documents/Code/Rust/im/crates/im-web/src/i18n.rs).

  • crates/forge-web/src/i18n.rs defines Lang { En, Tr }, a Key enum, and an exhaustive t(lang, key) -> &'static str, plus fmt helpers for strings with arguments.
  • Language selection: the user's language column (T1 adds it; until it lands, read it defensively or wait), else Accept-Language, else En. shell sets <html lang>.
  • Wave 1 scope: the shell, nav, footer, login, setup, register, settings pages, and the 404 and error pages. Turkish translations must be real Turkish (the user is Turkish).
  • Expose Lang to page code through a Cx helper (lang(cx)).
  • Wave 3 (later): one worker per page group converts the rest. Don't do that now.
  • Tests: an exhaustiveness test (every Key non-empty in every Lang), Accept-Language parsing, and an e2e that the Turkish login page renders Turkish.

A1: security advisories (wave 1). Opt-in, because it downloads data.

  • Config: [advisories] enabled = false, ecosystems = ["crates.io","npm","Go"], refresh_hours = 24.
  • Data: OSV per-ecosystem dumps from https://osv-vulnerabilities.storage.googleapis.com/<ecosystem>/all.zip, downloaded with the existing HTTP client setup (timeouts, size cap 500 MiB) into <data_dir>/advisories/<eco>/. Parse only id, summary, aliases, severity, affected[].package.{ecosystem,name}, affected[].ranges (SEMVER and ECOSYSTEM events) and affected[].versions.
  • 0031_advisories.sql:
    • advisories (id, ecosystem, package, summary, severity, ranges_json, versions_json, modified);
    • repo_dependencies (repo_id, ecosystem, package, version, manifest_path);
    • repo_alerts (repo_id, advisory_id, package, version, state ('open','dismissed','fixed'), created_at, updated_at).
  • Scanning: after a push to the default branch (hook into post_update.rs), read Cargo.lock, package-lock.json (v2 and v3 packages) and go.sum from the tree (root and up to depth 3). Recompute dependencies, open new alerts, and mark disappeared ones fixed. Rescan all repos after each advisory refresh.
  • Version matching: semver for crates.io and npm, and Go pseudo-versions as semver. Use the semver crate.
  • Page: /{o}/{r}/security (Write+ only) lists alerts with dismiss and reopen. New alerts notify the repo's admins.
  • Tests: parsers for all three lockfiles, range matching (introduced, fixed, last_affected), and an e2e with a hand-written OSV fixture dir and a pushed Cargo.lock.

P1: repo transfer acceptance (wave 2, after O1)

  • 0032_repo_transfers.sql: repo_transfers (repo_id PRIMARY KEY, from_owner_id, to_owner_id, initiated_by, created_at, expires_at).
  • Flow:
    • A transfer now creates a pending row with a 7-day expiry and notifies the target (the user, or an org's owners).
    • The target accepts or declines on /{o}/{r}/transfer or the notification. Accepting runs the existing transfer_repo.
    • The initiator may cancel.
    • Admin transfers skip acceptance.
  • Org targets: the target may be an org where the actor is an owner (this moved here from O2). For an org target, any owner of the org may accept.
  • E2e: initiate, the repo is unchanged until accepted, accept moves it and the redirect works, decline, expiry, and a transfer to an org.

Round-2 follow-ups (from the round-1 reports)

  • F-A (advisories):
    • add a Security tab in the repo tabs, shown to Write+ only and only when advisories are enabled;
    • new alerts create a notification for the repo admins (owner, or the org owners plus admin collaborators) through notify.rs, plus a test.
  • F-D (discussions):
    • the notifications inbox lists discussion_notifications (mentions, and replies to your discussion);
    • lock is enforced in the store too, not only in the pages;
    • deleting a user reassigns their discussions and comments to the ghost user, the same way issues do (store/user_delete.rs);
    • tests for each.
  • F-W (wiki):
    • CI job tokens can read (clone) the wiki of the job's repo;
    • wiki pushes write an audit entry;
    • document in docs/ (wherever mirrors are documented) that mirrors don't include wikis;
    • add a backup round-trip test that includes a wiki.