Tahinli/forge
public0 watching
★ 0
| 1 | # Backups |
| 2 | |
| 3 | This is the operator guide for forge backups. The design is in `docs/architecture.md` §8. Read this page before you rely on backups, and run the restore drill below at least once a quarter. |
| 4 | |
| 5 | ## What a backup set contains |
| 6 | |
| 7 | A set is a directory `forge-backup-<UTC stamp>/` with: |
| 8 | |
| 9 | | Path | Contents | |
| 10 | | --- | --- | |
| 11 | | `manifest.json` | Format version, forge version, creation time, base URL, the sha256 and size of every file, and for every repo its id, owner, name, `refs_sha256` and bundle path (or `null` for an empty repo). | |
| 12 | | `db/forge.db` | A consistent copy of the metadata database (users, issues, pull requests, settings, audit log, and so on), made with `VACUUM INTO`. | |
| 13 | | `repos/<repo id>.bundle` | `git bundle create --all` for every non-empty repo. Each bundle is imported and checked before the set is finished. | |
| 14 | | `keys/forge.key` | Only with `--include-key`. See below. | |
| 15 | |
| 16 | What a set does **not** contain: |
| 17 | |
| 18 | - **The instance key `forge.key`, unless you ask for it.** The key encrypts sealed values: 2FA secrets, webhook secrets and similar. A set without the key restores the data, but those sealed values stay unreadable. Without the key they are lost for good. Escrow the key separately, offline, for example in a password manager and on paper in a safe. Do not put it on the same off-site storage as the sets unless that storage is encrypted with a key you hold elsewhere. |
| 19 | - Server configuration (`config/forge.toml`), SSH host keys, the managed gitconfig and hooks. These are rewritten on boot, and the restore writes them again. |
| 20 | - Attachments and avatars that live outside the repos, if your deployment stores any. Check `data_dir` before you trust a restore is complete. |
| 21 | - **Release assets under `data_dir/blobs/`.** A set does not copy them. Back them up with the separate `rsync` step in "Release assets" below. |
| 22 | - **Git LFS objects under `data_dir/lfs/`.** A set does not copy them either. The `lfs_objects` rows are in the set, so a restore without the files gives LFS pointers that cannot be downloaded. Back them up with the `rsync` step in "LFS objects" below. |
| 23 | |
| 24 | Files in a set are mode 0600 and directories are mode 0700. Treat every set as private data, because it contains the full contents of every repo and the password hashes of every user. |
| 25 | |
| 26 | A set is written under a `.partial-…` name and renamed into place only when it is complete. A crash never leaves something that looks like a finished backup. |
| 27 | |
| 28 | ## Making a backup |
| 29 | |
| 30 | ```sh |
| 31 | forge --config /etc/forge/forge.toml admin backup --out /var/backups/forge |
| 32 | ``` |
| 33 | |
| 34 | Options: |
| 35 | |
| 36 | - `--include-key`: also copy `forge.key` into the set. Use this only when the set is stored where the key may live. |
| 37 | - `--repos-only` or `--db-only`: a partial set. These cannot be restored with `admin restore`. They exist for special cases such as shipping just the bundles. |
| 38 | |
| 39 | The command prints the path of the set and a summary. It exits non-zero on any failure, and then nothing is left behind. |
| 40 | |
| 41 | Automatic backups are configured in the `[backup]` section of `forge.toml`. The scheduler runs inside the server and writes the same kind of set. Its status and the last 20 runs are on `/admin/backups`: |
| 42 | |
| 43 | ```toml |
| 44 | [backup] |
| 45 | dir = "/var/backups/forge" # empty = disabled |
| 46 | interval_hours = 24 |
| 47 | keep = 7 # keeps the newest 7 complete sets; never touches other files |
| 48 | healthcheck_url = "" # optional, healthchecks.io style |
| 49 | fsck_interval_days = 7 |
| 50 | ``` |
| 51 | |
| 52 | > The `admin backup` command opens the metadata database from a second process. Turso does not support several processes using one database file at the same time, so run the CLI form with the server **stopped**. While the server is running, use the scheduler or "Run backup now" on `/admin/backups`. |
| 53 | |
| 54 | ## Checking a set |
| 55 | |
| 56 | ```sh |
| 57 | forge admin backup-verify /var/backups/forge/forge-backup-20261015T031500Z |
| 58 | ``` |
| 59 | |
| 60 | This recomputes every sha256 and size, checks that no unlisted file is present, imports every bundle into a scratch repo and compares its refs with the manifest, runs `integrity_check` on the database copy, and compares the repo list with the database. It prints every problem and exits non-zero if there is any. Scratch space is created next to the set, not in `/tmp`. |
| 61 | |
| 62 | Run it on every set you ship off-site, and after every copy you make. A set that has not been verified is not a backup. |
| 63 | |
| 64 | ## Shipping off-site |
| 65 | |
| 66 | Keep at least one copy at a second provider and, ideally, one copy that the server cannot delete (3-2-1-1-0 from `docs/architecture.md` §8.4). The server should hold no delete credentials for the backup storage. |
| 67 | |
| 68 | ### Option A: restic with an append-only rest-server |
| 69 | |
| 70 | On the backup host, run rest-server in append-only mode. A client can add snapshots but cannot delete or rewrite them: |
| 71 | |
| 72 | ```sh |
| 73 | rest-server --path /srv/restic --private-repos --append-only --listen 127.0.0.1:8000 |
| 74 | ``` |
| 75 | |
| 76 | Put it behind TLS (a reverse proxy) and give the forge host a password that only has access to its own repo. On the forge host: |
| 77 | |
| 78 | ```sh |
| 79 | export RESTIC_REPOSITORY='rest:https://forge-backup:PASSWORD@backup.example.invalid/forge' |
| 80 | export RESTIC_PASSWORD_FILE=/etc/forge/restic-password # 0600, root or the forge user only |
| 81 | restic init # once |
| 82 | restic backup /var/backups/forge |
| 83 | restic check --read-data-subset=1/20 # spot-check a slice of the data |
| 84 | ``` |
| 85 | |
| 86 | Append-only means `restic forget` and `restic prune` fail on the forge host. Do retention from a separate machine that holds the full credentials, which should not be the forge server. Keep a second restic key (`restic key add`) and store its password offline. |
| 87 | |
| 88 | ### Option B: rclone to object storage with object lock |
| 89 | |
| 90 | Use an `rclone` remote whose bucket has object lock in compliance mode (Backblaze B2, Cloudflare R2 bucket locks, or Wasabi compliance mode). Use `copy`, never `sync`: `sync` deletes remote files that are not present locally, which turns the server's own pruning into remote deletion. |
| 91 | |
| 92 | ```sh |
| 93 | rclone copy /var/backups/forge remote:forge-backups --immutable --checksum |
| 94 | rclone check /var/backups/forge remote:forge-backups --checksum |
| 95 | ``` |
| 96 | |
| 97 | `--immutable` stops rclone from overwriting or deleting existing remote files. Object lock is what makes the copies permanent, so set the retention period on the bucket and not only in rclone. |
| 98 | |
| 99 | ### Scheduling with systemd |
| 100 | |
| 101 | The job is a small script, so that the off-site upload runs only after the new set has verified. `/usr/local/bin/forge-backup-run`: |
| 102 | |
| 103 | ```sh |
| 104 | #!/bin/sh |
| 105 | set -eu |
| 106 | config=/etc/forge/forge.toml |
| 107 | out=/var/backups/forge |
| 108 | forge --config "$config" admin backup --out "$out" |
| 109 | # Stamps sort lexically, so the last name is the newest complete set. Partial sets are named .partial-… and do not match. |
| 110 | set_dir=$(ls -1d "$out"/forge-backup-* | sort | tail -n 1) |
| 111 | forge --config "$config" admin backup-verify "$set_dir" |
| 112 | restic backup "$out" |
| 113 | ``` |
| 114 | |
| 115 | `/etc/systemd/system/forge-backup.service`: |
| 116 | |
| 117 | ```ini |
| 118 | [Unit] |
| 119 | Description=Forge backup set and off-site copy |
| 120 | OnFailure=forge-backup-failed.service |
| 121 | |
| 122 | [Service] |
| 123 | Type=oneshot |
| 124 | User=forge |
| 125 | Group=forge |
| 126 | ExecStart=/usr/local/bin/forge-backup-run |
| 127 | ``` |
| 128 | |
| 129 | `/etc/systemd/system/forge-backup.timer`: |
| 130 | |
| 131 | ```ini |
| 132 | [Unit] |
| 133 | Description=Nightly forge backup |
| 134 | |
| 135 | [Timer] |
| 136 | OnCalendar=*-*-* 03:15:00 |
| 137 | Persistent=true |
| 138 | RandomizedDelaySec=10min |
| 139 | |
| 140 | [Install] |
| 141 | WantedBy=timers.target |
| 142 | ``` |
| 143 | |
| 144 | Enable it with `systemctl enable --now forge-backup.timer`. The `OnFailure=` unit should ping your alerting, or a healthchecks.io `/fail` URL. A missed success ping must alert too, so use a dead-man switch: the job pings `/start`, then the plain URL on success. |
| 145 | |
| 146 | ## Restoring |
| 147 | |
| 148 | Restoring is for a fresh machine or a fresh directory. The restore refuses a target that is not empty, and it refuses a set that fails verification. It never deletes anything. |
| 149 | |
| 150 | 1. **Get the set.** Copy the newest verified set from off-site storage to the restore host, for example with `restic restore latest --target /restore` or `rclone copy`. |
| 151 | 2. **Verify it.** |
| 152 | ```sh |
| 153 | forge admin backup-verify /restore/forge-backup-20261015T031500Z |
| 154 | ``` |
| 155 | Stop here if the result is not `OK`. |
| 156 | 3. **Get the key.** Place your escrowed `forge.key` where you will restore it. You do not have to place it before the restore: the restore can copy it from the set if the set has one. If the set has no key, you must copy the key in by hand before the server starts (step 6). |
| 157 | 4. **Restore into an empty directory.** |
| 158 | ```sh |
| 159 | forge admin restore --from /restore/forge-backup-20261015T031500Z --data-dir /var/lib/forge |
| 160 | ``` |
| 161 | The command verifies the set again, copies the database to `/var/lib/forge/forge.db`, recreates each repo with `init --bare`, fetches its bundle with `+refs/*:refs/*`, sets HEAD to the repo's default branch, and checks each repo's refs against the manifest. It also writes the managed gitconfig and hooks, and runs `git fsck --full` on each repo. It prints `WARNING:` lines for anything you must act on. |
| 162 | 5. **Point the config at the restored data.** Set `data_dir` and `database` in `forge.toml` to the values the restore printed, for example `database = "/var/lib/forge/forge.db"`. The database path is independent of `data_dir` in the config, so it must be set explicitly. |
| 163 | 6. **If the restore printed `NO INSTANCE KEY WAS RESTORED`:** copy the escrowed key to `/var/lib/forge/forge.key` with mode 0600, owned by the forge user. Do this before starting the server. If the server starts without the key, it creates a new one, and the sealed data stays unreadable. |
| 164 | 7. **Run the integrity check.** |
| 165 | ```sh |
| 166 | forge --config /etc/forge/forge.toml admin fsck --full |
| 167 | ``` |
| 168 | It runs `git fsck` on every repo and `integrity_check` on the database, and exits non-zero on any problem. |
| 169 | 8. **Start the server** and check: you can log in, the issues and pull requests are there, a clone over HTTP and SSH works, and a push works. |
| 170 | 9. **Record the time.** The time from the start of step 1 to the end of step 8 is your real recovery time. Write it down. |
| 171 | |
| 172 | If the restore fails, the target directory is left as it was when the failure happened. Do not start the server from it. Read the error, fix the cause (for example, free disk space or the git version), and restore into a new empty directory. Do not delete the failed directory until someone has looked at it. |
| 173 | |
| 174 | ## Quarterly restore drill |
| 175 | |
| 176 | Do this on a machine that is not the forge server, once every three months. Keep the answers in your ops log. |
| 177 | |
| 178 | - [ ] Pick the newest set in each off-site location, and the set from about 90 days ago. |
| 179 | - [ ] Download each one. Record the download time and size. |
| 180 | - [ ] Run `admin backup-verify` on each. Record the result. Any problem is an incident. |
| 181 | - [ ] Restore the newest set into an empty directory with `admin restore`. Time it. |
| 182 | - [ ] Copy the escrowed `forge.key` in by hand, from the offline copy, not from the set. Confirm the key matches the one in use: a sealed value (for example, a 2FA login) must decrypt. |
| 183 | - [ ] Run `admin fsck --full`. Expect no problems. |
| 184 | - [ ] Start a throwaway server on the restored data. Log in as an admin, open one issue, clone one repo over HTTP, clone one over SSH. |
| 185 | - [ ] Check that the sets you have are still readable with the keys you hold offline (restic: `restic snapshots`, and `restic check` from the second key). |
| 186 | - [ ] Confirm that the monitoring for the backup jobs and the fsck job still reports to you. |
| 187 | - [ ] Update the recovery time and the date of the last drill in your ops log. If the measured time is longer than your target, fix the cause before the next drill. |
| 188 | |
| 189 | ## Failover fencing |
| 190 | |
| 191 | Restoring a backup on another machine does not by itself make that machine the only writer. Before you bring up a restored standby as the primary, follow `docs/architecture.md` §8.5: |
| 192 | |
| 193 | 1. Stop writes on the old primary (block the HTTP and SSH listeners, or put the instance in read-only mode). |
| 194 | 2. Fence the old primary: revoke its replication and backup credentials, and make sure it cannot accept writes. Do this before promotion. A DNS change or a health check does not stop the old primary from coming back as a second writer. |
| 195 | 3. Confirm the last snapshot you restored and the last ref reconcile. Record the data-loss window. |
| 196 | 4. Restore on the standby, then run `quick_check` and `fsck`. |
| 197 | 5. Promote the standby and switch DNS. |
| 198 | 6. Record the incident and the actual recovery time. |
| 199 | |
| 200 | ## Standby replication |
| 201 | |
| 202 | A standby is a passive SSH host at another provider. It holds bare copies of every repo and a |
| 203 | rolling set of database snapshots. It does not run forge. Replication is off unless `[replication] |
| 204 | target` is set. |
| 205 | |
| 206 | ### What the standby receives |
| 207 | |
| 208 | - Every repo, as a bare repo at `<base>/repos/<repo id>.git`. A push is sent after each ref update, |
| 209 | debounced by 5 seconds. Every `reconcile_minutes` (default 60) every repo is pushed again, and the |
| 210 | refs checksum of the standby copy is compared with the local one. A mismatch triggers one re-push. |
| 211 | A second mismatch is recorded as an error. |
| 212 | - A database snapshot every `db_snapshot_minutes` (default 15). It is taken with `VACUUM INTO`, |
| 213 | checked with `PRAGMA integrity_check`, uploaded to `<base>/db/forge-<UTC stamp>.db` through a |
| 214 | temporary name, and then `<base>/db/latest` is pointed at it. The standby keeps the newest 24. |
| 215 | - Deleted repos: at the next reconcile, the standby copy of exactly that repo is removed. |
| 216 | |
| 217 | Nothing else is written. `<base>/keys/` belongs to the operator. Forge never writes to it, and the |
| 218 | instance key should be escrowed there by hand, as described under "Restoring". |
| 219 | |
| 220 | Status is on `/admin/replication` (admin only): the target, the last database snapshot, and one row |
| 221 | per repo (ok, lagging, error, or never replicated). "Reconcile now" runs a full pass at once. It is |
| 222 | audited as `replication_reconcile_started`. If `[replication] healthcheck_url` is set, it is pinged |
| 223 | after each reconcile: success when every step worked, `/fail` with the errors otherwise. |
| 224 | |
| 225 | ### Setting up the standby account |
| 226 | |
| 227 | 1. On the standby host, create a dedicated unix account with no password and a login shell that can |
| 228 | run commands (`/bin/sh`). Create `<base>` owned by that account, mode 0700. Git must be installed, |
| 229 | version 2.38 or newer. |
| 230 | 2. On the forge host, generate a key used only for replication, with no passphrase: |
| 231 | `ssh-keygen -t ed25519 -N '' -f /etc/forge/standby_key`, and set it to mode 0600, owned by the |
| 232 | forge user. |
| 233 | 3. On the standby, add the public key to `~<account>/.ssh/authorized_keys` with the `restrict` option |
| 234 | and nothing that grants more: |
| 235 | |
| 236 | ``` |
| 237 | restrict ssh-ed25519 AAAA...rest-of-public-key forge-replication |
| 238 | ``` |
| 239 | |
| 240 | `restrict` turns off port forwarding, agent forwarding, X11 forwarding and the pty. It does not |
| 241 | force a command, because replication runs `git receive-pack`, `git for-each-ref`, `cat`, `ls`, |
| 242 | `rm` and `mkdir` through ssh. The account's file permissions are what keep it inside `<base>`. |
| 243 | Keep `<base>` the only directory the account can write to, and do not give it sudo. |
| 244 | 4. Pin the standby's host key on the forge host. Get it over a channel you trust (the provider's |
| 245 | console, or a fingerprint you already hold), compare the fingerprint with the server's own |
| 246 | `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub`, and then write the key line to a file of its |
| 247 | own: |
| 248 | |
| 249 | ``` |
| 250 | ssh-keyscan -t ed25519 -p 22 standby.example > /etc/forge/standby_known_hosts |
| 251 | ``` |
| 252 | |
| 253 | Do not use `ssh-keyscan` without the fingerprint check. Forge always runs ssh with |
| 254 | `StrictHostKeyChecking=yes`, `UserKnownHostsFile` set to this file, and `IdentitiesOnly=yes`, so |
| 255 | a changed host key stops replication instead of being accepted. If the standby is rebuilt, repeat |
| 256 | this step and check the new fingerprint. |
| 257 | 5. Configure forge: |
| 258 | |
| 259 | ```toml |
| 260 | [replication] |
| 261 | target = "forge-sync@standby.example:/srv/forge-standby" |
| 262 | ssh_port = 22 |
| 263 | identity_file = "/etc/forge/standby_key" |
| 264 | known_hosts_file = "/etc/forge/standby_known_hosts" |
| 265 | db_snapshot_minutes = 15 |
| 266 | reconcile_minutes = 60 |
| 267 | healthcheck_url = "" |
| 268 | ``` |
| 269 | |
| 270 | The target is `[user@]host:/absolute/path`. The path may use only `A-Z a-z 0-9 . _ - /`, with no |
| 271 | `..`. Restart forge. The worker starts at boot and runs a snapshot and a reconcile straight away. |
| 272 | 6. Check `/admin/replication`. Every repo should show "ok" within a few minutes. On the standby, |
| 273 | `ls <base>/db` should show `latest` and at least one `forge-*.db`. |
| 274 | |
| 275 | ### Failover runbook |
| 276 | |
| 277 | Follow `docs/architecture.md` §8.5. The steps below are specific to the standby layout. |
| 278 | |
| 279 | 1. Stop writes on the primary, and fence it. Revoke its replication key (remove the line from the |
| 280 | standby's `authorized_keys`), and block its HTTP and SSH listeners. Do this before anything else. |
| 281 | Without fencing, the old primary can come back and accept pushes. |
| 282 | 2. Record the data-loss window: the time of `db/latest` (from its file name), and the last |
| 283 | successful reconcile on `/admin/replication`, if the primary still answers. |
| 284 | 3. On a new host with the same forge version, create the data directory, and copy the escrowed |
| 285 | `forge.key` into `<data_dir>/forge.key`. Without this key sealed values cannot be read. |
| 286 | 4. Put the snapshot in place at the path the config's `database` setting names: |
| 287 | `cp -L <base>/db/latest <database path>`. `-L` follows the `latest` link, so the copy is a plain file. |
| 288 | 5. Move each repo into the local layout. The local path is `<data_dir>/repos/<id[0..2]>/<id[2..4]>/<id>.git`, |
| 289 | where `<id>` is the lowercase repo id. For example: |
| 290 | |
| 291 | ```sh |
| 292 | for repo in <base>/repos/*.git; do |
| 293 | id=$(basename "$repo" .git) |
| 294 | lower=$(printf '%s' "$id" | tr 'A-Z' 'a-z') |
| 295 | dest="<data_dir>/repos/$(printf '%s' "$lower" | cut -c1-2)/$(printf '%s' "$lower" | cut -c3-4)/$lower.git" |
| 296 | mkdir -p "$(dirname "$dest")" && cp -a "$repo" "$dest" |
| 297 | done |
| 298 | ``` |
| 299 | |
| 300 | 6. Run `forge admin fsck --full`. Expect no problems. Run `quick_check` on the database. |
| 301 | 7. Start forge on the new host, log in as an admin, and clone one repo over HTTP and one over SSH. |
| 302 | 8. Promote the new host and switch DNS. Record the incident and the actual recovery time. |
| 303 | |
| 304 | ## Release assets |
| 305 | |
| 306 | Release asset bytes live in `<data_dir>/blobs/assets/<asset id>`, outside the database and the repos. The database rows that point at them are in every backup set, so a restore without the blobs leaves downloads returning 404. Keep the blobs in step with the sets: |
| 307 | |
| 308 | - Blobs are immutable. An asset is written once under its id and never rewritten, only added or deleted. The writer streams to a temporary file and renames it into place when complete, so a copy taken at any moment never holds half of an asset that is listed in the database. |
| 309 | - Copy the directory with a plain `rsync -a --delete <data_dir>/blobs/ <backup target>/blobs/`. Do it after each backup set, or on the same schedule. Copying during an upload is safe: the in-flight file is a temporary name that rsync may skip or copy, and the row that references it is written only after the rename. |
| 310 | - On restore, copy `blobs/` back into the new `data_dir` before starting the server. A row whose blob is missing is harmless to the server but its download returns 404; check the release pages after a drill. |
| 311 | - Deleting a release or a repo deletes its blobs at once. The blob directory has no other index, so it can be checked against the database with a `SELECT id FROM release_assets` list. |
| 312 | |
| 313 | ## LFS objects |
| 314 | |
| 315 | Git LFS object bytes live in `<data_dir>/lfs/<oid[0..2]>/<oid[2..4]>/<oid>`, outside the database and the repos. The `lfs_objects` rows that link them to repos are in every backup set. Copy the directory with the same `rsync` step as release blobs, after each backup set or on the same schedule: |
| 316 | |
| 317 | ```sh |
| 318 | rsync -a --delete <data_dir>/lfs/ <backup target>/lfs/ |
| 319 | ``` |
| 320 | |
| 321 | - **Restore:** copy `lfs/` back into the new `data_dir` before starting the server. Check one LFS file after a drill with `git lfs fetch` on a clone, or by opening a blob page that shows "Stored with Git LFS" and downloading it. |
| 322 | - **Why the copy is safe:** objects are named by their sha256, so a name always holds the same bytes. An object is written to a temporary file, checked against its hash and size, fsynced, and then renamed into place. A copy taken at any moment sees either the whole object or nothing under its final name. Objects are never rewritten, only added, and removed by `admin lfs-gc`, which deletes only files with no row in any repo that are older than a day. |