Backups

This is the operator guide for forge backups. The design is in docs/architecture.md §8. Read this page before you rely on backups, and run the restore drill below at least once a quarter.

What a backup set contains

A set is a directory forge-backup-<UTC stamp>/ with:

Path	Contents
`manifest.json`	Format version, forge version, creation time, base URL, the sha256 and size of every file, and for every repo its id, owner, name, `refs_sha256` and bundle path (or `null` for an empty repo).
`db/forge.db`	A consistent copy of the metadata database (users, issues, pull requests, settings, audit log, and so on), made with `VACUUM INTO`.
`repos/<repo id>.bundle`	`git bundle create --all` for every non-empty repo. Each bundle is imported and checked before the set is finished.
`keys/forge.key`	Only with `--include-key`. See below.

What a set does not contain:

The instance key forge.key, unless you ask for it. The key encrypts sealed values: 2FA secrets, webhook secrets and similar. A set without the key restores the data, but those sealed values stay unreadable. Without the key they are lost for good. Escrow the key separately, offline, for example in a password manager and on paper in a safe. Do not put it on the same off-site storage as the sets unless that storage is encrypted with a key you hold elsewhere.
Server configuration (config/forge.toml), SSH host keys, the managed gitconfig and hooks. These are rewritten on boot, and the restore writes them again.
Attachments and avatars that live outside the repos, if your deployment stores any. Check data_dir before you trust a restore is complete.
Release assets under data_dir/blobs/. A set does not copy them. Back them up with the separate rsync step in "Release assets" below.
Git LFS objects under data_dir/lfs/. A set does not copy them either. The lfs_objects rows are in the set, so a restore without the files gives LFS pointers that cannot be downloaded. Back them up with the rsync step in "LFS objects" below.

Files in a set are mode 0600 and directories are mode 0700. Treat every set as private data, because it contains the full contents of every repo and the password hashes of every user.

A set is written under a .partial-… name and renamed into place only when it is complete. A crash never leaves something that looks like a finished backup.

Making a backup

forge --config /etc/forge/forge.toml admin backup --out /var/backups/forge

Options:

--include-key: also copy forge.key into the set. Use this only when the set is stored where the key may live.
--repos-only or --db-only: a partial set. These cannot be restored with admin restore. They exist for special cases such as shipping just the bundles.

The command prints the path of the set and a summary. It exits non-zero on any failure, and then nothing is left behind.

Automatic backups are configured in the [backup] section of forge.toml. The scheduler runs inside the server and writes the same kind of set. Its status and the last 20 runs are on /admin/backups:

[backup]
dir = "/var/backups/forge"   # empty = disabled
interval_hours = 24
keep = 7                     # keeps the newest 7 complete sets; never touches other files
healthcheck_url = ""         # optional, healthchecks.io style
fsck_interval_days = 7

The admin backup command opens the metadata database from a second process. Turso does not support several processes using one database file at the same time, so run the CLI form with the server stopped. While the server is running, use the scheduler or "Run backup now" on /admin/backups.

Checking a set

forge admin backup-verify /var/backups/forge/forge-backup-20261015T031500Z

This recomputes every sha256 and size, checks that no unlisted file is present, imports every bundle into a scratch repo and compares its refs with the manifest, runs integrity_check on the database copy, and compares the repo list with the database. It prints every problem and exits non-zero if there is any. Scratch space is created next to the set, not in /tmp.

Run it on every set you ship off-site, and after every copy you make. A set that has not been verified is not a backup.

Shipping off-site

Keep at least one copy at a second provider and, ideally, one copy that the server cannot delete (3-2-1-1-0 from docs/architecture.md §8.4). The server should hold no delete credentials for the backup storage.

Option A: restic with an append-only rest-server

On the backup host, run rest-server in append-only mode. A client can add snapshots but cannot delete or rewrite them:

rest-server --path /srv/restic --private-repos --append-only --listen 127.0.0.1:8000

Put it behind TLS (a reverse proxy) and give the forge host a password that only has access to its own repo. On the forge host:

export RESTIC_REPOSITORY='rest:https://forge-backup:PASSWORD@backup.example.invalid/forge'
export RESTIC_PASSWORD_FILE=/etc/forge/restic-password   # 0600, root or the forge user only
restic init                                              # once
restic backup /var/backups/forge
restic check --read-data-subset=1/20                     # spot-check a slice of the data

Append-only means restic forget and restic prune fail on the forge host. Do retention from a separate machine that holds the full credentials, which should not be the forge server. Keep a second restic key (restic key add) and store its password offline.

Option B: rclone to object storage with object lock

Use an rclone remote whose bucket has object lock in compliance mode (Backblaze B2, Cloudflare R2 bucket locks, or Wasabi compliance mode). Use copy, never sync: sync deletes remote files that are not present locally, which turns the server's own pruning into remote deletion.

rclone copy /var/backups/forge remote:forge-backups --immutable --checksum
rclone check /var/backups/forge remote:forge-backups --checksum

--immutable stops rclone from overwriting or deleting existing remote files. Object lock is what makes the copies permanent, so set the retention period on the bucket and not only in rclone.

Scheduling with systemd

The job is a small script, so that the off-site upload runs only after the new set has verified. /usr/local/bin/forge-backup-run:

#!/bin/sh
set -eu
config=/etc/forge/forge.toml
out=/var/backups/forge
forge --config "$config" admin backup --out "$out"
# Stamps sort lexically, so the last name is the newest complete set. Partial sets are named .partial-… and do not match.
set_dir=$(ls -1d "$out"/forge-backup-* | sort | tail -n 1)
forge --config "$config" admin backup-verify "$set_dir"
restic backup "$out"

/etc/systemd/system/forge-backup.service:

[Unit]
Description=Forge backup set and off-site copy
OnFailure=forge-backup-failed.service

[Service]
Type=oneshot
User=forge
Group=forge
ExecStart=/usr/local/bin/forge-backup-run

/etc/systemd/system/forge-backup.timer:

[Unit]
Description=Nightly forge backup

[Timer]
OnCalendar=*-*-* 03:15:00
Persistent=true
RandomizedDelaySec=10min

[Install]
WantedBy=timers.target

Enable it with systemctl enable --now forge-backup.timer. The OnFailure= unit should ping your alerting, or a healthchecks.io /fail URL. A missed success ping must alert too, so use a dead-man switch: the job pings /start, then the plain URL on success.

Restoring

Restoring is for a fresh machine or a fresh directory. The restore refuses a target that is not empty, and it refuses a set that fails verification. It never deletes anything.

Get the set. Copy the newest verified set from off-site storage to the restore host, for example with restic restore latest --target /restore or rclone copy.

Verify it.

forge admin backup-verify /restore/forge-backup-20261015T031500Z

Stop here if the result is not OK.

Get the key. Place your escrowed forge.key where you will restore it. You do not have to place it before the restore: the restore can copy it from the set if the set has one. If the set has no key, you must copy the key in by hand before the server starts (step 6).
Restore into an empty directory.
```
forge admin restore --from /restore/forge-backup-20261015T031500Z --data-dir /var/lib/forge
```
The command verifies the set again, copies the database to /var/lib/forge/forge.db, recreates each repo with init --bare, fetches its bundle with +refs/*:refs/*, sets HEAD to the repo's default branch, and checks each repo's refs against the manifest. It also writes the managed gitconfig and hooks, and runs git fsck --full on each repo. It prints WARNING: lines for anything you must act on.
Point the config at the restored data. Set data_dir and database in forge.toml to the values the restore printed, for example database = "/var/lib/forge/forge.db". The database path is independent of data_dir in the config, so it must be set explicitly.
If the restore printed NO INSTANCE KEY WAS RESTORED: copy the escrowed key to /var/lib/forge/forge.key with mode 0600, owned by the forge user. Do this before starting the server. If the server starts without the key, it creates a new one, and the sealed data stays unreadable.
Run the integrity check.
```
forge --config /etc/forge/forge.toml admin fsck --full
```
It runs git fsck on every repo and integrity_check on the database, and exits non-zero on any problem.
Start the server and check: you can log in, the issues and pull requests are there, a clone over HTTP and SSH works, and a push works.
Record the time. The time from the start of step 1 to the end of step 8 is your real recovery time. Write it down.

If the restore fails, the target directory is left as it was when the failure happened. Do not start the server from it. Read the error, fix the cause (for example, free disk space or the git version), and restore into a new empty directory. Do not delete the failed directory until someone has looked at it.

Quarterly restore drill

Do this on a machine that is not the forge server, once every three months. Keep the answers in your ops log.

Pick the newest set in each off-site location, and the set from about 90 days ago.
Download each one. Record the download time and size.
Run admin backup-verify on each. Record the result. Any problem is an incident.
Restore the newest set into an empty directory with admin restore. Time it.
Copy the escrowed forge.key in by hand, from the offline copy, not from the set. Confirm the key matches the one in use: a sealed value (for example, a 2FA login) must decrypt.
Run admin fsck --full. Expect no problems.
Start a throwaway server on the restored data. Log in as an admin, open one issue, clone one repo over HTTP, clone one over SSH.
Check that the sets you have are still readable with the keys you hold offline (restic: restic snapshots, and restic check from the second key).
Confirm that the monitoring for the backup jobs and the fsck job still reports to you.
Update the recovery time and the date of the last drill in your ops log. If the measured time is longer than your target, fix the cause before the next drill.

Failover fencing

Restoring a backup on another machine does not by itself make that machine the only writer. Before you bring up a restored standby as the primary, follow docs/architecture.md §8.5:

Stop writes on the old primary (block the HTTP and SSH listeners, or put the instance in read-only mode).
Fence the old primary: revoke its replication and backup credentials, and make sure it cannot accept writes. Do this before promotion. A DNS change or a health check does not stop the old primary from coming back as a second writer.
Confirm the last snapshot you restored and the last ref reconcile. Record the data-loss window.
Restore on the standby, then run quick_check and fsck.
Promote the standby and switch DNS.
Record the incident and the actual recovery time.

Standby replication

A standby is a passive SSH host at another provider. It holds bare copies of every repo and a rolling set of database snapshots. It does not run forge. Replication is off unless [replication] target is set.

What the standby receives

Every repo, as a bare repo at <base>/repos/<repo id>.git. A push is sent after each ref update, debounced by 5 seconds. Every reconcile_minutes (default 60) every repo is pushed again, and the refs checksum of the standby copy is compared with the local one. A mismatch triggers one re-push. A second mismatch is recorded as an error.
A database snapshot every db_snapshot_minutes (default 15). It is taken with VACUUM INTO, checked with PRAGMA integrity_check, uploaded to <base>/db/forge-<UTC stamp>.db through a temporary name, and then <base>/db/latest is pointed at it. The standby keeps the newest 24.
Deleted repos: at the next reconcile, the standby copy of exactly that repo is removed.

Nothing else is written. <base>/keys/ belongs to the operator. Forge never writes to it, and the instance key should be escrowed there by hand, as described under "Restoring".

Status is on /admin/replication (admin only): the target, the last database snapshot, and one row per repo (ok, lagging, error, or never replicated). "Reconcile now" runs a full pass at once. It is audited as replication_reconcile_started. If [replication] healthcheck_url is set, it is pinged after each reconcile: success when every step worked, /fail with the errors otherwise.

Setting up the standby account

On the standby host, create a dedicated unix account with no password and a login shell that can run commands (/bin/sh). Create <base> owned by that account, mode 0700. Git must be installed, version 2.38 or newer.
On the forge host, generate a key used only for replication, with no passphrase: ssh-keygen -t ed25519 -N '' -f /etc/forge/standby_key, and set it to mode 0600, owned by the forge user.
On the standby, add the public key to ~<account>/.ssh/authorized_keys with the restrict option and nothing that grants more:
```
restrict ssh-ed25519 AAAA...rest-of-public-key forge-replication
```
restrict turns off port forwarding, agent forwarding, X11 forwarding and the pty. It does not force a command, because replication runs git receive-pack, git for-each-ref, cat, ls, rm and mkdir through ssh. The account's file permissions are what keep it inside <base>. Keep <base> the only directory the account can write to, and do not give it sudo.
Pin the standby's host key on the forge host. Get it over a channel you trust (the provider's console, or a fingerprint you already hold), compare the fingerprint with the server's own ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub, and then write the key line to a file of its own:
```
ssh-keyscan -t ed25519 -p 22 standby.example > /etc/forge/standby_known_hosts
```
Do not use ssh-keyscan without the fingerprint check. Forge always runs ssh with StrictHostKeyChecking=yes, UserKnownHostsFile set to this file, and IdentitiesOnly=yes, so a changed host key stops replication instead of being accepted. If the standby is rebuilt, repeat this step and check the new fingerprint.

Configure forge:

[replication]
target = "forge-sync@standby.example:/srv/forge-standby"
ssh_port = 22
identity_file = "/etc/forge/standby_key"
known_hosts_file = "/etc/forge/standby_known_hosts"
db_snapshot_minutes = 15
reconcile_minutes = 60
healthcheck_url = ""

The target is [user@]host:/absolute/path. The path may use only A-Z a-z 0-9 . _ - /, with no ... Restart forge. The worker starts at boot and runs a snapshot and a reconcile straight away.

Check /admin/replication. Every repo should show "ok" within a few minutes. On the standby, ls <base>/db should show latest and at least one forge-*.db.

Failover runbook

Follow docs/architecture.md §8.5. The steps below are specific to the standby layout.

Stop writes on the primary, and fence it. Revoke its replication key (remove the line from the standby's authorized_keys), and block its HTTP and SSH listeners. Do this before anything else. Without fencing, the old primary can come back and accept pushes.
Record the data-loss window: the time of db/latest (from its file name), and the last successful reconcile on /admin/replication, if the primary still answers.
On a new host with the same forge version, create the data directory, and copy the escrowed forge.key into <data_dir>/forge.key. Without this key sealed values cannot be read.
Put the snapshot in place at the path the config's database setting names: cp -L <base>/db/latest <database path>. -L follows the latest link, so the copy is a plain file.

Move each repo into the local layout. The local path is <data_dir>/repos/<id[0..2]>/<id[2..4]>/<id>.git, where <id> is the lowercase repo id. For example:

for repo in <base>/repos/*.git; do
  id=$(basename "$repo" .git)
  lower=$(printf '%s' "$id" | tr 'A-Z' 'a-z')
  dest="<data_dir>/repos/$(printf '%s' "$lower" | cut -c1-2)/$(printf '%s' "$lower" | cut -c3-4)/$lower.git"
  mkdir -p "$(dirname "$dest")" && cp -a "$repo" "$dest"
done

Run forge admin fsck --full. Expect no problems. Run quick_check on the database.
Start forge on the new host, log in as an admin, and clone one repo over HTTP and one over SSH.
Promote the new host and switch DNS. Record the incident and the actual recovery time.

Release assets

Release asset bytes live in <data_dir>/blobs/assets/<asset id>, outside the database and the repos. The database rows that point at them are in every backup set, so a restore without the blobs leaves downloads returning 404. Keep the blobs in step with the sets:

Blobs are immutable. An asset is written once under its id and never rewritten, only added or deleted. The writer streams to a temporary file and renames it into place when complete, so a copy taken at any moment never holds half of an asset that is listed in the database.
Copy the directory with a plain rsync -a --delete <data_dir>/blobs/ <backup target>/blobs/. Do it after each backup set, or on the same schedule. Copying during an upload is safe: the in-flight file is a temporary name that rsync may skip or copy, and the row that references it is written only after the rename.
On restore, copy blobs/ back into the new data_dir before starting the server. A row whose blob is missing is harmless to the server but its download returns 404; check the release pages after a drill.
Deleting a release or a repo deletes its blobs at once. The blob directory has no other index, so it can be checked against the database with a SELECT id FROM release_assets list.

LFS objects

Git LFS object bytes live in <data_dir>/lfs/<oid[0..2]>/<oid[2..4]>/<oid>, outside the database and the repos. The lfs_objects rows that link them to repos are in every backup set. Copy the directory with the same rsync step as release blobs, after each backup set or on the same schedule:

rsync -a --delete <data_dir>/lfs/ <backup target>/lfs/

Restore: copy lfs/ back into the new data_dir before starting the server. Check one LFS file after a drill with git lfs fetch on a clone, or by opening a blob page that shows "Stored with Git LFS" and downloading it.
Why the copy is safe: objects are named by their sha256, so a name always holds the same bytes. An object is written to a temporary file, checked against its hash and size, fsynced, and then renamed into place. A copy taken at any moment sees either the whole object or nothing under its final name. Objects are never rewritten, only added, and removed by admin lfs-gc, which deletes only files with no row in any repo that are older than a day.

Tahinli/forge