Back up and restore

Take a base or snapshot backup of a running server, archive the WAL, verify and expire backups, and restore to a backup's stop or a later LSN.

A backup is a directory that holds a copy of the data file, the log segments from the backup’s start to its stop, and a manifest. You take it from a running kds_server with tools/kds_backup.py. Restoring it replays the log from the start, and the restored instance opens only after recovery has passed the stop. With a WAL archive you can also restore to a later point in the log.

The commands on this page were not run for this page; they and their messages are quoted from the server’s backup manual.

Before you begin

The backup statements need the admin role and run only outside an explicit transaction. kds_backup connects without SCRAM, so it reaches the admin role only on a server with auth = off.
Backup directories are paths on the server’s host. Use a new, empty directory for each backup.
One backup runs at a time.
The users file and the TLS key and certificate are configuration, not data. They are never in a backup. Keep them with your configuration.

What a backup contains

<dir>/backup_manifest       written last, synced; its last line checksums the rest
<dir>/kds.db                the data file
<dir>/wal/wal-0-<n>.log     the log segments from the backup's start to its stop

A directory without backup_manifest is not a backup. A backup carries its own log for its window, so it restores without the WAL archive.

There are two ways to take one:

Path	The data file comes from	Needs
B: base backup	The server, which copies it while it keeps serving.	Nothing.
D: snapshot backup	An atomic snapshot you take between start and stop.	A snapshot that freezes the filesystem (LVM, ZFS, btrfs), or `scripts/backup/snapshot-freeze.sh`.

Take a base backup (path B)

Run the base backup into a new directory:

tools/kds_backup.py base --target /backups/2026-10-12

The tool runs these statements on the server:

Statement	What it does
`BACKUP START '<label>'`	Pins the log from the durable redo start (the backup’s start LSN).
`BACKUP COPY TO '<dir>'`	The server copies its data file into `<dir>/kds.db` on a background thread, under a write fence so no page is copied half-written. Returns at once.
`BACKUP STATUS`	Shows `copying=1` until the copy is done, with `pages_copied=` and `data_pages=`.
`BACKUP STOP TO '<dir>'`	Copies the pages the file grew, marks the stop, copies the log and writes the manifest.

Wait for the backup to finish. BACKUP STATUS shows state=idle last_outcome=done when the backup is complete.

The copier runs at up to 256 MiB/s. While it copies, a write to a page range being read waits for that read. SHOW META counts these waits in backup_fence_writes_waited.

To cancel an open backup, run BACKUP ABORT. An open backup that is idle for an hour is aborted, and a server restart ends any open backup. Neither can be mistaken for a finished backup.

Take a snapshot backup (path D)

Run the snapshot backup with a command that snapshots the volume holding the data file:
```
tools/kds_backup.py snapshot --target /backups/snap-1 \
    --command 'scripts/backup/snapshot-lvm.sh vg0 data /srv/kds/kds.db'
```
kds_backup runs BACKUP START, then your command, then BACKUP STOP TO '<dir>'. Your command gets KDS_BACKUP_DIR and KDS_BACKUP_ID in its environment. It either copies the data file to $KDS_BACKUP_DIR/kds.db, or keeps the snapshot elsewhere; you then name its data file when you verify and restore.
Make sure the snapshot is atomic. LVM, ZFS and btrfs snapshots are. On a host without them, scripts/backup/snapshot-freeze.sh <server-pid> <data_file> pauses the server process, copies the data file and resumes it; the server stalls for the copy. A cloud disk snapshot that does not freeze the filesystem can hold a torn page, and the restore refuses it.

The log does not come from the snapshot. BACKUP STOP copies the segments into the backup after the snapshot, so a separate WAL volume needs no second snapshot.

Turn on the WAL archive

The archive keeps every sealed log segment, so you can restore past a backup’s stop. Set exactly one of these keys in the config file and restart:

# kds.conf
archive_dir = /archive/kds                  # copy, fsync the file, fsync the directory
# or
archive_command = cp %p /archive/kds/%f     # exit 0 means durable; %p is the path, %f the name

Setting both is refused at startup. Segments are archived in order as kds-<lineage, 16 hex>-<segment>.log, and no segment is removed from the log before it is archived.

archive_dir never overwrites. A file already there with the same bytes is skipped; one with different bytes stops archiving with an error.
An archive_command must be idempotent.
A failure is retried with backoff, from 1 s doubling to 30 s.
If archiving keeps failing and the WAL filesystem has less than 4 segments (256 MiB) free, every statement that writes the log is refused (INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, PURGE). Reads continue. Fix the archive destination; nothing else is needed.

Monitor the archive in SHOW META: wal_archive_next_segment, wal_archive_lag_segments, wal_archived, wal_archive_failures, wal_archive_blocking, and, while failing, wal_archive_last_error.

Every restore that opens gets a new lineage id (SHOW META: wal_lineage), so a restored instance that archives to the same place never overwrites the original’s segments.

Verify, list and expire backups

tools/kds_backup.py verify /backups/2026-10-12          # same as kds_server --verify-backup
tools/kds_backup.py list /backups
tools/kds_backup.py expire /backups --keep 7 --archive /archive/kds
tools/kds_backup.py expire /backups --keep-since 1760000000 --archive /archive/kds

verify checks the manifest, every file’s size and CRC32C, that every segment from start to stop is present, every log record’s CRC32C, and every data page’s own checksum. For a snapshot kept outside the backup, add --snapshot-data <file>. On success it records last_verified, which list shows.
The same page check runs when a backup stops (a failing page fails the backup, and no manifest is written) and when a restore lays out a backup.
expire removes backups outside the retention and the archived segments that no retained backup needs. It refuses a retention that keeps none, and never touches a lineage no retained backup names.

Restore to a backup’s stop

Prepare a config file whose data_file and wal_dir point to locations that hold nothing.
Run the restore:
```
tools/kds_backup.py restore /backups/2026-10-12 --config /etc/kds/new.conf
# or directly:
kds_server --restore-from /backups/2026-10-12 --config /etc/kds/new.conf
```
The restore checks the manifest, verifies every file, copies the data file, copies the log segments and mounts the instance. Recovery replays from the backup’s start, and the instance opens only once it has passed the stop. It then serves. Add --restore-only to exit after recovery instead.
For a path D backup that does not hold the data file, name the snapshot’s data file with --restore-data <file>.

A restore interrupted partway never opens short of the stop.

Restore to a later point (point-in-time recovery)

The target is an LSN. Commit times are not recorded in the log, so there is no time target.

kds_server --restore-from /backups/b --restore-archive /archive/kds \
           --restore-target 812345678 --config new.conf
kds_server --restore-from /backups/b --restore-archive /archive/kds \
           --restore-target latest --config new.conf

The restore reads the archived segments after the backup’s stop, cuts the log at the first record at or past the target, and recovers. Everything committed before that record is restored; a transaction open at the cut is rolled back. latest replays every archived segment. A target below the backup’s stop is refused.

To find an LSN to return to, read wal_durable_lsn from SHOW META before an operation you may want to undo.

Verify the restore

SHOW META on the restored instance shows recovery_restore_backup=, recovery_restore_start= and recovery_restore_stop= for the mount that completed it.

Fix common refusals

Message	Cause	What to do
`BACKUP START: backup N (...) is open`	One backup at a time.	`BACKUP STOP` or `BACKUP ABORT` it.
`BACKUP ... inside a transaction is not implemented`	Run inside `BEGIN`.	Run it in autocommit.
`BACKUP STOP: ...'s data copy has X of Y pages`	The copy is still running.	Wait for `copying=0`.
`BACKUP STOP: ...'s data copy failed`	The copier hit an I/O error.	`BACKUP ABORT`, fix the target, take a new backup.
`BACKUP STOP: <dir> holds <name>`	The directory is not new.	Use a new directory.
`backup: <dir> has no backup_manifest`	Not a backup, or the stop never finished.	Take a new backup.
`backup: manifest format N is not this build's`	Written by another build.	Restore with the build that wrote it.
`backup N: wal/wal-0-K.log is ... the manifest says ...`	A file was changed or damaged.	Use another backup. The restore wrote nothing.
`backup N: segment K (lsn ...) is not in the backup`	A segment is missing.	Use another backup.
`data file: N of M written page(s) fail their checksum (page ...)`	A torn page (a snapshot that did not freeze the filesystem) or damage.	Take the backup again; for path D use a freezing snapshot.
`backup N: its log decodes only to lsn ...`	A damaged segment.	Use another backup.
`restore: ... already holds data`	The target is not empty.	Use an empty target.
`restore: target lsn T is below ... stop lsn`	The target is before the backup’s stop.	Use an older backup.
`restore: the archive's history ... ends at lsn E ... segment K is missing`	The archive does not reach the target.	Use a lower target, or find segment K.
`restore of backup N (start ..., stop ...) did not complete; the instance does not open`	The log does not reach the stop, or recovery failed.	The end of the message says which.
`ERR ... the WAL archive is failing (...)` on a write	The archive fail-closed rule above.	Fix the archive destination.

Limitations

Not built yet: incremental backups, compression, encryption, object-storage targets, parallel copy, concurrent backups, logical dumps and standby servers. A torn page from the device itself is refused, not repaired.

Next steps

Choose a durability class: what a crash can lose without a backup.
Server configuration: archive_dir, archive_command and the restore flags.
Known gaps.