Backup and restore: what is in the archive, what is not, and restoring one mailbox

Progressiverobot

This topic covers the built-in backup, what its archive holds and leaves out, and restoring a server, one mailbox or one message. Know this first: a restore is destructive and has no transaction. It deletes every domain, every account and every public folder before it puts the archive's back, so whatever an account has that is not in the archive is deleted and not replaced. And the archive does not include your messages unless you tick the box.

What one run writes

One 7-Zip archive per run, named HMBackup YYYY-MM-DD HHMMSS.7z, in the configured destination: an XML index and, with Back up messages on, the whole data directory beside it. The data directory is C:\Program Files\hMailServer\Data by default on Windows and /var/lib/hmailserver on Linux, and what lives inside it travels with the messages: Sieve scripts and their state (<data directory>\Sieve), and ACME certificates and keys (<data directory>\ACME, unless AcmeCertificateDirectory moved them). Since 6.3.4, messages held in an object store (MessageStoreBackend) are read into the archive too: a store that cannot be read fails the backup, a restore puts the files back on the local disk, and the store's credentials are not carried.

What it cannot hold: files the database only points at. Manually placed TLS certificates, private keys and DKIM keys are stored as paths. Restored onto a machine where those paths hold nothing, the server starts, serves no TLS and signs nothing. Keep them inside the data directory, or copy them yourself.

What is in the archive

Area	Carried
Domains	Aliases, distribution lists and members, signatures, DKIM selectors and key paths; since 6.3.4 the domain's allow and block lists, its spam traps, the domain transforms, the S/MIME authority and the recovery key of a domain encrypted at rest
Accounts	Address, password hash and its algorithm, forwarding, vacation, signature, quota, admin level, anti-spam thresholds, TOTP secret
Per account, every release	Rules, fetch accounts with their passwords, application passwords as hashes and, with Back up messages, folders and messages with UID and flags
Per account, since 6.3.4	Message keywords (the webmail's labels, second and third stars, follow-up flags, mute and pin), the address book, webmail preferences, scheduled sends and snoozes, file-link records, S/MIME keys, the password history, calendars with their events and tasks, the first-contact memory, safe and blocked senders, the mailbox key of an account encrypted at rest, the role assignment
Public folders	With the permissions on them. Backup.md states no condition; section 15.2 of the wiki lists them under Back up settings. A restore deletes every public folder first
Settings, with Back up settings	The `hm_settings` property set and the `[Settings]` rows of `hm_inisettings`; routes, IP ranges, global rules, TCP/IP ports, SSL certificate records, incoming relays, groups, blocked attachments, the DNSBL, SURBL and white lists; since 6.3.4 the remote domain policies and IP warm-up schedules
Custom administrator roles	With domains or settings; restored first, so the assignments accounts carry find their role

Nothing identity-shaped is written. A restore gives every row a new id, which makes the archive portable between database backends; a reference that cannot be resolved drops that one row and names the account in hmailserver_backup.log.

Secrets travel opened. Route, fetch-account and SSL private-key passwords are carried in a portable encoding and protected again on the restoring machine (since 6.3.8 one the encoding cannot carry, holding a character above U+00FF or longer than 248 characters, as an external account's OAuth token often is, is left out of the archive and named in the error log and the backup log as HM7280, to be entered again after a restore, where 6.3.7 and earlier wrote it empty); since 6.3.4 each TOTP secret and the [Settings] secrets (SRSSecret, OAuth2HmacSecret and the rest) are carried opened and sealed again under the restoring server's stored-secrets key, so a restore onto new hardware keeps every second factor. An unencrypted archive therefore hands those secrets to whoever can read the destination folder.

In 6.3.3 and earlier the per-account rows marked since 6.3.4 are in no archive, and a restore deletes them. Until you upgrade, back up the database itself with its own tool (pg_dump, mysqldump, SQL Server's backup, or a copy of hMailServer.sdf with the service stopped), restore from that when recovering a whole server, and tell users before they find their address book and labels gone. An archive written by 6.3.3 or earlier restores on a newer server as it always did.

What is not in the archive, deliberately

Not carried	Consequence of a restore
`[Database]` and `[Directories]` in `hMailServer.INI`; the administrator password and its second factor	Per machine. Keep a copy of the file, and never copy one machine's file over another's
`PasswordPepper` (since 6.3.4)	Kept in the file only. A restore onto new hardware needs the same pepper in that machine's file before an Argon2id or scrypt password can sign in; PBKDF2, SHA-256 and MD5 hashes, and the administrator password, do not use it
The stored-secrets key (since 6.3.4)	The archive does not need it. A database-level backup does: restored without the exported key, every second factor and `[Settings]` secret in it is unopenable. Export it once (`hMailServer.exe --export-stored-secrets-key FILE`, elevated, or `sudo hmailserver --export-stored-secrets-key FILE`) and keep it with the database backups, stored apart from them
On Linux, the stored-secret key file `<DataFolder>/.hmailserver-secret-key`	The archive does not carry it. Its passwords travel in a portable encoding and do not need it, but a domain's S/MIME authority key, and the mailbox key of an account encrypted at rest that has not yet signed in, are carried as envelopes that open only where this file is: restored without it, the authority must be rotated before it signs. A database-level backup needs it as well: back it up with the database, because a secret moved to a machine without it cannot be opened (HM6412)
The delivery queue	Queued files are copied with the data directory, but nothing re-queues them. Let the queue drain before a migration
Per-domain relay settings, and per-domain message retention days	Silently lost, and silently reset to `0` (keep for ever). Re-enter them
Greylisting triplets; auto-ban and login-failure state	Senders are greylisted afresh; bans lapse. Permanent IP ranges are carried
Quarantine, message trace, metric history; since 6.3.4 the audit trail, alert rules and events	Operational records. Back the database up with its own tool if you need them
Deleted items kept for recovery, eDiscovery cases and holds, messages that expire, password-protected messages	The `Recoverable`, `Discovery`, `Confidential` and `Protected` folders are left out of the copy and left in place by a restore. A restore with domains is refused while an account or a domain is under a legal hold. Archives taken by earlier 6.3.4 builds, before `Confidential` and `Protected` were left out, do hold both: a message that has expired or been withdrawn since is still in them, so apply the mail's own retention to those archives, or delete them
The full-text index, IMAP annotations, the QRESYNC expunge log	Rebuilt, or keyed on ids a restore reassigns. IMAP clients resynchronise
Passkeys; calendar and address-book shares; the backup passphrase, the SNMP passwords, the object store's secret, the sandbox token	Enrol, share or enter them again

A domain named recoverable, discovery, confidential or protected keeps its mail in a folder the backup leaves out. Such names are refused for a new domain since 6.3.4, and an existing one is reported as HM7597; rename it before relying on the backup.

The options, and the combinations that fail

Maintenance → Backup & restore in the Control Panel (Windows-only), the Control Deck's Backup view, PUT /api/v1/settings/backup and hmctl settings set backup set the destination and what goes in. A fresh install seeds backupoptions as 0: nothing is ticked. On a packaged Linux server the destination has to be inside the unit's ReadWritePaths, or the write fails; add it with a drop-in.

Option	What it changes
Back up domains	The domain tree goes into the index. Required for messages to mean anything
Back up messages	Copies the whole data directory. Only with Back up domains as well: messages without domains stages nothing and then fails
Message metadata only (`BackupMessagesDBOnly=1`)	Records which messages exist, not their contents; a restore expects the files still to be in the data directory. It governs the restore as well as the backup, and must be the same for both
Back up settings	The settings row above and, per the wiki, public folders
Compress destination files	Puts the message files inside the archive. Off, every run overwrites one `DataBackup` folder beside the archives: exactly one generation of mail, whatever retention says
Verify each backup (`BackupVerifyRestore=1`, the default)	The next section

Tick domains and messages together, always, and Back up settings as well if the installation has public folders. The index is written to hMailServerBackup.xml, added to the archive and deleted, so what you restore from is the .7z; the Control Panel's file picker defaults to *.xml, so choose All files.

How a run is verified

Every refusal happens before anything in the destination is written, or deletes only the archive this run created. A run refuses when a message file lies outside the data directory, when the destination is unreachable, and when the database is down or any read did not run, so an outage cannot produce a hollow archive. Then the archive is read back:

The index opens, through the same code a restore uses, and describes this run. If not, the backup fails and the archive is deleted.
The per-account rows are all there (since 6.3.4), counted per table. A difference fails the backup.
The message store extracts to exactly the files and bytes that were staged. Fewer is a truncated archive, whatever 7-Zip's exit code said.
Every message row has a file. Reported, never failed: mail flow does not stop for a backup. The log names the first ten.

Steps 3 and 4 run only when domains and messages with their files are included and BackupVerifyRestore is 1. The wiki puts step 2 under the same conditions; Backup.md says only that BackupVerifyRestore=0 turns off steps 3 and 4. Step 3 costs disk: the store exists twice in the temp directory during the check, and a temp volume that cannot hold it skips the step with an explanation rather than failing a good backup. A successful run ends Backup completed successfully.; a failed one has a BACKUP ERROR line.

Schedule and retention

Since 6.2.25. Nothing is configured on a fresh install.

Setting	Default	Meaning
`ScheduledBackupTime`	empty (off)	`HH:MM`, local time; at most one backup per calendar day
`ScheduledBackupIntervalMinutes`	`0` (off)	Used only when no daily time is set; measured against the newest archive in the destination
`ScheduledBackupKeepCount`	`0` (keep all)	How many of this server's own archives to keep
`ScheduledBackupMaxAgeDays`	`0` (keep all)	Delete archives older than this
`UpdateBackupBeforeApply`	`1`	An unattended live-update apply (in `UpdateWindow`) backs up to the destination first and is cancelled without a successful backup; an apply you start yourself takes none

Since 6.3.4 these are stored in the database: set them on the Backup & restore page, with PUT /api/v1/settings/ini/{name} or hmctl ini set, not in the file (how the settings store works). Retention has four rules: nothing is deleted until the replacement has completed successfully; the newest archive is never deleted; the age rule always leaves the two newest alone; and only files whose names the server generated are considered, the timestamp read from the name. A scheduled run certain to fail (no destination, nothing selected, database unavailable, destination unreachable) is refused before it starts and backs off 5, 10, 20, 40 and then 60 minutes. Test backup directory on the Control Panel's Diagnostics page catches a schedule that has been failing silently.

Encrypted archives, since 6.3.4

Off until a passphrase is set; then every archive is 7-Zip AES-256 with its headers encrypted (-mhe=on), so it cannot even be listed without the passphrase. Set it in the Control Panel (Backup & restore → Encryption), the Deck's Backup view, encryption_passphrase in PUT /api/v1/settings/backup, or hmctl backup passphrase set. It is 12 to 256 printable ASCII characters, kept in the database under the machine's secret store (DPAPI on Windows, the key file on Linux), never written into the archive, a command line or a log, and never read back. If it is lost, everything in every archive written with it is lost. Changing it does not touch archives already written, so write the passphrase and its date down together, away from the backups.

What it changes about a run. Compression must be on when messages are backed up, or the run is refused. Staging moves into a directory under TempDirectory, so the temp volume needs room for the message store plus a little, and the run is refused with both numbers if it is not there. A stored passphrase this machine cannot open fails the backup; an unencrypted copy is never written in its place. 7zr, the reduced p7zip build, cannot encrypt: a Linux server needs 7-Zip (7zz) or p7zip-full. The archive's file name, its size and the fact that a backup ran are not hidden.

Restoring the whole server

Phase one decides and changes nothing. Since 6.3.4 the archive is listed first (7za l -slt, the headers only): an entry with an absolute path, a .. segment, or a link with somewhere to point refuses the whole archive as HM7540, because no backup the server writes has such an entry. Then the archive must be one of ours, written by an older or equal version (a newer one is refused, with no override), and hold what you asked to restore; then the store is extracted to a scratch directory. Phase two acts: delete every domain, delete every public folder, empty the data directory and copy the staged store in, load domains and accounts, load settings, reinitialise. A failure here leaves a half-restored server. The listing check works whichever 7-Zip is installed; the Windows installer's is 26.03 in 6.3.4, replacing 19.00.

Where. The Control Panel's Restore card and COM (BackupManager.LoadBackup, LoadEncryptedBackup) on Windows; POST /api/v1/backup/restore with file, passphrase and the three parts, which is the only restore a Linux server has; the Deck's Backup view; hmctl backup restore FILE --domains --messages --passphrase-file FILE. The REST route's check is bounded at two minutes; past that the route answers 504 and starts nothing.

Install the same version, at the same drive letter and directory layout.
Restore or recreate [Database] in hMailServer.INI and confirm you can connect.
Choose the .7z, tick the categories, start, and watch hmailserver_backup.log.
Copy back what the archive could not hold: certificate, key and DKIM files, and PasswordPepper.
Re-enter the per-domain relay settings.
Check Server status, then Diagnostics.
Send a message in from outside, read it over IMAP, send one out.

Rehearse this yearly on a scratch machine against a new, empty database, never production, and keep the list of what you fixed by hand with the backup.

Restoring one mailbox, or one message, since 6.3.4

An archive in the backup destination can give back one account's mailbox, every folder and message it held when the backup was taken, or one message of it, and touch nothing else: not the account's settings, rules, contacts or calendars, and no other account.

Where	How
REST	`GET /api/v1/backup/archives`, `GET /api/v1/backup/{archive}/mailboxes/{address}`, `POST .../restore` with `mode`, or `message` and `folder`, `GET /api/v1/backup/restore-mailbox/{job}`
Control Deck, and Control Panel (Windows)	The Restore a mailbox and Restore one message cards
COM (Windows)	`BackupManager.RestoreMailbox`, `RestoreMessage`, `DescribeMailbox`, `MailboxRestoreStatus`
hmctl	`hmctl backup archives`, `hmctl backup mailbox ARCHIVE ADDRESS`, `hmctl backup restore-mailbox ARCHIVE ADDRESS --merge\\|--replace [--message ID [--folder F]]`, `hmctl backup restore-status JOB`

Which archives. Only one inside the destination, named as the listing names it and never by an arbitrary path. It must have been taken with domains and messages; an archive taken with BackupMessagesDBOnly holds rows and no files, and the job fails. An encrypted one needs its passphrase.

Merge creates each archived folder that is missing and adds each archived message whose file is not in the live mailbox; nothing is removed, and merging twice adds nothing. Replace empties every live folder first, so the mailbox is the archive's and mail that arrived since the backup is gone; what it removes is kept only for RecoverDeletedDays, which is off by default. Replace is refused with 400 when the archive holds no messages for the account.

The identity rule. Every message file is named once, with a GUID, and the same name anywhere in the live mailbox is the same message: a message the person moved is not brought back to its old folder, and one they deleted is. A restored message keeps its flags, keywords and dates, gets a fresh UID, and is charged to the account as a delivery is.

Refused, and counted. The account must exist on this server and the archive must hold it (404); restoring into a different account is not offered, and neither is a public folder. A second restore while a backup or restore runs is 409. The outcome (folders, added, skipped, failed) is in the job's status and the backup log. Restoring needs manage_server; listing an account's messages out of an archive needs view_mailbox.

Where to read more

Wiki: Backup and Restore, with COM commands in section 15.9; Installing on Linux for the Linux paths and key file.
Repository: Backup.md; SecretsAtRest.md.
The public documentation: Backup and restore, Backups on Linux.
On this forum: Moving to new hardware, Upgrading to 6.3.4, and Hardening a public mail server for exporting the stored-secrets key.