Warm standby and high availability: what a second machine can and cannot do

Progressiverobot

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) adds no schema step, so a 6.3.6 node and a 6.3.7 node both require schema 6108 and the version check at start does not tell them apart: upgrade both nodes all the same. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, so the statement after a database restart or a failover no longer fails; and the first units of the active-active work are in the tree, and it is still not built. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026), a security release, adds no schema step either, so a node of 6.3.6, 6.3.7 or 6.3.8 requires 6108 and the check at start does not tell them apart: upgrade both nodes all the same. See the 6.3.8 announcement.

This topic covers what hMailServer 6.3.4 offers for a second machine. Two operator documents describe one topology from two angles: WarmStandby.md, on what has to reach the standby and the constraints that decide the design, and HighAvailabilityRunbook.md, the active/passive procedure with a virtual IP and a health check. The one thing to know before anything else: exactly one hMailServer service may run against the database at a time. There is no clustering code and no active/active mode. Failover is starting a service on the other machine, which takes minutes, and a second service started against the same database while the first still runs delivers mail twice, with no error anywhere.

Why exactly one service may run

The code makes two servers destructive to each other in three ways:

Startup unlocks the whole queue. PersistentMessage::UnlockAll runs update hm_messages set messagelocked = 0 for every queued message when the service starts, because locks held by a crashed server would otherwise stall those messages for ever. A second server started against the same database releases every message the first is delivering, and both then deliver them.
Delivery selection is not coordinated. The queue is driven by select ... where messagelocked = 0 and messagenexttrytime <= now, and the lock is taken after selection, so two servers race that window even after both have started cleanly.
External fetch has the same shape. PersistentFetchAccount::UnlockAll runs at fetch-manager start, and a POP3 account fetched twice with delete-after-download is mail split between two stores.

The rehearsal of 9 September 2026 measured the first. With the primary holding a queued message locked, the standby was started on a second machine: it bound the same ports, because the listeners are rows in hm_tcpipports and both servers read them, and the lock went to 0. Nothing warned, on either side.

So the standby is warm: installed, configured, patched along with the primary, and its service set to Manual start and left stopped. WarmStandby.md says Manual rather than Disabled, because a disabled service cannot be started by the runbook without an extra step. The HA runbook accepts either in its section 4 and its check-list, though the wiki's copy of it lists Disabled among the ways the pair is got wrong, for the same reason. Use Manual.

What lives where

State	Lives in	Reaches the standby by
Domains, accounts, aliases, lists, rules, routes, IP ranges	The database	Sharing the database
Message metadata, folders, IMAP state	The database	Sharing the database
The delivery queue	`hm_messages` rows and spool files on disk	The database and the data directory
Greylisting triplets, auto-ban ranges, login-failure state	The database	Sharing the database
Message files (`.eml`)	The data directory	Replication or shared storage, below
The `[Settings]` section	Since 6.3.4 the database (`hm_inisettings`, schema 6042); the INI section is a cache	Sharing the database
`[Directories]`, `[Database]`, `[Security]`, `[GUILanguages]`	Each machine's own `hMailServer.ini`	Configured per machine, deliberately
`[SettingsOverride]`	Each machine's own INI	Not shared, on purpose
`PasswordPepper`	Each machine's own INI; no database and no backup holds it	Copied by hand
TLS certificates and keys, DKIM keys	Files on disk; the database stores their paths	The same files at the same paths
Route, external-fetch and per-domain-relay passwords	The database, DPAPI-protected	They do not travel
An account's second factor and the `[Settings]` secrets	The database, sealed under the stored-secrets key (schema 6103)	Sharing the database, once the standby holds the key

The settings store, since 6.3.4. A standby that shares the database inherits the primary's server settings at its next start, without anything copying INI files. Each setting the primary changed since the standby's file was written is named in the standby's error log as HM5804 and its line rewritten: that is the copy catching up, not a fault, and it appears once. A stale file can still do two things. A setting deleted on the primary whose line the standby's file still carries is adopted back into the shared store when the standby starts, so delete it from the standby's file as well. And a key the standby's file holds that the primary never had is adopted the same way.

Do not copy the primary's INI over the standby's. On Windows [Database] Password is sealed with the machine's own DPAPI key, and the copy breaks the standby's database connection outright. Treat "no [SettingsOverride] section on either machine" as part of the check-list: a key in that section is applied over whatever the database holds.

On 6.3.3 it was different. The wiki records that the [Settings] section was not carried by the shared database there, and had to be copied into the other machine's INI after every settings change. The wiki also notes that [LDAP], [SendingLimits] and [SendingLimitsOverrides] stay in each node's file in 6.3.4, to be kept the same by hand.

Paths must resolve on both machines. If the primary's row says C:\certs\mail.pem, the standby needs that file at that path. Rotated DKIM keys are written into <DataDirectory>\DKIM; a standby promoted with an out-of-date copy of that folder signs with a key file it does not have.

What does not travel

Passwords the server presents to somebody else. Route smart-host credentials, external fetch accounts, the per-domain relay password and SSL private-key passphrases are stored protected by Windows DPAPI with CRYPTPROTECT_LOCAL_MACHINE, and a blob written by one machine cannot be decrypted by another. After a failover the server keeps running, with local delivery, IMAP, POP3 and unauthenticated relaying untouched, but each such credential fails until it is re-entered on the standby. Failback breaks them the other way.

Keep the list of credentialed routes, fetch accounts and relay-configured domains, with the passwords in a password manager. It is between zero and five entries on a typical installation. The wiki has a PowerShell script over COM that lists them; none of its getters returns a password.
On Linux the server seals these with a key file, <DataFolder>/.hmailserver-secret-key. In the rehearsal the route arrived without its password, the standby reported HM6412, and the delivery attempt went out unauthenticated. A replication that copies the data directory wholesale copies that file too, and the standby then reads every secret the primary could. Decide which you want; the rehearsal excluded it.

The administrator password. It lives in hMailServer.ini, not in the database. Until it is set on the standby, the standby has no REST API and no COM authentication. On Linux the command is sudo hmailserver --set-admin-password. On Windows WarmStandby.md names the Control Panel, but a standby installed with 6.3.4 is a new installation, whose service runs as NT SERVICE\hMailServer by default, and under that account the Control Panel's password card is refused: use hMailServer.exe --set-admin-password from an elevated prompt, which is new in 6.3.4. The rehearsal found this when the standby came up mute.

PasswordPepper. With a different value on the standby, or none, no account whose password is an Argon2id or scrypt hash signs in there.

The stored-secrets key, since 6.3.4. It is kept in each server's INI as [Security] StoredSecretsKey, never in the database, so the standby needs it carried: export it on the primary, elevated, with hMailServer.exe --export-stored-secrets-key FILE (sudo hmailserver ... on Linux), and import it on the standby with --import-stored-secrets-key FILE. If the standby's installer made a key of its own, remove its [Security] StoredSecretsKey line first, or the import is refused. Never copy the line itself; it is sealed under the primary's own DPAPI. Without the key every second factor refuses every code on the standby, and SRS, BATV, HS256 tokens and /metrics stop working there; the standby says so as HM7495 at each start. Hardening a public mail server has what the key seals, and what to do on an upgraded server, which has none until you make one.

The message store: the two documents differ

WarmStandby.md says replicate, and do not share. It argues against putting the data directory on an SMB share or a Cluster Shared Volume:

The accept path flushes the spool file to disk before answering 250. Over SMB that guarantee depends on the file server and on the network between the machines, which is the failure the topology exists for.
Local delivery copies the spool file once per local recipient, and the delivery loop rewrites message files; each becomes network I/O.
Handles and oplocks held by a crashed primary take time to expire, so the standby can find its files locked by a machine that no longer exists.
Cluster Shared Volumes are built for large block-oriented I/O, not millions of small files.

It recommends a one-way scheduled robocopy /MIR of the data directory every few minutes, or DFS-R or storage-level replication, stopped as part of failover, so that a stale mirror pass from a half-dead primary cannot delete newly accepted mail. Messages accepted after the last sync are rows whose files are missing on the standby. The server treats a missing file as an error on access, not a crash, so a five-minute interval bounds the loss to five minutes of message bodies; metadata, folders and flags lose nothing. The document calls the SMB and CSV reasoning operational judgement built on code facts, not itself a code fact.

HighAvailabilityRunbook.md says share. Both nodes must see the same [Directories] DataFolder on shared storage (SAN, NAS, a clustered file system or a cloud file share), with identical read and write access for both service accounts, and the database and the store on the same failover boundary. Its validation check-list begins with that.

Neither document withdraws the other. Choose knowing what each claims.

The database

Use a server backend, SQL Server, MySQL or MariaDB, or PostgreSQL, with its own availability. SQL Server Compact is excluded: it is a single local file. Both machines must run the same build: REQUIRED_DB_VERSION is checked at every start and refuses a mismatch in both directions, so a half-upgraded pair fails safe at startup where the two builds require different schemas; 6.3.6, 6.3.7 and 6.3.8 all require 6108, so that check does not catch a pair split between them.

Health probes and the load balancer

The probes are on the metrics listener, enabled on both nodes. Since 6.3.4 these are stored settings: set them in the Control Panel (Windows), over /api/v1/settings/ini, with hmctl, or on a Linux node whose REST listener is not yet on with hmailserver --set-setting NAME=VALUE. Then reinitialise (Apply now, or POST /api/v1/server/reinitialize) or restart.

[Settings]
MetricsServerPort=8080
MetricsServerBindAddress=0.0.0.0
MetricsServerAuthToken=<32+ random characters>
ShutdownDrainSeconds=30

Point the health check at /readyz. It answers 200 only when the server is Running and the database is connected, and 503 while stopping, while draining, or when the database is lost. The passive node's service is stopped, so its connection is refused and it is never routed to.
/metrics answers 503 on a non-loopback bind until MetricsServerAuthToken is set; the probes need no credential.
ShutdownDrainSeconds defaults to 0, which waits for nothing. Set it.
The drain is /api/v1/server/drain, on the REST listener: GET its state, POST to enter, DELETE to leave.

Monitoring and health has the three probes and the listener's rules. Two matter here: MetricsServerBindAddress takes an IP literal and nothing else, since a host name keeps the listener, and so the probes, from starting; and the listener serves one connection at a time, so no sub-second health check.

Building the standby

Install the same build as the primary, with the same drive letter and directory layout.
Point its [Database] section at the same database server, verify that it connects, then stop the service and set it to Manual.
Set up the store: replication from primary to standby, or the shared directory.
Record the credential list somewhere that survives the primary.
Set the administrator password, copy PasswordPepper, and import the stored-secrets key.
Rehearse the failover at least once before it is real.

Planned failover

The drain as a state of its own and UpgradeOnStart are new in 6.3.4. On 6.3.3, step 1 is a plain stop with ShutdownDrainSeconds set.

Drain A: POST /api/v1/server/drain (manage_server), or the Drain button in the Deck's update view. A closes its SMTP, IMAP, POP3 and ManageSieve listeners, pauses its delivery queue, and reports /readyz 503 draining. A new connection is refused at the socket, which every MTA retries. REST, metrics and web services stay up. Watch GET /api/v1/server/drain until drained is true.
Stop A.
Verify A is down: /readyz on A is refused.
Stop the replication job, in whichever direction it runs, where the store is replicated rather than shared. Do it before B is started. The runbook has no such step, because it assumes a shared store; WarmStandby.md's failover has it. In a planned failover A's machine stays up, so a scheduled mirror pass from it would delete the mail B has just accepted.
Move the VIP to B.
Start B and wait for /readyz 200. If B's binaries are newer than the shared schema, B refuses to start (HM5011) unless its INI says [Database] UpgradeOnStart=1. Take the database backup before this step. One node at a time.
Check the key on B: no HM7495, HM7492 or HM7493 in its error log. /readyz is 200 either way, so this is the check that finds a missing key.
Confirm: a test message through the VIP, and /healthz reporting database: up and state: running. Then re-enter the route, fetch and relay passwords on B, as after an unplanned failover.

Reverse the steps to fail back, the check of the key included; with a replicated store, follow Failback below, which resynchronises the store first. To take A off the air without a failover, do step 1 and then DELETE /api/v1/server/drain, which reopens the same ports.

Unplanned failover

The health check on /readyz fails; stop routing to A.
Fence A: power it off, or isolate it from the storage and database network. Unreachable is not the same as dead.
Stop the replication job, where the store is replicated.
Move the VIP, or the DNS record or port forward, to B.
Start the service on B. Its startup unlock now works for you: messages the dead primary held mid-delivery are released and retried. A message the primary had delivered but not yet deleted from the queue may be delivered again, which is the known, bounded cost.
Re-enter the route, fetch and relay passwords on B.
When A is repaired, bring it back stopped, as the new passive node.

The two documents order the middle differently: the runbook moves the VIP and then starts B, and WarmStandby.md starts the standby and then moves the endpoint. Both put the fence first.

Keep MX TTLs modest, 300 to 3600 s, as a standing decision. Sending servers retry for days, so mail queued remotely during the gap is delayed, not lost.

Failback is the same with the roles swapped, plus one step first where the store is replicated: resynchronise the data directory backwards, because the standby has been accepting mail and the primary's store is now the stale one. Then stop the standby's service, replicate from standby to primary once more for the final delta, start the primary, move the endpoint back, re-enter the route, fetch and relay passwords on the primary, and set up the replication from primary to standby again.

Verify, after either direction

GET /readyz answers 200: the service is running and the database has answered a real round trip within 20 s.
A message sent in from outside, read back over IMAP or POP3.
A message sent out through each credentialed route.
No ERROR entries in the application log since the failover.
TLS on 25, 143, 110, 587, 993 and 995 presents the right certificate.

What it does not do, and what has not been rehearsed

No active/active, no sharding, no clustering code. The runbook puts them out of scope.
Not seconds. Failover is minutes, and the timings on the wiki pages are the shape of a rehearsed failover, not a promise.
No rolling restart on one node. A single node refuses connections for the length of an install.
The drain's state is per process, so a pair drains one node at a time.
The rehearsal was a Linux pair: an Ubuntu 26.04 standby, one PostgreSQL 18 database, an SSH tunnel standing in for a LAN route. Not rehearsed: a Windows-to-Windows pair, DPAPI itself, a file-replication product rather than a copy at the moment of failure, DNS or IP takeover, and failback.

Where to read more

The wiki pages: Warm Standby, with the worked commands and the recovery-window table, and High Availability Runbook, with the probe-lifecycle table and load-balancer configurations
The documents: WarmStandby.md and HighAvailabilityRunbook.md
The documentation page: A warm standby, active/passive high availability and Prometheus metrics and health probes
On this forum: Upgrading to 6.3.4, Moving to new hardware, and Settings: what is not stock and when a save takes effect, for the settings store and [SettingsOverride]