Hardening a public mail server: secure defaults, checklist, sealed secrets and roles

Progressiverobot

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) is a security release that bears on this guide. On Windows, hMailServer.exe /ProtectFolders, which setup runs at every installation and upgrade, now hardens the program folder, Bin and ControlPanel: where hMailServer was installed in a folder that took its permissions from the root of a drive, any account that could sign in to the computer could replace the programs the service runs (CVE-2026-107571); the default folder under Program Files was never affected, and an installation at a drive's root is hardened by the upgrade itself. The serveradmin role given with a list of domains is confined to those domains over COM as over REST, a credential confined to named domains is refused the changes that reached beyond its domains over REST and COM, and a state-changing REST request that a browser reports as cross-site is refused. SMTP, POP3 and ManageSieve refuse a SASL PLAIN sign-in that asks to act as another account, and the stored-secrets key is no longer left in memory freed without being written over. An upgraded Windows server keeps the .NET 10 Desktop Runtime it had, 10.0.12 where 6.3.4 to 6.3.6 installed it, against which Microsoft published advisories on 13 October 2026: bring it to 10.0.13 through Microsoft Update or Microsoft's installer. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) is a security release too: upgrade every server running 6.3.7 or earlier. On Windows setup now makes or hardens the program folder, Bin and ControlPanel before it copies a file or runs a program of the installation (the four narrower cases of work item 78 that 6.3.7's notes left open), and a program folder, Bin or ControlPanel, a folder or a link on the way to them, or a program, library, script or settings file in them, that another account than SYSTEM, the Administrators and TrustedInstaller controls ends the installation with exit code 7, naming the icacls that deals with it and, where it says so, a restart of Windows before setup runs again. The auto-ban saves the IP range it adds under a name of its own, so that it bans the address; the web services listener (CardDAV, CalDAV, Autodiscover) refuses an address an auto-ban or an IP range bans; and a sign-in's name is matched to its account as the per-name lockout and the compromised-account hold count it, so a mail client that signs in with another spelling of the account's address than the stored one, in more than the case of the letters A to Z, or with white space at either end, may no longer sign in: correct the name in the client. See the 6.3.8 announcement.

This guide covers what stands between a mail server on the public internet and the people who guess passwords at it. All of it is new in 6.3.4 except auto-ban, account lockout, the administrator's second factor and the IP range option Require TLS for auth, which are older; 6.3.4 sets that option on a new installation and judges it in the checklist. The one thing to know first: the secure defaults are applied only to a database the installation itself created, and an upgraded server is never changed. After an upgrade your listeners, IP ranges and service account are as they were and nothing is sealed until you make a key; the checklist says what is left.

What a new installation starts with, and what an upgrade keeps

In 6.3.3 and earlier a new installation listened on ports 25, 587, 110 and 143 with no TLS, and neither shipped IP range required TLS for authentication. The create scripts still make that database; the Windows installer, and the Linux packages' first start, now change it before the server first listens.

	A new installation since 6.3.4	An upgraded server
Ports 25 and 587 (SMTP), 110 (POP3), 143 (IMAP)	STARTTLS offered, optional	as it was
IP range Internet (every IPv4 address)	TLS required for authentication	as it was; the checklist flags it
IP range My computer (127.0.0.1)	unchanged	as it was
Certificates	Self-signed (made at installation)	as they were
Windows service account	`NT SERVICE\hMailServer`	the account it had (LocalSystem, if it began before 6.3.4)

What a client sees. From another machine, AUTH on SMTP is not offered before STARTTLS and is refused with 530 A SSL/TLS-connection is required for authentication. if it is sent; IMAP LOGIN and AUTHENTICATE answer BAD A SSL/TLS-connection is required for authentication.; POP3 USER and AUTH answer -ERR. After STARTTLS each works as before. From the server itself, on 127.0.0.1, nothing has changed, and mail from other servers on port 25 is taken with or without TLS.

Why STARTTLS is optional. A port's requirement applies to every address: on port 25 it would refuse every server that cannot start TLS, on the others this computer's own clients. The range's rule protects the password.

The certificate. self-signed-certificate.pem and its key self-signed-key.pem are in <data directory>/Certificates: C:\Program Files\hMailServer\Data\Certificates on a default Windows installation, /var/lib/hmailserver/Certificates from the Linux packages, where the key is mode 0600 and owned by the service's user. It is RSA 2048, runs 825 days, names the host and gives the organisation hMailServer, self-signed at installation. It encrypts, but nothing vouches for it, so mail clients warn. Replace it with ACME (set AcmeEnabled, AcmeContactEmail and AcmeDomains; the first certificate goes to every listener that holds the self-signed one) or with your own, set on each port. See Let's Encrypt certificates are automatic.

Where the defaults are applied

The create step marks its database with the hm_settings row NewInstallDefaultsPending; the verb --apply-new-install-defaults requires the mark and removes it once it has decided.

Platform	When it runs	What to know
Windows	The installer, after the database step of a new installation, with the service stopped	A database DBSetup attached, rather than created, carries no mark: exit 5, logged, nothing changed
Linux (`.deb`, `.rpm`)	The unit's first start after `sudo -u hmailserver hmailserver --create-database`, which marks the database and leaves a one-line file, `.new-installation`, in the `DataFolder`	The unit runs `ExecStartPre=-/usr/bin/hmailserver --apply-new-install-defaults --if-marked`: nothing unless that file is there, then one try at the database, given room by `TimeoutStartSec=300` (the leading `-` forgives a failure, not a timeout). It ignores a mark made more than a day from its clock; a run by hand has no such limit
A container (since 6.3.8)	The image's entrypoint, before whatever command the container runs; the Compose file and the Helm chart run the same step before the server	`hmailserver --apply-new-install-defaults --if-marked` secures a database the entrypoint's `--create-database` made a moment before and opens nothing at any other start; a failure (run as root, a database not reached) is said and forgiven, and the server starts all the same
By hand	A container from an image before 6.3.8, a server started by hand rather than by the unit, or defaults left for later	With the service stopped; on Linux as the service's user, because the verb refuses to run as root

By hand, on Windows and then on Linux:

Stop-Service hMailServer
& "C:\Program Files\hMailServer\Bin\hMailServer.exe" --apply-new-install-defaults
Start-Service hMailServer

sudo systemctl stop hmailserver
sudo -u hmailserver hmailserver --apply-new-install-defaults
sudo systemctl start hmailserver

Exit codes: 0 applied (with --if-marked, also nothing to do); 1 something could not be made or written (HM7510, HM7511 in the error log); 2 no database named, or not readable enough to judge, or on Linux run as root; 4 the Windows service is running; 5 not a database this installation created, or not as the create script left it: nothing changed.

On Linux, mind the data directory. A DataFolder outside the unit's ReadWritePaths (/var/lib/hmailserver and /var/log/hmailserver) is read-only to the service, so the first start can neither write the certificate nor store mail there. Add the directory with a drop-in (sudo systemctl edit hmailserver, then ReadWritePaths=/srv/mail under [Service]) before that first start.

Not yet proved on a clean machine. SecureDefaults.md's checks that the installer applies the defaults are not yet written, and those for the .deb ran only on GitHub Actions, so nothing runs them today.

The setup checklist

The checklist judges every server, new or upgraded, in twelve items.

Key	Done when
`profile`	a setting profile has been applied; asked for on a server with fewer than ten accounts
`domain`	an active domain exists, and none has a name the server reserves for its own directories
`account`	at least one account exists
`submission`	every SMTP listener other than port 25 has STARTTLS or implicit TLS, and a certificate other than the self-signed one
`sign_in_tls`	every IP range that lets a client connect from another address requires TLS for authentication, or reaches only listeners that start TLS before a password
`spf`, `dkim`, `dmarc`	every active domain has a TXT record beginning `v=spf1`, signs with a key whose record is published at `<selector>._domainkey.<domain>`, and has a TXT record beginning `v=DMARC1` at `_dmarc.<domain>`
`backup`	the destination is set and the folder exists
`second_factor`	the administrator password is enrolled with a one-time code
`stored_secrets_key`	this server holds the key the database records, and an export of it has been recorded
`service_account`	it is not LocalSystem on Windows, not root on Linux

Each item is done, action_needed, cannot_tell or not_needed. Port 25 is never the submission port, and a plain submission port beside a secured one still needs action.

Where it is. GET /api/v1/setup/checklist (reading needs view_configuration; ?dns=0 skips the lookups), the Deck's Checklist view and, on Windows, COM's Utilities.GetSetupChecklist(LookUpDns) and the Control Panel's External setup page and dashboard.

Passwords only over TLS, on a server you upgraded

"Require TLS for auth" is an option of an IP range, not of a port, and covers SMTP, POP3, IMAP and ManageSieve alike.

Read the sign_in_tls item: it names each range, and the listeners where a password from there would cross the network in the clear.
If the finding is sign_in_no_tls, a listener has no TLS to start, and requiring TLS would refuse every client on it: give those listeners a certificate and STARTTLS first. This finding carries no fix.
Set every mail client that signs in from another machine to STARTTLS first. Once the range requires TLS, a client that signs in without TLS is refused; that is why an upgrade is never changed by itself.
When the finding is sign_in_plain, apply its fix: the Deck's button Require TLS for sign-in from this range, or PUT /api/v1/ipranges/{id} with {"require_tls_for_auth": true}, which needs manage_settings. On Windows the Control Panel opens its IP ranges page, where the setting is Require TLS for auth.
Replace the self-signed certificate if you have one, so that submission_self_signed clears.

Secrets at rest and the stored-secrets key

Until 6.3.4 the administrator password (unsalted MD5, from the Windows installer), each account's second factor and the [Settings] secrets were kept as written.

The administrator password is now written as PBKDF2. An older MD5 or SHA-256 value is rewritten as PBKDF2 at the first successful administrator sign-in where the service can write the file. Where it cannot (NT SERVICE\hMailServer, or the Linux package's root-owned file) the old value stays, HM7494 says so once, and the verb rewrites it: on Windows hMailServer.exe --set-admin-password from an elevated prompt, with the password in HM_ADMIN_PASSWORD, or on one line of standard input from a pipe or a file for a password in plain ASCII (Windows PowerShell 5.1 turns every other character into ?), never as an argument; on Linux sudo hmailserver --set-admin-password, which reads standard input.

Second factors and ten [Settings] secrets (SRSSecret, BATVSecret, OAuth2HmacSecret and the rest) are sealed with AES-256-GCM under the installation's stored-secrets key, in the form HMK1:<key id>:<base64>, once the installation has a key. The key is kept in each server's hMailServer.ini as [Security] StoredSecretsKey, and never in the database or a backup archive.

Who gets a key without asking. The Windows installer makes one on a new installation. On Linux, hmailserver --create-database makes one when it creates the schema and can write the configuration: in a container, whose /etc/hmailserver the service owns, it does (no image of 6.3.4 is published); run by the package as the service's user, it cannot. An upgraded installation, and a package installation on Linux, seal nothing until an administrator makes the key:

Make the key on one node. Windows, from an elevated prompt beside hMailServer.exe: .\hMailServer.exe --create-stored-secrets-key. Linux: sudo hmailserver --create-stored-secrets-key.
Export it at once: .\hMailServer.exe --export-stored-secrets-key D:\Escrow\hmailserver-stored-secrets.key, or sudo hmailserver --export-stored-secrets-key FILE. The export writes a new file, never over one that exists, readable only by SYSTEM and the administrators on Windows and mode 0600 on Linux, and prints the key id, never the key.
Restart the service on that node (Restart-Service hMailServer, or sudo systemctl restart hmailserver). Its next start seals what is stored as written; until then nothing is sealed.
On every other node that shares the database, --import-stored-secrets-key FILE, then restart each. An import is refused while the node holds another key that opens: a standby or new node whose installer made a key of its own removes its [Security] StoredSecretsKey line first, and until then the shared database refuses it (HM7495).
Keep the exported file with the database backups, stored apart from them.

Why the export matters. The product's own backup archive carries these secrets opened, and a restore seals them under the restoring server's key, so set a backup passphrase. A database-level backup (a dump, a copy of the .sdf) restored without the key brings back every second factor and [Settings] secret unopenable. A copy of hMailServer.ini is no copy of the key on Windows, since its line is sealed under that machine's DPAPI; never copy that line from one machine to another.

When a node lacks the key it says HM7495 at every start and makes no key of its own; until the key is imported, a [Settings] secret sealed under it runs as though unset (HM7492) and a second factor refuses every code (HM7493). A key lost for good is recovered from with --forget-sealed-secrets <key id>, with the service stopped (exit 4 otherwise), which clears only what that key sealed: every second factor is enrolled again, every [Settings] secret entered again.

ProtectStoredSecretsWithDPAPI=0 (default 1) has a cost. The next start opens what was sealed and stores it as written, readable by anyone who can read the file or the database, and the machine-bound secrets are written with the legacy Blowfish scheme, whose key is in the source code: the way back, not a setting for a new server.

Two things to do after an upgrade. Since 6.3.4 PasswordPepper stays in hMailServer.ini only and is left out of every backup, so keep a copy of that value with the backups. And a password hash stored before 6.3.4 from a password with a character above U+00FF is refused, to its own password too: an administrator sets such passwords again, and the application log names each account as its user tries to sign in.

Checking new passwords against published breaches

Off by default, because it sends something off the server. When on, a new password found in the published corpus of breached passwords is refused.

Setting (REST key)	Default	What it means
`breach_check_enabled`	off	The whole feature
`breach_check_url`	`https://api.pwnedpasswords.com/range/`	Where the five characters are sent
`breach_check_minimum_count`	`1`	Refuse a password seen this many times or more
`breach_check_unreachable_action`	`allow`	`allow` lets a change through unchecked; `refuse` refuses it

What leaves the server is the first five characters of the password's SHA-1 and nothing else. The default endpoint is a third-party service, Have I Been Pwned's Pwned Passwords, and an organisation that cannot send it those five characters, its address and the time runs its own mirror.

When it runs. When a password is chosen, by the person or by an administrator, and never at logon. An endpoint that gives no complete answer within eight seconds counts as unusable, and the error log records HM6600 (let through) or HM6601 (refused). Under refuse, an endpoint that is down refuses every password change on the server. Bulk imports make one request per account, so run them against a mirror or with the check off.

Where it is set. The four settings are hm_settings rows, not hMailServer.ini lines, and a change applies from the next password chosen. They are on the Control Panel's Administrative access → Password page (the Passwords found in published breaches card), in the Deck's Settings view, over REST and in hmctl settings set:

hmctl settings set --set breach_check_enabled=true

Compromised-account detection

This judges every authenticated submission (SMTP, the webmail's send, a scheduled draft) against the account's own history of the last 30 days, without reading the message. It starts in watch on every installation, new or upgraded: the baseline is built, the account.compromised alert says what enforcing would have done, and nothing else happens.

Four signals score: volume (2 points), the last hour's messages reaching compromise_volume_floor (30) and exceeding compromise_volume_factor (4) times the busiest baseline hour; recipients (2), the last hour's distinct recipients reaching compromise_recipient_floor (20), at least 80 per cent of them new to the account; hour (1), an hour of the day (UTC) the account has not submitted at in the baseline; address (1), a client network (the /24 of an IPv4 address, the /64 of an IPv6 one) not seen for this account. compromise_score_to_lock (3) is a takeover. In enforce, the message is held, the account is locked until an administrator unlocks it, and the alert and an account.locked webhook are raised. The client is still answered 250 Queued. Held mail is reviewed on the Deck's Held for review page and under /api/v1/review; a held message nobody has looked at in compromise_held_days days (7) is discarded. Watch it before enforcing: the false positive is a person locked out of their own mail. There is no Control Panel page for it.

Roles instead of the administrator password

Rather than share the administrator password, give each administrator a role in the domains they look after, on the Deck's Roles view, over REST, or on Windows in the Control Panel or over COM. Nothing changes on the day you upgrade: an account with no assignment holds the authority its admin level has always given it, and a narrow role cannot take over a wider one, so a help desk is refused a domain administrator's password reset. Administrator roles end to end has the permissions, how to give a role, and the two stores only the administrator password opens.

The rest of the list

Auto-ban is on by default: 3 failures within 30 minutes, a 60-minute ban. MaxInvalidLogonAttempts at 0 disables it, and AutoBanMinutes at 0 disconnects the client but bans nothing. Account lockout (AccountLockoutThreshold) is off until set, and counts by the name being guessed at. IP ranges, auto-ban, lockout and the rest has both in full.
The server-enforced administrator second factor refuses the password alone everywhere, the REST API included.
On Windows, move a service still on LocalSystem with the installer, the Control Panel's button, or from an elevated Command Prompt in Bin: hMailServer.exe --confirm /GrantServiceAccess, then, only if that ended in 0, hMailServer.exe --service-account virtual /Register, then net stop hMailServer && net start hMailServer. Check first what the virtual account cannot do, listed in Installing on Windows.
The event-script flaw fixed in 6.3.4 (CVE-2026-100741; its record was not yet public when this was written) is in every release from 6.0.0 to 6.3.3: upgrade, and until you can, switch event scripting off. See Event scripts and the 6.3.4 security fix and the 6.3.4 security announcement.

What it does not do

Passkeys, single sign-on and the session policies do not reach IMAP, POP3 or SMTP, which still take the password or an app password.
The breach check does not cache, and does not expire a password that enters the corpus later.
Compromised-account detection does not drop a session that is already up.
The stored-secrets key protects against a copy of the database that leaves without the file, not against a local account on the server.

Where to read more

Wiki: Security Hardening.
Documents: SecureDefaults.md, SetupChecklist.md, SecretsAtRest.md, PasswordBreachCheck.md, CompromisedAccounts.md and Roles.md.
The public documentation: chapter 17, Security hardening.
On this forum: Upgrading to 6.3.4, The Control Deck and hMailServer on Linux.

A suspected vulnerability is reported privately, never in a forum post: by a confidential GitLab issue, or by email to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue (SECURITY.md).