TLS on the listeners: your own certificate, versions, ciphers and client certificates

Progressiverobot

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) renames two Control Panel pages this topic names: SSL certificates is TLS certificates and SSL/TLS is TLS versions & ciphers; Ctrl+K takes every old title and route to its page. See the 6.3.7 announcement. On Windows, hMailServer 6.3.8 (2 November 2026) reads each certificate and private key file, dh2048.pem and a port's client-certificate CA file from the file configured, and refuses with HM7536 a name it cannot hand over as it is named (work item 115): a port whose certificate or key such a name gives does not start, a client-certificate CA file verifies no certificate, and dh2048.pem is left out as a missing one is. Before upgrading, give each such file whose path holds characters outside ASCII, and the folders above it, an ASCII name; the Control Panel shows a certificate or key file whose name is refused as Name refused. On Linux every name goes over in UTF-8, so on a server that ran under a LANG that is not UTF-8, rename such a file to its UTF-8 name before upgrading. See the 6.3.8 announcement.

This topic is the manual half of TLS on the SMTP, IMAP and POP3 listeners: your own certificate and where the Windows service may read it, then versions, ciphers, key exchange, session resumption, client certificates with SASL EXTERNAL, and certificates for IP addresses. Know this first: a port set to use TLS whose certificate cannot be loaded does not fall back to plain text. It does not start. The application log says HM5113, and the only other symptom is a listener missing.

Where each thing is set, and when it applies

What	Set in	Applies
Certificates, and which port uses which	Control Panel: TLS & certificates → SSL certificates and Connections & protocols → TCP/IP ports; the Deck's certificates and listeners views; `/api/v1/certificates` and `/api/v1/ports`; `hmctl`'s `certificate` and `port` resources; COM	At a reinitialise, or a service restart
TLS 1.0 to 1.3, the cipher list for TLS 1.2 and below, the two cipher-order switches	Control Panel: TLS & certificates → SSL/TLS; `PUT /api/v1/settings`; `hmctl settings set`; COM `Settings`	See the note below
`TlsCipherSuites13`, `TlsKeyExchangeGroups` and the four session settings	The same page; `PUT /api/v1/settings/ini/{name}`; `hmctl ini set`; on Linux `hmailserver --set-setting`	At a reinitialise

A reinitialise is Apply now on the Control Panel's pending bar or in the Deck's Settings view, POST /api/v1/server/reinitialize, or Application.Reinitialize over COM. Every service stops and starts again in the same process; it ends the Deck's session and any drain in progress. GET /api/v1/server/settings/pending lists the [Settings] keys saved and not yet running, such as TlsKeyExchangeGroups; a port, a certificate, the versions and the cipher list are never in it. The Control Panel and COM are Windows-only; on Linux use the Deck, REST or hmctl.

Do not edit hMailServer.ini. Since 6.3.4 the database holds the [Settings] keys, so a line changed in the file is put back (HM5804) or left unused; Settings: when a save takes effect explains the store and the reinitialise.

When the versions and the cipher list apply. They and the two order switches are database settings (hm_settings rows), not [Settings] keys, so SettingsApply.md gives them no mark. Roadmap.md says they are applied in the call that builds each listener's TLS context, and SettingsApply.md says those contexts are built when the servers start; the wiki's Control Panel Pages page, written against 6.2.28, says the SSL/TLS page's COM settings apply at once. Press Apply now after changing any of them, and check from outside.

Adding your own certificate

Two PEM files. The certificate file must hold the leaf and the issuer chain: the server loads it twice, once as the certificate and once as the chain, and reports each failure separately. The private key is the second file.

A passphrase-protected key is supported, per certificate. Give it in the add dialog, on the SSL certificates page's Private key passphrase card, in the Deck, or as private_key_password in POST /api/v1/certificates. It is write-only and kept under the machine's own secret store. The built-in backup and restore carries it to a new machine; a copied or shared database does not, so on a warm standby enter it again after a failover, and again on the primary after failing back, or the key does not load there and the TLS ports that use it do not start. An encrypted key with no passphrase is HM6170, and its port does not start.

Where the files go. On Windows, a new 6.3.4 installation runs the service as NT SERVICE\hMailServer, which reads only what it has been granted (next section); put the files in Data\Certificates (C:\Program Files\hMailServer\Data\Certificates on a default installation), where a new 6.3.4 installation's self-signed pair is. An upgraded Windows server keeps the account it had; LocalSystem can read a file anywhere. On Linux the service runs as User=hmailserver under a unit with ProtectHome=yes, so nothing under /home is visible to it. The self-signed pair of a new installation is in /var/lib/hmailserver/Certificates, its key mode 0600 owned by the service's user; keep yours beside it, readable by that user.

Copy the certificate file and the key into that folder.
Add the certificate: on Windows, TLS & certificates → SSL certificates → Add certificate (a name, the PEM certificate file, the PEM key file, an optional passphrase); or the Deck's certificates view; or POST /api/v1/certificates with name, certificate_file and private_key_file. A file that is not there is refused (400). The Control Panel's grid then shows whether the certificate parses, when it expires, and whether the key is encrypted and matches.
On Windows under the virtual account, run Grant access (next section), so it reads the new paths.
Set the certificate on every TLS port: TCP/IP ports → Edit, the certificate list. Over REST, PUT /api/v1/ports/{id} is the whole record and fields left out take their defaults, so send protocol, address, port, connection security, certificate_id and the two client-certificate fields every time.
Reinitialise.
Check the application log for the codes below and, on Windows, Monitoring & troubleshooting → Server status for the listeners actually running and TLS & certificates → Transport encryption overview, which judges ports, certificates, versions and ciphers together.

From a shell over the REST listener: add the certificate, then, once the ports are set, reinitialise and read the banner over implicit TLS. --from-json with - reads the body from standard input, which keeps a passphrase off the process list; the JSON holds the fields of step 2 and, for an encrypted key, private_key_password.

hmctl certificate create --from-json - < certificate.json
hmctl api POST /api/v1/server/reinitialize
openssl s_client -connect mail.example.com:465 -quiet

Code	Message	Meaning
`HM5113`	`Failed to load certificate file` / `…certificate chain…` / `…private key file`	The path, the format or the passphrase is wrong, the file cannot be read, or the key does not match. The port does not start
`HM5113`	`Error initializing SSL. Certificate not set.`	A TLS port has no certificate. The port does not start
`HM6170`	—	An encrypted key with no passphrase stored. The port does not start
`HM5991`	`The TLS certificate … expired on …` / `… is not valid until …`	Reported at startup; the port starts and offers it anyway, and clients refuse it. A future date usually means the server's clock is wrong
`HM5603`	`Unable to enable Diffie-Hellman key agreement`	`dh2048.pem` is missing: from `Bin` on Windows, from beside the binary in `/usr/bin` on Linux

Replacing it later. When ACME renews, it logs ACME: Restarting servers to load the new certificate.; after replacing a file by hand, reinitialise likewise. If you publish DANE records, publish the new TLSA record before you rotate the certificate, never after; GET /api/v1/tlsa gives one 3 1 1 record per configured certificate, for _25._tcp only. Since 6.3.4 the alert certificate.expiring is on as shipped (7 days ahead), but nothing is mailed until AlertRecipient and AlertSenderAddress are set; the metrics listener exports hmailserver_tls_certificate_expiry_seconds. Since 6.3.4 a COM script's change to an SSL certificate is not in effect until Save(), and one never saved changes nothing, so a script that relied on an unsaved change must now save; the listeners still load the certificate at a reinitialise.

ACME alongside. The built-in client gives its certificate to TLS ports that have none or hold the installation's self-signed one, and leaves alone a port that names a certificate somebody chose.

The HTTPS listeners take files, not a certificate record: RestApiCertificateFile, WebServicesCertificateFile and MetricsServerCertificateFile, each with its private-key partner, all waiting for a reinitialise. ManageSieve reuses the certificate configured on an IMAP, POP3 or SMTP port.

The service account's folder rules (Windows, since 6.3.4)

hMailServer.exe /GrantServiceAccess, run elevated, grants NT SERVICE\hMailServer Modify on the program folder's own Data, Logs, Temp and Database, Read on its other folders, and what an administrator approved. It is the Grant access… button on Maintenance → Server limits & expert settings, card Windows service account, or, from an elevated Command Prompt in Bin:

hMailServer.exe --confirm /GrantServiceAccess

It checks every certificate and key file the configuration names (the SSL certificates, DKIM keys, and the REST, metrics and web-services files) against those grants. A file in a folder they reach needs nothing; where the file's own permissions keep the folder's grant from it, the grant gives Read on that file alone. A file they do not reach is listed OUTSIDE and is never given Read where it is, because the configuration could name any file on the machine. Move it into Data\Certificates, or approve its folder for Read with --approve-read "<folder>" or Yes at the question --confirm asks; approve only a folder you recognise. Some folders are refused even when approved, a share and a drive's root among them. ServiceAccount.md does not list the client-certificate CA bundle among the files the grant checks, so keep that bundle in Data\Certificates too. The account's rules in full, from what is never granted to the exit codes, are in Installing on Windows.

Protocol versions and the legacy switches

Version	Bit in `SslVersions`	COM	REST, in `PUT /api/v1/settings`
TLS 1.0 (legacy)	2	`TlsVersion10Enabled`	`tls_version_10_enabled`
TLS 1.1 (legacy)	4	`TlsVersion11Enabled`	`tls_version_11_enabled`
TLS 1.2	8	`TlsVersion12Enabled`	`tls_version_12_enabled`
TLS 1.3	16	`TlsVersion13Enabled`	`tls_version_13_enabled`

The shipped value is 24, TLS 1.2 and 1.3 only; SSLv2 and SSLv3 cannot be switched on at all. Do not re-enable TLS 1.0 or 1.1 unless a specific old client forces you to, and plan to replace that client. The versions and the cipher list also govern the connections this server makes when it delivers. The REST, web-services HTTPS and metrics listeners add a TLS 1.2 minimum of their own, so TLS 1.0 switched on for an old mail client stays off for the management API, and a server left with only TLS 1.0 and 1.1 leaves those three listeners with no version at all. Every version switched off is HM5990 and falls back to TLS 1.2 and 1.3; turning TLS off is done per port.

Cipher suites: `SslCipherList`, `AEAD-ONLY` and `TlsCipherSuites13`

SslCipherList is an OpenSSL cipher string for TLS 1.2 and below only; it does not touch TLS 1.3. The shipped value is the long-standing list, which also admits CBC-mode suites and static-RSA key exchange. A string OpenSSL rejects is HM5511, and the previous list stays.
AEAD-ONLY is a named preset, not an OpenSSL string: set SslCipherList to that word and the server applies a vetted list of forward-secret AES-GCM and ChaCha20-Poly1305 suites (ECDHE and DHE), excluding every CBC-mode suite and static-RSA key exchange. The cost: TLS 1.2 clients that speak only CBC cannot connect, and TLS 1.0 and 1.1 are left with no usable suite, so never enable them alongside it; the Transport encryption overview flags that combination. A misspelled preset goes to OpenSSL, which rejects it, and HM5511 names the string.
TlsCipherSuites13 takes RFC 8446 names, colon separated, for example TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256. Empty, the shipped value, leaves OpenSSL's defaults. A list rejected outright is HM6120 and the defaults stand; names this build does not know are named in HM6121 and the rest of the list is applied.
Prefer server cipher order and Prioritize ChaCha20-Poly1305 (TlsOptionPreferServerCiphersEnabled, TlsOptionPrioritizeChaChaEnabled; over REST tls_prefer_server_ciphers, tls_prioritize_chacha) are both off as shipped. The second works only with the first on and TLS 1.2 or 1.3 enabled.

A compliance baseline from a shell, then a reinitialise:

hmctl settings set --set ssl_cipher_list=AEAD-ONLY --set tls_prefer_server_ciphers=true
hmctl ini set TlsCipherSuites13 TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
hmctl api POST /api/v1/server/reinitialize

The ini line takes the administrator password (--password-file or HMAIL_PASSWORD); an API key is refused there with 401.

Key exchange: the hybrid post-quantum groups

TlsKeyExchangeGroups ships as X25519MLKEM768:SecP256r1MLKEM768:X25519:secp384r1:secp256r1: two ML-KEM hybrids first, then the classical curves, so a modern peer agrees a hybrid and an older one falls back. The list reaches every TLS context in the server: the mail listeners, ManageSieve, the three HTTPS listeners and the outbound client, so mail this server sends uses it too. A list OpenSSL rejects, an empty one, or one that leaves no group, is HM5720 and falls back to secp384r1:x25519:secp256r1, without the two hybrids; if even that is refused, HM5721 and OpenSSL's own defaults.

Session caches, tickets and ticket-key rotation

All four are on the SSL/TLS page's Session resumption tab and wait for a reinitialise. At their defaults three of them make no OpenSSL call at all.

Setting	Default	A non-default value
`TlsSessionTicketsEnabled`	on	Off: no tickets on any version. TLS 1.2 clients fall back to the in-process cache; TLS 1.3 clients get no ticket
`TlsSessionCacheSize`	0, OpenSSL's cap of 20480	A positive value replaces the cap; `-1` turns the server-side cache off, leaving tickets alone
`TlsSessionTimeoutSeconds`	0, OpenSSL's 300	Bounds how long a leaked resumption secret stays useful
`TlsTicketKeyRotationSeconds`	0, one key for the life of the process	A captured ticket can be decrypted for at most two intervals; `86400` is a day

Rotation is the one worth turning on for forward secrecy. Without it every ticket is sealed under one key made at startup, and that key, recovered later from a memory dump or a hibernation file, opens every ticket ever recorded. Rotation is process-wide. Each listener keeps its own session-ID context, so a session made on a port that asks for no client certificate is never resumed on one that requires it.

Client certificates (mutual TLS) and SASL `EXTERNAL`

The policy is per port, on the TCP/IP port dialog (COM ClientCertificatePolicy and ClientCertificateCAFile; REST client_certificate_policy and client_certificate_ca_file).

Policy	Behaviour	CA bundle cannot be loaded
Off (0)	Nothing is asked. The default	—
Request (1)	Asked, verified if offered, the outcome logged; the handshake never fails over it	`HM6140`, and the port starts anyway
Require (2)	The handshake fails unless the client presents a certificate chaining to the port's CA bundle	The listener is not started

The bundle is the only trust anchor for that port, not the Windows certificate store. A policy on a port without TLS is refused when you save it; Roadmap.md adds a policy with no CA file, and Require on a STARTTLS (optional) port, where a client that never issues STARTTLS is never asked. A hand-edited database that has one anyway is HM6142 (require stops the listener, request only logs). Start with Request to see which clients could survive Require; Require on the port ordinary mail clients use would lock out every one without a certificate.

EXTERNAL. A verified client certificate that names an address makes AUTH EXTERNAL (RFC 4422) available on that connection, on SMTP, IMAP and POP3, and to nobody else. It signs in as the mailbox the certificate names: a subjectAltName rfc822Name, the subject's emailAddress, or an address-shaped common name; a certificate for one user is never a credential for another. Since 6.3.4 such a sign-in also meets the domain's and the account's country policy, and an SMTP session signed in with it is no longer refused its second message with 550. Since 6.3.4 a mailbox encrypted at rest refuses EXTERNAL on IMAP and POP3, opening with its own password only.

A certificate for an IP address (Linux, since 6.3.4)

This is about certificates this server checks when it connects out, not the one it serves. An endpoint given by its address (an object store, a webhook, the update feed, an SMTP route, the syslog sink) presents a certificate with the address in an iPAddress entry, where RFC 5280 puts it. The Linux check read only dNSName entries and the common name, so such a certificate was refused; it is accepted now, and the HTTPS client no longer sends an address as the server name (SNI), as the IMAP migration client already did not. Newly refused on Linux: a wildcard taken to cover an address, and the common name of a certificate whose iPAddress entries name other addresses. Windows always accepted such certificates.

What it does not do

It does not obtain or renew certificates: that is ACME, in Let's Encrypt certificates are automatic.
Connection security per port, and what a mail client is told: Connecting a mail client.
MTA-STS, DANE, TLS-RPT and remote domain policies: Requiring TLS. The hardening checklist: Hardening a public mail server.
TLS to the database is [Database] TlsMode and TlsCaFile, with no Control Panel or Deck editor: Choosing a database and encrypting the link.

Where to read more

Wiki: Encryption and Certificates §9.2 to §9.4 and §9.12, Ports Reference §3.4, Settings Reference, Security Hardening §17.3.
Documents: ServiceAccount.md, SettingsApply.md, SecureDefaults.md, SecretsAtRest.md, DatabaseTls.md, AlertsAndAuditTrail.md.
On the forum, for the certificate-expiry alert and where it is mailed: Alerts, the audit trail, syslog and the journal.
The documentation page, 9.2 The manual way.