Requiring TLS: MTA-STS and TLS-RPT, DANE with ACME, and remote domain policies

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) fixes DANE enforcement deferring mail to a DNSSEC-signed domain whose MX answer came compressed, live.com among them (work item 80), and treats a DNS lookup that fails as a failure rather than a name with no addresses: a delivery whose mail servers' lookup failed tells the sender that the DNS query failed (4.4.4) and retries, a DNSSEC validation that fails says where and why in the debug log, and the application log's lines for a skipped host end with the reason in brackets. The ACME HTTP-01 challenge responder, which served one connection at a time so that idle clients could make a certificate's issuance or renewal fail, now serves up to 64 connections at once (work item 97). See the 6.3.7 announcement. In hMailServer 6.3.8 (2 November 2026) the web services listener refuses an address an auto-ban or an IP range bans (work item 105), so an IP range of your own that allows some addresses none of SMTP, POP3 and IMAP now closes that listener to them, and with it the MTA-STS policy this topic publishes and ACME's http-01 validation where that listener answers it: check that the sending MTAs and the certificate authority do not come from such a range. See the 6.3.8 announcement.

This topic is about mail travelling encrypted between servers, in both directions, and about being able to show that it did: honouring the MTA-STS and DANE policies other domains publish, publishing your own, adding a requirement of your own for a partner that publishes nothing, and sending the TLS-RPT and DMARC reports that tell other operators what became of their mail. It assumes a certificate is already installed and the listeners already use it; that is Let's Encrypt certificates are automatic, and the basics of SPF, DKIM and DMARC are in Being trusted by the servers you send to. The one thing to know first: the three outbound policies - MtaStsEnabled, DaneEnforcementEnabled and DnssecValidationEnabled - are on by default, and every outbound path described here that cannot get the encryption it requires defers the message with 4.7.0. Where encryption is required, nothing falls back to sending in the clear.

What is on already, and what is yours to do

Mechanism	Which mail	State as shipped	What you do
Honouring other domains' MTA-STS policies	mail you send	on, MtaStsEnabled=1	nothing
Honouring other domains' DANE records	mail you send	on, DaneEnforcementEnabled=1 and DnssecValidationEnabled=1	nothing
Publishing your own MTA-STS policy	mail you receive	MtaStsHostingEnabled=1, but inert until WebServicesHttpsPort is set	set the port, have a certificate for mta-sts.<domain>, publish the _mta-sts TXT record
Publishing your own TLSA record	mail you receive	nothing to switch on	publish the record the ACME log prints, in a DNSSEC-signed zone
Remote domain policies (since 6.3.4)	both directions	none exist	create one per partner
TLS-RPT and DMARC aggregate reports you send	reports to other operators	inert until a sender address is set	set TlsRptFromAddress and DmarcRptFromAddress

The three outbound policies, the authentication-results headers and the two reporters are on the Control Panel's TLS & certificates → Transport security page (Windows-only); on either platform, PUT /api/v1/settings/ini/{name} and hmctl ini set write them. Of the keys in this topic, WebServicesHttpsPort and AcmeEnabled wait for a reinitialise (Apply now) and the rest apply when saved. Do not edit hMailServer.ini for them: since 6.3.4 the database holds these keys, and a line edited in the file is ignored and named in the error log as HM5804. Settings: when a save takes effect explains the store and [SettingsOverride].

Honouring another domain's MTA-STS policy

It applies to an MX delivery, not to a fixed relay. For each message due for a domain, the server looks up the TXT record at _mta-sts under that domain. With no record, "no policy" is cached for 30 minutes and the message is delivered as usual. With a record, the policy is fetched over HTTPS from mta-sts.<domain> at /.well-known/mta-sts.txt, and its mode decides:

mode	What the server does
none	Delivers as usual
testing	Writes a debug log line only; nothing is enforced
enforce	Drops every MX host the policy does not name. If none is left, the message is deferred with 4.7.0. Otherwise TLS with certificate verification is required to the hosts that remain

How long a policy is believed. A policy is cached until its own max_age and revalidated by re-reading the DNS record's policy id at most once an hour. A DNS failure or a failed fetch during revalidation keeps the cached policy (RFC 8461 §5.1). The mx: patterns match either exactly or with a leading *. that matches exactly one label.

What you see in the log.

SMTPDeliverer - Message 4711: MTA-STS policy mode 'enforce' active for example.com. TLS with certificate validation is required.
SMTPDeliverer - Message 4711: Skipping MX host mx9.example.com - not permitted by the MTA-STS policy of example.com.
SMTPDeliverer - Message 4711: Delivery to example.com deferred. No MX host matches the domain's MTA-STS policy.

Honouring DANE

DANE applies to an MX delivery when DaneEnforcementEnabled=1. The MX lookup for the recipient domain is DNSSEC-validated, and the chain status of the MX RRset decides what happens next.

Step	Status	What happens
The MX RRset	insecure	Delivered as usual
The MX RRset	bogus	The host is skipped entirely
The MX RRset	secure	A TLSA lookup is made per host, at _25._tcp.<mx-host>
The TLSA lookup	bogus	The host is skipped
The TLSA lookup	no records	Delivered as usual
The TLSA lookup	validated	TLS is required and the certificate is pinned to the TLSA records

If no host could be attempted, the message is deferred with 4.7.0.

Three points that matter when you read a log:

• The MX RRset must be DNSSEC-validated too, not only the TLSA record (RFC 7672 §2.2). A TLSA record found under a host name learned from a forged MX answer proves nothing, so the server validates the MX RRset and checks that the host it is about to contact is one of the names that RRset published.
• Only DANE-EE, usage 3, is honoured. A record of any other usage is skipped, issuer certificates are ignored, and the leaf is matched against selector 0 (whole certificate) or 1 (SubjectPublicKeyInfo) with matching type 0 (exact), 1 (SHA-256) or 2 (SHA-512).
• It fails open except for one value. Bogus - a published DNSSEC chain that does not validate - is the only status that can stop a delivery. Every other outcome lands on the ordinary delivery path.

An empty DnssecTrustAnchors means the built-in IANA root anchors. Setting DnssecValidationEnabled=0 reverts to opportunistic, unvalidated TLSA use. With enforcement on, a recipient whose DNSSEC chain is bogus will not receive your mail rather than receive it unencrypted.

SMTPDeliverer - Message 4712: DNSSEC-validated DANE-EE TLSA records found for mx1.example.com. TLS with TLSA certificate matching is required.
SMTPDeliverer - Message 4712: TLSA records for mx1.example.com failed DNSSEC validation. Skipping this host (RFC 7672).
SMTPDeliverer - Message 4712: Delivery to example.com deferred. TLSA records of all MX hosts failed DNSSEC validation.

And in the TCP/IP log, per handshake:

DANE: Session 812: Certificate matched DANE-EE TLSA record (selector 1, matching type 1).
DANE: Session 813: Certificate did not match any DANE-EE TLSA record. Failing TLS handshake.

Both deferral paths use 4.7.0, "other or undefined security status" (RFC 3463).

Publishing your own MTA-STS policy

MtaStsHostingEnabled defaults to 1. The built-in web services server serves /.well-known/mta-sts.txt for any hosted, active domain when the request's Host is mta-sts.<domain>, and generates the body from your live MX records unless MtaStsPolicyMx overrides them:

version: STSv1
mode: enforce
mx: mail.yourcompany.com
max_age: 604800

Setting	Default	Notes
MtaStsHostingEnabled	1	On, but inert until the HTTPS listener exists
MtaStsPolicyMode	enforce	enforce, testing or none; anything else is read as enforce
MtaStsPolicyMaxAge	604800 (7 days)	Clamped to 86400 … 31557600
MtaStsPolicyMx	empty	Comma-separated override; empty derives the list from live MX records
WebServicesHttpsPort	0	Must be set. RFC 8461 §3.3 requires the policy to be fetched over HTTPS

WebServicesHttpsPort is what makes it live. While it is 0 the server logs that the policy cannot be served and that sending servers will treat the domain as having no policy.

1. Set WebServicesHttpsPort.
2. Add mta-sts.yourcompany.com to AcmeDomains, so that the HTTPS listener has a certificate for the name it will be asked for. Every name in AcmeDomains becomes a SAN on one certificate, and each must already resolve to this server (Let's Encrypt certificates are automatic).
3. Decide MtaStsPolicyMode. The default is enforce, under which a sender that cannot verify the certificate on your MX host defers its mail to you. The wiki's DNS page says to start at testing, which, once you publish a _smtp._tls TXT record (v=TLSRPTv1; rua=mailto:...), gives you TLS-RPT reports without losing mail while you find out whether every sender can reach you over TLS; then move to enforce.
4. Publish _mta-sts.yourcompany.com. IN TXT "v=STSv1; id=20260908000000Z".
5. Bump the id whenever you change the policy.

DANE for your own server, with ACME

Every successful ACME issuance writes the exact record to publish into the log:

ACME: DANE TLSA record for this certificate: _25._tcp.<mx-host>. IN TLSA 3 1 1 3b1f…

That is 3 1 1: DANE-EE, SubjectPublicKeyInfo, SHA-256. The digest is over the key, so the record stays valid across a renewal only when the key is kept, which is what AcmeReuseKey (default 1) does. When it is on and a privkey.pem already exists, the log says ACME: Reusing the existing certificate key (keeps published TLSA records valid). Otherwise a fresh 2048-bit RSA key is generated. Leave it on if you publish TLSA records.

The wiki's DNS page adds three things. Your zone must be DNSSEC-signed for the record to mean anything. Publish the record for every host named in your MX records. And re-publish before you rotate a certificate, never after. The Control Panel's DNS records page does not generate the TLSA value.

A requirement of your own: remote domain policies

Since 6.3.4 (schema 6045). In 6.3.3 the only ways to insist on TLS to a destination are the server-wide delivery security setting, a route's own connection security, and the MTA-STS or DANE policy the domain publishes itself. A remote domain policy is you saying the same for a domain that publishes nothing: a partner, a bank.

Where it lives. Policies are rows of hm_remotedomainpolicies, take effect for the next message without a restart, are part of the configuration backup, and have nothing in hMailServer.ini. Four surfaces write the same records:

Where	How
Control Panel (Windows)	Mail flow & delivery → Remote domains, beside Routes
Control Deck	Remote domains in the navigation
REST API	GET/POST /api/v1/remote-domains, PUT/DELETE /api/v1/remote-domains/{id}, GET /api/v1/remote-domains/effective?domain=...
COM (Windows)	Application.Settings.RemoteDomainPolicies

All are server-administrator only. An API key restricted to named domains is refused, because a policy names somebody else's domain and decides this server's posture.

A policy is not a route. A route says where mail for a domain goes and replaces the MX lookup. A policy says what this server will and will not do when it gets there, however the server was found. A domain may have both, and the policy applies to a routed delivery exactly as it applies to an MX one.

Which policy applies. A policy names a domain (bank.example) or a pattern (*.example, *), and the most specific active one governs:

1. an exact name beats any pattern, however long the pattern;
2. between two patterns, the one with more literal characters wins;
3. an inactive policy is skipped, and the next most specific active one governs in its place. Switching off the entry for one domain beneath a * that requires TLS removes an exception; it does not exempt that domain.

GET /api/v1/remote-domains/effective?domain=mail.bank.example (and PolicyForDomain over COM) answers which record a delivery will use, before a message has to find out. When one connection carries recipients of several domains, the strictest combination applies: the highest TLS requirement, and the smallest of each non-zero limit.

Outbound TLS.

Value	What delivery to the domain requires
none (default)	Nothing beyond what the server does anyway: the delivery security setting, a route's security, and MTA-STS or DANE if the domain publishes them
encrypted	STARTTLS must be offered and the handshake must succeed. The certificate is not judged - the level for a partner with a self-signed or internal certificate
verified	Encrypted, and the certificate must chain to a trusted root and name the host connected to - what MTA-STS enforce asks
dane	Encrypted, and the certificate must match a DNSSEC-validated TLSA record. The TLSA lookup is made even with DANE enforcement off and even for a routed delivery; a host with no validated record is skipped, so with DnssecValidationEnabled=0 every delivery under the policy is deferred

It can only raise the requirement. The policy, MTA-STS enforce and a usable TLSA record combine as the strictest of the three, never the newest. A domain that publishes MTA-STS enforce and is named here at encrypted still gets certificate verification. No value here weakens what a domain published for itself.

A delivery it cannot satisfy is deferred, never sent in the clear. The recipient's status is 4.7.0, the message stays queued and is retried on the usual schedule, and the log line - and the bounce, if the retries run out - names the rule:

Delivery deferred: the remote domain policy for bank.example requires TLS with a
certificate that verifies to this destination, and it could not be established.
The message has NOT been sent in the clear.

It is a deferral rather than a bounce on purpose: the far end may fix its STARTTLS or its certificate before the queue gives up, which is how RFC 8461 §5 reasons about MTA-STS.

Inbound. With Require TLS on mail from this domain set, an unencrypted session whose envelope sender is at the domain is answered 530 5.7.0 Must issue STARTTLS first at MAIL FROM. It is not applied to an authenticated submission, nor to a connection from a configured incoming relay: behind a front-end filter the hop the policy is about happened before that machine, and TLS has to be required there. The envelope sender is whatever the connecting server claims, so this keeps honest partners encrypted and is not an anti-spoofing control.

The rest of the record. The same row carries the largest message this server will send to the domain (beyond it no connection is opened and the message is bounced at once with 5.3.4), the simultaneous connections and the messages a minute (beyond either, deferred with 4.4.5 for one to five minutes), whether automatic replies and forwards may go there, a backup-MX recipient callout, and since 6.3.4 (schema 6094) Encrypt mail to this domain at this server. The ceilings are in Outbound shaping and IP warm-up, gateway encryption in End-to-end encryption.

Telling other operators what happened: the reports you send

Two aggregate reporters, both scheduled hourly and both inert until a sender address is set. They flush their statistics either way, so switching one on does not start with an empty history.

	TLS-RPT (RFC 8460)	DMARC aggregate (RFC 7489)
Address setting	TlsRptFromAddress	DmarcRptFromAddress
Organisation name	TlsRptOrganizationName, default hMailServer	DmarcRptOrganizationName, default hMailServer
Recorded by	ExternalDelivery, per delivery attempt, classified tlsa, sts or no-policy-found	SpamTestDMARC, for every message evaluated against a published policy, passes included
Schema	-	DmarcRptSchemaVersion, default 1 = RFC 7489 Appendix C; 2 = RFC 9990

DMARC reports are sent to an address outside the policy domain only when that address's own DNS says it wants them, so a domain cannot use its DMARC record to aim this server's reports at somebody else.

Recording what this server concluded. Three related settings, the first two off by default:

Setting	Default	What it writes
AuthenticationResultsEnabled	off	One Authentication-Results header per accepted inbound message (RFC 8601), with the SPF, DKIM and DMARC verdicts this server reached. Only checks that ran are reported
ReceivedSpfHeaderEnabled	off	A Received-SPF header (RFC 7208 §9.1), written only when the SPF check ran
AuthenticationResultsIdentity	empty = this computer's name	The authserv-id, always lower-cased

An arriving message that carries an Authentication-Results header claiming this server's identity has that header removed before the server writes its own; a header naming any other identity is left untouched.

The reports other servers send about your domains go to the rua= address in your _dmarc and _smtp._tls records. Since 6.3.4 a domain's report mailbox reads them: see DKIM rotation, DMARC and TLS reports.

A checklist

1. AcmeEnabled=1, AcmeDomains naming mail., mta-sts. and autoconfig., and port 80 reachable (the ACME topic).
2. Check TLS & certificates → Transport encryption overview, which says whether anything still carries a password in the clear.
3. Leave MtaStsEnabled, DaneEnforcementEnabled and DnssecValidationEnabled at 1.
4. Set WebServicesHttpsPort, start MtaStsPolicyMode at testing, and publish _mta-sts and _smtp._tls in DNS.
5. Publish the TLSA record the ACME log prints, if your zone is signed.
6. Since 6.3.4, name the partners that must never get mail in the clear as remote domain policies at encrypted or verified.
7. Turn on AuthenticationResultsEnabled, and set TlsRptFromAddress and DmarcRptFromAddress.
8. Diary the first renewal: it falls two-thirds of the way through the certificate's life, and the log line to look for is ACME: Certificate issued successfully.

What this topic does not cover

• The connection to the database ([Database] TlsMode and TlsCaFile, since 6.3.4): Choosing a database and encrypting the link.
• Protocol versions, cipher suites, session resumption and client certificates: TLS on the listeners.
• Encrypting the message rather than the connection. TLS ends at the next server; End-to-end encryption is about what happens after that.

Where to read more

• Wiki: Encryption and Certificates, §9.6 to §9.10; DNS for Your Domain, for the records themselves; Routes and Relays, §14.2a, for the whole remote domain record.
• Operator document at the v6.3.4 tag: RemoteDomains.md.
• Public documentation: 9.4 Outbound transport security and 8.7 MTA-STS.
• On this forum: Let's Encrypt certificates are automatic, Being trusted by the servers you send to, and Upgrading to 6.3.4 for what changes about hMailServer.ini.