Troubleshooting by symptom: no mail in, no mail out, a service that will not start

Progressiverobot

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) changes what an upgraded server does without anything being switched on, and some of it looks like a fault. A forward, a redirect or a rule's forward of mail from outside that did not pass DMARC for the forwarding domain's organisation keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. An authenticated submission is signed as another hosted domain only when the account may send as the address its From names, and is otherwise sent unsigned as that domain. A domain's or an account's rule may no longer set From, Sender, Resent-From, Resent-Sender, DKIM-Signature or the loop count, and a Sieve script may no longer add or delete them: one stored before the upgrade still runs without that action, and the application log names it. RuleLoopLimit 0 stops out-of-office replies. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries. On Windows the x64 installer's MariaDB Connector/C 3.4.11 has seven authentication plugins where 6.3.6's had nine, so a database account that signs in with MySQL Enterprise's Windows authentication can no longer sign in and the server cannot open its database. An error log that was quiet may now name database failures that were hidden (HM5032), and the Control Panel's API & monitoring page is Monitoring & alerts. Fixed from reports: the Control Panel's IP ranges page that could show no list at all, DANE enforcement deferring mail to a DNSSEC-signed domain whose MX answer came compressed (live.com among them), and classic Outlook's POP3 downloads stopping at the UTF8 it sends after the login. See the 6.3.7 announcement. After the upgrade to hMailServer 6.3.8 (2 November 2026), TlsMode=verify-full on MySQL or MariaDB with no TlsCaFile set is refused with HM7535 on a connection to a database server on this machine, and the database is not reached; on Windows a TlsCaFile or PostgreSQLSslRootCert whose path holds characters outside ASCII may be refused with HM7534, with the same result, and a TLS certificate or key file so named with HM7536, so that the port it is for does not start; and a Linux server that ran under a LANG that is not UTF-8 no longer finds a file or folder named outside ASCII until it is renamed to its UTF-8 name. A client that signs in with another spelling of the account's address than the stored one, in more than the case of the letters A to Z, or with white space at either end, may no longer sign in. With the web services listener on, an address whose IP range allows none of SMTP, POP3 and IMAP - an auto-ban's, or one of your own that allows nothing - is closed there with no answer, and with it CalDAV, CardDAV, Autodiscover, autoconfiguration, MTA-STS and, when this listener answers it, ACME's http-01 validation. See the 6.3.8 announcement.

This topic is the wiki's Troubleshooting chapter arranged by the sentence support threads open with: mail from outside does not arrive, a client cannot send, the Control Panel or the Deck will not connect, the service will not start. It also covers the server's replies, the Diagnostics self-test, the message trace and what only looks like a fault. The one thing to know first: on Windows the service tells the Service Control Manager it is running before it has opened its database or a single port, so "the service is running" proves nothing. Today's ERROR_hmailserver_<date>.log does: it almost always holds the reason a start failed.

Start here

Three pages, in this order, under Monitoring & troubleshooting in the Control Panel (Windows):

Diagnostics, the built-in self-test (below). It names most problems outright.
Live logs, while you reproduce the problem, with debug messages on if the answer is not obvious, and off afterwards.
Delivery queue: is mail stuck, and what do NextTry and Tries say?

On Linux the Deck's Diagnostics, Logs and queue views read the same, and three commands start every investigation:

systemctl status hmailserver
journalctl -u hmailserver -n 100 --no-pager
sudo tail -n 100 /var/log/hmailserver/ERROR_hmailserver_$(date +%F).log

Which problem is it? Not running, or nothing listening: The service will not start. A sender got an SMTP error: the reply names the cause. A sender timed out, or outbound mail sits in the queue: Diagnosing stalled mail. Inbound with nothing at all: I can't receive. Outbound that never reaches the queue: I can't send.

"I can't receive mail from outside"

Work down the table in order, cheapest check first.

Check	How (Windows)	A failure means
Is the service running?	`Get-Service hMailServer`	Start it, then read the error log for why it stopped
Is anything listening on 25?	`Get-NetTCPConnection -LocalPort 25 -State Listen`	The port is not configured, is bound to one address only, or another program has it: Connections & protocols → TCP/IP ports
Is 25 open in Windows Firewall?	`Get-NetFirewallRule -Enabled True \\| Where-Object DisplayName -like "hMail"`	Open it: see the firewall paragraph below
Does the machine itself answer on 25?	`Test-NetConnection 127.0.0.1 -Port 25`	Then it is the server, not the network
Is 25 forwarded by your router or cloud firewall?	Its configuration	The far end sees a refusal or a timeout
Does your ISP block 25?	Ask them	Common on consumer lines: a relay host or a business line
Is the MX record right?	The MX query page, or `Resolve-DnsName example.com -Type MX`	Fix DNS
Can the world reach you?	From outside: `Test-NetConnection mail.example.com -Port 25`	It is the path in
Is the domain in hMailServer and Active?	Accounts & domains → Domains	An inactive domain is not accepted for
Is the mailbox Active, and full?	The account dialog	A full mailbox gets a temporary failure, and the sender retries

A port bound to one internal address answers only on the LAN. Server status lists the listeners the service actually brought up.

The firewall. The installer opens no port, so nothing from outside reaches the server until you open them, on the machine and on the router or cloud firewall in front of it. Which ports, and the rules for Windows and Linux, are in Every port hMailServer listens on or connects to.

"I can't send mail"

The reply names most sending faults. In an ESMTP session an enhanced status code follows the number: 530 Must issue STARTTLS first. arrives as 530 5.7.0 Must issue STARTTLS first.

Reply	Likely cause	What to do
`550 Delivery is not allowed to this address.` ("relay denied")	The client is not authenticating, or uses 25 instead of 587	Port 587 with authentication; a printer that cannot authenticate gets its own range under Access & abuse protection → IP ranges
`530 SMTP authentication is required.`	The same, said earlier	As above
`530 Must issue STARTTLS first.`	The port requires TLS first	STARTTLS or SSL/TLS in the client
`535 Authentication failed.`	Wrong password, not the full address as user name, or an inactive account	Also check the auto-ban
`535 Authentication failed. Too many invalid logon attempts.`	The auto-ban: 3 failures, 60 minutes, by default	Below
`550 Too many recipients.`	Too many recipients in one transaction	Below
`552 5.3.4 Message size exceeds fixed maximum message size. Size: N KB, Max size: M KB`	Over the server's maximum message size (20480 KB), or the domain's own	Connections & protocols → Protocols, SMTP tab, or the domain dialog
`452 4.3.1 Insufficient system storage.`	Free space on the store's volume below `MinimumFreeDiskSpaceMB` (100 MB)	Free space; senders retry

A banned client. The ban is an expiring IP range: Access & abuse protection → IP ranges lists it, and deleting it releases the client (DELETE /api/v1/ipranges/<id>, or the Deck's IP ranges view).

550 Too many recipients. Two wiki pages disagree. Troubleshooting §19.3 ties it to maxsmtprecipientsinbatch (100) and says split the batch or raise the limit; the Architecture page's RCPT TO chart refuses at 50,000 recipients and uses MaxSMTPRecipientsInBatch to batch outbound deliveries. Splitting the message is right either way; raising the setting also changes the outbound batch size.

What the server's replies mean

A 4xx is temporary and the sender retries; a 5xx is permanent and the sender bounces.

Reply	Meaning
`421 4.7.0 Too many invalid commands. Bye!`	Disconnected for misbehaving: a scanner, or a broken client
`451 4.3.1 Server temporarily overloaded while accepting the message; please retry.`	The accept pipeline exceeded `FinalizationTimeout`: Diagnosing stalled mail
`451 4.3.2 Unable to verify the recipient at the moment. Please retry later.`	The database did not answer, so the mail is deferred, not bounced. Check `/readyz`
`451 4.3.2 The server is shutting down and cannot accept the message.`	A graceful stop with `ShutdownDrainSeconds` set
`452 4.2.2 Mailbox is full.`	At or over quota, refused during the conversation; the holder had a `QUOTA_WARNING`
`550 Unknown user`	No such mailbox, alias or list
`553` or `554` from a remote server	The far end refused you; its text is the diagnosis, usually SPF, DKIM, PTR or a block list

"My mail goes to spam"

Almost always DNS, in this order: the PTR record (nothing compensates for a missing or generic one), SPF, DKIM, DMARC even at p=none, then the block lists. With split-horizon DNS, the outside view is the one that counts. Being trusted by the servers you send to covers it.

The Control Panel, the Deck or the REST listener will not connect

The Control Panel (Windows) talks to the running service over COM: no service, no connection. Use the administration password, not a mailbox's. After a restart it reconnects by itself; after you stop the service it deliberately does not. Remotely it needs port 135 and DCOM. Every page saying "Server unavailable" means the engine is paused: Server status → Resume. Since 6.3.4, under NT SERVICE\hMailServer, a change of the administrator password is refused because the service may not write the file: run hMailServer.exe /SetAdministratorPassword from an elevated prompt, then restart the service. More in The Control Panel on Windows.

The Deck and the REST API share one listener, off until RestApiPort is set. It refuses to start without an administrator password, beyond loopback without a certificate, or with a certificate it cannot load (HM5113 names the file), each logged once as RestApi: Refusing to start; the three lines and their fixes are in The REST API: turning it on. Only a RestApi: Listening on line proves it is up. Since 6.3.4 the port waits for a reinitialise (Apply now, or POST /api/v1/server/reinitialize), and a line edited into hMailServer.ini after the first start loses to the stored value (HM5804). With the listener off, on Linux:

sudo hmailserver --set-setting RestApiPort=8045
sudo systemctl restart hmailserver

On Windows, the Control Panel's API & monitoring page (Monitoring & alerts since 6.3.7). "Web administration page not installed" means the WebAdmin files are missing, or ProgramFolder does not point at the directory holding them. Repeated bad passwords get the address refused for five minutes; loopback is exempt. The Deck itself is The Control Deck.

The service will not start

The start, in order; where it stops names the problem:

Windows is told the service is running, before any of what follows.
hMailServer.INI is read, alone, which is how the database is found. A [Database] Type it does not recognise abandons the start, with nothing logged about a database. On Linux, sudo -u hmailserver hmailserver --check-config then prints Database type: 0.
HM5005: Server or Database is missing for the configured Type.
The database is connected, retried ConnectionAttempts times (6), ConnectionAttemptsDelay seconds apart (5). HM4354: the database server is down or moved, or the credentials changed. On Windows the [Database] password is DPAPI-protected per machine and cannot be copied from another server. Since 6.3.4, HM7530 (an unknown TlsMode) and HM7531 (verify-full with a missing TlsCaFile) stop it too, and since 6.3.8 HM7535 (verify-full on MySQL or MariaDB with no TlsCaFile set, to a database server on this machine: name the CA that issued its certificate in TlsCaFile, or set TlsMode=require) and, on Windows, HM7534 (a TlsCaFile or PostgreSQLSslRootCert name the client cannot be handed as named: give the file and the folders above it ASCII names); --check-database makes one connection and says what it got.
The schema version, compared both ways. HM5010: Database version could not be detected. HM5011 too old: DBUpdater.exe from Bin on Windows, sudo -u hmailserver hmailserver --upgrade-database on Linux. HM5011 too new: install the matching or a newer server; never downgrade the database. Since 6.3.4, [Database] UpgradeOnStart=1 makes the service upgrade the schema as it starts: off by default, no backup first, one node at a time.
Settings are read again from the database, since 6.3.4 with any [SettingsOverride] keys over them. A configuration that will not load abandons the start.
One listener per TCP/IP ports row. A row whose protocol is off on Protocols → Services is skipped in silence, the commonest reason for nothing on 143. HM4316: something else holds the port (IIS on 25 or 443, another mail server, a leftover instance); that port alone stays down. HM5113 with Error initializing SSL. Certificate not set.: that listener does not start.
The optional listeners (metrics, REST, web services, ManageSieve), each only with a non-zero port. MetricsServer: Invalid bind address means MetricsServerBindAddress is not an IP literal (localhost is refused too); that listener stays down and the server runs.

A failure in steps 2 to 6 stops the whole server, and one with a code writes it to the ERROR log; steps 7 and 8 cost one listener and go to the application log.

Windows. A service that stops at once with nothing logged is the account's rights on Bin, Data, Logs and Database. Since 6.3.4 a new installation runs as NT SERVICE\hMailServer (an upgrade keeps its account), and a folder outside the program folder needs approving: run hMailServer.exe --confirm /GrantServiceAccess elevated in Bin, and approve only a folder you recognise, since the service may then write in it. Local SQL Server with integrated authentication needs a login for NT SERVICE\hMailServer. What the account may not do, and moving between it and LocalSystem, are in Installing on Windows. The installer's recovery settings restart a failed service after 1, 2 and 5 minutes (sc.exe qfailure hMailServer).

Linux, under systemd. A start that fails before any listener opens puts hMailServer could not start: in the journal, with the reason on standard error in case the log directory is the fault. The unit runs as hmailserver, and ProtectSystem=strict makes anything outside its ReadWritePaths (/var/lib/hmailserver and /var/log/hmailserver) read-only to the service: a data, backup, archive or ACME directory elsewhere needs a drop-in (sudo systemctl edit hmailserver). HM5094: the MySQL or MariaDB client library is not installed. HM5603 on every TLS listener: /usr/bin/dh2048.pem is missing; reinstall the package.

The Diagnostics self-test

Monitoring & troubleshooting → Diagnostics in the Control Panel; since 6.3.4 also the Deck's Diagnostics view and POST /api/v1/diagnostics/run, on either platform. Ten tests:

Test	Fails when
Server details	Never; since 6.3.4 it also says whether the database connection is encrypted
Installation paths (6.3.0)	A configured path does not exist
Test IPv6	Never
Test outbound port	Outbound 25 is blocked, or the smart host's name, port or TLS mode is wrong
Test backup directory	No destination, or the service cannot write there
Test MX records	Public DNS is wrong, or the resolver is not answering
Test local connect	Hairpin NAT fails, 25 is not forwarded, or the MX points elsewhere
Test message file locations	Files outside the data directory, or full paths stored, which stops the built-in backup
Test IP range configuration	`127.0.0.1` has been auto-banned
Error logs	Any `ERROR*` file exists

Fill both domain boxes first: the MX and local-connect tests, and the outbound test without a smart host, are skipped rather than failed when they are empty. Copy puts the result on the clipboard for a post.

"Did we ever get it?": the message trace

The message trace records what the server did with each message, keyed on its queue id. It is off by default (MessageTraceEnabled), because it records who corresponds with whom, and nothing is recorded while it is off: turn it on before you need it.

Turning it on. Since 6.3.4 it applies when saved: the Control Panel's Message trace card on Monitoring & troubleshooting → Logging, PUT /api/v1/settings/ini/{name}, or hmctl, both with the administrator password:

hmctl ini set MessageTraceEnabled 1

On 6.3.3 and earlier it is MessageTraceEnabled=1 under [Settings] and a restart. Rows are kept MessageTraceRetentionDays (30): since 6.3.4 an hourly task deletes older ones; on 6.3.3 and earlier only Remove expired on the Message trace page, or MessageTrace.DeleteExpired over COM, did.

Reading it. The Control Panel's Message trace page searches an address as sender or recipient, newest first, and Follow this message shows one queue id. Over REST since 6.3.4: GET /api/v1/message-trace, /api/v1/message-trace/address/{address} and /api/v1/message-trace/queue/{id}, which need the view_message_trace permission.

hmctl api GET /api/v1/message-trace/address/[email protected]

What a row says. One row per recipient per delivery attempt: when, the event, sender, recipient, source address and status code. Events include delivered and failed; on 6.3.4 also deferred (an attempt that will be retried) and suppressed (stopped by the suppression list). A refusal at RCPT carries queue id 0. A forward, a list or a rule that re-sends makes a new queue entry.

Answering the question. A delivered row: you had it; look at the account's folders and rules. A row with queue id 0: refused at RCPT, and the status says why. No row: either it never reached you, or it is still queued or held, came while the trace was off, or is past the retention. The trace records delivery attempts only, with no acceptance, spam or virus event, so an empty trace does not prove the server never had the message. The per-domain reports' traffic, failures, senders and recipients are counted from it, and say so while it is off: Alerts, the audit trail and per-domain reports.

Things that look like faults and are not

What you see	Why
A new sender's first message is minutes late	Greylisting doing its job
(6.3.4) A value typed into `[Settings]` is back after a restart, HM5804 in the ERROR log	The database is the settings store: set it in the Control Panel, over REST (`PUT /api/v1/settings/ini/{name}`) or with `hmctl ini set`
(6.3.4) HM5807 at every start	A `[SettingsOverride]` section is forcing keys; remove it once the stored value is right
(6.3.4, Linux package) HM5803 at every start	The service cannot write its copy into the root-owned file; the stored value is in effect
6.2.28 to 6.3.3: "The update feed answered HTTP 404."	Set `UpdateFeedUrl=https://updates.progressiverobot.com/releases.json` and restart
"Release 6.3.4 carries no Sigstore bundle for its installer, so it cannot be verified."	Install 6.3.4 by hand: Upgrading to 6.3.4

Where a message waits, since 6.3.4. Neither delivered nor queued may mean held on purpose: by the sandbox, by compromised-account detection, in a back-off window, under an IP warm-up ceiling or behind a drain; Diagnosing stalled mail says where to look. And 550 5.7.1 Too many delivery attempts before the greylisting delay had passed is greylisting's abuse lock, lifted after GreyListingAbuseLockMinutes (60); a 5.7.10 bounce to a partner domain is gateway encryption refusing to send in the clear, the partner's key having expired.

What it does not do

It does not diagnose a mail client (Connecting a mail client), read logs line by line (Reading the logs), or work a stalled queue. The Control Panel and COM are Windows-only. When nothing here names the fault, Read this before you post says what a request needs. Without a GitLab account, email the Service Desk, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue.

Where to read more

Wiki: Troubleshooting, Ports Reference, Monitoring and Health, Installing hMailServer, Installing on Linux, The Control Panel, The REST API, Upgrading hMailServer, Architecture.
Documents: ServiceAccount.md, DatabaseTls.md, Reports.md, SettingsApply.md.
On the forum: Settings: when a save takes effect (HM5804 and the reinitialise), IP ranges, auto-ban and lockout, Choosing a database and encrypting the link (TlsMode, HM7530 and HM7531) and The hmailserver command line on Linux (the unit and its drop-ins).
Public documentation: Troubleshooting and Troubleshooting a Linux installation.