Diagnosing stalled mail: a sender that times out, and a message that never leaves

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) adds reasons a message stays where it is. A forward, a redirect or a rule's forward of mail from outside whose From names the forwarding domain's own organisation, and that did not pass DMARC for it when it arrived, keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. RuleLoopLimit 0 now stops out-of-office replies, as it stopped every other generated message. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries, where a failed lookup was read as a name with no addresses. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, and an error log that was quiet may now name database failures that were hidden (HM5032). See the 6.3.7 announcement.

This topic covers the case where mail is not moving and the server looks healthy: no crash, no error, the service running, and a log that seems to stop mid-transaction. The server used to be silent about it; it is no longer, and the answer is usually in one line of the log. The one thing to establish first is which half is stuck: the server accepting a message from a sender, or delivering one it has already accepted. The two fail differently, for different causes, under different settings.

Which half is stuck?

Accepting. The sending server connects, sends the message, and then waits. From its side there is a timeout after the message body was transmitted; Postfix reports it as timed out while sending end of data. The message never appears in your queue.

Delivering. The message was accepted (the sender got a 250), is visible in the delivery queue, and never leaves.

Monitoring & troubleshooting → Delivery queue in the Control Panel tells the two apart in one look, and so does hmailserver_delivery_queue_messages on the metrics endpoint once it has a port (see Commands). The Control Panel is Windows-only; on Linux the queue is GET /api/v1/queue, hmctl queue list (since 6.3.4) and the Control Deck's queue page.

Turn on debug logging first

Control Panel → Logging → tick Debug messages, and make sure logging is enabled; then reproduce the problem once. The stage lines below are written at debug level. The slow ones (ten seconds or more, for a whole stage or for any single spam test) are also written at application level, so a server with only application logging still records the important half. Turn debug logging off again afterwards on a busy server. Reading the logs covers where the logging keys live on each platform.

Accepting: the sender times out after sending the message

Between the 354 and the 250 the server runs the accept pipeline: spam tests, message modifications, archiving, the OnAcceptMessage script and the database save. All of it runs on a bounded pool of threads, and the reply is sent only when it finishes. The log times each stage:

SMTPConnection - accept: start spam-protection.
Spam test: SpamTestDNSBlackLists, Score: 0, Time: 12 ms
Spam test: SpamTestSpamAssassin, Score: 0, Time: 120000 ms
SMTPConnection - accept: done spam-protection in 120047 ms (session 5).
SMTPConnection - accept: done message-modifications in 0 ms (session 5).
SMTPConnection - accept: start script/save.
SMTPConnection - accept: done script/save in 3 ms (session 5).

How to read it. Read it as a sequence and look for the last line written; the stage that is stuck is the one after it. Two stages announce themselves with a start line (spam-protection and script/save), so for those a start with no matching done is the whole diagnosis. The middle stage, message modifications, has no start line, so a stall there shows as done spam-protection followed by silence. If every stage completes but one took a long time, the Time: or done ... in value names the culprit.

What the log shows	Cause
SpamTestSpamAssassin with a large Time:	spamd is unreachable, overloaded, or accepting connections without answering
SpamTestDNSBlackLists, SpamTestSURBL or SpamTestSPF slow	The resolver is not answering. Look in the TCP/IP log for DNS - Query timed out and DNS - Query failure. Do not reach for DNSServer as the fix (below)
done spam-protection, then silence	Message modifications: the spam headers, the signature, the List-* headers, or the write of the modified message back to disk
script/save slow	An OnAcceptMessage event script, or the database
Nothing between 354 and silence	A version older than 6.2.17, which added these lines. Upgrade

The DNSServer setting. It was broken in 6.2.17 and 6.2.18 (CHANGELOG's 6.2.19 section names 6.2.16 as well) and fixed in 6.2.19: on the affected versions every lookup through a configured custom server failed, and DNSBL, SURBL, SPF, DKIM and MX lookups failed in silence, so such a server had quietly stopped most of its spam filtering while still accepting mail. One caveat survives the fix: a custom server list is only honoured together with DNS_QUERY_BYPASS_CACHE, so every lookup pays full price. Leave DNSServer empty unless you need a specific resolver.

SPF, and SPF on Linux. In 6.3.3 and earlier the SPF evaluator resolved names through the Windows DNS client rather than through the server's resolver. On Windows an SPF check therefore went to the system resolvers whatever DNSServer said; on Linux every SPF check returned a temporary error, reported once per process as HM6406, and because SpamTestDMARC evaluates SPF, DMARC there lost its SPF half. The RFC 7208 evaluator that replaced it in 6.3.4 resolves through the same resolver as every other lookup. If you run 6.3.3 or earlier on Linux, an SPF or DMARC verdict from it is not a verdict.

The deadline. Since 6.2.17 acceptance is bounded: if the pipeline exceeds FinalizationTimeout (240 seconds by default) the server answers 451 4.3.1 Server temporarily overloaded while accepting the message; please retry. and the sender retries rather than waiting for a reply that never comes. The accompanying error entry names the deadline that was hit, and the stage timings above it name what consumed the time.

If every message stalls, not just one

Look for these two lines:

Task SMTP-accept session=42 ip=203.0.113.9 waited 8 seconds for a thread in
work queue Asynchronous task queue. 14 task(s) are still queued.

All 15 threads in work queue Asynchronous task queue have been busy for at
least 120 seconds, so no further task on this queue can start. 14 task(s) are
queued. Running: SMTP-accept session=42 ip=203.0.113.9 (thread 4120, 168s), ...

Every worker is occupied, and one slow dependency does this to the whole server. The task names say which sessions are stuck; the stage timings say what they are stuck on. AsyncQueueStallThreshold (120 seconds by default) is how long every worker may be busy before the second report is written; 0 turns the report off.

Delivering: how the queue works

There is no in-memory queue to inspect. The delivery queue is the set of rows in hm_messages whose type is delivering, and the delivery manager is a loop that polls for them. Three properties of that loop explain most of what a stuck queue looks like:

• The selection is where messagetype = 1 and messagelocked = 0 and messagenexttrytime <= now, ordered by size, then try count, then id. Small and new goes first.
• The delivery pool is separate and smaller than the accept pool: maxdelivertythreads, 10 by default, on the work queue named SMTP delivery queue. It is not the Asynchronous task queue that finalises inbound mail, so a saturated delivery pool does not stop mail being accepted; it stops it leaving.
• The manager wakes on demand, or once a minute otherwise. A message whose next-try time has just passed is picked up within a minute, not instantly.

At startup every message in the queue is unlocked (update hm_messages set messagelocked = 0), because locks held by a server that crashed would otherwise strand those messages for ever. That is also why exactly one hMailServer service may run against one database.

The retry schedule, and when a message gives up

Setting	Where	Default	Effect
Number of retries	Mail flow & delivery → Delivery of e-mail (smtpnoofretries)	4	How many further attempts after the first
Minutes between every retry	The same page (smtpminutesbetweenretries)	60	The gap between them
QuickRetries	[Settings]	0 (off)	The first N retries use the short gap below instead, for greylisting
QuickRetriesMinutes	[Settings]	6	That short gap
QueueRandomnessMinutes	[Settings]	0 (off)	Adds a random 1..N minutes to every gap, so a burst does not retry in lockstep
Route number of tries and minutes between tries	Mail flow & delivery → Routes	—	Overrides both, per matching route; where several routes match, the largest of each wins

With the shipped defaults that is five attempts an hour apart: a destination that is refusing temporarily holds a message for roughly four hours and then bounces it. The try count in the queue is the same messagecurnooftries the arithmetic uses. Two things do not advance that counter: Deliver now resets the next-try time without touching the count, and a message held because the virus scanner could not examine it is held on its own budget (below).

Reading a stuck queue

What the queue looks like	What it means	Where to look next
Depth climbing, oldest message seconds old	A busy server draining normally. Not a fault	—
Depth flat, oldest message hours old, try counts climbing	The far end is deferring you	The SMTP log for that destination; your IP's reputation and PTR
Every stuck message shares one recipient domain	That destination, not your server	The route settings for it, and its MX
Try counts stuck at 0, next-try times in the past, nothing moving	Delivery threads are not running the tasks	hmailserver_workqueue_depth{queue="SMTP delivery queue"}, and the saturation report above
Messages appear and vanish without arriving	Delivered to the wrong place, or discarded by a rule	Mail flow & delivery → Rules; enable MessageTraceEnabled and read the trace
Try count 1 and a next-try time of 1901-01-01	An ETRN hold: the one route these recipients match is an ETRN route	Mail flow & delivery → Routes; the log line is SMTPDeliverer - Route Message: HOLD for later delivery..
Queue empty, but the sender got a 250 and nobody received it	Delivered locally into a folder you are not looking at, or the row was removed	The account's folders; the message trace
Message N could not be delivered since the data file does not exist	The row and the store disagree: a restore that missed files, or a store on a share that vanished	Turn on MessageStoreConsistencyCheck and read the report

When the virus scanner cannot examine a message. With AVFailAction=1 such a message is not delivered. It is held on its own schedule rather than on the SMTP retry budget: AVFailRetryMinutes apart (15 by default), at most AVFailMaxHolds times (16), and then returned to the sender. The log says where in that budget each hold is (... this was hold 3 of 16 before it is returned to the sender.). Each hold counts as a deferral in hmailserver_messages_deferred_total, so a scanner that has stopped answering shows as a deferral rate with no matching delivery rate.

The usual causes of a genuinely stuck delivery

• A virus scanner that stops responding. ClamAV is contacted after the message is accepted, so a wedged clamd shows up as accepted but never delivered. Each socket operation is bounded by ClamMinTimeout / ClamMaxTimeout, but that is a deadline on one read or write, armed fresh each time, and the message is streamed in chunks, so one large message can take considerably longer than ClamMaxTimeout in total.
• A remote server that answers extremely slowly. The idle timeout is re-armed on every byte received, so a host that sends one byte occasionally used to hold a delivery thread indefinitely. Since 6.2.18 there is an absolute ceiling, ClientSessionCeiling (30 minutes by default), armed once and never re-armed.
• A custom virus scanner or external tool that hangs, bounded by ExternalProcessTimeout.
• A message held for a sandbox's verdict (since 6.3.4). With SandboxEnabled on, a message none of your own signed-in users submitted, whose attachment is of a submitted type, waits for the sandbox you named to open the file, up to SandboxTimeoutMinutes (15 by default). While it waits it is in no mailbox, not in the quarantine and not the queue's to retry: its Retry and Delete answer 409. It is listed by GET /api/v1/sandbox/held, by GET /api/v1/queue marked held: sandbox (the held column of hmctl queue list), and in the Deck's queue page under Held for a sandbox, where it can be delivered without the verdict or discarded.
• The database. Check for errors mentioning the connection pool.

Since 6.3.4 a message can also wait in other places, each on purpose: held for review by compromised-account detection in enforce (/api/v1/review); in a destination's back-off window, with the last error Delivery deferred by this server's own shaping, not by a remote server. (GET /api/v1/queue/shaping; such a deferral does not spend the message's retries); under an IP warm-up ceiling; or behind a drain, when /readyz says draining. Section 19.11a of the Troubleshooting page lists each with what to do.

Settings that bound each stage

All are in seconds. On 6.3.3 and earlier they are [Settings] keys in hMailServer.ini. Since 6.3.4 they are stored in the database and changed in the Control Panel, over the REST API at /api/v1/settings/ini/<name> or with hmctl, and the save applies them. They still appear in the file's [Settings] section, but the database's value is the one used: where the service can write the file, an edit made there is reported (HM5804) and put back at the next start.

Setting	Default	Bounds	0 means
FinalizationTimeout	240	The whole accept pipeline, after which the sender gets a 451	no bound
SAMaxTimeout	90	SpamAssassin: idle timeout, and a session ceiling of this plus 30s	not "no bound"
ClamMaxTimeout	90	ClamAV: idle timeout per socket operation	not "no bound"
DNSQueryTimeout	10	A single DNS query	no bound
ScriptTimeout	60	One event script invocation	no bound
ExternalProcessTimeout	300	An external scanner process	no bound
ClientSessionCeiling	1800	An entire outbound delivery session	no bound
AsyncQueueStallThreshold	120	How long every worker may be busy before the saturation report	reporting off
DBConnectionAcquireTimeout	60	Waiting for a pooled database connection	no bound

The two exceptions matter, because 0 tightens them instead of removing them. SAMaxTimeout and ClamMaxTimeout are not passed straight through: they go to a calculator that returns the minimum whenever the maximum is lower than it. So SAMaxTimeout=0 gives a 30-second idle timeout (SAMinTimeout) and a 30-second session ceiling, and ClamMaxTimeout=0 gives 15 seconds (ClamMinTimeout). To lengthen either, raise the value; to lengthen it a long way, raise the matching ...MinTimeout too. There is no way to make either unbounded.

When the pool deadline expires, a recipient lookup answers 451 4.3.2 rather than 550: the server can tell "the database did not answer" from "no such address", so a database locked by a backup defers the mail instead of bouncing it.

Commands

The COM examples below are Windows-only. The metrics endpoint and the REST routes work on both platforms.

Is it accepting, or is it delivering? The metrics listener is off until MetricsServerPort is set (9100 here; since 6.3.4 the port, the bind address and the listener's other settings, its credentials included, wait for a reinitialise). Once it is on, the endpoint needs no credential on a loopback bind:

(Invoke-RestMethod http://127.0.0.1:9100/metrics) -split "`n" |
  Select-String '^hmailserver_delivery_queue_(messages|oldest_message_age_seconds) |^hmailserver_workqueue_depth|^hmailserver_messages_(delivered|deferred|bounced)_total '

A rising oldest_message_age_seconds with a flat delivered_total is the delivering half. A rising hmailserver_workqueue_depth{queue="Asynchronous task queue"} is the accepting half.

The COM examples start from a signed-in $app:

$app = New-Object -ComObject hMailServer.Application
$app.Authenticate("Administrator", "<password>") | Out-Null

Push one message to the front of the queue (resets its next-try time, leaves the try count alone, and releases an ETRN hold):

$q = $app.GlobalObjects.DeliveryQueue
$q.ResetDeliveryTime(4711)
$q.StartDelivery()

Wake the delivery manager without touching any message:

$app.GlobalObjects.DeliveryQueue.StartDelivery()

Remove one message from the queue (it is locked first, so a message a thread already has is not pulled from under it):

$app.GlobalObjects.DeliveryQueue.Remove(4711)

Follow the log while you reproduce it by tailing the day's file and filtering on accept: |Spam test:|waited .* for a thread|have been busy|SMTPDeliverer; Reading the logs has the tail command for each platform.

Turn the message trace on (MessageTraceEnabled, off by default) to see what was done with one message. It has no acceptance, spam or virus event, so an empty trace does not show that the server never had the message; Troubleshooting by symptom says how to turn it on and what a row says.

What to include in a report

If none of the above identifies it, post in General support or open an issue with: the log from the 354 (or from the delivery attempt) onwards, including the stage timing lines; the ERROR_hmailserver_<date>.log for the same window; which scanners and event scripts are enabled; and whether the message eventually arrives, arrives twice, or never arrives. Say which half you have. Read this before you post covers the rest. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email.

Where to read more

• Wiki: Diagnosing Stalled Mail and Troubleshooting, sections 19.7 and 19.11a.
• Repository: DiagnosingStalledMail.md; SettingsApply.md; Sandbox.md; SUPPORT.md.
• The public documentation: The delivery queue and the retry schedule and Troubleshooting.
• On this forum: Being trusted by the servers you send to, for when the far end is deferring you; Outbound shaping and IP warm-up, for when your own server is holding mail back; Monitoring and health, for turning the metrics listener on.