---
title: "Diagnosing stalled mail: a sender that times out, and a message that never leaves"
url: https://www.hmailserver.co.uk/topic/28219/diagnosing-stalled-mail-a-sender-that-times-out-and-a-message-that-never-leaves
author: "Progressiverobot"
published: 2026-09-27T19:01:13.731Z
updated: 2026-09-27T19:01:13.731Z
category: "General support"
tags: ["diagnostics","logs","smtp","spam"]
site: hMailServer Community Forum
---

# Diagnosing stalled mail: a sender that times out, and a message that never leaves

*Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) adds reasons a message stays where it is. A forward, a redirect or a rule's forward of mail from outside whose From names the forwarding domain's own organisation, and that did not pass DMARC for it when it arrived, keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. `RuleLoopLimit` 0 now stops out-of-office replies, as it stopped every other generated message. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries, where a failed lookup was read as a name with no addresses. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, and an error log that was quiet may now name database failures that were hidden (`HM5032`). See [the 6.3.7 announcement](/topic/28277).*

This topic covers the case where mail is not moving and the server looks healthy: no crash, no error, the service running, and a log that seems to stop mid-transaction. The server used to be silent about it; it is no longer, and the answer is usually in one line of the log. The one thing to establish first is **which half is stuck**: the server *accepting* a message from a sender, or *delivering* one it has already accepted. The two fail differently, for different causes, under different settings.

### Which half is stuck?

**Accepting.** The sending server connects, sends the message, and then waits. From its side there is a timeout *after* the message body was transmitted; Postfix reports it as `timed out while sending end of data`. The message never appears in your queue.

**Delivering.** The message was accepted (the sender got a `250`), is visible in the delivery queue, and never leaves.

**Monitoring & troubleshooting → Delivery queue** in the Control Panel tells the two apart in one look, and so does `hmailserver_delivery_queue_messages` on the metrics endpoint once it has a port (see Commands). The Control Panel is Windows-only; on Linux the queue is `GET /api/v1/queue`, `hmctl queue list` (since 6.3.4) and the Control Deck's queue page.

### Turn on debug logging first

Control Panel → **Logging** → tick **Debug messages**, and make sure logging is enabled; then reproduce the problem once. The stage lines below are written at debug level. The *slow* ones (ten seconds or more, for a whole stage or for any single spam test) are also written at application level, so a server with only application logging still records the important half. Turn debug logging off again afterwards on a busy server. [Reading the logs](/topic/28220) covers where the logging keys live on each platform.

### Accepting: the sender times out after sending the message

Between the `354` and the `250` the server runs the accept pipeline: spam tests, message modifications, archiving, the `OnAcceptMessage` script and the database save. All of it runs on a bounded pool of threads, and the reply is sent only when it finishes. The log times each stage:

```
SMTPConnection - accept: start spam-protection.
Spam test: SpamTestDNSBlackLists, Score: 0, Time: 12 ms
Spam test: SpamTestSpamAssassin, Score: 0, Time: 120000 ms
SMTPConnection - accept: done spam-protection in 120047 ms (session 5).
SMTPConnection - accept: done message-modifications in 0 ms (session 5).
SMTPConnection - accept: start script/save.
SMTPConnection - accept: done script/save in 3 ms (session 5).
```

**How to read it.** Read it as a sequence and look for the last line written; the stage that is stuck is the one after it. Two stages announce themselves with a `start` line (`spam-protection` and `script/save`), so for those a `start` with no matching `done` is the whole diagnosis. The middle stage, message modifications, has no `start` line, so a stall there shows as `done spam-protection` followed by silence. If every stage completes but one took a long time, the `Time:` or `done ... in` value names the culprit.

| What the log shows | Cause |
|---|---|
| `SpamTestSpamAssassin` with a large `Time:` | spamd is unreachable, overloaded, or accepting connections without answering |
| `SpamTestDNSBlackLists`, `SpamTestSURBL` or `SpamTestSPF` slow | The resolver is not answering. Look in the TCP/IP log for `DNS - Query timed out` and `DNS - Query failure`. Do not reach for `DNSServer` as the fix (below) |
| `done spam-protection`, then silence | Message modifications: the spam headers, the signature, the List-* headers, or the write of the modified message back to disk |
| `script/save` slow | An `OnAcceptMessage` event script, or the database |
| Nothing between `354` and silence | A version older than 6.2.17, which added these lines. Upgrade |

**The `DNSServer` setting.** It was broken in 6.2.17 and 6.2.18 (CHANGELOG's 6.2.19 section names 6.2.16 as well) and fixed in 6.2.19: on the affected versions every lookup through a configured custom server failed, and DNSBL, SURBL, SPF, DKIM and MX lookups failed in silence, so such a server had quietly stopped most of its spam filtering while still accepting mail. One caveat survives the fix: a custom server list is only honoured together with `DNS_QUERY_BYPASS_CACHE`, so every lookup pays full price. Leave `DNSServer` empty unless you need a specific resolver.

**SPF, and SPF on Linux.** In 6.3.3 and earlier the SPF evaluator resolved names through the Windows DNS client rather than through the server's resolver. On Windows an SPF check therefore went to the system resolvers whatever `DNSServer` said; on Linux every SPF check returned a temporary error, reported once per process as `HM6406`, and because `SpamTestDMARC` evaluates SPF, DMARC there lost its SPF half. The RFC 7208 evaluator that replaced it in 6.3.4 resolves through the same resolver as every other lookup. If you run 6.3.3 or earlier on Linux, an SPF or DMARC verdict from it is not a verdict.

**The deadline.** Since 6.2.17 acceptance is bounded: if the pipeline exceeds `FinalizationTimeout` (240 seconds by default) the server answers `451 4.3.1 Server temporarily overloaded while accepting the message; please retry.` and the sender retries rather than waiting for a reply that never comes. The accompanying error entry names the deadline that was hit, and the stage timings above it name what consumed the time.

### If every message stalls, not just one

Look for these two lines:

```
Task SMTP-accept session=42 ip=203.0.113.9 waited 8 seconds for a thread in
work queue Asynchronous task queue. 14 task(s) are still queued.
```

```
All 15 threads in work queue Asynchronous task queue have been busy for at
least 120 seconds, so no further task on this queue can start. 14 task(s) are
queued. Running: SMTP-accept session=42 ip=203.0.113.9 (thread 4120, 168s), ...
```

Every worker is occupied, and one slow dependency does this to the whole server. The task names say which sessions are stuck; the stage timings say what they are stuck on. `AsyncQueueStallThreshold` (120 seconds by default) is how long every worker may be busy before the second report is written; `0` turns the report off.

### Delivering: how the queue works

There is no in-memory queue to inspect. The delivery queue *is* the set of rows in `hm_messages` whose type is delivering, and the delivery manager is a loop that polls for them. Three properties of that loop explain most of what a stuck queue looks like:

- **The selection is `where messagetype = 1 and messagelocked = 0 and messagenexttrytime <= now`, ordered by size, then try count, then id.** Small and new goes first.
- **The delivery pool is separate and smaller than the accept pool**: `maxdelivertythreads`, 10 by default, on the work queue named `SMTP delivery queue`. It is not the `Asynchronous task queue` that finalises inbound mail, so a saturated delivery pool does not stop mail being accepted; it stops it leaving.
- **The manager wakes on demand, or once a minute otherwise.** A message whose next-try time has just passed is picked up within a minute, not instantly.

At startup every message in the queue is unlocked (`update hm_messages set messagelocked = 0`), because locks held by a server that crashed would otherwise strand those messages for ever. That is also why exactly one hMailServer service may run against one database.

### The retry schedule, and when a message gives up

| Setting | Where | Default | Effect |
|---|---|---|---|
| Number of retries | **Mail flow & delivery → Delivery of e-mail** (`smtpnoofretries`) | `4` | How many *further* attempts after the first |
| Minutes between every retry | The same page (`smtpminutesbetweenretries`) | `60` | The gap between them |
| `QuickRetries` | `[Settings]` | `0` (off) | The first N retries use the short gap below instead, for greylisting |
| `QuickRetriesMinutes` | `[Settings]` | `6` | That short gap |
| `QueueRandomnessMinutes` | `[Settings]` | `0` (off) | Adds a random 1..N minutes to every gap, so a burst does not retry in lockstep |
| Route *number of tries* and *minutes between tries* | **Mail flow & delivery → Routes** | — | Overrides both, per matching route; where several routes match, the largest of each wins |

With the shipped defaults that is five attempts an hour apart: a destination that is refusing temporarily holds a message for roughly four hours and then bounces it. The try count in the queue is the same `messagecurnooftries` the arithmetic uses. Two things do not advance that counter: *Deliver now* resets the next-try time without touching the count, and a message held because the virus scanner could not examine it is held on its own budget (below).

### Reading a stuck queue

| What the queue looks like | What it means | Where to look next |
|---|---|---|
| Depth climbing, oldest message seconds old | A busy server draining normally. Not a fault | — |
| Depth flat, oldest message hours old, try counts climbing | The far end is deferring you | The SMTP log for that destination; your IP's reputation and PTR |
| Every stuck message shares one recipient domain | That destination, not your server | The route settings for it, and its MX |
| Try counts stuck at `0`, next-try times in the past, nothing moving | Delivery threads are not running the tasks | `hmailserver_workqueue_depth{queue="SMTP delivery queue"}`, and the saturation report above |
| Messages appear and vanish without arriving | Delivered to the wrong place, or discarded by a rule | **Mail flow & delivery → Rules**; enable `MessageTraceEnabled` and read the trace |
| Try count `1` and a next-try time of `1901-01-01` | An ETRN hold: the one route these recipients match is an ETRN route | **Mail flow & delivery → Routes**; the log line is `SMTPDeliverer - Route Message: HOLD for later delivery..` |
| Queue empty, but the sender got a `250` and nobody received it | Delivered locally into a folder you are not looking at, or the row was removed | The account's folders; the message trace |
| `Message N could not be delivered since the data file does not exist` | The row and the store disagree: a restore that missed files, or a store on a share that vanished | Turn on `MessageStoreConsistencyCheck` and read the report |

**When the virus scanner cannot examine a message.** With `AVFailAction=1` such a message is not delivered. It is held on its own schedule rather than on the SMTP retry budget: `AVFailRetryMinutes` apart (15 by default), at most `AVFailMaxHolds` times (16), and then returned to the sender. The log says where in that budget each hold is (`... this was hold 3 of 16 before it is returned to the sender.`). Each hold counts as a deferral in `hmailserver_messages_deferred_total`, so a scanner that has stopped answering shows as a deferral rate with no matching delivery rate.

### The usual causes of a genuinely stuck delivery

- **A virus scanner that stops responding.** ClamAV is contacted after the message is accepted, so a wedged clamd shows up as accepted but never delivered. Each socket operation is bounded by `ClamMinTimeout` / `ClamMaxTimeout`, but that is a deadline on *one* read or write, armed fresh each time, and the message is streamed in chunks, so one large message can take considerably longer than `ClamMaxTimeout` in total.
- **A remote server that answers extremely slowly.** The idle timeout is re-armed on every byte received, so a host that sends one byte occasionally used to hold a delivery thread indefinitely. Since 6.2.18 there is an absolute ceiling, `ClientSessionCeiling` (30 minutes by default), armed once and never re-armed.
- **A custom virus scanner or external tool that hangs**, bounded by `ExternalProcessTimeout`.
- **A message held for a sandbox's verdict** (since 6.3.4). With `SandboxEnabled` on, a message none of your own signed-in users submitted, whose attachment is of a submitted type, waits for the sandbox you named to open the file, up to `SandboxTimeoutMinutes` (15 by default). While it waits it is in no mailbox, not in the quarantine and not the queue's to retry: its Retry and Delete answer 409. It is listed by `GET /api/v1/sandbox/held`, by `GET /api/v1/queue` marked `held: sandbox` (the `held` column of `hmctl queue list`), and in the Deck's queue page under *Held for a sandbox*, where it can be delivered without the verdict or discarded.
- **The database.** Check for errors mentioning the connection pool.

Since 6.3.4 a message can also wait in other places, each on purpose: held for review by compromised-account detection in `enforce` (`/api/v1/review`); in a destination's back-off window, with the last error `Delivery deferred by this server's own shaping, not by a remote server.` (`GET /api/v1/queue/shaping`; such a deferral does not spend the message's retries); under an IP warm-up ceiling; or behind a drain, when `/readyz` says `draining`. Section 19.11a of the Troubleshooting page lists each with what to do.

### Settings that bound each stage

All are in seconds. On 6.3.3 and earlier they are `[Settings]` keys in `hMailServer.ini`. Since 6.3.4 they are stored in the database and changed in the Control Panel, over the REST API at `/api/v1/settings/ini/<name>` or with `hmctl`, and the save applies them. They still appear in the file's `[Settings]` section, but the database's value is the one used: where the service can write the file, an edit made there is reported (HM5804) and put back at the next start.

| Setting | Default | Bounds | `0` means |
|---|---|---|---|
| `FinalizationTimeout` | 240 | The whole accept pipeline, after which the sender gets a `451` | no bound |
| `SAMaxTimeout` | 90 | SpamAssassin: idle timeout, and a session ceiling of this plus 30s | **not** "no bound" |
| `ClamMaxTimeout` | 90 | ClamAV: idle timeout per socket operation | **not** "no bound" |
| `DNSQueryTimeout` | 10 | A single DNS query | no bound |
| `ScriptTimeout` | 60 | One event script invocation | no bound |
| `ExternalProcessTimeout` | 300 | An external scanner process | no bound |
| `ClientSessionCeiling` | 1800 | An entire outbound delivery session | no bound |
| `AsyncQueueStallThreshold` | 120 | How long every worker may be busy before the saturation report | reporting off |
| `DBConnectionAcquireTimeout` | 60 | Waiting for a pooled database connection | no bound |

**The two exceptions matter, because `0` tightens them instead of removing them.** `SAMaxTimeout` and `ClamMaxTimeout` are not passed straight through: they go to a calculator that returns the *minimum* whenever the maximum is lower than it. So `SAMaxTimeout=0` gives a 30-second idle timeout (`SAMinTimeout`) and a 30-second session ceiling, and `ClamMaxTimeout=0` gives 15 seconds (`ClamMinTimeout`). To lengthen either, raise the value; to lengthen it a long way, raise the matching `...MinTimeout` too. There is no way to make either unbounded.

When the pool deadline expires, a recipient lookup answers `451 4.3.2` rather than `550`: the server can tell "the database did not answer" from "no such address", so a database locked by a backup defers the mail instead of bouncing it.

### Commands

The COM examples below are Windows-only. The metrics endpoint and the REST routes work on both platforms.

**Is it accepting, or is it delivering?** The metrics listener is off until `MetricsServerPort` is set (9100 here; since 6.3.4 the port, the bind address and the listener's other settings, its credentials included, wait for a reinitialise). Once it is on, the endpoint needs no credential on a loopback bind:

```powershell
(Invoke-RestMethod http://127.0.0.1:9100/metrics) -split "`n" |
  Select-String '^hmailserver_delivery_queue_(messages|oldest_message_age_seconds) |^hmailserver_workqueue_depth|^hmailserver_messages_(delivered|deferred|bounced)_total '
```

A rising `oldest_message_age_seconds` with a flat `delivered_total` is the delivering half. A rising `hmailserver_workqueue_depth{queue="Asynchronous task queue"}` is the accepting half.

The COM examples start from a signed-in `$app`:

```powershell
$app = New-Object -ComObject hMailServer.Application
$app.Authenticate("Administrator", "<password>") | Out-Null
```

**Push one message to the front of the queue** (resets its next-try time, leaves the try count alone, and releases an ETRN hold):

```powershell
$q = $app.GlobalObjects.DeliveryQueue
$q.ResetDeliveryTime(4711)
$q.StartDelivery()
```

**Wake the delivery manager without touching any message:**

```powershell
$app.GlobalObjects.DeliveryQueue.StartDelivery()
```

**Remove one message from the queue** (it is locked first, so a message a thread already has is not pulled from under it):

```powershell
$app.GlobalObjects.DeliveryQueue.Remove(4711)
```

**Follow the log while you reproduce it** by tailing the day's file and filtering on `accept: |Spam test:|waited .* for a thread|have been busy|SMTPDeliverer`; [Reading the logs](/topic/28220) has the tail command for each platform.

**Turn the message trace on** (`MessageTraceEnabled`, off by default) to see what was done with one message. It has no acceptance, spam or virus event, so an empty trace does not show that the server never had the message; [Troubleshooting by symptom](/topic/28225) says how to turn it on and what a row says.

### What to include in a report

If none of the above identifies it, post in [General support](/category/8) or open an issue with: the log from the `354` (or from the delivery attempt) onwards, including the stage timing lines; the `ERROR_hmailserver_<date>.log` for the same window; which scanners and event scripts are enabled; and whether the message eventually arrives, arrives twice, or never arrives. Say which half you have. [Read this before you post](/topic/28185) covers the rest. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email.

### Where to read more

- Wiki: [Diagnosing Stalled Mail](https://gitlab.com/hmailserver/hmailserver/-/wikis/Diagnosing-Stalled-Mail) and [Troubleshooting](https://gitlab.com/hmailserver/hmailserver/-/wikis/Troubleshooting), sections 19.7 and 19.11a.
- Repository: [DiagnosingStalledMail.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/DiagnosingStalledMail.md); [SettingsApply.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/SettingsApply.md); [Sandbox.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Sandbox.md); [SUPPORT.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/SUPPORT.md).
- The public documentation: [The delivery queue and the retry schedule](https://www.progressiverobot.com/hmailserver-documentation/#267-the-delivery-queue-and-the-retry-schedule) and [Troubleshooting](https://www.progressiverobot.com/hmailserver-documentation/#19-troubleshooting).
- On this forum: [Being trusted by the servers you send to](/topic/28188), for when the far end is deferring you; [Outbound shaping and IP warm-up](/topic/28259), for when your own server is holding mail back; [Monitoring and health](/topic/28272), for turning the metrics listener on.