Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) moves the Control Panel's Audit trail page, with journaling and eDiscovery, into a new Compliance group, and the alerts' settings, the outbound webhooks and the SNMP agent onto its Monitoring & alerts page (API & monitoring until 6.3.6; 6.3.5 and 6.3.6 had them on Logging); Ctrl+K takes an old title or route to its page. It also fixes the SMTP log, whose lines the syslog sink copies, which wrote some credentials in the clear with SMTP logging on, in every version from 6.0.0 to 6.3.6 (work item #95). See the 6.3.7 announcement. This topic covers four things 6.3.4 adds for being told what the server is doing and who changed it: the alerts, the audit trail, the syslog sink (or the systemd journal on Linux) and the per-domain reports, with a paragraph on following one connection live. The one thing to know first: after the upgrade to 6.3.4 the audit trail and the alerts are on, with seven of the eleven conditions enabled, but nothing is mailed until AlertRecipient and AlertSenderAddress are set. What 6.3.4 switches on without a switch From the CHANGELOG's list of what changes on upgrade: the audit trail is on (schema 6043), and keeps every row until a retention is set. Compromised-account detection starts in watch mode (schema 6071): it raises account.compromised but holds nothing until compromise_detection_mode is set to enforce. The reputation watch, which raises reputation.listed, is on in a new installation and off after an upgrade. Alerts A condition is a row in hm_alertrules (on or off, threshold, mail or webhook or both, cool-down, at once or in the digest); an event is a row in hm_alertevents. Delivery is AlertTask, once a minute, and nothing else sends, so an alert about the delivery queue is never itself submitted into the stuck queue. Condition On as shipped Raised by Threshold backup.failed yes A backup that failed, and a scheduled backup that could not run. Sent at once — certificate.expiring yes The notAfter of every configured certificate, read once a minute 7 days disk.low no The free-space warning band, and again below the floor; cleared on recovery — queue.stalled no A work queue whose every thread has been in the same task past AsyncQueueStallThreshold — autoban.storm no Auto-bans in the last hour 25 minidump.written no A crash dump newer than the last one recorded, in the log directory — complaints.rate.high yes A hosted domain's complaint rate over the feedback loops' seven-day window; inert while complaint processing is off, and a domain that sent fewer than 100 messages is not judged 1,000 per million dkim.rotation.stalled yes A rotation whose next key's record has not been seen for the threshold, or a running rotation whose last move failed 7 days account.compromised yes Compromised-account detection at submission, one event per account per hour; in watch mode the detail says what enforcing would have done — account.password-reset yes A password reset completed through an account's confirmed recovery address; no cool-down and no digest — reputation.listed yes The reputation watch's stored rows; sent at once, and a check that was refused or timed out does not clear it — The four that are off are noisy on a healthy server. The settings. All nine are rows of hm_settings, none an hMailServer.ini key. They are in the server group of GET/PUT /api/v1/settings, on the Deck's Settings page, and on Windows over COM through Settings. The CHANGELOG and AlertsAndAuditTrail.md spell some in their hm_settings form (AlertRecipient, AlertSenderAddress and so on); the REST keys are these. REST key Default Meaning alerts_enabled on The master switch alert_recipient empty Where alerts and the digest go; a rule can name an address of its own instead alert_sender_address empty What they are sent from; a rule with no sender sends nothing alert_digest_enabled on Whether rules marked for the digest wait for it alert_digest_hour 7 The hour, UTC, the digest goes out; a digest with nothing in it is not sent alert_max_per_hour 20 Past this many single alerts in an hour, everything goes into the digest alert_webhook_max_attempts 5 Webhook attempts before an event is dead-lettered (1 to 20) audit_trail_enabled on Whether administrative changes are recorded audit_retention_days 0 Days of trail to keep; 0 keeps everything Nothing may flap. A state condition already raised is not raised again; an occurrence inside the rule's cool-down is recorded but held for the digest; and the digest is one message listing what fired. Webhooks. A rule whose actions is 2 (webhook) or 3 (mail and webhook) POSTs one JSON object per event (condition, state, severity, time, time_utc, summary, detail, server) with three headers: X-hMailServer-Timestamp, X-hMailServer-Signature: sha256=<hex> and X-hMailServer-Event. It is signed under the rule's shared secret, as every webhook the server sends is; the topic on integrating applications shows how to verify it. A rule that calls a webhook is refused without an address and without a secret; webhook_secret is write-only and the listing says only webhook_secret_set. Only https, or plain http to a loopback address, is used. An attempt that does not get a 2xx is retried on the next pass, then after 1, 5 and 15 minutes, which is the default five attempts, and 60 minutes apart after that if the maximum is raised. After the last attempt the event is dead-lettered: kept, no longer retried, and reported once in the error log as HM6543. Route Does Permission GET /api/v1/alerts/rules Every condition and its rule view_configuration PUT /api/v1/alerts/rules/{condition} Change one, or create one for a condition nothing ships; a field left out keeps its value manage_settings GET /api/v1/alerts/events What has fired, newest first, with the webhook's state view_configuration POST /api/v1/alerts/test Raise a condition through the path a real one takes: {"condition":"disk.low","summary":"...","occurrence":false} manage_settings POST /api/v1/alerts/run Run one pass now manage_server All five are refused to every API key. The administrator password reaches them, and so does an account whose role holds the permission and reaches every domain. Neither the Control Panel nor the Deck has a page for the rules, and hmctl and the PowerShell module have no alerts or audit verbs; hmctl api reaches the routes. The audit trail One row per administrative change made over COM, REST or the Deck, in hm_audit: when (audittime, UTC); who (auditactor: Administrator over COM, administrator or key:<id> over REST, and an account signed in with its own password is its address over COM and account:<address> over REST and the Deck); the interface and, except over COM, the peer address; the object's type and name; the action (created, updated or deleted, also unlocked, cleared, moved and move failed, the last for a mailbox move that failed and could not be put back); the detail (the columns the change carried, or for a setting Name: old -> new, ending [role: <name>] under a role); and the two hashes of the chain. Both kinds of setting are recorded: the hm_settings values and the [Settings] keys (in hm_inisettings since schema 6042), with the value they had. A key set for the first time reads SmtpTarpitCount: (default) -> 5; returned to its default it reads SmtpTarpitCount: 5 -> (default), with the action deleted. A write of the value a setting already has records nothing. What is never recorded as a value. A password, key, token, secret, hash, salt or passphrase is recorded as (changed), decided by the name of the column or setting, matched as a substring, so MetricsServerAuthToken and PasswordPepper are among them. Fifteen keys whose names only look like secrets, the seven PasswordPolicy* keys among them, are recorded with their values. The server's own writes (delivery, retention, greylisting, the index) are never recorded. Verifying the chain. Each row carries the SHA-256 of the row before it and of itself. GET /api/v1/audit/verify answers {"intact","rows_checked","first_broken_id","reason"}, and the reason tells a row that was edited from one that was removed. This makes tampering detectable, not impossible: whoever can write the table can recompute the whole chain. Rows hashed the older way, which only a build from before the 6.3.4 release can have written, did not cover a character above U+00FF; they still verify, with that weakness. Reading it. GET /api/v1/audit?actor=&object_type=&action=&since=&until=&limit=&offset= answers newest first; actor is a substring, so key: finds every change made with an API key, and limit is 1 to 1,000, default 100. It needs the administrator password, or an account whose role holds view_audit (the auditor role) and is not confined to named domains. Every API key is refused, with 401 rather than 403, so that it learns nothing from asking. Putting a change back. POST /api/v1/audit/{id}/revert writes a setting's old value back through the same setter, so the put-back is itself a row in the trail. It needs manage_settings and is refused with 409 for a row that is not a setting's change, a secret's row, a value the trail cut at 120 characters, and a setting this version no longer has. Where to see it. The Deck's Audit trail view has the filters, paging, Verify the chain and Put back. On Windows the Control Panel has the same page, over COM, at Monitoring & troubleshooting → Audit trail (Compliance → Audit trail since 6.3.7; Ctrl+K takes the old route to it). Switching recording off is itself recorded. The trail, the rules and the events are not in the built-in backup. What it holds for compliance. A download of a discovery export is recorded on the trail alone, so with the trail off it is recorded nowhere. An erasure is recorded with the address it erased wherever it deletes from recorded tables (aliases, distribution-list recipients and, with the archive included, the archive index), and audit_retention_days is how those rows age out. The compliance topic covers eDiscovery and the erasure. What it does not record. Reads; what a restore replaces (the request is one row, restore started); creating or revoking a REST API key; releasing or deleting a quarantined message; creating, renaming or deleting a public folder; the [Security], [Directories] and [Database] sections of hMailServer.ini; and a change nobody made through an interface. A change to the twelve syslog settings over COM, from the Control Panel's Logging page for one, is recorded in 6.3.4: the settings property setters carry an audit scope of their own. The wiki's syslog section, 16.2a, still describes the earlier state. Syslog and the journal (since 6.3.4) Every entry the server logs can also be sent as RFC 5424 to a syslog collector over UDP, TCP or TLS. It is a copy: the files under LogFolder are written as before. On Windows the settings are the syslog (RFC 5424) card on the Control Panel's Logging page; on Linux they are keys of PUT /api/v1/settings/logging and the Control Deck, and hmctl settings get logging prints them. They take effect when the sink starts, at the next service start or Reinitialize, not when they are saved. hm_settings name REST key Default Meaning SyslogEnabled syslog_enabled off The whole feature SyslogHost syslog_host empty Empty is not "off": on Linux under systemd it selects the journal; on Windows nothing is sent, which the server says once in the application log SyslogPort syslog_port 514 514 for UDP and plain TCP; 6514 is the registered port for TLS SyslogTransport syslog_transport 0 (UDP) 0 UDP, 1 TCP, 2 TCP over TLS; the REST key takes udp, tcp or tls SyslogFacility syslog_facility 2 (mail) 0 to 23; 16 to 23 are local0 to local7 SyslogMinimumSeverity syslog_minimum_severity 6 (informational) Entries less severe than this are not sent SyslogLogTypes syslog_log_smtp, syslog_log_pop3, syslog_log_imap, syslog_log_application, syslog_log_tcpip, syslog_log_debug 86 One bit mask: SMTP, POP3, application and IMAP on; TCP/IP and debug off Errors are sent whatever the mask says. A message looks like this: <22>1 2026-09-15T14:21:09.284+01:00 mail.example.com hMailServer 4812 SMTPD [hmailserver@32473 thread="7412" session="41" client="203.0.113.9"] <BOM>RECEIVED: MAIL FROM:<[email protected]> The message id is the log category (SMTPD and so on) or, on an error line carrying an hMailServer error number, that number, so msgid == HM5165 is a usable filter at the collector. The structured data under hmailserver@32473 carries thread, and session and client on a conversation line. UDP truncates each message to 1024 bytes, cutting the text and never the header; TCP and TLS frame each message with its length and truncate nothing below 65000 bytes. TLS verifies the collector's certificate against the machine's trust store and the configured host name, so set the host to the name in the certificate, not an IP address; no client certificate is offered. A collector that is down costs log lines, not mail. Entries go to a background thread over a queue of 4096 that drops the oldest when full; a stream that cannot connect backs off from one second to a minute; and the application log gets one line when an outage starts and one when it ends, with the number dropped. Nothing is retried from disk. Each step of a stream transport (the connection, the TLS handshake, one entry written) is given ten seconds, so a collector that accepts the connection and then goes quiet is a failure like a refused one, and a reinitialise or a service stop waits for that step at most. For rsyslog, in /etc/rsyslog.d/30-hmailserver.conf (UDP for the default transport, TCP for SyslogTransport = 1; the stop keeps the lines out of /var/log/mail.log too, so drop it if you want both): module(load="imudp") input(type="imudp" port="514") module(load="imtcp") input(type="imtcp" port="514") if ($app-name == "hMailServer") then { action(type="omfile" file="/var/log/hmailserver.log" template="RSYSLOG_SyslogProtocol23Format") stop } Without RSYSLOG_SyslogProtocol23Format rsyslog reformats to the older RFC 3164 shape and the structured data is lost. For TLS (SyslogTransport = 2 module(load="imtcp" StreamDriver.Name="gtls" StreamDriver.Mode="1" StreamDriver.AuthMode="anon") input(type="imtcp" port="6514") global(DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca.pem" DefaultNetstreamDriverCertFile="/etc/rsyslog.d/collector-cert.pem" DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/collector-key.pem") A self-signed collector certificate has to be installed as trusted on the hMailServer machine, or the handshake is refused and nothing is sent. The journal, on Linux only, needs no collector. Where the server runs as the packaged hmailserver.service, switch the sink on and leave the collector host empty: $ hmctl settings set logging --set syslog_enabled=true $ sudo systemctl restart hmailserver $ journalctl -u hmailserver -f Beside the standard fields the journal gets HMAILSERVER_CATEGORY, HMAILSERVER_MSGID, HMAILSERVER_THREAD, HMAILSERVER_SESSION and HMAILSERVER_CLIENT: $ journalctl -u hmailserver HMAILSERVER_SESSION=41 $ journalctl -u hmailserver HMAILSERVER_MSGID=HM5165 If a collector host is configured the journal is not used. The start of the sink is itself logged, as Syslog: sending RFC 5424 log entries to <host>:<port> over UDP, facility 2. or Syslog: writing log entries to the systemd journal.; a server that logs neither has the feature off. Not sent: the AWStats log, the backup log and the event-script log. Per-domain reports (since 6.3.4) GET /api/v1/reports is an index of the sections, the state of their three sources and what this server cannot answer. A report drawn while its source was off is empty, which is not the same statement as "nothing happened"; each section's note says which it is. GET /api/v1/reports/{section} answers one section, the Deck's Reports view draws the summary, and hmctl report prints a section or saves it as CSV. There is no Control Panel page. Section Counted from Says traffic the message trace Per day and per local domain: incoming, outgoing, failed_incoming, failed_outgoing failures the message trace Per local domain and SMTP status code senders, recipients the message trace The local addresses that sent or received the most, with their failures mailboxes hm_messages The largest mailboxes as they are now: a size, not a history volume the metric history, and the daily per-domain sample Processed, delivered, deferred, bounced, spam and viruses per day, with by_domain rows of spam and viruses storage the same The store's size per day and growth_bytes, with by_domain rows of each domain's size summary all three sources Every section the credential may see, with an omitted list and the reason quota the store Every mailbox with a limit and at least one message, nearest its limit first, as a share of the limit: a share now, not a history countries the metric history and the country counters What a country policy refused, per day; the split by country is since the server last started, not per day The last two are described in QuotaPolicy.md and CountryBlocking.md, not in Reports.md. Parameters. from and to as YYYY-MM-DD, both inclusive, defaulting to the last 30 days and never more than 366; domain, one hosted domain (a domain not hosted here is a 404); top, 1 to 200, default 10; format=csv, refused for summary. In CSV a cell starting with =, +, - or @ is prefixed with an apostrophe. Three things decide whether a report means anything. The message trace is off by default (MessageTraceEnabled=0), and while it is off the four sections counted from it are empty. Processed, delivered, deferred and bounced cannot be given per domain, because those counters carry no domain; spam, viruses and storage can, since schema 6062 added the daily sample in hm_domainsamples, which starts the day it was first taken. And only hosted domains are counted. Authorisation. The administrator password reads everything; every other credential needs both view_configuration and view_message_trace. domainadmin and auditor hold both; helpdesk is refused. A key restricted to named domains sees only those and is refused 403 for another. On volume and storage Reports.md contradicts itself: its Authorisation section refuses them to such a key outright, while its section What cannot be answered, and what it would take (and, for volume, its table of sections under The route) gives a credential restricted to domains the per-domain rows, as the wiki's section 16.6a does. Test both with your own restricted key before a script relies on either. The cost. One report makes at most three grouped queries; the trace read is capped at 100 000 grouped rows and the answer says truncated when it reaches the cap. hmctl report sections hmctl report traffic --from 2026-09-01 --to 2026-09-15 --domain example.com Following one connection live Since 6.3.4 the Deck's Logs page lists the sessions open now and those recently ended, and Follow shows one session's lines as they are written, from an in-memory ring of the last 4,096 log entries that no restart keeps; it needs view_logs. Reading the logs covers following a connection in full, and the log files themselves. Where to read more Documents at v6.3.4: AlertsAndAuditTrail.md, Syslog.md, Reports.md and FollowConnection.md Wiki: Monitoring and Health, sections 16.2a, 16.6a, 16.7a and 16.7b The public documentation: 16.2 Logs on disk On this forum: Monitoring and health for the metrics, the probes, the dashboard and the SNMP agent; Administrator roles for the permissions named here. See also The Control Deck and hMailServer on Linux: what you get and what you do not

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) calls the anti-virus action that strips attachments Remove all attachments and deliver the text (Delete infected attachments only in 6.3.6's Control Panel), and under it a message whose one part is an attachment, and an S/MIME message signed as one part, now arrive as the action's note alone, where every version to 6.3.6 delivered them as they arrived with what the scanner had found in them; a message the server cannot read is deleted and each of its recipients sent the "message deleted" notice whatever the two notify settings say, while an encrypted message, and one that is one text part, are delivered as they are. In the Control Panel the sidebar has a Virus scanning group, and Ctrl+K takes every old title to its page. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) lets the Linux server reach clamd over its Unix socket: ClamAVHost takes the absolute path of clamd's LocalSocket, such as /var/run/clamav/clamd.ctl for Debian's and Ubuntu's clamav-daemon, which listens on nothing else by default, and ClamAVPort is then not used; no connection test is offered for it on Linux, so a test message and the log show the scan (ClamAV.md). See the 6.3.8 announcement. hMailServer does not scan for viruses itself. It hands each message to an engine you install (ClamAV, ClamWin or any command-line scanner) and acts on the answer, and nothing described here is enabled on a fresh install. This topic covers the three scanners, what happens when one finds something or cannot run, the blocked-attachment list, and what 6.3.4 added: the sandbox, and the handling of mail sealed end to end. The one thing to know first is the failure mode: a configuration that looks enabled and is inert. A scanner switched on with a wrong host or a missing executable does not refuse mail and does not disable itself; when every scanner has errored the verdict is NoVirusFound, which from the message's point of view is identical to having been examined and found clean. 6.3.4 fixed one such case in the custom scanner, described below. What runs, and when Scanning happens on the delivery side, not during the SMTP conversation: the message has already been accepted and spooled by the time any scanner sees it. Which messages are scanned. For SMTP, a message is flagged when the connecting IP's security range has the virus-protection option; both default ranges ("Internet" and "My computer") have it. For mail this server fetches over POP3 or IMAP, the flag is the external account's own "use anti-virus" setting. A flagged message larger than AntiVirus.MaximumMessageSize is not scanned at all, silently; the default is 0, no limit. Two passes. The whole spool file is scanned first; then every MIME attachment is written to a temporary file and scanned separately, which is what matters for a scanner that does not decode MIME itself. If the message cannot be parsed for the second pass, error HM6001 is reported and only the whole file was scanned; if one attachment cannot be written to the temp directory, HM6002 is reported and that attachment was not examined. Concurrency. At most 10 scans run at once (MaxRunningScanners). A thread that cannot get a slot waits up to 60 seconds and then scans anyway, writing Scanning message without an available virus scanner slot. to the debug log. The read-only overview. Spam & virus filtering → Virus scanning overview in the Control Panel (Windows only) answers the question the settings page cannot: will an infected message be caught here. The three scanners All three can be enabled at once. They run in the order ClamWin, custom, ClamAV, stopping at the first that reports a virus. A scanner that errors does not stop the others. ClamAV (clamd) ClamWin (clamscan.exe) Custom scanner Switch AntiVirus.ClamAVEnabled AntiVirus.ClamWinEnabled AntiVirus.CustomScannerEnabled Default off off off Reached by TCP socket, INSTREAM one process per file one process per file Settings AntiVirus.ClamAVHost (localhost), AntiVirus.ClamAVPort (3310) AntiVirus.ClamWinExecutable, AntiVirus.ClamWinDBFolder AntiVirus.CustomScannerExecutable "Infected" is a reply matching stream…: <name> FOUND exit code 1 exit code equal to AntiVirus.CustomScannerReturnValue Timeout ClamMinTimeout 15 s to ClamMaxTimeout 90 s ExternalProcessTimeout, 300 s ExternalProcessTimeout, 300 s Virus name reported the name clamd returned always Unknown always Unknown ClamAV. Install ClamAV, run freshclam to download signatures, run clamd listening on TCP 3310, set the host and port, and press Test ClamAV connection. The button does three things in order: It sends nPING and requires PONG, so "nothing is listening" is told apart from "something else is listening". It sends nVERSION and reports what answered. It streams a harmless sample and then the EICAR test string, both from memory, and requires the first to come back clean and the second FOUND. A success looks like ClamAV 1.4.2/27700/Mon Sep 8 09:12:11 2026. Result: stream: Eicar-Signature FOUND; a failure names the step, for example Unable to connect to ClamAV server at localhost:3310. The custom scanner. Any program that exits with a distinguishable code will do. Put %FILE% where the file to scan belongs; without it, the file name is appended as the single quoted argument. The Control Panel ships presets, all of them Windows programs: Preset Command line "Infected" exit code Microsoft Defender "C:\Program Files\Windows Defender\MpCmdRun.exe" -Scan -ScanType 3 -File "%FILE%" -DisableRemediation 2 Sophos "C:\Program Files\Sophos\Sophos Anti-Virus\savscan.exe" -ss -archive "%FILE%" 3 ESET "C:\Program Files\ESET\ESET Security\ecls.exe" "%FILE%" 50 Bitdefender "C:\Program Files\Bitdefender\Endpoint Security\bdscan.exe" "%FILE%" 1 Kaspersky "C:\Program Files (x86)\Kaspersky Lab\Kaspersky Endpoint Security\avp.com" SCAN "%FILE%" 2 The 6.3.4 fix: a bare %FILE%. CHANGELOG records that a custom scanner whose command line holds a bare, unquoted %FILE% was handed the message file's path cut at its first space, and the default data directory, under C:\Program Files, has one. The scanner therefore scanned no file, answered with an exit code other than the virus one, and the message was delivered as clean. Since 6.3.4 a %FILE% outside double quotes is replaced by the path in quotes. One already inside them, "%FILE%" as the documented examples above write it, was not affected and is replaced as before. If you run an earlier release with a command line of your own, check it for an unquoted %FILE%. The same fix removes a trailing backslash from ClamWin's database and temporary folders, which escaped the closing quote of --database="..." and took the file to scan with it. The service account must be able to run the scanner. A new 6.3.4 installation on Windows runs the service as NT SERVICE\hMailServer by default; an upgrade keeps the account it had, LocalSystem unless the installer's page is answered otherwise (docs/ServiceAccount.md; the wiki's "LocalSystem by default" describes earlier releases and upgraded servers). Under the virtual account a custom scanner or ClamWin runs as that account: it reads the message in the temp folder, which is granted, and its own program and signatures, which the Users group can normally read; a scanner that needs more has to be granted it (Installing on Windows covers the account and its grants). Whichever account runs yours, the executable, its database directory and the temp directory must all be reachable by it. The symptom of getting it wrong is Unable to launch the virus scanner… reported as HM5406, once per message. For an engine with no command-line interface, the wiki points to an OnAcceptMessage event script or the HTTP filter hook. If you edit hm_settings by hand, note that the custom scanner's return value is stored under the misspelled name customviursscannerreturnvalue. Testing it end to end The test buttons check the scanner. To check the pipeline, send yourself a message with the EICAR test file attached and watch the application log: Virus found in message from [email protected]. Taking actions SMTPDeliverer - Message 4711: Message will be deleted (contained virus Eicar-Signature). Two things spoil that test on Windows. The host antivirus eats the EICAR file before hMailServer sees it, so exclude the mail store and the temp directory from real-time scanning. And your own outbound mail is not scanned by this server if it never comes back in through SMTP from an unauthenticated range, so send the test from an external address. What happens when a virus is found AntiVirus.Action What happens Delete entire e-mail (0, the default) The message is delivered to nobody. Logged as Message will be deleted (contained virus X). Delete infected attachments only (1; Remove all attachments and deliver the text in 6.3.7's Control Panel) The message is rebuilt, not edited: the first text part is kept, everything else is discarded, the VIRUS_ATTACHMENT_REMOVED server message is prepended to the body and the subject is prefixed with the VIRUS_FOUND message. Logged as Message attachments stripped (contained virus X). Since 6.3.7 a message whose one part is an attachment arrives as the note alone, where every version to 6.3.6 delivered it as it arrived, and one the server cannot read is deleted, each recipient sent the "message deleted" notice whatever the two notify settings say Understand the rebuild before you choose it: an HTML alternative, inline images and every clean attachment go with the infected one. AntiVirus.NotifySender and AntiVirus.NotifyReceiver are both off by default, and the wiki's advice is to leave them off: the sender address on a virus is virtually always forged, so a notification is backscatter. When a scanner cannot run Setting Default Meaning AVFailAction 0 0 delivers the message anyway. 1 never delivers unscanned mail AVFailRetryMinutes 15 Minutes between scan attempts while a message is held AVFailMaxHolds 16 Holds before the message is returned to its sender. 0 tells the sender immediately They are on Anti-virus settings → General, card When a scanner cannot run. The wiki describes them as stored in hMailServer.INI; since 6.3.4 the database is the settings store and a [Settings] line edited in the file is put back, so change them in the Control Panel, over REST (PUT /api/v1/settings/ini/{name}) or, on Linux, with hmailserver --set-setting NAME=VALUE. At the defaults, AVFailAction 1 gives a message about four hours to be scanned before its sender is told. The hold counter is the policy's own, not the message's try counter, because incrementing that would make the pass that finally scans the message skip DKIM signing. It lives in memory, so a restart gives every held message a fresh budget, and it is bounded at 100,000 tracked messages; beyond that messages are still held, never delivered unscanned, and HM6195 is reported. Every scanner error is reported as HM5406 with the failing scanner's own text. Blocked attachments Spam & virus filtering → Blocked attachments refuses files by name pattern. The master switch is AntiVirus.EnableAttachmentBlocking, default off. The wiki's starting list: *.exe *.scr *.pif *.bat *.cmd *.com *.vbs *.js *.jse *.wsf *.wsh *.hta *.jar *.msi *.msp *.cpl *.lnk *.reg *.ps1 *.iso *.img The attachment is not deleted. Its file name has .txt appended and its content is replaced by the ATTACHMENT_REMOVED server message, which names the original file through the %MACRO_FILE% macro, so the recipient receives invoice.exe.txt containing an explanation. Matching is a case-insensitive wildcard over the declared file name. It does not look inside archives and cannot see the real type of a file whose extension lies: it is a policy control, not a scanner. Since 6.3.4 the rewrite is logged. A rewritten message produces VirusScanner - Message 1234 from [email protected]: the attachment(s) invoice.exe matched the blocked-attachment list. Their content has been replaced with a notice and the message rewritten. Before this the rewrite left no record anywhere but inside the message itself. Sealed mail: what the list and the scanner do with it Since 6.3.4. A message encrypted end to end, S/MIME or PGP/MIME, is opaque to the server, which decides that from the body, never from the Content-Type label, so a message only labelled encrypted is blocked, stripped and scanned like any other. What opaque means exactly, and what the rest of the server does with such a message, is in the topic on end-to-end encryption and sealed mail. For the list and the scanner, none of it is a setting: The blocked-attachment list never rewrites an opaque message. In 6.3.3 and earlier blocking *.asc, *.p7m, *.p7s or *.* replaced the content of every sealed message with the notice, with no log line, and the plaintext existed nowhere else. It is now delivered as it arrived, and the application log says it was not checked against the list. A signed message is checked like any other, with its detached signature left alone. The scanner still scans it, and what it examines is ciphertext, so a clean result is not a verdict on the message; the log says its content COULD NOT BE EXAMINED for viruses. It is deliberately not treated as a scanner failure, so a server with AVFailAction 1 does not hold every encrypted message it receives; a policy that unscannable mail must not be delivered belongs in a rule or a Sieve script, not the scanner. The "delete attachments" action never strips an encrypted message. When the scanner objects to one, the message is left as it arrived and the log ends Set the anti-virus action to delete the message if such a message must not be delivered. That is not a quiet pass: the message still carries what the scanner objected to. To 6.3.6 this held for every opaque message; since 6.3.7 an S/MIME message signed as one part (application/pkcs7-mime holding signed-data, which is opaque but not encrypted) is replaced by the action's note, as any other single-part attachment is. The sandbox Since 6.3.4 (schema 6091). ClamAV says whether a file is known to be bad. What an unknown file does when it is opened takes a sandbox, a service outside this server which you run or subscribe to. It does nothing until you name one: SandboxEnabled is off, and cannot be saved on while SandboxSubmitUrl or SandboxVerdictUrl is empty. Both addresses must be https, or plain http to this machine. After the anti-virus scan and before the rules, a message that no signed-in user of this server submitted, with an attachment whose extension or media type is in SandboxTypes, is held: in no mailbox, not in the quarantine, and in nobody's digest. The default types are: exe,dll,scr,js,jse,vbs,wsf,hta,msi,bat,cmd,ps1,jar,lnk,iso,img,docm,xlsm,pptm, doc,xls,ppt,rtf,pdf,zip,7z,rar The sandbox says What happens every file clean delivered, with an X-hMailServer-Sandbox header reading clean (<n> file(s), <seconds>s) any file malicious, or suspicious at or above SandboxScoreThreshold (70) quarantined with the verdict as the reason; returned to its sender with 5.7.1 while QuarantineEnabled is off an error, once every file has answered, or no answer within SandboxTimeoutMinutes (15) SandboxTimeoutAction: deliver (0, the default) delivers with a header saying so; bounce (1) returns it to its sender The contract. One POST per file to SandboxSubmitUrl as multipart/form-data with four fields (file, filename, sha256, message_id), then a GET to SandboxVerdictUrl every SandboxPollSeconds (30), answered with JSON such as {"state":"done","verdict":"clean","score":3,"detail":"opened, nothing happened"}. With SandboxToken set, every request carries Authorization: Bearer <token>; the token is not in the backup archive. No vendor adapter is shipped. The document notes that a service which shares what it is sent, as VirusTotal shares uploads with its subscribers, is a place the attachments of the mail you receive go. What is exempt. What your own signed-in users submit (authenticated SMTP, the webmail, the send API) is not uploaded unless SandboxInternalMail is on. The document states the threat model of that default: a stolen password used to send attachments to colleagues is not examined by the sandbox, though ClamAV, the blocked-attachment list and compromised-account detection still apply. Limits. An attachment larger than SandboxMaxSizeKB (20480) is not submitted. A message with more files to submit than SandboxMaxFiles (10) is not held, and SandboxTimeoutAction decides it at once. Archives are submitted whole; links are not opened anywhere. A clean or malicious verdict is remembered by the file's SHA-256 for SandboxCacheHours (24), server-wide. What the administrator sees. GET /api/v1/sandbox/held lists what is held; POST /api/v1/sandbox/held/{queue_id}/release delivers without the verdict and DELETE /api/v1/sandbox/held/{queue_id} discards, both on the audit trail. The Control Deck's queue page has a Held for a sandbox box, and hmctl queue list has a held column. The settings are on the Control Panel's Anti-virus page, Sandbox tab (Windows only), and on PUT /api/v1/settings. Rules, and the message store Since 6.3.4 the mail flow rules can match the attachment's name and, in the outbound content policy, its type read from its bytes, so a rule can hold or refuse what the blocked-attachment list would only rename; that is in the topic on mail flow rules. The message store options added in 6.3.4 change nothing for a scanner: a message is scanned as plain RFC 822 in the queue and only placed in the mailbox in its stored form afterwards. A scanner pointed at the data directory from outside the server reads a compressed or encrypted container, not a message. Error codes Code Meaning HM5406 An enabled scanner errored; the detail names the scanner and the failure HM6001 The message could not be parsed, so only the whole file was scanned HM6002 An attachment could not be written to the temp directory for scanning HM6195 100,000 messages are already held for want of a scan; further holds are not counted HM7272 The sandbox called a message malicious and the quarantine could not store it; it is held again, never delivered Where to read more Wiki: https://gitlab.com/hmailserver/hmailserver/-/wikis/Stopping-Viruses https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Sandbox.md https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/SealedMail.md Public documentation: https://www.progressiverobot.com/hmailserver-documentation/#11-stopping-viruses, https://www.progressiverobot.com/hmailserver-documentation/#111-clamav and https://www.progressiverobot.com/hmailserver-documentation/#112-blocked-attachments Upgrading to 6.3.4 and the 6.3.4 security announcement

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) fixes the SMTP log writing some credentials in the clear with SMTP logging on: an AUTH PLAIN sent before EHLO or not in its one exact form, the bearer token of AUTH XOAUTH2 and AUTH OAUTHBEARER, and the lines of SCRAM-SHA-256, OAUTHBEARER and GSSAPI went into the log as the client sent them, base64 and readable, in every version from 6.0.0 to 6.3.6. An SMTP log written by 6.3.6 or earlier may hold them: redact every line of a SASL exchange before posting one. A database write that failed is now reported (HM5032) where it was taken as done, so an error log that was quiet may name failures that were hidden; and the IMAP and POP3 listeners can take the client's address from a PROXY protocol header, as the SMTP listener does, both off by default. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) no longer reports a stop of the servers as errors of their own: work that outlasted a stop, a reinitialise among them, writes a line to the debug log where it wrote HM5136 or another code to the error log and ran an installed OnError script, so the error log of a server that reinitialises may be quieter; and an event script's call that meets a stop fails with "The server is stopping. The request was not completed.", its script error going to the debug log. A failed account sign-in at the REST listener from loopback, as through a reverse proxy on the same machine, is counted against the name alone and writes no Auto-ban: logon failure line for fail2ban to read. See the 6.3.8 announcement. This topic covers what hMailServer writes to disk and which file to open, the settings that decide how much is written and kept, the live views, following one connection (since 6.3.4), a copy to syslog or the journal, and the fail2ban filter the Linux package installs. The one thing to know first: open ERROR_hmailserver_<date>.log before anything else. It holds only the entries the server itself classed as an error, and its mere existence is a diagnostic, because the Control Panel's built-in self-test (Monitoring & troubleshooting → Diagnostics, so Windows) fails when any ERROR* file is present in the log directory. Where the logs are Platform Log directory Set by Windows C:\Program Files\hMailServer\Logs by default [Directories] LogFolder in hMailServer.INI Linux /var/log/hmailserver, created by the package as 0750 hmailserver:hmailserver and left in place when the package is removed [Directories] in /etc/hmailserver/hMailServer.ini The format is the same on both platforms. On Linux the journal (journalctl -u hmailserver) carries the startup line, anything written to standard error and the operational events; a memory-safety fault is recorded in /var/log/hmailserver/crash-oracle.log as one line (signal, address, thread, event number) before the process dies by the signal it caught. [Directories] is per machine and is never mirrored into the database, so do not copy another server's file over this one's. The files File Holds Rotates hmailserver_<date>.log Everything: application lines, and the protocol conversations unless they are split out Daily, by name ERROR_hmailserver_<date>.log Only entries the server classed as an error Daily, by name hmailserver_SMTP_<date>.log, hmailserver_IMAP_<date>.log, hmailserver_POP3_<date>.log The per-protocol conversations, when SepSvcLogs=1 Daily, by name hmailserver_backup.log Every backup and restore run, line by line Never: one growing file. LogDeleteDays matches it by name but deletes only a file nothing has written to for that many days hmailserver_events.log Event-script output Never, as above hmailserver_awstats.log The AWStats-format feed, when AWStats logging is on Never, as above hMailServer_messagestore_consistency.report The most recent message-store consistency scan: one line per message whose file is missing, as messageid<TAB>account<TAB>expected-path Overwritten each run The backup and event logs are among the files the server marks with a byte-order mark; they are UTF-16LE on both platforms, so a file written on one reads on the other. A log line looks like "IMAPD" 5220 41 "2026-09-17 09:00:00.120" "192.0.2.7" "SENT: A1 OK LOGIN completed", where IMAPD is the category and 41 is the session id that every line of that conversation carries. The conversation categories are SMTPD, SMTPC, POP3D, POP3C and IMAPD; APPLICATION, TCPIP, DEBUG and ERROR are the rest. The settings Setting Default What it does What it looks like when it is wrong LogLevel 9 Not the choice of log types, which is the Logging page's tick-boxes (or PUT /api/v1/settings/logging). The Settings Reference records two effects: long lines are cut only when it is 2 or lower with debug off, and IMAP SENT: lines for FETCH and STATUS are logged only when it is above 2 or debug is on — Debug messages (the Logging page) off Adds the LOG_DEBUG lines, including the accept-pipeline stage timings Left on, a busy server fills the disk in days, and a full disk stops mail SepSvcLogs 0 Splits SMTP, IMAP and POP3 into their own files Off, so a search for one protocol wades through all three LogDeleteDays 0 Deletes files named hmailserver_*.log or ERROR_hmailserver_*.log whose last write is older than this many days, and nothing else in the directory. 0 keeps everything for ever Left at 0 on a server with debug logging on: the usual cause of a full log volume JsonLogging 0 Writes JSON lines, for Elasticsearch, Loki or Splunk Overridden while Logging.LogFormat is NCSA: the pipeline gets NCSA lines and no error SlowQueryLogMilliseconds 0 (off) Logs, and counts in hmailserver_db_slow_queries_total, any database statement slower than this Off, so a database that has become slow leaves no trace but the latency histogram WindowsEventLogLevel 2 How much also goes to the Windows event log — Where they are set. On Windows, Monitoring & troubleshooting → Logging in the Control Panel. On either platform the table's [Settings] keys (all but Debug messages) are set with PUT /api/v1/settings/ini/{name} or hmctl ini set. On Linux there is no Control Panel, so the log categories, debug messages among them, are set with PUT /api/v1/settings/logging, hmctl settings set logging or the Deck's Settings page. There, WindowsEventLogEnabled=1 is a Windows inheritance in name only, because on Linux it means syslog, and the operational events reach the journal through it. Since 6.3.4 these keys are stored in the database, whose value is the one used: where the service can write hMailServer.INI, a line edited there is put back and reported as HM5804. How the settings store works has the rest, [SettingsOverride] (HM5807) among it. Rotation. On Windows, LogDeleteDays is the rotation. On Linux the package installs /etc/logrotate.d/hmailserver; the rule uses copytruncate because the server does not reopen its log on a signal, and the rule says so at length. Debug logging: on to diagnose, off afterwards. The stage-timing lines that diagnosing stalled mail depends on are debug lines; the slow ones (ten seconds or more) are written at application level as well. Leaving debug on matters because a full disk stops mail: the application log warns below DiskSpaceWarningThresholdMB (1024 MB), and below MinimumFreeDiskSpaceMB (100 MB) on the message store's volume the server answers 452 4.3.1 Insufficient system storage. and senders retry. The live views Windows: the Control Panel. Monitoring & troubleshooting → Live logs is a real-time, colour-coded tail of what the server is writing; every page in that group but Discovery reads the running service over COM (Discovery has no COM interface, so it reads the REST listener and needs RestApiPort set). Diagnosing stalled mail, in the same group, has a one-click debug-logging toggle. Both platforms: the Control Deck and hmctl. The browser Control Deck at / on the REST listener has a Logs page that lists the log files and shows any file's last 200 or 2,000 lines, without a share to the log directory. Since 6.3.4, hmctl log list and hmctl log tail <name> --lines N read the same over GET /api/v1/logs and GET /api/v1/logs/{name}, and work against a Windows or a Linux server because they go over HTTP rather than DCOM. The Linux packages install hmctl as /usr/bin/hmctl; the HmailServer PowerShell module is the same vocabulary for PowerShell. The files themselves, from a shell. On Windows: $log = "C:\Program Files\hMailServer\Logs\hmailserver_" + (Get-Date -Format "yyyy-MM-dd") + ".log" Get-Content $log -Tail 0 -Wait | Select-String "SMTPD|SMTPConnection" On Linux: tail -f /var/log/hmailserver/hmailserver_$(date +%F).log journalctl -u hmailserver -f Following one connection, since 6.3.4 Every SMTP, POP3 and IMAP conversation is a session with a number, and every line the log writes for it carries that number. Until 6.3.4 that meant searching the day's log for the number afterwards. Since 6.3.4 the Connections section at the top of the Deck's Logs page lists every connection the server has open now, newest first, and the last two hundred that ended: the id, the protocol, the address it came from, the port it came in on, the account once it has signed in, and when it started or ended. Follow beside any of them opens a pane holding every line the log has for that session and keeps adding lines as the server writes them; Copy puts the whole pane on the clipboard; Stop following ends it. A session that ended a second ago is still listed, so an SMTP conversation that lasted a quarter of a second can be picked and read. How it works, and what that means for you. The server keeps the last 4,096 log entries in memory, across every session; the ring is always on, and nothing survives a restart. A debug-logged IMAP client can fill it in minutes, so a follow whose cursor points before the oldest entry still held answers dropped: true and starts from what is left; the page says so. A line older than that is in the log files and nowhere else. The conversation lines (SMTPD, SMTPC, POP3D, POP3C, IMAPD) carry the session id in their own column. DEBUG and TCPIP lines carry none and are filed by the first number after the word session in their text (TCP connection started for session 41, Ending session 41); that is an inference, and the text is not always the server's own. A line that does neither is nobody's and is not shown. It shows what the log mask lets through, and no more. With IMAP logging off, an IMAP session is listed and its follow is empty; the answer's logged field says whether the log mask lets that protocol's lines through, and the page names the switch to turn on. Debug logging is what makes the TLS handshake, the timeouts and the disconnect appear beside the conversation. The connections the server opens itself are listed too, marked client: SMTP client, POP3 client, IMAP client, and the short-lived SpamAssassin and Diagnostics connections. Not listed: anything that is not a TCPConnection, so the REST API, the webmail, the DAV services, ManageSieve and the anti-virus scanner's socket to clamd are not sessions here. A delivery to a host that refuses the connection is not listed either, though with debug logging on its lines can still be followed by id, with known: false. A follow is a long poll, not a stream: GET /api/v1/logs/sessions/{id}?after=0&wait=10 answers the session's lines after sequence 0 and a cursor to ask after next time, waiting up to wait seconds for the next line. The wait is at most twenty seconds, and one waiting follow at a time is allowed, because a waiting follow holds one of the HTTP listener's four worker threads, which also serve the webmail; a second follow is answered at once with waited: false. An answer carries at most 500 lines, and the pane keeps the last 5,000. GET /api/v1/logs/sessions lists the sessions (the newest thousand when more are open, with truncated: true), with held_lines and logging_enabled. Both routes are under the view_logs permission, which the built-in read-only auditor role holds, and both are server-wide: a credential confined to some domains is refused them, as it is refused the log files. There is no Control Panel page and no hmctl verb for this; the Control Panel's Live logs is the older, whole-log form over COM. There is no filtering inside a session. A copy to syslog, or to the journal, since 6.3.4 Since 6.3.4 every entry the server logs can also be sent as RFC 5424 to a syslog collector or, on Linux, where the server runs as the packaged systemd unit with no collector host named, to the journal. It is a copy: the files above are written exactly as before, and nothing it does can stop a line reaching them, or delay or lose a message. Its settings are hm_settings rows, not [Settings] keys, read when the sink starts (the next service start or reinitialise), not when they are saved. With the journal on, journalctl -u hmailserver HMAILSERVER_SESSION=41 follows one session from the journal, across a restart. Alerts, the audit trail, syslog and the journal has the twelve settings, the transports, the rsyslog configurations and the journal's fields. Codes in the error log worth knowing since 6.3.4 Code What it says, and what to do HM5804 A value typed into hMailServer.INI [Settings] for a key the database holds was put back at the last start. Change it in the Control Panel, with PUT /api/v1/settings/ini/{name}, hmctl ini set or COM Settings.SetIniSetting HM7460 AutoBanFirewall is on and the Windows service account cannot write firewall rules (below) HM7495 The database records a stored-secrets key this server does not hold, and every account's second factor refuses every code: import the exported key with hMailServer.exe --import-stored-secrets-key FILE (elevated) or sudo hmailserver --import-stored-secrets-key FILE, then restart HM7540 A restore was refused because the archive holds an entry with an absolute path, a .. segment or a link with somewhere to point; nothing was extracted. Find out where the archive came from fail2ban on Linux, and the Windows contrast An auto-ban (repeated failed logons from one address) is enforced by the server's own listeners: the kernel accepts the connection and the server refuses it. Shipped in 6.3.2, the Linux package offers two ways to push the ban below the server, and one is enough. fail2ban needs no capability at all, because it runs as root and reads the application log. The package installs /etc/fail2ban/filter.d/hmailserver.conf and a jail at /etc/fail2ban/jail.d/hmailserver.conf, disabled as Debian asks of a shipped jail. The filter matches the two fixed-shape lines the server writes for every counted failure and every ban, whether or not protocol logging is on; auto-ban on logon failure (AutoBanOnLogonFailureEnabled, on as shipped; auto_ban_on_logon_failure over REST) is what writes them. Turn it on: printf '[hmailserver]\nenabled = true\n' > /etc/fail2ban/jail.d/hmailserver.local systemctl reload fail2ban fail2ban-client status hmailserver The server itself, with AutoBanFirewall=1, makes the firewall match its auto-ban ranges at start, on every ban, on every range deletion and once a minute. On Linux it runs /usr/lib/hmailserver/autoban-hook, which keeps an nftables set with a timeout (iptables where there is no nft) and drops TCP from the banned address to the ports the server listens on. Changing the firewall needs CAP_NET_ADMIN, which the unit deliberately does not grant a mail server by default. Grant it once with a drop-in, and the hook's check verb says whether it can act: systemctl edit hmailserver [Service] AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN systemctl restart hmailserver sudo -u hmailserver /usr/lib/hmailserver/autoban-hook check AutoBanCommand, on either platform, runs a program of yours instead of, or as well as, AutoBanFirewall (<command> ban <address> <minutes> <ports> and <command> unban <address>) for a cloud firewall or a router. Before turning either on, list the addresses you administer from in AutoBanNeverBan: a ban that reaches the firewall locks that address out of every mail port on the machine. Since 6.3.4 AutoBanFirewall, AutoBanCommand and AutoBanNeverBan are stored settings like the other [Settings] keys: set them over the API or with hmctl ini set, not in the file. The fail2ban filter and the hook are part of the Linux packages. On Windows, AutoBanFirewall writes one inbound block rule per banned address in Windows Defender Firewall, named hMailServer auto-ban <address> and scoped to the ports the server listens on, and since 6.3.4 a new installation's service account, NT SERVICE\hMailServer, cannot write them: HM7460 is logged once a start and no rules appear, while the listeners still refuse a banned address. Turn AutoBanFirewall off there, or use AutoBanCommand towards a router or cloud firewall; HM7461 names rules a LocalSystem run left behind, with the PowerShell that removes them. What to include in a report SUPPORT.md asks for, in order of value: the relevant log, not a description of it, with debug logging on and the window around the problem; the ERROR_hmailserver_<date>.log for the same window; the exact version, the platform and the database backend; and what you expected instead, and whether it is reproducible. Remove passwords, keys and real email addresses first, because a posted log is public. Read this before you post asks for the same log lines and adds the build number, the package, whether the server was upgraded and the Diagnostics output; Copy on the Diagnostics page puts the whole result on the clipboard. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email. Where to read more Wiki: Monitoring and Health, sections 16.2 and 16.2a; Troubleshooting, sections 19.10 and 19.11; Installing on Linux, for the journal and fail2ban. Repository: FollowConnection.md, Syslog.md, SUPPORT.md. The public documentation: Logs on disk, Logging and log rotation on Linux, Follow one message end to end in the logs, Codes in the error log. On this forum: hMailServer on Linux: what you get and what you do not, The Control Deck, Diagnosing stalled mail, and IP ranges, auto-ban and lockout for when a ban is made and how to lift one.

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) adds reasons a message stays where it is. A forward, a redirect or a rule's forward of mail from outside whose From names the forwarding domain's own organisation, and that did not pass DMARC for it when it arrived, keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. RuleLoopLimit 0 now stops out-of-office replies, as it stopped every other generated message. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries, where a failed lookup was read as a name with no addresses. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, and an error log that was quiet may now name database failures that were hidden (HM5032). See the 6.3.7 announcement. This topic covers the case where mail is not moving and the server looks healthy: no crash, no error, the service running, and a log that seems to stop mid-transaction. The server used to be silent about it; it is no longer, and the answer is usually in one line of the log. The one thing to establish first is which half is stuck: the server accepting a message from a sender, or delivering one it has already accepted. The two fail differently, for different causes, under different settings. Which half is stuck? Accepting. The sending server connects, sends the message, and then waits. From its side there is a timeout after the message body was transmitted; Postfix reports it as timed out while sending end of data. The message never appears in your queue. Delivering. The message was accepted (the sender got a 250), is visible in the delivery queue, and never leaves. Monitoring & troubleshooting → Delivery queue in the Control Panel tells the two apart in one look, and so does hmailserver_delivery_queue_messages on the metrics endpoint once it has a port (see Commands). The Control Panel is Windows-only; on Linux the queue is GET /api/v1/queue, hmctl queue list (since 6.3.4) and the Control Deck's queue page. Turn on debug logging first Control Panel → Logging → tick Debug messages, and make sure logging is enabled; then reproduce the problem once. The stage lines below are written at debug level. The slow ones (ten seconds or more, for a whole stage or for any single spam test) are also written at application level, so a server with only application logging still records the important half. Turn debug logging off again afterwards on a busy server. Reading the logs covers where the logging keys live on each platform. Accepting: the sender times out after sending the message Between the 354 and the 250 the server runs the accept pipeline: spam tests, message modifications, archiving, the OnAcceptMessage script and the database save. All of it runs on a bounded pool of threads, and the reply is sent only when it finishes. The log times each stage: SMTPConnection - accept: start spam-protection. Spam test: SpamTestDNSBlackLists, Score: 0, Time: 12 ms Spam test: SpamTestSpamAssassin, Score: 0, Time: 120000 ms SMTPConnection - accept: done spam-protection in 120047 ms (session 5). SMTPConnection - accept: done message-modifications in 0 ms (session 5). SMTPConnection - accept: start script/save. SMTPConnection - accept: done script/save in 3 ms (session 5). How to read it. Read it as a sequence and look for the last line written; the stage that is stuck is the one after it. Two stages announce themselves with a start line (spam-protection and script/save), so for those a start with no matching done is the whole diagnosis. The middle stage, message modifications, has no start line, so a stall there shows as done spam-protection followed by silence. If every stage completes but one took a long time, the Time: or done ... in value names the culprit. What the log shows Cause SpamTestSpamAssassin with a large Time: spamd is unreachable, overloaded, or accepting connections without answering SpamTestDNSBlackLists, SpamTestSURBL or SpamTestSPF slow The resolver is not answering. Look in the TCP/IP log for DNS - Query timed out and DNS - Query failure. Do not reach for DNSServer as the fix (below) done spam-protection, then silence Message modifications: the spam headers, the signature, the List-* headers, or the write of the modified message back to disk script/save slow An OnAcceptMessage event script, or the database Nothing between 354 and silence A version older than 6.2.17, which added these lines. Upgrade The DNSServer setting. It was broken in 6.2.17 and 6.2.18 (CHANGELOG's 6.2.19 section names 6.2.16 as well) and fixed in 6.2.19: on the affected versions every lookup through a configured custom server failed, and DNSBL, SURBL, SPF, DKIM and MX lookups failed in silence, so such a server had quietly stopped most of its spam filtering while still accepting mail. One caveat survives the fix: a custom server list is only honoured together with DNS_QUERY_BYPASS_CACHE, so every lookup pays full price. Leave DNSServer empty unless you need a specific resolver. SPF, and SPF on Linux. In 6.3.3 and earlier the SPF evaluator resolved names through the Windows DNS client rather than through the server's resolver. On Windows an SPF check therefore went to the system resolvers whatever DNSServer said; on Linux every SPF check returned a temporary error, reported once per process as HM6406, and because SpamTestDMARC evaluates SPF, DMARC there lost its SPF half. The RFC 7208 evaluator that replaced it in 6.3.4 resolves through the same resolver as every other lookup. If you run 6.3.3 or earlier on Linux, an SPF or DMARC verdict from it is not a verdict. The deadline. Since 6.2.17 acceptance is bounded: if the pipeline exceeds FinalizationTimeout (240 seconds by default) the server answers 451 4.3.1 Server temporarily overloaded while accepting the message; please retry. and the sender retries rather than waiting for a reply that never comes. The accompanying error entry names the deadline that was hit, and the stage timings above it name what consumed the time. If every message stalls, not just one Look for these two lines: Task SMTP-accept session=42 ip=203.0.113.9 waited 8 seconds for a thread in work queue Asynchronous task queue. 14 task(s) are still queued. All 15 threads in work queue Asynchronous task queue have been busy for at least 120 seconds, so no further task on this queue can start. 14 task(s) are queued. Running: SMTP-accept session=42 ip=203.0.113.9 (thread 4120, 168s), ... Every worker is occupied, and one slow dependency does this to the whole server. The task names say which sessions are stuck; the stage timings say what they are stuck on. AsyncQueueStallThreshold (120 seconds by default) is how long every worker may be busy before the second report is written; 0 turns the report off. Delivering: how the queue works There is no in-memory queue to inspect. The delivery queue is the set of rows in hm_messages whose type is delivering, and the delivery manager is a loop that polls for them. Three properties of that loop explain most of what a stuck queue looks like: The selection is where messagetype = 1 and messagelocked = 0 and messagenexttrytime <= now, ordered by size, then try count, then id. Small and new goes first. The delivery pool is separate and smaller than the accept pool: maxdelivertythreads, 10 by default, on the work queue named SMTP delivery queue. It is not the Asynchronous task queue that finalises inbound mail, so a saturated delivery pool does not stop mail being accepted; it stops it leaving. The manager wakes on demand, or once a minute otherwise. A message whose next-try time has just passed is picked up within a minute, not instantly. At startup every message in the queue is unlocked (update hm_messages set messagelocked = 0), because locks held by a server that crashed would otherwise strand those messages for ever. That is also why exactly one hMailServer service may run against one database. The retry schedule, and when a message gives up Setting Where Default Effect Number of retries Mail flow & delivery → Delivery of e-mail (smtpnoofretries) 4 How many further attempts after the first Minutes between every retry The same page (smtpminutesbetweenretries) 60 The gap between them QuickRetries [Settings] 0 (off) The first N retries use the short gap below instead, for greylisting QuickRetriesMinutes [Settings] 6 That short gap QueueRandomnessMinutes [Settings] 0 (off) Adds a random 1..N minutes to every gap, so a burst does not retry in lockstep Route number of tries and minutes between tries Mail flow & delivery → Routes — Overrides both, per matching route; where several routes match, the largest of each wins With the shipped defaults that is five attempts an hour apart: a destination that is refusing temporarily holds a message for roughly four hours and then bounces it. The try count in the queue is the same messagecurnooftries the arithmetic uses. Two things do not advance that counter: Deliver now resets the next-try time without touching the count, and a message held because the virus scanner could not examine it is held on its own budget (below). Reading a stuck queue What the queue looks like What it means Where to look next Depth climbing, oldest message seconds old A busy server draining normally. Not a fault — Depth flat, oldest message hours old, try counts climbing The far end is deferring you The SMTP log for that destination; your IP's reputation and PTR Every stuck message shares one recipient domain That destination, not your server The route settings for it, and its MX Try counts stuck at 0, next-try times in the past, nothing moving Delivery threads are not running the tasks hmailserver_workqueue_depth{queue="SMTP delivery queue"}, and the saturation report above Messages appear and vanish without arriving Delivered to the wrong place, or discarded by a rule Mail flow & delivery → Rules; enable MessageTraceEnabled and read the trace Try count 1 and a next-try time of 1901-01-01 An ETRN hold: the one route these recipients match is an ETRN route Mail flow & delivery → Routes; the log line is SMTPDeliverer - Route Message: HOLD for later delivery.. Queue empty, but the sender got a 250 and nobody received it Delivered locally into a folder you are not looking at, or the row was removed The account's folders; the message trace Message N could not be delivered since the data file does not exist The row and the store disagree: a restore that missed files, or a store on a share that vanished Turn on MessageStoreConsistencyCheck and read the report When the virus scanner cannot examine a message. With AVFailAction=1 such a message is not delivered. It is held on its own schedule rather than on the SMTP retry budget: AVFailRetryMinutes apart (15 by default), at most AVFailMaxHolds times (16), and then returned to the sender. The log says where in that budget each hold is (... this was hold 3 of 16 before it is returned to the sender.). Each hold counts as a deferral in hmailserver_messages_deferred_total, so a scanner that has stopped answering shows as a deferral rate with no matching delivery rate. The usual causes of a genuinely stuck delivery A virus scanner that stops responding. ClamAV is contacted after the message is accepted, so a wedged clamd shows up as accepted but never delivered. Each socket operation is bounded by ClamMinTimeout / ClamMaxTimeout, but that is a deadline on one read or write, armed fresh each time, and the message is streamed in chunks, so one large message can take considerably longer than ClamMaxTimeout in total. A remote server that answers extremely slowly. The idle timeout is re-armed on every byte received, so a host that sends one byte occasionally used to hold a delivery thread indefinitely. Since 6.2.18 there is an absolute ceiling, ClientSessionCeiling (30 minutes by default), armed once and never re-armed. A custom virus scanner or external tool that hangs, bounded by ExternalProcessTimeout. A message held for a sandbox's verdict (since 6.3.4). With SandboxEnabled on, a message none of your own signed-in users submitted, whose attachment is of a submitted type, waits for the sandbox you named to open the file, up to SandboxTimeoutMinutes (15 by default). While it waits it is in no mailbox, not in the quarantine and not the queue's to retry: its Retry and Delete answer 409. It is listed by GET /api/v1/sandbox/held, by GET /api/v1/queue marked held: sandbox (the held column of hmctl queue list), and in the Deck's queue page under Held for a sandbox, where it can be delivered without the verdict or discarded. The database. Check for errors mentioning the connection pool. Since 6.3.4 a message can also wait in other places, each on purpose: held for review by compromised-account detection in enforce (/api/v1/review); in a destination's back-off window, with the last error Delivery deferred by this server's own shaping, not by a remote server. (GET /api/v1/queue/shaping; such a deferral does not spend the message's retries); under an IP warm-up ceiling; or behind a drain, when /readyz says draining. Section 19.11a of the Troubleshooting page lists each with what to do. Settings that bound each stage All are in seconds. On 6.3.3 and earlier they are [Settings] keys in hMailServer.ini. Since 6.3.4 they are stored in the database and changed in the Control Panel, over the REST API at /api/v1/settings/ini/<name> or with hmctl, and the save applies them. They still appear in the file's [Settings] section, but the database's value is the one used: where the service can write the file, an edit made there is reported (HM5804) and put back at the next start. Setting Default Bounds 0 means FinalizationTimeout 240 The whole accept pipeline, after which the sender gets a 451 no bound SAMaxTimeout 90 SpamAssassin: idle timeout, and a session ceiling of this plus 30s not "no bound" ClamMaxTimeout 90 ClamAV: idle timeout per socket operation not "no bound" DNSQueryTimeout 10 A single DNS query no bound ScriptTimeout 60 One event script invocation no bound ExternalProcessTimeout 300 An external scanner process no bound ClientSessionCeiling 1800 An entire outbound delivery session no bound AsyncQueueStallThreshold 120 How long every worker may be busy before the saturation report reporting off DBConnectionAcquireTimeout 60 Waiting for a pooled database connection no bound The two exceptions matter, because 0 tightens them instead of removing them. SAMaxTimeout and ClamMaxTimeout are not passed straight through: they go to a calculator that returns the minimum whenever the maximum is lower than it. So SAMaxTimeout=0 gives a 30-second idle timeout (SAMinTimeout) and a 30-second session ceiling, and ClamMaxTimeout=0 gives 15 seconds (ClamMinTimeout). To lengthen either, raise the value; to lengthen it a long way, raise the matching ...MinTimeout too. There is no way to make either unbounded. When the pool deadline expires, a recipient lookup answers 451 4.3.2 rather than 550: the server can tell "the database did not answer" from "no such address", so a database locked by a backup defers the mail instead of bouncing it. Commands The COM examples below are Windows-only. The metrics endpoint and the REST routes work on both platforms. Is it accepting, or is it delivering? The metrics listener is off until MetricsServerPort is set (9100 here; since 6.3.4 the port, the bind address and the listener's other settings, its credentials included, wait for a reinitialise). Once it is on, the endpoint needs no credential on a loopback bind: (Invoke-RestMethod http://127.0.0.1:9100/metrics) -split "`n" | Select-String '^hmailserver_delivery_queue_(messages|oldest_message_age_seconds) |^hmailserver_workqueue_depth|^hmailserver_messages_(delivered|deferred|bounced)_total ' A rising oldest_message_age_seconds with a flat delivered_total is the delivering half. A rising hmailserver_workqueue_depth{queue="Asynchronous task queue"} is the accepting half. The COM examples start from a signed-in $app: $app = New-Object -ComObject hMailServer.Application $app.Authenticate("Administrator", "<password>") | Out-Null Push one message to the front of the queue (resets its next-try time, leaves the try count alone, and releases an ETRN hold): $q = $app.GlobalObjects.DeliveryQueue $q.ResetDeliveryTime(4711) $q.StartDelivery() Wake the delivery manager without touching any message: $app.GlobalObjects.DeliveryQueue.StartDelivery() Remove one message from the queue (it is locked first, so a message a thread already has is not pulled from under it): $app.GlobalObjects.DeliveryQueue.Remove(4711) Follow the log while you reproduce it by tailing the day's file and filtering on accept: |Spam test:|waited .* for a thread|have been busy|SMTPDeliverer; Reading the logs has the tail command for each platform. Turn the message trace on (MessageTraceEnabled, off by default) to see what was done with one message. It has no acceptance, spam or virus event, so an empty trace does not show that the server never had the message; Troubleshooting by symptom says how to turn it on and what a row says. What to include in a report If none of the above identifies it, post in General support or open an issue with: the log from the 354 (or from the delivery attempt) onwards, including the stage timing lines; the ERROR_hmailserver_<date>.log for the same window; which scanners and event scripts are enabled; and whether the message eventually arrives, arrives twice, or never arrives. Say which half you have. Read this before you post covers the rest. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email. Where to read more Wiki: Diagnosing Stalled Mail and Troubleshooting, sections 19.7 and 19.11a. Repository: DiagnosingStalledMail.md; SettingsApply.md; Sandbox.md; SUPPORT.md. The public documentation: The delivery queue and the retry schedule and Troubleshooting. On this forum: Being trusted by the servers you send to, for when the far end is deferring you; Outbound shaping and IP warm-up, for when your own server is holding mail back; Monitoring and health, for turning the metrics listener on.