Skip to content
hMailServer Community Forum
hMailServer 6.3.8 is the current release, signed 2 November 2026. Windows and Linux, and an in-place upgrade from 5.3 or later, but for the original hMailServer's 5.7.Download 6.3.8 Documentation FAQ
  • ProgressiverobotP

    Monitoring and health: Prometheus metrics, health probes, Grafana, OTLP and SNMPv3

    Pinned Monitoring, metrics & logs diagnostics administration rest-api linux windows
    1
    0 Votes
    1 Posts
    160 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.6 (6 October 2026) publishes the first container image, registry.gitlab.com/hmailserver/hmailserver:6.3.6, and a monitor that matches "The response carried no HTTP headers." needs the new sentence, which says who ended the connection, after how many bytes and seconds. See the 6.3.6 announcement. hMailServer 6.3.7 (17 October 2026) publishes registry.gitlab.com/hmailserver/hmailserver:6.3.7, tagged 6.3 and latest until 6.3.8; renames the Control Panel page that holds the metrics listener's settings from API & monitoring to Monitoring & alerts, with the alerts' settings, the outbound webhooks and the SNMP agent on it (Ctrl+K takes the old title to it); and refuses GET /api/v1/metrics/history, and the same history over COM, to a credential confined to named domains, such as a domain administrator, the auditor role given a list of domains or an API key issued for named domains. With IMAP and POP3 now taking the client's address from a PROXY protocol header as SMTP does, the High Availability Runbook says that a balancer's /readyz check must not carry that header: the metrics listener reads none and answers a request behind one 404. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) publishes registry.gitlab.com/hmailserver/hmailserver:6.3.8, also tagged 6.3 and latest. See the 6.3.8 announcement. This topic covers what hMailServer 6.3.4 offers a monitoring system: the metrics listener's Prometheus exposition and health probes, the repository's Grafana dashboard, the OpenTelemetry push, the server's own metric history, the new SNMPv3 agent and, briefly, the daily block-list check of the server's own addresses. The one thing to know first: the metrics listener is off until you give it a port, and once it is on, the health check to configure is /readyz, because it is proved by a real database round trip rather than by a flag. The metrics listener One small HTTP listener serves both the exposition and the probes. Its seven settings are [Settings] keys, set on the Control Panel's Monitoring & troubleshooting → API & monitoring page (Monitoring & alerts since 6.3.7; Windows only), with PUT /api/v1/settings/ini/{name}, with hmctl ini set, or, on a Linux server whose REST listener is not yet on, with hmailserver --set-setting NAME=VALUE; the topic on settings explains where they are stored and when a save applies. All seven are taken when the listener starts, the three credential settings included, so a saved change waits for a reinitialise (POST /api/v1/server/reinitialize, or Apply now in the Control Panel or the Deck) or a service restart. A token that has been saved and not applied leaves /metrics answering as it did before. Setting Default Effect When it is wrong MetricsServerPort 0 0 means the listener is never created No port, no probes, no metrics, and no complaint anywhere MetricsServerBindAddress 127.0.0.1 An IP literal only, parsed with inet_pton: 0.0.0.0, 127.0.0.1, :: or a specific IPv6 address A host name or localhost is rejected: the log says MetricsServer: Invalid bind address, the listener does not start and the probes go with it MetricsServerAuthToken empty Sent by the scraper as Authorization: Bearer <token> Absent on a non-loopback bind: /metrics answers 503, not 401, and the body names the settings that would open it. The probes keep answering MetricsServerAuthUsername and MetricsServerAuthPassword empty HTTP Basic; with both schemes configured, either satisfies the check As above MetricsServerCertificateFile and MetricsServerPrivateKeyFile empty Both set makes the whole port HTTPS, probes included Set but unusable: the listener still serves the probes in the clear and /metrics answers 503 :: serves both address families; a specific IPv6 literal serves IPv6 only. "Loopback" means the whole of 127.0.0.0/8 and ::1. The probes are answered before any branch that can refuse. A Kubernetes httpGet probe, an ELB or ALB health check and a Docker HEALTHCHECK have nowhere to keep a bearer token. Refused credentials on /metrics are counted in hmailserver_metrics_unauthorized_requests_total rather than logged. Nothing on the request path touches the database or the disk. A refresher thread probes the database every 5 s (every 30 s after a failure), refreshes the delivery-queue aggregates at most every 10 s and only when a scrape has asked for them, and re-reads the certificate files every 5 minutes. One exception since 6.3.4: the two message-store gauges are cached for an hour and recomputed by whichever caller finds them stale, so about once an hour a scrape can run select sum(messagesize), count(*) from hm_messages on the listener thread itself: on a large store, a slow answer to that scrape and to any probe queued behind it. It is a single accept loop serving one connection at a time. A request's header read is bounded at 5 s and a response write at 15 s. For a load balancer's check the High Availability Runbook gives an interval of 5 s and a timeout of 2 s; never point a sub-second health check at it. The three health probes No credential in any configuration; HTTP/1.0 responses that close the connection. Path 200 when Body Use it for /livez Always, if the listener answered; never 503 alive Liveness restarts. It checks nothing on purpose: a database outage must not get a healthy process killed /readyz State is Running and the database has proved within the last 20 s that it can answer; 503 otherwise ready, or not ready: <reason> Load-balancer and VIP routing /healthz The same conditions {"status":"ok\|unavailable","state":"...","database":"up\|down","uptime_seconds":N} Dashboards and human debugging Readiness is four conditions in order, each with its own reason: not ready: server not in running state; not ready: database not connected; not ready: database did not answer the last readiness probe; and not ready: database readiness probe has not completed recently. The last exists because a probe against a database that has stopped answering can hang for as long as DBConnectionAcquireTimeout allows, 60 s by default. Since 6.3.4 a server in maintenance mode answers 503 not ready: draining: POST /api/v1/server/drain enters it, GET reports {"draining","since","sessions","deliveries","drained"}, and DELETE leaves it. Start-up is a refused connection, not a 503: the listener is brought up only after the state has gone to Running. Shutdown is a 503 first, then a refusal: the state goes to Stopping before anything is torn down, so /readyz reports 503 while ShutdownDrainSeconds runs. The High Availability Runbook builds its active/passive design on this: both nodes have the listener on, with MetricsServerBindAddress=0.0.0.0 and a MetricsServerAuthToken, and the VIP is gated strictly on /readyz. The passive node's service is stopped, so its /readyz is refused and it is never routed to. Since the probes need no credential, a missing token shows as missing dashboards, not as a failed failover. Every metric family Thirty-eight metric families in 6.3.4 (thirty in 6.3.3). Counters are monotonic since server start; a restart resets them, which is what hmailserver_start_time_seconds is for. Mail flow Metric Type Meaning, and the trap in it hmailserver_processed_messages_total counter One per delivery pass started, not per message received: a message deferred three times and then delivered counts four hmailserver_messages_delivered_total counter Delivery passes that completed with no failed recipient hmailserver_messages_deferred_total counter Passes rescheduled: a temporary failure, greylisting, or a message the virus scanner could not examine (AVFailAction=1) hmailserver_messages_bounced_total counter Delivery-failure notifications queued to the sender; one can cover several recipients hmailserver_delivery_queue_messages gauge Rows in hm_messages in the delivering state hmailserver_delivery_queue_oldest_message_age_seconds gauge Age of the oldest queued message; 0 when the queue is empty. Alert on this, not on depth hmailserver_spam_messages_total, hmailserver_viruses_removed_total counter Messages classified as spam; viruses removed during delivery preprocessing hmailserver_store_bytes, hmailserver_store_messages gauge Since 6.3.4. The size and the number of the messages in hm_messages, recomputed at most once an hour hmailserver_domain_messages_received_total{domain}, hmailserver_domain_messages_sent_total{domain} counter Only present when MetricsPerDomainEnabled=1; only hosted domains are ever labelled Added in 6.3.4. hmailserver_country_blocked_total and hmailserver_country_blocked_by_country_total{country} count connections and sign-ins refused by a country policy, and hmailserver_spam_trap_hits_total counts hits on a declared spam trap. Three gauges cover the feedback loops' seven-day window: hmailserver_domain_complaints{domain}, hmailserver_domain_complaint_window_messages{domain} and hmailserver_domain_complaint_rate{domain}. They are present only when MetricsPerDomainEnabled is set and complaint processing is on. The rate gauge is a ratio, not parts per million: Complaints.md alerts on hmailserver_domain_complaint_rate > 0.001. Parts per million is the unit of the complaints.rate.high alert rule's threshold, so an expression written in parts per million against the gauge would never fire. Sessions, authentication and TLS Metric Type Meaning hmailserver_sessions{protocol} gauge Concurrent sessions; protocol is smtp, imap or pop3 hmailserver_auth_success_total, hmailserver_auth_failures_total counter A rising failure rate against a flat success rate is a password-guessing run hmailserver_tls_handshakes_total, hmailserver_tls_handshake_failures_total counter The mail protocols only; a failed handshake on the metrics port is never counted here hmailserver_tls_certificate_expiry_seconds{certificate} gauge notAfter as a Unix timestamp per configured certificate, re-read every 5 minutes; at most 128 series. Absent entirely when no certificate is configured. With no certificate row, the ACME certificate is labelled ACME (automatic) hmailserver_metrics_unauthorized_requests_total counter Always 0 when no credential is configured The server and the database Metric Type Meaning hmailserver_start_time_seconds gauge Uptime is time() - hmailserver_start_time_seconds hmailserver_build_info{version,architecture,database_schema_version} gauge Always 1; the information is in the labels hmailserver_state{state} gauge One series per state (unknown, stopped, starting, running, stopping and, since 6.3.4, draining), exactly one of which is 1 hmailserver_workqueue_depth{queue}, hmailserver_workqueue_blocking_tasks_waiting{queue} gauge Seven fixed queue values: Server queue, Maintenance queue, Asynchronous task queue, Name lookup queue, IOCPQueue, SMTP delivery queue, External fetch queue. Asynchronous task queue sends the final 250, so depth there is mail waiting to be acknowledged hmailserver_messagestore_missing_files gauge Always 0 unless MessageStoreConsistencyCheck=1 hmailserver_command_processing_seconds, hmailserver_db_query_seconds histogram Bucket bounds from 0.0001 s to 1.0 s, and from 0.001 s to 10.0 s; both emit the full _bucket, _sum and _count set hmailserver_database_connected gauge 1 when the database answered a real round trip recently hmailserver_database_probe_success_timestamp_seconds, hmailserver_database_probe_age_seconds gauge When it last answered, and how long ago on the server's own clock; /readyz reports 503 once the age passes 20 s hmailserver_db_connections{state} gauge busy and available; available pinned at zero is saturation hmailserver_db_slow_queries_total counter Statements slower than SlowQueryLogMilliseconds; 0 while that setting is 0 Scraping it scrape_configs: - job_name: hmailserver scrape_interval: 30s scrape_timeout: 10s metrics_path: /metrics authorization: credentials: "<the value of MetricsServerAuthToken>" static_configs: - targets: ["mail.example.com:9100"] Use scheme: https when the certificate pair is set, and keep scrape_timeout well under scrape_interval. The wiki gives nine alerting rules against these metrics, among them hmailserver_database_connected == 0 for 2m, hmailserver_delivery_queue_oldest_message_age_seconds > 3600 for 15m and hmailserver_tls_certificate_expiry_seconds - time() < 14 * 86400. The Grafana dashboard The repository ships hmailserver/docs/grafana-dashboard.json. Import it as it is and pick your Prometheus data source; the instance variable, from label_values(hmailserver_build_info, instance), selects the server. Its uid is hmailserver-overview; it refreshes every 1m and opens on now-24h. Its panels array has 19 entries: four row headers, five stat panels and ten time series. Every rate is over [5m] and every query is filtered by instance=~"$instance". Health: server state (hmailserver_state == 1), database, uptime, missing message files, and TLS certificate expiry, amber below 21 days and red below 7. Mail flow: delivery outcomes (delivered, deferred and bounced, stacked), queue depth with the oldest message's age on a right-hand axis, messages processed with spam and viruses removed, and active sessions by protocol. Authentication and TLS: authentication outcomes and TLS handshakes, as rates. Latency and database: command processing and database query latency (p50, p95 and p99 by histogram_quantile over the two histograms' _bucket series), the connection pool by state, and slow queries. What the dashboard does not draw: the two database probe gauges, the two work-queue gauges, hmailserver_metrics_unauthorized_requests_total, the per-domain counters, the two store gauges and the six families added in 6.3.4. hmailserver_build_info is read by the instance variable and by no panel. The metric history inside the server The server keeps a short history of its own, so the Control Panel's dashboard can show 24 hours, 7 days and 30 days without Prometheus. MetricsHistoryDays (default 7; 0 switches it off) is the days kept in hm_metricsamples: one row per metric per minute, plus a sample at service start. Fourteen metrics are stored in 6.3.3, named as the exposition's without the hmailserver_ prefix and with the label folded in, such as sessions_smtp and processed_messages_total. 6.3.4 adds store_bytes and store_messages, whose history begins the day the server first samples them, and country_blocked_total and spam_trap_hits_total. Read it with GET /api/v1/metrics/history?metric=…&range=24h|7d|30d, or on Windows over COM with Utilities.GetMetricHistory. It has no labels and no histograms and is not a replacement for Prometheus, but it survives a restart. Pushing to an OpenTelemetry collector Three OTLP/HTTP signals, each a no-op until its own endpoint is set. The pushed metric names are the Prometheus names verbatim. Setting Default Effect OtelMetricsEndpoint empty OTLP metrics; the default path is /v1/metrics OtelMetricsInterval 60 Seconds between exports, clamped to 5–3600 OtelEndpoint empty OTLP traces only: one span per delivery attempt, joined to the trace named by the message's own traceparent; default path /v1/traces OtelLogsEndpoint empty OTLP logs: the same lines, categories and mask the log files get; default path /v1/logs What is pushed: the twelve counters that predate 6.3.4 and, since 6.3.4, hmailserver_spam_trap_hits_total, sessions as one gauge with a protocol attribute, hmailserver_messagestore_missing_files, and the per-domain counters when they are enabled. The histograms, the database and queue gauges and certificate expiry are scrape-only. On Linux, in every release from 6.3.0 to 6.3.3, a collector that accepted the connection and never answered held the exporter, and the stop that waits for it, for good; 6.3.4 fixes that. The SNMPv3 agent (since 6.3.4) For a monitoring system that speaks SNMP and not Prometheus or OpenTelemetry, 6.3.4 adds a read-only agent answering the mail MIB (RFC 2789, MTA-MIB) and the standard SNMP groups from the same counters /metrics serves; an object /metrics has no counter for is answered noSuchObject, not made up. It speaks SNMPv3 with the User-based Security Model for one user at authPriv only (HMAC-SHA-256 and AES-128), and SNMPv2c as an option that stays off even when the agent is on. SNMPv1 is not answered, a Set is answered noAccess, and nothing is sent unasked: no traps, no informs. It is off as shipped, and an empty allow list answers nobody. Setting (hm_settings) REST key Default Meaning SnmpEnabled snmp_enabled off The whole agent; off, no socket is opened SnmpPort snmp_port 161 The UDP port SnmpBindAddress snmp_bind_address empty, every address 127.0.0.1 for a collector on the same machine; otherwise an address on a management network SnmpAllowedAddresses snmp_allowed_addresses empty Addresses and CIDR ranges, at most 64. Name the managers one by one: UDP does not prove a source address SnmpV3User snmp_v3_user empty The one SNMPv3 user SnmpV3AuthPassword, SnmpV3PrivPassword snmp_v3_auth_password, snmp_v3_priv_password (write-only) empty 8 to 64 ASCII characters each SnmpV2cEnabled, SnmpV2cCommunity snmp_v2c_enabled, snmp_v2c_community (write-only) off, empty The community crosses the network in the clear; an answer is never more than eight times its request SnmpContact, SnmpLocation snmp_contact, snmp_location empty sysContact and sysLocation They are on /api/v1/settings (the server group), on the Control Deck's Settings page under Server and in hmctl settings. There is no COM property for any of them, and in 6.3.4 no Control Panel page, so Windows and Linux are configured the same way; since 6.3.7 they are the SNMP tab of the Control Panel's Monitoring & alerts page on Windows, which 6.3.5 and 6.3.6 had on Logging. They can also be given as HM_PROPERTY_SnmpEnabled and so on in the environment, which reaches the database only when hmailserver --apply-environment is run (hMailServer.exe --apply-environment on Windows): the service does not read the environment itself. Each setting is read when the agent starts, at the next service start or Reinitialize. The two passwords and the community are stored protected, never read back, and not carried by a backup: enter them again after a restore. Snmp.md's example keeps every password off the command line. On the server, for a manager at 192.0.2.10: $ umask 077 $ cat > snmp.json {"snmp_allowed_addresses": "192.0.2.10", "snmp_v3_user": "monitor", "snmp_v3_auth_password": "...", "snmp_v3_priv_password": "...", "snmp_contact": "[email protected]", "snmp_location": "Rack 4, London", "snmp_enabled": true} ^D $ hmctl settings set --from-json snmp.json $ rm snmp.json $ hmctl api POST /api/v1/server/reinitialize On the manager, with Net-SNMP 5.9 or later, defVersion 3, the user (defSecurityName), both passphrases, defSecurityLevel authPriv, defAuthType SHA-256 and defPrivType AES go in ~/.snmp/snmp.conf, readable by you alone (chmod 600); snmpwalk mail.example.com 1.3.6.1.2.1.28 then walks the mail MIB. What it serves. In the MTA-MIB mtaGroupTable has one row per protocol the server runs: 1 SMTP, 2 POP3, 3 IMAP. What has a counter is answered: mtaStoredMessages is the delivery queue's depth, mtaTransmittedMessages is hmailserver_messages_delivered_total, mtaGroupOldestMessageStored is the oldest queued message's age, and mtaGroupInboundAssociations is hmailserver_sessions for the protocol. mtaReceivedMessages is noSuchObject, because no counter of messages received exists, and so are the volume, recipient and conversion columns. Sixteen scalars at 1.3.6.1.4.1.32473.1.2.<n>.0 carry /metrics series directly, among them hmailserver_state as a number at 14 (3 is running, 5 draining) and the two delivery-queue gauges at 15 and 16. The enterprise number is a placeholder. 1.3.6.1.4.1.32473 is the number RFC 5612 reserves for documentation and examples; it is not registered to this project with IANA. When a number is registered, sysObjectID, the scalars and the engine ID's prefix move to it in one release, and a manager configured for these OIDs will need them changed. There is no MIB module file yet, for the same reason. When it does not start. Error 7420: the port could not be bound, because another program holds it (the operating system's own SNMP service, on Windows or Linux, listens on 161 when it is installed) or, on Linux outside the packaged unit, the process lacks CAP_NET_BIND_SERVICE; the server runs on without the agent. 7421: the engine's seed or boot count in hm_settings could not be read or written. 7422: an exception left the agent's thread; a Reinitialize starts it again. On Linux the packaged hmailserver.service has CAP_NET_BIND_SERVICE for port 25, which is what lets it bind 161. No container image of 6.3.4 is published; a container built from the repository's Dockerfile publishes the port as UDP (-p 161:161/udp) and needs a host name of its own, since two started with one fixed hostname are one SNMP engine. The daily block-list check, briefly Since 6.3.4 the server looks the addresses it sends from up on DNS block lists once a day and raises the reputation.listed alert when one is listed. The watch is on in a new installation and off after an upgrade, and is read at GET /api/v1/reputation; the topic on DKIM rotation, reports and the reputation watch has the lists, the addresses checked and what a failed lookup means. Where to read more Wiki: Monitoring and Health and High Availability Runbook Documents at v6.3.4: Snmp.md, HighAvailabilityRunbook.md, ReputationWatch.md, Complaints.md and the dashboard itself, grafana-dashboard.json The public documentation: 16.3 Prometheus metrics and health probes On this forum: Alerts, the audit trail, syslog and the journal, and per-domain reports, Reading the logs, and Warm standby and high availability for the pair the probes gate. See also The Control Deck and hMailServer on Linux: what you get and what you do not
  • ProgressiverobotP

    DKIM rotation, DMARC and TLS reports, complaints, bounces and the reputation watch

    TLS, certificates & deliverability dkim dmarc deliverability dns diagnostics
    1
    0 Votes
    1 Posts
    119 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) no longer signs mail from outside whose From names a domain hosted here with that domain's DKIM key, and signs an authenticated submission as another hosted domain only when its account may send as the address its From names (CVE-2026-107567); the upgrade stops the signing, and only rotating the domain's key, as this topic describes, ends the signatures already made. See the 6.3.7 announcement. Five things new in 6.3.4 keep your signing keys fresh or read how the world is answering the mail your domains send: DKIM key rotation, the DMARC and TLS reports your domains receive, the providers' complaint feedback loops, bounce classification with a suppression list, and a daily check of your own addresses against block lists. The one thing to know first: each is its own switch and each is off until you turn it on - the reputation watch is the exception, on in a new installation and off after an upgrade - and only one of them, the suppression list, ever stops a message. The DKIM and DMARC records themselves are in Being trusted by the servers you send to. The Control Panel and the COM API named below are Windows-only. The Control Deck, the REST API and hmctl work against a Windows or a Linux server; the Linux packages install hmctl, and the Windows installer does not carry it. DKIM keys that rotate Schema 6048. A rotation policy per domain does, on a schedule or on Rotate now, what used to be done by hand. It is off on every domain until switched on, and it never edits DNS: it says what to publish, checks it, and waits. The walk. Three slots, each a selector with a key file: current (the only key that signs), next (published and waiting; it never signs before the switch) and retiring (kept for mail still on its way). The schedule starts a rotation when the current key is older than the interval, or somebody presses Rotate now. An RSA key is generated under a new selector: the one you give, or s and today's UTC date (s20260915, then s20260915b if that is taken). You publish a TXT record at <selector>._domainkey.<domain> with the value v=DKIM1; k=rsa; p=.... Every pass - every fifteen minutes, and on Check DNS now - looks the record up. The first pass that finds exactly the new key records the moment. Any pass that does not find it matching sends that moment back to zero. Signing moves once the record has matched for the whole propagation period and the retiring slot is free. The old selector's record stays published for the retention period. The old selector is retired. A key file the rotation generated is deleted; one it did not generate is left. The status lists the record with the action remove, and removing it is yours. Signing itself is still the domain's DKIM signing switch: the rotation moves keys, it does not switch signing on. Setting REST key Default Range Rotate on a schedule dkim_rotation_enabled off Days between rotations dkim_rotation_interval_days 180 1 to 3650 Propagation period, hours dkim_rotation_propagation_hours 72 1 to 720 Retention period, days dkim_rotation_retention_days 7 1 to 365 Key size, bits dkim_rotation_key_bits 2048 2048, 3072 or 4096 Every interface refuses a value out of range rather than clamping it. Set them in the Control Panel's domain dialog (DKIM tab), the Deck's domain editor, PUT /api/v1/domains/{domain}, or hmctl domain update example.com --set dkim_rotation_enabled=true. Nothing is written to hMailServer.ini. What the status says. GET /api/v1/domains/{domain}/dkim/rotation answers the phase (off, idle, awaiting_dns, propagating, ready, waiting_for_retiring_slot) and each slot's last check: unchecked, matches, not_found, different, lookup_failed or key_unreadable. different means a record exists and none carries this key, and waiting will not fix it. A match proves that this server's resolver sees the key, not that every receiver's does. Where the keys are, and who can read them. <DataDirectory>/DKIM, as <selector>._domainkey.<domain>.pem. Windows Linux Default directory C:\Program Files\hMailServer\Data\DKIM /var/lib/hmailserver/DKIM Protection A protected security descriptor naming only SYSTEM, the Administrators group and the service account; it does not inherit the data directory's permissions Created with mode 0600, never widened On Windows, a Control Panel that is not elevated cannot read a generated key; the records it shows come from the status document. A managed domain refuses a blind save. While rotation is on, or a next or retiring selector is in flight, a save that would change the selector or key file is refused: over COM with an error, over REST with 409. When it goes wrong. The dkim.rotation.stalled alert, on by default and held for the daily digest, is raised when a next key's record has not been seen for the rule's threshold (7 days by default) or the last attempt to move a running rotation failed. A record that is missing or different is never an error. The error log is for faults, codes 6610 to 6615. A key that has been compromised. If an earlier old selector is still being kept, Retire the old key now first. Rotate now, and publish the new record at once. Check DNS now until the record matches, then Switch now with the box that skips the propagation period (hmctl dkim switch example.com --force). Force skips the period and never the check. Retire the old key now again, which deletes the compromised key's file if the rotation made it. Do not remove the old record. Publish it with an empty key, v=DKIM1; p=, which says the key is revoked rather than missing. Backups. A backup that includes the data directory includes the key files. Since 6.3.4 an archive can be encrypted (7-Zip AES-256, headers included), but encryption is off until a backup passphrase is set, and until then the archive holds the private keys in the clear: treat it as you treat the keys. Set it on the Control Panel's Backup & restore → Encryption, the Deck's Backup view, PUT /api/v1/settings/backup (encryption_passphrase) or hmctl backup passphrase set. Lose the passphrase and everything in every archive written with it is lost. Not done: generating Ed25519 keys; signing twice during the switch; publishing a revocation. The reports your domains receive Schema 6070. Receivers send a DMARC aggregate report once a day, and senders that honour TLS-RPT report on their sessions to your MX hosts. Name the mailbox. Each domain has one report mailbox: the Reports received tab of the Control Panel's domain dialog, the Reports the domain receives group of the Deck's domain editor, or report_mailbox on PUT /api/v1/domains/{domain}. It is empty by default, and empty means nothing is read. The address has to be in a domain this server hosts, and is refused otherwise. Then publish it: _dmarc.example.com. TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]" _smtp._tls.example.com. TXT "v=TLSRPTv1; rua=mailto:[email protected]" Keep the p= your domain already publishes: this step adds only the rua= address. The wiki's DNS page says to start at p=none and tighten only once the reports show all your legitimate mail passes SPF and DKIM. What is read. A message is opened only when it is addressed to a report mailbox and to nothing else of this server's, and it is delivered to the mailbox unchanged, read or not. Each candidate part is unwrapped by its bytes (gzip, zip or plain), and the first character decides: < is a DMARC aggregate report, { a TLS-RPT report. The same report twice is one row. Nothing read here acts. No list is fed, no mail is refused, no setting changes. What carried a report is kept too. Nothing inside a report proves who sent it, so the dmarc= verdict and header.from= domain this server wrote on the carrying message are kept on the row (message_dmarc, message_from). Both are empty unless AuthenticationResultsEnabled is on; with it off, the reporter is shown as Not checked. The bounds. A message is not read past 32 MB, and an attachment is not inflated past 16 MB. A report keeps at most 2,000 records, and the log says when the chart under-counts. A domain takes in at most 20,000 records and 1,000 TLS-RPT rows in any twenty-four hours; a report past either ceiling is refused whole. The chart. GET /api/v1/domains/{domain}/dmarc-reports/summary?days=30, drawn by the Deck's DMARC page and hmctl dmarc summary: messages passed and failed, what receivers quarantined and rejected, the reporters, and up to 200 sources that sent as the domain. DomainReportRetentionDays (365 by default; 0 keeps for ever) decides how long a report is kept. The rows are not in the backup; the reports are still in the mailbox, which is. Not done: no reverse DNS or geography for a source; no forensic (ruf=) reports; no alert when the pass rate falls. The chart is how you see whether it is safe to move from p=none to p=quarantine to p=reject. Complaints: the feedback loops Schema 6055. The large mailbox providers post an abuse report (RFC 5965) to an address you registered with them whenever a reader presses this is spam. With processing on and a mailbox named, a report is recorded against the sending domain and account, counted, and past a threshold puts the address on the domain's suppression list. It is off by default, and inert without a mailbox. Why only a configured mailbox is an intake. A feedback report carries nothing known only to the provider. What tells a genuine report from a forged one is that it arrived where the provider was told to send it. So use a dedicated address the loops know and nobody else does - [email protected] rather than [email protected] - and do not publish it. Setting REST key Default Meaning ComplaintProcessingEnabled processing_enabled off The whole feature ComplaintMailbox mailbox empty The address the loops are registered against, or several separated by commas. Each must be one this server hosts ComplaintSuppressionThreshold suppression_threshold 0 0 to 100. 0 records and counts every report and lists nobody. Raise it to 1 once the entries have been read ComplaintRetentionDays retention_days 180 0 keeps for ever; at most 3650. It does not remove a suppression entry a complaint made They are keys of /api/v1/settings/complaints, a group on the Deck's Settings page, and hmctl settings get complaints. Three gates that are not settings. The report must be addressed to the complaint mailbox and to nothing else. The domain it is about must be one this server hosts. And an Original-Rcpt-To that this server hosts lists nobody. A complaint is not a bounce. It has a counter of its own, hm_bouncecounts.bccomplaints. Its list entry carries source complaint, and the refusal a sender meets uses 5.7.1, never 5.1.1. not-spam is recorded and counts towards neither the rate nor the list; every other feedback type counts. It is the same suppression list the bounces feed, and the list is applied whenever either feature is on. The rate is complaints divided by messages delivered, over a seven-day window that is not a setting, in parts per million. A domain that has sent fewer than 100 messages in the window is not judged. The complaints.rate.high alert ships on, seeded at 1000, with a cooldown of a day, held for the digest. With MetricsPerDomainEnabled set, the gauge hmailserver_domain_complaint_rate{domain} is exposed with two others. Reading them. The Deck's domain row has a Complaints button; GET /api/v1/domains/{domain}/complaints needs view_configuration and view_message_trace; hmctl complaint list --domain example.com and hmctl complaints rates do the same from a shell. A backup does not carry the complaints. Bounces read, and a suppression list Schema 6049. With bounce processing on, every refusal met while delivering and every RFC 3464 delivery report that arrives later is classified against its enhanced status code and counted per recipient and per sending domain. An address that hard-bounces as many times in a row as the threshold says goes on that domain's suppression list. It is off by default. Setting REST key Default BounceProcessingEnabled processing_enabled off BounceHardBounceThreshold hard_bounce_threshold 1 (1 to 100) BounceMailboxFullIsHard mailbox_full_is_hard off BounceSuppressionAction suppression_action 0 refuse; 1 drop; 2 observe BounceSuppressionExpiryDays suppression_expiry_days 0, entries kept until removed BounceReadDeliveryReports read_delivery_reports on BounceCountRetentionDays count_retention_days 90 BounceReportWindowDays report_window_days 7 (1 to 30) They are on the Control Panel's Delivery of e-mail page (the Bounces and the suppression list card), the Deck's Settings page, /api/v1/settings/bounces and hmctl settings get bounces. The safe way to start is suppression_action=observe for a week. The server classifies, counts and lists, delivers everything, and logs every message that would have been stopped. Read the list, then switch to refuse. What counts as which. Code Category Counts towards the list 4.2.2 or 5.2.2 mailbox_full only with BounceMailboxFullIsHard any other 4.x.x soft no 5.7.x policy never 5.1.1, 5.1.2, 5.1.3, 5.1.6, 5.1.10, 5.2.1 hard yes any other 5.x.x other no A bare 550 is not a dead address. The classifier never believes a code this server worked out for itself. When a remote's reply carries no enhanced code of its own, the words decide, policy words first: 550 Recipient address rejected: Access denied is policy, 550 No such user here is hard, and 550 <[email protected]> is other. hmctl bounces classify "550 5.1.1 User unknown" shows how a reply would be read, and changes nothing. A delivery report has to prove itself. It is counted only when it passes six checks, among them: its envelope sender is null, it was not classified as spam, it arrived over SMTP here and not from a signed-in user, and it reproduces a row of hm_bouncereportkeys, a SHA-256 of the Message-ID, recipient and envelope sender of a message this server really sent. What that does not stop: one recipient in To or Cc could forge a report for another visible recipient. Such an entry says source: report, and read_delivery_reports can be turned off. What the list does. Action At RCPT TO and on the webmail At delivery refuse Refused with 550 and the status that listed the address, with a sentence saying why The recipient is removed and returned to the sender in an ordinary non-delivery report drop Accepted Removed silently; a log line says so observe Accepted Delivered; a log line says it would have been stopped Mail already in the queue meets the list at its next attempt. If the list cannot be read, the message is sent as if the address were not listed, and HM6622 is reported at most once an hour. Looking at it. Control Panel → Monitoring & troubleshooting → Suppression list; Deck → Domains → Suppressions ›; GET /api/v1/domains/{domain}/suppressions; hmctl suppression list|create|delete --domain. The list's document says processing_enabled when bounces are read and list_applies when the entries act. Removing an entry lets the next message through at once and starts the run of hard bounces again from none. The reputation watch Schema 6057. Once a day the server asks the block lists about the addresses it sends from, and raises reputation.listed when one is listed. On in a new installation, off after an upgrade: an upgraded server may already have a delivery bind address, and switching this on beneath it would send that address to four third parties every day without anybody choosing it. Turn it on with reputation_watch_enabled, on the Deck's Settings page, PUT /api/v1/settings or hmctl settings set; switching it off clears a standing reputation.listed. Which addresses. SMTPDeliveryBindToIP, the bind to address action of any active global rule, and ReputationWatchAddresses, which exists for a server behind NAT. A private, loopback, link-local or shared address is never looked up. At most 32 addresses are checked; one past that is reported as too_many. Which lists. ReputationWatchLists ships with ipv4:zen.spamhaus.org, ipv4:bl.spamcop.net, ipv4:psbl.surriel.com and ipv4:bl.mailspike.net. Nothing shipped covers IPv6, on purpose: an IPv6 outbound address reads not_applicable against all four until you name a zone whose IPv6 coverage you have read. A subscriber key put in this setting is readable by everybody who can read the configuration. A refusal is not a listing, and neither is clean. Before any address is looked up, each list is asked about 127.0.0.2, which must be listed, and 127.0.0.1, which must not be. What comes back Status An answer in 127.255.255.0/24 refused The lookup failed lookup_failed 127.0.0.2 is not listed list_not_answering 127.0.0.1 is listed list_lists_everything An answer outside 127.0.0.0/8 unexpected_answer If DNSServer is one of the large public resolvers, Spamhaus will be refused every day, and the report says so rather than calling the server clean. Point DNSServer at a resolver of your own. A lookup that fails never clears a listing. After a week with no definite answer a pair becomes unknown, not clean: hmctl still exits 1, and the report answers listed as null. Where and when. GET /api/v1/reputation; POST /api/v1/reputation/check, answered 409 while a check is running; the Deck's Outbound reputation view; hmctl reputation status. A check is bounded to 120 seconds; pairs it does not reach are written as lookup_failed. The alert is sent at once rather than held for the digest. Not done: delisting; domain-based lists; watching between checks; seeing behind a smart host. Where to read more Operator documents at the v6.3.4 tag: DkimKeyRotation.md, DmarcReports.md, Complaints.md, BounceSuppression.md and ReputationWatch.md. Wiki: DNS for Your Domain, Outbound Shaping and Monitoring and Health. Public documentation: 8.5 DKIM and 8.6 DMARC. On this forum: Being trusted by the servers you send to, The Control Deck, Requiring TLS for the reports your server sends, and Alerts, the audit trail, syslog for how an alert reaches you.
  • ProgressiverobotP

    Relocating an installation to another drive, and moving or renaming a mailbox

    Guides & tutorials migration windows administration database diagnostics
    1
    0 Votes
    1 Posts
    120 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) bears on relocating a Windows installation to another drive. hMailServer.exe /ProtectFolders, which setup runs at every installation and upgrade, now hardens the program folder, Bin and ControlPanel: where hMailServer was installed in a folder that took its permissions from the root of a drive, any account that could sign in to the computer could replace the programs the service runs (CVE-2026-107571); the default folder under Program Files was never affected. The procedure below copies the tree and re-registers it without running setup, and a folder below a drive's root takes the permissions the root hands down, so an installation relocated there is hardened by the upgrade to 6.3.7, or on 6.3.7 by running hMailServer.exe /ProtectFolders as an administrator, not by the move (on 6.3.6 or earlier follow the icacls steps in ServiceAccount.md, since those versions' /ProtectFolders leaves the three program folders as they are); before upgrading an installation at a drive's root, follow the steps "Before upgrading an installation at a drive's root" in ServiceAccount.md. Where accounts that are not administrators could sign in to the computer while the installation was in such a folder, the release says to reinstall the programs into a new folder that only the administrators may change, not over the installation, with setup given the new folder by /DIR=: over an installation setup installs into the folder it finds, and a 32-bit installation's uninstall entry overrides /DIR= until it is taken away, which the "re-run the installer" repair below meets. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) has setup make or harden the program folder, Bin and ControlPanel before it copies a file or runs a program of the installation. One of them that another account could change, as a folder below a drive's root can be, setup hardens and then refuses, with exit code 7, until Windows has restarted; and a folder or a link on the way to them, or a program, library, script or settings file in them, that another account than SYSTEM, the Administrators and TrustedInstaller controls, it refuses with exit code 7, naming the icacls that deals with it and a restart of Windows before setup runs again. On Windows 6.3.8 may also refuse, with HM7534 or HM7536, TlsCaFile, PostgreSQLSslRootCert, a TLS certificate or private key file, OAuth2PublicKeyFile or AcmeCertificateDirectory where its path holds characters outside ASCII, so give the new root, and the folders above it, an ASCII name. See the 6.3.8 announcement. This guide covers two different moves. The first is relocating a configured installation to another directory or drive on the same machine, for example from C:\Program Files (x86)\hMailServer to C:\Program Files\hMailServer. The second is renaming a mailbox or moving it to another domain, which is an operation of its own since 6.3.4. Moving to new hardware is a third thing, and Moving to new hardware, and why Windows to Linux is a rebuild covers it. The one thing to know first: relocating is not one change. The installation path is recorded in the registry, in the service, in the COM registration, in hMailServer.ini and in a handful of database columns, and since 6.3.4 a [Settings] path edited in the file is ignored in favour of the value stored in the database. The relocation runbook is written in Windows terms: the registry, the service and the COM registration. The documents used here give no separate procedure for a Linux installation. Installing somewhere new is a different, solved problem: /DIR="<path>" on the installer puts a fresh installation wherever you want. The good news first The mail store itself is already relocatable. Message rows hold a bare {guid}.eml with no directory, and the path is rebuilt at every read against the data directory. Moving the data directory and pointing DataFolder at its new home moves every message with it. The exception is a database that predates hMailServer 5.4, where the column can hold a full path. Before you touch anything Take a backup and verify it restores (Backup and restore). Everything below is reversible only while the old tree and a good backup both exist. Ask the server whether its message rows are relative. In the Control Panel, Monitoring & troubleshooting → Diagnostics → Test message file locations. If it reports ERROR: Full paths are stored in the database, the move is still possible, but converting the paths becomes mandatory rather than optional. Write down what you are about to strand. The same Diagnostics page has an Installation paths section that lists the registry root, every [Directories] key and every path-shaped setting the running server holds, with the ones that do not exist marked. It changes nothing. The inventory Where What Repaired by Bin\hMailServer.ini, [Directories] ProgramFolder, DataFolder, DatabaseFolder, LogFolder, TempFolder, EventFolder Editing the file Bin\hMailServer.ini, [Database] PostgreSQLSslRootCert and TlsCaFile Editing the file. Leave the connection keys alone: they name a server, not a path The settings store (hm_inisettings) ArchiveDir, AcmeCertificateDirectory, OAuth2PublicKeyFile, UpdateTrustRootsFile, UpdateLogPublicKeyFile, and the certificate and private key files of the REST API, metrics and web services listeners Storing the new value, not editing the file (since 6.3.4) Registry, HKLM\SOFTWARE\hMailServer in the 32-bit view InstallLocation By hand Registry: the service's ImagePath, LocalServer32 (79 keys), InprocServer32 (15), the TypeLib values hMailServer.exe /Register Registry, HKLM\...\Uninstall\hMailServer_is1 InstallLocation, UninstallString, DisplayIcon Nothing; re-run the installer The database Certificate and key files, DKIM key files, the client-certificate CA bundle, backupdestination, the ClamWin and custom scanner paths, rule actions Each on its own page What breaks when a key is left behind. With a stale DataFolder the store appears empty. With a stale LogFolder logs are written to the old tree, silently. With a stale EventFolder event scripts are not found and events stop running. The 32-bit view is not optional. The server reads InstallLocation through KEY_WOW64_32KEY, and the installer writes it there. Writing the 64-bit view changes nothing and looks like it worked. The procedure Stop the service, and confirm it is stopped. Nothing here may run against a live server. Copy the tree. Copy, do not move: the old tree is the rollback. Include Bin\hMailServerApiKeys.ini if it exists, since the REST API key store lives there and no installer manifest mentions it. Grant the service account access to the new tree as 6.3.8's RelocatingAnInstallation.md does: a copy takes the permissions of the folder it lands in (under Program Files every user's Read on the new Data, Logs, Temp, Database, Events and hMailServer.INI) and not the old tree's entries for NT SERVICE\hMailServer, so the document makes the new root with the permissions setup gives a program folder before the copy, runs hMailServer.exe /ProtectFolders from the new Bin after step 5's /Register, and grants a service that runs as anything but LocalSystem its folders again with /GrantServiceAccess. A server that cannot write its log directory starts and then says nothing. Rewrite the file keys in <new root>\Bin\hMailServer.ini whose value was under the old root: the [Directories] folders, and [Database] PostgreSQLSslRootCert and TlsCaFile, the second named beside the first by 6.3.8's document, since with TlsMode=verify-full a CA file left in the old tree stops the server reaching its database once that tree is deleted (HM7531). Nothing else, and not a folder that was outside the old tree and has not moved: a DataFolder pointed at an empty directory makes the store appear empty. Repoint the registry root, from an elevated prompt: reg add "HKLM\SOFTWARE\hMailServer" /v InstallLocation /t REG_SZ /d "<new root>" /reg:32 /f. /reg:32 is the whole of the point. Re-register the server, from an elevated prompt, from the new Bin: hMailServer.exe /Register. It registers the path of the executable that runs it. Start it, and check what it read. Read the application log, and since 6.3.4 the ERROR log too. Confirm that the Diagnostics page's Installation paths shows the new paths (6.3.8's document names it; the Control Panel has no Settings → Directories page), that your domains are listed, and that the log file you are reading is under the new log directory. Convert full message paths, only if the diagnostic found any: run the Data Directory Synchronizer, in its import mode, on the old installation before step 2 copies anything, as step 0 of 6.3.8's RelocatingAnInstallation.md has it, not against the new data directory after the move: it converts only a row whose full path is where the file is found, so after the move it would import every such message a second time, and its other mode would delete the copies. A hand-written UPDATE hm_messages SET ... is not a supported converter. Store the [Settings] paths and fix the database-borne paths (both below). Verify before deleting anything: send and receive a message; complete a TLS handshake on every port that has a certificate; run a backup to completion; run an event script, if you use them; check the log directory is being written. Keep the old tree, unreachable by the service, for at least one full backup cycle. Do not make it unreachable until the [Settings] paths are stored and a reinitialise has picked up the new certificate and ACME paths. The failure this prevents. A server whose anchors are stale reports Running and serves an empty configuration. That failure is silent. The [Settings] paths: 6.3.3 and earlier, and since 6.3.4 On 6.3.3 and earlier these are [Settings] keys of hMailServer.ini like any other, and editing the file moves them. Since 6.3.4 it does not: the stored value is used, the edit is named in the error log as HM5804, and, where the service can write the file, the line is put back to the old path. RelocatingAnInstallation.md and the wiki page between them give three ways to store the new values. The first is the document's own, and the one to use: When How Where it is described After the moved server is running In the Control Panel under Settings, with Settings.SetIniSetting over COM, PUT /api/v1/settings/ini/{name}, or hmctl ini set <name> <value> RelocatingAnInstallation.md, step 9, and the wiki Before stopping the old server The same four ways, as the last thing before the service is stopped. A save applies ArchiveDir, OAuth2PublicKeyFile, UpdateTrustRootsFile and UpdateLogPublicKeyFile at once, not at the next start as the wiki page has it, so the running old server turns to a tree not yet copied: archiving stops, and token validation and update verification refuse. The database then holds the new paths, so going back to the old tree means storing the old values again The wiki, corrected by SettingsApply.md At the first start, if the service is already stopped A [SettingsOverride] section in the new hMailServer.ini, named in the error log as HM5807; then store the same values and remove the section before the next restart The wiki A save applies every one of these paths at once except the listeners' certificate paths and the ACME certificate directory, which are bound when the servers start and wait for a reinitialise (Apply now, or POST /api/v1/server/reinitialize). Until then the server runs on the old paths, which still resolve because the tree was copied rather than moved. PasswordPepper is the exception to all of this: it lives in the file only, is not a path, and comes across with the copied file. The database-borne paths SSL certificates. Since 6.3.7 the Control Panel's SSL certificates page has Edit…, which changes a certificate's name and files and keeps its id, so every port that presents it keeps it and reads the new files at the next reinitialise; deleting a certificate and adding it again, as the document had it until 6.3.8, leaves each port that presented it with none, and such a port does not start (HM5113). It can also be set over COM. DKIM keys. Domain → DKIM, the primary and, if used, the secondary. A domain whose key rotation manages its keys refuses that edit. Client-certificate CA bundle, per port. A port set to require a client certificate refuses every client when its bundle cannot be loaded. Archive index. hm_archiveindex.archivepath is matched on the exact string by retention, legal hold and the address eraser. Do not bulk-rewrite it. If the archive directory moved, move the archive back, or accept that the rows written before the move point at the old location; there is no supported converter for this column. What stays broken, and why Add/Remove Programs still points at the old unins000.exe. Do not edit unins000.dat. The supported repair is to re-run the installer at the new root. Start-menu shortcuts hold absolute targets and nothing repairs them. Message content. The old path appears inside messages the server itself wrote, such as a delivery report. Never rewrite message content. Renaming a mailbox, or moving it to another domain New in 6.3.4 (schema 6056, whose table hm_mailboxmoves is the move's journal). new_address carries the domain, so renaming and moving are the same operation: [email protected] to [email protected] is a rename, [email protected] to [email protected] is a move. A move keeps the account's row, so everything keyed on the account id follows without being touched: rules, folders with their subscriptions and access-control entries, application passwords, the second factor, contacts, calendars and the role assignment. Copying the mail into a second account loses all of it. Interface How Control Deck The Move or rename card in the account editor hmctl hmctl mailbox status [email protected], then hmctl mailbox move [email protected] [email protected] REST POST /api/v1/accounts/ann%40example.com/move with {"new_address": "[email protected]", "keep_alias": true}; GET on the same path says whether anything holds the mailbox PowerShell Move-HmMailbox -Address [email protected] -NewAddress [email protected] Control Panel (Windows) Move or rename… on the account's General tab COM (Windows) Account.MoveTo(NewAddress, KeepOldAddressAsAlias) or Utilities.MoveMailbox(Address, NewAddress, KeepOldAddressAsAlias) The old address is kept as an alias by default, pointing at the new address. Turn it off (keep_alias false, --drop-alias) only when the address is to stop working: mail to it is then refused at RCPT TO. What moves: the message store directory <data>/<domain>/<local part>; the Sieve tree; aliases in any domain that point at the account; distribution list memberships; recipients of mail still in the delivery queue; and quarantined mail, up to a ceiling of 200,000 rows. What does not move, on purpose. The archive and the message trace stay under the old address, as a record of the past. Erasure works by address, so erasing the new address will not reach the archive written under the old one: erase both (Compliance: personal data and erasure). The domains a role reaches are not rewritten either, so check a role after a cross-domain move (Administrator roles end to end). What is refused, before the first byte is moved: a domain this server does not hold, or one that is not active; a destination domain at its maximum number of accounts, or that would exceed its maximum total size; an account, an alias or a distribution list already at the new address; a mailbox encrypted at rest moving to another domain (409); a rename inside the domain is not refused; an installation whose message file names are still full paths; a credential that may not administer accounts in both domains; a mailbox in use. In use means a POP3 session, or an IMAP session with one of the account's folders selected. What is not counted: an IMAP session that has selected nothing, a webmail request in flight, a ManageSieve session and an SMTP delivery to the mailbox in flight. A message that arrives during the copy is written under the old directory, and the old directory is swept once afterwards; anything that could not be carried is HM6948. That is why a move of a large busy mailbox belongs out of hours. When a move fails or is interrupted There is no transaction across a filesystem and four database backends, so the order is chosen to make each step undoable: the store, then the account row, then the references, then the alias. A journal row is opened before the first step and closed after the last, and an interrupted move is said once per start as HM6947 in the application log, naming both addresses and the stage it stopped at. Stage What it means started Nothing has been changed moving the store The copy was under way; look at both directories. The database has not been touched store moved The directories are at the new address. While the account row says the old one, the account authenticates and its mail is unreadable account written The store and the account row agree; the references have not all been rewritten references rewritten Everything but the alias complete The move finished rolled back A step failed and everything before it was put back rollback failed Something could not be put back; HM6940 is in the application log To finish a move interrupted at moving the store or store moved, ask for the move again, naming the address the directories now have. Read the journal with hmctl mailbox status or GET /api/v1/accounts/<address>/move. The journal is not in the backup. Code Meaning HM6941 The move succeeded but the alias for the old address could not be created HM6942 Aliases pointing at the account could not be re-pointed HM6943 Distribution list memberships could not be moved HM6944 Messages in the delivery queue still name the old address HM6945 Quarantined messages still name the old address HM6946 The move could not be journalled. It still happened, and is on the audit trail Who may. manage_accounts, in both domains. The server administrator and a domain administrator of both domains have it; the help-desk role and the auditor do not. view_accounts is enough to read the journal. A move is one entry on the audit trail: object type account, action moved. What it does not do The relocation runbook does not repair the uninstall entry or the shortcuts, and it has no converter for the archive index. A mailbox move does not wait for a delivery in flight; nothing in the server records one. Setting Account.Address and saving goes through the same implementation, but leaves no alias, does not refuse while the mailbox is in use, and cannot cross domains. A domain rename (Domain.Name and Save()) is a different thing again, and keeps the path it has always taken. Where to read more Wiki: Relocating an Installation. Documents: RelocatingAnInstallation.md and MailboxMove.md. The public documentation: 33.1, the installation on disk. On this forum: Moving to new hardware, and why Windows to Linux is a rebuild, Upgrading to 6.3.4, and how the settings store works and when a save takes effect in Settings: what is not stock, when a save takes effect, and the three levels.
  • ProgressiverobotP

    How the service is built and what runs out first: pools, connections, large folders

    Guides & tutorials limitations hardware diagnostics administration webmail
    1
    0 Votes
    1 Posts
    119 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) changes three things this guide touches. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, so the statement after a database restart, a failover or wait_timeout no longer fails, and an insert that forgives a duplicate key forgives that alone, so an error log that was quiet may now name failures that were hidden. On Windows a stop or a reinitialise no longer waits past a work queue's ten seconds for a thread held by a host or a database that has stopped answering, and setup, the live update's silent install and the uninstaller wait ShutdownDrainSeconds plus 90 seconds for the stop, where they waited plus 30. The ACME HTTP-01 challenge responder, which served one connection at a time, serves up to 64 at once. See the 6.3.7 announcement. This topic describes a running hMailServer 6.3.4 service: its parts and pools, the connection ceilings in front of them, the scheduled work behind them, what the machine needs, and what a folder of a hundred thousand messages costs the webmail. The one thing to know first: nothing in the server is unbounded. There is no thread per connection and no pool that grows. Every piece of work runs on a pool of a set size, and a dependency that stops answering uses up a pool long before it uses up the processor, the memory or the disk. What it cannot give you is a throughput figure: none has been measured yet. The parts of the running service Everything runs in one process: the Windows service hMailServer, or the systemd unit hmailserver on Linux. Part What it is Notes Mail listeners SMTP, POP3 and IMAP on IOService + TCPServer Boost.Asio HttpServer The engine under two listeners: REST (/api/v1, the webmail at /portal, the Control Deck) and web services (MTA-STS policy, autoconfig, ACME http-01) Its own io_context, so a request storm cannot starve SMTP accept. Since 6.2.28 Metrics listener /metrics and the health probes Raw sockets, one worker thread ManageSieve Sieve script management Raw sockets, a thread of its own SNMP agent Read-only, UDP 161 Since 6.3.4; off, and answering nobody until SnmpAllowedAddresses names a manager Background Scheduler, SMTPDeliveryManager, ExternalFetchManager Long-lived tasks on the main server queue State The database (accounts, domains, message metadata and settings, since 6.3.4 including the [Settings] keys), one file per message, and hMailServer.ini as the bootstrap The database stays small next to the store The handover that matters for capacity: an SMTP session never delivers anything. The final 250 is sent from a worker on the asynchronous task queue once the message is written, and delivery happens later on a delivery thread, so the accepting side and the delivering side run out separately. Diagnosing stalled mail takes each half in turn; How email works follows one message through both. The pools, their sizes, and what a full one looks like Seven work queues are created at startup, and the two HTTP listeners have workers of their own. The first column is the queue label of hmailserver_workqueue_depth{queue} on the metrics endpoint, which counts tasks accepted and not yet started; the last row is not a work queue and has no label. Queue Threads Sized by What runs on it When it is full IOCPQueue 15 tcpipthreads Every socket read, write and TLS handshake for SMTP, POP3 and IMAP New connections are accepted but nothing progresses; clients time out mid-command Asynchronous task queue 15 MaxNumberOfAsynchronousTasks Message finalization, which sends the final 250 Senders sit after CRLF.CRLF with no reply Name lookup queue 15 the same setting Blocking reverse-DNS lookups for live sessions Reverse names become "Unknown", so a dead reverse zone cannot stall acknowledgements SMTP delivery queue 10 maxdelivertythreads The delivery of one queued message The queue grows and mail is late; nothing is lost, since the sender already has its 250 External fetch queue 15 MaxNumberOfExternalFetchThreads One run per fetch account Fetch accounts fall behind their schedule Maintenance queue 5, fixed nothing One-shot scheduled tasks; a backup started over REST A slow sweep delays the other one-shot sweeps Server queue 4, fixed nothing Scheduler, IOService, SMTPDeliveryManager, ExternalFetchManager Cannot fill in normal operation: each task holds one slot for the life of the server HttpServer workers 4 per listener, fixed nothing One REST or web-services request each Requests queue behind the four; SMTP is unaffected Where the four sizes are set. On Windows, the Control Panel's Maintenance → Performance page, Threads tab, holds the COM Settings properties TCPIPThreads, MaxAsynchronousThreads and MaxDeliveryThreads, and the [Settings] key MaxNumberOfExternalFetchThreads. On either platform the first three are in the server settings group: the Control Deck's Settings view, PUT /api/v1/settings with tcpip_threads, max_asynchronous_threads and max_delivery_threads, or hmctl settings set. The fourth goes through /api/v1/settings/ini/{name} (administrator password only), hmctl ini set, or on Linux hmailserver --set-setting as root. Since 6.3.4 the database is the settings store, so do not edit the [Settings] line in hMailServer.ini: a line edited there is ignored in favour of the stored value. Settings: when a save takes effect covers the store, what becomes of the file's line (HM5804) and when each key applies. The tab also shows WorkerThreadPriority, read-only: the server stores that value and never reads it. When a new size takes effect. Since 6.3.4 a save applies most settings at once, but MaxNumberOfExternalFetchThreads waits for a reinitialise, because the queue's size is set when the fetch manager is built; GET /api/v1/server/settings/pending lists it until then. For the other three the documents leave a gap: the Control Panel reports a save of its COM settings as "applied immediately", while the Architecture page has the pools created when the servers start, and no document says when a changed count reaches a running pool. Until one does, treat all four alike: after saving, reinitialise (the Control Panel's Reinitialize or Apply now, POST /api/v1/server/reinitialize, or hmctl api POST /api/v1/server/reinitialize) or restart the service. A reinitialise stops and starts every service in place, ends the Control Deck's session, and ends a drain. # The REST listener on this machine: read the server settings group, # then give delivery 25 threads (or: hmctl settings set --set max_delivery_threads=25) curl -u Administrator:pw http://127.0.0.1:8443/api/v1/settings curl -u Administrator:pw -X PUT -H "Content-Type: application/json" \ -d '{"max_delivery_threads":25}' http://127.0.0.1:8443/api/v1/settings # The fetch pool is a [Settings] key; it waits for a reinitialise hmctl ini set MaxNumberOfExternalFetchThreads 20 hmctl api POST /api/v1/server/reinitialize Two asynchronous threads are kept back. Finalization runs the spam tests and event scripts, which wait on things outside the process, so AsyncQueueReservedThreads (2) keeps that many threads free of it: with the defaults at most 13 finalizations run at once. A task waiting for a slot holds no thread and is counted by hmailserver_workqueue_blocking_tasks_waiting{queue}; a count that stays above zero means something those tasks depend on is slow. On Windows the key is on Server limits & expert settings, card Timeouts and queue bounds; anywhere, hmctl ini set. Only two queues report a stall: the asynchronous and name lookup queues write the "every thread busy" report that Diagnosing stalled mail reads. The others are left out deliberately, since their tasks are long-lived by design, so a full IOCPQueue, SMTP delivery queue or External fetch queue shows only in hmailserver_workqueue_depth{queue} and in its symptoms. The database connections every pool shares The deliveries, the sign-ins and the rest of the server share one pool of database connections, NumberOfConnections in the [Database] section of hMailServer.ini, 5 as shipped. A wait for one longer than DBConnectionAcquireTimeout (60 seconds) fails. In hmailserver_db_connections{state}, available pinned at zero with busy at the pool size is saturation, and every mail path is then waiting out that timeout. Raise it with the delivery threads. The relay profile sets maxdelivertythreads to 25 and the sender profile to 50, and each tells you to raise NumberOfConnections to about the delivery thread count. The built-in database has one connection. On Windows, NumberOfConnections is forced to 1 for MSSQLCE. Before You Install calls the built-in engine adequate for anything up to a few dozen mailboxes, and its single connection the first thing to change if the server feels slow under concurrency. Linux has no built-in database; it uses PostgreSQL or MariaDB. Choosing a database covers the choice. It is a bootstrap key. [Database] stays in the file, never in the database, and its keys wait for a restart. On Windows the Performance page's Database tab edits it, on the server machine only. On Linux: sudoedit /etc/hmailserver/hMailServer.ini # [Database] NumberOfConnections=25 sudo systemctl restart hmailserver Connections and requests: the ceilings in front of the pools Where Limit As shipped Set on SMTP sessions MaxSMTPConnections 0, no cap Connections & protocols → Protocols, SMTP tab; max_smtp_connections IMAP sessions MaxIMAPConnections 0, no cap The IMAP tab; max_imap_connections POP3 sessions MaxPOP3Connections not given in the documents; GET /api/v1/settings shows it The POP3 tab; max_pop3_connections Recipients in one message 50,000 50,000 No setting is named; beyond it, 550 Too many recipients REST listener, with the webmail and the Deck 64 connections, closed at accept before any TLS handshake; 4 workers; 64 KB a request (16 MB, with 300 seconds, on the two attachment routes); 30 seconds a request; 300 seconds a connection fixed no key Web-services listener 128 connections; 4 workers; 64 KB a request; 15 seconds a request; 120 seconds a connection fixed no key The webmail shares the REST listener: its users, the Control Deck and every script that calls the API meet the same 64 connections and four workers. Its request rate is in the topic on the REST API. Outbound, since 6.3.4, MaxOutboundConnsPerDestination and MaxOutboundConnsPerTenant (both 0, off) cap sessions per destination and per sending domain; a tenant share larger than the delivery thread count is no share at all (Outbound shaping and IP warm-up). Scheduled work The Scheduler polls its recurring tasks once a minute; one-shot tasks go through the Maintenance queue at once. External fetch is not one of them: the ExternalFetchManager runs its own one-minute loop. Roadmap.md recorded in August 2026 that a due task runs on the scheduler's own thread, so one long task delays the others, and that the real period is the interval plus the run time. The tasks, grouped by cadence: Task Cadence Registered GreyListCleanerTask every GreylistingRecordExpirationInterval minutes (240 as shipped) when SMTP is on RemoveExpiredRecords, WorkQueueHealthTask every minute always; the second writes the stall report TlsRptReporterTask, DmarcRptReporterTask hourly always; each sends only when its from-address is set LogRetentionTask, MailboxRetentionTask start + 6 hours always ArchiveRetentionTask, IMAPExpungeRetentionTask start + 12 hours always MetricsHistoryTask start + every minute always UpdateCheckTask start + every 15 minutes always; does nothing until UpdateCheckEnabled=1 MessageStoreConsistencyTask, DiskSpaceMonitorTask start + hourly always BackupScheduleTask a one-minute tick against the wall clock only when a schedule is set DirectorySyncScheduleTask start + every [LDAP] SyncScheduleMinutes only when that is set AcmeRenewalTask start + hourly only when AcmeEnabled=1 Sizing the machine Memory. Before You Install asks for 2 GB as a minimum on Windows, more for a busy server. It is guidance; nothing enforces it, and memory per session has not been measured. Disk. Size it for how much mail you will keep, not how much you have today. The store is Data\ under the program folder on Windows and /var/lib/hmailserver on Linux, one file per message. For scale, the webmail bench's hundred thousand messages, 5 % of them with an attachment of 20 to 512 KB, are 1,323,987,827 bytes. Below MinimumFreeDiskSpaceMB (100) new mail is refused temporarily, 452 4.3.1 over SMTP and NO [UNAVAILABLE] to an IMAP APPEND, so senders retry; below DiskSpaceWarningThresholdMB (1024) the server logs a warning and keeps accepting. Antivirus. On Windows, exclude Data\, Database\ and Temp\ from real-time scanning. A scanner holding a message file open is a documented cause of delivery that silently does nothing; the server detects one shape of it and logs error 5760. Before You Install names no exclusions for Linux, where the store is /var/lib/hmailserver with its temp and database subdirectories. Durability. MessageStoreFsync (0 as shipped) flushes the spool file to disk before the 250, and a failed flush refuses the message. The published-numbers method runs with it on and labels the shipped default "not durable". Set it on Windows on Server limits & expert settings (card Message store durability), or anywhere with hmctl ini set. A folder of a hundred thousand messages in the webmail Every webmail path reads the folder's shared message collection, the one IMAP, POP3 and delivery use: one small object per message. Until 6.3.4 opening a folder copied the whole of it, under the lock IMAP sessions and delivery wait on, for every page of 200. Since 6.3.4: Operation What it does now What still grows with the folder Open, GET /api/v1/me/folders/{id}/messages One pass under the lock counts the folder and copies only the page of 200; each row's file head is read twice, not three times That pass; and the first open after a start, or after thirty idle minutes in the cache, reads every row of the folder from the database, as an IMAP SELECT does Scroll Older and Newer by before_uid; from three pages up, Oldest in one request (oldest_before_uid) Every page costs what the first costs, wherever it is in the folder: the one pass that counts the folder Search At most 2,000 message files and about two and a half seconds a request, then answers where to go on from; across every folder it walks the mailbox newest first and answers next_before_id; with IndexerFullText and message indexing on (off as shipped), a message the index rules out is passed without opening its file Performance.md does not say Select-all and one action The selection goes to POST /api/v1/me/folders/{id}/bulk: at most 1,000 messages a call for a flag, 500 for a move or a delete, nothing new taken on after about two and a half seconds A changed flag is a row written, a move is a file copied, and a selection narrowed to an inbox tab reads the head of every message Junk over a whole folder is a hundred thousand lessons. With SpamAssassin on, each message moved into or out of Junk queues one lesson, as the same move over IMAP does. In the Settings Reference the lesson on a move is SpamAssassinLearnOnMove (off as shipped), run as a task on the asynchronous work queue. A move holds the mail twice, briefly: original and copy, both counted against the quota, for no longer than one slice of the work. Not changed. The Storage, Subscriptions and Scheduled views still copy each folder they look at; emptying Junk or Trash copies the folder once per 500 messages removed; a quick step over a whole-folder selection still reads at most five thousand messages. For other callers of GET /api/v1/me/search, it now answers complete: false, more: true with next_before_id whenever it is not the end, and its hits come newest first across every folder, where they came folder by folder. Over IMAP, one SEARCH or SORT is bounded by IMAPSearchTimeout (60 seconds) and IMAPSearchMaxMegabytes (2048 MB examined). The webmail portal at /portal covers the rest of the client. What has not been measured The webmail at a hundred thousand. Performance.md has the method and a results table for the Windows bench (SQL Server Compact, 100,000 messages in the inbox, the full-text index off and on). Every cell reads to be measured on the bench; nothing was estimated. Its tools, build/perf/seed-folder.py and build/perf/webmail-timing.py, are in the repository and also run against a Linux server. Record the machine, database, message store, index state and build with any numbers. How much the server carries. PublishedNumbers.md defines messages in and out per second, latency percentiles, IMAP sessions per core with every operation's 95th percentile under 200 ms, memory per session and Dovecot's imaptest, measured against Postfix and Dovecot on the same machine under ten fairness rules. Its results say Not measured yet, and until they are filled the first line of Roadmap2.md is a goal, not a claim. The load generator, in the repository since 6.3.4, is build/perf/load/loadgen.py; a run in which the generator was busy for more than 85 per cent of it gives only a floor and is not published. The Windows bench gives hMailServer's own numbers only, not to be compared with the Linux table. python build/perf/load/loadgen.py --help python build/perf/load/loadgen.py self-test # proves the generator, says nothing about any server Size from the ceilings above, watch the queue depths and the database pool, and treat any throughput figure quoted elsewhere as unmeasured by the project. What it does not do No document gives a rule for sizing the pools to a machine; the only sized examples are the relay and sender profiles' 25 and 50 delivery threads. The main server queue, the maintenance queue and the HTTP listeners' workers, connection caps and deadlines are fixed; no key changes them. Where to read more Wiki: Architecture, Before You Install, Control Panel Reference section 7.8, Settings Reference, The REST API section 8, APIs Reference section 1 and Monitoring and Health. Documents: Performance.md, PublishedNumbers.md, SettingsApply.md, Profiles.md and OutboundShaping.md. The public documentation: Before you install, A database and Prometheus metrics and health probes. On this forum: Diagnosing stalled mail, Reading the logs, Monitoring and health, The webmail in depth, The Control Deck and hMailServer on Linux. Without a GitLab account, a bug or a question can go to the Service Desk address contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the replies arrive by email.
  • ProgressiverobotP

    Every port hMailServer listens on or connects to, and the firewall rules to match

    General support security administration windows linux diagnostics
    1
    0 Votes
    1 Posts
    169 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) lets the IMAP and POP3 listeners take the client's address from a PROXY protocol header, as the SMTP listener does, each from the peers on a list of its own; both are off by default, and in the Control Panel the PROXY protocol and XCLIENT are a card on Incoming relays, while API & monitoring is Monitoring & alerts. The ACME http-01 challenge responder serves up to 64 connections at once and closes one past that, where it served one at a time. The REST listener refuses a state-changing request that carries a credential when the browser reports it as cross-site (Sec-Fetch-Site, or an Origin that is not this server's); scripts, hmctl and the PowerShell module send no such header and are unaffected. 6.3.7's container image is registry.gitlab.com/hmailserver/hmailserver:6.3.7. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) has the web services listener close a connection, with no answer and before the TLS handshake, from an address whose IP range allows none of SMTP, POP3 and IMAP - an auto-ban's, or one of your own that allows nothing - while loopback and the addresses in WebServicesTrustedProxies are never refused by a range; so a range of your own that allows some addresses nothing now closes this listener to them as well, and with it ACME's http-01 validation when this listener answers it, MTA-STS, CalDAV, CardDAV, Autodiscover, autoconfiguration and the Web Key Directory: check that the certificate authority, the sending MTAs and these services' clients do not come from one. 6.3.8's container image is registry.gitlab.com/hmailserver/hmailserver:6.3.8, also tagged 6.3 and latest. See the 6.3.8 announcement. This topic lists every socket hMailServer 6.3.4 can open, inbound and outbound, and the firewall rules that follow. The one thing to know first: a new server listens on four mail ports and on nothing else. Every other listener is off until a setting gives it a port, three of them bind only to 127.0.0.1 when you do, and almost every outbound connection beyond DNS, SMTP delivery and the recipient's MTA-STS policy exists because an administrator turned a feature on. Write the firewall for what you turned on. Connection security per mail port, and what a new 6.3.4 installation sets on each, is in Connecting a mail client: IMAP, POP3, SMTP submission and the ports. Two kinds of listener, set in two places Mail-protocol listeners (SMTP, POP3, IMAP) are rows of the table hm_tcpipports, edited on Connections & protocols → TCP/IP ports in the Control Panel (Windows), on the Control Deck, over COM (Settings.TCPIPPorts, Windows) and over REST at /api/v1/ports, where a new row's address defaults to 0.0.0.0. A port you add, change or delete takes effect at a reinitialise (POST /api/v1/server/reinitialize, or the Deck's button that restarts the services in place) or a service restart. Service listeners are one key each, all off by default. Set their [Settings] keys in the Control Panel (Windows), over REST (PUT /api/v1/settings/ini/{name}), with hmctl ini set or hmailserver --set-setting NAME=VALUE; the Deck has no page for them. Since 6.3.4 the database holds these keys, so a line edited in hMailServer.ini turns no listener on (HM5804); Settings: what is not stock, when a save takes effect explains the store. Each listener's port, bind address and certificate files are among the 28 keys that wait for a reinitialise: Apply now in the Control Panel or the Deck, POST /api/v1/server/reinitialize, Application.Reinitialize over COM, or a restart. AcmeHttpPort applies when it is saved. The SNMP agent's settings are hm_settings rows instead, set on the Deck's Settings page under Server, over REST (/api/v1/settings) or with hmctl settings set, with no Control Panel page, and read when the agent starts, at a service start or a Reinitialize. The mail listeners Port Protocol Created by Default bind What it says on connect 25 SMTP Seeded in every new database The row's IP address (0.0.0.0 or :: is every address) 220 MAIL01 ESMTP: the Welcome banner from Connections & protocols → Protocols → SMTP, or the computer name, with ESMTP appended 587 SMTP submission Seeded The row's IP address The same banner 110 POP3 Seeded The row's IP address +OK POP3, or your welcome message 143 IMAP Seeded The row's IP address * OK IMAPrev1, or your welcome message 465, 993, 995 SMTP, IMAP, POP3 over implicit TLS You, once you have a certificate What you choose Nothing until the TLS handshake completes A listener bound to one internal address is the usual reason mail works from the LAN and not from outside. Two failures leave one port dead and the rest running: a TLS port with no certificate selected does not start (HM5113), and a port something else holds does not bind (HM4316). On Windows with WSL2 in mirrored networking mode, a socket bound inside WSL takes the port from Windows too, shows in neither Get-NetTCPConnection nor netstat -ano, and on the project's test bench only wsl --shutdown gave it back. The service listeners Listener Port key, default Bind key, default TLS How to tell it is up REST API, the Control Deck at /, the webmail at /portal, SCIM at /scim/v2 (since 6.3.4) RestApiPort, 0; 8045 is the conventional port RestApiBindAddress, 127.0.0.1 Required unless bound to 127.0.0.1, localhost or ::1 RestApi: Listening on <endpoint> (https) or (http, loopback only) in the application log Public web services WebServicesHttpPort, WebServicesHttpsPort, both 0; 80 and 443 to enable WebServicesBindAddress, 0.0.0.0 WebServicesCertificateFile and WebServicesPrivateKeyFile on the HTTPS port, or the ACME certificate on disk A URL it serves answers Prometheus metrics and health probes MetricsServerPort, 0 MetricsServerBindAddress, 127.0.0.1, an IP literal only MetricsServerCertificateFile + MetricsServerPrivateKeyFile; plain HTTP otherwise /livez answers 200 alive ManageSieve (RFC 5804) ManageSieveServerPort, 0; 4190 is the standard port ManageSieveServerBindAddress, 127.0.0.1 STARTTLS, only when an IMAP, POP3 or SMTP port has a certificate Its capability response ACME http-01 challenge AcmeHttpPort, 80 Every address Never Open only during an issuance SNMP agent, UDP, since 6.3.4 SnmpPort, 161, with SnmpEnabled off SnmpBindAddress, empty = every address SNMPv3 at authPriv instead of TLS A stranger gets no answer at all One port carries the REST API, the Deck and the webmail; there is no second port to open. It will not start without an administrator password, nor beyond loopback without a certificate, and with RestApiCertificateFile empty it takes the ACME certificate when one exists. For users to reach /portal, or an identity provider SCIM, it needs a certificate they trust, a reachable bind address and a firewall rule, or a TLS-terminating proxy in front (Behind a gateway, load balancer or reverse proxy). Turning it on is in The REST API: turning it on, credentials, roles, one-time codes. The web services listener binds every address by default. It serves the MTA-STS policy at /.well-known/mta-sts.txt (HTTPS only), ACME challenges, Thunderbird autoconfig, Outlook autodiscover and the Apple profile (HTTPS only), security.txt, CardDAV and CalDAV under /dav/ (HTTPS only; CalDAV since 6.3.4), and, since 6.3.4, the OpenPGP Web Key Directory (HTTPS only) and an S/MIME authority's root and CRL (plain HTTP). MtaStsHostingEnabled and AutoconfigEnabled default to 1 while both ports stay 0, which the server reports once at startup; the ports stay 0 on purpose, because port 80 may belong to IIS. The HTTPS port waits for a certificate, its own or ACME's, and says so in the log. Metrics. A host name or localhost as the bind address is refused (MetricsServer: Invalid bind address) and the probes go with the listener. Beyond loopback with no credential, /metrics answers 503 while the probes still answer. More, including the SNMP agent, is in Monitoring and health: Prometheus metrics, health probes, Grafana, OTLP and SNMPv3. ManageSieve has no certificate of its own and offers no STARTTLS, saying so in the log, until a mail port has one. The ACME challenge listener is not permanent. When WebServicesHttpPort equals AcmeHttpPort, the web services listener answers /.well-known/acme-challenge/; otherwise a temporary listener binds AcmeHttpPort for the issuance, and a failed bind logs Failed to start the http-01 challenge listener. Is the port in use? The client uses the http-01 challenge only, so port 80 must be reachable from the internet (Let's Encrypt certificates are automatic). The SNMP agent (since 6.3.4) answers nobody until SnmpAllowedAddresses names a manager; a datagram from any other address is dropped before it is read. Bind it to 127.0.0.1 or a management address, and name managers one by one: UDP does not prove a source address. Error 7420 is a port it could not bind, such as one the operating system's own SNMP service holds. Snmp.md lists the Linux bind to 161 under the packaged unit as not proved by any test. A Control Panel on another machine (Windows) reaches the COM API over DCOM, which must be permitted, after probing TCP port 135 (the RPC endpoint mapper). hmctl and the PowerShell module use the REST listener instead. Where Windows and Linux differ Where you set a listener. On Windows, the TCP/IP ports page, Connections & protocols → Web services & autoconfiguration, TLS & certificates → Certificates (ACME) and Monitoring & troubleshooting → API & monitoring (Monitoring & alerts since 6.3.7). Linux has no Control Panel and no COM: the Deck for the mail listeners and the SNMP agent, and REST, hmctl or hmailserver --set-setting for every listener's [Settings] key. The packaged RestApiPort=0 is stored at the first start, and a line edited afterwards turns nothing on; First install on Linux has the order. Low ports. The packaged hmailserver.service runs as its own user with CAP_NET_BIND_SERVICE, which it has for port 25 and which also lets the agent bind UDP 161. The database port. On Windows [Database] Port of 0 means the provider's own default. On Linux write it out (5432, 3306 it goes into the connection string as it stands. DNS. On Windows the custom servers under Connections & protocols → DNS resolver are given port 0, because the Windows DNS client supplies 53 itself and rejects an entry that names one; DNSServer is one IPv4 address with no port syntax. Containers. The image exposes 25, 110, 143, 465, 587, 993, 995, 4190 and 8045, and SNMP is published as UDP. No image of 6.3.4 is published; 6.3.8's is registry.gitlab.com/hmailserver/hmailserver:6.3.8, also tagged 6.3 and latest, and In a container or on Kubernetes has the rest. Proving what is listening On Windows, on the server and then from outside: Get-NetTCPConnection -State Listen -LocalPort 587 | Select-Object LocalAddress, LocalPort, OwningProcess Test-NetConnection 127.0.0.1 -Port 25 Get-NetFirewallRule -Enabled True | Where-Object DisplayName -like "*hMail*" Test-NetConnection mail.example.com -Port 25 openssl s_client -connect mail.example.com:465 -quiet On Linux, ss -ltnp lists listening sockets with their process, and the REST listener answers curl -u Administrator http://127.0.0.1:8045/api/v1/status. On Windows, Monitoring & troubleshooting → Server status lists the listeners the service brought up; senders timing out with nothing in the SMTP log means nothing reached a listener. What the server connects to On every server, with nothing turned on: Destination Port Setting If it is blocked DNS resolvers 53, UDP and TCP The machine's resolvers, or DNSServer Nothing is delivered externally, and DNSBL, SPF, DKIM, DMARC, MTA-STS and DANE stop at once Every recipient domain's MX host 25 None Messages stay queued and are retried, by default 4 times at 60-minute intervals, then bounced The recipient domain's MTA-STS policy, from mta-sts.<domain> 443 MtaStsEnabled, 1 Delivery to a domain with an enforcing policy falls back to what DANE and your TLS settings allow DANE and TLS-RPT need no port of their own: TLSA records come over DNS, and TLS-RPT reports are sent as ordinary mail. Delivery leaves from SMTPDeliveryBindToIP, or a rule's bind to address action; since 6.3.4 SMTPDeliveryBindToIP may name one address of each family, separated by a comma. Only when you configure it: Destination Port Setting, as shipped Smart host, or a route's host SMTPRelayerPort (25; 0 is read as 25), or the route's own Empty Database server The provider's port [Database]; since 6.3.4 MySQL and MariaDB use TLS on it when offered SpamAssassin spamd, ClamAV clamd 127.0.0.1:783, localhost:3310 Both off; no TLS An HTTP filter engine The URL's FilterHookUrl, empty The ACME directory 443 AcmeEnabled, 0; AcmeDirectoryUrl is Let's Encrypt The update feed, https://updates.progressiverobot.com, and the installer from the project's package registry on gitlab.com 443 UpdateCheckEnabled, 0 The password breach check, https://api.pwnedpasswords.com/range/ 443 BreachCheckEnabled, 0 (since 6.3.4) An object store: https://s3.<region>.amazonaws.com, https://<account>.blob.core.windows.net or MessageStoreObjectEndpoint 443, or the endpoint's MessageStoreObjectService, empty (since 6.3.4) OAuth2: a key set, an introspection endpoint, the outbound relay's token endpoint The URL's OAuth2JwksUrl, OAuth2IntrospectionUrl, OutboundOAuth2TokenUrl, empty An OpenID Connect issuer: <issuer>/.well-known/openid-configuration, then its token endpoint The issuer's oidc_issuer, empty (since 6.3.4) Outbound webhooks, alert webhooks, a sandbox service, a route of mail in to a webhook The URL's No subscription, rule or route; SandboxEnabled, 0 (all since 6.3.4) A syslog collector SyslogPort, 514 (UDP, TCP); 6514 is the registered TLS port SyslogEnabled, off (since 6.3.4) An OpenTelemetry collector The endpoint's OtelEndpoint, OtelMetricsEndpoint, OtelLogsEndpoint, empty The browser vendor's push service The service's WebPushEnabled, 0 (since 6.3.4) External accounts and migrations: another server's POP3 or IMAP; imap.gmail.com or outlook.office365.com on 993; graph.microsoft.com; CardDAV and CalDAV over https The account's; a person's own account 110, 995, 143 or 993 at public addresses unless reader_fetch_accounts_private_addresses is on Nothing until an account or a migration is made At a person's request in the webmail: since 6.3.4, a correspondent's OpenPGP key from their domain's Web Key Directory (lookup_wkd, on; never an address inside your network) or from https://keys.openpgp.org (lookup_keyserver, off), and a sender's BIMI logo (public addresses only, and only with AuthenticationResultsEnabled, which is 0); and, since 6.3.2, a one-click unsubscribe, posted to a public address only. Where an outbound request may go. Every outbound web request is https, or plain http to this machine only; https is verified against the machine's root store, host name checked, TLS 1.2 at least, so a private CA must be in that store (or in MessageStoreObjectCaFile for an object store). Outbound webhooks may reach a private address, on purpose; a route of mail in to a webhook never reaches a link-local or cloud metadata address, and one not made by the server administrator reaches public addresses only. Through a proxy. HttpProxy (host:port or [ipv6]:port, since 6.2.28) carries the update feed and its downloads, the JWKS and introspection fetches, the breach check and the object store. It sends no proxy credentials, and a value with no port fails with HttpProxy must be host:port (or [ipv6]:port). rather than going direct. ACME issuance, the outbound relay's token request, a mail-in-to-webhook route and the MTA-STS policy fetch never use it; the MTA-STS fetch is on by default, so a proxy-only egress stops it. Whether the other requests above use the proxy is not documented: prove a proxy-only egress before relying on it. UpdateFeed.md, written before the feed site's first deployment, says the address answers nothing usable until then. The firewall matrix Direction Port From or to Only when In TCP 25 Anywhere Always In TCP 587, and 465, 993, 995 Anywhere, or your users' networks 587 always; the others once you have created them In TCP 143, 110 Your users A client still uses them; deny them afterwards In TCP 80 Anywhere ACME, or web services over HTTP In TCP 443 Anywhere WebServicesHttpsPort is 443 In TCP REST port (8045) Your management network; users for /portal; the identity provider for SCIM The listener is bound beyond loopback, with TLS In TCP metrics port, ManageSieve 4190 Prometheus; your management network Bound beyond loopback (or keep 127.0.0.1 and a tunnel) In UDP 161 The SNMP managers only, on the bound address The agent is on In TCP 135 and DCOM The administrator's machine A remote Control Panel (Windows) Out TCP 25 Anywhere, or only the smart host on its port Always Out UDP and TCP 53 Your resolvers Always Out TCP 443 Anywhere, or the hosts in the tables above MTA-STS is on (the default) or a feature needs it Out Database, spamd, clamd, syslog, OTLP ports Those hosts They are not on this machine Out TCP 110, 995, 143, 993, or the account's port The servers mail is collected or migrated from An external account or a migration exists Windows. The installer opens no firewall port, so nothing reaches the server until you add the rules. In an elevated PowerShell: New-NetFirewallRule -DisplayName "hMailServer SMTP" -Direction Inbound -Protocol TCP -LocalPort 25 -Action Allow New-NetFirewallRule -DisplayName "hMailServer Submission" -Direction Inbound -Protocol TCP -LocalPort 587 -Action Allow New-NetFirewallRule -DisplayName "hMailServer IMAPS" -Direction Inbound -Protocol TCP -LocalPort 993 -Action Allow New-NetFirewallRule -DisplayName "hMailServer POP3S" -Direction Inbound -Protocol TCP -LocalPort 995 -Action Allow Linux. With firewalld, firewall-cmd --permanent --add-port={25,587,993,995}/tcp and firewall-cmd --reload; with ufw, ufw allow 25,587,993,995/tcp. The documentation page's chapter 4.4 also opens 110, 143 and 465; leave them out. On either system, add 143, 110, 465, 80 or 443 only when the matrix says so, and forward the same ports on the router or cloud firewall. The Linux package's own firewall hook (AutoBanFirewall=1) only blocks, and needs CAP_NET_ADMIN, which the unit does not grant by default; on Windows the same switch is refused under NT SERVICE\hMailServer (HM7460). Put the addresses you administer from in AutoBanNeverBan first; Reading the logs, and fail2ban has auto-ban in the firewall. What it does not do The SNMP agent sends no trap or inform and does not answer SNMPv1; nothing leaves it unasked. Country blocking downloads no geographic database; the .mmdb file is yours to fetch and keep fresh. No AI or machine-learning service is called by the server. Where to read more Wiki: Ports Reference, Before You Install, Installing hMailServer, Installing on Linux, Settings Reference, Troubleshooting, Encryption and Certificates. Documents at v6.3.4: EvidencePack.md (what leaves the server), SettingsApply.md, Snmp.md, UpdateFeed.md, InboundWebhooks.md, ObjectStorage.md, PasswordBreachCheck.md, Migration.md. The documentation page's ports reference and Open the firewall chapters. On the forum: Let's Encrypt certificates are automatic, First install on Linux, The Control Deck, Installing on Windows: wizard pages, service account, profile and the first hour, Your first domain and mailbox, and the DNS records your domain needs, TLS on the listeners: your own certificate, versions, ciphers and Troubleshooting by symptom: no mail in, no mail out.
  • ProgressiverobotP

    Troubleshooting by symptom: no mail in, no mail out, a service that will not start

    General support diagnostics smtp windows linux systemd
    1
    0 Votes
    1 Posts
    170 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) changes what an upgraded server does without anything being switched on, and some of it looks like a fault. A forward, a redirect or a rule's forward of mail from outside that did not pass DMARC for the forwarding domain's organisation keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. An authenticated submission is signed as another hosted domain only when the account may send as the address its From names, and is otherwise sent unsigned as that domain. A domain's or an account's rule may no longer set From, Sender, Resent-From, Resent-Sender, DKIM-Signature or the loop count, and a Sieve script may no longer add or delete them: one stored before the upgrade still runs without that action, and the application log names it. RuleLoopLimit 0 stops out-of-office replies. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries. On Windows the x64 installer's MariaDB Connector/C 3.4.11 has seven authentication plugins where 6.3.6's had nine, so a database account that signs in with MySQL Enterprise's Windows authentication can no longer sign in and the server cannot open its database. An error log that was quiet may now name database failures that were hidden (HM5032), and the Control Panel's API & monitoring page is Monitoring & alerts. Fixed from reports: the Control Panel's IP ranges page that could show no list at all, DANE enforcement deferring mail to a DNSSEC-signed domain whose MX answer came compressed (live.com among them), and classic Outlook's POP3 downloads stopping at the UTF8 it sends after the login. See the 6.3.7 announcement. After the upgrade to hMailServer 6.3.8 (2 November 2026), TlsMode=verify-full on MySQL or MariaDB with no TlsCaFile set is refused with HM7535 on a connection to a database server on this machine, and the database is not reached; on Windows a TlsCaFile or PostgreSQLSslRootCert whose path holds characters outside ASCII may be refused with HM7534, with the same result, and a TLS certificate or key file so named with HM7536, so that the port it is for does not start; and a Linux server that ran under a LANG that is not UTF-8 no longer finds a file or folder named outside ASCII until it is renamed to its UTF-8 name. A client that signs in with another spelling of the account's address than the stored one, in more than the case of the letters A to Z, or with white space at either end, may no longer sign in. With the web services listener on, an address whose IP range allows none of SMTP, POP3 and IMAP - an auto-ban's, or one of your own that allows nothing - is closed there with no answer, and with it CalDAV, CardDAV, Autodiscover, autoconfiguration, MTA-STS and, when this listener answers it, ACME's http-01 validation. See the 6.3.8 announcement. This topic is the wiki's Troubleshooting chapter arranged by the sentence support threads open with: mail from outside does not arrive, a client cannot send, the Control Panel or the Deck will not connect, the service will not start. It also covers the server's replies, the Diagnostics self-test, the message trace and what only looks like a fault. The one thing to know first: on Windows the service tells the Service Control Manager it is running before it has opened its database or a single port, so "the service is running" proves nothing. Today's ERROR_hmailserver_<date>.log does: it almost always holds the reason a start failed. Start here Three pages, in this order, under Monitoring & troubleshooting in the Control Panel (Windows): Diagnostics, the built-in self-test (below). It names most problems outright. Live logs, while you reproduce the problem, with debug messages on if the answer is not obvious, and off afterwards. Delivery queue: is mail stuck, and what do NextTry and Tries say? On Linux the Deck's Diagnostics, Logs and queue views read the same, and three commands start every investigation: systemctl status hmailserver journalctl -u hmailserver -n 100 --no-pager sudo tail -n 100 /var/log/hmailserver/ERROR_hmailserver_$(date +%F).log Which problem is it? Not running, or nothing listening: The service will not start. A sender got an SMTP error: the reply names the cause. A sender timed out, or outbound mail sits in the queue: Diagnosing stalled mail. Inbound with nothing at all: I can't receive. Outbound that never reaches the queue: I can't send. "I can't receive mail from outside" Work down the table in order, cheapest check first. Check How (Windows) A failure means Is the service running? Get-Service hMailServer Start it, then read the error log for why it stopped Is anything listening on 25? Get-NetTCPConnection -LocalPort 25 -State Listen The port is not configured, is bound to one address only, or another program has it: Connections & protocols → TCP/IP ports Is 25 open in Windows Firewall? Get-NetFirewallRule -Enabled True \| Where-Object DisplayName -like "*hMail*" Open it: see the firewall paragraph below Does the machine itself answer on 25? Test-NetConnection 127.0.0.1 -Port 25 Then it is the server, not the network Is 25 forwarded by your router or cloud firewall? Its configuration The far end sees a refusal or a timeout Does your ISP block 25? Ask them Common on consumer lines: a relay host or a business line Is the MX record right? The MX query page, or Resolve-DnsName example.com -Type MX Fix DNS Can the world reach you? From outside: Test-NetConnection mail.example.com -Port 25 It is the path in Is the domain in hMailServer and Active? Accounts & domains → Domains An inactive domain is not accepted for Is the mailbox Active, and full? The account dialog A full mailbox gets a temporary failure, and the sender retries A port bound to one internal address answers only on the LAN. Server status lists the listeners the service actually brought up. The firewall. The installer opens no port, so nothing from outside reaches the server until you open them, on the machine and on the router or cloud firewall in front of it. Which ports, and the rules for Windows and Linux, are in Every port hMailServer listens on or connects to. "I can't send mail" The reply names most sending faults. In an ESMTP session an enhanced status code follows the number: 530 Must issue STARTTLS first. arrives as 530 5.7.0 Must issue STARTTLS first. Reply Likely cause What to do 550 Delivery is not allowed to this address. ("relay denied") The client is not authenticating, or uses 25 instead of 587 Port 587 with authentication; a printer that cannot authenticate gets its own range under Access & abuse protection → IP ranges 530 SMTP authentication is required. The same, said earlier As above 530 Must issue STARTTLS first. The port requires TLS first STARTTLS or SSL/TLS in the client 535 Authentication failed. Wrong password, not the full address as user name, or an inactive account Also check the auto-ban 535 Authentication failed. Too many invalid logon attempts. The auto-ban: 3 failures, 60 minutes, by default Below 550 Too many recipients. Too many recipients in one transaction Below 552 5.3.4 Message size exceeds fixed maximum message size. Size: N KB, Max size: M KB Over the server's maximum message size (20480 KB), or the domain's own Connections & protocols → Protocols, SMTP tab, or the domain dialog 452 4.3.1 Insufficient system storage. Free space on the store's volume below MinimumFreeDiskSpaceMB (100 MB) Free space; senders retry A banned client. The ban is an expiring IP range: Access & abuse protection → IP ranges lists it, and deleting it releases the client (DELETE /api/v1/ipranges/<id>, or the Deck's IP ranges view). 550 Too many recipients. Two wiki pages disagree. Troubleshooting §19.3 ties it to maxsmtprecipientsinbatch (100) and says split the batch or raise the limit; the Architecture page's RCPT TO chart refuses at 50,000 recipients and uses MaxSMTPRecipientsInBatch to batch outbound deliveries. Splitting the message is right either way; raising the setting also changes the outbound batch size. What the server's replies mean A 4xx is temporary and the sender retries; a 5xx is permanent and the sender bounces. Reply Meaning 421 4.7.0 Too many invalid commands. Bye! Disconnected for misbehaving: a scanner, or a broken client 451 4.3.1 Server temporarily overloaded while accepting the message; please retry. The accept pipeline exceeded FinalizationTimeout: Diagnosing stalled mail 451 4.3.2 Unable to verify the recipient at the moment. Please retry later. The database did not answer, so the mail is deferred, not bounced. Check /readyz 451 4.3.2 The server is shutting down and cannot accept the message. A graceful stop with ShutdownDrainSeconds set 452 4.2.2 Mailbox is full. At or over quota, refused during the conversation; the holder had a QUOTA_WARNING 550 Unknown user No such mailbox, alias or list 553 or 554 from a remote server The far end refused you; its text is the diagnosis, usually SPF, DKIM, PTR or a block list "My mail goes to spam" Almost always DNS, in this order: the PTR record (nothing compensates for a missing or generic one), SPF, DKIM, DMARC even at p=none, then the block lists. With split-horizon DNS, the outside view is the one that counts. Being trusted by the servers you send to covers it. The Control Panel, the Deck or the REST listener will not connect The Control Panel (Windows) talks to the running service over COM: no service, no connection. Use the administration password, not a mailbox's. After a restart it reconnects by itself; after you stop the service it deliberately does not. Remotely it needs port 135 and DCOM. Every page saying "Server unavailable" means the engine is paused: Server status → Resume. Since 6.3.4, under NT SERVICE\hMailServer, a change of the administrator password is refused because the service may not write the file: run hMailServer.exe /SetAdministratorPassword from an elevated prompt, then restart the service. More in The Control Panel on Windows. The Deck and the REST API share one listener, off until RestApiPort is set. It refuses to start without an administrator password, beyond loopback without a certificate, or with a certificate it cannot load (HM5113 names the file), each logged once as RestApi: Refusing to start; the three lines and their fixes are in The REST API: turning it on. Only a RestApi: Listening on line proves it is up. Since 6.3.4 the port waits for a reinitialise (Apply now, or POST /api/v1/server/reinitialize), and a line edited into hMailServer.ini after the first start loses to the stored value (HM5804). With the listener off, on Linux: sudo hmailserver --set-setting RestApiPort=8045 sudo systemctl restart hmailserver On Windows, the Control Panel's API & monitoring page (Monitoring & alerts since 6.3.7). "Web administration page not installed" means the WebAdmin files are missing, or ProgramFolder does not point at the directory holding them. Repeated bad passwords get the address refused for five minutes; loopback is exempt. The Deck itself is The Control Deck. The service will not start The start, in order; where it stops names the problem: Windows is told the service is running, before any of what follows. hMailServer.INI is read, alone, which is how the database is found. A [Database] Type it does not recognise abandons the start, with nothing logged about a database. On Linux, sudo -u hmailserver hmailserver --check-config then prints Database type: 0. HM5005: Server or Database is missing for the configured Type. The database is connected, retried ConnectionAttempts times (6), ConnectionAttemptsDelay seconds apart (5). HM4354: the database server is down or moved, or the credentials changed. On Windows the [Database] password is DPAPI-protected per machine and cannot be copied from another server. Since 6.3.4, HM7530 (an unknown TlsMode) and HM7531 (verify-full with a missing TlsCaFile) stop it too, and since 6.3.8 HM7535 (verify-full on MySQL or MariaDB with no TlsCaFile set, to a database server on this machine: name the CA that issued its certificate in TlsCaFile, or set TlsMode=require) and, on Windows, HM7534 (a TlsCaFile or PostgreSQLSslRootCert name the client cannot be handed as named: give the file and the folders above it ASCII names); --check-database makes one connection and says what it got. The schema version, compared both ways. HM5010: Database version could not be detected. HM5011 too old: DBUpdater.exe from Bin on Windows, sudo -u hmailserver hmailserver --upgrade-database on Linux. HM5011 too new: install the matching or a newer server; never downgrade the database. Since 6.3.4, [Database] UpgradeOnStart=1 makes the service upgrade the schema as it starts: off by default, no backup first, one node at a time. Settings are read again from the database, since 6.3.4 with any [SettingsOverride] keys over them. A configuration that will not load abandons the start. One listener per TCP/IP ports row. A row whose protocol is off on Protocols → Services is skipped in silence, the commonest reason for nothing on 143. HM4316: something else holds the port (IIS on 25 or 443, another mail server, a leftover instance); that port alone stays down. HM5113 with Error initializing SSL. Certificate not set.: that listener does not start. The optional listeners (metrics, REST, web services, ManageSieve), each only with a non-zero port. MetricsServer: Invalid bind address means MetricsServerBindAddress is not an IP literal (localhost is refused too); that listener stays down and the server runs. A failure in steps 2 to 6 stops the whole server, and one with a code writes it to the ERROR log; steps 7 and 8 cost one listener and go to the application log. Windows. A service that stops at once with nothing logged is the account's rights on Bin, Data, Logs and Database. Since 6.3.4 a new installation runs as NT SERVICE\hMailServer (an upgrade keeps its account), and a folder outside the program folder needs approving: run hMailServer.exe --confirm /GrantServiceAccess elevated in Bin, and approve only a folder you recognise, since the service may then write in it. Local SQL Server with integrated authentication needs a login for NT SERVICE\hMailServer. What the account may not do, and moving between it and LocalSystem, are in Installing on Windows. The installer's recovery settings restart a failed service after 1, 2 and 5 minutes (sc.exe qfailure hMailServer). Linux, under systemd. A start that fails before any listener opens puts hMailServer could not start: in the journal, with the reason on standard error in case the log directory is the fault. The unit runs as hmailserver, and ProtectSystem=strict makes anything outside its ReadWritePaths (/var/lib/hmailserver and /var/log/hmailserver) read-only to the service: a data, backup, archive or ACME directory elsewhere needs a drop-in (sudo systemctl edit hmailserver). HM5094: the MySQL or MariaDB client library is not installed. HM5603 on every TLS listener: /usr/bin/dh2048.pem is missing; reinstall the package. The Diagnostics self-test Monitoring & troubleshooting → Diagnostics in the Control Panel; since 6.3.4 also the Deck's Diagnostics view and POST /api/v1/diagnostics/run, on either platform. Ten tests: Test Fails when Server details Never; since 6.3.4 it also says whether the database connection is encrypted Installation paths (6.3.0) A configured path does not exist Test IPv6 Never Test outbound port Outbound 25 is blocked, or the smart host's name, port or TLS mode is wrong Test backup directory No destination, or the service cannot write there Test MX records Public DNS is wrong, or the resolver is not answering Test local connect Hairpin NAT fails, 25 is not forwarded, or the MX points elsewhere Test message file locations Files outside the data directory, or full paths stored, which stops the built-in backup Test IP range configuration 127.0.0.1 has been auto-banned Error logs Any ERROR* file exists Fill both domain boxes first: the MX and local-connect tests, and the outbound test without a smart host, are skipped rather than failed when they are empty. Copy puts the result on the clipboard for a post. "Did we ever get it?": the message trace The message trace records what the server did with each message, keyed on its queue id. It is off by default (MessageTraceEnabled), because it records who corresponds with whom, and nothing is recorded while it is off: turn it on before you need it. Turning it on. Since 6.3.4 it applies when saved: the Control Panel's Message trace card on Monitoring & troubleshooting → Logging, PUT /api/v1/settings/ini/{name}, or hmctl, both with the administrator password: hmctl ini set MessageTraceEnabled 1 On 6.3.3 and earlier it is MessageTraceEnabled=1 under [Settings] and a restart. Rows are kept MessageTraceRetentionDays (30): since 6.3.4 an hourly task deletes older ones; on 6.3.3 and earlier only Remove expired on the Message trace page, or MessageTrace.DeleteExpired over COM, did. Reading it. The Control Panel's Message trace page searches an address as sender or recipient, newest first, and Follow this message shows one queue id. Over REST since 6.3.4: GET /api/v1/message-trace, /api/v1/message-trace/address/{address} and /api/v1/message-trace/queue/{id}, which need the view_message_trace permission. hmctl api GET /api/v1/message-trace/address/[email protected] What a row says. One row per recipient per delivery attempt: when, the event, sender, recipient, source address and status code. Events include delivered and failed; on 6.3.4 also deferred (an attempt that will be retried) and suppressed (stopped by the suppression list). A refusal at RCPT carries queue id 0. A forward, a list or a rule that re-sends makes a new queue entry. Answering the question. A delivered row: you had it; look at the account's folders and rules. A row with queue id 0: refused at RCPT, and the status says why. No row: either it never reached you, or it is still queued or held, came while the trace was off, or is past the retention. The trace records delivery attempts only, with no acceptance, spam or virus event, so an empty trace does not prove the server never had the message. The per-domain reports' traffic, failures, senders and recipients are counted from it, and say so while it is off: Alerts, the audit trail and per-domain reports. Things that look like faults and are not What you see Why A new sender's first message is minutes late Greylisting doing its job (6.3.4) A value typed into [Settings] is back after a restart, HM5804 in the ERROR log The database is the settings store: set it in the Control Panel, over REST (PUT /api/v1/settings/ini/{name}) or with hmctl ini set (6.3.4) HM5807 at every start A [SettingsOverride] section is forcing keys; remove it once the stored value is right (6.3.4, Linux package) HM5803 at every start The service cannot write its copy into the root-owned file; the stored value is in effect 6.2.28 to 6.3.3: "The update feed answered HTTP 404." Set UpdateFeedUrl=https://updates.progressiverobot.com/releases.json and restart "Release 6.3.4 carries no Sigstore bundle for its installer, so it cannot be verified." Install 6.3.4 by hand: Upgrading to 6.3.4 Where a message waits, since 6.3.4. Neither delivered nor queued may mean held on purpose: by the sandbox, by compromised-account detection, in a back-off window, under an IP warm-up ceiling or behind a drain; Diagnosing stalled mail says where to look. And 550 5.7.1 Too many delivery attempts before the greylisting delay had passed is greylisting's abuse lock, lifted after GreyListingAbuseLockMinutes (60); a 5.7.10 bounce to a partner domain is gateway encryption refusing to send in the clear, the partner's key having expired. What it does not do It does not diagnose a mail client (Connecting a mail client), read logs line by line (Reading the logs), or work a stalled queue. The Control Panel and COM are Windows-only. When nothing here names the fault, Read this before you post says what a request needs. Without a GitLab account, email the Service Desk, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue. Where to read more Wiki: Troubleshooting, Ports Reference, Monitoring and Health, Installing hMailServer, Installing on Linux, The Control Panel, The REST API, Upgrading hMailServer, Architecture. Documents: ServiceAccount.md, DatabaseTls.md, Reports.md, SettingsApply.md. On the forum: Settings: when a save takes effect (HM5804 and the reinitialise), IP ranges, auto-ban and lockout, Choosing a database and encrypting the link (TlsMode, HM7530 and HM7531) and The hmailserver command line on Linux (the unit and its drop-ins). Public documentation: Troubleshooting and Troubleshooting a Linux installation.
  • ProgressiverobotP

    Reading the logs: levels, files, the Logs page, following a connection, and fail2ban

    General support logs diagnostics linux windows systemd
    1
    0 Votes
    1 Posts
    143 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) fixes the SMTP log writing some credentials in the clear with SMTP logging on: an AUTH PLAIN sent before EHLO or not in its one exact form, the bearer token of AUTH XOAUTH2 and AUTH OAUTHBEARER, and the lines of SCRAM-SHA-256, OAUTHBEARER and GSSAPI went into the log as the client sent them, base64 and readable, in every version from 6.0.0 to 6.3.6. An SMTP log written by 6.3.6 or earlier may hold them: redact every line of a SASL exchange before posting one. A database write that failed is now reported (HM5032) where it was taken as done, so an error log that was quiet may name failures that were hidden; and the IMAP and POP3 listeners can take the client's address from a PROXY protocol header, as the SMTP listener does, both off by default. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) no longer reports a stop of the servers as errors of their own: work that outlasted a stop, a reinitialise among them, writes a line to the debug log where it wrote HM5136 or another code to the error log and ran an installed OnError script, so the error log of a server that reinitialises may be quieter; and an event script's call that meets a stop fails with "The server is stopping. The request was not completed.", its script error going to the debug log. A failed account sign-in at the REST listener from loopback, as through a reverse proxy on the same machine, is counted against the name alone and writes no Auto-ban: logon failure line for fail2ban to read. See the 6.3.8 announcement. This topic covers what hMailServer writes to disk and which file to open, the settings that decide how much is written and kept, the live views, following one connection (since 6.3.4), a copy to syslog or the journal, and the fail2ban filter the Linux package installs. The one thing to know first: open ERROR_hmailserver_<date>.log before anything else. It holds only the entries the server itself classed as an error, and its mere existence is a diagnostic, because the Control Panel's built-in self-test (Monitoring & troubleshooting → Diagnostics, so Windows) fails when any ERROR* file is present in the log directory. Where the logs are Platform Log directory Set by Windows C:\Program Files\hMailServer\Logs by default [Directories] LogFolder in hMailServer.INI Linux /var/log/hmailserver, created by the package as 0750 hmailserver:hmailserver and left in place when the package is removed [Directories] in /etc/hmailserver/hMailServer.ini The format is the same on both platforms. On Linux the journal (journalctl -u hmailserver) carries the startup line, anything written to standard error and the operational events; a memory-safety fault is recorded in /var/log/hmailserver/crash-oracle.log as one line (signal, address, thread, event number) before the process dies by the signal it caught. [Directories] is per machine and is never mirrored into the database, so do not copy another server's file over this one's. The files File Holds Rotates hmailserver_<date>.log Everything: application lines, and the protocol conversations unless they are split out Daily, by name ERROR_hmailserver_<date>.log Only entries the server classed as an error Daily, by name hmailserver_SMTP_<date>.log, hmailserver_IMAP_<date>.log, hmailserver_POP3_<date>.log The per-protocol conversations, when SepSvcLogs=1 Daily, by name hmailserver_backup.log Every backup and restore run, line by line Never: one growing file. LogDeleteDays matches it by name but deletes only a file nothing has written to for that many days hmailserver_events.log Event-script output Never, as above hmailserver_awstats.log The AWStats-format feed, when AWStats logging is on Never, as above hMailServer_messagestore_consistency.report The most recent message-store consistency scan: one line per message whose file is missing, as messageid<TAB>account<TAB>expected-path Overwritten each run The backup and event logs are among the files the server marks with a byte-order mark; they are UTF-16LE on both platforms, so a file written on one reads on the other. A log line looks like "IMAPD" 5220 41 "2026-09-17 09:00:00.120" "192.0.2.7" "SENT: A1 OK LOGIN completed", where IMAPD is the category and 41 is the session id that every line of that conversation carries. The conversation categories are SMTPD, SMTPC, POP3D, POP3C and IMAPD; APPLICATION, TCPIP, DEBUG and ERROR are the rest. The settings Setting Default What it does What it looks like when it is wrong LogLevel 9 Not the choice of log types, which is the Logging page's tick-boxes (or PUT /api/v1/settings/logging). The Settings Reference records two effects: long lines are cut only when it is 2 or lower with debug off, and IMAP SENT: lines for FETCH and STATUS are logged only when it is above 2 or debug is on — Debug messages (the Logging page) off Adds the LOG_DEBUG lines, including the accept-pipeline stage timings Left on, a busy server fills the disk in days, and a full disk stops mail SepSvcLogs 0 Splits SMTP, IMAP and POP3 into their own files Off, so a search for one protocol wades through all three LogDeleteDays 0 Deletes files named hmailserver_*.log or ERROR_hmailserver_*.log whose last write is older than this many days, and nothing else in the directory. 0 keeps everything for ever Left at 0 on a server with debug logging on: the usual cause of a full log volume JsonLogging 0 Writes JSON lines, for Elasticsearch, Loki or Splunk Overridden while Logging.LogFormat is NCSA: the pipeline gets NCSA lines and no error SlowQueryLogMilliseconds 0 (off) Logs, and counts in hmailserver_db_slow_queries_total, any database statement slower than this Off, so a database that has become slow leaves no trace but the latency histogram WindowsEventLogLevel 2 How much also goes to the Windows event log — Where they are set. On Windows, Monitoring & troubleshooting → Logging in the Control Panel. On either platform the table's [Settings] keys (all but Debug messages) are set with PUT /api/v1/settings/ini/{name} or hmctl ini set. On Linux there is no Control Panel, so the log categories, debug messages among them, are set with PUT /api/v1/settings/logging, hmctl settings set logging or the Deck's Settings page. There, WindowsEventLogEnabled=1 is a Windows inheritance in name only, because on Linux it means syslog, and the operational events reach the journal through it. Since 6.3.4 these keys are stored in the database, whose value is the one used: where the service can write hMailServer.INI, a line edited there is put back and reported as HM5804. How the settings store works has the rest, [SettingsOverride] (HM5807) among it. Rotation. On Windows, LogDeleteDays is the rotation. On Linux the package installs /etc/logrotate.d/hmailserver; the rule uses copytruncate because the server does not reopen its log on a signal, and the rule says so at length. Debug logging: on to diagnose, off afterwards. The stage-timing lines that diagnosing stalled mail depends on are debug lines; the slow ones (ten seconds or more) are written at application level as well. Leaving debug on matters because a full disk stops mail: the application log warns below DiskSpaceWarningThresholdMB (1024 MB), and below MinimumFreeDiskSpaceMB (100 MB) on the message store's volume the server answers 452 4.3.1 Insufficient system storage. and senders retry. The live views Windows: the Control Panel. Monitoring & troubleshooting → Live logs is a real-time, colour-coded tail of what the server is writing; every page in that group but Discovery reads the running service over COM (Discovery has no COM interface, so it reads the REST listener and needs RestApiPort set). Diagnosing stalled mail, in the same group, has a one-click debug-logging toggle. Both platforms: the Control Deck and hmctl. The browser Control Deck at / on the REST listener has a Logs page that lists the log files and shows any file's last 200 or 2,000 lines, without a share to the log directory. Since 6.3.4, hmctl log list and hmctl log tail <name> --lines N read the same over GET /api/v1/logs and GET /api/v1/logs/{name}, and work against a Windows or a Linux server because they go over HTTP rather than DCOM. The Linux packages install hmctl as /usr/bin/hmctl; the HmailServer PowerShell module is the same vocabulary for PowerShell. The files themselves, from a shell. On Windows: $log = "C:\Program Files\hMailServer\Logs\hmailserver_" + (Get-Date -Format "yyyy-MM-dd") + ".log" Get-Content $log -Tail 0 -Wait | Select-String "SMTPD|SMTPConnection" On Linux: tail -f /var/log/hmailserver/hmailserver_$(date +%F).log journalctl -u hmailserver -f Following one connection, since 6.3.4 Every SMTP, POP3 and IMAP conversation is a session with a number, and every line the log writes for it carries that number. Until 6.3.4 that meant searching the day's log for the number afterwards. Since 6.3.4 the Connections section at the top of the Deck's Logs page lists every connection the server has open now, newest first, and the last two hundred that ended: the id, the protocol, the address it came from, the port it came in on, the account once it has signed in, and when it started or ended. Follow beside any of them opens a pane holding every line the log has for that session and keeps adding lines as the server writes them; Copy puts the whole pane on the clipboard; Stop following ends it. A session that ended a second ago is still listed, so an SMTP conversation that lasted a quarter of a second can be picked and read. How it works, and what that means for you. The server keeps the last 4,096 log entries in memory, across every session; the ring is always on, and nothing survives a restart. A debug-logged IMAP client can fill it in minutes, so a follow whose cursor points before the oldest entry still held answers dropped: true and starts from what is left; the page says so. A line older than that is in the log files and nowhere else. The conversation lines (SMTPD, SMTPC, POP3D, POP3C, IMAPD) carry the session id in their own column. DEBUG and TCPIP lines carry none and are filed by the first number after the word session in their text (TCP connection started for session 41, Ending session 41); that is an inference, and the text is not always the server's own. A line that does neither is nobody's and is not shown. It shows what the log mask lets through, and no more. With IMAP logging off, an IMAP session is listed and its follow is empty; the answer's logged field says whether the log mask lets that protocol's lines through, and the page names the switch to turn on. Debug logging is what makes the TLS handshake, the timeouts and the disconnect appear beside the conversation. The connections the server opens itself are listed too, marked client: SMTP client, POP3 client, IMAP client, and the short-lived SpamAssassin and Diagnostics connections. Not listed: anything that is not a TCPConnection, so the REST API, the webmail, the DAV services, ManageSieve and the anti-virus scanner's socket to clamd are not sessions here. A delivery to a host that refuses the connection is not listed either, though with debug logging on its lines can still be followed by id, with known: false. A follow is a long poll, not a stream: GET /api/v1/logs/sessions/{id}?after=0&wait=10 answers the session's lines after sequence 0 and a cursor to ask after next time, waiting up to wait seconds for the next line. The wait is at most twenty seconds, and one waiting follow at a time is allowed, because a waiting follow holds one of the HTTP listener's four worker threads, which also serve the webmail; a second follow is answered at once with waited: false. An answer carries at most 500 lines, and the pane keeps the last 5,000. GET /api/v1/logs/sessions lists the sessions (the newest thousand when more are open, with truncated: true), with held_lines and logging_enabled. Both routes are under the view_logs permission, which the built-in read-only auditor role holds, and both are server-wide: a credential confined to some domains is refused them, as it is refused the log files. There is no Control Panel page and no hmctl verb for this; the Control Panel's Live logs is the older, whole-log form over COM. There is no filtering inside a session. A copy to syslog, or to the journal, since 6.3.4 Since 6.3.4 every entry the server logs can also be sent as RFC 5424 to a syslog collector or, on Linux, where the server runs as the packaged systemd unit with no collector host named, to the journal. It is a copy: the files above are written exactly as before, and nothing it does can stop a line reaching them, or delay or lose a message. Its settings are hm_settings rows, not [Settings] keys, read when the sink starts (the next service start or reinitialise), not when they are saved. With the journal on, journalctl -u hmailserver HMAILSERVER_SESSION=41 follows one session from the journal, across a restart. Alerts, the audit trail, syslog and the journal has the twelve settings, the transports, the rsyslog configurations and the journal's fields. Codes in the error log worth knowing since 6.3.4 Code What it says, and what to do HM5804 A value typed into hMailServer.INI [Settings] for a key the database holds was put back at the last start. Change it in the Control Panel, with PUT /api/v1/settings/ini/{name}, hmctl ini set or COM Settings.SetIniSetting HM7460 AutoBanFirewall is on and the Windows service account cannot write firewall rules (below) HM7495 The database records a stored-secrets key this server does not hold, and every account's second factor refuses every code: import the exported key with hMailServer.exe --import-stored-secrets-key FILE (elevated) or sudo hmailserver --import-stored-secrets-key FILE, then restart HM7540 A restore was refused because the archive holds an entry with an absolute path, a .. segment or a link with somewhere to point; nothing was extracted. Find out where the archive came from fail2ban on Linux, and the Windows contrast An auto-ban (repeated failed logons from one address) is enforced by the server's own listeners: the kernel accepts the connection and the server refuses it. Shipped in 6.3.2, the Linux package offers two ways to push the ban below the server, and one is enough. fail2ban needs no capability at all, because it runs as root and reads the application log. The package installs /etc/fail2ban/filter.d/hmailserver.conf and a jail at /etc/fail2ban/jail.d/hmailserver.conf, disabled as Debian asks of a shipped jail. The filter matches the two fixed-shape lines the server writes for every counted failure and every ban, whether or not protocol logging is on; auto-ban on logon failure (AutoBanOnLogonFailureEnabled, on as shipped; auto_ban_on_logon_failure over REST) is what writes them. Turn it on: printf '[hmailserver]\nenabled = true\n' > /etc/fail2ban/jail.d/hmailserver.local systemctl reload fail2ban fail2ban-client status hmailserver The server itself, with AutoBanFirewall=1, makes the firewall match its auto-ban ranges at start, on every ban, on every range deletion and once a minute. On Linux it runs /usr/lib/hmailserver/autoban-hook, which keeps an nftables set with a timeout (iptables where there is no nft) and drops TCP from the banned address to the ports the server listens on. Changing the firewall needs CAP_NET_ADMIN, which the unit deliberately does not grant a mail server by default. Grant it once with a drop-in, and the hook's check verb says whether it can act: systemctl edit hmailserver [Service] AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_ADMIN CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_NET_ADMIN systemctl restart hmailserver sudo -u hmailserver /usr/lib/hmailserver/autoban-hook check AutoBanCommand, on either platform, runs a program of yours instead of, or as well as, AutoBanFirewall (<command> ban <address> <minutes> <ports> and <command> unban <address>) for a cloud firewall or a router. Before turning either on, list the addresses you administer from in AutoBanNeverBan: a ban that reaches the firewall locks that address out of every mail port on the machine. Since 6.3.4 AutoBanFirewall, AutoBanCommand and AutoBanNeverBan are stored settings like the other [Settings] keys: set them over the API or with hmctl ini set, not in the file. The fail2ban filter and the hook are part of the Linux packages. On Windows, AutoBanFirewall writes one inbound block rule per banned address in Windows Defender Firewall, named hMailServer auto-ban <address> and scoped to the ports the server listens on, and since 6.3.4 a new installation's service account, NT SERVICE\hMailServer, cannot write them: HM7460 is logged once a start and no rules appear, while the listeners still refuse a banned address. Turn AutoBanFirewall off there, or use AutoBanCommand towards a router or cloud firewall; HM7461 names rules a LocalSystem run left behind, with the PowerShell that removes them. What to include in a report SUPPORT.md asks for, in order of value: the relevant log, not a description of it, with debug logging on and the window around the problem; the ERROR_hmailserver_<date>.log for the same window; the exact version, the platform and the database backend; and what you expected instead, and whether it is reproducible. Remove passwords, keys and real email addresses first, because a posted log is public. Read this before you post asks for the same log lines and adds the build number, the package, whether the server was upgraded and the Diagnostics output; Copy on the Diagnostics page puts the whole result on the clipboard. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email. Where to read more Wiki: Monitoring and Health, sections 16.2 and 16.2a; Troubleshooting, sections 19.10 and 19.11; Installing on Linux, for the journal and fail2ban. Repository: FollowConnection.md, Syslog.md, SUPPORT.md. The public documentation: Logs on disk, Logging and log rotation on Linux, Follow one message end to end in the logs, Codes in the error log. On this forum: hMailServer on Linux: what you get and what you do not, The Control Deck, Diagnosing stalled mail, and IP ranges, auto-ban and lockout for when a ban is made and how to lift one.
  • ProgressiverobotP

    Diagnosing stalled mail: a sender that times out, and a message that never leaves

    General support diagnostics logs smtp spam
    1
    0 Votes
    1 Posts
    131 Views
    ProgressiverobotP
    Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) adds reasons a message stays where it is. A forward, a redirect or a rule's forward of mail from outside whose From names the forwarding domain's own organisation, and that did not pass DMARC for it when it arrived, keeps the envelope sender it came with, with no SRS address or rewritten sender; where the copy would still carry an identity of that organisation (an empty envelope sender under this server's host name, or an envelope sender at this server's host name or at a hosted domain in that organisation) it is not sent to an outside address at all: the message stays in the mailbox, and the application log names each recipient. RuleLoopLimit 0 now stops out-of-office replies, as it stopped every other generated message. A delivery whose mail servers' DNS lookup failed tells the sender that the DNS query failed (4.4.4) and retries, where a failed lookup was read as a name with no addresses. On MySQL and MariaDB a lost database connection is reconnected and the statement run again, and an error log that was quiet may now name database failures that were hidden (HM5032). See the 6.3.7 announcement. This topic covers the case where mail is not moving and the server looks healthy: no crash, no error, the service running, and a log that seems to stop mid-transaction. The server used to be silent about it; it is no longer, and the answer is usually in one line of the log. The one thing to establish first is which half is stuck: the server accepting a message from a sender, or delivering one it has already accepted. The two fail differently, for different causes, under different settings. Which half is stuck? Accepting. The sending server connects, sends the message, and then waits. From its side there is a timeout after the message body was transmitted; Postfix reports it as timed out while sending end of data. The message never appears in your queue. Delivering. The message was accepted (the sender got a 250), is visible in the delivery queue, and never leaves. Monitoring & troubleshooting → Delivery queue in the Control Panel tells the two apart in one look, and so does hmailserver_delivery_queue_messages on the metrics endpoint once it has a port (see Commands). The Control Panel is Windows-only; on Linux the queue is GET /api/v1/queue, hmctl queue list (since 6.3.4) and the Control Deck's queue page. Turn on debug logging first Control Panel → Logging → tick Debug messages, and make sure logging is enabled; then reproduce the problem once. The stage lines below are written at debug level. The slow ones (ten seconds or more, for a whole stage or for any single spam test) are also written at application level, so a server with only application logging still records the important half. Turn debug logging off again afterwards on a busy server. Reading the logs covers where the logging keys live on each platform. Accepting: the sender times out after sending the message Between the 354 and the 250 the server runs the accept pipeline: spam tests, message modifications, archiving, the OnAcceptMessage script and the database save. All of it runs on a bounded pool of threads, and the reply is sent only when it finishes. The log times each stage: SMTPConnection - accept: start spam-protection. Spam test: SpamTestDNSBlackLists, Score: 0, Time: 12 ms Spam test: SpamTestSpamAssassin, Score: 0, Time: 120000 ms SMTPConnection - accept: done spam-protection in 120047 ms (session 5). SMTPConnection - accept: done message-modifications in 0 ms (session 5). SMTPConnection - accept: start script/save. SMTPConnection - accept: done script/save in 3 ms (session 5). How to read it. Read it as a sequence and look for the last line written; the stage that is stuck is the one after it. Two stages announce themselves with a start line (spam-protection and script/save), so for those a start with no matching done is the whole diagnosis. The middle stage, message modifications, has no start line, so a stall there shows as done spam-protection followed by silence. If every stage completes but one took a long time, the Time: or done ... in value names the culprit. What the log shows Cause SpamTestSpamAssassin with a large Time: spamd is unreachable, overloaded, or accepting connections without answering SpamTestDNSBlackLists, SpamTestSURBL or SpamTestSPF slow The resolver is not answering. Look in the TCP/IP log for DNS - Query timed out and DNS - Query failure. Do not reach for DNSServer as the fix (below) done spam-protection, then silence Message modifications: the spam headers, the signature, the List-* headers, or the write of the modified message back to disk script/save slow An OnAcceptMessage event script, or the database Nothing between 354 and silence A version older than 6.2.17, which added these lines. Upgrade The DNSServer setting. It was broken in 6.2.17 and 6.2.18 (CHANGELOG's 6.2.19 section names 6.2.16 as well) and fixed in 6.2.19: on the affected versions every lookup through a configured custom server failed, and DNSBL, SURBL, SPF, DKIM and MX lookups failed in silence, so such a server had quietly stopped most of its spam filtering while still accepting mail. One caveat survives the fix: a custom server list is only honoured together with DNS_QUERY_BYPASS_CACHE, so every lookup pays full price. Leave DNSServer empty unless you need a specific resolver. SPF, and SPF on Linux. In 6.3.3 and earlier the SPF evaluator resolved names through the Windows DNS client rather than through the server's resolver. On Windows an SPF check therefore went to the system resolvers whatever DNSServer said; on Linux every SPF check returned a temporary error, reported once per process as HM6406, and because SpamTestDMARC evaluates SPF, DMARC there lost its SPF half. The RFC 7208 evaluator that replaced it in 6.3.4 resolves through the same resolver as every other lookup. If you run 6.3.3 or earlier on Linux, an SPF or DMARC verdict from it is not a verdict. The deadline. Since 6.2.17 acceptance is bounded: if the pipeline exceeds FinalizationTimeout (240 seconds by default) the server answers 451 4.3.1 Server temporarily overloaded while accepting the message; please retry. and the sender retries rather than waiting for a reply that never comes. The accompanying error entry names the deadline that was hit, and the stage timings above it name what consumed the time. If every message stalls, not just one Look for these two lines: Task SMTP-accept session=42 ip=203.0.113.9 waited 8 seconds for a thread in work queue Asynchronous task queue. 14 task(s) are still queued. All 15 threads in work queue Asynchronous task queue have been busy for at least 120 seconds, so no further task on this queue can start. 14 task(s) are queued. Running: SMTP-accept session=42 ip=203.0.113.9 (thread 4120, 168s), ... Every worker is occupied, and one slow dependency does this to the whole server. The task names say which sessions are stuck; the stage timings say what they are stuck on. AsyncQueueStallThreshold (120 seconds by default) is how long every worker may be busy before the second report is written; 0 turns the report off. Delivering: how the queue works There is no in-memory queue to inspect. The delivery queue is the set of rows in hm_messages whose type is delivering, and the delivery manager is a loop that polls for them. Three properties of that loop explain most of what a stuck queue looks like: The selection is where messagetype = 1 and messagelocked = 0 and messagenexttrytime <= now, ordered by size, then try count, then id. Small and new goes first. The delivery pool is separate and smaller than the accept pool: maxdelivertythreads, 10 by default, on the work queue named SMTP delivery queue. It is not the Asynchronous task queue that finalises inbound mail, so a saturated delivery pool does not stop mail being accepted; it stops it leaving. The manager wakes on demand, or once a minute otherwise. A message whose next-try time has just passed is picked up within a minute, not instantly. At startup every message in the queue is unlocked (update hm_messages set messagelocked = 0), because locks held by a server that crashed would otherwise strand those messages for ever. That is also why exactly one hMailServer service may run against one database. The retry schedule, and when a message gives up Setting Where Default Effect Number of retries Mail flow & delivery → Delivery of e-mail (smtpnoofretries) 4 How many further attempts after the first Minutes between every retry The same page (smtpminutesbetweenretries) 60 The gap between them QuickRetries [Settings] 0 (off) The first N retries use the short gap below instead, for greylisting QuickRetriesMinutes [Settings] 6 That short gap QueueRandomnessMinutes [Settings] 0 (off) Adds a random 1..N minutes to every gap, so a burst does not retry in lockstep Route number of tries and minutes between tries Mail flow & delivery → Routes — Overrides both, per matching route; where several routes match, the largest of each wins With the shipped defaults that is five attempts an hour apart: a destination that is refusing temporarily holds a message for roughly four hours and then bounces it. The try count in the queue is the same messagecurnooftries the arithmetic uses. Two things do not advance that counter: Deliver now resets the next-try time without touching the count, and a message held because the virus scanner could not examine it is held on its own budget (below). Reading a stuck queue What the queue looks like What it means Where to look next Depth climbing, oldest message seconds old A busy server draining normally. Not a fault — Depth flat, oldest message hours old, try counts climbing The far end is deferring you The SMTP log for that destination; your IP's reputation and PTR Every stuck message shares one recipient domain That destination, not your server The route settings for it, and its MX Try counts stuck at 0, next-try times in the past, nothing moving Delivery threads are not running the tasks hmailserver_workqueue_depth{queue="SMTP delivery queue"}, and the saturation report above Messages appear and vanish without arriving Delivered to the wrong place, or discarded by a rule Mail flow & delivery → Rules; enable MessageTraceEnabled and read the trace Try count 1 and a next-try time of 1901-01-01 An ETRN hold: the one route these recipients match is an ETRN route Mail flow & delivery → Routes; the log line is SMTPDeliverer - Route Message: HOLD for later delivery.. Queue empty, but the sender got a 250 and nobody received it Delivered locally into a folder you are not looking at, or the row was removed The account's folders; the message trace Message N could not be delivered since the data file does not exist The row and the store disagree: a restore that missed files, or a store on a share that vanished Turn on MessageStoreConsistencyCheck and read the report When the virus scanner cannot examine a message. With AVFailAction=1 such a message is not delivered. It is held on its own schedule rather than on the SMTP retry budget: AVFailRetryMinutes apart (15 by default), at most AVFailMaxHolds times (16), and then returned to the sender. The log says where in that budget each hold is (... this was hold 3 of 16 before it is returned to the sender.). Each hold counts as a deferral in hmailserver_messages_deferred_total, so a scanner that has stopped answering shows as a deferral rate with no matching delivery rate. The usual causes of a genuinely stuck delivery A virus scanner that stops responding. ClamAV is contacted after the message is accepted, so a wedged clamd shows up as accepted but never delivered. Each socket operation is bounded by ClamMinTimeout / ClamMaxTimeout, but that is a deadline on one read or write, armed fresh each time, and the message is streamed in chunks, so one large message can take considerably longer than ClamMaxTimeout in total. A remote server that answers extremely slowly. The idle timeout is re-armed on every byte received, so a host that sends one byte occasionally used to hold a delivery thread indefinitely. Since 6.2.18 there is an absolute ceiling, ClientSessionCeiling (30 minutes by default), armed once and never re-armed. A custom virus scanner or external tool that hangs, bounded by ExternalProcessTimeout. A message held for a sandbox's verdict (since 6.3.4). With SandboxEnabled on, a message none of your own signed-in users submitted, whose attachment is of a submitted type, waits for the sandbox you named to open the file, up to SandboxTimeoutMinutes (15 by default). While it waits it is in no mailbox, not in the quarantine and not the queue's to retry: its Retry and Delete answer 409. It is listed by GET /api/v1/sandbox/held, by GET /api/v1/queue marked held: sandbox (the held column of hmctl queue list), and in the Deck's queue page under Held for a sandbox, where it can be delivered without the verdict or discarded. The database. Check for errors mentioning the connection pool. Since 6.3.4 a message can also wait in other places, each on purpose: held for review by compromised-account detection in enforce (/api/v1/review); in a destination's back-off window, with the last error Delivery deferred by this server's own shaping, not by a remote server. (GET /api/v1/queue/shaping; such a deferral does not spend the message's retries); under an IP warm-up ceiling; or behind a drain, when /readyz says draining. Section 19.11a of the Troubleshooting page lists each with what to do. Settings that bound each stage All are in seconds. On 6.3.3 and earlier they are [Settings] keys in hMailServer.ini. Since 6.3.4 they are stored in the database and changed in the Control Panel, over the REST API at /api/v1/settings/ini/<name> or with hmctl, and the save applies them. They still appear in the file's [Settings] section, but the database's value is the one used: where the service can write the file, an edit made there is reported (HM5804) and put back at the next start. Setting Default Bounds 0 means FinalizationTimeout 240 The whole accept pipeline, after which the sender gets a 451 no bound SAMaxTimeout 90 SpamAssassin: idle timeout, and a session ceiling of this plus 30s not "no bound" ClamMaxTimeout 90 ClamAV: idle timeout per socket operation not "no bound" DNSQueryTimeout 10 A single DNS query no bound ScriptTimeout 60 One event script invocation no bound ExternalProcessTimeout 300 An external scanner process no bound ClientSessionCeiling 1800 An entire outbound delivery session no bound AsyncQueueStallThreshold 120 How long every worker may be busy before the saturation report reporting off DBConnectionAcquireTimeout 60 Waiting for a pooled database connection no bound The two exceptions matter, because 0 tightens them instead of removing them. SAMaxTimeout and ClamMaxTimeout are not passed straight through: they go to a calculator that returns the minimum whenever the maximum is lower than it. So SAMaxTimeout=0 gives a 30-second idle timeout (SAMinTimeout) and a 30-second session ceiling, and ClamMaxTimeout=0 gives 15 seconds (ClamMinTimeout). To lengthen either, raise the value; to lengthen it a long way, raise the matching ...MinTimeout too. There is no way to make either unbounded. When the pool deadline expires, a recipient lookup answers 451 4.3.2 rather than 550: the server can tell "the database did not answer" from "no such address", so a database locked by a backup defers the mail instead of bouncing it. Commands The COM examples below are Windows-only. The metrics endpoint and the REST routes work on both platforms. Is it accepting, or is it delivering? The metrics listener is off until MetricsServerPort is set (9100 here; since 6.3.4 the port, the bind address and the listener's other settings, its credentials included, wait for a reinitialise). Once it is on, the endpoint needs no credential on a loopback bind: (Invoke-RestMethod http://127.0.0.1:9100/metrics) -split "`n" | Select-String '^hmailserver_delivery_queue_(messages|oldest_message_age_seconds) |^hmailserver_workqueue_depth|^hmailserver_messages_(delivered|deferred|bounced)_total ' A rising oldest_message_age_seconds with a flat delivered_total is the delivering half. A rising hmailserver_workqueue_depth{queue="Asynchronous task queue"} is the accepting half. The COM examples start from a signed-in $app: $app = New-Object -ComObject hMailServer.Application $app.Authenticate("Administrator", "<password>") | Out-Null Push one message to the front of the queue (resets its next-try time, leaves the try count alone, and releases an ETRN hold): $q = $app.GlobalObjects.DeliveryQueue $q.ResetDeliveryTime(4711) $q.StartDelivery() Wake the delivery manager without touching any message: $app.GlobalObjects.DeliveryQueue.StartDelivery() Remove one message from the queue (it is locked first, so a message a thread already has is not pulled from under it): $app.GlobalObjects.DeliveryQueue.Remove(4711) Follow the log while you reproduce it by tailing the day's file and filtering on accept: |Spam test:|waited .* for a thread|have been busy|SMTPDeliverer; Reading the logs has the tail command for each platform. Turn the message trace on (MessageTraceEnabled, off by default) to see what was done with one message. It has no acceptance, spam or virus event, so an empty trace does not show that the server never had the message; Troubleshooting by symptom says how to turn it on and what a row says. What to include in a report If none of the above identifies it, post in General support or open an issue with: the log from the 354 (or from the delivery attempt) onwards, including the stage timing lines; the ERROR_hmailserver_<date>.log for the same window; which scanners and event scripts are enabled; and whether the message eventually arrives, arrives twice, or never arrives. Say which half you have. Read this before you post covers the rest. Without a GitLab account, email the report to the project's Service Desk address, contact-project+hmailserver-hmailserver-86730559-issue-@incoming.gitlab.com, which GitLab files as a confidential issue; the maintainers answer by email. Where to read more Wiki: Diagnosing Stalled Mail and Troubleshooting, sections 19.7 and 19.11a. Repository: DiagnosingStalledMail.md; SettingsApply.md; Sandbox.md; SUPPORT.md. The public documentation: The delivery queue and the retry schedule and Troubleshooting. On this forum: Being trusted by the servers you send to, for when the far end is deferring you; Outbound shaping and IP warm-up, for when your own server is holding mail back; Monitoring and health, for turning the metrics listener on.

hMailServer Community

The community support forum for hMailServer — the free, open-source mail server for Windows and Linux — maintained by Progressive Robot Ltd. The software is AGPL-3.0 and stays that way; this is where it is discussed, debugged and documented.

The software

Getting help

Legal

Progressive Robot Ltd · 220 Chester Business Park, Heronsway, Chester, Cheshire CH4 9GB, United Kingdom · +44 (0)1244 911212 · [email protected] · VAT 506152326

Powered by NodeBB