Monitoring and health: Prometheus metrics, health probes, Grafana, OTLP and SNMPv3

Progressiverobot

Written for 6.3.4. hMailServer 6.3.6 (6 October 2026) publishes the first container image, registry.gitlab.com/hmailserver/hmailserver:6.3.6, and a monitor that matches "The response carried no HTTP headers." needs the new sentence, which says who ended the connection, after how many bytes and seconds. See the 6.3.6 announcement. hMailServer 6.3.7 (17 October 2026) publishes registry.gitlab.com/hmailserver/hmailserver:6.3.7, tagged 6.3 and latest until 6.3.8; renames the Control Panel page that holds the metrics listener's settings from API & monitoring to Monitoring & alerts, with the alerts' settings, the outbound webhooks and the SNMP agent on it (Ctrl+K takes the old title to it); and refuses GET /api/v1/metrics/history, and the same history over COM, to a credential confined to named domains, such as a domain administrator, the auditor role given a list of domains or an API key issued for named domains. With IMAP and POP3 now taking the client's address from a PROXY protocol header as SMTP does, the High Availability Runbook says that a balancer's /readyz check must not carry that header: the metrics listener reads none and answers a request behind one 404. See the 6.3.7 announcement. hMailServer 6.3.8 (2 November 2026) publishes registry.gitlab.com/hmailserver/hmailserver:6.3.8, also tagged 6.3 and latest. See the 6.3.8 announcement.

This topic covers what hMailServer 6.3.4 offers a monitoring system: the metrics listener's Prometheus exposition and health probes, the repository's Grafana dashboard, the OpenTelemetry push, the server's own metric history, the new SNMPv3 agent and, briefly, the daily block-list check of the server's own addresses. The one thing to know first: the metrics listener is off until you give it a port, and once it is on, the health check to configure is /readyz, because it is proved by a real database round trip rather than by a flag.

The metrics listener

One small HTTP listener serves both the exposition and the probes. Its seven settings are [Settings] keys, set on the Control Panel's Monitoring & troubleshooting → API & monitoring page (Monitoring & alerts since 6.3.7; Windows only), with PUT /api/v1/settings/ini/{name}, with hmctl ini set, or, on a Linux server whose REST listener is not yet on, with hmailserver --set-setting NAME=VALUE; the topic on settings explains where they are stored and when a save applies. All seven are taken when the listener starts, the three credential settings included, so a saved change waits for a reinitialise (POST /api/v1/server/reinitialize, or Apply now in the Control Panel or the Deck) or a service restart. A token that has been saved and not applied leaves /metrics answering as it did before.

Setting	Default	Effect	When it is wrong
`MetricsServerPort`	`0`	`0` means the listener is never created	No port, no probes, no metrics, and no complaint anywhere
`MetricsServerBindAddress`	`127.0.0.1`	An IP literal only, parsed with `inet_pton`: `0.0.0.0`, `127.0.0.1`, `::` or a specific IPv6 address	A host name or `localhost` is rejected: the log says `MetricsServer: Invalid bind address`, the listener does not start and the probes go with it
`MetricsServerAuthToken`	empty	Sent by the scraper as `Authorization: Bearer <token>`	Absent on a non-loopback bind: `/metrics` answers 503, not 401, and the body names the settings that would open it. The probes keep answering
`MetricsServerAuthUsername` and `MetricsServerAuthPassword`	empty	HTTP Basic; with both schemes configured, either satisfies the check	As above
`MetricsServerCertificateFile` and `MetricsServerPrivateKeyFile`	empty	Both set makes the whole port HTTPS, probes included	Set but unusable: the listener still serves the probes in the clear and `/metrics` answers 503

:: serves both address families; a specific IPv6 literal serves IPv6 only. "Loopback" means the whole of 127.0.0.0/8 and ::1.

The probes are answered before any branch that can refuse. A Kubernetes httpGet probe, an ELB or ALB health check and a Docker HEALTHCHECK have nowhere to keep a bearer token. Refused credentials on /metrics are counted in hmailserver_metrics_unauthorized_requests_total rather than logged.

Nothing on the request path touches the database or the disk. A refresher thread probes the database every 5 s (every 30 s after a failure), refreshes the delivery-queue aggregates at most every 10 s and only when a scrape has asked for them, and re-reads the certificate files every 5 minutes. One exception since 6.3.4: the two message-store gauges are cached for an hour and recomputed by whichever caller finds them stale, so about once an hour a scrape can run select sum(messagesize), count(*) from hm_messages on the listener thread itself: on a large store, a slow answer to that scrape and to any probe queued behind it.

It is a single accept loop serving one connection at a time. A request's header read is bounded at 5 s and a response write at 15 s. For a load balancer's check the High Availability Runbook gives an interval of 5 s and a timeout of 2 s; never point a sub-second health check at it.

The three health probes

No credential in any configuration; HTTP/1.0 responses that close the connection.

Path	200 when	Body	Use it for
`/livez`	Always, if the listener answered; never 503	`alive`	Liveness restarts. It checks nothing on purpose: a database outage must not get a healthy process killed
`/readyz`	State is `Running` and the database has proved within the last 20 s that it can answer; 503 otherwise	`ready`, or `not ready: <reason>`	Load-balancer and VIP routing
`/healthz`	The same conditions	`{"status":"ok\\|unavailable","state":"...","database":"up\\|down","uptime_seconds":N}`	Dashboards and human debugging

Readiness is four conditions in order, each with its own reason: not ready: server not in running state; not ready: database not connected; not ready: database did not answer the last readiness probe; and not ready: database readiness probe has not completed recently. The last exists because a probe against a database that has stopped answering can hang for as long as DBConnectionAcquireTimeout allows, 60 s by default. Since 6.3.4 a server in maintenance mode answers 503 not ready: draining: POST /api/v1/server/drain enters it, GET reports {"draining","since","sessions","deliveries","drained"}, and DELETE leaves it.

Start-up is a refused connection, not a 503: the listener is brought up only after the state has gone to Running. Shutdown is a 503 first, then a refusal: the state goes to Stopping before anything is torn down, so /readyz reports 503 while ShutdownDrainSeconds runs.

The High Availability Runbook builds its active/passive design on this: both nodes have the listener on, with MetricsServerBindAddress=0.0.0.0 and a MetricsServerAuthToken, and the VIP is gated strictly on /readyz. The passive node's service is stopped, so its /readyz is refused and it is never routed to. Since the probes need no credential, a missing token shows as missing dashboards, not as a failed failover.

Every metric family

Thirty-eight metric families in 6.3.4 (thirty in 6.3.3). Counters are monotonic since server start; a restart resets them, which is what hmailserver_start_time_seconds is for.

Mail flow

Metric	Type	Meaning, and the trap in it
`hmailserver_processed_messages_total`	counter	One per delivery pass started, not per message received: a message deferred three times and then delivered counts four
`hmailserver_messages_delivered_total`	counter	Delivery passes that completed with no failed recipient
`hmailserver_messages_deferred_total`	counter	Passes rescheduled: a temporary failure, greylisting, or a message the virus scanner could not examine (`AVFailAction=1`)
`hmailserver_messages_bounced_total`	counter	Delivery-failure notifications queued to the sender; one can cover several recipients
`hmailserver_delivery_queue_messages`	gauge	Rows in `hm_messages` in the delivering state
`hmailserver_delivery_queue_oldest_message_age_seconds`	gauge	Age of the oldest queued message; `0` when the queue is empty. Alert on this, not on depth
`hmailserver_spam_messages_total`, `hmailserver_viruses_removed_total`	counter	Messages classified as spam; viruses removed during delivery preprocessing
`hmailserver_store_bytes`, `hmailserver_store_messages`	gauge	Since 6.3.4. The size and the number of the messages in `hm_messages`, recomputed at most once an hour
`hmailserver_domain_messages_received_total{domain}`, `hmailserver_domain_messages_sent_total{domain}`	counter	Only present when `MetricsPerDomainEnabled=1`; only hosted domains are ever labelled

Added in 6.3.4. hmailserver_country_blocked_total and hmailserver_country_blocked_by_country_total{country} count connections and sign-ins refused by a country policy, and hmailserver_spam_trap_hits_total counts hits on a declared spam trap. Three gauges cover the feedback loops' seven-day window: hmailserver_domain_complaints{domain}, hmailserver_domain_complaint_window_messages{domain} and hmailserver_domain_complaint_rate{domain}. They are present only when MetricsPerDomainEnabled is set and complaint processing is on. The rate gauge is a ratio, not parts per million: Complaints.md alerts on hmailserver_domain_complaint_rate > 0.001. Parts per million is the unit of the complaints.rate.high alert rule's threshold, so an expression written in parts per million against the gauge would never fire.

Sessions, authentication and TLS

Metric	Type	Meaning
`hmailserver_sessions{protocol}`	gauge	Concurrent sessions; `protocol` is `smtp`, `imap` or `pop3`
`hmailserver_auth_success_total`, `hmailserver_auth_failures_total`	counter	A rising failure rate against a flat success rate is a password-guessing run
`hmailserver_tls_handshakes_total`, `hmailserver_tls_handshake_failures_total`	counter	The mail protocols only; a failed handshake on the metrics port is never counted here
`hmailserver_tls_certificate_expiry_seconds{certificate}`	gauge	`notAfter` as a Unix timestamp per configured certificate, re-read every 5 minutes; at most 128 series. Absent entirely when no certificate is configured. With no certificate row, the ACME certificate is labelled `ACME (automatic)`
`hmailserver_metrics_unauthorized_requests_total`	counter	Always `0` when no credential is configured

The server and the database

Metric	Type	Meaning
`hmailserver_start_time_seconds`	gauge	Uptime is `time() - hmailserver_start_time_seconds`
`hmailserver_build_info{version,architecture,database_schema_version}`	gauge	Always `1`; the information is in the labels
`hmailserver_state{state}`	gauge	One series per state (`unknown`, `stopped`, `starting`, `running`, `stopping` and, since 6.3.4, `draining`), exactly one of which is `1`
`hmailserver_workqueue_depth{queue}`, `hmailserver_workqueue_blocking_tasks_waiting{queue}`	gauge	Seven fixed `queue` values: `Server queue`, `Maintenance queue`, `Asynchronous task queue`, `Name lookup queue`, `IOCPQueue`, `SMTP delivery queue`, `External fetch queue`. `Asynchronous task queue` sends the final `250`, so depth there is mail waiting to be acknowledged
`hmailserver_messagestore_missing_files`	gauge	Always `0` unless `MessageStoreConsistencyCheck=1`
`hmailserver_command_processing_seconds`, `hmailserver_db_query_seconds`	histogram	Bucket bounds from 0.0001 s to 1.0 s, and from 0.001 s to 10.0 s; both emit the full `_bucket`, `_sum` and `_count` set
`hmailserver_database_connected`	gauge	`1` when the database answered a real round trip recently
`hmailserver_database_probe_success_timestamp_seconds`, `hmailserver_database_probe_age_seconds`	gauge	When it last answered, and how long ago on the server's own clock; `/readyz` reports 503 once the age passes 20 s
`hmailserver_db_connections{state}`	gauge	`busy` and `available`; `available` pinned at zero is saturation
`hmailserver_db_slow_queries_total`	counter	Statements slower than `SlowQueryLogMilliseconds`; `0` while that setting is `0`

Scraping it

scrape_configs:
  - job_name: hmailserver
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    authorization:
      credentials: "<the value of MetricsServerAuthToken>"
    static_configs:
      - targets: ["mail.example.com:9100"]

Use scheme: https when the certificate pair is set, and keep scrape_timeout well under scrape_interval. The wiki gives nine alerting rules against these metrics, among them hmailserver_database_connected == 0 for 2m, hmailserver_delivery_queue_oldest_message_age_seconds > 3600 for 15m and hmailserver_tls_certificate_expiry_seconds - time() < 14 * 86400.

The Grafana dashboard

The repository ships hmailserver/docs/grafana-dashboard.json. Import it as it is and pick your Prometheus data source; the instance variable, from label_values(hmailserver_build_info, instance), selects the server. Its uid is hmailserver-overview; it refreshes every 1m and opens on now-24h. Its panels array has 19 entries: four row headers, five stat panels and ten time series. Every rate is over [5m] and every query is filtered by instance=~"$instance".

Health: server state (hmailserver_state == 1), database, uptime, missing message files, and TLS certificate expiry, amber below 21 days and red below 7.
Mail flow: delivery outcomes (delivered, deferred and bounced, stacked), queue depth with the oldest message's age on a right-hand axis, messages processed with spam and viruses removed, and active sessions by protocol.
Authentication and TLS: authentication outcomes and TLS handshakes, as rates.
Latency and database: command processing and database query latency (p50, p95 and p99 by histogram_quantile over the two histograms' _bucket series), the connection pool by state, and slow queries.

What the dashboard does not draw: the two database probe gauges, the two work-queue gauges, hmailserver_metrics_unauthorized_requests_total, the per-domain counters, the two store gauges and the six families added in 6.3.4. hmailserver_build_info is read by the instance variable and by no panel.

The metric history inside the server

The server keeps a short history of its own, so the Control Panel's dashboard can show 24 hours, 7 days and 30 days without Prometheus. MetricsHistoryDays (default 7; 0 switches it off) is the days kept in hm_metricsamples: one row per metric per minute, plus a sample at service start. Fourteen metrics are stored in 6.3.3, named as the exposition's without the hmailserver_ prefix and with the label folded in, such as sessions_smtp and processed_messages_total. 6.3.4 adds store_bytes and store_messages, whose history begins the day the server first samples them, and country_blocked_total and spam_trap_hits_total. Read it with GET /api/v1/metrics/history?metric=…&range=24h|7d|30d, or on Windows over COM with Utilities.GetMetricHistory. It has no labels and no histograms and is not a replacement for Prometheus, but it survives a restart.

Pushing to an OpenTelemetry collector

Three OTLP/HTTP signals, each a no-op until its own endpoint is set. The pushed metric names are the Prometheus names verbatim.

Setting	Default	Effect
`OtelMetricsEndpoint`	empty	OTLP metrics; the default path is `/v1/metrics`
`OtelMetricsInterval`	`60`	Seconds between exports, clamped to 5–3600
`OtelEndpoint`	empty	OTLP traces only: one span per delivery attempt, joined to the trace named by the message's own `traceparent`; default path `/v1/traces`
`OtelLogsEndpoint`	empty	OTLP logs: the same lines, categories and mask the log files get; default path `/v1/logs`

What is pushed: the twelve counters that predate 6.3.4 and, since 6.3.4, hmailserver_spam_trap_hits_total, sessions as one gauge with a protocol attribute, hmailserver_messagestore_missing_files, and the per-domain counters when they are enabled. The histograms, the database and queue gauges and certificate expiry are scrape-only. On Linux, in every release from 6.3.0 to 6.3.3, a collector that accepted the connection and never answered held the exporter, and the stop that waits for it, for good; 6.3.4 fixes that.

The SNMPv3 agent (since 6.3.4)

For a monitoring system that speaks SNMP and not Prometheus or OpenTelemetry, 6.3.4 adds a read-only agent answering the mail MIB (RFC 2789, MTA-MIB) and the standard SNMP groups from the same counters /metrics serves; an object /metrics has no counter for is answered noSuchObject, not made up. It speaks SNMPv3 with the User-based Security Model for one user at authPriv only (HMAC-SHA-256 and AES-128), and SNMPv2c as an option that stays off even when the agent is on. SNMPv1 is not answered, a Set is answered noAccess, and nothing is sent unasked: no traps, no informs. It is off as shipped, and an empty allow list answers nobody.

Setting (`hm_settings`)	REST key	Default	Meaning
`SnmpEnabled`	`snmp_enabled`	off	The whole agent; off, no socket is opened
`SnmpPort`	`snmp_port`	`161`	The UDP port
`SnmpBindAddress`	`snmp_bind_address`	empty, every address	`127.0.0.1` for a collector on the same machine; otherwise an address on a management network
`SnmpAllowedAddresses`	`snmp_allowed_addresses`	empty	Addresses and CIDR ranges, at most 64. Name the managers one by one: UDP does not prove a source address
`SnmpV3User`	`snmp_v3_user`	empty	The one SNMPv3 user
`SnmpV3AuthPassword`, `SnmpV3PrivPassword`	`snmp_v3_auth_password`, `snmp_v3_priv_password` (write-only)	empty	8 to 64 ASCII characters each
`SnmpV2cEnabled`, `SnmpV2cCommunity`	`snmp_v2c_enabled`, `snmp_v2c_community` (write-only)	off, empty	The community crosses the network in the clear; an answer is never more than eight times its request
`SnmpContact`, `SnmpLocation`	`snmp_contact`, `snmp_location`	empty	`sysContact` and `sysLocation`

They are on /api/v1/settings (the server group), on the Control Deck's Settings page under Server and in hmctl settings. There is no COM property for any of them, and in 6.3.4 no Control Panel page, so Windows and Linux are configured the same way; since 6.3.7 they are the SNMP tab of the Control Panel's Monitoring & alerts page on Windows, which 6.3.5 and 6.3.6 had on Logging. They can also be given as HM_PROPERTY_SnmpEnabled and so on in the environment, which reaches the database only when hmailserver --apply-environment is run (hMailServer.exe --apply-environment on Windows): the service does not read the environment itself. Each setting is read when the agent starts, at the next service start or Reinitialize. The two passwords and the community are stored protected, never read back, and not carried by a backup: enter them again after a restore.

Snmp.md's example keeps every password off the command line. On the server, for a manager at 192.0.2.10:

$ umask 077
$ cat > snmp.json
{"snmp_allowed_addresses": "192.0.2.10", "snmp_v3_user": "monitor",
 "snmp_v3_auth_password": "...", "snmp_v3_priv_password": "...",
 "snmp_contact": "[email protected]", "snmp_location": "Rack 4, London",
 "snmp_enabled": true}
^D
$ hmctl settings set --from-json snmp.json
$ rm snmp.json
$ hmctl api POST /api/v1/server/reinitialize

On the manager, with Net-SNMP 5.9 or later, defVersion 3, the user (defSecurityName), both passphrases, defSecurityLevel authPriv, defAuthType SHA-256 and defPrivType AES go in ~/.snmp/snmp.conf, readable by you alone (chmod 600); snmpwalk mail.example.com 1.3.6.1.2.1.28 then walks the mail MIB.

What it serves. In the MTA-MIB mtaGroupTable has one row per protocol the server runs: 1 SMTP, 2 POP3, 3 IMAP. What has a counter is answered: mtaStoredMessages is the delivery queue's depth, mtaTransmittedMessages is hmailserver_messages_delivered_total, mtaGroupOldestMessageStored is the oldest queued message's age, and mtaGroupInboundAssociations is hmailserver_sessions for the protocol. mtaReceivedMessages is noSuchObject, because no counter of messages received exists, and so are the volume, recipient and conversion columns. Sixteen scalars at 1.3.6.1.4.1.32473.1.2.<n>.0 carry /metrics series directly, among them hmailserver_state as a number at 14 (3 is running, 5 draining) and the two delivery-queue gauges at 15 and 16.

The enterprise number is a placeholder. 1.3.6.1.4.1.32473 is the number RFC 5612 reserves for documentation and examples; it is not registered to this project with IANA. When a number is registered, sysObjectID, the scalars and the engine ID's prefix move to it in one release, and a manager configured for these OIDs will need them changed. There is no MIB module file yet, for the same reason.

When it does not start. Error 7420: the port could not be bound, because another program holds it (the operating system's own SNMP service, on Windows or Linux, listens on 161 when it is installed) or, on Linux outside the packaged unit, the process lacks CAP_NET_BIND_SERVICE; the server runs on without the agent. 7421: the engine's seed or boot count in hm_settings could not be read or written. 7422: an exception left the agent's thread; a Reinitialize starts it again. On Linux the packaged hmailserver.service has CAP_NET_BIND_SERVICE for port 25, which is what lets it bind 161. No container image of 6.3.4 is published; a container built from the repository's Dockerfile publishes the port as UDP (-p 161:161/udp) and needs a host name of its own, since two started with one fixed hostname are one SNMP engine.

The daily block-list check, briefly

Since 6.3.4 the server looks the addresses it sends from up on DNS block lists once a day and raises the reputation.listed alert when one is listed. The watch is on in a new installation and off after an upgrade, and is read at GET /api/v1/reputation; the topic on DKIM rotation, reports and the reputation watch has the lists, the addresses checked and what a failed lookup means.

Where to read more

Wiki: Monitoring and Health and High Availability Runbook
Documents at v6.3.4: Snmp.md, HighAvailabilityRunbook.md, ReputationWatch.md, Complaints.md and the dashboard itself, grafana-dashboard.json
The public documentation: 16.3 Prometheus metrics and health probes
On this forum: Alerts, the audit trail, syslog and the journal, and per-domain reports, Reading the logs, and Warm standby and high availability for the pair the probes gate. See also The Control Deck and hMailServer on Linux: what you get and what you do not