---
title: "Monitoring and health: Prometheus metrics, health probes, Grafana, OTLP and SNMPv3"
url: https://www.hmailserver.co.uk/topic/28272/monitoring-and-health-prometheus-metrics-health-probes-grafana-otlp-and-snmpv3
author: "Progressiverobot"
published: 2026-09-27T19:54:13.731Z
updated: 2026-09-27T19:54:13.731Z
category: "Monitoring, metrics & logs"
tags: ["diagnostics","administration","rest-api","linux","windows"]
site: hMailServer Community Forum
---

# Monitoring and health: Prometheus metrics, health probes, Grafana, OTLP and SNMPv3

*Written for 6.3.4. hMailServer 6.3.6 (6 October 2026) publishes the first container image, `registry.gitlab.com/hmailserver/hmailserver:6.3.6`, and a monitor that matches "The response carried no HTTP headers." needs the new sentence, which says who ended the connection, after how many bytes and seconds. See [the 6.3.6 announcement](/topic/28276). hMailServer 6.3.7 (17 October 2026) publishes `registry.gitlab.com/hmailserver/hmailserver:6.3.7`, tagged `6.3` and `latest` until 6.3.8; renames the Control Panel page that holds the metrics listener's settings from API & monitoring to Monitoring & alerts, with the alerts' settings, the outbound webhooks and the SNMP agent on it (Ctrl+K takes the old title to it); and refuses `GET /api/v1/metrics/history`, and the same history over COM, to a credential confined to named domains, such as a domain administrator, the auditor role given a list of domains or an API key issued for named domains. With IMAP and POP3 now taking the client's address from a PROXY protocol header as SMTP does, the High Availability Runbook says that a balancer's `/readyz` check must not carry that header: the metrics listener reads none and answers a request behind one 404. See [the 6.3.7 announcement](/topic/28277). hMailServer 6.3.8 (2 November 2026) publishes `registry.gitlab.com/hmailserver/hmailserver:6.3.8`, also tagged `6.3` and `latest`. See [the 6.3.8 announcement](/topic/28278).*

This topic covers what hMailServer 6.3.4 offers a monitoring system: the metrics listener's Prometheus exposition and health probes, the repository's Grafana dashboard, the OpenTelemetry push, the server's own metric history, the new SNMPv3 agent and, briefly, the daily block-list check of the server's own addresses. The one thing to know first: the metrics listener is **off until you give it a port**, and once it is on, the health check to configure is `/readyz`, because it is proved by a real database round trip rather than by a flag.

### The metrics listener

One small HTTP listener serves both the exposition and the probes. Its seven settings are `[Settings]` keys, set on the Control Panel's **Monitoring & troubleshooting → API & monitoring** page (**Monitoring & alerts** since 6.3.7; Windows only), with `PUT /api/v1/settings/ini/{name}`, with `hmctl ini set`, or, on a Linux server whose REST listener is not yet on, with `hmailserver --set-setting NAME=VALUE`; [the topic on settings](/topic/28222) explains where they are stored and when a save applies. **All seven are taken when the listener starts, the three credential settings included**, so a saved change waits for a reinitialise (`POST /api/v1/server/reinitialize`, or **Apply now** in the Control Panel or the Deck) or a service restart. A token that has been saved and not applied leaves `/metrics` answering as it did before.

| Setting | Default | Effect | When it is wrong |
|---|---|---|---|
| `MetricsServerPort` | `0` | `0` means the listener is never created | No port, no probes, no metrics, and no complaint anywhere |
| `MetricsServerBindAddress` | `127.0.0.1` | An IP literal only, parsed with `inet_pton`: `0.0.0.0`, `127.0.0.1`, `::` or a specific IPv6 address | A host name or `localhost` is rejected: the log says `MetricsServer: Invalid bind address`, the listener does not start and the probes go with it |
| `MetricsServerAuthToken` | empty | Sent by the scraper as `Authorization: Bearer <token>` | Absent on a non-loopback bind: `/metrics` answers **503**, not 401, and the body names the settings that would open it. The probes keep answering |
| `MetricsServerAuthUsername` and `MetricsServerAuthPassword` | empty | HTTP Basic; with both schemes configured, either satisfies the check | As above |
| `MetricsServerCertificateFile` and `MetricsServerPrivateKeyFile` | empty | Both set makes the whole port HTTPS, probes included | Set but unusable: the listener still serves the probes in the clear and `/metrics` answers 503 |

`::` serves both address families; a specific IPv6 literal serves IPv6 only. "Loopback" means the whole of `127.0.0.0/8` and `::1`.

**The probes are answered before any branch that can refuse.** A Kubernetes `httpGet` probe, an ELB or ALB health check and a Docker `HEALTHCHECK` have nowhere to keep a bearer token. Refused credentials on `/metrics` are counted in `hmailserver_metrics_unauthorized_requests_total` rather than logged.

**Nothing on the request path touches the database or the disk.** A refresher thread probes the database every 5 s (every 30 s after a failure), refreshes the delivery-queue aggregates at most every 10 s and only when a scrape has asked for them, and re-reads the certificate files every 5 minutes. One exception since 6.3.4: the two message-store gauges are cached for an hour and recomputed by whichever caller finds them stale, so about once an hour a scrape can run `select sum(messagesize), count(*) from hm_messages` on the listener thread itself: on a large store, a slow answer to that scrape and to any probe queued behind it.

**It is a single accept loop serving one connection at a time.** A request's header read is bounded at 5 s and a response write at 15 s. For a load balancer's check the High Availability Runbook gives an interval of 5 s and a timeout of 2 s; never point a sub-second health check at it.

### The three health probes

No credential in any configuration; HTTP/1.0 responses that close the connection.

| Path | 200 when | Body | Use it for |
|---|---|---|---|
| `/livez` | Always, if the listener answered; never 503 | `alive` | Liveness restarts. It checks nothing on purpose: a database outage must not get a healthy process killed |
| `/readyz` | State is `Running` **and** the database has proved within the last 20 s that it can answer; 503 otherwise | `ready`, or `not ready: <reason>` | Load-balancer and VIP routing |
| `/healthz` | The same conditions | `{"status":"ok\|unavailable","state":"...","database":"up\|down","uptime_seconds":N}` | Dashboards and human debugging |

Readiness is four conditions in order, each with its own reason: `not ready: server not in running state`; `not ready: database not connected`; `not ready: database did not answer the last readiness probe`; and `not ready: database readiness probe has not completed recently`. The last exists because a probe against a database that has stopped answering can hang for as long as `DBConnectionAcquireTimeout` allows, 60 s by default. Since 6.3.4 a server in maintenance mode answers `503 not ready: draining`: `POST /api/v1/server/drain` enters it, `GET` reports `{"draining","since","sessions","deliveries","drained"}`, and `DELETE` leaves it.

**Start-up is a refused connection, not a 503**: the listener is brought up only after the state has gone to `Running`. **Shutdown is a 503 first, then a refusal**: the state goes to `Stopping` before anything is torn down, so `/readyz` reports 503 while `ShutdownDrainSeconds` runs.

The High Availability Runbook builds its active/passive design on this: both nodes have the listener on, with `MetricsServerBindAddress=0.0.0.0` and a `MetricsServerAuthToken`, and the VIP is gated strictly on `/readyz`. The passive node's service is stopped, so its `/readyz` is refused and it is never routed to. Since the probes need no credential, a missing token shows as missing dashboards, not as a failed failover.

### Every metric family

Thirty-eight metric families in 6.3.4 (thirty in 6.3.3). Counters are monotonic since server start; a restart resets them, which is what `hmailserver_start_time_seconds` is for.

**Mail flow**

| Metric | Type | Meaning, and the trap in it |
|---|---|---|
| `hmailserver_processed_messages_total` | counter | One per delivery pass started, not per message received: a message deferred three times and then delivered counts four |
| `hmailserver_messages_delivered_total` | counter | Delivery passes that completed with no failed recipient |
| `hmailserver_messages_deferred_total` | counter | Passes rescheduled: a temporary failure, greylisting, or a message the virus scanner could not examine (`AVFailAction=1`) |
| `hmailserver_messages_bounced_total` | counter | Delivery-failure notifications queued to the sender; one can cover several recipients |
| `hmailserver_delivery_queue_messages` | gauge | Rows in `hm_messages` in the delivering state |
| `hmailserver_delivery_queue_oldest_message_age_seconds` | gauge | Age of the oldest queued message; `0` when the queue is empty. Alert on this, not on depth |
| `hmailserver_spam_messages_total`, `hmailserver_viruses_removed_total` | counter | Messages classified as spam; viruses removed during delivery preprocessing |
| `hmailserver_store_bytes`, `hmailserver_store_messages` | gauge | Since 6.3.4. The size and the number of the messages in `hm_messages`, recomputed at most once an hour |
| `hmailserver_domain_messages_received_total{domain}`, `hmailserver_domain_messages_sent_total{domain}` | counter | Only present when `MetricsPerDomainEnabled=1`; only hosted domains are ever labelled |

**Added in 6.3.4.** `hmailserver_country_blocked_total` and `hmailserver_country_blocked_by_country_total{country}` count connections and sign-ins refused by a country policy, and `hmailserver_spam_trap_hits_total` counts hits on a declared spam trap. Three gauges cover the feedback loops' seven-day window: `hmailserver_domain_complaints{domain}`, `hmailserver_domain_complaint_window_messages{domain}` and `hmailserver_domain_complaint_rate{domain}`. They are present only when `MetricsPerDomainEnabled` is set **and** complaint processing is on. **The rate gauge is a ratio, not parts per million**: [Complaints.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Complaints.md) alerts on `hmailserver_domain_complaint_rate > 0.001`. Parts per million is the unit of the `complaints.rate.high` alert rule's threshold, so an expression written in parts per million against the gauge would never fire.

**Sessions, authentication and TLS**

| Metric | Type | Meaning |
|---|---|---|
| `hmailserver_sessions{protocol}` | gauge | Concurrent sessions; `protocol` is `smtp`, `imap` or `pop3` |
| `hmailserver_auth_success_total`, `hmailserver_auth_failures_total` | counter | A rising failure rate against a flat success rate is a password-guessing run |
| `hmailserver_tls_handshakes_total`, `hmailserver_tls_handshake_failures_total` | counter | The mail protocols only; a failed handshake on the metrics port is never counted here |
| `hmailserver_tls_certificate_expiry_seconds{certificate}` | gauge | `notAfter` as a Unix timestamp per configured certificate, re-read every 5 minutes; at most 128 series. **Absent entirely when no certificate is configured.** With no certificate row, the ACME certificate is labelled `ACME (automatic)` |
| `hmailserver_metrics_unauthorized_requests_total` | counter | Always `0` when no credential is configured |

**The server and the database**

| Metric | Type | Meaning |
|---|---|---|
| `hmailserver_start_time_seconds` | gauge | Uptime is `time() - hmailserver_start_time_seconds` |
| `hmailserver_build_info{version,architecture,database_schema_version}` | gauge | Always `1`; the information is in the labels |
| `hmailserver_state{state}` | gauge | One series per state (`unknown`, `stopped`, `starting`, `running`, `stopping` and, since 6.3.4, `draining`), exactly one of which is `1` |
| `hmailserver_workqueue_depth{queue}`, `hmailserver_workqueue_blocking_tasks_waiting{queue}` | gauge | Seven fixed `queue` values: `Server queue`, `Maintenance queue`, `Asynchronous task queue`, `Name lookup queue`, `IOCPQueue`, `SMTP delivery queue`, `External fetch queue`. `Asynchronous task queue` sends the final `250`, so depth there is mail waiting to be acknowledged |
| `hmailserver_messagestore_missing_files` | gauge | Always `0` unless `MessageStoreConsistencyCheck=1` |
| `hmailserver_command_processing_seconds`, `hmailserver_db_query_seconds` | histogram | Bucket bounds from 0.0001 s to 1.0 s, and from 0.001 s to 10.0 s; both emit the full `_bucket`, `_sum` and `_count` set |
| `hmailserver_database_connected` | gauge | `1` when the database answered a real round trip recently |
| `hmailserver_database_probe_success_timestamp_seconds`, `hmailserver_database_probe_age_seconds` | gauge | When it last answered, and how long ago on the server's own clock; `/readyz` reports 503 once the age passes 20 s |
| `hmailserver_db_connections{state}` | gauge | `busy` and `available`; `available` pinned at zero is saturation |
| `hmailserver_db_slow_queries_total` | counter | Statements slower than `SlowQueryLogMilliseconds`; `0` while that setting is `0` |

### Scraping it

```yaml
scrape_configs:
  - job_name: hmailserver
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    authorization:
      credentials: "<the value of MetricsServerAuthToken>"
    static_configs:
      - targets: ["mail.example.com:9100"]
```

Use `scheme: https` when the certificate pair is set, and keep `scrape_timeout` well under `scrape_interval`. The wiki gives nine alerting rules against these metrics, among them `hmailserver_database_connected == 0` for 2m, `hmailserver_delivery_queue_oldest_message_age_seconds > 3600` for 15m and `hmailserver_tls_certificate_expiry_seconds - time() < 14 * 86400`.

### The Grafana dashboard

The repository ships `hmailserver/docs/grafana-dashboard.json`. Import it as it is and pick your Prometheus data source; the `instance` variable, from `label_values(hmailserver_build_info, instance)`, selects the server. Its uid is `hmailserver-overview`; it refreshes every `1m` and opens on `now-24h`. Its `panels` array has 19 entries: four row headers, five stat panels and ten time series. Every rate is over `[5m]` and every query is filtered by `instance=~"$instance"`.

- **Health**: server state (`hmailserver_state == 1`), database, uptime, missing message files, and TLS certificate expiry, amber below 21 days and red below 7.
- **Mail flow**: delivery outcomes (delivered, deferred and bounced, stacked), queue depth with the oldest message's age on a right-hand axis, messages processed with spam and viruses removed, and active sessions by protocol.
- **Authentication and TLS**: authentication outcomes and TLS handshakes, as rates.
- **Latency and database**: command processing and database query latency (p50, p95 and p99 by `histogram_quantile` over the two histograms' `_bucket` series), the connection pool by state, and slow queries.

What the dashboard does not draw: the two database probe gauges, the two work-queue gauges, `hmailserver_metrics_unauthorized_requests_total`, the per-domain counters, the two store gauges and the six families added in 6.3.4. `hmailserver_build_info` is read by the `instance` variable and by no panel.

### The metric history inside the server

The server keeps a short history of its own, so the Control Panel's dashboard can show 24 hours, 7 days and 30 days without Prometheus. `MetricsHistoryDays` (default `7`; `0` switches it off) is the days kept in `hm_metricsamples`: one row per metric per minute, plus a sample at service start. Fourteen metrics are stored in 6.3.3, named as the exposition's without the `hmailserver_` prefix and with the label folded in, such as `sessions_smtp` and `processed_messages_total`. 6.3.4 adds `store_bytes` and `store_messages`, whose history begins the day the server first samples them, and `country_blocked_total` and `spam_trap_hits_total`. Read it with `GET /api/v1/metrics/history?metric=…&range=24h|7d|30d`, or on Windows over COM with `Utilities.GetMetricHistory`. It has no labels and no histograms and is not a replacement for Prometheus, but it survives a restart.

### Pushing to an OpenTelemetry collector

Three OTLP/HTTP signals, each a no-op until its own endpoint is set. The pushed metric names are the Prometheus names verbatim.

| Setting | Default | Effect |
|---|---|---|
| `OtelMetricsEndpoint` | empty | OTLP metrics; the default path is `/v1/metrics` |
| `OtelMetricsInterval` | `60` | Seconds between exports, clamped to 5–3600 |
| `OtelEndpoint` | empty | OTLP **traces** only: one span per delivery attempt, joined to the trace named by the message's own `traceparent`; default path `/v1/traces` |
| `OtelLogsEndpoint` | empty | OTLP logs: the same lines, categories and mask the log files get; default path `/v1/logs` |

What is pushed: the twelve counters that predate 6.3.4 and, since 6.3.4, `hmailserver_spam_trap_hits_total`, sessions as one gauge with a `protocol` attribute, `hmailserver_messagestore_missing_files`, and the per-domain counters when they are enabled. The histograms, the database and queue gauges and certificate expiry are scrape-only. On Linux, in every release from 6.3.0 to 6.3.3, a collector that accepted the connection and never answered held the exporter, and the stop that waits for it, for good; 6.3.4 fixes that.

### The SNMPv3 agent (since 6.3.4)

For a monitoring system that speaks SNMP and not Prometheus or OpenTelemetry, 6.3.4 adds a read-only agent answering the mail MIB (RFC 2789, MTA-MIB) and the standard SNMP groups **from the same counters `/metrics` serves**; an object `/metrics` has no counter for is answered `noSuchObject`, not made up. It speaks SNMPv3 with the User-based Security Model for one user at authPriv only (HMAC-SHA-256 and AES-128), and SNMPv2c as an option that stays off even when the agent is on. SNMPv1 is not answered, a Set is answered `noAccess`, and nothing is sent unasked: no traps, no informs. It is off as shipped, and **an empty allow list answers nobody**.

| Setting (`hm_settings`) | REST key | Default | Meaning |
|---|---|---|---|
| `SnmpEnabled` | `snmp_enabled` | off | The whole agent; off, no socket is opened |
| `SnmpPort` | `snmp_port` | `161` | The UDP port |
| `SnmpBindAddress` | `snmp_bind_address` | empty, every address | `127.0.0.1` for a collector on the same machine; otherwise an address on a management network |
| `SnmpAllowedAddresses` | `snmp_allowed_addresses` | empty | Addresses and CIDR ranges, at most 64. Name the managers one by one: UDP does not prove a source address |
| `SnmpV3User` | `snmp_v3_user` | empty | The one SNMPv3 user |
| `SnmpV3AuthPassword`, `SnmpV3PrivPassword` | `snmp_v3_auth_password`, `snmp_v3_priv_password` (write-only) | empty | 8 to 64 ASCII characters each |
| `SnmpV2cEnabled`, `SnmpV2cCommunity` | `snmp_v2c_enabled`, `snmp_v2c_community` (write-only) | off, empty | The community crosses the network in the clear; an answer is never more than eight times its request |
| `SnmpContact`, `SnmpLocation` | `snmp_contact`, `snmp_location` | empty | `sysContact` and `sysLocation` |

They are on `/api/v1/settings` (the server group), on the Control Deck's **Settings** page under **Server** and in `hmctl settings`. **There is no COM property** for any of them, and in 6.3.4 no Control Panel page, so Windows and Linux are configured the same way; since 6.3.7 they are the **SNMP** tab of the Control Panel's **Monitoring & alerts** page on Windows, which 6.3.5 and 6.3.6 had on **Logging**. They can also be given as `HM_PROPERTY_SnmpEnabled` and so on in [the environment](/topic/28231), which reaches the database only when `hmailserver --apply-environment` is run (`hMailServer.exe --apply-environment` on Windows): the service does not read the environment itself. Each setting is read when the agent starts, at the next service start or *Reinitialize*. The two passwords and the community are stored protected, never read back, and **not carried by a backup**: enter them again after a restore.

[Snmp.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Snmp.md)'s example keeps every password off the command line. On the server, for a manager at 192.0.2.10:

```console
$ umask 077
$ cat > snmp.json
{"snmp_allowed_addresses": "192.0.2.10", "snmp_v3_user": "monitor",
 "snmp_v3_auth_password": "...", "snmp_v3_priv_password": "...",
 "snmp_contact": "postmaster@example.com", "snmp_location": "Rack 4, London",
 "snmp_enabled": true}
^D
$ hmctl settings set --from-json snmp.json
$ rm snmp.json
$ hmctl api POST /api/v1/server/reinitialize
```

On the manager, with Net-SNMP 5.9 or later, `defVersion 3`, the user (`defSecurityName`), both passphrases, `defSecurityLevel authPriv`, `defAuthType SHA-256` and `defPrivType AES` go in `~/.snmp/snmp.conf`, readable by you alone (`chmod 600`); `snmpwalk mail.example.com 1.3.6.1.2.1.28` then walks the mail MIB.

**What it serves.** In the MTA-MIB `mtaGroupTable` has one row per protocol the server runs: 1 SMTP, 2 POP3, 3 IMAP. What has a counter is answered: `mtaStoredMessages` is the delivery queue's depth, `mtaTransmittedMessages` is `hmailserver_messages_delivered_total`, `mtaGroupOldestMessageStored` is the oldest queued message's age, and `mtaGroupInboundAssociations` is `hmailserver_sessions` for the protocol. `mtaReceivedMessages` is `noSuchObject`, because no counter of messages received exists, and so are the volume, recipient and conversion columns. Sixteen scalars at `1.3.6.1.4.1.32473.1.2.<n>.0` carry `/metrics` series directly, among them `hmailserver_state` as a number at 14 (3 is running, 5 draining) and the two delivery-queue gauges at 15 and 16.

**The enterprise number is a placeholder.** 1.3.6.1.4.1.32473 is the number RFC 5612 reserves for documentation and examples; it is not registered to this project with IANA. When a number is registered, `sysObjectID`, the scalars and the engine ID's prefix move to it in one release, and a manager configured for these OIDs will need them changed. There is no MIB module file yet, for the same reason.

**When it does not start.** Error `7420`: the port could not be bound, because another program holds it (the operating system's own SNMP service, on Windows or Linux, listens on 161 when it is installed) or, on Linux outside the packaged unit, the process lacks `CAP_NET_BIND_SERVICE`; the server runs on without the agent. `7421`: the engine's seed or boot count in `hm_settings` could not be read or written. `7422`: an exception left the agent's thread; a Reinitialize starts it again. On Linux the packaged `hmailserver.service` has `CAP_NET_BIND_SERVICE` for port 25, which is what lets it bind 161. No container image of 6.3.4 is published; a container built from the repository's Dockerfile publishes the port as UDP (`-p 161:161/udp`) and needs a host name of its own, since two started with one fixed hostname are one SNMP engine.

### The daily block-list check, briefly

Since 6.3.4 the server looks the addresses it sends from up on DNS block lists once a day and raises the `reputation.listed` alert when one is listed. The watch is **on in a new installation and off after an upgrade**, and is read at `GET /api/v1/reputation`; [the topic on DKIM rotation, reports and the reputation watch](/topic/28258) has the lists, the addresses checked and what a failed lookup means.

### Where to read more

- Wiki: [Monitoring and Health](https://gitlab.com/hmailserver/hmailserver/-/wikis/Monitoring-and-Health) and [High Availability Runbook](https://gitlab.com/hmailserver/hmailserver/-/wikis/High-Availability-Runbook)
- Documents at v6.3.4: [Snmp.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Snmp.md), [HighAvailabilityRunbook.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/HighAvailabilityRunbook.md), [ReputationWatch.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/ReputationWatch.md), [Complaints.md](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/Complaints.md) and the dashboard itself, [grafana-dashboard.json](https://gitlab.com/hmailserver/hmailserver/-/blob/v6.3.4/hmailserver/docs/grafana-dashboard.json)
- The public documentation: [16.3 Prometheus metrics and health probes](https://www.progressiverobot.com/hmailserver-documentation/#163-prometheus-metrics-and-health-probes)
- On this forum: [Alerts, the audit trail, syslog and the journal, and per-domain reports](/topic/28273), [Reading the logs](/topic/28220), and [Warm standby and high availability](/topic/28232) for the pair the probes gate. See also [The Control Deck](/topic/28192) and [hMailServer on Linux: what you get and what you do not](/topic/28189)