Alerts, the audit trail, syslog and the journal, and per-domain reports

Progressiverobot

Written for 6.3.4. hMailServer 6.3.7 (17 October 2026) moves the Control Panel's Audit trail page, with journaling and eDiscovery, into a new Compliance group, and the alerts' settings, the outbound webhooks and the SNMP agent onto its Monitoring & alerts page (API & monitoring until 6.3.6; 6.3.5 and 6.3.6 had them on Logging); Ctrl+K takes an old title or route to its page. It also fixes the SMTP log, whose lines the syslog sink copies, which wrote some credentials in the clear with SMTP logging on, in every version from 6.0.0 to 6.3.6 (work item #95). See the 6.3.7 announcement.

This topic covers four things 6.3.4 adds for being told what the server is doing and who changed it: the alerts, the audit trail, the syslog sink (or the systemd journal on Linux) and the per-domain reports, with a paragraph on following one connection live. The one thing to know first: after the upgrade to 6.3.4 the audit trail and the alerts are on, with seven of the eleven conditions enabled, but nothing is mailed until AlertRecipient and AlertSenderAddress are set.

What 6.3.4 switches on without a switch

From the CHANGELOG's list of what changes on upgrade: the audit trail is on (schema 6043), and keeps every row until a retention is set. Compromised-account detection starts in watch mode (schema 6071): it raises account.compromised but holds nothing until compromise_detection_mode is set to enforce. The reputation watch, which raises reputation.listed, is on in a new installation and off after an upgrade.

Alerts

A condition is a row in hm_alertrules (on or off, threshold, mail or webhook or both, cool-down, at once or in the digest); an event is a row in hm_alertevents. Delivery is AlertTask, once a minute, and nothing else sends, so an alert about the delivery queue is never itself submitted into the stuck queue.

Condition	On as shipped	Raised by	Threshold
`backup.failed`	yes	A backup that failed, and a scheduled backup that could not run. Sent at once	—
`certificate.expiring`	yes	The notAfter of every configured certificate, read once a minute	7 days
`disk.low`	no	The free-space warning band, and again below the floor; cleared on recovery	—
`queue.stalled`	no	A work queue whose every thread has been in the same task past `AsyncQueueStallThreshold`	—
`autoban.storm`	no	Auto-bans in the last hour	25
`minidump.written`	no	A crash dump newer than the last one recorded, in the log directory	—
`complaints.rate.high`	yes	A hosted domain's complaint rate over the feedback loops' seven-day window; inert while complaint processing is off, and a domain that sent fewer than 100 messages is not judged	1,000 per million
`dkim.rotation.stalled`	yes	A rotation whose next key's record has not been seen for the threshold, or a running rotation whose last move failed	7 days
`account.compromised`	yes	Compromised-account detection at submission, one event per account per hour; in `watch` mode the detail says what enforcing would have done	—
`account.password-reset`	yes	A password reset completed through an account's confirmed recovery address; no cool-down and no digest	—
`reputation.listed`	yes	The reputation watch's stored rows; sent at once, and a check that was refused or timed out does not clear it	—

The four that are off are noisy on a healthy server.

The settings. All nine are rows of hm_settings, none an hMailServer.ini key. They are in the server group of GET/PUT /api/v1/settings, on the Deck's Settings page, and on Windows over COM through Settings. The CHANGELOG and AlertsAndAuditTrail.md spell some in their hm_settings form (AlertRecipient, AlertSenderAddress and so on); the REST keys are these.

REST key	Default	Meaning
`alerts_enabled`	on	The master switch
`alert_recipient`	empty	Where alerts and the digest go; a rule can name an `address` of its own instead
`alert_sender_address`	empty	What they are sent from; a rule with no sender sends nothing
`alert_digest_enabled`	on	Whether rules marked for the digest wait for it
`alert_digest_hour`	7	The hour, UTC, the digest goes out; a digest with nothing in it is not sent
`alert_max_per_hour`	20	Past this many single alerts in an hour, everything goes into the digest
`alert_webhook_max_attempts`	5	Webhook attempts before an event is dead-lettered (1 to 20)
`audit_trail_enabled`	on	Whether administrative changes are recorded
`audit_retention_days`	0	Days of trail to keep; 0 keeps everything

Nothing may flap. A state condition already raised is not raised again; an occurrence inside the rule's cool-down is recorded but held for the digest; and the digest is one message listing what fired.

Webhooks. A rule whose actions is 2 (webhook) or 3 (mail and webhook) POSTs one JSON object per event (condition, state, severity, time, time_utc, summary, detail, server) with three headers: X-hMailServer-Timestamp, X-hMailServer-Signature: sha256=<hex> and X-hMailServer-Event. It is signed under the rule's shared secret, as every webhook the server sends is; the topic on integrating applications shows how to verify it.

A rule that calls a webhook is refused without an address and without a secret; webhook_secret is write-only and the listing says only webhook_secret_set. Only https, or plain http to a loopback address, is used. An attempt that does not get a 2xx is retried on the next pass, then after 1, 5 and 15 minutes, which is the default five attempts, and 60 minutes apart after that if the maximum is raised. After the last attempt the event is dead-lettered: kept, no longer retried, and reported once in the error log as HM6543.

Route	Does	Permission
`GET /api/v1/alerts/rules`	Every condition and its rule	`view_configuration`
`PUT /api/v1/alerts/rules/{condition}`	Change one, or create one for a condition nothing ships; a field left out keeps its value	`manage_settings`
`GET /api/v1/alerts/events`	What has fired, newest first, with the webhook's state	`view_configuration`
`POST /api/v1/alerts/test`	Raise a condition through the path a real one takes: `{"condition":"disk.low","summary":"...","occurrence":false}`	`manage_settings`
`POST /api/v1/alerts/run`	Run one pass now	`manage_server`

All five are refused to every API key. The administrator password reaches them, and so does an account whose role holds the permission and reaches every domain. Neither the Control Panel nor the Deck has a page for the rules, and hmctl and the PowerShell module have no alerts or audit verbs; hmctl api reaches the routes.

The audit trail

One row per administrative change made over COM, REST or the Deck, in hm_audit: when (audittime, UTC); who (auditactor: Administrator over COM, administrator or key:<id> over REST, and an account signed in with its own password is its address over COM and account:<address> over REST and the Deck); the interface and, except over COM, the peer address; the object's type and name; the action (created, updated or deleted, also unlocked, cleared, moved and move failed, the last for a mailbox move that failed and could not be put back); the detail (the columns the change carried, or for a setting Name: old -> new, ending [role: <name>] under a role); and the two hashes of the chain.

Both kinds of setting are recorded: the hm_settings values and the [Settings] keys (in hm_inisettings since schema 6042), with the value they had. A key set for the first time reads SmtpTarpitCount: (default) -> 5; returned to its default it reads SmtpTarpitCount: 5 -> (default), with the action deleted. A write of the value a setting already has records nothing.

What is never recorded as a value. A password, key, token, secret, hash, salt or passphrase is recorded as (changed), decided by the name of the column or setting, matched as a substring, so MetricsServerAuthToken and PasswordPepper are among them. Fifteen keys whose names only look like secrets, the seven PasswordPolicy* keys among them, are recorded with their values. The server's own writes (delivery, retention, greylisting, the index) are never recorded.

Verifying the chain. Each row carries the SHA-256 of the row before it and of itself. GET /api/v1/audit/verify answers {"intact","rows_checked","first_broken_id","reason"}, and the reason tells a row that was edited from one that was removed. This makes tampering detectable, not impossible: whoever can write the table can recompute the whole chain. Rows hashed the older way, which only a build from before the 6.3.4 release can have written, did not cover a character above U+00FF; they still verify, with that weakness.

Reading it. GET /api/v1/audit?actor=&object_type=&action=&since=&until=&limit=&offset= answers newest first; actor is a substring, so key: finds every change made with an API key, and limit is 1 to 1,000, default 100. It needs the administrator password, or an account whose role holds view_audit (the auditor role) and is not confined to named domains. Every API key is refused, with 401 rather than 403, so that it learns nothing from asking.

Putting a change back. POST /api/v1/audit/{id}/revert writes a setting's old value back through the same setter, so the put-back is itself a row in the trail. It needs manage_settings and is refused with 409 for a row that is not a setting's change, a secret's row, a value the trail cut at 120 characters, and a setting this version no longer has.

Where to see it. The Deck's Audit trail view has the filters, paging, Verify the chain and Put back. On Windows the Control Panel has the same page, over COM, at Monitoring & troubleshooting → Audit trail (Compliance → Audit trail since 6.3.7; Ctrl+K takes the old route to it). Switching recording off is itself recorded. The trail, the rules and the events are not in the built-in backup.

What it holds for compliance. A download of a discovery export is recorded on the trail alone, so with the trail off it is recorded nowhere. An erasure is recorded with the address it erased wherever it deletes from recorded tables (aliases, distribution-list recipients and, with the archive included, the archive index), and audit_retention_days is how those rows age out. The compliance topic covers eDiscovery and the erasure.

What it does not record. Reads; what a restore replaces (the request is one row, restore started); creating or revoking a REST API key; releasing or deleting a quarantined message; creating, renaming or deleting a public folder; the [Security], [Directories] and [Database] sections of hMailServer.ini; and a change nobody made through an interface. A change to the twelve syslog settings over COM, from the Control Panel's Logging page for one, is recorded in 6.3.4: the settings property setters carry an audit scope of their own. The wiki's syslog section, 16.2a, still describes the earlier state.

Syslog and the journal (since 6.3.4)

Every entry the server logs can also be sent as RFC 5424 to a syslog collector over UDP, TCP or TLS. It is a copy: the files under LogFolder are written as before. On Windows the settings are the syslog (RFC 5424) card on the Control Panel's Logging page; on Linux they are keys of PUT /api/v1/settings/logging and the Control Deck, and hmctl settings get logging prints them. They take effect when the sink starts, at the next service start or Reinitialize, not when they are saved.

`hm_settings` name	REST key	Default	Meaning
`SyslogEnabled`	`syslog_enabled`	off	The whole feature
`SyslogHost`	`syslog_host`	empty	Empty is not "off": on Linux under systemd it selects the journal; on Windows nothing is sent, which the server says once in the application log
`SyslogPort`	`syslog_port`	`514`	514 for UDP and plain TCP; 6514 is the registered port for TLS
`SyslogTransport`	`syslog_transport`	`0` (UDP)	`0` UDP, `1` TCP, `2` TCP over TLS; the REST key takes `udp`, `tcp` or `tls`
`SyslogFacility`	`syslog_facility`	`2` (mail)	0 to 23; 16 to 23 are `local0` to `local7`
`SyslogMinimumSeverity`	`syslog_minimum_severity`	`6` (informational)	Entries less severe than this are not sent
`SyslogLogTypes`	`syslog_log_smtp`, `syslog_log_pop3`, `syslog_log_imap`, `syslog_log_application`, `syslog_log_tcpip`, `syslog_log_debug`	`86`	One bit mask: SMTP, POP3, application and IMAP on; TCP/IP and debug off

Errors are sent whatever the mask says. A message looks like this:

<22>1 2026-09-15T14:21:09.284+01:00 mail.example.com hMailServer 4812 SMTPD [hmailserver@32473 thread="7412" session="41" client="203.0.113.9"] <BOM>RECEIVED: MAIL FROM:<[email protected]>

The message id is the log category (SMTPD and so on) or, on an error line carrying an hMailServer error number, that number, so msgid == HM5165 is a usable filter at the collector. The structured data under hmailserver@32473 carries thread, and session and client on a conversation line. UDP truncates each message to 1024 bytes, cutting the text and never the header; TCP and TLS frame each message with its length and truncate nothing below 65000 bytes. TLS verifies the collector's certificate against the machine's trust store and the configured host name, so set the host to the name in the certificate, not an IP address; no client certificate is offered.

A collector that is down costs log lines, not mail. Entries go to a background thread over a queue of 4096 that drops the oldest when full; a stream that cannot connect backs off from one second to a minute; and the application log gets one line when an outage starts and one when it ends, with the number dropped. Nothing is retried from disk. Each step of a stream transport (the connection, the TLS handshake, one entry written) is given ten seconds, so a collector that accepts the connection and then goes quiet is a failure like a refused one, and a reinitialise or a service stop waits for that step at most.

For rsyslog, in /etc/rsyslog.d/30-hmailserver.conf (UDP for the default transport, TCP for SyslogTransport = 1; the stop keeps the lines out of /var/log/mail.log too, so drop it if you want both):

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

if ($app-name == "hMailServer") then {
   action(type="omfile" file="/var/log/hmailserver.log"
          template="RSYSLOG_SyslogProtocol23Format")
   stop
}

Without RSYSLOG_SyslogProtocol23Format rsyslog reformats to the older RFC 3164 shape and the structured data is lost. For TLS (SyslogTransport = 2

module(load="imtcp"
       StreamDriver.Name="gtls"
       StreamDriver.Mode="1"
       StreamDriver.AuthMode="anon")
input(type="imtcp" port="6514")

global(DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca.pem"
       DefaultNetstreamDriverCertFile="/etc/rsyslog.d/collector-cert.pem"
       DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/collector-key.pem")

A self-signed collector certificate has to be installed as trusted on the hMailServer machine, or the handshake is refused and nothing is sent.

The journal, on Linux only, needs no collector. Where the server runs as the packaged hmailserver.service, switch the sink on and leave the collector host empty:

$ hmctl settings set logging --set syslog_enabled=true
$ sudo systemctl restart hmailserver
$ journalctl -u hmailserver -f

Beside the standard fields the journal gets HMAILSERVER_CATEGORY, HMAILSERVER_MSGID, HMAILSERVER_THREAD, HMAILSERVER_SESSION and HMAILSERVER_CLIENT:

$ journalctl -u hmailserver HMAILSERVER_SESSION=41
$ journalctl -u hmailserver HMAILSERVER_MSGID=HM5165

If a collector host is configured the journal is not used. The start of the sink is itself logged, as Syslog: sending RFC 5424 log entries to <host>:<port> over UDP, facility 2. or Syslog: writing log entries to the systemd journal.; a server that logs neither has the feature off. Not sent: the AWStats log, the backup log and the event-script log.

Per-domain reports (since 6.3.4)

GET /api/v1/reports is an index of the sections, the state of their three sources and what this server cannot answer. A report drawn while its source was off is empty, which is not the same statement as "nothing happened"; each section's note says which it is. GET /api/v1/reports/{section} answers one section, the Deck's Reports view draws the summary, and hmctl report prints a section or saves it as CSV. There is no Control Panel page.

Section	Counted from	Says
`traffic`	the message trace	Per day and per local domain: `incoming`, `outgoing`, `failed_incoming`, `failed_outgoing`
`failures`	the message trace	Per local domain and SMTP status code
`senders`, `recipients`	the message trace	The local addresses that sent or received the most, with their failures
`mailboxes`	`hm_messages`	The largest mailboxes as they are now: a size, not a history
`volume`	the metric history, and the daily per-domain sample	Processed, delivered, deferred, bounced, spam and viruses per day, with `by_domain` rows of spam and viruses
`storage`	the same	The store's size per day and `growth_bytes`, with `by_domain` rows of each domain's size
`summary`	all three sources	Every section the credential may see, with an `omitted` list and the reason
`quota`	the store	Every mailbox with a limit and at least one message, nearest its limit first, as a share of the limit: a share now, not a history
`countries`	the metric history and the country counters	What a country policy refused, per day; the split by country is since the server last started, not per day

The last two are described in QuotaPolicy.md and CountryBlocking.md, not in Reports.md.

Parameters. from and to as YYYY-MM-DD, both inclusive, defaulting to the last 30 days and never more than 366; domain, one hosted domain (a domain not hosted here is a 404); top, 1 to 200, default 10; format=csv, refused for summary. In CSV a cell starting with =, +, - or @ is prefixed with an apostrophe.

Three things decide whether a report means anything. The message trace is off by default (MessageTraceEnabled=0), and while it is off the four sections counted from it are empty. Processed, delivered, deferred and bounced cannot be given per domain, because those counters carry no domain; spam, viruses and storage can, since schema 6062 added the daily sample in hm_domainsamples, which starts the day it was first taken. And only hosted domains are counted.

Authorisation. The administrator password reads everything; every other credential needs both view_configuration and view_message_trace. domainadmin and auditor hold both; helpdesk is refused. A key restricted to named domains sees only those and is refused 403 for another. On volume and storage Reports.md contradicts itself: its Authorisation section refuses them to such a key outright, while its section What cannot be answered, and what it would take (and, for volume, its table of sections under The route) gives a credential restricted to domains the per-domain rows, as the wiki's section 16.6a does. Test both with your own restricted key before a script relies on either.

The cost. One report makes at most three grouped queries; the trace read is capped at 100 000 grouped rows and the answer says truncated when it reaches the cap.

hmctl report sections
hmctl report traffic --from 2026-09-01 --to 2026-09-15 --domain example.com

Following one connection live

Since 6.3.4 the Deck's Logs page lists the sessions open now and those recently ended, and Follow shows one session's lines as they are written, from an in-memory ring of the last 4,096 log entries that no restart keeps; it needs view_logs. Reading the logs covers following a connection in full, and the log files themselves.

Where to read more

Documents at v6.3.4: AlertsAndAuditTrail.md, Syslog.md, Reports.md and FollowConnection.md
Wiki: Monitoring and Health, sections 16.2a, 16.6a, 16.7a and 16.7b
The public documentation: 16.2 Logs on disk
On this forum: Monitoring and health for the metrics, the probes, the dashboard and the SNMP agent; Administrator roles for the permissions named here. See also The Control Deck and hMailServer on Linux: what you get and what you do not