Message storage: compression, de-duplication, object storage and encryption at rest

Written for 6.3.4. hMailServer 6.3.8 (2 November 2026), a security release, reads on Windows the CA file MessageStoreObjectCaFile names from the file configured, or refuses its name: a name it cannot hand to OpenSSL as it is named, as a path holding characters outside ASCII may be, is refused with HM7536 in the error log, and then no request to the object store is allowed, so give the file, and the folders above it, an ASCII name before upgrading. See the 6.3.8 announcement.

This topic covers the three things 6.3.4 can do to a stored message that 6.3.3 could not: store it compressed with its large attachments held once, keep it in an S3-compatible or Azure Blob store instead of under the data directory, and store it encrypted to a key of the mailbox's own. All three are new in 6.3.4, and all three are off until an administrator turns them on: a server that turns none of them on stores every message exactly as 6.3.3 did. The one thing to know first: the delivery queue is never compressed, encrypted or put in the store. A queued message is plain RFC 822 on the local disk, so the event scripts, the virus scanners, SpamAssassin, the sandbox and the filter hook read what they always read.

One door, and what a client sees

Every place that reads, sizes, copies, moves or deletes a stored message now goes through one seam, MessageStore. Everything that hands a message out reads the original bytes back through it, so a client sees nothing different: IMAP FETCH BODY[], RFC822.SIZE, BODYSTRUCTURE, SEARCH, SORT and THREAD; POP3 RETR, TOP, LIST and STAT; the webmail and the REST API; the rules, Sieve, the quota (which counts the original sizes), and a DKIM signature checked against the stored copy.

What changes is the file itself. Message.Filename over COM still answers the stored file's path. With compression on, a COM client or a script that opens that path reads a container, not RFC 822; with an object store, once the message is committed there is no file there. Read the message through the API instead.

Compression and attachments held once

Schema 6098. With MessageStoreCompression on, a message delivered to a mailbox, or put there by IMAP APPEND, is stored as a container: the message compressed with deflate, and each large MIME part, an attachment of at least MessageStoreDedupMinKB once decoded, cut out and held once as a blob that every message carrying the same bytes shares. A mail sent to forty people on this server with a 10 MB presentation keeps the presentation once.

Where	What
Control Panel (Windows)	Settings > Maintenance > Performance, the Message store tab
Control Deck	The two settings on the server settings page, and a Message store section on the dashboard
REST	message_store_compression and message_store_dedup_min_kb in /api/v1/settings; GET /api/v1/store; POST /api/v1/store/measure
COM (Windows)	Settings.MessageStoreCompression, Settings.MessageStoreDedupMinKB, Settings.MessageStoreStatus

Both are rows of hm_settings and take effect for the next message stored. MessageStoreDedupMinKB is 256 unless changed, 0 to 65536, and 0 holds nothing once and compresses only.

• The switch rewrites nothing. The reader never asks the setting what a file is: it reads either form, per file, by the file's first eight bytes. Turning it off leaves every stored message readable.
• Only delivery and IMAP APPEND write the new form. A webmail draft, a message saved through COM, the importers and the IMAP mirror write plain.
• A message larger than 64 MB is stored as it came, because a container is decoded whole in memory.
• Every container is checked before a byte is served: its structure, each blob against its hash, and the SHA-256 of the whole. A damaged one is refused and named in the error log (HM7350), never served as something else.
• Two messages share a blob when the part's encoded bytes are identical. The same file attached twice by a client that wraps its base64 differently is two blobs.

The command line, run as the user the server runs as (hMailServer.exe takes the same switches on Windows):

hmailserver --measure-store [--account <address> | --domain <name>]
hmailserver --compress-store [--account <address> | --domain <name>] [--decompress]
hmailserver --recount-store

Verb	What it does	Server
--measure-store	Changes nothing; prints what the store takes now, what it would take, and how much more it would save	May be running
--compress-store	Rewrites every stored file in place, each written beside itself, read back and compared byte for byte, then renamed over its name	Stopped
--compress-store --decompress	Puts the files back as plain RFC 822	Stopped
--recount-store	Sets every blob's count to what the stored files reference	Stopped

Exit 0 is a finished walk, 1 a file that could not be read or written, 2 no database or no such account or domain, and 4 a running server.

How much it saves has not been measured on a real mailbox. MessageStore.md holds a placeholder where the figures are to go, and the wiki claims none. Run --measure-store on your own store and read the answer. The document also records that its regression fixture had not yet run against a bench when it was written.

Object storage: S3 and Azure Blob

The messages in the mailboxes, the blobs and the recover-deleted store's kept copies can be kept in an S3-compatible store (AWS S3, MinIO, Wasabi, Backblaze B2's S3 API, Ceph RGW), signed with AWS Signature Version 4, or in Azure Blob Storage, with Shared Key or a SAS. The database stays the index, and an object is named by the path its file had. There is no schema step. Nothing changes until a store is named.

Every write is a local one. A message is placed on the local disk exactly as without a store, and a committer puts it into the store afterwards, and only then removes the local file. It runs every minute; what it has been handed is journalled in <data directory>\_ObjectStore\pending.txt and taken up again after a restart. So a delivery never waits for the store and never fails for it, and every read looks at the local disk first.

Property	REST key	Default
MessageStoreBackend	message_store_backend	file; the other value is object
MessageStoreObjectService	message_store_object_service	empty; s3 or azure
MessageStoreObjectEndpoint	message_store_object_endpoint	empty, which is the service's own
MessageStoreObjectRegion	message_store_object_region	us-east-1
MessageStoreObjectBucket	message_store_object_bucket	empty
MessageStoreObjectAddressing	message_store_object_addressing	auto; also path, virtual
MessageStoreObjectPrefix	message_store_object_prefix	empty
MessageStoreObjectAccessKey	message_store_object_access_key	empty
MessageStoreObjectSecret	message_store_object_secret	empty
MessageStoreObjectSasToken	message_store_object_sas_token	empty
MessageStoreObjectCaFile	message_store_object_ca_file	empty
MessageStoreObjectCacheMB	message_store_object_cache_mb	32

These are rows of hm_settings, set over /api/v1/settings or on the Control Deck's Settings > Server form. There is no COM member and no Control Panel page for them. The secret and the SAS are protected, never emitted, and left out of a backup, so after a restore onto another server they are entered again.

TLS verification is always on, and no setting turns it off. With MessageStoreObjectCaFile set, the certificate is verified against that file: beside the system's bundle on Linux, and alone on Windows.

Turning it on:

1. Make the bucket or container. This server makes none. For S3, an access key allowed s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket and the multipart upload calls, and a lifecycle rule that aborts incomplete multipart uploads after a day.
2. Name it, with message_store_backend still file.
3. Copy the store into it: hmailserver --copy-store-to-object-storage. It runs beside the server, verifies every object, and goes on where it stopped when run again.
4. Switch new mail to it: message_store_backend to object, taken within five seconds.
5. Remove the local copies when you are ready: stop the server and run the copy again with --remove-source.

Going back is the same the other way: message_store_backend to file, then hmailserver --copy-store-from-object-storage, then, once it says nothing failed, message_store_object_service to empty.

When the store cannot be reached, mail is not lost.

Door	Answer for a message only the store holds
A delivery	Placed on the local disk and done; committed when the store answers again
IMAP FETCH	NO [UNAVAILABLE]
POP3 RETR and TOP	-ERR [SYS/TEMP]
The webmail and REST	503

HM7473 says, once an hour at most, how many files wait on the local disk. HM7471 is the store refusing this server's credentials (401, 403), which is never taken for an absent message. HM7476 names a message that is neither on the disk nor in the store; no placeholder is written for it while a store is named.

What is thinner than it could be: one connection per request; one committer, a pass at a time; credentials are the two the settings hold, with no instance role, no workload identity and no Azure AD token; the Import Tool and the data directory synchronizer walk the local disk only; a changed prefix is not a move. The requests were proved against MinIO and Azurite on the project's lab on 25 September 2026. The server's fixture needs that lab, is skipped on a bench without one, and has not run in the Linux suite.

Mailbox encryption at rest

Schema 6099. A mailbox whose messages are stored encrypted to an X25519 key of the account's own, with AES-256-GCM, as the container's last stage, after compression. The key opens when the person signs in with their password and closes MailboxKeyIdleMinutes after the owner's last use (15 by default, 1 to 1440). Mail that arrives while nobody is signed in is encrypted all the same, because encrypting needs only the key's public half. When a password is lost, the domain's recovery key opens the key again. It is off for every account until an administrator switches it on for that account.

What it protects: a disk taken out of the machine, a snapshot of the volume, a backup archive, the data directory copied somewhere, the database dumped, once the account has signed in with its password after the switch.

What it does not protect. It does not protect the mail from the running server, which reads it while the key is open, nor from whoever runs the server, who can capture a password at sign-in. A person who needs that wants a zero-access mailbox, and the two are never on one mailbox.

Until the first sign-in the key is pending, held under this machine's own secret: the machine's DPAPI key on Windows, and the key file .hmailserver-secret-key inside the data directory on Linux. On Linux a copy of the data directory together with the database therefore opens every pending key. An account that only receives mail and never signs in with its password stays pending.

Switching it on, on the Control Deck or over REST:

1. The domain's recovery key first: the domain page's Mailbox recovery key card, or PUT /api/v1/domains/{domain}/mailbox-recovery with {"passphrase": ..., "hint": ...}. The passphrase is at least 12 characters, and this server never stores it. Keep it where this server is not.
2. Then the account: the Encryption at rest card, or PUT /api/v1/accounts/{address}/mailbox-encryption with {"enabled": true}. The switch-on ends every session the account had. A mail client sees its connection drop; the person signs in again with the password.
3. Then what is already there: hmailserver --encrypt-store --account <address>, with the server stopped.

Refused, with a sentence: an account whose domain has no recovery key; a zero-access mailbox; an account that signs in against a Windows directory; and an account with a TOTP second factor enrolled. The last runs both ways: a second factor cannot be enrolled on an account that has a mailbox key.

Sign-in. IMAP and POP3 sign a keyed mailbox in only with its own password. An app password, an OAuth2 token, a client certificate or a Kerberos ticket is answered NO [UNAVAILABLE] (POP3: -ERR [SYS/PERM]), and the IMAP master user is refused whether or not the owner's key is open. The webmail refuses a passkey, single sign-on or an app password (403, mailbox_key). SMTP, ManageSieve and DAV are not refused, since they do not read the mailbox.

Passwords. The person's own change wraps the key by the new password. An administrator's reset does so while the key is open or pending, and is refused while it is closed, with a sentence naming the recovery: POST /api/v1/accounts/{address}/mailbox-encryption/recover with the domain's passphrase and the new password.

A lost password and a lost recovery passphrase together are the mail lost. Nobody can open it. DELETE /api/v1/accounts/{address}/mailbox-encryption?confirm=forfeit removes the key, and with it every message still encrypted to it, for ever. It is refused while a legal hold covers the account.

Switching it off stores new mail as it comes. What was stored encrypted stays encrypted until hmailserver --decrypt-store, run with the server stopped and the domain's passphrase in the environment variable HMAILSERVER_RECOVERY_PASSPHRASE, never on the command line.

A keyed mailbox stays in its domain. A move to another domain is refused (409); a rename inside the domain goes ahead.

What stays in the clear: the queue and the copy a delivery works on, which is overwritten with zeros once placed, though a copy-on-write filesystem, a snapshot or an SSD may not honour that; the calendar and the address book; the quarantine, the archive and journal copies, and a forward; each message's row in the database, with its size, flags, folder and dates; logs and the trace.

An encrypted mailbox holds no part once. A blob shared between accounts cannot be under one account's key, so every part stays inside the message's own container. The cost is the de-duplication, for these mailboxes only.

Backups

• A backup with messages copies the data directory as always, with the containers, the Blobs directory and encrypted files as they are. It copies file by file while the server runs, so an archive taken while mail is delivered and deleted can hold a container whose part was written after the Blobs directory was copied: take the backup when the server is quiet. With an object store it also reads every message and blob that is only in the store into the archive; a store that cannot be read fails the backup.
• A full restore rebuilds the blobs' counts from the stored files. hm_storeblobs is not in the archive.
• An encrypted mailbox's key is carried with its account, and the domain's recovery key with its domain. A pending key on Windows opens only on the machine that wrote it; elsewhere the mailbox is opened with the recovery. On Linux it opens wherever the key file goes, and the archive does not carry that file.
• A restore onto a server that never turned compression on reads the containers the archive brought.

Known limits in 6.3.4

Limit	Where it is stated
Encryption at rest is on REST and the Control Deck, with no webmail card and no Control Panel page. The person's own view is GET /api/v1/me/mailbox-encryption	CHANGELOG, EncryptionAtRest.md
No COM for the switch, the recovery or MailboxKeyIdleMinutes	EncryptionAtRest.md
A search over an encrypted mailbox reads every message, and works only while its key is open. The full-text index holds none of its words	EncryptionAtRest.md
With compression on, Message.Filename over COM names the stored form; with an object store it names a path that holds no file once committed	CHANGELOG, MessageStore.md, ObjectStorage.md
An account's key is never rotated; the domain's recovery key can be replaced	EncryptionAtRest.md
The encryption fixture runs on Windows and is not in the Linux suite	EncryptionAtRest.md
No zstd: deflate is the one stage	MessageStore.md

Where to read more

• The wiki page: Message Storage
• The documents: MessageStore.md, ObjectStorage.md and EncryptionAtRest.md, each with its error codes and what proves each claim
• Zero-access mailboxes: the wiki's Encryption for People, and on this forum End-to-end encryption
• The documentation page: Backup and restore
• On this forum: The Control Deck, where the cards and the settings form are, and Backup and restore, for what the archive holds and what it leaves out