Outbound shaping and IP warm-up: one slow destination must not stall the queue

Progressiverobot

Since 6.3.4 the delivery queue shapes what it sends. It pauses a destination that asked it to slow down, caps how many sessions it holds open to one destination and how many deliveries one tenant has in flight, and holds a new outbound address to a daily ceiling at each large provider. Everything in this topic is new in 6.3.4 except MaxOutboundPerDestinationPerMinute, a fixed per-destination rate that 6.3.3 already has. What to know first: of what this topic covers, two things change behaviour on upgrade without a switch. The back-off is on by default, and a deferral this server chose does not spend a message's retries until the message has been queued for longer than its retry schedule spans; only a failure at a remote does. The second covers every hold in this topic, and so changes an upgraded server that already sets MaxOutboundPerDestinationPerMinute. The connection ceilings, the tenant share and the warm-up do nothing until you set them.

The order of the checks

For each batch of recipients on one target, the server judges in this order, before any connection is opened:

The remote domain policy's largest message: 5.3.4, permanent, measured on the queued message.
The destination's back-off window. Held until the window ends; then one delivery goes through as the probe, and any other arriving meanwhile waits a minute.
The policy's messages a minute (max_messages_per_minute).
The connection ceiling: the policy's max_connections where the policy sets one, otherwise the server-wide MaxOutboundConnectionsPerDestination.
The tenant share, MaxOutboundConnectionsPerTenant.
The server-wide MaxOutboundPerDestinationPerMinute.
DNS: the MX lookup (or the route's host) and MTA-STS; then the IP warm-up's daily ceiling; then, where DANE applies, whether the MX set is DNSSEC-validated.
Gateway encryption, for a domain whose policy encrypts. The copy is made here, so a batch that is only going to be deferred is never encrypted.
The connection attempts.

Every one of 2 to 6 is a deferral with 4.4.5 ("mail system congestion") and a sentence naming the rule. The target each check is keyed by is what the batch is delivered to: the recipient domain of an MX delivery, or the host of a route or a relay, or its address when the route names an IP literal, lower-cased.

The back-off: a destination said slow down

When a destination answers a delivery with a request to slow down, it gets a back-off window: nothing is sent to it until the window ends, one delivery then probes it, and the same answer doubles the window. Deliveries to every other destination are untouched.

What is read as a signal.

Reply	Read as
Code 421, unless an exclusion below applies	A signal, whatever the text, at the banner or anywhere after it
A 4xx whose enhanced status is `4.7.0`, `4.7.28`, `4.7.500`, `4.7.650`, `4.7.652` or `4.3.2`	A signal
A 4xx whose text, lower-cased, contains one of the phrases below	A signal

The phrases: rate limit, rate-limit, ratelimit, too many messages, too many connections, too much, unusual rate, unexpected volume, temporarily deferred, temporarily rate, throttl, try again later, server busy, back off, slow down. They are English, and a remote that says the same in another language is read by its codes alone. The table is fixed and cannot be edited.

What is deliberately not a signal, each of which wins over the text and over the 421 rule:

any 5xx: a refusal is not a request to wait;
4.7.1, which greylisting and policy refusals answer, and any reply whose text says greylist or graylist;
4.2.x: one mailbox, never the destination;
4.5.3: too many recipients for one message;
4.4.x: 421 4.4.2 timeout exceeded is the remote's idle timeout on a slow DATA;
a connection that was refused or timed out, a DNS failure, and this server's own deferrals.

The window and the probe. The first slow-down reply opens a window of OutboundBackoffMinutes. While it is open a batch for that destination is held with 4.4.5, and the application log says:

Delivery to <target> deferred: the destination asked this server to slow
down at <time> (<reply>); next attempt after <time>.

When the window has passed, the first delivery goes through as the probe. If it is delivered, or fails with anything that is not a slow-down reply, the window is cleared. If it is answered with a slow-down reply, the window doubles, up to OutboundBackoffMaxMinutes. A probe nobody ended - a delivery thread that died with it - goes stale after fifteen minutes and the next delivery takes it over.

Setting	Default	Range	Meaning
`OutboundBackoffMinutes`	`2`	0 to 10080	The first window. `0` turns the back-off off
`OutboundBackoffMaxMinutes`	`60`	1 to 10080	The ceiling a doubling window stops at

The two refuse to cross each other on every surface: to move both, move the ceiling first when raising and the base first when lowering.

A relay is one destination. With SMTPRelayer set, or a route through a smart host, one slow-down reply from it holds every message that goes through it for the window.

One miss, stated in the document. The delivery client keeps a recipient's first non-fatal reply, so a genuine 421 from a second MX host after a plain 4xx from the first is not seen for that recipient.

Connection ceilings and the tenant share

Both use the same counting semaphore, and it never blocks: a delivery thread that waited for a slot would be a thread not delivering anybody else's mail. Over the ceiling the batch is deferred one to five minutes.

Setting	Default	Range	What it bounds
`MaxOutboundConnectionsPerDestination`	`0`, off	0 to 1000	Sessions open at once to one target that no remote domain policy sets `max_connections` for
`MaxOutboundConnectionsPerTenant`	`0`, off	0 to 1000	Deliveries one tenant may have in flight at once

The per-domain ceilings are on the remote domain policy, whose TLS half is in Requiring TLS:

Policy field	0 means	Beyond it
Largest message (KB)	no limit of ours	Not attempted: no connection is opened, and the message is bounced at once with `5.3.4`, naming the policy and both sizes
Simultaneous connections	unlimited	Deferred with `4.4.5` for one to five minutes. The count covers every domain the policy's pattern matches
Messages a minute	unlimited	Deferred with `4.4.5` for one to five minutes. Independent of `MaxOutboundPerDestinationPerMinute`; both apply

A policy's connection ceiling wins over the server-wide one wherever the policy sets it.

Who a tenant is. The hosted domain of the envelope sender, with an alias domain resolved to its domain. A sender this server does not host - mail relayed for an IP range or a route, forwarded external mail, the null sender - is the one tenant (external), so relayed mail cannot starve the hosted domains either.

The tenant share is not a volume control. It is a share of connections, bounded by MaxDeliveryThreads: a share larger than the thread count is no share at all. The per-account sending limits are the volume control. This server's own delivery reports have the null sender, so they share (external) with relayed mail.

Where the settings are. The four new settings are on the Control Panel's Delivery of e-mail page beside MaxOutboundPerDestinationPerMinute (Windows), on the Deck's Settings view, over COM as Settings.OutboundBackoffMinutes, OutboundBackoffMaxMinutes, MaxOutboundConnectionsPerDestination and MaxOutboundConnectionsPerTenant (Windows), and over REST on /api/v1/settings as outbound_backoff_minutes, outbound_backoff_max_minutes, max_outbound_connections_per_destination and max_outbound_connections_per_tenant. In hm_settings the two connection rows are named MaxOutboundConnsPerDestination and MaxOutboundConnsPerTenant, because the column holds thirty characters.

A deferral this server chose is not counted

When every non-fatal failure of a delivery pass was one of this server's own deferrals, the message's next try is set to the longest deferral of the pass with the tries counter untouched, and the log says so:

Delivery deferred by this server's own shaping, not by a remote server.
Scheduling it for later delivery in N minute(s); the attempt is not
counted against the retry limit.

When any recipient of the pass failed at a remote, the counted path runs as it always has. Before this, with a small retry count, a message could be bounced having never been offered to anybody.

The hold is bounded. A message queued for longer than the span its counted schedule would have allowed - (retries + 1) times the minutes between tries, from the time it was accepted - takes the counted path from then on, so that a destination that says "slow down" for ever still ends in a bounce:

Deferred by this server's own shaping, but it has been queued for N
minute(s), longer than the M minute(s) its retry schedule spans. This
deferral is counted, so that the message is not held for ever.

A held pass is a first pass only once. DKIM and ARC signing, the mirror copy and a rule's policy notice happen once, however many passes a message is held through. Nor does a held pass forward, copy, reply or call a rule's script function again: those four actions run once per counted attempt.

IP warm-up

Schema 6093. A schedule holds one outbound address to a daily ceiling of recipients per provider, rising day by day along a fixed plan; the queue sends up to the ceiling and holds the rest, usually until the next UTC day. Nothing is held until an administrator makes a schedule.

A schedule is about one address, and the address has to be bound. It is the local address a delivery leaves from: a global rule's bind to address action when one applies, otherwise SMTPDeliveryBindToIP (Bind outbound connections to IP, smtp_delivery_bind_to_ip). A delivery that binds no address leaves from whatever the operating system chooses, and no schedule is consulted for it. So a schedule does nothing, even enabled, until the address is configured on the host and one of those binds it. The server can check the setting and nothing else: the listing says in_bind_setting, and the Deck warns when it is false.

The provider is decided from the mail host, never from the recipient's domain. A company on Google Workspace is reached through Google's MX hosts, and it is Google that judges the address. The first host the batch will try decides.

Provider	Key	Mail host names
Google	`google`	`google.com`, `googlemail.com`, `aspmx.l.google.com`
Microsoft	`microsoft`	`outlook.com`, `hotmail.com`, `protection.outlook.com`, `olc.protection.outlook.com`
Yahoo	`yahoo`	`yahoodns.net`, `yahoo.com`, `aol.com`
Apple	`apple`	`icloud.com`, `apple.com`
Proton	`proton`	`protonmail.ch`, `proton.me`
Every other provider	`other`	everything else, counted together at five times the named providers' ceiling

Mail relayed through a smart host that is not one of those names is other, whoever the smart host delivers it to.

The plans, in recipients per named provider per day. After a plan's last day there is no ceiling, and the schedule holds and counts nothing.

Day of `standard`	Ceiling	`other`
1	50	250
2	100	500
3	200	1,000
4	500	2,500
5	1,000	5,000
6	2,000	10,000
7	5,000	25,000
8 to 10	10,000	50,000
11 to 14	20,000	100,000
15 to 21	50,000	250,000
22 to 30	100,000	500,000

The plans side by side:

Plan	Length	Day 1	Last day	Rule
`standard`	30 days	50	100,000	the table above
`gentle`	60 days	25	50,000	half of each ceiling, twice the days
`fast`	15 days	100	200,000	twice each ceiling, half the days

Days are UTC days, whatever the server's time zone, and a start later than today is refused.

Recipients, not messages. A message to forty people at Gmail is forty to Google. Before a delivery connects, its recipients reserve room out of the day's allowance; when the attempts are over, every recipient that was delivered is added to sent and the reservation is let go. A recipient that was not delivered is not counted. So a provider that refuses the warming address is offered as many recipients a day as the queue has; its slow-down replies are answered by the back-off. The counts are written to hm_warmupcounts at every change, so a restart resumes the day. Local delivery is not counted.

The hold. Recipients that do not fit - the last of the batch, not chosen by any priority - are marked 4.4.5. When the day's ceiling was reached, the next try is one to five minutes after the next midnight UTC:

SMTPDeliverer - Message 1234: 1 recipient(s) held by the IP warm-up of
192.0.2.10: day 1 of its standard schedule allows 50 recipient(s) a day at
Google, and today's have been sent. Next attempt after midnight UTC, in
723 minute(s); the attempt is not counted against the retry limit.

How long a backlog is carried. With the default schedule - four retries, sixty minutes apart, a five-hour span - a recipient queued in the morning comes back on each of the next five UTC days and is returned if the fifth still has no room. Send no more from a warming address than the sum of those days' ceilings, or raise the retry count for the duration of the warm-up.

Managing schedules. On the Control Deck's Delivery queue page, in the Warm-up box, or over REST. The desktop Control Panel has nothing for it, and there is no COM interface.

Route	What it does
`GET /api/v1/warmup`	Today's UTC date and the schedules, each with its day, `in_bind_setting`, and per provider `sent`, `deferred` and `ceiling`
`POST /api/v1/warmup`	Adds one: `address` (required), `plan` (`standard` by default), `started` (`YYYY-MM-DD`, today by default), `enabled`
`PUT /api/v1/warmup/{id}`	Changes `plan`, `started` or `enabled`. The address cannot be changed
`DELETE /api/v1/warmup/{id}`	Removes a schedule and its counts
`GET /api/v1/warmup/{id}/history`	The counts, newest first, with the day and ceiling in force then

Reading takes view_queue and changing takes manage_queue; a credential confined to named domains is refused. A second schedule for one address is 409. Back up settings carries the schedules but not the counts.

When the database does not answer, the queue does not stop: schedules that cannot be read (7210) and a day's count that cannot be read or started (7211, 7212) leave the delivery going as though the address had no schedule. A count that cannot be saved (7213) is kept in memory; if the service restarts first, the day can go past its ceiling by that much.

Where to see it

GET /api/v1/queue/shaping (view_queue, server-wide): the open back-off windows - destination, since, until, the window's minutes, the reply and the host that gave it, how many deliveries it has held, whether a probe is out - and the connections open under each ceiling and each tenant.
DELETE /api/v1/queue/shaping/{destination} (manage_queue) clears one window once you have fixed the cause: 200 with {"cleared":true}, 404 when no window is open for that name, and an event on the audit trail.
The Deck's Delivery queue shows the same as Slowed destinations, with a Resume button, and the connection counts as they were when the page was drawn.
The application log says, for each batch held, which rule held it and until when.

What is per process. The windows and the connection counts live in memory. A restart forgets every window, which costs at most one more slow-down reply per destination. Nothing is shared between the nodes of a warm-standby pair.

What it does not do

The signal phrases, the providers and the plans are fixed. A sixth provider, or a plan of your own, is not possible.
No per-tenant volume quota beyond the per-account one: this shares connections, not messages a day.
No pool of addresses. A schedule is about one address, and mail that did not fit waits.
No reputation feedback. The ceilings rise by the calendar, not by how the providers are answering.
No delayed delivery report. A sender learns of a long hold only when it ends in a bounce.
A deferral by a rate or a ceiling costs one delivery pass per message every one to five minutes for as long as the backlog lasts.
The warm-up counts are not exported as metrics.

Not proved by any test, according to IpWarmup.md: room held by deliveries in progress; a later host of the other address family skipped for being reached from another warming address; what a database failure does (7210 to 7213); the audit trail's entry; the configuration backup's round trip of the schedules; and an MX lookup's own host names reaching the classification, since every delivery in the fixture goes through a route.

Where to read more

Operator documents at the v6.3.4 tag: OutboundShaping.md, IpWarmup.md and RemoteDomains.md.
Wiki: Outbound Shaping and Routes and Relays.
Public documentation: 26.7 The delivery queue and the retry schedule and 14.1 Sending through a smart host.
On this forum: The Control Deck, Upgrading to 6.3.4, Requiring TLS for the remote domain policy's TLS half, and DKIM rotation, DMARC and TLS reports, complaints, bounces and the reputation watch for bounces, complaints and block lists.