Staged rollout and automatic rollback

About 7 min

Configuration and node-program changes are never pushed to every node at once. They are delivered in stages, so a problem stops where its blast radius is smallest.

Step 1 / 3
1 node

The next stage starts only when every node in this stage has installed it with no new alert

Step 2 / 3
10% of nodes

The same bar

Step 3 / 3
All nodes

Rollout complete

Configuration published
Nodes
MergingEdits within 10 seconds merge into one version
Pause auto publishing30 minutes
Local health watch5 minutes after taking effect
Configuration moves 1 node → 10% → all nodes; if a node rolls back by itself, the rollout stops at the current stage

Configuration changes: three stages

Configuration changes such as sites and rules move 1 node → 10% of nodes → all nodes. The next stage starts only when every node in the current stage has installed the version with no new alert. If a node fails to load it, the rollout stops at the current stage and raises an alert.

  • Several edits made within 10 seconds are merged into one version, so a run of saves does not create a string of tiny versions.
  • You can pause automatic publishing for 30 minutes to fit a change window.

Under System Admin › Upgrade & Maintenance › Node Upgrade, the "Config publishing" section shows the version being delivered and its stage, and lets you intervene:

ActionEffect
Advance nowSkip the wait and move to the next stage
Pause / ResumePause or resume automatic advancing
Roll back to previousReturn the configuration to the previous version
Push to all nodes nowSkip the 1-node and 10% stages and deliver to all nodes; requires ticking a confirmation box and is written to the operation log
Caution "Push to all nodes now" gives up staged protection. Use it only for urgent fixes.

Node-local automatic rollback

For 5 minutes after a new configuration takes effect, each node watches its own health. If the 5xx ratio, worker crashes or the rule budget clearly worsen, the node returns to the previous configuration by itself and refuses that version. The master raises a "node automatically rolled back config" alert and the rollout stays at the current stage instead of widening.

The node list and node edit page show "Config rolled back" with the version and the reason. After the cause is fixed, click "Re-push" to deliver the same configuration again.

Node program upgrades: staged and reversible

Upgrading the node program in System Admin › Upgrade & Maintenance › Node Upgrade is also staged: first one node, then 10%, then all; every stage must fully succeed before the next. Each upgrade creates a rollout record, and an in-progress record can be paused ("Pause", "Resume") or reversed with "Roll Back": this stops the rollout, and nodes already upgraded return to their pre-upgrade version.

The upgrade replaces the whole package and switches atomically. If the new config fails its check or nginx is not ready, the node switches back to the old version and keeps serving. To go back, use "Roll Back" on the rollout record for the node program, or "Roll back to previous" in the configuration release panel for the configuration.

Before restarting nginx, a node is taken out of DNS and waits for the DNS cache to expire, so there is basically no downtime; it restarts directly only when the master is unreachable or too old or the line group cannot spare it, so upgrade in batches at off-peak hours. Nodes refuse downgrades by default: only a rollback ("Roll Back" on a rollout record) or a rollback upgrade you confirmed can move a node to an older version. See Online upgrades.

Upgrading the master

Upgrade Master and Roll back master are also under System Admin › Upgrade & Maintenance › Master Upgrade. Tick "Auto-upgrade nodes when the Master is upgraded" to have nodes follow, going through the same stages. While a master upgrade is still queued or installing, "Stop Upgrade" cancels it; a switch that has already started will finish. See Online upgrades for the steps.

Tip When a node rolls back automatically, read the version and reason in the alert first. Don't re-push immediately; do it once you've confirmed the problem is fixed in the configuration.
Get started

Try the live demo

Open the live demo, or contact us for a quote and the install command