Operations

Upgrading

Replace the binary and restart. A restart is a first class state rather than an event to avoid: the only state the process carries is its identity map, which a restart is allowed to empty. An interrupted segment is recovered and sealed under the identity of the run that wrote it, and a fresh run beginning at sequence zero is read as a restart rather than as a gap.

What a restart costs, precisely:

• Records already written are safe. Sealed segments are immutable, and the active segment is recovered on next start with at most a torn final line discarded and counted.
• Events published during the gap are lost, and counted. The process is a subscriber, so nothing replays them. The window is the length of the restart.
• The identity map starts empty. Blocks whose prefix chains began before the restart cannot be given a portable identity until they are stored again, so content_unresolved rises on heartbeats and cross node figures are lower bounds until the engine's cache turns over. An engine restart clears its cache and resets this completely, so the cost never outlives the cache itself.

Roll one node first and confirm the record stream before continuing, the same way you would with anything else in the serving path, even though this is not in it.

Diagnostics

The record stream is the diagnostic. There is no local summary to collect, because the process holds no state that could differ from what it wrote.

When something looks wrong, three things answer most questions:

To hand a bundle to us: the heartbeats and lifecycle records from the window in question are sufficient and contain no block identities at all. Sending cache records is rarely necessary and they are pseudonymized under your key, which we do not hold.

Retention

On the node, one flag bounds the record directory:

Past the cap the oldest segments are reclaimed, oldest first by run and then by sequence number. What survives is therefore the newest data. The loss is declared in band, naming the run and the sequence range that went, and it leaves a structural gap in the sequence as well, because sustained pressure can reclaim the very segment that carried the declaration.

Configure your shipper so that a vanished source file is an expected outcome rather than a retryable error. Reclamation can remove a segment between the shipper listing it and opening it, and a shipper that stalls its queue on a missing file converts a bounded, recorded loss into total shipping stoppage.

Backup and restore verification

Records land on the node and are moved to wherever you collect them. Two properties make recovery from that collection possible, and they are worth knowing because they are what makes a restore complete rather than approximate.

• Re ingesting is a no operation. A batch is identified by the content hash of its file, checked before anything is written, so replaying a directory cannot double count.
• Segments are immutable once sealed. A file that arrived is final, so there is no version of it to reconcile.

Together those mean a reader that lost time recovers by walking the collection again. The practical consequence for your retention policy: keep shipped segments at least as long as your backup interval. They are the delta that closes a restore gap, because a restore returns to the last backup and everything after it is replayed from them. Keeping them for less makes a restore permanently lossy.

Running a warm spare analyzer

You do not need failover machinery, and you should not build any. The analyzer is a pure function of the segment collection: give two analyzers the same collection and they arrive at the same state without ever talking to each other. A warm spare is just a second analyzer reading the same collection. Failover is repointing your dashboard at the spare. That is the whole procedure.

Check convergence yourself rather than trusting the claim: run a report against each analyzer over the same window and diff the two snapshots. The analysis is identical line for line. The only differences are the generation timestamp and each analyzer's own ingest tally, which describe the run rather than the fleet. Our release pipeline holds this with a drill: two analyzers over one collection, the primary killed mid stream and brought back, and both required to produce the same analysis.

One deleter, and it pauses during overlaps

If you automate cleanup of the segment collection, follow one rule. A segment may be deleted only after it has been ingested and a completed backup covers it, and only one thing in your system may do the deleting. Age is never a reason to delete: an old segment that will not go away should page someone, not disappear.

The rule earns its keep during overlaps. While two analyzers read the same collection at once, during a spare promotion or an upgrade, pause the cleanup, and turn it back on when one analyzer owns acknowledgment again. If both delete, each can remove segments that only it has ingested, and the other has lost them permanently.

Upgrading or moving the analyzer

Blue green, with machinery you already have. Bring up the new machine, restore the latest backup onto it, let it walk the segments written since that backup, then repoint the dashboard and retire the old machine. Nothing coordinates, for the same reason the warm spare works.

A side effect worth noticing: every upgrade is also a restore test, because the upgrade path is the restore path. If upgrades are routine, proven restores are too. Keep cleanup paused while both machines run.

After an outage

Bring the analyzer back and it catches up on its own; there is nothing to replay by hand. Catch up reads files in parallel, which is safe here because a batch is identified by the content of its file and re ingesting one is a no operation, so the parallel result is byte identical to the sequential one. Watch the backlog drain on /metrics.

What recovery promises

After any restart or restore, the dashboard comes back reading the newest records first while history backfills behind it, and a multi day outage heals completely rather than leaving holes. A drill in our release pipeline kills the analyzer, hands it a simulated multi day backlog, and verifies that machinery on every release.

The stated targets are 15 minutes to a live dashboard and 12 hours to fully heal a 72 hour outage. The drill runs at reduced scale, so today those are targets we state, not figures we claim. The absolute numbers get published when the long calibration runs measure them on a real fleet, which is the same standard every other unmeasured figure on this site is held to.

Key rotation

Block identities are derived under your key, so a new key produces new identities for the same blocks. There is no bridge between the two, deliberately: any bridge could only connect a post rotation eviction back to its pre rotation store while the bridge was still active, and the blocks most worth accounting for are the ones that may never be evicted at all. A continuity guarantee whose completeness cannot be bounded is worse than a declared discontinuity.

So a rotation is a clean break, and the timing is what makes it free or expensive.

The free path

Rotate the key as part of the deploy that restarts the engines. An engine restart clears its cache, so every residency closes for that reason regardless and nothing spans the cut. There is nothing to bridge because there is nothing crossing.

The word engine is doing the work there. Swapping the key file while engines stay warm leaves blocks alive across the boundary, which is supported and honest but imprecise: those blocks appear as an open interval before the cut and a new block after it.

What you lose, and what you do not

You lose block level continuity across the boundary. You do not lose your history. Trends and totals carry no block identity at any horizon, so they cross every rotation untouched. The figures on either side of the boundary are sound; only figures spanning it are lower bounds, and only for questions about individual blocks.

During the roll

While a fleet is partly rotated it genuinely holds two identity spaces, and the report will say so rather than quietly returning zero. That is correct behavior, not an alarm: cross instance figures are unavailable until the roll completes, and per space figures remain sound throughout.

A single node on an identity nobody else shares is the different case, and it means the key file was generated twice rather than distributed.