Recovery after node loss

Recovery restores a cell’s SQLite database from snapshot and WAL records in the bucket, then continues under a valid ownership lease. There is no peren cell inspect command. Confirm recovery with an HTTP response that returns the expected state after restart. For a production bucket, run peren conformance storage first.

Recovery after node loss

Symptom

A process stops or restarts. Later requests miss earlier mutations, fail while acquiring ownership, or return errors while the bucket is unreachable.

Checks

1. Confirm the fleet file still points at the same bucket path or endpoint the node used when it published state.
2. For a production or multi-node bucket, run storage conformance before relying on recovery:

peren conformance storage fleet.toml

The command checks conditional writes and ranged reads. If it fails, do not use that provider for recovery.

3. Confirm the Worker bundle and Durable Object namespace configuration still address the same cell identity you expect.
4. Send the same HTTP request that previously mutated state and compare the response body to the last known value.

Restart on one node with a file bucket

On a single node using [bucket] kind = "file", stop the process, start it again against the same path, and repeat the request. Peren acquires ownership, restores published replica bytes into the cell’s SQLite file when records exist, then runs the Worker. A matching response proves local restart recovery for that cell.

Keep the data directory and file-bucket path intact across the restart. Deleting either removes the local database or the published records the next start needs.

Peer takeover

After a dispatch finishes, the owning node releases ownership so the owner field is clear. Another node that shares the same bucket can then acquire the cell, restore published snapshot and WAL records and serve the next request.

Ownership recovery that replaces a still-recorded owner advances the ownership epoch and fences the previous lease. A node holding the older epoch cannot publish or commit further state for that cell. The live request path acquires when no owner is recorded; it does not automatically run dead-owner recovery when a previous owner is still present. There is no operator CLI that moves cells between nodes.

If acquisition fails after a crash mid-dispatch, check whether the ownership record still names an owner, whether the bucket accepts conditional writes and whether published replica objects for the cell are readable. Fix storage reachability and ownership before expecting a peer to serve the cell.

Failure triage

Keep node.node_id, node.listen, the data directory and the bucket configuration stable across supervised restarts. Changing node.node_id creates a new peer identity; it does not prove that the previous process has safely drained.

When a cell does not recover as expected:

1. Run peren conformance storage fleet.toml against the same bucket configuration.
2. Check node logs for fencing, publish, checkpoint or restore errors.
3. Confirm the current bundle still contains the Durable Object namespace and service binding that address the cell.
4. Run peren storage collect fleet.toml --dry-run to see whether obsolete replica objects are collectible. The command keeps the current restore plan and honors limits.replica_retention_secs.
5. Use peren storage erase fleet.toml --cell <cell-id> --dry-run only when the product record for that cell has already been deleted and you intend to remove its durable recovery state.

Do not erase a cell to fix a stuck recovery unless the desired outcome is permanent loss of that cell’s prior state. Erasure removes restore inputs; it is not a repair operation.

Read Durability for publish and fencing details and Placement for ownership versus preference.