Durability

A cell keeps its mutable state in SQLite on the owning node. After a dispatch commits, Peren publishes recovery records to the bucket so a later acquisition can rebuild that SQLite database. Durability across process restart or node replacement therefore depends on both the local commit and a successful publish.

Durable turn

Commit and publish

1. The Worker mutates cell storage through Peren.storage inside a transaction or mutation.
2. SQLite records the committed revision on the owning node.
3. Peren reads the database image and any new WAL frames, then publishes them to the bucket under the current ownership epoch.
4. Publication returns a receipt that must match the cell, epoch and generation. A mismatched receipt leaves the cell draining and blocks further mutable dispatch for that resident.

Peren.storage.mutation records the callback result under the mutation id. A later call with the same id returns the stored result and does not run the callback again.

Replica manifest

Replica objects are immutable. The mutable restore pointer is cells/<cell>/root.json. That root names one complete restore plan: the snapshot generation, the latest revision and every WAL segment needed after the snapshot. Restore reads the root first and rebuilds only the state it names.

Checkpointing writes a new complete database image, resets the WAL boundary and prunes superseded replica objects. The limits.checkpoint_threshold_bytes setting, or a service-level override, triggers a checkpoint when a dispatch publishes at least that many WAL bytes. Set the threshold to 0 to disable threshold-triggered checkpoints.

peren storage collect uses the same root reachability rule. It keeps the root snapshot and WAL objects, then removes replica objects that are no longer reachable from the authoritative root after limits.replica_retention_secs has elapsed. peren storage prune remains an alias.

Checkpoint and collection decisions are per cell and are derived from the cell’s current root manifest. A checkpointed cell can be collected independently from other cells once its obsolete replica objects are outside the configured retention window.

peren storage erase is the explicit operator path for permanent cell erasure. It removes the authoritative restore pointer and replica objects for one cell id after a dry run or an explicit --force. Use it only when the application no longer needs that cell and you accept that Peren cannot restore its prior state from the fleet bucket.

Lifecycle trace points

Each mutable dispatch passes through the same observable lifecycle:

Service binding fetches and Durable Object fetches propagate the incoming trace id into the child cell dispatch. Use peren trace fleet.toml --request-id ... to inspect the local lifecycle stream, or configure [otlp] to export spans to an OTLP collector.

Ownership fencing

A cell has one current owner for mutable work. Peren records ownership with a conditional write on the bucket. Publish and checkpoint refuse when the stored owner is empty or the epoch no longer matches the lease. A fenced cell stops accepting mutable dispatch rather than writing under a stale epoch.

Provider requirements

The bucket provider must support conditional writes and ranged reads. Those behaviors back ownership updates and replica restore. Before trusting a provider for production recovery, run:

peren conformance storage fleet.toml

If peren conformance storage fails, do not use that bucket for fleet recovery. Ordinary puts and gets can still succeed.

Limits

A successful local SQLite commit is not durable for recovery until Peren publishes the corresponding snapshot or WAL records. If the bucket is unreachable during publish, the cell drains and refuses further mutable work until storage is fixed. File and memory buckets are fine for development. Multi-node recovery needs a shared bucket that passes peren conformance storage.

Read Recovery after node loss for operator checks and Placement for how ownership differs from preference.