6 min read
Why S3-compatible is not enough for durable state
The object-store behaviour Peren needs for cell ownership, replication and recovery—and why an endpoint URL proves none of it.
“S3-compatible” is a useful label. It usually tells you that a service accepts familiar requests, credentials and object operations. It does not tell you what happens when two nodes race to update the same record, a write times out halfway through, or recovery asks for an exact byte range.
Those details matter because Peren puts object storage in the live stateful path. Ownership records decide which node may execute a cell. Snapshots and write-ahead-log objects let another node recover acknowledged state. The bucket is doing more than holding backups.
The API gets Peren through the front door. The behaviour decides whether a write is safe.
Ownership needs conditional writes
When a cell has no owner, a node creates its ownership record only if the object is still absent. When ownership changes, the new node replaces the record only if the version it read is still current.
Those conditions choose one winner when nodes race. If A and B both read version 17, A may replace it with epoch 18. B’s later update based on version 17 must fail. An implementation that accepts both writes has turned a fenced ownership protocol into last-write-wins storage.
The provider may express that condition as an ETag, generation number or another opaque comparison token. The name does not matter to Worker code. The decision does: create this object only if it is absent, or replace the exact version Peren read.
Provider qualification
Compatibility is behaviour, not branding
Exactly one creator can establish the first ownership record.
Two nodes may both believe they created the cell owner.
A timeout is not a rejection
Distributed writes have three outcomes:
- applied: the provider confirms the write;
- rejected: the condition did not match; or
- ambiguous: the client lost the answer.
The third outcome causes trouble. If a connection times out after sending a write, the object may exist even though Peren never received the success response. Treating that timeout as a rejection and making a fresh decision can conflict with the decision already stored.
Peren leaves the result as unknown until it can reload the authoritative object and compare the expected identity, version and digest. Guessing would be easier, but it would also be unsafe.
Timeout after send
Unknown does not mean rejected
This rule also applies to replication. Publication uses deterministic object identity. Finding the expected bytes after an uncertain upload can confirm success. Finding different bytes at that identity is corruption, not an invitation to overwrite them.
Recovery needs exact ranges
Peren restores cell storage from a valid snapshot baseline and the ordered WAL material that follows it. Range reads let recovery fetch bounded sections while retaining frame and checksum boundaries.
The provider must honour the requested range exactly and return enough information to detect incomplete or malformed data. Returning a full object for every range, shifting the bytes or silently truncating a response changes what recovery is able to validate.
Peren also needs stable listing and reads for the objects in a recovery chain. If a confirmed object disappears from a listing, recovery sees what looks like a missing position. Diagnostics must tell the operator whether an object is absent, malformed, unsupported or unavailable because those problems have different fixes.
What Peren asks the object store to do
Peren does not ask several nodes to edit one SQLite file in object storage. It publishes versioned ownership records and immutable replication objects with identity and checksum information.
The object store does not execute a database transaction across the fleet. Its job is smaller and precise:
- serialize conditional ownership updates;
- retain published objects under deterministic identities;
- return exact bytes for verification and restore; and
- make uncertainty observable so Peren can reconcile it.
None of these operations looks dramatic on its own. Together, they decide whether the fleet has one owner or two, and whether an acknowledged write can be recovered.
Test the endpoint you will run
A compatibility table can tell you which providers have been exercised. It cannot test the endpoint in front of you. A gateway, proxy, emulator or storage mode can change semantics while the product name stays the same.
Peren’s conformance checks send the important operations to the configured service. If credentials are missing, the check is skipped and the provider remains unqualified. Passing against a local substitute says nothing about a hosted endpoint, and basic object CRUD says nothing about ownership.
Keep credentials in environment variables and name those variables in the fleet configuration:
[bucket]
kind = "s3"
endpoint = "https://objects.example.com"
bucket = "peren-fleet"
region = "us-east-1"
access_key_env = "PEREN_BUCKET_ACCESS_KEY"
secret_key_env = "PEREN_BUCKET_SECRET_KEY"Then exercise the storage operations Peren depends on:
peren conformance storage config.tomlUse peren diagnose config.toml --storage-test when investigating the same boundary operationally. Neither command should print the resolved secret values.
Run the check before the provider receives durable cells, and run it again after a provider migration, gateway replacement or storage-mode change.
Fail clearly
If the provider cannot perform the required operations, Peren should refuse qualification or startup for that durable path. Falling back to process memory or local files would leave nodes with different sources of truth.
Treat the object store as part of the stateful runtime. Watch its latency at the acknowledgement boundary, preserve its errors in diagnostics and call it compatible only after exercising the behaviours Peren relies on.
Read the provider compatibility requirements, durability model and degraded-operation guide before choosing storage for a fleet.