peren
Menu
Search Peren documentation
Start typing to search the documentation.
    Browse documentation
    On this page
    Search Peren documentation
    Start typing to search the documentation.

      Peren documentation

      Drain a node

      Record a draining membership state, or refuse new work on a live peer listener.

      Symptom: a node must leave service for maintenance, replacement, or shutdown, and new work must stop reaching it.

      Node drain

      Node drain
      Registry drain and live drain are different. REGISTRY DRAIN LIVE DRAIN writes sets peren node drain draining in registry admission unchanged POST /control/v1/node/drain peer listener admission draining new work returns 503

      There are two drain paths. They are not the same command.

      Fleet registry drain

      peren node drain writes draining into the local fleet registry under the data directory. It does not move cells. It does not change admission on a running process.

      peren node drain peren.toml --node 00000000-0000-0000-0000-000000000001 --reason maintenance

      Representative output:

      node 00000000-0000-0000-0000-000000000001 draining

      Live admission drain

      The live path that returns 503 for new work is the peer control endpoint:

      curl -sS -X POST http://127.0.0.1:7000/control/v1/node/drain

      That call sets readiness false and admission to draining on the running process. New top-level work on that node is refused with 503. The endpoint exists only on the peer listener.

      If [control].require_signed_mutations = true, the same request must include Peren control-signature headers. The signature targets the node id and covers the method, path, request body, timestamp, and nonce. Unsigned live drain requests return 401 in that mode.

      Response fields include admission, ready, active, websocket_resident, placement_weight, placement_pressure, disk_removal_safe, and retired. placement_pressure is maximum while the node is draining or control-only. disk_removal_safe is true when the listener is not ready, active work is zero, and no WebSocket-resident cells remain.

      Existing WebSocket sessions are not moved by drain. They remain attached to their resident cell until the socket closes, and messages on those sockets continue to dispatch on the drained node. New WebSocket upgrades are refused after drain because they require top-level admission.

      The peren CLI drain command performs the registry write. It does not call the peer control endpoint.

      Remove a node from the registry

      peren node remove updates the registry to removed. It does not wipe disk.

      peren node remove peren.toml --node 00000000-0000-0000-0000-000000000001

      Without --force, removal refuses unless a prior drain record exists for that node. Pass --force only when the operator has already accepted that the drain precondition will be skipped.

      Do not delete local data until durable recovery state exists elsewhere and the live control report shows disk_removal_safe after a peer drain. A node with active WebSocket residency is not disk-removal safe.

      Verification

      peren node health peren.toml
curl -sS http://127.0.0.1:7000/control/v1/node
curl -sS http://127.0.0.1:8080/readyz

      Related: Operations overview, Networking, Degraded operation.