Crate grafos_pipeline

Crate grafos_pipeline 

Source
Expand description

Relocatable stage edges with generation-tracked handoff and durable checkpoints for grafOS pipelines.

This crate provides the core primitives for building pipelines whose inter-stage edges can relocate transparently on failure. The key pattern:

  1. A producer pushes items into a grafos_collections::queue::FabricQueue via a RelocatableQueueEdge.
  2. When the queue’s lease expires or the connection drops, the producer allocates a new queue, publishes the new grafos_locator::locator::QueueLocator via a grafos_locator::handoff::HandoffWriter, and resumes pushing.
  3. A consumer polls a grafos_locator::handoff::HandoffReader to detect the generation bump, rebinds to the new queue, and resumes popping.

Stage state can be durably checkpointed to block storage via CheckpointedStageState, enabling at-least-once delivery with idempotent sinks.

§Feature flags

FeatureDefaultEffect
stdYesEnables std across all grafos dependencies
observeNoEnables grafos-observe integration

Structs§

CheckpointedStageState
A stage state wrapper with durable checkpoint/restore support.
EdgeCheckpoint
Checkpoint recording how far a stage has progressed through its input and output queues.
RelocatableQueueEdge
A queue edge that can relocate to a new backing queue on failure.
StageId
Unique identifier for a pipeline stage.

Enums§

EdgeError
Errors that can occur during edge operations.

Type Aliases§

EdgeGeneration
Alias for grafos_fence::FenceEpoch used as the edge generation counter.