Crate grafos_jobs 

grafos-jobs — Idempotent burst compute and retry scaffolding.

This crate provides a higher-level job runner on top of grafos-batch. It standardizes chunking, idempotent output capture, retries on lease failure, and teardown semantics — the “pop-up supercomputer” pattern.

Core types

Type	Purpose
WorkChunk	Trait for serializable work units with stable IDs
JobOutputStore	Trait for storing/retrieving outputs by chunk ID
RetryPolicy	Configures which errors to retry, max retries, backoff
JobCoordinator	Orchestrates chunk execution with retries and aggregation

Quick start

use grafos_jobs::*;

// Define a work chunk
#[derive(Clone, serde::Serialize, serde::Deserialize)]
struct Square { value: u64 }

impl WorkChunk for Square {
    fn chunk_id(&self) -> ChunkId { ChunkId(self.value) }
    fn to_bytes(&self) -> Vec<u8> {
        postcard::to_allocvec(self).unwrap()
    }
    fn from_bytes(bytes: &[u8]) -> grafos_std::Result<Self> {
        postcard::from_bytes(bytes).map_err(|_| grafos_std::FabricError::IoError(-200))
    }
}

// Create a job
let chunks: Vec<Box<dyn WorkChunk>> = vec![
    Box::new(Square { value: 2 }),
    Box::new(Square { value: 3 }),
];

let mut store = MemoryOutputStore::new();
let policy = RetryPolicy::default();

let mut coord = JobCoordinator::new(policy);
let result = coord.run(
    &chunks,
    &mut store,
    |chunk_bytes| {
        let sq: Square = postcard::from_bytes(chunk_bytes)
            .map_err(|_| grafos_std::FabricError::IoError(-200))?;
        let result = sq.value * sq.value;
        Ok(postcard::to_allocvec(&result).unwrap())
    },
    |outputs| {
        let sum: u64 = outputs.iter().map(|(_, v)| {
            postcard::from_bytes::<u64>(v).unwrap_or(0)
        }).sum();
        postcard::to_allocvec(&sum).unwrap()
    },
).unwrap();

let total: u64 = postcard::from_bytes(&result.aggregate).unwrap();
assert_eq!(total, 4 + 9);

Feature flags

Feature	Default	Effect
std	Yes	Enables std in dependent crates
observe	No	Enables grafos-observe hooks

Structs

ChunkId

Stable identifier for a work chunk.

JobCoordinator

Orchestrates chunk execution with retries and idempotent output capture.

JobResult

Aggregate result of a job run.

MemoryOutputStore

In-memory output store backed by a BTreeMap.

RetryPolicy

Retry policy for job chunk execution.

Enums

RetryableError

Classifies whether an error is retryable.

Traits

JobOutputStore

Trait for storing and retrieving chunk outputs by ID.

WorkChunk

A serializable unit of work with a stable identity.