Cookbook

This directory contains one markdown file per cookbook recipe. These are expanded versions of the ideas in docs/cookbook-ideas.md, written as standalone documents with design notes, failure modes, and testing guidance. Every published cookbook recipe should have corresponding compiled code under cookbook/ in a numbered workspace crate.

Recipe Standard

A cookbook recipe should help a reader solve a concrete job with the real public API surface. It should feel like a practical field guide: the reader has a problem, the recipe gives a direct solution, the code runs, and the discussion explains the boundaries and tradeoffs. It is not a roadmap note, an internal design sketch, a feature tour, or a workaround for missing runtime behavior.

A strong recipe is job-oriented rather than feature-oriented:

• Concrete outcome: name the useful thing the reader will build, such as a webhook receiver, replicated order pipeline, fenced leader, or checkpointed workflow. Avoid abstract goals like “demonstrate queues” or “show the lease API.”
• Problem first: open with the production problem in the reader’s terms: “reject stale writes after failover”, “lease a GPU for one request”, or “resume a pipeline after provider drain.” Do not open with internal package history or implementation sequencing.
• When to use it: state the production scenario, assumptions, and resource or failure-domain policy the recipe relies on.
• Real APIs only: use public grafos-* handles, CLI commands, and types that compile in the workspace. Do not use mocks, prose-only APIs, hidden sync loops, synthetic host setup, or deterministic state-core internals unless the recipe is explicitly for implementers.
• Correct abstraction boundary: application recipes must not call raw fabricBIOS wire-token minting/parsing functions or hold fabricBIOS signing keys. Use the grafOS lease/session/resource handles and, when a recipe is specifically modeling an authority service, use grafos-std capability APIs such as CapabilityAuthority.
• Program code: include the minimal complete code path readers can adapt: setup, core logic, policy construction, error handling, and tests.
• Core grafOS API path: when the runnable crate uses a recipe-specific helper or facade, show the public grafos-* calls inside that helper before or alongside the higher-level program flow. Readers should be able to see which resource handle is created, which placement/quorum policy is attached, which operation is invoked, and which typed error or refusal marker protects the failure boundary.
• Failure behavior: explain what happens on retries, duplicate inputs, stale epochs, quorum loss, unavailable domains, relocation, or degraded placement. Use the real typed errors or refusal markers.
• Run and verify: include the command, test, or harness that proves the behavior, plus the expected result. Commands should be copy-pasteable.
• Adaptation notes: call out the practical knobs a reader is likely to change, such as placement policy, quorum, TTL, idempotency key, consumer group, checkpoint interval, resource size, or tenant/provider selection.

Recommended page shape:

1. Problem: what the reader needs to accomplish and why the naive approach fails or is risky.
2. Solution: the short version of the grafOS pattern and the public APIs it uses.
3. Code: the minimal complete path readers can adapt. Keep setup, policy, execution, error handling, and tests together.
4. How it works: explain the resource handles, leases, placement or quorum policy, authority boundary, and data flow. This section should make the public API path visible, not hide it behind a recipe-specific facade.
5. Failure behavior: describe the real outcome for duplicate requests, retries, revocation, FENCED state, stale epochs, quorum loss, provider loss, or degraded placement. Name the typed errors or refusal markers.
6. Run it: give the exact command or harness and the expected successful result.
7. Adapt it: list the small set of knobs that change the recipe for a real deployment.

Anti-patterns:

• Do not write a recipe whose main value is “this API exists.” Build a useful program instead.
• Do not use mock-only examples, fake leases, synthetic scheduler state, hidden synchronization loops, or prose-only APIs to make the recipe look complete.
• Do not skip the failure path. A recipe that only shows the happy path is incomplete unless the failure behavior is genuinely irrelevant.
• Do not bury the real grafos-* calls behind an opaque helper without showing the helper implementation or the equivalent direct API path.
• Do not present an unimplemented primitive as runnable cookbook code. Build the primitive first, or keep the idea in a design document.

User-facing recipes must not mention phase numbers, TODO sequencing, or other internal project-management terms. If a primitive is not implemented, the runnable cookbook must not pretend it exists; build the primitive first or keep the idea in a design document outside the runnable recipe set.

Browse by Group

The recipes below are organized by what you are trying to build. Pick the group that matches your problem and read its overview page; within each group the recipes are ordered progressively from the smallest useful program to advanced variants.

• Start Here — first tasklet, local-dev loop, test harness, observability.
• State & Storage — caches, durable structures, session stores, KV, content-addressed object storage.
• Services, RPC & Web — microservices, ingress, typed RPC, async / durable APIs.
• Pipelines & Workflows — ETL, streams, message bus, MapReduce, event pipelines, durable workflow engines.
• Coordination & Consistency — leadership, fencing, stale-write rejection, lease-based conflict resolution.
• GPU & Inference — all GPU and LLM workloads; sub-grouped into basics, shared inference, correctness & memory, revocation & recovery, audit & attribution.
• Placement, Scaling & Operations — placement policy, multi-cloud deploy, drain / migration, scaling under load, borrowed hardware.
• Security, Audit & Cost — lease-scoped encryption, time-bounded secrets, attestation, fair-share quotas, audit chain, cost attribution.

Cross-cutting tags

Recipes also carry tags from a restricted vocabulary so you can browse across groups by concern:

gpu · inference · multi-tenant · revocation · fencing · audit-chain · cost-aware · multi-cloud · placement · durable-state · encryption · testing · observability · web · rpc · streaming · workflow · requires-cuda

A revocation tag, for example, gathers Recipe 28 (stale-write rejection in leader election) from Coordination, Recipe 58 (workload on node drain) from Operations, and Recipes 63 / 64 / 67 (intra-kernel revoke, FENCED detection, hot-rebind) from GPU & Inference into one logical bundle that the primary group structure would otherwise spread across the cookbook. The full per-tag listings live under ./tags/.

Related Operator Guides

For execution tracing and debugging (not recipe-style, but frequently needed alongside cookbook workloads):

• docs/runbooks/workload-tracing-runbook.md — debugging failed tasklets, placement decisions, fenced leases, module cache misses
• docs/grafos/execution-correlation-guide.md — end-to-end trace_id correlation from admission to execution
• docs/grafos/mixed-fleet-execution-model.md — Linux vs Pi5 runtime differences

API Documentation References

We do not currently have a single, generated “API reference” as markdown in docs/, but we do have:

• Markdown guides (architecture + usage): docs/grafos-std-guide.md, docs/grafos-collections-guide.md, docs/grafos-sync-guide.md, docs/grafos-rpc-guide.md, docs/grafos-net-guide.md, docs/grafos-tensor-guide.md, docs/grafos-observe-guide.md.
• Crate READMEs (API surface summaries + examples): crates/grafos-std/README.md, crates/grafos-collections/README.md, crates/grafos-sync/README.md, crates/grafos-rpc/README.md, crates/grafos-net/README.md, crates/grafos-tensor/README.md, crates/grafos-observe/README.md.

If you want full rustdoc output locally, run:

cargo doc --workspace --no-deps

Then open target/doc/index.html (or use cargo doc --open).