Graph engineering academy · module 6 of 7 · 12 min
Production graph engineering
What does it take to run a graph for real?
- Name the eleven properties a production graph needs.
- Explain idempotency and why resume depends on it.
- Plan how a graph is versioned while runs are in flight.
A graph that works in a demo and a graph that runs a team's releases differ in eleven properties. None is exotic; each one is a decision someone has to make before the first incident makes it for them.
-
Typed state
A schema for the notebook every node shares, with rules for how concurrent writes merge.
Check Can a node write a field another node does not expect? A test should say no.
Where this shows up → -
Execution durability
State and position saved at every step, in storage that outlives the process.
Check Kill the worker after each node in a test; the resumed run matches the uninterrupted one.
Where this shows up → -
Idempotency
A side effect keyed by run id, so running it twice does it once.
Check Every tool that writes, sends or deploys checks its key first.
Where this shows up → -
Retry policies
How many times, how far apart, and what happens after the last one.
Check Every cycle has a limit and a failure edge.
Where this shows up → -
Concurrency
Parallel branches, the join, and what happens when one branch is slow or fails.
Check A slow branch has a timeout; a failed branch has a defined effect on the join.
Where this shows up → -
Observability
A trace per run: nodes, edges taken, state diffs, model calls, cost, human waits.
Check From a run id you can answer "why did it take this path?" without reading code.
Where this shows up → -
Graph versioning
Runs in flight keep the graph they started on; new runs get the new one.
Check A checkpoint records the graph version; the runtime refuses to resume on a mismatched version.
Where this shows up → -
Human oversight
Reviewer nodes on the paths that need them, with the context the reviewer needs to decide.
Check The approval rate and the time-to-answer are tracked; a reviewer can see what was tried.
Where this shows up → -
Access controls
Each node runs with the narrowest credentials its job needs; reading nodes cannot write.
Check List each node’s credentials; none is the human’s own.
Where this shows up → -
Cost budgets
Limits on steps, tokens, time and money per run, enforced outside the models.
Check A runaway cycle stops at the budget in a test.
Where this shows up → -
Evaluation suites
Tasks from real failures, graded on outcomes over several trials, run on every change to a prompt, model or edge.
Check A change that lowers the pass rate cannot merge without a decision.
Where this shows up →
The stateful graph pattern's "Process crashes mid-run" scenario shows durability and idempotency working together. Run it.
The Agents & Humans Briefing
How humans and agents work together: the day's field notes and the patterns behind them, in one short email around noon UTC.
Free. No spam. Unsubscribe anytime.