Meet Evarness: The Bench Gets a Name, and Its First Capability Graduates

This series has been building a bench for proving AI agent harnesses — the bottleneck, the deterministic spine, the spine drawn as a graph, and finally the discipline of proving in simulation before going live. The bench has been working under the name Harness Lab. Today that work gets a proper name — Evarness — and a public open-source home at github.com/evarness-ai/evarness. Evarness is an open-source, simulation-first assurance framework: it turns an AI-agent harness’s guarantees into canonical traces, executable invariants, and portable proof bundles you can verify offline. But the name is the small news. The bigger news is how it goes open source: not as a repository dump, but one capability at a time — each one finished, tested, mastered, and introduced by a post like this one before it earns a place in the public repo. The first graduate is the piece everything else stands on: the assurance spine — canonical traces, invariant contracts, proof bundles, and offline verification. And it ships with a receipt: for the shipped approval-gated example, one canonical digest, byte-identical across five execution environments — carried by the v0.1.0 release, so the receipt is a download, not a screenshot.

Why graduation, not migration

The obvious way to open-source a working codebase is to flip the visibility switch. I decided not to, for a reason this series has been circling for months: a public repo is a set of claims. Every file in it implicitly says “this works, this is supported, you can build on this.” A wholesale transfer would have made hundreds of claims in one commit, most of them unexamined.

So Evarness follows a pipeline instead, and every capability walks it in order:

Idea → Experiment → Prototype → Rewrite → Master → Blog → OSS → Community → Production

A few of these stages carry rules that changed how I work:

  • Prototypes are never promoted. Experimental code optimizes for learning — hardcoded assumptions, one target environment, fast iteration. Product code optimizes for adoption — stable contracts, refusal behavior, portability. The Rewrite stage is where one becomes the other, mandatorily.
  • The blog is the introduction point. A capability is not “moved” until the post introducing it is live. If it can’t be explained, demonstrated, and reproduced by a reader, it isn’t ready — the post is the final test.
  • PyPI is a claim of readiness, and it comes last. Being on GitHub is an invitation to evaluate; being on a package index says “make this a dependency.” Evarness doesn’t publish to PyPI until a capability set has been tested and evaluated by people who aren’t me. The package name is reserved in the code; the upload waits for the Community stage to do its work.

The discipline has a cost — the public repo starts small — and that’s the point. Everything in it has cleared the same bar.

The first graduate: the assurance spine

The capability is deliberately narrow. Stated precisely:

Given a graph, a fixture, a seed, and an engine version, Evarness produces a canonical trace, evaluates declared invariants, emits a portable proof bundle, and verifies that bundle offline.

That’s it. No visual builder, no real model providers, no live tools, no adapters — those exist, and they arrive later, each with its own post. The spine goes first because every later capability depends on it to be provable at all: if traces aren’t canonical and contracts aren’t checkable, everything else can only be demonstrated, never proven.

What lands in the repo:

  • The engine — an event-sourced executor over harness graphs: thirty-one node types (routing, interception, redaction, policy gates, budgets, retrieval, memory, approval gates, judges), deterministic topological order, every failure and block a first-class trace event.
  • Canonical traces — the run’s identity is c1:sha256:…, a versioned digest over the canonical event stream. Wall-clock out, payloads in, serialization byte-stable.
  • Invariant contracts — the guarantees a harness claims, declared in YAML as temporal assertions over the event stream: never this after that, this precedes that. Verdicts are computed from the events but stored outside them — checking a run never changes its identity.
  • Proof bundles — evarness prove runs every scenario twice (reproduction demonstrated, not asserted), evaluates the contracts, pins the subject — graph hash, tool-manifest hashes, contract-definitions hash, engine version — and emits a bundle with a mandatory not-proven section. Renderers for HTML, JUnit, and SARIF turn it into something a CI pipeline gates on.
  • Offline verification — evarness verify recomputes digests and per-event chains and checks verdict consistency anywhere, no network — without trusting the producing machine’s precomputed results. prove --sign adds an Ed25519 attestation; on the verifying side, --require-signature refuses unsigned bundles and --pubkey pins who signed — without the pin, a tampered bundle re-signed with a different key would still carry a valid signature.

And what deliberately does not land: anything that touches the world. Every model response and tool result in this release is fixture-scripted. A graph that names a real provider is refused with an error that names the alternative; a tool set to mode: real is refused as a traced governance block. Not because real execution doesn’t exist — the previous post ran it live over the wire — but because it hasn’t cleared the bar yet, and a silent fallback to simulation would make the trace lie about what ran. The refusals are tested like any other behavior.

The receipt

The spine’s whole value rests on one property: the digest is a portable contract, not a description of my laptop. So that’s the property I verified hardest during the graduation. The shipped example — a send tool behind a human approval gate, with three declared invariants — pauses at that gate by design, and the paused run produces this digest:

c1:sha256:8c4b5f50a76a9ca232972a5c4cbadb6ca52b5cbf468d4bdc1d53a23cb1c422a0

Identical, byte for byte, in five places:

  1. the original development repository the code graduated from,
  2. the extracted package, run from a development checkout,
  3. the built wheel, installed into a clean virtual environment,
  4. macOS on Apple silicon, Python 3.14,
  5. Linux x86-64, Python 3.10, in a container.

Same graph, same fixture, same seed — same identity, across operating systems, Python versions, and packaging boundaries. That is what “reproducible” has to mean for a proof bundle to be worth carrying between machines. The repo’s CI re-earns the claim on every push: on both Linux and macOS, across the oldest and newest supported Pythons, a pipeline job builds the wheel, installs it into a clean venv, re-proves the shipped example, and verifies the bundle. And the digest above isn’t only printed here — the v0.1.0 release attaches the proof bundles it names, so you can download one and evarness verify it yourself.

Sixty seconds to your own verified proof

pip install "evarness @ git+https://github.com/evarness-ai/evarness.git@v0.1.0"
# expected: PROOF: PENDING and exit code 1 — the gate is the feature
evarness prove approval_gated_send -o proof.json
evarness prove approval_gated_send --approve n3=approve -o proof.json
evarness verify proof.json

The first command is a deliberate speed bump. The pattern’s entire contract is that nothing sends without a human approval — so with no decision supplied, the scenario pauses at the gate and the verdict says PENDING: zero invariants evaluated, nothing claimed, exit code 1. A proof that checked nothing refuses to call itself one.

Supply the decision, and the second command produces the real thing. The proof demonstrates that the pattern’s three contracts held — approval precedes send, no send after rejection, no model call after a block — that the run reproduces digest-for-digest, and, in the not_proven section, exactly what none of this establishes: scripted scenarios are not universal safety, and a signature proves integrity, not that the runs happened. The bundle says this about itself, because a proof that overstates its scope isn’t a proof — it’s marketing.

If sixty seconds turns into an afternoon, the docs site goes as deep as you want — its live demos are generated by the product at build time, and the build fails if their digests drift from the values the docs cite. ONBOARDING walks the whole surface, and PROVE-VERIFY walks the implementation of exactly the pipeline you just ran.

The quiet part: agents are the first domain, not the point

One thing became unmistakable while extracting the spine: the kernel — the executor, the canonical trace, the contracts, the proof machinery — never needed to know it was about agents. Events in, digests and verdicts out. So the package is layered accordingly: evarness.core is a domain-agnostic kernel, and the agent node set is a domain built on it through typed extension seams — node registries, contract libraries, config overlays, pip entry points.

That’s not an implementation detail; it’s the bigger bet. “A blocked run never reaches the model” and “no deploy without a passing eval” are the same kind of statement — a temporal contract over a governed workflow. A future ML-pipeline domain (dataset → transform → train → eval → deploy_gate) would inherit determinism, tracing, proof bundles, and offline verification for free, and so would any domain where “we always do X before Y” currently lives in a wiki instead of a per-run-verified contract. A domain author writes node types; the assurance machinery comes with the kernel. That claim stays a claim until a second domain runs end to end — the honesty ledger in the repo’s ARCHITECTURE.md says exactly that — but the seams are built, typed, and public.

What’s next

The pipeline has a queue: real model providers, sandboxed real-tool execution with OS-enforced containment, adapters that prove agents built on other frameworks, the governance benchmark — and, further out, that second domain. Each is somewhere between Rewrite and Master right now, and each will graduate the same way this one did — or not until it can. One companion is already public: Evarness Studio, an Alpha visual client for building graphs and browsing proof bundles. It’s optional by design — the CLI does everything headless — and its own introduction post is in the queue.

Run the approval-gated example, read its not_proven section, and compare your digest against the release’s receipt. If it differs — or anything else breaks — an issue with the graph, fixture, seed, Python version, and OS is a deterministic repro — the house currency. That’s the Community stage starting to do its work, and it’s the only road to PyPI.