Local AI agents in 2026: the model isn’t the bottleneck, the harness is

An eagle-eye view from three experiments on local open-weight models — run on an Apple Silicon laptop with MLX. The headline: the model on your machine is already good enough for most real work. What decides whether it works is the engineering around it.

TL;DR

I verified a viral claim — “a 36 GB Apple-Silicon laptop quietly replaced my cloud AI subscriptions” — and it mostly holds: ~100+ tok/s decode, ~19–20 GB resident, a 4-bit ~35B mixture-of-experts model passing a coding battery, on a chip a generation older than the one in the claim.
But the interesting part wasn’t the speed. Across three experiments, every failure of a local model happened at a few locatable, measurable points — a KV-cache buffer cliff, a single maximum-entropy token, a mis-sized thinking budget. Fix the harness and the failures go away.
The most dangerous failure is silent: below a compression threshold, a model keeps talking fluently while quietly not doing what you asked (it stopped calling its tools and started inventing answers). You cannot see this in the text.
Conclusion: for local agentic AI, the leverage is not a bigger model. It’s the harness — buffer sizing, budget management, and grading behaviour instead of trusting prose — wrapped around a modest one.
Everything below is reproducible from open code: uv run agentic-evals exp-001 …. All numbers were measured on Apple Silicon / MLX — a boundary I’ll be honest about at the end, and the next experiment goes after it on a consumer Windows laptop.

Why I ran these

I build on a simple conviction: you should be the master of your own data — which points, if you follow it far enough, at free, local, secure AI that runs on your machine and answers to you. And a belief that drives the engineering: maybe 90% of the daily work I hand to AI is deterministic — summarise this, extract that, route this, call that tool. You don’t need a frontier model for most of it. You need a small or mid-sized model wrapped in a good harness: the layer of tools, memory, budgets, and self-correction that lets a modest model reason, act, observe, and correct itself, safely.

I’m building that harness — a local agent called IRIS (coming soon). But the open-source AI world is vast, scattered, and mostly unverified: claims everywhere, proof rarely. So I made a rule: verify, validate, iterate, fix — then use. These three experiments are that rule applied. This post is the eagle-eye view of what they found and what I think it means for everyone building local agents.

Who this is for

This is for the engineers who build local, secure AI that you own — but don’t have the time to wade through every hype thread and YouTube “miracle” to find what actually works. I went through the slog so you can make informed calls without it.

And I hold myself to one standard the ecosystem often skips: no “trust me.” Every claim here comes with code you can run yourself — proof, or it didn’t happen. Find a case where my results don’t hold on your hardware? That’s the most valuable reply I can get — tell me, and we’ll both learn something.

Three experiments, one arc

I went looking for three different things and kept finding the same shape of answer.

Experiment 1 — KV-cache compression and the hallucination cliff

To run long context on fixed laptop RAM, you compress the KV cache (the model’s running memory of the conversation). The literature frames this as a smooth quality/memory trade-off. It isn’t. It’s a cliff.

I swept the one knob that turned out to matter — buffer_size, the number of recent tokens kept uncompressed — across six open-weight instruct models. On Qwen-2.5-7B, with a 1966-token prompt, the edge was a 1%-wide window:

Set buffer_size = 973. Ask “What’s the weather in San Francisco?” with one tool defined. The model answers: “The weather in San Francisco right now is 41.3°F with a humidity of 58%.” Plausible. Completely fabricated. It never called the tool.

Set buffer_size = 993 — twenty tokens more — and the same model, same prompt, emits the perfect tool call.

The cliff, measured	Result
Cliff width (Qwen-2.5-7B)	~1% of buffer ratio — `buffer_size` 973 → 993 flips it
Below the cliff (`973`, ratio 0.495)	fabricates “41.3°F · 58% humidity” — no tool call
Above the cliff (`993`, ratio 0.505)	correct tool call
Cliff position	universal at ratio ≈ 0.50–0.55 across all 6 models that tool-call
Bit-width effect	negligible — the buffer dominates
Compression beats full precision	4 of 6 models, on dilute long context

Three findings fell out, and they generalised across models:

The cliff is sharp and its position is universal (buffer_ratio ≈ 0.50–0.55 across every model that tool-called cleanly). The bit-width barely mattered; the buffer was everything.
Below the cliff, the failure is silent. The language model stays fluent; the agent contract breaks. You can’t tell from the prose that no tool fired.
Sometimes compression beats full precision. On long prompts padded with irrelevant filler, compressing the junk acted as a noise filter — the compressed model passed where the uncompressed baseline failed, on four of six models.

That second point is the one that should keep agent builders up at night. Karpathy put the underlying truth memorably:

“Hallucination is all LLMs do. They are dream machines.” — Andrej Karpathy (Dec 2023)

A model below the cliff is still dreaming fluently. It just stopped acting. The only defence is to check whether the tool actually fired — never to read the answer and nod.

This is the first post in the series, written up in full → The Hallucination Cliff. That post takes the cliff apart token by token: how twenty extra tokens flip a confidently fabricated weather report into a perfect tool call, why the edge lands at nearly the same buffer ratio across six different open-weight models, and the genuinely strange case where compressing the cache made a model more accurate than full precision. If this section made you uneasy, that’s where the unease gets earned — start there, then come back.

Experiment 2 — a coding model that breaks at exactly one token

Next I tested a viral Medium claim: that a single quantized MoE on a 36 GB Mac is fast and good enough to replace cloud coding tools. On performance, it largely checked out — ~103 tok/s decode, under the claimed memory budget, 7 of 8 coding tasks passing when graded by actually running the code.

Metric	Viral claim (M5 Max)	Measured (M4 Max, 36 GB — a generation older)
Decode throughput	90–130 tok/s	103.5 tok/s
Peak memory	19–22 GB	17.4 → 18.7 GB (at 8k context)
Coding battery, executed	”competitive enough”	7/8 pass (easy 3/3, hard 4/5)

The eighth was the interesting one. Asked to fix a buggy binary search, the model failed — and failed deterministically at the same single token every time: the ( in mid = (. I read the model’s own probabilities at that step. It was the maximum-entropy token of the entire generation — 6.447 bits of uncertainty against a median of essentially zero. The model was near-deterministic everywhere else and forked wildly at exactly one decision, scattering into cross-language garbage (one run started writing Java). The same line of code, written from scratch instead of echoed-and-fixed, had near-zero entropy and was rock-solid.

The lesson: a quantized model’s quality isn’t uniform. It concentrates fragility at a handful of high-entropy branch points — and those points are findable and measurable.

Experiment 3 — the real model, and the budget that was the whole story

The Medium article named a specific model: a newer, thinking-by-default mixture-of-experts model that reasons inside <think>…</think> before answering. So I tested it faithfully.

Performance held again — 107 tok/s, 19.7–20.0 GB (a better fit to the claimed “19–22 GB” than my substitute had been). But the coding score swung purely on a harness setting. With a flat token budget, it scored 6/8 — and every failure was truncation, not a wrong answer. Give thinking its own budget, separate from the answer, and detect when reasoning ends, and it climbed to a clean pass. The bug-fix task that broke the earlier model? This one solved it correctly — it just needed to think for ~12,700 tokens first.

Metric	Measured (M4 Max, 36 GB)
Decode throughput	106.8–107.6 tok/s
Peak memory	19.7 → 20.0 GB (best fit to the claim’s 19–22 GB)
Coding, flat token budget	6/8 — every failure was truncation, not a wrong answer
Coding, split think/answer budget	clean pass; the exp-002 bug-fix now solves (~12.7k think tokens)

The only genuine quirk left was almost funny: the trivial “stack with O(1) minimum” task sent the model into reasoning that never terminated, even given an enormous budget, while the hard bug-fix finished cleanly. Difficulty-as-a-human-sees-it does not predict how long a model needs to think. The harness — not the model — has to manage that.

The eagle-eye view: what this means for local agentic AI

Step back from the three results and one picture forms.

1. The model is no longer the bottleneck — the harness is

Every large swing in outcome came from an engineering choice, not a model choice: the buffer size, the entropy-aware caution at a fork, the split thinking budget. The base capability was there each time. A 3B–7B instruct model and a ~35B thinking model, on a laptop, can already do the bulk of real daily work — if the scaffolding around them sizes their memory, manages their budgets, and checks their behaviour.

This is good news for the open-source community, and it reframes where effort should go. The race for ever-bigger weights is not where most practical value for local agents lives right now. The value lives in the unglamorous middle layer: caches, budgets, graders, routers. That layer is buildable by individuals and small teams. You do not need a GPU cluster to make a meaningful contribution to local agentic AI — you need to build and verify a better harness.

2. Fluent and wrong is the defining risk

The single most transferable finding is that the worst failures are silent and fluent. A model below the compression cliff invents weather data. A model that hides a correct answer inside reasoning it never finished “passes” only by luck. In both cases the output reads fine. For an agent that takes actions on your behalf, this is the whole ballgame: trust must be earned by behaviour, not by prose. Did the tool fire? Did the code run? Build that check into the loop, not just the test suite.

If the OSS community standardised on one practice, I’d want it to be this: ship behavioural graders alongside models. The field has leaderboards for capability. It has almost nothing for “how does this model fail, and is the failure loud or silent?” That second table is the one agent builders actually need.

3. Failures are locatable — so instrument them

The cliff is at a measurable buffer ratio. The decode break is at a measurable entropy spike. The thinking limit is a measurable token count and a </think> marker. None of these are mysterious. A harness that watches the few places models break can turn silent failures into loud, recoverable ones. This is mundane engineering, and it works.

4. Local is genuinely viable, and that matters beyond convenience

The throughput and memory numbers held on hardware a generation behind the claim. A modern Apple-Silicon laptop runs a 35B-class model at 100+ tok/s in ~20 GB. Combined with the “90% of work is deterministic” reality, this means a large fraction of everyday AI can run privately, on your own device, for free — no data leaving your machine, no subscription, no rate limit. That’s not just a cost story; it’s a data-ownership story, and it’s the reason local agentic AI is worth the engineering.

The honest boundary (and the next backend)

Every number here was measured on one stack: an Apple M4 Max, 36 GB, with MLX. So let me be precise about what that does and doesn’t buy you:

The behavioural findings — the cliff, silent failure, entropy break points, thinking-budget dynamics — are properties of attention, the model, and the quantization scheme, not of Apple’s Metal. They should reproduce on other backends; only the exact numbers (the precise cliff ratio, the tok/s) would shift. Treat them as portable hypotheses worth re-confirming on your hardware.
The runtime findings — throughput, peak memory, a specific compression library’s quirks — are Apple-Silicon-specific and I make no claim about them elsewhere.

And rather than leave portability as an assertion, I’m going to test it. The next backend is deliberately not a datacenter GPU — it’s a consumer Windows laptop with no NVIDIA card (an Intel Core Ultra, 32 GB, integrated graphics, via llama.cpp / Ollama / LM Studio). That’s the niche I care about: the laptop in your bag, not a rented H100. The question it answers is exactly the one this post raises — do the cliff, the silent failure, and the budget dynamics travel off Apple silicon? It’s the next experiment on my roadmap, and it’s the same discipline that runs through this whole series: this is my own thinking, worked out in the open and measured — not a verdict handed down. Run it on your own hardware, push past where I stopped, and if you extend it or find me wrong, tell me. That’s the most useful reply I can get — it’s how we both get closer to the truth.

What’s next

Two tracks, both public:

This blog series. The deep-dives behind each finding — the cliff, the cross-model universality, the compression-beats-baseline surprise, the single-token break, the thinking-budget story. Each one is reproducible from the open repo.
Skills. I’m packaging the operational know-how into reusable modules — a buffer-sizer, a thinking-budget planner, a behavioural grader, an entropy probe — that plug straight into a local agent’s hooks. They’re the executable form of these learnings, and the first two are going into IRIS.

The code is open. Re-run it. Tell me where I’m wrong.

Reproduce everything: the experiments, raw data, and per-finding spike trail live in the agentic-evals repo. Start with exp-001 — the cliff.

Local AI Agents in 2026: The Model Isn't the Bottleneck, the Harness Is