Skip to main content
株式会社オブライト
株式会社オブライト

The Harness Can Double Your AI Coding Costs

Databricks' Internal Benchmark, Pi's Minimal Design, and pi-autoresearch's 53% Liquid Speedup (2026)

Published 2026/11/2010 min read

Databricks found the same model at the same effort cost over 2x more per task depending on the harness, with equal quality. Why minimal Pi uses less context, plus pi-autoresearch's Liquid 53% PR.

It's tempting to think AI coding costs come down to which model you pick. But in an internal study Databricks published on July 8, 2026, running the same model at the same thinking effort through a different tool — the harness — changed per-task cost by more than 2x in some cases, while quality stayed the same. The difference came from how much context each tool sends the model on every turn.

The harness Databricks singled out as performing best "in many cases" was Pi (pi.dev), a deliberately minimal open-source agent. This article covers what Databricks measured, why Pi uses less context, and how Shopify CEO Tobi Lütke used the Pi extension pi-autoresearch to make the Liquid template engine 53% faster — all from primary sources.

Key points

- The harness changes cost: In Databricks' study, the same model and effort cost more than 2x more per task in some cases via Claude Code/Codex than via Pi. Pi sent about one-third as much context per turn
- Pi is minimal: Only four tools are enabled by default — read, bash, edit, write. No sub-agents or plan mode; add them with extensions if needed
- Automated optimization via extension: pi-autoresearch loops "try, measure, keep or revert." A Shopify Liquid PR built with it cuts parse+render time 53% (not merged)

What is a harness?

A harness is the tool that runs an AI model — Claude Code, Codex, Pi, and so on. The model does the thinking and writing; the harness decides what instructions and tools it gets, and how much file content and command output goes into the conversation.

RoleExamplesEffect on cost
ModelOpus 4.8, Sonnet 5, GPT 5.4 Mini, GLM 5.2, etc.Price per token and how many tokens it uses to solve a task
HarnessClaude Code, Codex, Pi, etc.Size of system prompt and tool descriptions, context sent each turn, number of retries

AI usage is billed roughly by tokens exchanged. If a harness sends lots of information every turn, the same model costs more. Databricks' study showed this gap is larger than many expect.

Databricks' internal benchmark: what and how

Databricks' blog built a benchmark from real tasks engineers performed on its multi-million-line codebase (Python, Go, TypeScript, Scala, Rust, and more) and compared model-and-harness combinations. The method:

1. Select recent, human-written merged PRs (excluding bot and fully AI-generated changes) with good tests and changes confined to a few modules
2. Rewrite each PR's intent as a task prompt, removing the solution
3. Hold out the test files and have the agent write the rest of the change on its own
4. When the agent says it's done, apply the held-out tests to decide pass/fail (no LLM judge)
5. Run each harness with its standard setup. After noticing answers were recoverable from git history, history was sealed during runs

Databricks notes the study "isn't meant to be comprehensive," and draws four conclusions:

ConclusionDetails
1. No single vendor winsThe cost-quality Pareto frontier mixed OpenAI, Anthropic, and open models
2. Open models are readyGLM 5.2 landed in the top tier, statistically tied with Opus 4.8 on quality, at $1.28/task vs $1.94
3. Token price misleadsSonnet 5 is ~1.7x cheaper per token than Opus 4.8 but cost $2.09/task vs $1.94 and completed fewer tasks (81% vs 87%), using 1.9x more tokens
4. Harness matters a lotIn many cases, simple harnesses like Pi performed best

On the fourth point, the blog says running the same model with the same thinking effort through Claude Code/Codex vs Pi produced per-task costs that "differed significantly (more than 2x in some cases), while quality remained the same." The main difference was context per turn: Pi sent about 3x less, keeping a tighter working set and finishing in fewer runs.

Some write-ups cite specific figures such as "Opus 4.8 at high effort: Pi 85% vs Claude Code 87% quality at 2.08x lower cost" or "~742K vs ~237K context tokens per task." Those numbers don't appear in the blog's text and appear to be read from its charts, so this article focuses on what the text states.

Caveats when reading the results

- Not "Pi is always cheaper": Databricks explicitly says the lesson isn't that one harness is always cheaper or that native harnesses are worse — model choice is just one piece of the puzzle
- July 2026 configurations: Harnesses were measured with their standard setups at the time; updates such as smaller system prompts can change the gap
- One company's code: Results come from Databricks' 10+ language codebase and may not transfer directly to others
- Databricks' position: Databricks invests in its own tooling for swapping models and harnesses (Omnigent, Unity Gateway), which aligns with its conclusions

Why Pi uses less context

Pi was created by Mario Zechner and is now developed by Earendil Inc. under the MIT license (see What is Pi (pi.dev)?). Per the official docs, only four tools are enabled by default — read, bash, edit, and write; grep, find, ls, codemode, and others are opt-in. The site lists sub-agents, plan mode, permission popups, built-in to-dos, and background bash as things Pi deliberately doesn't include.

AspectPi's approach
Built-insMinimal (four default tools); short system prompt and tool descriptions
Missing featuresAdd via TypeScript extensions, on-demand skills, and prompt templates; distribute as packages via npm or Git
ModelsMany providers (Anthropic, OpenAI, Google, Azure, Bedrock, and more) plus local models; switch per task
SafetyNo permission popups or sandbox; running inside a container or VM is recommended

The shorter the preamble and the fewer the tools sent each turn, the fewer tokens each exchange consumes. Pi's add-only-what-you-need design plausibly explains the context gap Databricks observed.

pi-autoresearch: an automated try-measure-keep loop

pi-autoresearch gives Pi an autonomous optimization loop. Inspired by Andrej Karpathy's autoresearch, its README sums it up as: try an idea, measure it, keep what works, discard what doesn't, repeat. It works for any measurable target — test speed, bundle size, LLM training, build times, Lighthouse scores. It's MIT-licensed.

# Install and start Pi
pi install npm:pi-autoresearch
pi

# Start the loop inside Pi (example: speed up unit tests)
/autoresearch optimize unit test runtime, monitor correctness
ComponentRole
init_experimentOne-time setup: name, metric, unit, direction
run_experimentRuns any command, times it, captures output
log_experimentRecords the result, auto-commits, updates the dashboard
.auto/measure.shBenchmark script that outputs METRIC name=number lines
.auto/checks.sh (optional)Tests, types, lint; failures block keeping a change
/autoresearch finalizeSplits kept experiments into clean, independently reviewable branches

After three or more runs, a confidence score shows how the best improvement compares to the noise floor: 2.0x or more suggests a real gain, below 1.0x is within noise.

Shopify's case: Liquid parse+render 53% faster

In March 2026, Shopify CEO Tobi Lütke opened PR #2056 against Liquid, Shopify's open-source template engine. Per the PR, about 120 automated experiments with Pi and pi-autoresearch — edit, commit, test, benchmark, keep or discard — produced a 93-commit change. pi-autoresearch was released by Lütke and Shopify's David Cortés.

MetricMainPRChange
Parse+render7,469µs3,534µs−53%
Parse6,031µs2,353µs−61%
Render1,438µs1,146µs−20%
Object allocations62,62024,530−61%

The benchmark used real Shopify theme templates with production-like data on Ruby 3.4 with YJIT, and all 974 unit tests passed. With garbage collection consuming 74% of CPU time, the work focused on eliminating allocations.

- Benchmark numbers: 53% applies to this benchmark, not to every storefront. Per reports, Lütke himself said the result is probably somewhat overfit
- Not merged: As of November 20, 2026 the PR is still open
- Human review still needed: The loop keeps changes that pass tests; design quality and maintainability need human review

Applying this in your organization

Databricks' conclusions apply to any company scaling AI coding. Here's how to cut costs without sacrificing quality:

ActionDetails
Match models to task difficultyConfig tweaks and flag flips don't need top models. Databricks decided to push more work to Haiku and GPT 5.4 Mini class models
Compare cost per taskA cheap-per-token model can cost more if it works longer and uses more tokens
Compare harnesses tooThe same model costs differently by tool; avoid locking into one and keep switching easy
Measure on your own dataMerged PRs plus tests already form a benchmark no model has trained on
Automate measurable goalsFor test or build times, let a loop like pi-autoresearch propose changes and have people review them

Pi is free, but model API usage is billed separately. Because it has no permission popups, run it in a container or VM away from credentials and production systems. For setup and safety, see What is Pi (pi.dev)?; for running multiple agents in the terminal, see Rex (Superlogical) and Corgi.

FAQ

What is a harness?

The tool that runs an AI model — Claude Code, Codex, Pi, and so on. It decides the system prompt and tools the model gets, and how much file content and command output goes into the conversation. The same model can use very different token counts, and therefore costs, depending on the harness.

Will Pi cut my costs in half?

Not necessarily. Databricks saw per-task cost differ by "more than 2x in some cases" with the same model and effort, but explicitly says this doesn't mean one harness is always cheaper. The results reflect its own codebase and each tool's July 2026 configuration, and tool updates can change the gap. Measure on your own work.

Did quality differ in Databricks' test?

Per the blog, when only the harness changed (same model, same thinking effort), cost differed significantly while quality stayed the same. The main factor was how much context each harness sent per turn; Pi sent about 3x less.

What does pi-autoresearch do?

It's a Pi extension that automatically loops: try an idea, measure it, keep improvements, revert regressions. It works for any measurable target — test runtime, bundle size, build times, Lighthouse scores. It's MIT-licensed and installs with pi install npm:pi-autoresearch.

Did Shopify's Liquid really get 53% faster?

Per Tobi Lütke's PR, on a benchmark using real Shopify theme templates, parse+render dropped from 7,469µs to 3,534µs (-53%) and allocations fell 61%, with all 974 unit tests passing. These are benchmark numbers, and as of November 20, 2026 the PR has not been merged.

How can my company control AI coding costs?

Following Databricks' conclusions: route simpler work to cheaper models, compare cost per task rather than price per token, and include the harness in your comparison. Your own merged PRs and tests can form a small in-house benchmark.

Summary

Databricks' study shows AI coding costs depend heavily on the harness, not just the model. With the same model and effort, per-task cost can differ by more than 2x depending on how much context the tool sends each turn. Minimal Pi is a prime example, and its pi-autoresearch extension can keep testing improvements toward any measurable goal.

That doesn't mean switching to Pi will automatically save money. The results reflect Databricks' code and July 2026 settings, and tool updates change them. Measuring cost and quality per task on your own work — and keeping models and harnesses swappable — is the most reliable cost control.

References

株式会社オブライト

Oflight Inc. is a Tokyo-based IT company providing software development, digital signage, web development, network systems, and IoT/AR/VR technology services.

Related free tools (no sign-up, instant results)

Feel free to contact us

Contact Us