2. Why optimize the harness?
Getting an AI agent to work is only the beginning. As we add tools, instructions and more agents, the system around the model can become surprisingly expensive.
That surrounding system is the harness: the prompts, tools, memory and orchestration that turn a model into a working application. Repeated context, oversized tool responses and unnecessary handoffs all add up. A usage dashboard can show the bill. But what actually changes it?
That is the idea behind HarnessOpt, a Copilot CLI plugin I have been building with Amar Kumar and Chetan Sagare. It connects measurement to action: understand a harness, find worthwhile changes, and test whether those changes really help.
3. The autoresearch inspiration
A key inspiration is Andrej Karpathy's autoresearch: give an agent a real experimental setup, an objective and a budget, then let it modify code, evaluate the result, keep or discard the change, and repeat.
Hypothesis → edit → evaluate → keep or revert → repeat
Karpathy's implementation experiments with model-training code: each training run has a fixed five-minute budget, and validation bits per byte supplies the score. HarnessOpt applies that experiment-driven pattern to a different target: the code and context around an existing model. The objective is lower token usage and cost while preserving answer quality, rather than training a better model.
That is the distinction between an optimization report and an autonomous research loop. The loop acts on a hypothesis, reads the outcome, and uses the accumulated successes and failures to choose what to try next. Meta-Harness is another closely related reference: it searches harness code using prior candidates, scores and execution traces. The research references below connect this idea to prompt evolution, workflow search and cost-aware agent design.
4. Four components, one feedback loop
-
01 / Establish the baseline
Build benchmark
Capture real answers and the inputs and outputs of individual steps. Keep a fixed yardstick so every experiment has something concrete to compare against.
-
02 / Understand the system
Profile
An expert agent reads the harness and maps its components and workflows. Static token counts and runtime traces reveal where context, tools, models and handoffs consume tokens.
-
03 / Pick worthwhile changes
Review
Audit the measurements, then rank opportunities by impact, effort and risk. Explain what to change and why, rather than just flagging an expensive component.
-
04 / Experiment and check
Improve
Implement a change, test it against the benchmark and retain validated improvements. The autonomous research loop repeats this process, recording successes and rejected experiments.
These are reasoning agents, not just scripts that count tokens. They read the target harness and trace how a request becomes an answer. Deterministic tooling handles tokenization, evidence storage and dashboard rendering; the agents supply the discovery, diagnosis and experiment design.
5. Under the hood
5.1 Build benchmark: capture a replayable baseline
/harnessopt:build-benchmark
Run representative queries through the harness's normal interface and capture its actual final answers. Optional, opt-in instrumentation also records step-level fixtures: the real inputs and outputs of model calls, tools, retrievals and agent handoffs. Those fixtures let us test a changed component without rerunning the entire workflow.
The benchmark also supplies the captured execution trace and available usage records. Final-answer-only benchmarks work without deep instrumentation, but cannot provide the same isolated tests. Unavailable inputs or usage remain explicitly unavailable, rather than being reconstructed.
5.2 Profile: attribute usage to the right component
/harnessopt:profile
The profiler reads entry points, prompt assembly, tool registrations and agent definitions to map the actual execution flow. It separates static footprint from runtime usage: exact tokenization measures instructions and tool schemas, while the benchmark's trace supplies observed model calls, tokens, retries and available costs.
Attribution matters. If an agent receives a short digest of a configuration file, counting the entire file inflates its footprint. Likewise, a workflow total includes its child agents; adding both totals would double-count the same work. Runtime figures link to evidence rows and runnable SQL in a local evidence.db; static counts link to the measured text and tokenizer.
5.3 Review: audit first, recommend second
/harnessopt:review
The reviewer first checks the profile against the code and captured execution, correcting over-attribution before ranking changes. It then reads the actual step inputs and outputs to find waste that totals alone cannot explain: repeated instructions, large results copied between agents, duplicate retrievals, or a powerful model handling a narrow task.
Each recommendation identifies the component, supporting evidence, proposed change, expected effect and generalization risk. An unused tool in one run is not automatically disposable: it may serve a valid branch the benchmark never exercised. The goal is to preserve capabilities, not optimize only for the recorded examples.
5.4 Improve: run an experiment, not a guess
/harnessopt:improve auto-optimize
The improver forms a hypothesis, makes a reversible change and chooses the smallest trustworthy measurement. Re-tokenization measures a context trim; repeated component replays test an isolated model or tool change; repeated full-harness runs measure effects that cross stage boundaries. Variable runs need an average and spread, not a savings claim from one lucky execution.
A separate Checker assesses answer equivalence using the benchmark's configured method: harness validators, an LLM judge, numeric tolerance or exact match. Smaller text alone does not establish correctness. The keep-or-rollback decision considers both quality and cost, and the experiment history records the outcome. Autonomous mode repeats this loop within a budget, with cumulative checks to catch interactions between changes.
Equivalence is relative to the captured baseline, not proof that the baseline itself is correct. Repository tests and broader evaluation cases remain important; passing a small benchmark does not establish quality on unseen tasks.
6. Measured wins
Fewer tokens across four measured before-and-after comparisons of a data-analysis harness.
Fewer tokens across three complete review-stage comparisons for feature-building tasks.
The useful changes went beyond shorter prompts: pass compact result references instead of copying full datasets into every handoff, expose role-specific tool sets, match model capability to the task, and remove redundant stages. In the data-analysis example, the recorded research loop explored 25 rounds of improvements.
Case-study names are omitted from this text; the recording is presented unchanged. These are observed comparisons, not guaranteed savings. The feature-building figures cover three complete review-stage pairs (46.2%, 23.2% and 72.4%), not whole builds. A fourth pair is excluded here because its reported comparison omits a required fallback and uses different model versions. Token reductions do not necessarily imply equivalent cost reductions. The 25-round history is separate from the end-to-end comparisons; their savings should not be added together. Cost figures in the recording mix measured charges and repriced usage, rather than uniformly representing billed savings.
My main takeaway: optimize the whole harness, not just the prompt. Measure a real baseline, change one thing, and keep the improvement only when the evidence supports it.
Next, we want to broaden the benchmarks and try more harnesses, so the loop is tested against more workflows and edge cases.
7. Research and references
Karpathy's autoresearch and the papers cited in our deck provide useful ideas for the loop. They are references, not claims that HarnessOpt implements every algorithm below or inherits their published results.
- Andrej Karpathy — autoresearch. An autonomous code-editing and evaluation loop for model-training experiments: retain improvements, discard regressions, repeat within a budget.
- GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. Uses reflection on execution trajectories to propose prompt changes and combine lessons from successful candidates.
- AFlow: Automating Agentic Workflow Generation. Searches code-represented workflows with Monte Carlo Tree Search and execution feedback.
- MaAS: Multi-agent Architecture Search via Agentic Supernet. Samples query-dependent agent architectures to allocate inference resources according to the task.
- EvoAgentX: An Automated Framework for Evolving Agentic Workflows. Brings workflow generation, execution, evaluation and optimization of prompts, tools and topology into one framework.
- AgentEvo: Cost-aware Agentic Workflow Generation via Adaptive Multi-stage Evolution. Explores cost-aware workflow generation through an evolutionary process.
- HarnessOpt-Bench: Evaluating LLMs at Harness Optimization. A separate research benchmark that evaluates harness optimization under a fixed evaluation budget, with held-out tests and auditable candidate versions.
- Meta-Harness: End-to-End Optimization of Model Harnesses. An outer-loop search over harness code with access to prior source, scores and execution traces. Reference implementation.