The Harness Is Part of the Agent
What studying more than 40 agent harnesses taught me about the systems around the model.
I began comparing more than 40 public agent harnesses because I expected the search to reveal one dominant architecture. Instead, it revealed recurring control surfaces—context, tools, state, permissions, recovery—and different tradeoffs around each one. Coding agents, personal agents, research agents, and workflow systems kept arranging the same parts around different promises.
The governing question became: how much of an agent’s behavior belongs to the model, and how much belongs to the system around it?
Note: The comparison used public documentation, repositories, product behavior, and architecture notes. It is a developing design map, not a controlled benchmark, an exhaustive taxonomy, or 40 production systems I personally operated.
What I mean by a harness
The harness is the system around the model.
It decides what the model can see, which tools it can use, what survives between steps, when it should stop, and what happens after failure.
context
↓
permission → model → tools
↓
state, evidence, recovery
Change one of these pieces and the effective agent changes.
The same model can feel careful in one product and reckless in another. It can recover cleanly from a failed command or repeat the same side effect. It can remember why the work exists or lose the goal after a long session.
We often call these differences “prompting.” Many of them are systems design.
How I compared the systems
The corpus included coding agents, browser and computer-use agents, workflow runtimes, research agents, multi-agent orchestrators, and persistent personal agents.
I looked for a few practical things:
- Where does the control loop live?
- How is context assembled?
- Who grants permission?
- What survives a session?
- How does the system recover?
- What supports a claim that the work is done?
Public systems change quickly, and some internal behavior is invisible. So this is a map of recurring design choices, not a leaderboard.
Eight kinds of agent
The systems roughly fell into eight groups:
- Conversational copilots organize around a turn or thread.
- Coding agents organize around a repository task.
- Computer-use agents organize around actions on a screen.
- Workflow runtimes organize around a defined process.
- Research agents organize around a question and source set.
- Multi-agent systems organize around delegated subtasks.
- Persistent personal agents organize around an ongoing relationship.
- Ambient agents wake in response to time or events.
No group is universally better. Each makes a different promise about time, authority, and evidence.
A coding agent may be excellent at closing a local task and poor at knowing whether customers received the result. A persistent agent can reduce repetition, but a bad memory can influence unrelated work for weeks. An ambient agent can be useful without being asked, but it can also become an interruption machine.
The recurring parts
Across those groups, 13 patterns kept returning. They fit into five layers.
1. The loop
The run loop decides when the model observes, acts, inspects a result, retries, or stops.
A weak loop ends too early or retries forever. A useful one has budgets, clear stopping conditions, and a way to ask for help.
2. What the agent knows
This layer includes context composition, working state, long-term memory, and model routing.
More context is not always better. The harness has to choose what matters now, distinguish trusted instructions from untrusted content, and preserve enough state to resume without pretending the world stood still.
Persistent memory adds another question: should this information still influence the agent tomorrow?
3. What the agent can do
Tools, permissions, skills, and background triggers live here.
A tool schema tells the model how to call something. It does not fully describe the side effect, whether the action can be reversed, or what counts as real success.
Permissions turn “the model can call this tool” into “the agent may take this action, on this resource, for this purpose.” OpenAI’s account of running Codex safely shows why sandboxing, network controls, and review belong to the agent system rather than the prompt alone.
Reusable skills help too, but they are also code entering the system. Their source and permission scope matter.
4. What happens when things break
Long tasks fail in ordinary ways: a command times out, an API returns half a result, a session restarts, or two workers change the same state.
Recovery therefore needs checkpoints, safe replay, cancellation, and a way to separate completed work from work that only looked complete.
This is where multi-agent systems become difficult. Parallelism is useful only when the subtasks are clear and the results can be reconciled.
5. How anyone knows what happened
Logs help operators. People need a simpler view:
- what changed;
- why the agent acted;
- what evidence it used;
- what still needs judgment.
Raw traces are inspectable, but they are not automatically understandable. A good harness turns activity into a small, honest account of the current state.
What survived the comparison
Five lessons felt durable.
A better model does not fix a weak system
The model proposes actions. The harness decides what it sees, what it may do, and whether failure remains local. Model capability and system reliability are related, but they are not the same variable.
Persistence changes the product
A session agent can forget a mistake when the chat ends. A persistent agent can carry it into future planning and action.
Once state crosses time, correction and forgetting become part of the product.
Trust lives in the handoffs
The important moments are often between things: person and agent, model and tool, memory and action, local work and production, artifact and accepted outcome.
Each handoff needs a clear boundary.
Recovery is a feature
Demos optimize the happy path. Real systems spend much of their time in partial states.
An agent that can pause, show what it knows, and resume safely often feels more capable than one that simply produces a more impressive first answer.
The person should not become the orchestrator
Builders may enjoy agents, tools, branches, queues, and traces. Most people do not want a second job operating that machinery.
A personal agent should translate the system into responsibilities, outcomes, and the few moments that genuinely need a person.
What this means for Waldo
The comparison gave me a clearer set of choices.
I want to keep durable state, explicit permissions, recovery, source-aware memory, and completion backed by evidence.
I want to adapt the useful ideas from coding agents—plans, diffs, approvals, workspaces—to responsibilities that also involve messages, calendars, other people, and the physical world.
I want to avoid transcript-first navigation, hidden personality scores, unlimited capture, and dashboards that reward machine activity rather than useful outcomes.
These are directions, not proof that Waldo has solved the problem. The product still has to show that it creates less work for the person.
The next artifact
This essay is the short version.
The full comparison should become a living Harness Atlas: one page per system, a shared rubric, dated sources, original diagrams, and clear notes on what Waldo should adopt, adapt, or reject.
The editorial inspiration is Sebastian Raschka’s Big LLM Architecture Comparison: repeat the same comparison structure, show the architecture clearly, and let readers see where systems genuinely differ.
The atlas should not try to crown a winner. It should make a few questions easier to answer:
- What survives after a session?
- Where does authority live?
- How does the system recover?
- What can the user correct?
- What evidence supports “done”?
A model provides capability. What kind of harness lets that capability meet the world without hiding its tradeoffs?