Agent Harness #1 — From a Research Question to a Paper: Designing the Pipeline

Agent Harness, part 1. Series index

Systems that take a research question and return a paper already exist. Several of them have been published, benchmarked, and argued about. Building another one was never the point.

The question I actually wanted to answer was narrower. By the time I started, three open-source frameworks had become the default way people wrote multi-agent systems — CrewAI, AutoGen, LangGraph. Everybody picked one. Nobody seemed to have built the same non-trivial pipeline in all three and reported what the choice actually bought you. That was the contribution I was after: one long-horizon research pipeline, implemented three times, compared.

I built the CrewAI one first, because I had to have something working before there was anything to compare.

It took months, and by the time it ran end to end the ground had moved. All three frameworks had shipped breaking changes. A comparison of the versions I had started against would have described a state of the world that no longer existed, and would have kept decaying while I finished the other two ports. So I stopped.

What I had left was a single, working, unusually well-documented attempt: give it a research goal and a short config, and it plans an experiment, writes the code, runs it, analyses the result, and produces a paper — with a human approving at five points along the way. One person, one system, taken from an idea to something that actually ran.

That record seemed worth keeping, so I took it apart stage by stage. This series is that teardown: how it was designed, what those design decisions did and did not cover, and — by the end — what happened when I pointed the system’s own verification standard at my own write-up of it.

This first post is the architecture.

What automating research actually demands

There is a large and growing set of agent systems that complete tasks. Booking a flight, refactoring a module, filling a form. Research is a different shape, in three ways that turned out to drive every subsequent decision.

A task has a right answer; research does not. You can check a booking against the itinerary. You cannot check “is this the right experiment” against anything, which means the system cannot validate its own plan and something else has to.

A task takes minutes; research takes hours. A training run is not a request-response cycle. The harness has to stay alive across it, survive the machine going to sleep, and still know afterwards what happened.

A task that fails stops; research that fails produces output anyway. This is the one I did not see coming. If a flight booking fails you get an error. If an experiment fails, the next stage still has a result file to read, still has numbers in it, and still writes the paper. Failure does not interrupt the pipeline — it propagates through it, wearing the same clothes as success.

That third property is the seed of everything later in this series.

Seven stages

The pipeline I settled on:

question → plan → design → code → execute → analyze → write

In the implementation these collapse into five phase modules plus an exit gate. What matters more than the stage list is that each boundary forced a decision I did not anticipate having to make:

Each of those is a post in this series. The last one is where it comes apart.

The five human gates are not evenly spread, and that is the point: two of them sit inside the execution phase, because that is where a long run can go wrong in ways nobody planned for. The dashed lines are the boundaries. Everything that broke later broke on one of them.

What was built, and what was not

The completed implementation is in CrewAI, and it works end to end. Some shape:

Around 14,600 lines across the CrewAI implementation, a React streaming UI on a FastAPI backend, five human-approval gates, and fourteen architecture decision records written as I went.

The honest other half of that table:

I am putting that second table in the first post deliberately, because it contains the thing I got wrong.

Stopping because the frameworks had moved was a judgement call I still think was right. But when I went back through the code much later, I found two further reasons the comparison could not have worked even if I had finished it — one in the implementation, one in the measurement design. Neither was visible to me at the time. Those are the next post, and the repository has carried the withdrawal since it was archived.

One thing I found while writing this

Writing this post meant counting lines I had not counted before, and one of them contradicts my own record. The repository’s evidence map says the largest phase does not import the framework, citing phase2_coding.py at 1,209 lines. phase2_coding.py does not import crewai — that part holds, and it is the striking part. But it is the second-largest phase. phase3_execution.py is 1,342 lines.

Small, and it does not change the finding. I am mentioning it because it is the first thing this series turned up, on the first pass, in a record I wrote myself and considered finished. It will not be the last.

Where this goes

The next few posts are design: what the framework was actually doing, how state crosses the boundaries between stages, what a model may and may not author, where the human sits, and what it takes to keep a multi-hour run observable. Then: what “success” even means to a harness, the failures that report success anyway, and finally the same standard applied back to my own numbers.

The transferable point of this one is smaller than the architecture diagram suggests: the number of stages is not the design. The contract between them is. Every failure later in this series lives on a boundary, not inside a stage.

The system described here is archived at MARS; the analysis is in agent-harness-anatomy. The archived runs were driven by OpenAI gpt-5.2.