Research — study AI agents at work, not in a benchmark.

Real work, fully recorded

Each agent run with its model, time, cost and outcome; each review with its verdict and findings; each human decision with its reason and time. Nothing to scrape or clean: the record is the data.

The effect of the rules on the work

Change one rule — who reviews, how far an agent may act alone, when a person must approve — and measure what happens to quality (work sent back by reviewers, rounds per change) and to speed (time from task to accepted change, idle time). Every rule change is dated, so each one can be studied as a before and after.

Models side by side, under the same conditions

Several AI vendors on the same tasks, each checked by a reviewer from a different company, with every disagreement recorded.

Reproducible by design

Decisions are signed, the record is only ever added to, and every measurement comes with the command that produced it.

Volume with one approver

In two days, across two repositories: 819 commits and 140 changes accepted by the approver, each one reviewed before it was accepted.

The second reviewer matters

In a security review of the agents’ own code, two reviewers from different AI companies disagreed on two of the first three parts: one passed the code, the other rejected it and proved the problem with failing tests. Both problems were fixed the same day.

The bottleneck was approval, not agents

Work ran at full speed while the approver was at the keyboard and nearly stopped overnight, with agents and machines idle, waiting for a human signature. That finding led to a mode in which the approver signs a batch of work before leaving and the agents continue on their own, with every irreversible step still waiting for a person. We are measuring its effect now.

Six days with agents, 14 to 20 months for people

An effort analysis of the whole build: 139 hours of agent work against an estimate of 14 to 20 months for a team of about ten, with code quality measured against published baselines. Read the report →

The tool: Saphan Studio Team, free for research and teaching

For non-commercial research and teaching at universities and research institutes. The same orchestration and governance engine our own engineering runs on, with an experiment mode: you define variants of the rules, and every run is tagged with the variant it ran under. The team helps you with your first study.

The data: months of real agent work, as a dataset

Runs, reviews, human decisions, refusals and every rule change with its date, from agents of several vendors working side by side (Claude, Codex, Cursor, Qwen, GLM, Nemotron and others). Exported and anonymised: no machine names, keys or personal data.

Research programmes, part one: making agents work better

Study designs we have drafted from our own record. Each one names its question, the data that answers it, and what would have to change to measure it properly:

• Cross-vendor review: does a reviewer from a different vendor than the author find more problems, or is it simply a stricter reviewer?
• Fresh start or continued session: should each repair round begin with a new agent, or continue the earlier conversation?
• Frontier or cheaper models: where does a cheaper model do the same work under the same rules and the same review?
• Rules as experiments: change one rule, such as who reviews or how far an orchestrator may act alone, and measure quality and speed.
• Agents at a closed door: when an action is refused, do agents stop, ask, look for another way, or use the governed path?
• Instructions or checks: which lessons hold as written guidance, and which hold only when a check enforces them?

Research programmes, part two: people and agents together

The part almost nobody studies yet. Not whether a model solves a task, but how one person and a fleet of agents get better at working together, measured in the same record:

• Knowledge reinforcement: does the owner give better orders and decisions the longer the fleet works with them, when the fleet prepares the context behind each decision?
• The owner's capacity: how many parallel streams can one person govern with good judgment, where is the ceiling, and what raises it?
• How rules are learned: how many signed decisions does it take before agents absorb a new rule, and why do some rules land after one signature while others need reminders?
• Sparks: insights caught from the work at the moment they appear, then turned into rules, research or public material, or dropped. Which sparks actually improve the fleet?
• Catching what does not converge: when the fleet keeps failing in the same way, does showing it to the person change their decision, and how fast?
• Interventions as data: every time the owner steps in is recorded. What do those moments say about where a system still needs a person?

Our patents, under NDA

Saphan Studio is patent pending in the U.S. and Europe. For a joint study we can share the filings under NDA, so your work builds on the mechanisms instead of guessing at them.

One thing we ask

Share what you learn. Publish it, and tell us, so the next team that runs agents works better.