TLDR: Vasiliy Radostev, technical Cofounder of Metergraph, walks us through his multi-model software factory’s architecture. After he implemented multiple models to check one another’s work at different stages in his factory, the mistakes he found during human review went down by 80%.

The world is moving towards building software factories that could turn product plans into code while our laptops sleep. But, software factories often go wrong, leading to buggy products and wasted tokens.

‍Metergraph, a PSL startup, helps companies spend less on AI while keeping output quality high. It looks at a team's real AI traffic and finds the cheaper model that still clears their quality bar. Vasiliy Radostev, the technical cofounder of Metergraph, has now spent several months testing the strengths and quirks of different models. Overtime, he has turned into a big advocate for getting more than one model into a room and setting them upon each other to improve output quality. You've probably heard of multi-agent debate, where models argue their way to a better answer, or LLM-as-a-judge, where one model grades another's work.

A couple of weeks back, Vasiliy was testing his Claude's PR when he hit a very simple bug. The code was crashing, so he went back to Claude Code (who authored the current code) and asked it to fix the issue. Claude found the problem, moved one line of code up to prevent the crash, and announced it was done. On a hunch, Vasiliy took Claude's new output to Codex and asked what it thought. Codex saw it straight away, "if you do a fresh run, you would have a different failure now." Claude's fix hadn't fixed anything. It had only moved the crash elsewhere. When Vasiliy took that back to Claude, it agreed at once: "oh yeah, absolutely, I missed it."

A lot of companies we know bring in a code reviewer like Github Copilot to check their final PRs. Vasiliy however thinks the final step is a little too late. "Coding is not a hard task for the models anymore," Vasiliy says. The hard part is the ladder above it: high level solution, design details, and robust implementation plans. Debate pays off most on the way down to the code, not just in the code. This approach has reduced mistakes by about 80% compared to Vasiliy's older single-agent factory model. If you have yet to move to a software factory process with multiple models pitching in, this article is for you.

Build a basic Multi-Model software factory

Hire a Builder and a Reviewer

Start by picking two models: one to build and one to review. Vasiliy picked Codex and Claude for his factory, with Codex playing the part of the orchestrator and reviewer. “Claude, I find, is better with complex planning than Codex,” he said, “but hand Claude a review and it can’t leave anything alone. It is not very good at deciding what’s important, what’s not important, and always has to give, you know, some kind of feedback.” So a naming nitpick lands in the same list, with the same weight, as a crash. Codex and Grok, by contrast, are “more reserved, unlikely to go on an endless chase.” Models and their strengths will change with every release. So it makes sense to test new ones as they come out and reassign the roles from time to time.

Shape the Factory Floor

The factory runs on an always-on cloud VM (he uses GCP). Work enters it when Vasiliy moves a Linear ticket assigned to Codex (the orchestrator) into Todo. A small controller (a pre-written program) plays the role of a PM and manages the factory. For each issue, the controller sets up a new branch and git worktree and sends Codex a fixed snapshot of the issue and its evidence. As the models work, the controller moves the ticket through the factory’s five stages: Plan, Design, Implement, Review, Human approval. The last stage is a PR waiting for Vasiliy in the morning. The agents are not allowed to merge the PRs themselves.

Turn the Assembly Line into a Skill

The work inside each stage is defined by a skill called issue-resolve. Together with a fixed runner prompt, it gives Codex its identity and responsibilities. It tells Codex to “Fix the issue stage by stage” and to “Review as a senior engineer responsible for shipping: balance correctness, long-term maintainability, and time to deliver.”

At the start of the flow, Codex asks Claude for a short plan and design. The skill specifies how Codex should phrase that request: “a concise plan.md proportional to the issue,” covering “the customer problem, current evidence, system cause, proposed solution, observable acceptance criteria, material risks, and focused validation.”

If the issue names production data, the models check it first. If they can’t reach the data, or the problem no longer reproduces, they stop and explain the gap instead of sending the machines off on a wild goose chase. From there, the loop runs. Codex critiques the plan and design and asks for changes. Once corrections are made, the plan is cleared for Claude to implement. Claude builds, runs for a hot minute, and pushes a draft PR when it’s ready. Codex reviews the diff, checks CI, and requests fixes. Once CI passes on the current commit, Codex marks the PR ready for Vasiliy.

While explaining the model to us, Vasiliy highlighted the importance of getting the reviewer prompt right. "Unless you narrow down exactly what you want from the reviewer,” he warned, “you will not converge.” A loose prompt can spiral past 25 rounds of debate, into over-engineered code that ends in one big (useless) pull request. At the same time, it is not always easy to specificy what exactly the models should do upfront. So, for more ambiguous issues, Vasiliy stops the factory after the plan and design are in, and cuts scope after reading them.

A bloated PR, apart from being useless, also burns a lot of tokens. Vasiliy, in his enthusiasm to experiment, braves this risk. "Worst case, I'll run out of tokens for a week," he shrugged. We understand this might not be the answer for all of us (it's not for us, anyway). So for the more reserved, he suggested capping review rounds in the orchestrator's prompts, setting limits per run (Claude Code’s --max-turns flag helps here) or specifying a spend limit in our provider consoles.

Bridge the Models

To connect the models, Vasiliy built his own MCP bridge. Codex starts the bridge, which starts or resumes a Claude conversation in a registered workspace and hands back a conversation ID and turn ID. Codex uses these to check the run’s status, wait for a result, or cancel the turn. Vasiliy doesn’t think the specific bridge matters much, though. “The structure is more important than, you know, the specific bridge you use.” If you’d rather not build your own, off-the-shelf options include PAL MCP or the servers built into the tools themselves (Claude mcp server and Codex mcp-server).

Four Pairs of Eyes

At the end of the day, Vasiliy still reviews the factory's output at each stages. He looks at the high level design, and some parts of the final code if its a high risk PR. Issues that impact more customers in more fundamental ways get more attention from him. "I would never give up my human review," he admitted with an air of caution. For all that, he's found that his reviewer models have significantly taken the load off of him. From Vasiliy’s experiment, we learned that the biggest challenges are: breaking up the pipeline, having multiple models review each stage, creating traces for humans to review and course correct, and ensuring context passes on from one stage to another smoothly.

To close with an analogy from the banking industry, banks run a procedure called four-eyes principle; a rule that nothing goes through until a second person has checked the maker's work. In Vasiliy’s setup, the models now supply the first pair of eyes and he supplies the second. The dream future, only half-jokingly, is one where the models supply both pairs of eyes and push the work out the doors on their own. We're still a long way from that; but until then, let's teach these agents to debate, learn, and keep each other accountable.

About Vasiliy Radostev

Vasiliy Radostev is a Seattle-based engineering leader who has spent more than a decade building conversational and agentic systems. He was a Principal Engineering Manager at Microsoft, Amazon and HelloFresh. He also co-authored the AG2 team’s launch of native Agent-to-Agent (A2A) protocol support in AG2, the open-source multi-agent framework.

About Sahana Narendran

Sahana is a founding member of Lev, an MS HCDE student at the University of Washington, and a writer for the PSL blog. She builds AI agents, tools and workflows that multiply what people can get done, and she is curious about exploring new ways for humans and AI to work together.

‍