Inside the live factory
Watch uncertainty change the workflow.
A task moves through five phases: Design, Coding, Code Completion, Code Review, and Testing. Each pairs an instructor with an assistant: CEO and CTO, CTO and Programmer, Reviewer and Programmer, or Tester and Programmer.
- 01
GenerateAgents produce a response for the current development phase.
- 02
Measure uncertaintyToken log probabilities feed a rolling score, compared with that phase’s threshold.
- 03
Retrieve when neededA score above the threshold triggers a web lookup. Retrieved context informs subsequent phases.
- 04
Continue downstreamThe next agent receives the accumulated output and available retrieved context.
The demo makes these events inspectable through the token stream, uncertainty gauge, retrieval sources, and event log. Select a task and press Play to follow a run.
The prototype uses heuristic uncertainty scores. Generated code and tests are displayed, not executed; completion metrics are static quality checks, separate from the paper’s evaluation.
Explore live demo ↗01 / Research question
How can multi-agent coding systems identify unreliable decisions before hallucinations move from planning into code and testing?
02 / Method
- 01
Estimate response uncertainty from token-level log probabilities at critical phase transitions.
- 02
Calibrate phase-aware thresholds and trigger retrieval-based verification only when uncertainty is high.
- 03
Evaluate completeness, executability, consistency, overall quality, ablations, and agent communication on SRDD.
03 / Outcome
The study reports consistent gains over single- and multi-agent baselines, with the strongest improvements appearing in execution reliability and overall quality.