The study limitations say refined artifacts can preserve incorrect assumptions. The intervention also includes investigation, repair, review and packaging; it does not isolate the effect of rearranging identical outputs. Its five-reference protocol is a choice, not a guarantee of sufficient uncertainty estimation. Those limits matter when deciding what the inspection record must expose.
The first-evaluation case analysis records 813 raw-code failures becoming refined-artifact successes, alongside 136 reversals. These are paired task–model outcomes across the four agents; the 136 are not unique tasks, and the analysis does not assign every reversal to packaging. These counts belong to the first evaluation, separate from the three-evaluation means above.
The AuxiliaryMCMC example shows how the inspection record changes a decision. Its prescribed Gaussian sweep calls for 100 rounds at each of three acceptance rates. The Opus 5 refined-artifact attempt completes only 55 rounds per rate after the workload exceeds the available clock at its chosen concurrency. The primary audit invalidates numerical matches affected by that changed measurement procedure. The raw-code attempt also falls short of 100 rounds in part of its sweep, so this case does not establish that refinement alone caused the deviation.
Applied to that case, our unexecuted inspection record would retain the 100-round target, the 55-round execution, the affected output and audit finding, and the decision that those numerical matches do not meet the specified procedure. It would leave the case’s API, GPU, human-review and maintenance amounts unknown where unreported. A close score would not repair a changed measurement, and a completed setup would not establish that the intended experiment finished.
The useful application is to ask for an inspectable procedure and a complete budget, then assess them against the intended research task. The paper’s selected-task results support examining that approach under its tested conditions. They do not establish universal model rankings, total-cost savings, production reliability or successful reproduction of all ML research.
Illustration: four artifact-output icons adapted from Figure 2 in ReproBridge: Building Reproducibility into ML Research Artifacts, by Yubo Wang, Rui Meng, Jaehyun Nam, Tomas Pfister and Jinsung Yoon, CC BY 4.0. Icons rearranged in a new folio illustration; the original figure also depicts construction and independent review roles.