Regression testing
Source reports covering Regression testing. Stories will appear when analysis and community reactions are ready.
1 reports · page 1 of 1
Connected topics are temporarily unavailable. You can still browse stories below.
Source reports
AI agent fixed the bug, but what else did it change?
tool
9 min read
Trail of Bits has released patching-bugs and regression-hunt, two agent plugins developed through its Patch the Planet initiative with OpenAI. Patching-bugs reproduces reported defects, develops and tests fixes, and applies a three-part proof: the original code must fail, the patched code must pass, and a full revert must fail again. Regression-hunt traces changed code to real interfaces, reproduces suspected regressions on live deployments, replays evidence with a fresh-context agent, and requires adjudication before reporting a regression. Trail of Bits reviewed more than 130 upstream-merged Patch the Planet pull requests and confirmed 21 regressions: 16 functional, one performance-related, and four build or test regressions. None disabled a security protection. The 21 regressions came from 58 candidates in one run, or roughly 36% precision. In preliminary testing against those known cases, regression-hunt produced 25% fewer false positives than a baseline Codex agent asked to review a pull request for regressions, with a 95% confidence interval of 14% to 38%. Its stricter evidence requirements left some known regressions classified as borderline. The plugins use defined schemas for claims, revisions, checks, evidence, and open questions. They can incorporate a repository’s THREAT_MODEL.md and PATCHING_GUIDELINES.md or AGENTS.md files to define security assumptions, supported behavior, patch requirements, build and test procedures, and acceptable evidence.