Releases 2.4 to 2.14 ยท October 2026
What changed, and what comes next
Twelve releases in six days, all aimed at one question: can the patch being judged influence its own verdict? Each answer below closed a way it could.
What changed
-
2.14.1Oct 9, 2026
Two fail-opens, found by reviewing 2.14.0
- A mutant run that failed because another run touched the same thing outside the repository — a fixed
/tmppath, a port, a test database — counted as a killed mutant. With 2.14.0's mutants in parallel and the full suite alongside, a test that checks no value went from 0/8 kills to 8/8 and MERGE (AG-043). Mutants now wait for the suite and run one at a time;--mutation-workers Nis the operator's choice, for suites that share nothing outside the repository. - The baseline was built with
git archive, which applies the project's own export attributes: withtests/ export-ignorethe baseline had no tests, and the baseline oracle could not judge a rewritten one (AG-044). The baseline is now the commit's tree, every file as committed.
- A mutant run that failed because another run touched the same thing outside the repository — a fixed
-
2.14.0Oct 8, 2026
Same verdicts, less waiting
- The baseline and patch sides of each check, the full suite and the mutants now run at the same time. On five real more-itertools cases the gate's own time fell from 319 s to 127 s, 177 to 101, 285 to 109 and 285 to 108; the decision,
fix_provenand every mutant's killed/survived came out identical. - Running side by side can push a run past the wall clock, and a mutant that hangs counts as killed. So a mutant killed by the clock while anything else was running is rerun alone, and the rerun decides: it can only lower the score.
--serial-sidesturns all of it off. - Correction (2.14.1): the identical results held for tests that share nothing outside the repository. A mutant run that another run made fail was counted as killed; see 2.14.1.
- The baseline and patch sides of each check, the full suite and the mutants now run at the same time. On five real more-itertools cases the gate's own time fell from 319 s to 127 s, 177 to 101, 285 to 109 and 285 to 108; the decision,
-
2.13.0Oct 7, 2026
Tell BLOCK, INCONCLUSIVE and a broken run apart in a workflow
- The Action has an
exit-codeoutput: 0 merge, 1 block, 2 inconclusive, 3/4 usage or harness error, empty if the Action stopped before the gate ran. continue-on-erroralone let a BLOCK through, anddecisioncould not separate INCONCLUSIVE from a run that printed no verdict. The README now shows how to let INCONCLUSIVE pass only where the change cannot be measured. Pointed out by a reader of the write-up.
- The Action has an
-
2.12.0Oct 7, 2026
The gate tells an agent what would earn a MERGE
- Every result short of MERGE carries
next_steps: cover these lines, kill this named mutant, add a test, fix the code, ask a human about an edited baseline test. It never suggests editing a baseline test, the harness, a floor or the policy. - Mutants run only the claimed tests, not their whole files: a failing test outside the claims can no longer inflate the score, and a large file no longer leaves every mutant stillborn.
- A mutant that makes the tests hang counts as killed, but only when the unmutated run finishes well inside the limit.
- Blank and comment-only added lines no longer count as untested code.
- Validated on 40 real more-itertools runs: no reintroduced bug reached MERGE, and 7 of 8 real fixes were proven. An experimental exact count for small changes ships switched off.
- Every result short of MERGE carries
-
2.11.0Oct 5, 2026
Run as an agent would run it, on real bugs
- The MCP server was called on eight real more-itertools fixes, both ways. Reintroduced bugs blocked: 4 of 8 before, 7 of 8 now (the eighth no test in that project detects). No wrong MERGE in 16 runs.
- A test the patch adds now runs against the old code: failing there and passing on the patch proves the fix. 6 of 8 real fixes proven, up from 0. A test that only imports something new proves nothing.
- The whole-suite and whole-file checks get their own time and memory limits; under the per-test ones they died on any suite of real size.
- Tests defined on a mixin resolve to the classes that run them; a git worktree's
.gitfile is no longer foreign code; the reason always explains the decision. - Evidence collection reads only the changed lines from coverage data instead of writing a full report: 411 s to 278 s on one real call. Calls still take minutes on medium suites; that is the next thing.
-
2.10.0Oct 5, 2026
The MCP tools say when to call them
- Tool descriptions cover when to call, what runs, side effects and the timeout; titles and behaviour hints mark
gate_policyread-only andverify_repoopen-world, since the repository's tests run. - The
[mcp]extra now needs MCP SDK 1.14 or later. Measured: 1.9.4, 1.12 and 1.13 failed to registerverify_repo.
- Tool descriptions cover when to call, what runs, side effects and the timeout; titles and behaviour hints mark
-
2.9.0Oct 5, 2026
The MCP server runs with one command
uvx adversary-gate-mcpstarts the same server aspip install "adversary-gate[mcp]", without installing anything first. It is a separate PyPI package that ships no code and pins the gate at its own version.- The core package still depends only on pytest: the MCP SDK stays out of CI installs.
-
2.8.0Oct 4, 2026
Coding agents can call the gate, but cannot configure it
- MCP server (
adversary-gate-mcp):verify_repojudges a git working tree, uncommitted changes included. A skill for Hermes Agent ships with the package. - The agent names what to judge. The baseline, the interpreter and the floors are set by whoever runs it, because whoever picks the baseline picks the oracle.
- Risk triage with Jev (opt-in): a confident "high risk" raises the floors for that run. No answer can lower them.
- Listed on Glama, with a Docker image tested in CI.
- MCP server (
-
2.7.0Oct 4, 2026
A mutation score is a sample, so it gets a confidence interval
- The strength floor reads the lower bound of an 80% Wilson interval. One killed mutant out of one is no longer "strong"; five out of five is the minimum (AG-023).
- Mutants are spread one per changed line before any line gets two.
- A passing claim says how it passed:
fixedwhen the test failed before the patch,no_regressionwhen it passed both times (AG-030).
-
2.6.0Oct 4, 2026
The oracle covers helpers and the whole suite
--test-supportdeclares helpers whose path doesn't say "test", so they are restored from the baseline too.- The collateral full-suite run also executes the baseline's tests against the new code. A bent test that belongs to no claim is a BLOCK.
-
2.5.0Oct 4, 2026
A rewritten test is judged by the baseline's copy
- When a patch changes a test that already existed, the gate runs the original against the new code. A bug hidden behind a rewritten assertion becomes BLOCK, and an honest test refactor becomes VERIFIED (AG-021).
-
2.4.0Oct 4, 2026
The patch can't configure the pytest that judges it
- A patch could ship a
.pytest.iniand a plugin that lied only for the exact bytes of its bug, and reach MERGE. Every test-harness file is now compared tree against tree with the baseline (AG-032). - One package,
adversary_gate, instead of four top-level modules (AG-027).
- A patch could ship a
Full detail, with how each finding was reproduced: CHANGELOG and findings ledger.
What comes next
In order of how much each one limits the gate today.
-
Biggest gap
Measure languages other than Python
Today a C++, Rust or SQL change is INCONCLUSIVE. Reaching MERGE needs a coverage adapter and, harder, a mutation adapter per language. Rust comes first, through
cargo-mutants. -
Planned
End-to-end runs with a real agent
The MCP server is tested over the real protocol. The next step is a full loop with Hermes Agent: the agent writes a patch, calls the gate, and acts on the answer.
-
Planned
Validate Jev triage against the live API
The client follows TypeSafe's published API and is tested against a local server that speaks it. It has not run against the live service yet.
-
Always open
Find a wrong answer
A wrong MERGE is the failure this tool exists to prevent. If you find one, it becomes the next finding in the public ledger, with a reproduction and a fix.