AdversaryGate

Releases 2.4 to 2.14 ยท October 2026

What changed, and what comes next

Twelve releases in six days, all aimed at one question: can the patch being judged influence its own verdict? Each answer below closed a way it could.

What changed

  1. 2.14.1Oct 9, 2026

    Two fail-opens, found by reviewing 2.14.0

    • A mutant run that failed because another run touched the same thing outside the repository — a fixed /tmp path, a port, a test database — counted as a killed mutant. With 2.14.0's mutants in parallel and the full suite alongside, a test that checks no value went from 0/8 kills to 8/8 and MERGE (AG-043). Mutants now wait for the suite and run one at a time; --mutation-workers N is the operator's choice, for suites that share nothing outside the repository.
    • The baseline was built with git archive, which applies the project's own export attributes: with tests/ export-ignore the baseline had no tests, and the baseline oracle could not judge a rewritten one (AG-044). The baseline is now the commit's tree, every file as committed.
  2. 2.14.0Oct 8, 2026

    Same verdicts, less waiting

    • The baseline and patch sides of each check, the full suite and the mutants now run at the same time. On five real more-itertools cases the gate's own time fell from 319 s to 127 s, 177 to 101, 285 to 109 and 285 to 108; the decision, fix_proven and every mutant's killed/survived came out identical.
    • Running side by side can push a run past the wall clock, and a mutant that hangs counts as killed. So a mutant killed by the clock while anything else was running is rerun alone, and the rerun decides: it can only lower the score. --serial-sides turns all of it off.
    • Correction (2.14.1): the identical results held for tests that share nothing outside the repository. A mutant run that another run made fail was counted as killed; see 2.14.1.
  3. 2.13.0Oct 7, 2026

    Tell BLOCK, INCONCLUSIVE and a broken run apart in a workflow

    • The Action has an exit-code output: 0 merge, 1 block, 2 inconclusive, 3/4 usage or harness error, empty if the Action stopped before the gate ran.
    • continue-on-error alone let a BLOCK through, and decision could not separate INCONCLUSIVE from a run that printed no verdict. The README now shows how to let INCONCLUSIVE pass only where the change cannot be measured. Pointed out by a reader of the write-up.
  4. 2.12.0Oct 7, 2026

    The gate tells an agent what would earn a MERGE

    • Every result short of MERGE carries next_steps: cover these lines, kill this named mutant, add a test, fix the code, ask a human about an edited baseline test. It never suggests editing a baseline test, the harness, a floor or the policy.
    • Mutants run only the claimed tests, not their whole files: a failing test outside the claims can no longer inflate the score, and a large file no longer leaves every mutant stillborn.
    • A mutant that makes the tests hang counts as killed, but only when the unmutated run finishes well inside the limit.
    • Blank and comment-only added lines no longer count as untested code.
    • Validated on 40 real more-itertools runs: no reintroduced bug reached MERGE, and 7 of 8 real fixes were proven. An experimental exact count for small changes ships switched off.
  5. 2.11.0Oct 5, 2026

    Run as an agent would run it, on real bugs

    • The MCP server was called on eight real more-itertools fixes, both ways. Reintroduced bugs blocked: 4 of 8 before, 7 of 8 now (the eighth no test in that project detects). No wrong MERGE in 16 runs.
    • A test the patch adds now runs against the old code: failing there and passing on the patch proves the fix. 6 of 8 real fixes proven, up from 0. A test that only imports something new proves nothing.
    • The whole-suite and whole-file checks get their own time and memory limits; under the per-test ones they died on any suite of real size.
    • Tests defined on a mixin resolve to the classes that run them; a git worktree's .git file is no longer foreign code; the reason always explains the decision.
    • Evidence collection reads only the changed lines from coverage data instead of writing a full report: 411 s to 278 s on one real call. Calls still take minutes on medium suites; that is the next thing.
  6. 2.10.0Oct 5, 2026

    The MCP tools say when to call them

    • Tool descriptions cover when to call, what runs, side effects and the timeout; titles and behaviour hints mark gate_policy read-only and verify_repo open-world, since the repository's tests run.
    • The [mcp] extra now needs MCP SDK 1.14 or later. Measured: 1.9.4, 1.12 and 1.13 failed to register verify_repo.
  7. 2.9.0Oct 5, 2026

    The MCP server runs with one command

    • uvx adversary-gate-mcp starts the same server as pip install "adversary-gate[mcp]", without installing anything first. It is a separate PyPI package that ships no code and pins the gate at its own version.
    • The core package still depends only on pytest: the MCP SDK stays out of CI installs.
  8. 2.8.0Oct 4, 2026

    Coding agents can call the gate, but cannot configure it

    • MCP server (adversary-gate-mcp): verify_repo judges a git working tree, uncommitted changes included. A skill for Hermes Agent ships with the package.
    • The agent names what to judge. The baseline, the interpreter and the floors are set by whoever runs it, because whoever picks the baseline picks the oracle.
    • Risk triage with Jev (opt-in): a confident "high risk" raises the floors for that run. No answer can lower them.
    • Listed on Glama, with a Docker image tested in CI.
  9. 2.7.0Oct 4, 2026

    A mutation score is a sample, so it gets a confidence interval

    • The strength floor reads the lower bound of an 80% Wilson interval. One killed mutant out of one is no longer "strong"; five out of five is the minimum (AG-023).
    • Mutants are spread one per changed line before any line gets two.
    • A passing claim says how it passed: fixed when the test failed before the patch, no_regression when it passed both times (AG-030).
  10. 2.6.0Oct 4, 2026

    The oracle covers helpers and the whole suite

    • --test-support declares helpers whose path doesn't say "test", so they are restored from the baseline too.
    • The collateral full-suite run also executes the baseline's tests against the new code. A bent test that belongs to no claim is a BLOCK.
  11. 2.5.0Oct 4, 2026

    A rewritten test is judged by the baseline's copy

    • When a patch changes a test that already existed, the gate runs the original against the new code. A bug hidden behind a rewritten assertion becomes BLOCK, and an honest test refactor becomes VERIFIED (AG-021).
  12. 2.4.0Oct 4, 2026

    The patch can't configure the pytest that judges it

    • A patch could ship a .pytest.ini and a plugin that lied only for the exact bytes of its bug, and reach MERGE. Every test-harness file is now compared tree against tree with the baseline (AG-032).
    • One package, adversary_gate, instead of four top-level modules (AG-027).

Full detail, with how each finding was reproduced: CHANGELOG and findings ledger.

What comes next

In order of how much each one limits the gate today.

  • Biggest gap

    Measure languages other than Python

    Today a C++, Rust or SQL change is INCONCLUSIVE. Reaching MERGE needs a coverage adapter and, harder, a mutation adapter per language. Rust comes first, through cargo-mutants.

  • Planned

    End-to-end runs with a real agent

    The MCP server is tested over the real protocol. The next step is a full loop with Hermes Agent: the agent writes a patch, calls the gate, and acts on the answer.

  • Planned

    Validate Jev triage against the live API

    The client follows TypeSafe's published API and is tested against a local server that speaks it. It has not run against the live service yet.

  • Always open

    Find a wrong answer

    A wrong MERGE is the failure this tool exists to prevent. If you find one, it becomes the next finding in the public ledger, with a reproduction and a fix.