quick-lru · round 1
We planted … bugs. The green suite caught 0 of them.
Each bug is modelled on a common production failure mode: stale data served as fresh, the same key counted twice, an expiry boundary off by one millisecond. Every one of … original tests still passed on the survivors below.
IBM Bob · Guardian mode
Bob writes one test per survivor. Caught: 0 / …
Guardian is a custom Bob mode that can only write to tests-added/. Each test must pass on the original code and fail on the bug. Tests use a frozen clock, because one of the original tests fails under CPU load with no bug at all.
quick-lru · unseen bugs
Then we planted bugs Bob never saw. The improved suite caught 0 / ….
The held-out bugs were hidden from Bob with .bobignore before any test was written. The commit that hides them comes first in the git history.
Who checks the checker?
Bob's test fixed the line it was shown. Move the same bug one function over, and it misses.
Bob's test was written for this bug. Caught.
Same mistake, different function. Bob's test alone: …
Fix the class, not the line
Bob rewrote the tests for the whole failure pattern. Unseen bugs caught: 0 / …
On the way there, review found a test that could never fail: size() caps its answer at maxSize, which hid the double count. Bob first swapped in an easier test; review asked again, and Bob found a real state where the bug shows. Every step is in the verification log.
Red team
Saboteur read the new tests and aimed at their gaps. Caught: 0 / …
Saboteur is the second custom Bob mode. It can read code and tests but can only write bug files and its pattern catalog. This round, the combined suite (original plus Bob's tests) held.
Second library · dot-prop
A second, well-tested library. First round: 0 / … caught.
So Bob built a scout. It ranks each public function by how many lines of the test file mention it, and the next attack goes where tests are thin: this round hit escapePath and two helpers behind deepKeys. unflatten, the thinnest, was not attacked.
Scout-guided round: 0 / … caught. One real blind spot.
Why it matters, on one more input. This is a demo script (docs/examples/dp-r2-3-collision.mjs), not one of Bob's tests.
const list = ['a', 'b']; list['1abc'] = 'collides';
deepKeys({list})
original: list[0], list[1], list.1abc
with bug: list[0], list[1], list[1] ← two different keys, one pathBob's test for the same bug uses an oversized index; the '1abc' input above is not covered yet. Caught: 0 / …
How IBM Bob built it
Bob is the engine. Every planted bug, every added test and the runner itself, with one runner fix after review.
- Plan modethe first plan, reviewed before any code
- Agent moderunner, scout, every test run
- Saboteurcustom mode that can only write bug files and its pattern catalog
- Guardiancustom mode that can only write tests
- realistic-mutantcustom skill for planting a bug honestly
- Subagentstwo in parallel drafted the upstream pull requests
We checked Bob, and ourselves
0 times a result looked done and was not.
Invalid JSON behind a clean summary. A timing test that made our own tool report false catches. A test that could not fail. Twice, the human was the one who was wrong. And a cold review by a second AI found our runner scoring a syntax error as a catch. Every entry has a commit.