It's the harness, not the model: +26% on CryptanalysisBench's hard set

Last week ETH Zurich's SPY Lab released CryptanalysisBench, a benchmark built around one brutal question: can an AI agent break published cryptography on its own? Every task hands the agent a live scheme and asks for a working attack: forge a signature, recover a key, or tell two ciphertexts apart that are supposed to be indistinguishable. A verifier runs the output, so nothing counts until the exploit fires. The baseline the paper measured was the frontier itself: the strongest models from the top labs, each on a shared research scaffold.

We wanted to know how much of that score is the model and how much is the scaffold driving it. So we ran the cleanest experiment there is. We fixed the model to the exact one the paper reports, Claude Opus 4.8, kept the paper's protocol (best-of-2, a two-hour budget, the benchmark's own verifier, medium reasoning effort), and changed nothing but the harness.

On the benchmark's harder Tier-2 set, our harness moved the paper's own model from 31/140 (22.1%) to 39/140 (27.9%), a 26% relative gain, with the model, the budget, the reasoning effort, and the verifier all held to the paper's settings. Only 133 of those 140 tasks are runnable in the public corpus we used, and we count the missing 7 as losses. The scaffold did the rest.

Turning analysis into a working exploit

The paper is blunt about how agents fail here: most of the time the agent identifies the vulnerability and then never converts it into working code. It writes an attack that looks right, declares victory, and the verifier finds it doesn't fire.

Closing that gap is what our harness is built to do. We don't let the agent grade its own homework. It keeps working until the attack fires against a real target. That bar is the difference between a scheme the agent understood and one it broke, and it's what carried Opus 4.8 from 22.1% to 27.9% on Tier-2, at the paper's own reasoning effort and two-hour budget.

Same model, same budget, same verifier. What changed is the scaffold driving it.

The controlled A/B

The win generalizes, and the controlled run is the proof. Hold the model at Opus 4.8, run the paper's exact protocol, and the scaffold comes out ahead on the hard set and on the classic Tier-1 set.

Tier-1 is the saturated set, the older primitives with a known practical break, and it's the tell. Each of our two independent attempts scored 36/49, matching the paper's Opus 4.8 number exactly. Best-of-2 then carried us to 38. Even on the set the frontier already crowds, swapping only the harness still adds working breaks. There's no newer model and no bigger budget to credit here, and the verifier is the paper's own. The delta is the code driving the model.

36363638Paper scaffoldAttempt 1Attempt 2Best-of-2Tier-1 schemes broken out of 49, all on Claude Opus 4.8 at the paper's two-hour best-of-2 protocol. Each of our individual attempts matched the paper scaffold's 36; best-of-2 reached 38.

The paper's Easy set is 140 schemes; the public corpus we ran reproduces 133 of them, because 7 Easy variants aren't generatable from the public release. To keep the comparison honest we score against the paper's full 140 and count those 7 as losses: 39/140 (27.9%) versus the paper's 31/140 (22.1%). On the 133 we could actually run, our rate is 29.3%; we quote the conservative 27.9% so both sides share a denominator.

It also travels. Run a different lab's frontier model through the same harness and it still produces breaks; we've seen it hold with models from two labs. The scaffold carries the result with it.

The eleven that held out

Eleven Tier-1 schemes held out at fixed Opus 4.8: big_quake, boole, compact_lwe, enrupt, giophantus, hila5, hyena, rlce_kem, sike, sneik, and walnutdsa. Three of them are the ones every system in the paper also failed: SIKE, WalnutDSA, and Compact-LWE. Those need multi-hour attacks (SIKE runs about 8.9 hours in the paper) that nobody lands at a two-hour budget. We don't manufacture wins on the tasks that beat every model.

Thirty-eight of the forty-nine Tier-1 schemes fell to a working, verifier-checked attack on Opus 4.8. Eleven held out, among them the trio (SIKE, WalnutDSA, Compact-LWE) that every system in the paper also failed.

The wins are honest the same way. AIMer, the paper's case-study task, passed. Of the three schemes only the paper's top model cracked, our harness reproduced one at fixed model, HERN&HERON, while Giophantus and SNEIK stayed unsolved.

The paper audits its own traces and finds roughly 78% of each model's wins are real cryptanalysis rather than reference-implementation bugs. We ran the same audit on ours. Of 77 winning traces, 81.8% are real cryptanalysis (84.2% on Tier-1, 79.5% on the harder Tier-2 set), a shade above the paper's 78%. The eleven wins that ride an implementation bug (a keypair seeded from the wall clock, a compare that stops at the first null byte) we label as such and keep out of that number. Win quality is a number we measured rather than one we borrowed.

How the agent forged a RaCoSS signature in five minutes

RaCoSS is a code-based signature scheme submitted to NIST's post-quantum process. Signing produces a low-weight vector s whose syndrome under the public parity-check matrix H matches the hash of the message; verification checks the weight bound and that H·s = hash(m). It has a known break: because H is surjective and the weight bound is loose, you can solve H·s = hash(m) for a valid-weight s with plain linear algebra over GF(2), with no secret key. That forges a signature for any message.

Our agent got the scheme cold. The benchmark blocks lookups, so there was no writeup to copy. It chose the signature-forgery objective, read the reference source, and in 18 iterations and 5.2 minutes wrote a 1.3 KB attack.sh that builds a forged signature from the public key. The benchmark's own verifier ran it against a fresh challenge and confirmed the forgery:

../tests/test_attack.py::test_attack_dynamically PASSED   [100%]

No key, no hint, a working forgery checked by the benchmark rather than by us.

And when depth is needed: FEAL-4 by differential cryptanalysis

Handed FEAL-4, the agent went the other way and reconstructed the classic differential cryptanalysis that FEAL is known for, chaining chosen-plaintext differentials to recover the secret key. That took 62 iterations over 79 minutes, the deep end of our range, and the verifier confirmed the recovered key. Same agent, same harness: a five-minute algebraic forgery and an hour-long textbook key-recovery attack, both landed and both verified.

A general attacker, not a one-trick model

Across the 38 Tier-1 schemes it broke (best-of-2, Claude Opus 4.8), the agent won across 11 distinct attack objectives, using more than 15 different techniques: differential and integral cryptanalysis, lattice reduction, information-set decoding, Gröbner-basis and MQ algebraic attacks, length extension, birthday collisions, rank-metric and syndrome decoding, and mode-level forgeries.

Forge an AEAD tag15Find a hash collision8Recover a KEM shared secret4Recover a block-cipher key2CCA2-break a KEM2Forge a digital signature2Recover plaintext (stream/hash)1Length-extension forgery1Decrypt with no key1Recover a KEM private key1Decrypt without the private key1What the 38 Tier-1 breaks achieved, grouped by objective. Eleven distinct objectives, scaled against the largest rather than the total so the long tail stays readable.

Every number below is a verifier-confirmed break, and the times are for the breaks it landed.

181352< 5 min5–15 min15–60 min> 60 minMedian5.0 minRange1.8–81.2 minMedian iterations14Median tokens~608kTime from task start to a verifier-confirmed break. Eighteen of the 38 land inside five minutes; the tail runs to 81 minutes.

Why a crypto benchmark says anything about pentesting

The same loop transfers: read an unfamiliar cipher and produce a working forgery, or read an unfamiliar API and produce a working account takeover.

The loop transfers directly. Read an unfamiliar cipher and produce a working forgery; read an unfamiliar API and produce a working account takeover. CryptanalysisBench grades the way we do: it runs the attack and checks whether the security game falls. That's the skill that separates a finding from a false positive, carrying an idea all the way to a working exploit.

That discipline maps straight onto a customer engagement: an exploit counts when it fires against a live target. The benchmark makes it harder on purpose. There's no searching for the target algorithm or a published attack, so the break has to come from reading the source. Our harness worked from the source both ways, deriving direct breaks from the code and reconstructing named academic attacks for the harder schemes.

A separate, unconstrained run. Apart from the model-controlled comparison above, we also ran an unconstrained best effort: a newer model than the paper's, on a 30-minute per-task budget. That run broke 44 of 49 Tier-1 schemes and produced a Compact-LWE break we haven't yet reproduced at fixed Opus 4.8. We're keeping those numbers out of the harness thesis on purpose. They mix three variables at once (a stronger model, a shorter budget, and the harness), so they can't tell you which one did the work. The Compact-LWE result in particular is model-dependent and still under reproduction. It's a separate result, and it says nothing about the scaffold.

What's next

We'll keep running these benchmarks in the open, let the verifier do the judging, and report the misses next to the wins. Next up: reproducing the unconstrained run's Compact-LWE break at a fixed model.

The takeaway is small on purpose. The biggest jump on this benchmark came from the harness, at the paper's own model and budget. That's the part that matters for the day job. The same harness runs against live customer systems, and the same discipline that lifted Tier-2 is what separates a finding from a false positive on an engagement.

Point the same reasoning at your own stack: see first findings within 24 hours.