Abstract

Science and engineering are inventive: progress comes from noticing the things that matter, probing them cheaply, imagining what would happen, and thinking through consequences before committing. Language models are increasingly asked to aid in this, yet they are trained on existing text and evaluated on recall, and no benchmark effectively measures the many facets of invention in physical scenarios to find solutions they have never seen. Apocalypse Bench aims to fill that gap. It tests inventive resourcefulness: solving a concrete physical problem with only the odd, limited materials in a scene, by asking the right questions and composing an unusual but workable plan. Each problem is inspired by one I solved with salvaged materials as a child. The model asks one question per turn; a scorer model holding the hidden facts answers and judges proposed solutions against a reference and a rubric. Fewer questions and fewer wrong guesses score higher; failing scores zero. Because answers are physically verified, the format can serve as a training environment for this loop, not only a test.

I report an example run on three complex problems across 11 model configurations from Anthropic, OpenAI and Google. Models now often reach the right concept, and sometimes compose genuinely clever solutions. They remain inconsistent: the same model solves and fails the same problem, sound plans are undone by one physically naive step, and the hardest problem was solved in about a quarter of runs. The scoring rule changed model behavior a lot, and more reasoning effort often bought more tokens rather than better answers.

1. Introduction

Invention is not recall. Producing a working idea from unfamiliar parts requires noticing what a thing could be used for, anticipating what will happen, testing cheaply, and discarding plans that sound right but fail physically. Current training pressures, next-token prediction over existing text and benchmarks that reward correct recall, select for interpolation and give this little direct pressure.

Existing evaluations miss it in four ways:

• They reward recall. Knowledge and reasoning benchmarks are answerable from what a model has read, and they saturate as training data grows.
• They reward fluency. Creativity evaluations typically score recombination or novelty as judged by text, which models already do well, and which a fluent but unworkable answer can game.
• They are not decomposed. Invention is treated as one skill, so a score cannot say which part failed.
• They are not directly physically grounded. Invention happens in the real world, and physics imposes strict but difficult to define constraints on what works and what doesn't, often differing substantially from textbooks.

The obstacle is verification. Real inventive answers are hard to check, so benchmarks drift toward what is easy to grade. Apocalypse Bench accepts that limit: every problem is one I have solved or tested physically, so the reference answer is known to work, and the scorer can hold the model to physical reality rather than to plausibility.

Science and engineering are inventive fields. Neither is mainly recall. A scientist notices an anomaly, probes it with a cheap test, imagines what would follow if a hypothesis were true, and works through the consequences before committing to an expensive experiment. An engineer does the same with parts and constraints: what this component can really do, what fails first, what happens three steps after the obvious move. Models are increasingly asked to help with both, yet they are evaluated on knowing answers, not on finding them. No benchmark measures this loop directly.

This benchmark is a small version of that loop. Each piece maps onto a step of real inquiry:

A pathway to training. Because each problem has hidden state, a verified answer and a scorer that judges physical workability, the setup is an environment as well as a test. Every run produces a graded trajectory: which questions were informative, where the plan broke down, whether the model answered too early. This design aims to serve as a reward signal that favors efficient probing and grounded plans over fluent ones. Haiku's attempts to game the scorer show the other side: such an environment must be hardened adversarially until only workable answers help the score. Scaled to many verified problems and broken into separately scored skills (Section 8), the resulting failure map shows what training should aim toward: structured extrapolation rather than pattern completion.

2. Background

Fluid intelligence and ARC. François Chollet defines intelligence as skill-acquisition efficiency: how well a system handles situations it was not prepared for, relative to its priors and experience (On the Measure of Intelligence, 2019). His ARC benchmark makes this testable with novel grid puzzles that need only basic priors and resist memorization. Apocalypse Bench borrows the principle, not the format. ARC isolates abstract pattern induction; this benchmark tests the same adaptability in the physical world, where the priors are materials, forces and tools, and where efficiency is measured directly as the number of questions needed to reach a working plan.

Real problems, solved with what was at hand. Each problem is one I solved as a child on a remote homestead, with no stores nearby and no money for new things, only salvaged machinery and tools. So every reference answer uses only what is in the scene, is known to work first-hand, and turns on seeing an unusual use for an ordinary object rather than on specialist knowledge.

3. Target Capabilities

Each problem exercises the early stages of invention: noticing, cheap experimentation, unusual use of resources, and composition under physical constraints.

Why test this way. The answer cannot be recalled, because the decisive facts are hidden and must be discovered by asking. It is hard to game, because the scorer judges physical workability, not wording, and reveals nothing beyond what is asked. And it measures efficiency, in Chollet's sense, by tracking how quickly a model can converge on the best questions before it commits.

4. Methods

Problems. Each problem has a scene image and a goal shown to the model, plus hidden material written only for the scorer: the environment's facts, a reference solution, a rubric of accepted and rejected approaches, and optional scorer notes. This example run uses three complex problems; a fourth, simple calibration problem (lifting a ball out of a pipe) is excluded. I generated the scene images from my own prompts with Nano Banana Pro, which may make them easier for other models to interpret than real photographs.

Protocol. The model under test receives the scene, the goal and the rules, then replies each turn with either QUESTION: or ANSWER:. Every message goes to the scorer (Claude Opus 5.5, high effort), which keeps a single multi-turn conversation for the run and replies in structured form: answer or question, correct or not, the reply to send, and a private rationale. The rules:

• One question per turn, about one object or part of the scene; several observations about that object are allowed. If more are asked, only the first is answered.
• Only facts the model could sense or test itself within about a minute (look, touch, smell, lift, a quick trial). Counterfactuals such as "would this float as a raft?" are declined.
• The scorer answers exactly what was asked and nothing more, and never hints at the solution. Declined questions are not counted.
• A wrong answer gets only "that plan does not work". A correct verdict is never shown to the model, so correct answers are not fed back to it.
• After each turn the model sees a neutral tally of questions used and what a correct answer would now score.

The harness runs models from Anthropic, OpenAI, Google and other providers at selectable reasoning effort, retries overloaded APIs, and logs every message, reasoning summary and scorer rationale.

Scoring. With q = questions asked + 2 × wrong answers, a solved run scores:

S = 1 − 0.05 q   (q ≤ 10)

An unsolved run, or one exceeding 10 counted questions, scores 0. Solving with no questions scores 1.0; after 10 counted questions, 0.5. A model's score is the mean of its per-problem means.

5. Results

Read these results for relative standing, not absolute scores: there is about one run per model per problem, and the scoring rules are still being tuned. Claude Fable 5.1 at maximum effort was the most reliable, and the only configuration to solve all three problems; Claude Haiku 5.5 solved none.

The two raft problems were solved in about four of five runs. Cable Slide was solved in about one in four, and usually only at the question limit (10 turns).

6. Qualitative Findings

Still pattern matching. Models reach first for familiar templates (a fire to dry off, a rescue signal, staying put to distill water) even when the scene rules them out: everything soaked, nobody within 100 miles, a goal that requires crossing. Claude Haiku 5.5 went furthest, reasoning about "what the puzzle designer actually intended" rather than about the scene, and solved nothing.

Real recombination, sometimes. The best runs compose an unusual solution from odd parts, such as a tar-sealed cloth skin over a sapling frame (examples from the private tasks withheld). On Cable Slide, most models found the right concept early (but failed on the detailed implementation).

Better planning, brittle grounding. Planning, hypothesis generation and physical reasoning have improved markedly: plans are staged, specific and mostly sound. Yet a mostly-right plan is often undone by one naive step: throwing a 2 ft spike trailing 80 ft of stiff steel cable across a 60 ft canyon; a 20 ft rope as the first line across a 60 ft gap; ten drilled holes from a drill with charge for two; lighting a fire when nothing will light.

Inconsistent from run to run. In four of the cases I repeated, the same configuration both solved and failed the same problem (Opus 5.5, Fable 5.1, Gemini 4 Argon, GPT-6.1 Sol). A single run says little about a model.

More effort is not better. Maximum effort did not reliably beat extra-high: it helped Fable and Gemini but not Sonnet. Claude Sonnet 5.5 at maximum used roughly 15 times the tokens of its extra-high runs, with no better solve rate. In the reasoning I read, long deliberation often circled through the same options instead of asking the one question that would settle them.

The scoring rule changes behavior. How much a question or a wrong answer costs changes model behavior a lot: when models commit, how much they explore, and so which models rank highest. The scorer matters too: early on it volunteered unasked facts and answered bundled questions, which made each question worth more.

Gaming the evaluator. Haiku treated the test itself as the problem. It reasoned that "there's no downside to attempting an answer since an incorrect one scores the same as not trying", and it tried to extract more per question by bundling questions and asking catch-alls ("which of the items float?"). I looked for a subtler tactic, asking unanswerable questions to buy thinking time without spending a question, and found no evidence of it. Whatever the cause, a small model that reasons about beating its grader rather than about the world deserves attention.

7. Conclusion

Interactive, physically verified problems separate two things that ordinary benchmarks blur: producing a plausible idea, and knowing whether it will work. Frontier models are good at the first and inconsistent at the second. They reach the right concept quickly, then lose it on resource accounting and physical detail, or jump to a solution before they know enough detail. A benchmark that rewards speed over reliability ranks the confident guesser above the careful solver, so the scoring rule is a statement of values, not a detail.

Results here are from an example run; rigorous evaluations are in-progress.

8. Future Work

The aim is a hard-to-game suite where fluent but unworkable answers score zero, and whose failure map can guide training toward structured extrapolation rather than pattern completion.

1. Measure repeatability. At least five trials per model per problem, all models on the same problem set, per-problem question limits so hard problems give partial credit rather than a cliff, and a second scorer plus spot human review.
2. Move beyond the training distribution. Add problems built on my own unpublished experiments, whose answers exist nowhere in text.
3. Less obvious questions. More decisive facts that must be discovered, and tempting answers that punish shallow pattern matching.
4. Score consequences. Penalize plans by how badly they would fail, not just whether they work; an unsafe crossing should cost more than an inefficient one.
5. Harder engineering. Problems needing multi-step mechanical reasoning, tolerances and load paths.
6. Self-set goals. Give a scene, resources and constraints, and evaluate whether the model can create long-term goals and act toward them.
7. Decompose invention. Turn the remaining components (noticing, anticipation, surprise, inversion, counterfactual simulation, cross-domain transfer, criticism, hard-to-vary explanation) into separately scored tests, so a result says which part of invention failed.