Results — including the ones that say “no”
Most world-model write-ups lead with a win. The two findings below are the ones that actually changed how we build with this library, and both are negative or complicating results. We think that’s the more useful thing to publish.
1. Aggregate accuracy on a transition-prediction task is rigged by class imbalance
We trained MLPWorldModel to predict (state, action) → next_state and compared it against the simplest possible baseline: a two-line self-updating lookup table keyed on (state_desc, action).
Naive aggregate accuracy said the trained model was competitive. It wasn’t a fair comparison — bucket the same rows by whether that exact (state_desc, action) pair had been seen during training:
| bucket | share of rows | lookup accuracy | trained-model accuracy |
|---|---|---|---|
| seen before | 75.2% | ~0.973 | ~0.973 |
| unseen | 24.8% | 0.402 | — |
| aggregate | 100% | 0.972 | 0.936 |
Three-quarters of the evaluation set was already at ceiling for a zero-parameter lookup table, so the aggregate number is dominated by a bucket where neither method has to generalize at all. It can’t show a real win or a real regression — an aggregate metric here answers a different question than the one you’re asking.
Once we corrected the loop that fits the model (see below) and re-ran with the bucket split in place, the trained model still lost to the lookup table on 98.9% of rows (0.9357 vs 0.9720). It won only on the 1.1% of rows carrying a genuinely novel action — which is exactly the case a lookup table structurally cannot handle, and the only case where a model is worth the extra machinery.
Takeaway: if you’re evaluating a learned transition model, bucket by seen/unseen (state, action) before you trust an aggregate number, and always compare against the dumbest possible baseline first. A cache that already covers 75% of your evaluation set at ceiling will make almost anything look “competitive.”
The bug that made the first version of this comparison meaningless
The training loop that produced the first “trained model” here declared “3 epochs” in its own metadata while a bug in the batch sampler meant each “epoch” drew one random batch of 64 rows — 192 rows actually touched, out of a 20,000-row dataset that was reported as fully consumed. The reported loss curve looked fine (it was the loss on those 192 rows), and the regression gate that compares a new checkpoint’s loss to the previous checkpoint’s loss could only ever catch “worse than an untrained model” — it structurally could not tell a genuinely fitted model from a randomly-initialized one scoring against 192 samples. Fixing the loop (3 epochs of 192 rows → a real 10,296 gradient steps over the full set) is what produced the 0.9357 number above; the number before the fix was not measuring what it claimed to.
Takeaway: a green test suite over your training pipeline — shapes, checkpoint fields, loop plumbing — proves the plumbing works. It cannot prove the model was actually fit, because a plumbing test can’t distinguish a fitted model from an unfit one initialized the same way. Every trainer needs at least one positive assertion that it beats a named baseline, on held-out data, with the bucket split above applied.
2. An ensemble’s aggregate “champion” model was the worst individual predictor
Comparing several candidate world-model configurations by pooled/aggregate accuracy across an evaluation set picked a “champion” that, decomposed per-scenario, was in fact the worst individual predictor of state change among the candidates. The aggregate metric rewarded a model that did well on the easy majority of cases (again, largely the “seen before” bucket from finding #1) and masked that it was actively worse than its competitors on the harder minority — the cases the metric exists to measure in the first place.
Takeaway: the same class of bug as #1, from a different angle — a single pooled number over an imbalanced evaluation set will reliably pick the wrong winner if “wrong” means “worse where it matters.” We now score every world-model candidate on a stratified breakdown before trusting any ranking between them.
3. ARC-AGI-3 solving agent — where it actually stands
Running this world model as the planning core of an ARC-AGI-3 solving agent gets a measured solve rate of 2.12%, with the agent building reusable memory inside exactly one game so far. This number survived three separate attempts to argue it was an artifact of measurement (a stricter scoring pass, a check for silent no-ops in the harness, a check that the eval could distinguish “the model is wrong” from “the endpoint is dead”) — each one adversarially trying to find a reason the number was better or worse than it looked, and each one leaving the number where it started. We’re publishing it at that value rather than waiting for a better one, because a solve rate nobody adversarially checked is worth less than one that survived someone trying to break it.
If you use this library and get a result — especially a negative one — we’d rather hear about it than not.