ldraw-nova: AI agents build LEGO models by writing Python geometry
In ldraw-nova, models write a plan and a Python generator instead of raw coordinates, but nothing checks physics or stability.
livenerf on Claude Opus 5.5: output tokens catch model drift first
livenerf's validation found accuracy noise bigger than the real change, while output token counts clearly showed effort shifts.
Huseyin Kaplan · 1 min read
Every few weeks someone swears a model got worse after launch. Almost nobody kept a launch-day baseline to check.
A developer started one for Claude Opus 5.5 on release day. It's called livenerf, it reached the Hacker News front page overnight, and its validation run is the part worth reading.
Two identical runs differed by about 6 accuracy points. Dropping effort from high to medium moved accuracy about 4. The noise was bigger than the change.
Output tokens caught it. Medium effort cut them by about a quarter, low by about 60%. Swapping in the older Opus 5 wasn't distinguishable at all.
To catch drift in your own app:
- Log output tokens per request next to eval scores.
- Pin your harness and CLI version. A changed harness looks like a changed model.
- Keep a frozen set of questions the model only sometimes gets right.
Accuracy is noisy. Token counts are cheap evidence.
