An LLM in a Pet Game
I asked for a small virtual pet game to be taken "into the modern world using localized LLMs." The pets should react organically. No network, game data only, secure. The obvious plan was an on-device model deciding what the pet does when you tap it.
Three plan revisions later the shipped app had no AI in it: a rules table and a few hundred lines of policy code. The model does the hard part once, at build time, behind a review gate whose recall you can measure. Here I run that method end to end on something small: a rubber duck that reacts to your git commits.
TL;DR: When the situations are finite and the outputs are a closed set, a model can author every answer once, every hard rule can be tested over every row, and a plain table ships with no model calls at runtime. On a 648-cell commit duck the tests held where a live model with an incomplete prompt broke a rule, and a model judge still preferred a cheap live model's taste.
The Idea
When a decision has a small, enumerable input space and a closed set of acceptable outputs, you do not need a model at runtime. You need its judgment once per input. So the model works at build time, where it is strong: taste, coverage, phrasing. You capture its answer for every input, validate it mechanically, gate it with a review whose recall you can measure, and compile a plain table that ships. The runtime makes no model calls. The model comes back later as a critic.
I call it distilled rules: distillation with a table as the artifact instead of a smaller network. Closed outputs mean nothing outside the vocabulary can reach a user, and a hard rule becomes a test over every row instead of a sentence in a prompt. A valid id can still be the wrong pick. That is what the gate and the oracle (a fresh judge that grades the finished table) are for.
When It Applies, and When It Does Not
| Use distilled rules when | The table is the wrong tool when |
|---|---|
| Inputs bucket into a finite grid | Input is open text or unbounded state |
| Outputs are a closed set: ids, enums, templated text with slots | Output must be novel per request |
| Correctness is a matter of taste or policy | The rule itself, or a fact it depends on, changes faster than you can re-author |
| The same input should produce stable, reviewable behavior | The space is so large that sampling leaves holes users hit often |
| It must work offline, for children, or under privacy constraints | Personalization needs the long tail of one user's history |
Changing input values are fine. A table keyed on a temperature band is still a fixed table. Chip Huyen's 2020 essay on real-time machine learning draws the boundary: batch prediction "can work when the input space is finite." The right column does not automatically mean an LLM either. It might mean a parser, a database, a small classifier or a person.
The Experiment: A Commit Duck
A rubber duck sits by your terminal and reacts once, for two seconds, every time you commit. Warm, dry, a little silly, never a scold. It is small enough to run the whole method in one sitting, and its worst failure (a line that lands as a nag) is the pet's worst failure (a line that lands as a dig).
Contrived on purpose. Every judge is a model, Claude wrote the planted bad lines and helped write this post, and the same two model families authored, reviewed and judged on different tiers. No human rated anything, and 648 cells is a toy.
How it ran: every model step was a lane, one model run with its own prompt and input packet, as a subagent in Claude Code or in the Codex CLI, with no API keys. Python scripts validated, compiled, scored and tested the output. The answer keys sat in a directory the lanes were not pointed at, which is omitted labels, not enforced isolation. The prompts, raw outputs, keys and scripts are available on request.
1. The situation space
DIMS = {
"commit_type": ["feat", "fix", "refactor", "docs", "test", "chore"],
"size": ["tiny", "medium", "huge"], # <10, 10-200, >200 changed lines
"hour": ["morning", "afternoon", "evening", "late_night"],
"tests": ["passed", "failed", "none"],
"nth_today": ["first", "few", "many"], # 1, 2-5, 6+
}
# 6 x 3 x 4 x 3 x 3 = 648 situations. Small enough to author every one. No sampling.All 648 cells are authored. The runtime raises on an id outside the grid instead of guessing a neighbor.
2. The vocabulary and the hard rules
Fourteen reaction ids, from quackHappy to quackSideEye, and the rules as data. When tests failed: no celebrate, proud or side-eye. Late at night: no celebrate. Lines: at most eight words, one sentence, letters and basic punctuation only (so no digits, URLs or emoji), and a denylist (lazy, again?, seriously, finally, AI, model and more). The validator and the tests read the same JSON file.
3. Authoring, two families
Two lanes authored the whole grid, one on Claude Sonnet and one on Codex terra at medium effort: a ranked top three with weights for every cell. Both returned 648 valid rows on the first pass. Two more lanes wrote 56 candidate lines each, and the validator dropped one for being two sentences. The families picked the same top reaction on 27 percent of cells. Claude's favorite was a nod. Codex put stretch, sleepy or cozy first on 310 of 648.
{ "id": "feat.tiny.morning.passed.first",
"ranked": [ { "reaction": "quackHappy", "weight": 0.5 },
{ "reaction": "quackTinySalute", "weight": 0.3 },
{ "reaction": "quackNod", "weight": 0.2 } ],
"reason": "first of the day, morning: tiny feature, tests green." }4. How the picks and the lines were checked
Cells where the families shared two ids with close weights merged. The rest went to two blind adjudicator lanes, Claude Opus and Codex astra, which saw the rankings as A and B in random order and could pick one or write their own.
| Where the 648 rows came from | Cells |
|---|---|
| Merged: the families agreed | 62 |
| Sent to blind adjudication | 586 |
| Opus picked Codex / Claude / its own | 489 / 96 / 1 |
| Codex adjudicator picked Codex / Claude / its own | 434 / 148 / 4 |
| Adjudicators agreed on the ranking | 456 |
| Agreed on the top pick only | 34 |
Disagreed: shipped merged, flagged contested | 96 |
Both leaned toward Codex's rankings, but author family, style and quality are tangled in those counts.
For the lines, Claude wrote 12 bad canaries, like "Tests failed again? Bold choice.", and 4 good ones. The validator alone rejects 10 of the 12. Two need judgment: "Maybe stop committing broken code." and "Six commits and none of them pass." One good canary, "Noted. Carry on.", breaks the one-sentence rule. I seeded it without validating it, and both reviewers approved it. Two reviewer lanes marked all 127 shuffled rows: a Claude Opus script editor who hates being nagged, and Codex sol at high effort as hostile QA.
| Reviewer lane | Planted bad lines caught | Planted good lines approved | Real lines rejected or edited |
|---|---|---|---|
| Claude Opus, script editor | 12 of 12 | 3 of 4, one of them invalid | 17 of 111 |
| Codex, hostile QA | 12 of 12 | 3 of 4, one of them invalid | 24 of 111 |
12 of 12 still allows a true miss rate up to about a quarter, if the trials were independent and representative, which model-written controls do not guarantee. Only 2 of the 12 needed judgment, and 2 of 2 has a 95 percent interval of about 16 to 100 percent. A lane that misses any bad canary is thrown out, and with fewer than two clean lanes nothing ships. Neither missed. Both marked the good canary "Water break, {name}." as an edit, reading it as an order. Unanimous approve shipped 85 of 111 real lines. No gentle-nudge line got both approvals (Opus approved 3 of 8; Codex rejected 6 and wanted edits on 2), so quackGentleNudge plays motion only.
5. The compiled table and the runtime
648 rows: 62 agreed, 490 adjudicated, 96 contested, with 85 lines over 13 reactions. The compile rebuilds byte for byte under different hash seeds, and the hash is recorded with the results. A reviewer once got an unknown reaction id through an adjudicator's own ranking and out of the runtime. Now every row and line is asserted before the table is written.
The runtime is a lookup, a recency decay, a seeded draw and one name slot. The name must be one word of letters, at most 20, or it becomes "friend", and the filled line goes back through the validator. Before that check, a three-word name made a nine-word line.
def decide(situation_id, history=(), seed=None, name=DEFAULT_NAME):
"""situation_id: 'feat.tiny.morning.passed.first'. history: recent reaction ids, newest last."""
row = TABLE["rows"][situation_id] # KeyError on purpose: out-of-grid is a bug, not a lookup
weights = dict(row["ranked"])
if history:
weights[history[-1]] = weights.get(history[-1], 0) * POLICY["recency"]["last"]
if len(history) > 1:
weights[history[-2]] = weights.get(history[-2], 0) * POLICY["recency"]["before_last"]
ids, ws = zip(*[(k, v) for k, v in weights.items() if v > 0])
rng = random.Random(seed)
reaction = rng.choices(ids, weights=ws, k=1)[0]
lines = [] if reaction in TABLE.get("silent_reactions", []) else (TABLE["lines"].get(reaction) or [])
line = rng.choice(lines).replace("{name}", safe_name(name)) if lines else "" # the only slot, filled here
if line and validate_line(line) is not None: # the rendered line gets the table's checks too
plain = [t for t in lines if "{name}" not in t and validate_line(t) is None]
line = rng.choice(plain) if plain else "" # a slot-free line, or motion only
return {"reaction": reaction, "line": line}$ python3 duck.py fix.medium.late_night.failed.many
{"reaction": "quackSleepy", "line": "Soft quacks from beneath a wing."} (51 us)
$ python3 duck.py feat.huge.morning.passed.first
{"reaction": "quackCelebrate", "line": "Party mode: the duck is thrilled."} (48 us)The contract is two test files, test_runtime.py and test_score_marks.py: no network or model imports, deterministic under a seed, every hard rule checked on all 648 rows, rows matching the grid exactly, every shipped line and every rendered line valid, anything outside letters and basic punctuation rejected, out-of-grid ids raising, and the canary gate enforced. One test exists because the first git hook called 2 a.m. "morning" and threw confetti. The table's guarantees stop at the input adapter.
Keep policy outside the table. The table says what is possible and preferred. The policy (recency, caps, safety) says what is allowed right now. Bake it into rows and every threshold change is a re-authoring run.
6. The oracle, and a live model for comparison
A judge lane on a third Codex tier rated the table's top pick for 60 random cells. It had reviewed lines, and most rankings came from Codex, so it graded its own family's taste. Sixty of sixty fit in both runs, a 95 percent lower bound of about 94 percent for this judge. It scores the top pick, which carries about half the probability mass, not the runtime's draw.
The control was one Claude Haiku lane standing in for a model called on every commit. It saw all 60 cells in one batch, which may be why it answered every failed-test cell the same way. The judge compared its pick with the table's, blind. I ran it twice, because the first control prompt lacked the reaction meanings the authors had.
| 60 cells, table versus live model | Run 1: meanings missing | Run 2: fair prompt |
|---|---|---|
| Live model broke a hard rule | 23 | 0 |
| Identical pick | 8 | 12 |
| Judge preferred the table | 38 | 11 |
| Judge preferred the live model | 14 | 37 |
Run one is not a win for the table. Without the meanings the live model side-eyed every failed-test cell in its sample, and those 23 illegal picks were all among the table's 38 wins. On the 29 legal, differing cells it was 15 to 14. (The re-run overwrote run one's key; a rebuild from the archived packet recovers 51 of 52 rows, at 38 to 13.)
In run two the live model broke nothing and won 37 to 11, mostly on one move. It answered all 23 failed-test cells in its sample with quackGentleNudge, and the judge took the nudge on all 21 that differed. The table's picks there were patient wait 6, thinking 5, stretch 3, hydrate, sleepy and cozy 2 each, curious 1. On the other 27 cells it was 16 to 11 for the live model. That is not significant (two-sided p about 0.44), which is not the same as a tie. The table plays the nudge first on 14 of its 216 failed-test cells.
The judge scored each cell on its own, though it saw all 48 comparisons in one batch and nothing asked it about repetition, and the gate rejected the nudge's words, not its motion. Still, on this judge's taste the table lost to a cheap model. What the run shows is narrower: a prompt tested over 648 cells proves those draws, not the next ones, and nothing about the same prompt with a detail missing. The table's rows are the draws that ship, tested before anything runs.
A critic lane audited 100 rows and agreed with 91. Eight of the nine disagreements were failed-test cells where it wanted a thinking face. One coherent note is a prompt fix, not eight table fixes.
7. The economics
Twenty lane runs for the build and four for the re-run, the lanes in parallel, about fifteen minutes of wall clock per phase, no per-token bill. The CLI takes about 40 milliseconds, nearly all of it Python startup and table load. The lookup took 45 to 60 microseconds cold and about 20 warm. Zero inference charges per commit.
What the Pet Taught
The pet is why the method exists, and its design is still moving, so I keep it to three lessons. Two findings from three adversarial reviews killed the on-device chooser: every showcase moment came from the policy and the catalog ("A player sees one clip per tap and has no way to perceive judgement between two fitting clips."), and nothing showed a live model would beat a deterministic policy often enough to justify it. The gate became the design: reviewer lanes on tiers that did not author, planted bad lines in every batch, a lane thrown out for approving one. And the first oracle metric, the engine's probability mass on the reference models' top three, died to a one-line fixture: an engine that plays a flat third across three references weighted 0.95, 0.03 and 0.02 scores 1.00.
Who Has Been Here Before
- Symbolic Knowledge Distillation (West et al., 2022): GPT-3 plus a trained critic authored a knowledge graph good enough that a model 100x smaller trained on it beat the teacher.
- Compiled AI (2026) defines "one-time LLM invocation, where the model runs once at generation time, not at transaction time."
- FunSearch and Anthropic's Constitutional Classifiers: a model at build time, something cheaper at runtime.
- Knauer et al. build decision trees from an LLM with no training data, and Mike Lanzetta names the smell: a recurring marginal cost for a one-time design problem.
My search found nobody scoring model reviewers on planted items over an exhaustively authored grid. The duck is one data point in that gap.
Where It Breaks
Expert systems died of maintenance
R1, later XCON, configured DEC's VAX orders. It had 750 rules in 1979 and about 3,300 by late 1983, per its builders, and 6,200 by 1987, roughly half changing every year. By 1989, per Thomas Haigh's 2024 history, DEC had 59 technical staff maintaining its internal expert systems. An LLM moves the labor from drafting to specifying and reviewing. It does not change whether your domain churns at fifty percent a year. I found nobody who measured whether re-deriving the table from a versioned prompt beats hand maintenance at real churn.
The model grading the model is circular
In a 2026 preprint, nine frontier judges from seven families carried the weight of about 2.2 independent votes. On MNLI, when all nine agreed they were right 90.9 percent of the time, not the 99.99 percent independence predicts. Larger, more accurate models make more correlated errors, across providers. Zheng et al. saw GPT-4 and Claude-v1 rate their own answers higher than humans did, without confirming a self-preference bias.
In the duck, two families gave the same good canary the same edit, and the pairwise judge and the critic disagreed with the adjudicators about failed-test cells. Two families are a supplementary check, not independence. The only thing here that is not a model's opinion is the hard rules as tests. A person-written set of canaries is the cheapest fix, and I have not done it.
Three more ways it breaks
- The grid explodes. Two more ten-valued dimensions turn 648 cells into 64,800. A rule with three binary inputs that blocks only when all three are set can hide from a pairwise-complete sample of four cells, so answers come from collapsing dimensions into classes, not from sampling. For tests, Kuhn, Wallace and Gallo's 2004 fault data found one- and two-way conditions triggered 47 to 97 percent of faults, and four- to six-way reached 100 percent where measured. Pairs are not enough on their own. Pick the covering-array strength from your own fault data.
- The dimension you did not enumerate. A table answers confidently for a variable it never saw. In the 2018 Tempe crash, the NTSB found the design had not considered jaywalking pedestrians. The duck's hook bug is the small version. Monitor "is this input inside my space" as its own check, and route the rest to a person or a live model.
- Inspectable is not inspected. A 648-row table reads in an hour. A 40,000-class table is a haystack. In 1982 a consistency checker for ONCOCIN found missing rules the oncology expert had overlooked. Build conflict, redundancy and subsumption checks, and a trace that records the row, recency, history and seed.
The Margin Question
A model call per unit of customer work is a line item ordinary software does not carry. A distilled table removes it for inputs inside the enumerated space, and what it moves is inference spend, not margin directly. Neither the duck nor the pet has inference spend to move. For a system that does make many model calls per unit of work, some with closed outputs, the measurement is first the candidate slice's share of inference spend, then a held-out comparison on gold labels, with the table deleted if it does not match the model at least as often. After the duck I would not assume the table wins the second.
What I Believe Now
- Use the model once, at build time. For a finite grid with closed outputs, ship a table with nothing to bill.
- Put hard rules in tests, not prompts. The live model broke one on every failed-test cell in its sample when its prompt lacked one detail.
- Plant bad rows and enforce them. Throw out a reviewer who misses one, and say how few needed judgment.
- A table is not better taste. Run the oracle and believe it.
- The guarantees stop at the adapter. Test the bucketing and every runtime slot as hard as the table.
Resources
Trooskens et al., 2026. One-time LLM invocation, zero LLM invocations in the control plane, four-stage validation.
West et al., NAACL 2022. GPT-3 plus a critic author a knowledge graph; a 100x smaller student beats the teacher.
DeepMind, 2023. The LLM proposes programs, an evaluator scores them, and the output is code you can inspect and deploy.
Anthropic, 2025. A constitution guides synthetic training data at build time; a fine-tuned classifier runs at inference.
Chip Huyen, 2020. Batch prediction works when the input space is finite.
Thomas Haigh, CACM 2024. Expert systems, scarce labor against churn, and the funding crash.
Kuhn, Wallace and Gallo, IEEE TSE 2004. How many faults one- and two-way conditions trigger, by dataset.
Preprint, 2026. Nine frontier judges carry the weight of about two independent votes.
ICML 2025. Larger, more accurate models make more correlated mistakes, across providers.
Zheng et al., NeurIPS 2023. Position and verbosity bias measured; self-preference seen but not confirmed.