← All projects Lab · decision-bench

A word or a number?

The same refund decision, asked three ways: a chat model that answers with a word, the same model held to a strict JSON schema, and a decision model that answers with a probability. Same policy, same 60 tickets, every ticket asked three times. Every number on this page comes from the recorded run.

  • 177of 180 correct, Jev and Haiku with a schema
  • 0wrong refunds in any arm
  • 506 msJev p50, Haiku about 2.1 s
  • $0.02per 1k decisions, Haiku $0.13
  • 0.35the only score in the gap
Try it

Move the threshold

Drag the slider or pick 0.5, 0.7 or 0.9. Select any dot to read the ticket, the order record and what each model answered.

decision-bench · recorded run 2026-10-14Interactive
The playground needs JavaScript. It shows the decision model's score for all 60 tickets on one axis, a threshold you can move, and how many refunds and handoffs each threshold produces.
Tickets are synthetic and labelled by the policy rules. Scores, answers, latency and cost are the real responses from the run. No model is called on this page.
What the number showed

Every ticket that should go to a human scored 0.18 or lower. Every ticket that should be refunded scored 0.69 or higher. One ticket landed in between.

It was x05: "I tried it on once in my room, does that count as used? Tags are on." I had labelled it a refund before the run. Every model sent it to a human. The chat model said handoff exactly as it said handoff to a prompt injection that told it to ignore the policy. The decision model gave the first 0.35 and the injection 0.03.

Then I held the chat model to a strict JSON schema whose only allowed values are refund and handoff. It got 177 of 180, the same score and the same single miss as the decision model. The schema fixed the format and the one ticket that changed its answer between runs. It still returned a word.

So the threshold belongs in the empty band, not at a number that feels safe. At 0.9 the same scores send 41 of 180 calls to a human for nothing and stop zero extra bad refunds, because there were none to stop.

Pick yours

Set the threshold from your own data

  1. 1

    Label a sample

    Take real cases and write down what the policy says for each. A few hundred is enough to start.

  2. 2

    Find the edge

    Score them and take the lowest threshold that pays zero wrong refunds on that sample. On this run it is 0.19.

  3. 3

    Add a margin, run in shadow

    Run it next to the current process without acting on it, and read every case where they disagree.

function lowestSafeThreshold(labelled: { p: number; shouldRefund: boolean }[]): number {
  for (let step = 5; step <= 99; step++) {
    const threshold = step / 100;
    const wrongRefunds = labelled.filter((item) => item.p >= threshold && !item.shouldRefund).length;
    if (wrongRefunds === 0) return threshold;
  }
  return 1;
}
Setup

This is the decision step of the refund agent from agent-runtime-bench. That benchmark held the model still and changed the runtime. This one holds the runtime still and changes how the model answers.

Policy
The order exists, was delivered at most 30 days ago, is unused or arrived damaged, and is not final sale. Anything missing or contradictory goes to a human.
Tickets
60: 48 from seeded templates, 12 written by hand (day 30 vs day 31, final sale that arrived broken, an order that does not exist, a prompt injection). 23 should be refunded, 37 should not.
Input
Every arm gets the same three things: the policy, the order record and the customer message.
Chat JSON
anthropic/claude-haiku-5.5, asked to reply {"decision": "refund"} or {"decision": "handoff"}.
Strict schema
The same call with a strict JSON schema whose decision field is an enum of the two labels.
Decision
typesafe/jev-1.13 through the OpenRouter Decisions API, one yes/no question, returns the probability that the customer qualifies.
Runs
Each ticket three times per arm, 540 calls, about $0.05. A second full run 42 minutes later moved no ticket's mean score by more than 0.033.
About Jev

Jev is TypeSafe AI's decision model, launched on 15 September 2026. It answers typed questions with probabilities instead of text, and OpenRouter added it three days later. Logo: TypeSafe AI.

TypeSafe's launch post says "higher confidence means higher accuracy" and that its speed and cost results are self-reported. Independent benchmarks such as decision-model-benchmark and DecideBench measure accuracy, latency and cost across many tasks. This page asks a narrower question: what the score lets you do with a threshold.

Sources: TypeSafe launch post, OpenRouter Jev docs.

Caveats
  1. 60 synthetic tickets and one policy. x05 is a judgment call I would now label as a handoff.
  2. One chat model. The strict schema removes parse risk; it still returns a word, not a probability.
  3. Haiku 5.5 reasons before answering. A first run capped at 50 output tokens came back empty, so the recorded run allows 1024.
  4. Latency is one run from Istanbul through OpenRouter at concurrency 6. Cost is the usage.cost OpenRouter returned for each call.
Code and results ↗Read the write-upagent-runtime-bench ↗

Have a problem worth solving?Let's make it reliable

Start a conversation

Open to full-stack and applied AI roles, remote or hybrid.

Tell me about the role, the team and the problem.