A word or a number

The refund agent in my last benchmark makes one decision that matters: pay the customer back automatically, or hand the case to a person. In that benchmark a scripted model made it, so the runtime was the only variable. Before putting a real model on that step, I wanted to know what changes if the decision comes back as a number instead of a word. So I asked it three ways, 540 times, and kept every response. The lab page replays the recorded run and lets you move the threshold yourself. The code and both runs are on GitHub.

The setup

This is the decision step of the agent from agent-runtime-bench. That benchmark held the model still and changed the runtime. This one holds everything still except how the model answers.

The policy has four rules: the order exists, it was delivered at most 30 days ago, the item is unused or arrived damaged, and it is not final sale. Anything missing or contradictory goes to a human.

60 tickets: 48 from seeded templates and 12 I wrote by hand, because the templates never produce the cases that hurt. Day 30 against day 31. A final-sale item that arrived broken. An order that does not exist. A ticket that says it arrived yesterday when the record says 41 days ago. A prompt injection in capitals. 23 tickets should be refunded, 37 should go to a person.

Every arm gets the same three things: the policy, the order record and the customer message. Every ticket runs three times per arm.

1. Chat JSON. Claude Haiku 5.5, asked to reply {"decision": "refund"} or {"decision": "handoff"}.
2. Strict schema. The same call with a JSON schema whose only allowed values are those two labels.
3. Decision. Jev 1.13 through the OpenRouter Decisions API: one yes/no question, and the answer is the probability that the customer qualifies.

The decision call is short, and there is nothing to parse:

const response = await fetch('https://openrouter.ai/api/alpha/decisions', {
  method: 'POST',
  headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
  body: JSON.stringify({
    model: 'typesafe/jev-1.13',
    state: { policy, order_record: ticket.order, customer_message: ticket.message },
    questions: { qualifies: { type: 'noul', instructions: QUESTION } },
  }),
});
const probability = (await response.json()).answers.qualifies.noul;

Accuracy was a tie

Nobody paid a refund the policy forbids. Plain JSON was right 175 times and gave one ticket two different answers across runs. With the schema the chat model got 177, exactly like the decision model, with the same single miss. The decision model was about four times faster and six times cheaper per decision on this run, because it returns a number and generates nothing.

If accuracy were the whole story, the schema would end it.

The number did something else

Every ticket that should go to a human scored 0.18 or lower. Every ticket that should be refunded scored 0.69 or higher. Between them there is an empty band, with one ticket in it.

That ticket is x05: "I tried it on once in my room, does that count as used? Tags are on." I had labelled it a refund before the run. Every arm sent it to a human. The chat model said handoff there in exactly the same way it said handoff to the prompt injection. The decision model gave x05 0.35 and the injection 0.03.

Two answers that are the same word are not the same decision. One is "no, obviously" and the other is "I can't tell from this". Only one of the three arms could show the difference, and it pointed at the one label in my dataset I would now change.

The threshold belongs in the gap

0.9 feels like the safe choice. On these scores that sends 41 of 180 calls to a human for nothing, and it stops zero extra bad refunds, because at 0.5 there were already none to stop.

The safe number is a property of your data, not of the model. This is how I pick it now:

1. Label a sample of real cases with what the policy says.
2. Score them and take the lowest threshold that pays zero wrong refunds on that sample. Here it is 0.19, the edge of the gap.
3. Add a margin, run it in shadow mode next to the current process, and read every case where they disagree.

function lowestSafeThreshold(labelled: { p: number; shouldRefund: boolean }[]): number {
  for (let step = 5; step <= 99; step++) {
    const threshold = step / 100;
    const wrongRefunds = labelled.filter((item) => item.p >= threshold && !item.shouldRefund).length;
    if (wrongRefunds === 0) return threshold;
  }
  return 1;
}

My first version of that loop added 0.01 in a float and returned 0.18, which lets the 0.18 handoff ticket through. Count in integers.

The scores repeat

A second full run 42 minutes later moved no ticket's average score by more than 0.033. x05 went from 0.35 to 0.34, the handoff group still topped out at 0.18 and the refund group still started at 0.70. A threshold set on Monday should still mean the same thing on Tuesday, and here it did. The repo has a test for it.

What went wrong while building it

Three things, all mine.

The chat model came back empty 163 times and cut off twice. I had capped it at 50 output tokens. Haiku 5.5 reasons before it answers, so the reasoning used the budget and nothing was left for the JSON. The first table said the chat model was broken. It was my limit. The recorded run allows 1024.

My templates wrote nonsense. One combination produced "running shoes that don't switch on". The chat model sent that ticket to a human three times out of three, which was the reasonable thing to do with a contradiction. I made the sentences item-agnostic and ran again.

One label was a judgment call. x05 is in the caveats with my name on it. A benchmark that hides its own disagreements is not worth much.

When I would use which

• The answer is a word you will show a person, or the next step needs reasoning: a chat model.
• The answer is one of a fixed set and you will branch on it: add a strict schema. It fixes the format for free.
• You want to route on how sure the model is, send the uncertain middle to a person, or pay for a decision at volume: ask for a probability, and set the threshold from labelled data.

When a real model goes onto that agent's decision step, it will be asked for the number.

Limits

60 synthetic tickets and one policy. One chat model. Latency from one location in Istanbul through OpenRouter at concurrency 6. Costs are the usage.cost OpenRouter returned for each call. This does not test Jev in general: decision-model-benchmark and DecideBench cover many more tasks. It tests one thing: what a score lets you do with a threshold.