Skip to content
JevAgentHarness
Decision models for LLM agents

JevAgentHarness Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Kunlun Zhu1, Tao Feng1, Xu Han1, Heng Wang1, Ling Yang2, Xiangru Tang3, Jiaxuan You1, Heng Ji1

1University of Illinois Urbana-Champaign2Princeton University3Google

Paper arXiv HF Paper Code Models Benchmark
TL;DR

Let a small model make the routine choices of an agent

Most steps of an LLM agent end in a multiple-choice decision, such as which tool to call next. A small decision model can answer many of them in a fraction of an LLM call's time and price. We test such models inside running agents, deploy them behind a confidence threshold fitted on the agent's own trajectories, and train an open 4B model on the states an agent actually visits.

Figure 1 of the paper. Top: today the LLM makes every choice (tool, arguments, stop), one LLM call per step, slow and costly. Middle: most steps are multiple choice; a decision model scores the options (get_order, cancel_order, ask_user) and decides when its confidence clears a threshold, otherwise the LLM decides. Bottom: with a decision model, most steps run without the LLM, which is faster and cheaper.
Figure 1. Most of an agent's LLM calls end in a multiple-choice decision, and a small decision model can make the routine ones in a fraction of the time and price. The LLM makes the choices the decision model is unsure of.Icons: Microsoft Fluent Emoji (MIT).
Abstract

Large language model (LLM) agents now fix software, operate computers and serve customers, but their harnesses, the programs that run them, call a large LLM at every step, which makes long tasks slow and costly. Most of these calls end in a multiple-choice decision, such as which tool to call next. Decision models such as Jev answer these questions directly, at a fraction of an LLM call's time and price. We ask how a harness can delegate these decisions to a decision model with little loss of success. To evaluate decision models in agentic tasks, we build AgentDecisionBench, which tests them from isolated questions up to control of a running agent, on public benchmarks and our DecisionGym environment. The decision model most accurate on isolated questions solved only 69.5% of agent tasks, and without a fitted confidence threshold, delegating tool choices lowered success by 19.0 points. Building on these findings, we then design JevAgentHarness, which fits this threshold for each choice on the agent's own trajectories and lets the LLM decide below it. Because existing decision models never see the states an agent reaches after its own mistakes, we train AgentDecision-4B, an open 4B model, on such states with Jev's answers as labels. On tasks unlike its training data, the agent solved 91.5% of them with AgentDecision-4B and 82.0% with Jev. On tasks like their training data, 4B models trained this way made agents 81.1-89.0% cheaper and up to 2.5 times faster, without losing 3 or more points of task success. The models, data, and harness will be released at https://github.com/JevAgentHarness/JevAgentHarness.

Method

Three parts: evaluate, deploy, train

Each part answers one question. Which decision models work inside an agent? How should a harness use them? Can a small open model learn the decisions?

Benchmark

AgentDecisionBench

Tests decision models at three levels, each closer to real use, on public benchmarks and on DecisionGym. DecisionGym is a customer-service environment that knows the acceptable next steps at every state, so each choice can be scored.

  • L1Isolated questions. One question with no agent around it.
  • L2Step level. States from recorded agent runs. Answers are scored but not carried out.
  • L3Task level. The model makes a running agent's choices. Scored by task success.
Dataset on Hugging Face
Harness

JevAgentHarness

Puts a decision model into an existing harness. At a decision point the harness asks a typed multiple-choice question. It uses the answer only when the model's confidence clears a threshold fitted on the agent's own logs. Below the threshold, the LLM decides.

Decision points
tool choiceargumentsstopreflectionroutingmemoryaction
Modes
replacecascadeassistguardtrigger
Code on GitHub
Model

AgentDecision-4B and -0.8B

Existing decision models never see the states an agent reaches after its own mistakes. We train on exactly those states, with Jev's answers as labels. AgentDecision-0.8B applies the same recipe to StartLux-Decision-0.8B.

  1. Agent states labelled with Jev's answers
  2. SFT learns Jev's answers
  3. GRPO rewards success with fewer LLM calls
  4. Public tool-use data for the released 4B model
Models on Hugging Face
Figure 2 of the paper. Panel 1, Procedure (Stage 1, what to delegate): record the decision model silently on calibration tasks, fit a threshold per decision point, test each delegation (is it as good and cheaper? yes: delegate it, no: the LLM keeps it), select and freeze the passing delegations. Panel 2, Benchmark and harness: 8 decision points, 5 integration modes (replace, assist, guard, cascade, trigger), 3 benchmark levels (isolated questions, recorded agent runs, a running agent), cost and time of every call, adapters for existing harnesses. Panel 3, Post-training AgentDecision-4B: visited states labelled by Jev and the answer key, SFT to learn Jev's answers, GRPO for success with fewer LLM calls; AgentDecision-4B replaces Jev.
Figure 2. Overview of the pipeline. Stage 1 tests each kind of choice alone, and Stage 2 tests the passing delegations together on new tasks. These were the tool and argument cascades and a reflection trigger.
Open-source proxy

Routing for coding agents, with no change to the agent

JevAgentHarness also ships as a local OpenAI-compatible proxy. Codex, pi, DeepSeek Harness, mini-swe-agent and Terminus-2 work through it unchanged.

  1. The harness sends each model call to the proxy instead of the provider.
  2. A decision model reads the agent's state and answers one question: is the next step routine or hard?
  3. Routine steps go to a fast model and hard steps to the strong one. The first step and the decision that the task is done stay on the strong model, and any error falls back to it.
  4. Every step is logged with its tier, tokens, cost and latency, so jah report shows the saving on your own tasks.
Architecture of the JevAgentHarness proxy. An unmodified agent harness (Codex CLI, pi, DeepSeek Harness, mini-swe-agent, Terminus-2) sends every step through the OpenAI-compatible API to the proxy. The proxy keeps per-session state, asks a decision model whether the next step is routine or hard, and applies router rules: first step strong, escalate when p(hard) is at least tau, finish check, fail-safe. Hard steps go to the strong model, routine steps to the fast model. Every step is logged.
Results

What the experiments show

Every number on this page is read from the released data files. Hover or focus a mark for details, or open the numbers under each chart.

AgentDecisionBench

Decision-model leaderboard

Accuracy on isolated questions says little about success inside the agent

Each row is one decision model at the three levels (%). Rows are sorted by task success.

Tasks
Show

Source: AgentDecisionBench, configs leaderboard and models (data/leaderboard.json). Paper: Table 3, Figure 4b, App. G.

Full leaderboard

All models with their numbers at each level. Click a column to sort. A dash means the model was not run at that level.

Show
Paper, Figure 4

Isolated accuracy, cost, and success inside the agent

Cost against accuracy on isolated questions

Show the numbers
Source: data/frontier.json, built from e30_cost.tex and e9_frontier.tex. Paper: Figure 4a, App. G.6 and G.7.

Isolated scores do not rank decision models for agents

Show the numbers
Source: data/frontier.json (AgentDecisionBench leaderboard rows; Kendall from e21_shift.tex). Paper: Figure 4b, Sec. 7.1.
Paper, Figure 5

Where a decision model can take over

Delegating tool and argument choices in a cascade saved cost without losing success

Colour by
What the rows and columns mean (paper Figure 3)
Figure 3 of the paper. (a) Where a decision model can act: eight decision points in one step of the loop (cheaper LLM, reflect, which tool, arguments, stop, write memory, read memory, and which action in text games). (b) How it acts, five modes: Replace, the decision model always decides with no LLM call; Cascade, it decides when sure, otherwise the LLM decides; Assist, it shortlists or advises and the LLM picks; Guard, the LLM proposes and it approves or sends it back; Trigger, it decides whether an optional step runs.
Figure 3. A decision model can act at eight points of the agent loop, in five modes. Filled boxes choose the agent's action, and outlined boxes control the loop.
Show the numbers
Source: data/module_map.json, built from module_map.tex. Paper: Figure 5, Sec. 7.2; full map with intervals in App. G.2.
Paper, Table 1

Delegation in harder settings

Delegation in harder settings

Task success of the LLM alone and with the delegation, the share of the cost saved, and the mean time per task.

Show the numbers
Source: data/hard_settings.json, built from the paper's generated tables (research/paper/generated). Paper: Table 1, Sec. 7.3.
Open-source proxy

Coding agents on Terminal-Bench 2.0

Coding agents on Terminal-Bench 2.0

Setting

Show the numbers
Source: data/tb2.json, copied from bench/results and the run configs. Protocol written before the test runs: bench/PROTOCOL.md.

Decision-point modes on the development split

Show the numbers
Source: data/dev_modes.json, a copy of bench/results/dev_modes.json. Details: docs/modes.md.
Quickstart

Route your coding agent in three steps

The proxy holds the provider keys. Your harness gets a local URL and a dummy key, so keys never enter the agent's sandbox.

1

Install and start the proxy

git clone https://github.com/JevAgentHarness/JevAgentHarness.git && cd JevAgentHarness
pip install -e .
export GEMINI_API_KEY=...              # the LLM tiers (Gemini by default; any OpenAI-compatible API works)
export OPENROUTER_API_KEY=...          # the decision model (Jev on OpenRouter); or use {type: rules}, no key
jah init                               # optional: writes ~/.jah/config.yaml (commented) to edit
jah serve                              # http://127.0.0.1:8787/v1, uses ~/.jah/config.yaml if it exists
2

Point your harness at it

Use the base URL http://127.0.0.1:8787/v1, any dummy key and the model name jah-auto. jah setup <harness> prints the snippet for your harness.

jah-autorouted: the decision model picks the tier for every step
jah-strongevery step on the strong tier (your baseline)
jah-fastevery step on the fast tier
~/.codex/config.toml
model_provider = "jah"
model = "jah-auto"
web_search = "disabled"

[model_providers.jah]
name = "JevAgentHarness"
base_url = "http://127.0.0.1:8787/v1"
env_key = "JAH_API_KEY"
wire_api = "responses"
stream_idle_timeout_ms = 600000
JAH_API_KEY=jah codex exec --skip-git-repo-check "fix the failing test"
3

Measure the saving on your own tasks

Run the same task with jah-strong and jah-auto, then compare them. Shadow mode logs what every decision point would do without changing anything, and jah fit fits the thresholds on those logs.

jah report ~/.jah/logs            # per model name: sessions, steps, strong share, $ and LLM seconds per session
jah serve --preset shadow         # every point logs its answer next to what the agent did; nothing changes
jah fit ~/.jah/logs               # fit each point's threshold on those logs and print a config snippet
Release

Models, data and code

The models and the training and coding data are released for non-commercial use under CC BY-NC 4.0. The benchmark is released under CC BY 4.0 and the code under MIT.

ReleaseWhat it isLicenceLink
AgentDecision-4BModelQwen3-4B trained on agent states with Jev's answers as labels (SFT, then GRPO), then on public tool-use data. The main 4B model of the paper.CC BY-NC 4.0Hugging Face
AgentDecision-4B-v1ModelThe SFT and GRPO checkpoint, frozen before the Stage 2 tests.CC BY-NC 4.0Hugging Face
AgentDecision-0.8BModelStartLux-Decision-0.8B trained with the same recipe. The base model is licensed CC BY-NC 4.0.CC BY-NC 4.0Hugging Face
AgentDecisionBenchDatasetStep-level questions with answer keys, every model's answers at the isolated and step levels, the DecisionGym task lists for the task level, and the paper's leaderboard.CC BY 4.0Hugging Face
AgentDecision-trainDatasetThe training data: DecisionGym agent states with Jev's soft labels and gold labels, plus development splits.CC BY-NC 4.0Hugging Face
coding-agent-decisionsDatasetTerminal-Bench 2.0 runs through the proxy: the decision model's inputs, every step with its tier and cost, per-task trials and the decision-point logs.CC BY-NC 4.0Hugging Face
JevAgentHarnessCodeThe routing proxy, the benchmark scripts, and in research/ the code behind every experiment in the paper.MITGitHub
Citation

Cite this work

@misc{zhu2026jevagentharnessbenchmarkingdeployingtraining,
      title={JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs},
      author={Kunlun Zhu and Tao Feng and Xu Han and Heng Wang and Ling Yang and Xiangru Tang and Jiaxuan You and Heng Ji},
      year={2026},
      eprint={2610.21412},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2610.21412},
}