SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents

Yang Wan1, Zhenhao Zhang2, Jierui Wang1, Xiangxin Zhou3, Linchao Zhu1*
1Zhejiang University, 2University of Rochester, 3Tencent
*Corresponding author
Preprint
Paper arXiv Code Model Benchmark Leaderboard

Abstract

Deciding whether a trajectory actually fulfills its instruction determines how we measure computer-use agents on long-horizon graphical-user-interface tasks and how we train them with reinforcement learning (RL). This judgment has long relied on rule-based evaluation. Rule-based evaluation often disagrees with human intention and becomes outdated when an app updates or its online content drifts. Existing model-based judges attempt to address these problems, but their judging accuracy remains limited. We propose the SeekJudge framework, whose four agents, a Condense, a Ground, a Seek and an Analyze agent, reach a verdict through a Seek–Analyze loop over the trajectory. To train a specialized model on densely labeled trajectories, we propose a seed-driven distillation pipeline that expands a few human-labeled seed trajectories into 10K such trajectories. To evaluate step-level judgments, we build CUAStepBench, a human-annotated benchmark that pairs trajectory verdicts with dense step labels. Beyond accuracy, SeekJudge costs far less than a closed-source large model. We further propose rollout overlap, a reward-server design that reduces the overhead of a reward model in RL training. To our knowledge, SeekJudge is the first model-based reward to match or surpass native rule-based reward in online RL, measured by downstream success rate on held-out RL test goals. In offline judging, SeekJudge-9B also exceeds rule-based evaluation by 9.5 F1 on AgentRewardBench.

Motivation

We revisit what it takes to judge a long trajectory and frame it as the composition of two subtasks: localization, which identifies the image holding the decisive evidence, and extraction, which reads the decisive detail from that image accurately.

We observe that, even when the decisive image is always provided, adding more images from the same trajectory to a single forward pass degrades performance increasingly with their number. Competing content rather than context length drives the drop: as trajectory images accumulate, precision collapses while recall barely moves, so the judge reads accumulating plausible-looking details as evidence of success and the one decisive screenshot is drowned out.

Multi-image information noise: accuracy, precision, and recall as images accumulate.
Padding the input with real trajectory images (blue) degrades the verdict as their number grows, while padding with content-free images of equal token budget (red) does not.

The SeekJudge Framework

SeekJudge realizes this decomposition as a multi-agent framework whose four agents share one backbone model. The Condense and Ground agents first compress the whole trajectory into a compact timeline. The Seek agent takes this timeline as its initial context and runs a Seek–Analyze loop, querying the Analyze agent over several rounds for the evidence it still needs before emitting the final judgment.

The SeekJudge framework: Condense, Ground, Seek, and Analyze agents on one shared backbone.
The SeekJudge framework, where four agents share one backbone model.

Seek agent

The controller and the only agent that carries state across a run. It reads the condensed timeline and judges whether the accumulated evidence settles the instruction. When it is enough, it emits the verdict; when not, it names a single step and image id to inspect and sends a focused question to the Analyze agent.

Analyze agent

Answers one focused question about the single screenshot the Seek agent names and returns the decisive detail as text. It is stateless: every call starts fresh with only that question and image, so it acts as a tool the Seek agent invokes for a detail.

Condense agent

For each step it reads the two screenshots before and after the action and writes a few lines describing the state transition, the T entry of the timeline, turning the whole trajectory into a compact text timeline before the loop begins.

Ground agent

For each step it reads the pre-action screenshot together with the raw action the actor executed, such as click 128,453, and identifies from the screenshot which element those coordinates actually hit, the A entry of the timeline.

Robust to distraction Each call sees few images yet reads them at high fidelity, minimizing interference from other images and observations.
Scales to long trajectories Decoupling localization from extraction keeps each call on a much smaller context, 4–6× smaller than existing reward-model methods.
Adaptive compute The Seek agent issues only as many extraction calls as a case needs, so an easy trajectory settles in a few rounds while a longer one triggers more.

CUAStepBench

To support step-level evaluation we build CUAStepBench, a human-annotated benchmark of 278 tasks over 177 applications. Every trajectory carries a human verdict, and beyond that verdict each step receives a label.

278Annotated tasks
177Applications
3Platforms: Web, OS, Android
StepDense per-step labels

Reinforcement Learning Results

Each reward model drives RL training of an actor across three task domains. Averaged over the six backbone-domain settings, SeekJudge exceeds native rule-based reward by 3.3 points of test success, and among model-based rewards it gives the highest average test success on both actor backbones.

Test success rate (%) of policies trained under each reward model, UI-TARS 1.5 7B actor.
Reward Chrome Impress OS
Rule-based12.7530.4325.56
CUAJudge (GPT-5-mini)13.9133.4826.11
OS-Themis10.1432.4625.00
SeekJudge16.2338.8428.89

Offline Reward Benchmark

Training the SeekJudge-9B specialist sharpens the trajectory verdict and the step-level reading together. On CUAStepBench it lifts trajectory F1 over the Qwen3-VL-8B base and surpasses the closed-source CUAJudge despite its GPT-5-mini calls.

Trajectory- and step-level F1 (%) on CUAStepBench. Step-level metrics are with respect to error steps.
Framework Model Trajectory F1 Step F1
Human expert–82.2–
Large models
CodexGPT-5.577.9–
CUAJudgeGPT-5-mini68.537.9
SeekJudgeDeepSeek-V4-Pro-0423 + Gemini-3.0-Flash74.945.7
Small models
OSRewardOS-Shepherd-9B68.4–
CUAJudgeQwen3-VL-8B66.721.2
OS-ThemisQwen3-VL-8B59.815.5
SeekJudgeQwen3-VL-8B71.327.3
SeekJudgeSeekJudge-9B74.238.0
Lowest inference cost SeekJudge judges at the lowest cost at every image count. CUAJudge incurs extra GPT-5-mini calls, and the agentic Codex baseline costs two orders of magnitude more.
Small, flat context Peak context stays near 12K tokens at 52 images, where OS-Themis reaches ~48K and CUAJudge ~80K.
Denser training signal A continuous score with per-step terms gives a graded reward, improving RL test success over a binary pass-or-fail bit.

BibTeX

@article{wan2026seekjudge,
  author    = {Wan, Yang and Zhang, Zhenhao and Wang, Jierui and Zhou, Xiangxin and Zhu, Linchao},
  title     = {SeekJudge: A Practical Reward Framework for Reinforcement Learning in Computer-Use Agents},
  journal   = {arXiv preprint arXiv:2607.23263},
  year      = {2026},
}