L1deterministic
Static rule-based training
Build a working post-training pipeline when correctness can be checked with deterministic rules.
GSM8KHumanEval
Diagnostic benchmark · arXiv preprint
Can an LLM agent build the loop that improves another model?
Agent² RL-Bench evaluates the full post-training engineering process: diagnose a task, choose a route, write and debug training code, collect feedback, and iterate within a fixed budget.
Wanyi Chen* · Xiao Yang* · Xu Yang* · Tianming Sha* · Qizheng Li · Zhuo Wang · Bowen Xian · Fang Kong · Weiqing Liu† · Jiang Bian
Soochow University · Microsoft Research Asia · Peking University · Stony Brook University · University of Chicago
Time-ordered experiment lineage
best_so_far
scores.json
code revisions
model artifacts
run reports
Illustrative audit view based on the reported ALFWorld RL-oriented run.
The evaluation gap
Writing RL code is easy to claim.
Closing a working RL loop is observable.
Endpoint scores alone hide the engineering route. A model may improve through SFT, preference optimization, trajectory distillation, or genuine online interaction. Agent² RL-Bench keeps the process visible so researchers can study what agents tried, where they failed, and how they adapted.
A diagnostic ladder
Three levels. One increasingly difficult loop.
Each level adds a feedback structure that the previous one does not require, separating static pipeline construction from stateful agentic RL engineering.
L2preference signal
Static judge-based optimization
Optimize against a preference-like evaluator when reward is mediated by a model judge.
AlpacaEval
L3stateful online loop
Interactive rollout–reward–update
Collect trajectories, interpret rewards, update the model, and recover when the environment loop breaks.
ALFWorldWebShopDeepSearchQA
Inside one run
The benchmark evaluates engineering under feedback.
Every run starts from an isolated workspace and a fixed budget. The agent can train and submit repeatedly; diagnosis, repair, and resubmission are part of the measured capability.
01 Isolated workspace
workspace/
description.md
data/read only
base_model/read only
code/
output/
scores.json
02 Iterative agent loop
scalarfeedback
1Inspect
2Train
3Submit
4Revise
03 Grading API
POST
/submit{
"submission_id": 8,
"score": 93.28,
"best": 93.28,
"improvement": 88.43
}
Test data stays outside the workspace. The agent receives a scalar result and best-so-far feedback.
What the benchmark reveals
Capability appears. Reliability does not—yet.
The results expose a gap between producing plausible training code and repeatedly engineering a stable online RL system.
Online RL is possible
baseline4.85
+88.43
best run93.28
An agent closed the ALFWorld loop.
An RL-oriented run combined SFT warm-up with GRPO and online rollouts to produce the strongest reported gain.
01
Supervised routes still dominate.
Under fixed budgets, many successful routes rely on SFT or SFT-initialized composite pipelines.
common best routeSFT / hybrid
rare final-best routeonline RL
02
Interactive tasks separate agent stacks.
Driver and scaffold choices produce much larger observed differences on stateful tasks than on static ones.
03
DeepSearchQA remains structurally hard.
Agents construct diverse pipelines, yet search–retrieve–judge behavior remains difficult to improve through post-training alone.
Process, preserved
Every score has an experimental history.
Runtime recording and post-hoc summarization turn a long agent run into an inspectable research object. The result is a trajectory of decisions—not a single opaque endpoint.
- 01Submission trajectoryScores, validity, timing, and best-so-far state in
scores.json. - 02Code and model lineageTraining scripts, candidate artifacts, and Git-linked revisions when the stack exposes them.
- 03Structured run reportsHuman- and machine-readable summaries of routes, failures, and progress signals.
summary.mdscores.jsongit / revision logschematic
t₀
inspect · task diagnosisread instructions, data, and the base-model result
t₁
train · SFT warm-upbuild the first evaluable candidate model
t₂
interact · online rollout loopcollect trajectories and repair the reward path
t₃
update · GRPO refinementsubmit, compare feedback, and retain the best artifact
Current release
Six tasks across distinct feedback regimes.
The interface remains fixed while each task supplies the evaluator and feedback structure its training regime requires.
LevelTaskSignalPrimary challenge
L1GSM8KExact-match rulesReasoning-focused post-training
L1HumanEvalUnit testsCode generation and execution
L2AlpacaEvalModel judgePreference-oriented optimization
L3ALFWorldEnvironment episodesTrajectory collection and action grounding
L3WebShopStateful task rewardLong-horizon shopping interaction
L3DeepSearchQASearch-and-judgeTool-augmented retrieval trajectories
Build on the benchmark
Study the route, not only the result.
Run an agent, inspect its experimental lineage, or add a new task and evaluator through the plugin interface.
Citation
Agent² RL-Bench
@misc{chen2026agent2rlbench,
title={Agent$^2$ RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?},
author={Wanyi Chen and Xiao Yang and Xu Yang and Tianming Sha and Qizheng Li and Zhuo Wang and Bowen Xian and Fang Kong and Weiqing Liu and Jiang Bian},
year={2026},
eprint={2604.10547},
archivePrefix={arXiv},
primaryClass={cs.AI}
}