This is the follow up to our first blog post, The Agent Said It Was Done. The Database Disagreed. ThinkingBox-Training is now available on GitHub.

We built ThinkingBox to evaluate an agent's work by comparing the final database state with the expected outcome. We then used the same kind of executable signal as a reward on training tasks outside ThinkingBox-Bench. For Qwen3.8-27B, average success rose from 51.70% to 60.78%, while tasks completed correctly on all twenty attempts rose from 38 to 107 with 50 GRPO steps.

Figure 1. ThinkingBox turns executable checks of terminal database state into rewards for post-training agents on tasks outside ThinkingBox-Bench.

This is a joint blog by Microsoft and Hugging Face, special thanks to Mirco Milletari and Xiang Jiang from Microsoft, Sergio Paniego from Hugging Face and our former intern Zhuochun Li (University of Pittsburgh) for co-authoring/reviewing efforts.

We built ThinkingBox to catch agents that say they are done when they are not.

Each of its 507 tasks in ThinkingBox-Bench (our benchmark dataset) hands an agent a real backend and a set of tools, lets it work, then checks what really changed in the database. In the first post, we caught a model closing a customer's ticket as resolved while the carrier exception behind it was still open. Nine competent-looking tool calls, one wrong database row.

Then we noticed what we had built. For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.⁺

So we constructed a training pool from ThinkingBox-format tasks outside the benchmark and used their executable outcomes as rewards. The catch was choosing which tasks are suitable for agent training: most tasks are either already solved or hopeless for the model in front of you, and both teach it nothing.

Difficulty is policy-relative: a task is hard for the current model only until that model learns it.

We call this offline policy-relative difficulty. We estimated each task's success probability once using the base policy before training and kept the selected training pool fixed throughout the run.

What happened after training

We used offline policy-relative difficulty to build the training pool, then fine-tuned Qwen3.8-27B with GRPO for 50 steps. To recap, we evaluate each of the 507 benchmark tasks 20 times. pass@1 measures average success across all 10,140 attempts. pass@20 measures the share of tasks solved at least once. Observed 20/20 measures the share of tasks that pass every recorded attempt.

Table 1. Qwen3.8-27B results before and after 50 steps of GRPO on the offline-selected training pool.

Training sharply increased repeatable success, with a small decline in at-least-once coverage. The rest of the post explains how we selected practice tasks, built the reward loop, and measured that tradeoff.

What does that improvement look like on one task? A traveler requests a room downgrade, but company policy requires manager approval first. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After RL fine-tuning, it preserved the booking and payment state in all 20.

Figure 2. One corporate hotel-booking task, 20 attempts per model. The base Qwen3.8-27B model changed the booking without required manager approval in 15 attempts. The RL fine-tuned model preserved booking and payment state in all 20; all expected records matched in 19.

Contents

1. What the benchmark found, in sixty seconds
2. Results: the dependability column moved
3. Where the gains landed
4. What dependability costs
5. The training loop
6. Offline policy-relative difficulty
7. What earns a reward
8. Run the training stack
9. Where this goes next

In a hurry? The headline result is above. The implementation is in Run the training stack.

What the benchmark found, in sixty seconds

ThinkingBox-Bench grades agents on the database records they leave behind. Each of its 507 tasks starts a real backend, hands the agent a set of MCP tools and a simulated LLM user with private context, and at the end runs executable checks against the terminal database state. Checks are against the backend record that needs to be correct, not the specific tool calls. This incentivizes the model to discover possibly new solution paths rather than “rediscover” the human-backed solution, that might be suboptimal albeit correct.

In our first post we ran eighteen models through that benchmark twenty times each and reported a gap that headline accuracy hides. Kimi-K3 solved 93.89% of tasks at least once but completed only 68 of 507 tasks (13.41%) on all twenty recorded attempts. Claude Opus 5 inverted it: narrower coverage, far more consistency.

And bigger or better model did not close it. Claude Opus 5.5 scored 0.66 percentage points higher than Claude Opus 5 on pass@1, yet passed exactly the same number of tasks, 241, on all twenty attempts. That gain in headline accuracy bought zero additional dependability.

If you are choosing a model for work that touches production database records, pass@1 should not be read alone. Observed 20/20 shows which tasks succeed on every attempt, which is closer to what production reliability requires.

The executable terminal-state verdict was also the training reward. Before unpacking the training loop, here is what changed on the sealed benchmark.

Results: the dependability column moved

After training Qwen3.8-27B, we evaluated the selected step-50 checkpoint on the 507-task benchmark it had never trained on. The 507 tasks represent an out-of-distribution test set: their domains, databases, and most MCP tools were not encountered during RL training.

Table 2. ThinkingBox-Bench pass@1 (%) by domain, for selected models from our first blog's Table 2 and our fine-tuned model for different domains and overall. Observed 20/20 numbers are over the entire benchmark with 507 tasks. Each model is evaluated on all 507 tasks for 20 repeated trials. Bold marks the column leader; underline marks the runner-up.

The headline result is the observed-20/20 row in Table 1. A 27B model that initially completed 38 (7.5%) of ThinkingBox-Bench on every attempt now completes 107 (21.1%) on every attempt, almost thrice. For comparison, observed 20/20 for Kimi-K3 is 68, GPT-5.4 with 128, GPT-6 Astra has 231, and Claude Opus 5.5 leads with 241.

Average success and repeatability produce different orderings. In Table 2, we notice that GPT-5.6 Sol scored slightly higher on pass@1 than the RL fine-tuned Qwen3.8-27B (61.91% versus 60.78%) but passed fewer tasks on all 20 recorded attempts: 82 versus 107. Claude Sonnet 4.6 scored lower on pass@1, at 59.19%, yet passed nearly the same number of tasks every time, at 102. Neither metric alone captures the full picture.

The observed pattern is consistent with sharpening: repeatable success rose while at-least-once coverage fell slightly. This is in-line with prior work. One explanation is that probability distribution moved toward trajectories that work repeatedly while some low-probability paths disappeared.

Overall, the model did not become broader. It became more repeatable. The next question is where those gains landed.

Where the gains landed

Figure 3. Domain-level base scores, RLFT scores, and percentage-point changes. Green change cells show gains and red cells show declines; blue shading indicates the score level within each metric.

Pass@1 and observed 20/20 rose in all five domains. Pass@20 was mixed: it increased in neobank and consulting, but declined in retail, travel, and auto insurance. Auto insurance had the largest breadth loss (−7.0 points), while retail had the largest observed-20/20 gain (+24.5 points).

What dependability costs

The first post priced consistency using simple math: divide the estimated cost of the full 20-run campaign by the number of tasks the model passed every single time.

Cost per dependable task = estimated cost of the 507 × 20 campaign ÷ tasks passing 20/20

Applying the same cost math to the fine-tuned checkpoint's measured token usage:

Table 3. Estimated evaluation cost and dependability for base and RL fine-tuned Qwen3.8-27B. Cost per successful attempt divides total estimated campaign cost (507 x 20) by the number of passing attempts.

At $8.00 per dependable task, the fine-tuned model came close to GPT-5.4 ($6.80), GPT-6 Astra ($7.45), and Claude Opus 5.5 ($7.80) in the companion comparison.

Read the full original article:

HuggingFace Blog