Microsoft Details ThinkingBox RL Training for Agents
Microsoft has released a follow-up to its evaluation work, showing how executable database state checks can serve as reinforcement learning rewards to post-train agents.
Radwan AbdulhadiEditor and technology writer
From Database Grading to Reinforcement Learning Rewards
As a follow-up to initial findings regarding agent evaluation, the Microsoft team has made ThinkingBox-Training available on GitHub. This work builds upon the core concept that an agent might claim a task is complete while the underlying database reveals a different reality. Additional details and background can be found on the Hugging Face Blog.
The core methodology relies on ThinkingBox to evaluate an agent's work by comparing the terminal database state against expected outcomes. Because these executable checks return a clear pass or fail signal directly from the database, the system acts as more than a simple grader. It functions as a viable reward function for post-training agent tasks.
Offline Policy-Relative Difficulty and Task Selection
When constructing a training pool from format tasks outside of the benchmark, researchers faced the challenge of filtering out tasks that were either already solved or completely hopeless for the model. To address this, the team utilized offline policy-relative difficulty. A task's success probability was estimated once using the base policy prior to training, and the selected training pool remained fixed throughout the run.
This carefully selected pool enabled the fine-tuning of Qwen3.8-27B using GRPO for 50 steps. Evaluators tracked performance using multiple metrics, including pass@1 for average success across all attempts and observed 20/20 to measure the share of tasks passing every recorded attempt.
Sharpening Repeatability and Reducing Low-Probability Paths
Training sharply increased repeatable success, accompanied by a small decline in at-least-once coverage. This pattern indicates a process of sharpening, where the probability distribution shifts toward trajectories that work repeatedly while low-probability paths disappear. For example, in corporate hotel-booking scenarios involving room downgrades, the base model frequently changed bookings without required manager approval, whereas the RL fine-tuned model consistently preserved booking and payment states across all repeated attempts.
Further exploration of benchmark datasets and evaluation methods can be reviewed through the dedicated benchmark resources available on Hugging Face.
Evaluating Out-of-Distribution Performance
Following the training phase, the selected step-50 checkpoint of Qwen3.8-27B was evaluated on the 507-task benchmark. These tasks formed an out-of-distribution test set encompassing domains, databases, and Model Context Protocol tools not encountered during the reinforcement learning training phase.
The results highlighted a substantial improvement in reliability. The 27B model initially completed only 38 tasks on every attempt, a figure that rose to 107 tasks after training. The findings emphasize that headline accuracy metrics alone do not capture an agent's true production reliability.
Sources
- Hugging Face BlogTurning Database Grading into RL Rewards: Post-Training Agents with ThinkingBox
Continue chronologically
Related entity coverage
Related stories
Building Custom AI Models Using ML Intern
Discover how developers are leveraging HuggingChat with ML Intern to autonomously plan, train, evaluate, and publish custom open-source AI models.
AI Engineer Compiles 32-Bit RISC-V CPU Into Qwen LLM Weights
An AI engineer has successfully compiled a 32-bit RISC-V CPU directly into the weights of the Qwen 3.5 4B large language model, enabling it to run C code and DOOM.
FairStress Benchmark Measures LLM Demographic Fairness
Discover FairStress, a new benchmark designed to evaluate demographic fairness and hidden asymmetries in large language model decision-making.
Hugging Face Skills Automate SageMaker Model Deployments
Hugging Face and AWS release open-source agent skills, letting coding agents automate production-ready model deployments on Amazon SageMaker AI.
