Training AI Agents with Multi-Turn Reinforcement Learning
A new hands-on guide demonstrates how Multi-Turn Reinforcement Learning on Amazon SageMaker AI optimizes models based on the outcomes of complete multi-step tasks rather than isolated, labeled steps.
Radwan AbdulhadiEditor and technology writer
Introduction to Multi-Turn Reinforcement Learning
Traditional fine-tuning typically teaches an artificial intelligence model to generate a better answer for a single prompt. However, agentic tasks—such as filing expense reports or triaging support tickets—require a sequence of interdependent actions where success can only be judged by the final outcome. Single-turn reinforcement learning and standard supervised fine-tuning often fail to credit decisions made multiple steps prior to a task's conclusion. To address this, Multi-Turn Reinforcement Learning (MTRL) on Amazon SageMaker AI trains models on the definitive signal of a complete multi-step process.
Developers can explore the complete workflow by reviewing the official AWS Machine Learning Blog, which provides deep technical insights into deploying and evaluating these agentic pipelines.
The Wordle Demonstration Architecture
To illustrate the capabilities of MTRL, the guide utilizes Wordle as an objective benchmark environment. In the game, a player attempts to guess a hidden five-letter word within six tries, receiving direct positional feedback after every single guess. Because each guess strictly depends on the previous step's feedback, the game serves as an ideal testbed for multi-step agent behaviors.
While frontier models already play the game effortlessly, developers can apply these exact techniques to train smaller, cost-effective models tailored to proprietary internal business workflows. The foundational architecture splits responsibilities cleanly between Amazon SageMaker AI, which manages the training job and policy model, and Amazon Bedrock AgentCore, which hosts the agent environment, tools, and scoring functions.
Training Pipeline and Optimization Techniques
During training rollouts, the MTRL service invokes the Amazon Bedrock AgentCore runtime with training data containing secret words. An integrated agent calls the policy model through an OpenAI-compatible endpoint, making up to six sequential guesses. Once a game concludes, the agent computes an outcome-dominant reward. Group Relative Policy Optimization (GRPO) then leverages the reward distribution across multiple rollouts to update a Low-Rank Adaptation (LoRA) adapter on the underlying policy model.
Word lists integrated into the agent environment are derived from open-source SCOWL resources strictly for objective scoring purposes. Furthermore, user datasets, custom agents, and trained weights remain strictly within the developer's AWS account and are not used to train the base models.
Serverless Infrastructure and Pricing Dimensions
MTRL operates in a serverless manner where developers select a supported base model and let the service automatically provision the required training and sampling infrastructure. Billing is calculated across three distinct token dimensions: prefill tokens processed by the model, sample tokens generated during reasoning and execution, and train tokens processed during backward passes to update the LoRA adapter.
Because training costs scale directly with trajectory lengths and reasoning verbosity, optimizing game length helps manage expenditures. Detailed cost breakdowns, current billing rates, and complete pricing tiers can be reviewed directly through Amazon SageMaker AI pricing documentation.
Prerequisites and Implementation Resources
Deploying a multi-turn reinforcement learning pipeline requires specific toolsets, including Python 3.10 or later, Node 20 or later, the AgentCore CLI, and botocore version 1.43.37 or newer. Administrators must also configure appropriate IAM roles—such as the agent runtime role and the training job role—alongside an Amazon S3 bucket situated in the exact same AWS Region as the agent.
Developers looking to test the complete end-to-end pipeline can access all implementation scripts, configuration templates, and deployment guides via the dedicated aws-samples repository.
Sources
- AWS Machine Learning BlogMulti-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide
Continue chronologically
Related entity coverage
Related stories
Stream Low-Latency Speech With Kokoro TTS on Amazon SageMaker AI
Learn how to stream low-latency speech from AI voice agents using the Kokoro-82M text-to-speech model hosted on Amazon SageMaker AI.
Migrating Multi-Model AI Agents to Amazon Bedrock AgentCore
AWS introduces a streamlined path to migrate multi-model AI agents from self-managed ECS to Amazon Bedrock AgentCore, reducing operational complexity.
Fine-Tune Search Agents with Multi-Turn RL on SageMaker AI
Learn how to fine-tune LLM-powered search agents using multi-turn reinforcement learning on Amazon SageMaker AI for improved retrieval quality.
Build Efficient Agents Using Sequence-Level Knowledge Distillation
Learn how to transfer frontier model behavior to lower-cost student models using sequence-level knowledge distillation on Amazon SageMaker AI.
