Tackling the Content Selection Challenge

Generative artificial intelligence has drastically reduced the cost and time required to produce large volumes of personalized content. As detailed in the AWS Machine Learning Blog, a previous post showed how generative AI on Amazon Bedrock can generate personalized material at scale while adhering to brand guidelines and guardrails. However, this proliferation creates a new selection challenge: deciding which content variation to show each customer and determining how quickly the system can learn the optimal choice.

To address this hurdle, Amazon Payments deployed a multi-objective contextual multi-armed bandit (MAB) architecture using Amazon SageMaker AI. During a seven-week online A/B test, the system achieved a high single-digit percentage relative lift in final-funnel conversion for a specific customer population. Interestingly, another customer segment experienced no improvement over the existing experience, leading the team to conclude that the content itself, rather than the underlying model, was the constraining factor.

The Role of Multi-Armed Bandits in Personalization

A multi-armed bandit is a reinforcement learning method tailored for environments featuring numerous options and limited traffic. It continually learns which variation performs best while simultaneously serving live customers. By treating each content variation as an individual arm, the algorithm tests options against live traffic and gradually shifts impressions toward high-performing arms while reserving a fraction of traffic for ongoing testing.

This mechanism manages the fundamental trade-off between exploitation—serving the current best arm—and exploration, which tries less-certain arms to gather evidence. Because a bandit continuously performs both tasks, it adapts as new variations are introduced without requiring traditional testing cycles to conclude.

Transitioning from Standard Bandits to Contextual Models

While standard bandit algorithms select a single best arm for an entire audience, true personalization requires conditioning decisions on specific visitor attributes. Contextual bandits accomplish this by evaluating feature vectors representing visitor characteristics. Rather than maintaining separate instances for hand-defined groups, a contextual bandit applies patterns learned in one context to any similar visit.

For its production pipeline, Amazon Payments selected Linear Upper Confidence Bound (LinUCB), a battle-tested method noted for computational efficiency and deterministic arm selection. To understand broader deployment paradigms and deployment strategies on AWS, developers often reference resources covering dynamic A/B testing frameworks for machine learning models.

Optimizing the Entire Funnel and Handling Delayed Feedback

The customer journey in this application comprised three distinct phases: application start, submission, and approval. Because these stages do not move in unison, optimizing a single stage in isolation can degrade another. To overcome this seesaw problem, the team optimized the entire funnel simultaneously by running one LinUCB model per stage and combining their scores through a linear combination.

Handling delayed feedback, such as approval decisions that lag by days, requires careful ledger updates. Developers interested in experimenting with this architecture can explore a code repository provided by AWS, which includes a quick start Jupyter notebook designed to demonstrate the multi-objective contextual bandit method hands-on.

Sources

• AWS Machine Learning BlogUplifting conversion across the acquisition funnel with personalization using contextual bandits on AWS