Transforming Video Analysis with Agentic AI

With video intelligence powered by agentic AI, organizations across media, security, insurance, and professional services can ask natural language questions about uploaded videos and receive answers within seconds. As meeting recordings accumulate in shared drives, security cameras capture weeks of unreviewed footage, and field inspection videos sit in object storage, accessing valuable information has traditionally required watching hours of content manually or building custom machine learning pipelines.

A complete implementation of this architecture is available in the companion GitHub repository, providing the blueprint for developers looking to implement runtime orchestration rather than fixed pre-built workflows. The solution uses an agentic architecture that decides at runtime which AWS services to invoke, allowing responses for previously analyzed content to return in under a second.

Orchestrating Services at Runtime

Rather than pre-building a fixed pipeline for each question type, developers can use the Strands Agents SDK to create a single AI agent that dynamically orchestrates tools based on user inquiries. A major media and entertainment company adopted this approach during an AWS Professional Services engagement to query recorded discovery session content and extract design decisions, action items, and stakeholder positions.

According to the customer’s internal before-and-after comparison of analyst hours per recording, this implementation resulted in an approximate 80 percent reduction in manual review time across a backlog of over 200 multi-hour recordings. The system handles diverse queries, from meeting summaries to security footage checks, without requiring separate processing pipelines for each individual use case.

Core AWS Services and Components

The agentic framework relies on several core components. The agent orchestrator acts as the reasoning engine, powered by large language models like Anthropic Claude Sonnet through <a href="https://aws.amazon.com/bedrock/">Amazon Bedrock</a>. It receives natural language queries, determines which tools to invoke, sequences multiple service calls when necessary, and maintains conversation history for follow-up questions.

Visual analysis is handled by <a href="https://aws.amazon.com/rekognition/">Amazon Rekognition</a>, which detects objects, scenes, activities, and faces in video frames when a user's question concerns visual elements. Meanwhile, <a href="https://aws.amazon.com/transcribe/">Amazon Transcribe</a> converts spoken audio to text with automatic language detection and speaker diarization for audio-related queries.

Storage for uploaded videos and cached analysis outputs is managed by <a href="https://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a>, utilizing per-user prefixes to ensure multi-tenant isolation.

Implementation Requirements and Prerequisites

To follow along with the implementation, users require an AWS account with access to <a href="https://aws.amazon.com/bedrock/">Amazon Bedrock</a> configured with Anthropic Claude Sonnet enabled, alongside <a href="https://aws.amazon.com/s3/">Amazon Simple Storage Service (Amazon S3)</a> storage. For document processing, either Amazon Bedrock Data Automation or <a href="https://aws.amazon.com/rekognition/">Amazon Rekognition</a> and <a href="https://aws.amazon.com/transcribe/">Amazon Transcribe</a> are required.

Developers also need Python 3.11 or later with the Strands Agents SDK installed, the AWS Command Line Interface configured with appropriate IAM permissions, and basic familiarity with AI agent concepts such as tool use and reasoning loops.

Inverting the Traditional Video Pipeline

In conventional video analysis applications, developers define a fixed processing pipeline that runs transcription and visual analysis on every video regardless of specific queries. The agentic approach inverts this model by limiting pre-processing to file uploads and allowing the agent to reason about each question independently. For production deployments, AWS recommends adding guardrails to enforce content filtering and grounding checks, particularly for face-matching and surveillance use cases.

Sources

• AWS Machine Learning BlogAgentic conversational video intelligence built on AWS