IBM Research Details CUGA Open-Source Agent Harness
A new evaluation by IBM Research demonstrates how changing an AI agent harness can dramatically alter task completion rates while highlighting the specific enterprise benefits of the open-source CUGA framework.
Radwan AbdulhadiEditor and technology writer
Evaluating Agent Harnesses
With nearly 200 agent harnesses currently available to developers, industry analysis increasingly points to the harness as a critical factor in AI performance alongside the underlying model. Community lists track hundreds of these tools, demonstrating that changing a harness can substantially boost performance on complex benchmarks like SWE-bench Pro. Further details are available from Hugging Face Blog in the original source material.
To better understand these differences, IBM Research conducted a controlled experiment comparing three distinct harnesses—CUGA, Hermes from Nous Research, and Deep Agents from LangChain—using the exact same open-weight model and a standardized set of tasks. Further details are available from Back to Articles in the original source material.
AppWorld Benchmark Experiment
The evaluation utilized AppWorld, a benchmark comprising multi-step tasks across nine consumer applications and 457 APIs, such as email, file storage, and payment platforms. Every harness ran the GLM-5.3-flash open-weight model across 90 tasks in 30 scenarios, executing a single attempt per task under the Harbor evaluation framework.
While Hermes and Deep Agents are optimized for different use cases—such as always-on personal assistants or long multi-step research tasks—the AppWorld comparison tested multi-app API execution capabilities. The results showed that task completion rates fluctuated widely based entirely on the harness used, with consistency metrics further highlighting performance variations across scenarios.
Enterprise Focus and Runtime Enforcement
Enterprise environments require agents to interact with critical business systems like CRM and payment portals while adhering to strict governance rules. While teams often attempt to enforce rules through system prompts, instructions can be misinterpreted or bypassed. CUGA addresses this by moving business rules out of prompts and into the runtime harness.
The framework's policy layer enforces guidelines at multiple execution points, incorporating features like tool guards and payment approvals that operate independently of model decisions. Additional details regarding this governance model have been discussed in academic literature, including an ACM CAIS paper detailing governance by construction for generalist agents.
Deployment and Capabilities
Licensed under Apache 2.0, CUGA is designed to operate with any OpenAI-compatible endpoint. It can be deployed locally, on Kubernetes within private or sovereign clouds, or inside air-gapped sovereign clusters equipped with OIDC sign-in and separated chat and management roles.
Furthermore, IBM Sovereign Core builds upon the CUGA framework to enable fully contained agent execution within customer boundaries. The system also supports measurable cost controls via spending budgets and detailed run receipts tracking tokens, time, and tool calls.
Choosing the Right Harness
Industry guidance suggests matching the harness carefully to the intended workload. Terminal coding tasks benefit from tools like opencode, Codex, or Claude Code, while personal assistants rely on OpenClaw or Hermes.
For enterprise business automation across APIs, web UIs, and applications under strict enforceable rules, CUGA provides an Apache 2.0 open-source option built on top of LangGraph. Further information and community discussions regarding these harness comparisons can be explored through professional contributor profiles.
Sources
- Hugging Face BlogSame Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?
Continue chronologically
Related stories
Agentic Retrieval with LangChain and Amazon Bedrock
Explore building Retrieval Augmented Generation applications using LangChain and Amazon Bedrock Managed Knowledge Base for complex multi-part questions.
How Quantum Computers Stay Colder Than Deep Space
Explore how extreme cooling technologies, dilution refrigerators, and innovative designs keep quantum computers running near absolute zero.
NVIDIA: Security Requires Engineering for AI Agents
NVIDIA emphasizes a rigorous engineering approach to AI security, advocating for enforceable controls, agent identity, and standardized testing across the stack.
IBM Granite Time-Series Models Gain Ensemble Forecasting
IBM Research introduces ensemble forecasting for Granite time-series models, combining four complementary architectures with no retraining required.
