Enterprise-Bench: Measuring AI Agent Reliability at Production Scale

Enterprise AI agents are proliferating faster than the methods to evaluate them. Benchmarks such as SWE-Bench, Terminal-Bench, Agents' Last Exam, and TAU-Bench have advanced the understanding of agent capability on genuinely difficult problems, establishing principles that matter: use real-world artifacts, measure consistency rather than peak performance, and verify state rather than text.

The paper argues that these frameworks nonetheless share three structural characteristics that limit their use for enterprise deployment decisions. They operate within bounded, self-contained data environments; they evaluate with binary pass/fail metrics; and they test at a single, fixed problem size. These are described as boundaries of scope rather than flaws in design.

Enterprise work is structurally different: an agent must navigate heterogeneous data across CRM, engineering tools, knowledge bases and communication records, resolve entity identity across systems, and cope with a signal-to-noise ratio that shifts as operational history accumulates.

The paper's most analytically important design decision is the treatment of cross-system tasks that are deterministic in nature. Consider joining each open P1 support ticket to its product component, to any related open engineering issue, and to the customer's account and ARR. The task spans three systems and a multi-hop join, yet every operation is a deterministic lookup with one correct answer. Under the framework this is Level 1 - specifically, wide L1.

This classification separates two independent failure modes. Cognitive failure is the inability to reason through a problem, an L2+ limitation addressed by a better model or prompting. Architectural failure is the inability to reliably access or integrate data across systems, a wide-L1 limitation addressed by a better data-access strategy. Conflating them, the paper argues, leads to improving the model when the data architecture is the bottleneck.

The authors report that wide-L1 tasks produced the sharpest differentiation between agent architectures in their runs - governed data access reaching 90-100% while API tool-chains reached 0-50% on the same tasks - with the gap attributed to architecture rather than cognitive demand.

Enterprise-Bench models a mid-market B2B payments platform - API-driven infrastructure for payment processing, billing, subscription management and fraud detection. The paper chooses this archetype deliberately: in a B2B platform every team's data connects to revenue, so answering one question correctly often requires data from multiple functions, which is the structural pattern the benchmark is built to test.

Following the TPC-C principle of modeling specific scenarios that generalize, the payments vertical is described as 'the skin' while the underlying relationships - accounts, tickets, issues, components, opportunities - are presented as universal across any B2B enterprise. The 1x seed environment comprises 42 accounts, 40 product parts in a hierarchy, five connected systems, 128 tickets, 33 issues, 34 opportunities and 61 users, with cross-system links maintained through shared product-component identifiers.

Scaling adds only noise records - solved tickets, closed issues, closed opportunities - that carry terminal states excludable by any correct query, so the correct answer is preserved while the retrieval surface grows.

When an agent fails an enterprise task, the paper notes two possible explanations: it could not reason through the problem, or it could not operate the interface through which it reaches data. Without controlling for interface complexity, these are indistinguishable. Enterprise-Bench introduces tool-interface complexity as an explicitly controlled variable, running the same agent, data and questions across tiers of increasing realism.

Tier 1 (MCP1) provides curated, high-level helper tools with clean, predictable responses, approximating the interfaces most agent benchmarks offer. Tier 2 (MCP2) provides protocol-realistic replicas of production API surfaces - Jira REST with JQL, Salesforce with SOQL, Google Drive with bearer-token auth, pagination and nested payloads. Tier 3, genuine production vendor MCP servers, is characterized from public evidence rather than directly benchmarked.

The paper reports that this interface effect was independently corroborated by eight external research sources documenting degradation and token inflation under realistic API complexity.

A consistent pattern is reported across the runs: agents perform comparably on simple single-step questions and diverge sharply on multi-hop joins and cross-source synthesis. Simple lookups - ticket counts, SLA lookups, account summaries - reach 90-100% regardless of data-access approach, so they do not discriminate between architectures.

Multi-hop joins expose the divide. Trace-level analysis attributes failures to a specific mechanism: the agent retrieves all necessary data successfully, then mis-joins it during synthesis, collapsing precise field-level relationships into coarse hierarchical groupings - a 'Cartesian shortcut' that pairs every issue with every account in a product area rather than matching on the specific shared component. The paper reports that this is not a context-window limitation and that it worsens with scale.

The architectural implication offered is that an agent whose retrieval architecture enforces join precision through a query engine cannot fire this shortcut, while an agent that must maintain join precision through multi-step reasoning fires it with increasing probability as data grows.

The current release evaluates L1 and L2 across a 14-task suite spanning three persona clusters - Engineering, Support, and Sales/Operations. Each task is run at five data scales (1x through 256x) with ten independent trials per scale, producing 700 evaluation points per agent configuration. The headline metric is pass^k, the probability that an agent succeeds on all k consecutive trials, following the TAU-Bench formulation.

The paper frames the deliberately small task count as a design choice: statistical power comes from measurement depth per task - reliability, scale-sensitivity and cost trajectory - rather than task breadth, citing Item Response Theory and MLPerf as precedent that a few high-discrimination items can measure capability more precisely than many non-discriminative ones.

Acknowledged limitations include a single synthetic enterprise, mid-market scale, observational-only safety, LLM-judge non-determinism, and L1-L2 coverage, with L3/L4, adversarial noise layers and additional verticals named as future work.