How G2 compares foundation models with AI CX agents on the leaderboard

G2 agent evaluations for CX measure the outcome of an AI agent doing customer support work. This note covers a second set of runs: foundation models given the same work as the agents, and how we decided which of those models to show by default on the Customer Experience leaderboard (in beta).

What we ran

We ran 14 foundation models from Anthropic, OpenAI and Alibaba through the same evaluation as the CX agents on the leaderboard, most at several effort levels (low, medium, high, xhigh). Each model received the one-page support policy as its instructions, MCP access to the same 38 tools, and worked the same 46 scenarios as the agents. Each model and effort level ran four times; the leaderboard shows the mean across runs. Scoring is unchanged from the agents and is described in my September 22 Build Note.

The models ran without a product around them: no integrations beyond the 38 tools, no guardrails other than the policy text, no human handoff, no vendor configuration. We call this the model baseline.

Which models we show

Buyers asked us whether to build a support agent on a foundation model or buy one. The comparison that answers that question is against a model a support desk would run at volume, so the leaderboard shows two models by default: Claude Haiku 4.5 (default effort) and GPT-4o mini. Both are priced for high-volume use and are the models we see cited most in build-it-yourself conversations.

The selector lets you replace the default with any model we ran, including current frontier models at each effort level. Frontier models score above the top agents on every dimension in our runs. The gap between a frontier model and a cheap one is itself a data point for the build vs. buy decision; their cost per conversation at CX volume is a separate question we do not yet answer on the leaderboard.

Results as of October 9, 2026

What this does not tell you

The comparison doesn’t show what vendors add or the costs. Vendors add integrations to the ticket system and CRM, escalation logic around the model, and edge cases learned from real deployments. We are building more complex scenarios to capture that work in the scores.

Cost per conversation is not yet shown on the leaderboard and we are working on incorporating it.

Next

We plan to publish additional performance insights about token efficiency and time to first token with a select number of vendors.

If you have run a model vs. agent comparison for a support desk, we would like to compare approaches. Comment below or write to agent-evals@g2.com