What AI foundation models cost to run on G2's CX support tasks

The chart shows the judge score against measured model cost for 22 configurations of 12 models. Judge score is the mean of the four dimensions we score on every task: accuracy, policy compliance, relevance, and completeness. Cost is in dollars per one million conversations, where $10,000 equals one cent per conversation. One task in our environment is one simulated customer conversation.

Claude Sonnet 5.5 from Anthropic scored highest, and it was not the most expensive model to get there. Six configurations from Anthropic and Alibaba’s Qwen3.8 Max, served through Fireworks, score within two points of each other, yet the cost of running them differs by about three times. The cheapest of the six, Sonnet 5.5 at low effort, comes within 0.1 point of the best score. Claude Opus 5.5 costs 2.4 times as much as Sonnet 5.5 at the same effort setting, for a slightly lower score. OpenAI’s models trail, with GPT-5.4 mini the best at 94.5%. Every previous-generation model scored below 97%.

Only five of the 22 configurations are not beaten on both score and cost by another, and above 94.5% Sonnet 5.5 is the only one among them. Top accuracy also costs little in absolute terms: moving from GPT-5 mini at low effort (93.7%) to Sonnet 5.5 at low effort (98.5%) adds about 2¢ per conversation.

Best-scoring configuration per model

Scores and costs are means across runs. Prices are the standard-tier list prices published by each provider in October 2026.

Seven models ran at more than one effort or reasoning setting. Between the lowest and highest setting of the same model, the judge score moved by 0.1 to 1.7 points. In two of the seven, Qwen3.8 Max and Claude Sonnet 4.6, the lowest setting scored highest.

Cost moved more. GPT-5 mini at high effort produced 13,102 output tokens per conversation against 2,312 at low effort, and cost four times as much for 0.7 points. Claude Haiku 4.5 with reasoning cost 60% more than its default setting for 1.7 points. Claude Sonnet 5.5 at xhigh cost 27% more than at low for 0.1 points. For Claude Sonnet 4.6 and Claude Sonnet 5, all settings landed within 4% of each other on cost and within 0.7 points on score.

The tasks likely explain this. A support conversation in our environment is a sequence of policy checks, record lookups, and tool calls, with short replies to the customer. More thinking per turn does not appear to change what the agent looks up or which action it takes. With two to five runs per configuration, differences of under one point should not be read as a ranking of effort settings.

Newer models scored higher and cost less to run. Across four releases of Anthropic’s Sonnet line, from Claude Sonnet 4.5 to Claude Sonnet 5.5, the judge score rose by about six points while the cost per conversation fell by 27% to 43%, depending on the effort setting. The gains were not even: Sonnet 5 improved on Sonnet 4.6 by less than a point, which is within the run-to-run noise, while Sonnet 5.5 added about two more.

OpenAI’s models show the same pattern. GPT-4o, a 2024 model, costs 1.8 times as much as Claude Sonnet 5.5 at its top setting and scores 13 points lower. The newer GPT-5.4 mini scores nine points higher than GPT-4o at a third of the cost. The cheapest model we ran, GPT-4o mini, is also the lowest scoring.

Vendors on the leaderboard see the same pattern. Tidio's Lyro agent is built on Claude, per Anthropic's case study.

"We've been using the Sonnet line of models basically since Lyro was released globally. They keep releasing new models. They keep the price the same. So we're able to not change the economics while getting the benefits of having the stronger capabilities."

Each conversation consumed between 43,590 and 101,679 tokens, of which 1% to 17% were output tokens. The rest is the policy, the tool definitions, and the conversation so far, resent to the model on every turn. Cached-input pricing therefore sets most of the bill. Across the providers in this run, a cached input token is priced 50% to 95% below an uncached one, depending on the model. GPT-4o's effective input price in our runs came to about $1.33 per million tokens; Claude Sonnet 5.5's came to about $0.17. That difference, more than output pricing, is why a 2024 model costs more per conversation than the best current one.

Token counts are not comparable across providers because tokenizers differ, which is why this page reports dollars rather than tokens. Cost per conversation also depends on the harness: a longer policy, more tools, or longer conversations raise the input tokens on every turn.

At the most expensive configuration we ran, the model cost 8.7¢ per conversation. At the top-scoring configuration, 3.4¢. Fin lists its price at $0.99 per outcome, charged once per conversation that ends in a resolution or a procedure handoff. A human-handled ticket costs $5–$10. The model is a few percent of the vendor price and well under 1% of the human cost.

The differences between models on this page are real but small in dollars. The gap between the cheapest configuration and the best-scoring one is about $30,000 per million conversations, or 3¢ per conversation. The gap in score is 16 points. For a support desk, the decision is about quality: which model follows the policy and completes the action. The model bill does not change the answer.

Buyers building their own agent still want the number, because token usage is the part of a build they cannot see in advance.

"What we didn't have a good handle on is what was going to be the type of token usage. And that's something that we're measuring pretty aggressively right now."

Time per task is not on this page. We record latency for every run and plan to report it with the next refresh.

Vendor product cost is not comparable with these figures. Vendors price per resolution, per conversation, or per seat, and the model is one input among integrations, escalation logic, and support. We do not derive a vendor's model cost from its price.

This run covers Anthropic, OpenAI, and Alibaba models for which we have token-level usage.

Prices are standard-tier list prices in October 2026. Batch, enterprise, and committed-use discounts are not included. Prompt-cache behavior depends on the harness, so another harness would see different effective prices.

The leaderboard ranks agents by points earned on tasks; this page uses the judge score, the mean of the four scored dimensions, because it is the measure we have for every configuration. Frontier models score above the top agents on every dimension in our runs, as reported in the build note.

GPT-5 mini ran on dataset version 1.0.6; every other configuration ran on 1.0.7. Each configuration ran two to five times, and figures are means across runs.

The tasks are Tier 1 support scope. Longer tasks with more tool calls will likely raise the cost per conversation for every model, and we expect the differences between models to grow as tasks get harder.

Use the frontier as the starting point for a build, then measure token usage in your own harness. The policy length, the number of tools, and the length of a conversation set the input tokens, and the input tokens set the bill.