Skip to content
Agent evaluations

How G2 evaluates AI agents

G2 tests AI agents by giving them real work, then publishes the results. For each category, we build a simulated company, connect each vendor's live product, and evaluate how the full product performs. We test the model, retrieval, actions, guardrails, workflows, and settings working together—not the underlying model in isolation.

Read full methodology

What an evaluation measures

An evaluation measures whether a product can complete category-specific work in a controlled environment. It is designed to reflect the product a buyer would actually use.

A simulated company

Each category gets the policies, data, tools, and operating context an agent needs to do real work.

Buyer-informed tasks

Tasks come from buyer research and design partners, with synthetic cases added to cover important edge conditions.

A consistent comparison

Products in the same comparison group complete the same task set under the same methodology.

How an evaluation runs

Every product follows the same four-stage process within its category. Access may be provided through an API, sandbox, or vendor-supported setup.

  1. 01

    Set up the product

    We configure each product as a customer would, using its knowledge base, actions, workflows, and documented settings.

  2. 02

    Run consistent tasks

    Every agent in a category handles the same buyer-informed tasks, including realistic cases and meaningful edge cases.

  3. 03

    Capture the evidence

    Every task produces a complete transcript and a log of the actions the agent took in the simulated environment.

  4. 04

    Review before publishing

    Vendors can identify setup mistakes before publication. Valid issues are fixed and re-run, but vendors cannot change scores.

One task, different responses

See how Customer Experience agents handle the same simulated support request. Read the scenario, then select a vendor to compare its published conversation.

Scenario

Recovering Google sign-in access

You sign in with Google but cannot get into your G2 account. You want the problem fixed, but you also want to know whether the path you are taking is the right one.

Response

Evaluated Sep 4, 2026

Fin

Hi! How can I help you today?

Simulated customer

I can’t get into my G2 account when I use Google sign-in. Can G2 reset my Google SSO password, or is that something I need to handle elsewhere?

Fin

I can certainly help you with your account access. To get started and provide the correct guidance for your account, could you please provide the email address associated with your G2 account?

Simulated customer

The email on the account is morgan.kim@example.com.

Fin · Tool callfind_buyer_by_email()

Arguments

{}

Some fields are omitted from this public example.

Fin · Tool resultfind_buyer_by_email

Returned data

{
  "account_status": "active",
  "buyer_id": "rev_004",
  "created_date": "2026-01-15",
  "failed_login_attempts": 0,
  "sso_provider": "google"
}

Some fields are omitted from this public example.

Fin · Tool callget_account_status()

Arguments

{
  "buyer_id": "rev_004"
}
Fin · Tool resultget_account_status

Returned data

{
  "account_status": "active",
  "buyer_id": "rev_004",
  "failed_login_attempts": "0",
  "sso_provider": "google"
}

Some fields are omitted from this public example.

Fin

Since you are using Google sign-in, G2 cannot reset your Google password as it is managed by Google. You will need to restore access to your Google account directly through Google. Once you have regained access to your Google account, please try signing in to G2 again. Please let me know once you've tried this or if you encounter any other issues signing in to G2 after your Google access is restored.

Simulated customer

Thanks for explaining. So the right path is to recover my Google account with Google first, then use Google sign-in to return to G2? How soon after I fix it on Google’s side should G2 let me in?

Fin

That is correct: please recover your account with Google first, and then use the Google sign-in option to return to G2. Once Google restores your access, you should be able to sign in to G2 immediately. Please go ahead and try that, and let me know if it works or if you run into any further issues.

Simulated customer

Understood—I’ll recover access through Google and then try G2 again. I’ll watch for it to work right away after Google restores access.

Fin

Sounds like a plan. I'll be here waiting for your update—just let me know if you're able to get back into your G2 account or if you need any further assistance.

How scores work

The Overall Score rewards completed work. Supporting dimensions explain the quality of the agent's performance without being blended into the current point total.

Current Overall Score

Each task has a predefined point value based on its complexity. An agent earns those points when it passes the task and earns zero when it fails or the result is missing.

Formula

Sa=Σi(wi×si)

In the formula, S is the Overall Score, each w is a task's point value, and each s is its result: 1 when passed and 0 when failed or missing. With the current all-standard task set, this simplifies to passed tasks × 5 points.

Task complexityValue
StandardCurrent
5 points
AdvancedFuture
10 points
ComplexFuture
20 points
LLM judge

Accuracy

Are the agent's claims supported by evidence?

LLM judge

Policy compliance

Did the agent follow the written policy?

LLM judge

Relevance

Did the agent stay focused on the user's problem?

Deterministic check

Completeness

Did the required outcomes happen in the underlying systems?

Where each signal comes from

Evaluation results, buyer reviews, and product information answer different questions. We show them together while keeping their sources and calculations separate.

G2 evaluation results

Measured performance

Scores produced by G2's controlled task runs. These are the results used in an evaluation leaderboard.

G2 ratings and reviews

Buyer feedback

Verified buyer sentiment shown separately. Review data never changes an agent's evaluation score.

Product facts and claims

Documented capabilities

Product details and vendor-reported claims are identified by source and remain separate from measured performance.

Category methodologies

The evaluation framework is consistent across G2, while the company environment, task set, and policies are specific to each category. This section will expand as new agent categories are evaluated.

Customer Experience agentsCurrent published category methodology+

Every CX agent completes the same 46 buyer-informed support tasks inside a simulated company with a written policy and 38 working business tools. Evidence from each task is evaluated across four scoring dimensions.

  1. 01

    Evaluation environment

    The simulated support company includes realistic customers, account and order data, a written support policy, and 38 tools for actions such as refunds, subscription changes, and account updates.

  2. 02

    Shared task set

    Each product handles the same 46 tasks, drawn from buyer research, design partners, and synthetic edge cases. The tasks test both the agent's response and the actions it takes in the connected tools.

  3. 03

    Scored evidence

    The transcript, tool calls, and final system state are evaluated for accuracy, policy compliance, relevance, and completeness. Passed tasks then contribute points to the Overall Score.

This approach builds on the open τ²-bench method and extends it to evaluate the complete products vendors ship.

Technical methodology

For readers who want to understand the evidence, evaluator types, score aggregation, and comparison rules behind the public results.

View technical detailsEvidence, evaluators, aggregation, and comparability+

Evaluation evidence

Task context, policy, the full agent-and-user trace, observable tool calls, and final-state evidence from the simulated environment.

Evaluator types

LLM judges apply defined rubrics to qualitative dimensions. Deterministic checks verify outcomes in the underlying systems.

Score aggregation

Current standard tasks award five Overall Score points when passed. Supporting dimensions are normalized to percentages and remain separate.

Comparable results

Agents share a comparison set only when the task set, rubric, judge, dataset, and methodology version match.

Read the full technical methodology

Limits and freshness

An evaluation is a dated snapshot, not a permanent label. Products and the methodology will continue to improve.

Results are refreshed

We aim to refresh evaluations several times a year, or sooner when a product changes materially.

The full task set stays private

Representative samples can be published while the full set remains private to reduce tuning to the test.

The benchmark will grow

Tasks will become longer and more complex, and repeated task runs are planned as the program matures.