CX Agent Eval Report

Gorgias customer support evaluation

Shopify customer support

Archived evaluation · Oct 13, 2026

Strengths

Relevant responses and successful order changes.

Gorgias earned the highest relevance rating on 35 of 48 responses and passed all five tasks grouped as order changes in this run. In the selected exchange case, it confirmed the customer’s changed request before acting.

Overall, Gorgias passed 30 of 48 Shopify support tasks and earned an average judge score of 80.4%. Policy compliance was inconsistent: 9 of the 30 passing tasks received only 1–2/5 for following required policies.

Inspect the missed safeguard and other cases

Tasks passed: 30 / 48
Task points: 150/ 240
Average judge score: 80.4%
Passed, with low compliance: 9 / 30

49 assigned tasks · 48 scored · 1 infrastructure error excluded. These are simulated support requests from one evaluation run.

Task outcomes

What happened when customers asked for help?

Requests ranged from straightforward returns to changing several items across orders. 18 scored tasks did not pass their outcome checks.

30 passed18 not passed

Primary customer requestPassed / scored

Exchanges: 4/11
Returns & refunds: 6/9
Order changes: 5/5
Address & delivery: 4/5
Payments: 1/3
Lost or damaged items: 2/3
Multiple requests: 8/12

Grouped by the primary request for this report. Each scored task appears once; group sizes vary.

Response quality

Relevance was a strength; policy adherence varied.

35 of 48 responses earned 5/5 for relevance. Policy adherence was less consistent: 12 earned only 1–2/5. In the return case below, the agent submitted the request without the required confirmation.

The averages hide that variation. Here’s the spread of the judge’s ratings across tasks.

Relevance: 92% avg · 48 tasks rated
Accuracy: 86% avg · 48 tasks rated
Policy compliance: 74% avg · 48 tasks rated
Completeness: 85% avg · 22 tasks rated

Judge rating · low to high1/52/53/54/55/5

Three cases behind the scores

(Showing 3 of 48 tasks)

A clean pass, a missed safeguard, and a disagreement between measures. Open a case to inspect the conversation, recorded actions, and grading. Email headers and contact details are omitted.

A changed request, confirmed before acting

Exchanges

Passed

The customer rejected the proposed keyboard variant. Gorgias narrowed the request to the thermostat, asked for confirmation, and passed the task with full marks on every applicable quality dimension.

The Scenario

You wish to exchange the mechanical keyboard for the same one but with clicky switches and the smart thermostat for one compatible with Google Assistant instead of Apple HomeKit. If there is no keyboard that is clicky, RGB backlight, full size, you'd rather only exchange the thermostat.

The Conversation

Simulated customer and Gorgias. Tool calls and results appear in sequence when available. Some details are omitted.

Gorgias

Hi! How can I help you today?

Simulated customer

Hi, I need to make exchanges for two items from my order. I’d like to swap a mechanical keyboard for the same type with clicky switches, and exchange a smart thermostat for one that works with Google Assistant rather than Apple HomeKit. I want to make sure we can take care of both in the same request.

Gorgias

Hi Customer,

I can help with both items from order #3868 in the same exchange request.

Here is the exchange I have ready:

Mechanical Keyboard: from linear / RGB / full size to clicky / RGB / 80%, quantity 1
Smart Thermostat: from Apple HomeKit / black to Google Assistant / black, quantity 1

The keyboard replacement I selected is the available clicky option for this product. The thermostat replacement is also available.

The total price difference will be a refund of $54.21 to your original payment method.

If you would like me to proceed with this exchange and the automatic refund to your original payment method, please reply with your confirmation.

Best,

Simulated customer

Thanks for checking. I only want the keyboard exchanged if it’s clicky, RGB backlit, and full size. Since the available option is 80%, please leave the keyboard as is and proceed with the thermostat exchange to the black Google Assistant-compatible version only.

Gorgias

Hi Customer,

Thanks for clarifying. I have updated the exchange request for order #3868.

Here is the full order context:

Mechanical Keyboard: unchanged
Smart Thermostat: exchange from Apple HomeKit / black to Google Assistant / black, quantity 1
All other items on the order: unchanged

Any price difference will be charged or refunded automatically to your original payment method. For this thermostat exchange, the difference is a refund of $13.46 to your original payment method.

Please reply with an explicit confirmation if you would like me to proceed with this exchange.

Best,

Simulated customer

I confirm that I’d like to proceed with the thermostat exchange only, from the black Apple HomeKit version to the black Google Assistant-compatible version, with the $13.46 refund to my original payment method.

Gorgias · Tool callexchange_delivered_order_items()

Arguments

{
  "order_id": "#3868",
  "item_ids": [
    45358951923755
  ],
  "new_item_ids": [
    45358951858219
  ]
}

Gorgias · Tool resultexchange_delivered_order_items

Returned data

{
  "order_id": "#W2378156",
  "status": "exchange requested",
  "user_id": "yusuf_rossi_9620"
}

Some fields are omitted from this public example.

Gorgias

Hi Customer,

Thanks for confirming. The thermostat exchange for order #3868 has been processed.

Your order is now exchange requested. You will receive an email with instructions on how and where to return the Smart Thermostat, and no new order is needed. The price difference is a $13.46 refund to your original payment method.

Best,

How It Was Graded

Task outcome and response quality are scored separately. A passed task can still contain mistakes.

Relevance: 5/5
Accuracy: 5/5
Policy Compliance: 5/5

View Grading Evidence

Relevance 5/5

relevance: 5 — Agent correctly identified both exchange requests, proposed matching items, adapted precisely when user narrowed to thermostat-only after the keyboard size mismatch, and completed the thermostat exchange with correct refund details. No irrelevant detours.

Completeness Not scored

completeness: N/A — No env_assertions or nl_assertions were available for completeness scoring.

Accuracy 5/5

factual_correctness: 5 — All key factual claims in the agent's customer-facing turns are supported by evidence. The exchange was processed only for the Smart Thermostat (item 4983901480) as confirmed by the tool result and final state snapshot. The $13.46 refund figure matches exchange_price_difference of -13.46 in both the tool result and snapshot. Order #3868 maps to internal id #W2378156 per identifier_aliases. The order status 'exchange requested' matches the final state. The keyboard was correctly left unchanged. No material factual claim is contradicted by the evidence.

Policy Compliance 5/5

policy_compliance: 5 — Agent looked up order details, presented accurate exchange options (keyboard variants, thermostat options, price differences), correctly adjusted scope when customer restricted to thermostat-only, obtained explicit confirmation before executing the write action, made only one tool call, and completed the exchange. No policy rule was violated.

A passed return with a missed safeguard

Returns & refunds

Passed

The return was submitted and the task passed. The judge rated policy compliance 1/5 because the required confirmation step was missing.

The Scenario

You want to know exactly how many tshirt options are available in the online store right now. You want to also return the cleaner, headphone, and smart watch.

The Conversation

Simulated customer and Gorgias. Tool calls and results appear in sequence when available. Some details are omitted.

Gorgias

Hi! How can I help you today?

Simulated customer

Hi, I’d like to know exactly how many T-shirt options are currently available in the online store. I also need to return a cleaner, headphones, and a smartwatch.

Gorgias · Tool callreturn_delivered_order_items()

Arguments

{
  "order_id": "18942873337899",
  "item_ids": [
    45358964015147,
    45358964211755,
    45358957363243
  ]
}

Gorgias · Tool resultreturn_delivered_order_items

Returned data

{
  "order_id": "#W2378156",
  "status": "return requested",
  "user_id": "yusuf_rossi_9620"
}

Some fields are omitted from this public example.

Gorgias

Hello,

There are 10 T-shirt variants currently in stock.

The return for the headphones, vacuum cleaner, and smart watch from order #3873 has been submitted. The order is now marked as return requested, and the return includes those three items. The mechanical keyboard and smart thermostat were not included.

You’ll receive an email with instructions for returning the items.

Best regards

How It Was Graded

Task outcome and response quality are scored separately. A passed task can still contain mistakes.

Relevance: 5/5
Completeness: 5/5
Accuracy: 3/5
Policy Compliance: 1/5

View Grading Evidence

CriterionMet

Agent should tell the user that there are 10 t-shirt options available.

Grader’s Explanation

The agent explicitly states, 'There are 10 T-shirt variants currently in stock,' which directly answers the user's question about the number of T-shirt options available.

Relevance 5/5

relevance: 5 — Agent addressed both user needs directly: answered the T-shirt count question and initiated the return for all three requested items (cleaner, headphones, smartwatch). No detours or irrelevant paths.

Completeness 5/5

completeness: 5.00 — Derived deterministically from env/nl assertion coverage.

Accuracy 3/5

factual_correctness: 3 — The return claim is well-supported: tool result and final state confirm order #W2378156 (alias #3873) has status 'return requested' with items headphones, vacuum cleaner, and smart watch included, mechanical keyboard and smart thermostat excluded. However, the agent claims '10 T-shirt variants currently in stock.' This is a specific numeric fact presented as checked and confirmed. The evidence shows only two T-shirt items across the orders snapshot (item 8349118980 and 3799046073), but the snapshot does not cover the full product catalog. The domain facts note product lookups go through Shopify's native integration without a logged tool call, so absence of a tool call doesn't disprove the claim. Still, '10 T-shirt variants' is a precise, task-critical claim (it was the user's primary question) presented as confirmed, and there is no positive evidence supporting it — only the two T-shirt order line items visible in the snapshot, which are order records not catalog inventory. This is a weakly-grounded, important claim with no supporting evidence, but not directly contradicted by the evidence either. Score 3 reflects important claim with uncertain grounding but no clear proof of error.

Policy Compliance 1/5

policy_compliance: 1 — The agent executed a write action (return_delivered_order_items) without first obtaining explicit customer confirmation. The policy explicitly states: 'Before taking any action that updates the order (cancel, modify, return, exchange), you must list the action details and obtain explicit customer confirmation (yes) to proceed.' The transcript shows the user requested a return in Turn 1, and by Turn 4 the agent had already submitted the return and reported it as completed — with no intervening confirmation step visible. The tool evidence confirms the write action was executed at turn_ref 2, between the user's request and the agent's response, with no confirmation exchange in between. This is a critical safeguard bypass on an irreversible (or at minimum material) state-changing action.

A perfect judge score, but an unmatched outcome

Exchanges

Not Passed

The judge awarded full marks on the applicable dimensions. The outcome checks still marked the task failed: the expected exchange action did not match. The conversation and outcome checks measure different things; this case needs further review.

The Scenario

You want to return the office chair because it came with some broken pieces. But if the agent asks you for confirm, you say you want to rethink for a while, and then change your mind to exchange for the same item. If the agent mentions that the same product is not available, ask for a gray office chair with fixed armrest made of leather instead.

The Conversation

Simulated customer and Gorgias. Tool calls and results appear in sequence when available. Some details are omitted.

Gorgias

Hi! How can I help you today?

Simulated customer

I need to return an office chair—it arrived with broken pieces.

Gorgias

Hello,

I’m sorry the office chair arrived with broken pieces. For order #3921, I have the following item to return:

Office Chair, leather / blue / none, quantity 1

Please confirm that you want to return this item. Once you confirm, the return will be submitted, and any refund will go to the original payment method.

Simulated customer

I want to rethink it for a little while.

Gorgias

Hello,

Of course. Take the time you need. I won’t submit the return unless you confirm that you’d like to proceed.

Simulated customer

Actually, I’d rather exchange it for the same chair.

Gorgias

Hello,

I’m sorry the chair arrived damaged. For order #3921, I can help with an exchange for the same chair.

Here is the complete exchange request:

Original item: Office Chair, leather / blue / none, quantity 1
Replacement item: Office Chair, leather / blue / none, quantity 1

Any price difference will be charged or refunded automatically to the original payment method. In this case, the replacement is priced lower, so a refund of $27.46 would be applied to the original payment method.

Please confirm that you want to proceed with this exchange.

Best,

Simulated customer

Yes, please proceed with the exchange.

Gorgias · Tool callexchange_delivered_order_items()

Arguments

{
  "order_id": "#3921",
  "item_ids": [
    45358956806187
  ],
  "new_item_ids": [
    45358956740651
  ]
}

Gorgias · Tool resultexchange_delivered_order_items

Returned data

{
  "order_id": "#W2890441",
  "status": "exchange requested",
  "user_id": "mei_davis_8935"
}

Some fields are omitted from this public example.

Gorgias

Hello,

I’m sorry this chair arrived damaged, but the exchange has now been processed for order #3921.

Your order is now exchange requested. You will receive an email with instructions on how and where to return the item, and no new order is needed. A refund of $27.46 will be applied automatically to the original payment method.

Best,

How It Was Graded

Task outcome and response quality are scored separately. A passed task can still contain mistakes.

Relevance: 5/5
Accuracy: 5/5
Policy Compliance: 5/5

View Grading Evidence

Relevance 5/5

relevance: 5 — Agent correctly identified the user's need at each stage: initial return request, pause to reconsider, pivot to exchange for same item, and confirmation. Stayed tightly focused throughout with no irrelevant detours.

Completeness Not scored

completeness: N/A — No env_assertions or nl_assertions were available for completeness scoring.

Accuracy 5/5

factual_correctness: 5 — All important factual claims made by the agent are supported by the evidence. The exchange tool call succeeded for order #3921 (alias for #W2890441). The item exchanged was the Office Chair (leather/blue/none). The refund amount of $27.46 matches the exchange_price_difference of -27.46 in the tool result. The order status changed to 'exchange requested', consistent with both the tool result and final state snapshot. The payment method used ([original payment method]) matches the original payment. No factual errors are present.

Policy Compliance 5/5

policy_compliance: 5 — Agent looked up order details (reflected accurate specifics: order #3921, Office Chair leather/blue/none), listed full action details before acting, obtained explicit customer confirmation ('Yes, please proceed'), then executed the exchange tool. No policy rules violated.

Read the two measures together

8 of the 18 failed tasks still received judge scores of at least 80%. The chair-exchange case received a perfect judge score, yet its expected outcome did not match. That disagreement needs further review; a quality rating alone does not establish task success.

How we evaluated

Evaluation scope: Shopify Base · Dataset 1.0.2
Tasks evaluated: 49 assigned · 48 scored · 30 passed
Task scoring: Each passed task earns five points. The 240-point denominator excludes 1 infrastructure error.
Response quality: Scored by an LLM judge on a five-point scale and displayed as percentages. Each dimension uses the tasks where it applies.

These findings apply to this evaluation’s tasks and configuration. Reports using different task sets are assessed individually.

Explore G2’s evaluation methodology