Customer Experience Blueprint
Safety checkedFix Wrong AI Agent Answers Before They Recur
Stop the same support mistake from frustrating customers again: trace incorrect Yuma replies to the Handbook, missing order data, failed actions, or bugs, then test a proposed fix before publishing it. A weekly report highlights recurring causes and the next tickets to investigate.
What problem does this solve?
Finds incorrect Yuma AI replies, traces each reply to its cause, and prepares a Handbook fix with a test. It is for teams running Yuma AI support agents and replaces guesswork about which setting or information caused a wrong answer.
How does it work?
- Connect Yuma to your helpdesk and Shopify, and connect Claude to Yuma through the Yuma MCP. Set the ticket search period and investigation count, which default to 7 days and 10 tickets.
- Run prompt 1 to find tickets where the customer indicated Yuma's answer was wrong, including cases where the customer asked for a person, repeated the issue, or gave a low satisfaction rating.
- Run prompt 2 for each ticket to trace the reply to the relevant Handbook page or rule and identify missing or conflicting information, missing order data, a failed action, or a possible Yuma bug. Review the diagnosis before proceeding.
- For setup issues, run prompt 3 to draft a Handbook change and create and run a test against the draft. Review and publish the fix. Run prompt 4 weekly for a report of recurring causes and tickets to investigate next. Output: wrong-answer diagnoses, draft Handbook fixes, ticket-based tests, and a weekly report. Customers continue to receive the published version until a person publishes a fix.
What's the biggest win?
Each wrong answer gets a diagnosis, a drafted fix, and a test that can flag the same mistake after later changes.
What's required to run this?
- Yuma account running on the Handbook: provides the support AI, Handbook, tests, and ticket data.
- Ask Yuma access for the person reviewing diagnoses and fixes. Investigations use Ask Yuma credits.
- Helpdesk and Shopify connected to Yuma: the workflow uses ticket data and Shopify order data.
- Claude connected to Yuma through the Yuma MCP: runs the prompts.
- Prompt 1, find answers that went wrong:
Search every ticket in Yuma from the last [7] days where Yuma replied and then something suggests the answer was wrong: the customer said the answer was wrong or didn't make sense, asked for a person right after Yuma's reply, wrote back about the same issue, or gave a low satisfaction rating.
Return the [10] clearest cases. For each give the ticket ID, the customer's question, Yuma's reply in one line, and what went wrong in one line. Leave out tickets where the customer was simply unhappy with a correct policy answer.
- Prompt 2, explain why Yuma answered that way:
For ticket [ticket ID], explain why Yuma sent that reply.
Tell me:
- what Yuma saw on the ticket (customer message, order data, past conversations)
- which Handbook page or rule it followed, with a link
- what was missing, wrong or in conflict: a missing rule, two rules that disagree, missing order data, or an action that failed
- whether this looks like our setup or a Yuma bug
If it looks like a Yuma bug, say so and stop. Don't propose a Handbook change for it.
- Prompt 3, draft the fix and test it:
For ticket [ticket ID], using the diagnosis from prompt 2:
- Draft the smallest Handbook change that would have produced the right answer. Keep it as a draft and don't publish.
- Turn this ticket into a test on the Handbook page that owns the situation, with the expected outcome in one sentence.
- Run that test and the page's existing tests against the draft, and report what passed and what failed.
If the fix needs a policy decision we haven't made, list it as an open question instead of guessing.
- Prompt 4, weekly wrong-answer report:
For the last 7 days in Yuma, report in under 200 words:
- how many answers went wrong, grouped by cause (missing rule, conflicting rules, missing data, failed action, Yuma bug)
- the Handbook pages behind the most wrong answers
- fixes drafted, fixes published, and fixes still waiting for review
- tests failing right now
- the 3 tickets to investigate next, with ticket IDs
What are the constraints?
The workflow explains Yuma's decisions on a ticket. It does not debug other apps or helpdesk automations. If the cause appears to be a Yuma bug, report it to Yuma rather than changing the Handbook. A diagnosis can be incomplete, so review each one before approving a fix. Keep fixes specific to the case and add a test for each fix. Nothing changes for customers until a person publishes the draft.
Tools in this Blueprint
About This Blueprint
- Industry
- Computer Software
More Blueprints to explore
Send Weekly Product Defect Reports
Catch product defects before scattered support tickets obscure a rising trend. Weekly reports group faults by product and type, compare them with the prior four weeks, and include customer photos and quotes for faster quality investigations.
Urska Blagojevic
GTM Lead at Yuma AI (YC W23)
Onboard a new agency client in Hive from a project template
Avoid a client’s first week starting with unowned tasks or a missing kickoff. The setup shifts template dates, flags weekend deadlines and open owners, then verifies approved updates in Hive before handing over a week-one list and welcome-note draft.
Find the date a project runs out of budget, and the cards that used the most time
Catch a budget overrun before month-end by seeing when logged hours or project costs will run out at the current pace. The breakdown shows the arithmetic, flags the cards consuming the most time, and waits for your approval before posting to Hive.