Methodology
How G2 and Bluejay evaluate voice AI agent platforms: the environment every platform runs in, how the tasks were built, the exact definitions of task success, word error rate and latency, how models are chosen, and how the evaluation is kept fair.
Version 0.4, 27 Oct 2026 (manifest v5)
1Overview
This document specifies how G2 and Bluejay evaluate voice AI agent platforms. The unit of evaluation is the platform: the hosted runtime that answers a call, decides when the caller has finished speaking, executes tool calls, and connects the speech-to-text, LLM and text-to-speech layers. The agent itself is held constant. Every platform hosts the same system prompt, greeting and tool schemas, takes the same simulated calls, and is graded by the same deterministic verifier, so a difference in outcome is a difference in how platforms run an agent.
The task suite and its environments are derived from the methodology described in MIVAS-Bench, Bluejay's benchmark of multi-step, tool-using voice agents.
1.1How platforms are graded
Each platform takes the same 630 simulated calls: 90 tasks run five times each, plus 36 copies of easy tasks under background noise or a degraded phone line, also run five times. Every call is graded by software, not by a person or a language model, and the results add up to a platform score out of 100:
80points
Task success
Did the agent do the job? A call passes only if our tool server's log shows every required tool called with the right details, and the business's database after the call matches the expected result. No partial credit.
80 × passes ÷ (450 − calls failed by a model we chose)
10points
Word error rate
Did the platform hear the caller? Its own transcript of the caller is compared with the caller's script: word errors, meaning-changing errors, and the names, dates and numbers it got right, on clean and on degraded audio.
10 × composite C, half clean, half degraded
10points
Latency
How long does the caller wait? Measured on our own recording, on every turn, from the end of the caller's speech to the start of the agent's reply.
Full at ≤ 1.0 s median and ≤ 2.0 s p95; zero at 4.0 s and 6.0 s
Failures that are not the platform's own do not count against it. Calls broken by our side (a task defect, a caller that left its script, a harness fault) are voided, fixed and re-run on every platform, and on platforms where Bluejay chose the speech or language models, failures caused by those models are reported separately rather than scored. §4 gives every formula, §7 how each failure is attributed, and a qualitative profile of each platform (§5) is reported beside the score without entering it.
2Evaluation environment
Every call runs in the same closed environment: a simulated business with its own database and tool API, a simulated caller, and the platform under test between them. Only the platform and its transport change from one run to the next.
Figure 2.1Evaluation environment
2.1Simulated businesses
Each industry is a synthetic company with operating policies, a relational database and a tool API, modeled on a real deployment and on the rules that govern that kind of business:
| Industry | Company | Modeled on | What the model preserves |
|---|---|---|---|
| Healthcare | Straus Dermatology: 160 locations, 380 providers | A deployed multi-agent dermatology front desk | Identity verification before any chart, balance or result is disclosed; plan- and office-dependent coverage; cancellation-fee disclosure before commitment; a clinical desk that reports status but never interprets a result or approves a refill |
| Legal | Halverson & Reed: plaintiff-side injury, employment, workers' compensation and consumer firm | ABA Model Rule 1.18, duties to prospective clients [1] | Conflict screening before any facts are heard; practice-area, jurisdiction and filing-deadline checks before intake; hold, read-back and explicit confirmation before an evaluation is booked; no legal advice or case valuation |
| Customer support | Kestrel Electronics: about 1,000 stores, two membership tiers, delivery, installation and protection plans | The public capability surface of a national electronics retailer's generative-AI support line | Tier- and product-dependent return windows, state-dependent restocking fees, price-match exclusions, cancellation proration, recalls and impersonation reports |
Company identities, people and records are synthetic; the organizational boundaries and workflow constraints are the real ones.
2.2The agent under test
Each company's phone line was designed as a department of specialist agents (reception, identity, scheduling, billing and so on), each with production-length instructions and only the tools its role needs (Figure 2.2). Industry research came first: published agent deployments, operating policies, applicable regulation, call-centre request taxonomies and recurring failure cases from production evaluations, which together define what callers ask for and where an automated agent must defer to a person. The company, its workflows and its authorization gates were then specified from that research, and implemented as prompts, typed tools, a seeded database and server-side policy guards.
Workflow mapping assigned every kind of call to the specialist that owns it and encoded the authorization gates, decision points and escalation routes between them. The result is a different topology per industry: a privacy-gated, acyclic front desk in healthcare; a sequential admissibility funnel in legal; a cyclic service network for compound requests in customer support.
Figure 2.2Agent graphs
7 specialist agents · 35 tools · greeting: “Thank you for calling Straus Dermatology. This is Robin, an AI assistant. How can I help you?”
reception · handoffs: transfer_to_identity, transfer_to_scheduling, transfer_to_coverage, transfer_to_cosmetic
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
list_locations | When to call this tool: the caller asks for an office address, floor, suite, hours, parking, or transit; names an office; or you need an office id before booking or a coverage check |
identity · handoffs: transfer_to_scheduling, transfer_to_billing, transfer_to_clinical, transfer_to_coverage, transfer_to_cosmetic
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
identify_patient | When to call this tool: the caller may already be a patient and you need to find their chart before verify_identity |
verify_identity | When to call this tool: you have the caller's name and date of birth and must confirm they are the patient before any protected chart work |
get_patient_summary | When to call this tool: identity verification just succeeded — load the chart before handing off or starting specialist work |
scheduling · handoffs: none (terminal desk)
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
send_sms | When to call this tool: they asked for an appointment confirmation text after a medical booking, reschedule, or allergy service was just committed |
classify_visit_request | When to call this tool: the caller described a medical, surgical, or allergy reason for a visit and you need the appointment type before searching slots |
list_locations | When to call this tool: the caller asks for an office address, floor, suite, hours, parking, or transit; names an office; or you need an office id before booking or a coverage check |
find_slots | When to call this tool: you are ready to offer appointment times at specific offices |
book_appointment | When to call this tool: the caller said yes after you read back the day, time, office with floor, and provider with credentials for a medical, Mohs, or allergy evaluation visit |
reschedule_appointment | When to call this tool: the caller wants to move an existing appointment — offer this before cancelling |
cancel_appointment | When to call this tool: they still want to cancel after you offered to move the visit instead |
join_waitlist | When to call this tool: they want an earlier opening than the slots you can offer, or they decline to rebook after a cancel |
schedule_allergy_service | When to call this tool: book a dedicated allergy service (skin testing, patch testing, shots, and the like) after an explicit yes |
coverage · handoffs: transfer_to_scheduling
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
list_locations | When to call this tool: the caller asks for an office address, floor, suite, hours, parking, or transit; names an office; or you need an office id before booking or a coverage check |
check_plan_accepted | When to call this tool: the caller asks whether you take a plan at a specific office |
run_eligibility_check | When to call this tool: they want a live copay, deductible, or coinsurance and have given a member ID |
capture_insurance_update | When to call this tool: they have new or changed insurance to put on file |
cosmetic · handoffs: none (terminal desk)
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
quote_cosmetic_service | When to call this tool: they ask what a cosmetic treatment costs |
list_locations | When to call this tool: the caller asks for an office address, floor, suite, hours, parking, or transit; names an office; or you need an office id before booking or a coverage check |
find_slots | When to call this tool: you are ready to offer appointment times at specific offices |
book_cosmetic_consult | When to call this tool: they are ready to book a cosmetic consult |
send_payment_link | When to call this tool: they are ready to pay a medical balance or a cosmetic deposit |
offer_financing | When to call this tool: they cannot pay the full amount at once and the balance or quoted amount is over two hundred fifty dollars |
billing · handoffs: transfer_to_scheduling
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
get_account_balance | When to call this tool: they ask about a bill, balance, or charge — call this first, before explaining or collecting |
explain_charge | When to call this tool: they ask why a specific charge is on the account |
send_payment_link | When to call this tool: they are ready to pay a medical balance or a cosmetic deposit |
offer_financing | When to call this tool: they cannot pay the full amount at once and the balance or quoted amount is over two hundred fifty dollars |
request_fee_waiver | When to call this tool: they dispute a missed-visit fee and want it reviewed |
clinical · handoffs: transfer_to_scheduling
| Tool | What it does |
|---|---|
create_callback_task | When to call this tool: you need to promise a callback with a real tracked SLA — after a tool cannot finish the work, or they need a team to call back |
get_results_status | When to call this tool: they ask about pathology, lab, or allergy testing results |
request_rx_refill | When to call this tool: immediately after identity verification for a named-medication refill that maps to a closed-formulary slug, or refill-related prior authorization, before policy talk, extra details, a clinical message, callback, or handoff |
create_clinical_message | When to call this tool: a nurse, results, or refill question needs the clinical team — including when they push for result content you cannot read |
send_portal_activation | When to call this tool: the portal is not active and they need access — on every results call where it is inactive, before the call ends |
5 specialist agents · 24 tools
reception · handoffs: transfer_to_screening, transfer_to_client_services
| Tool | What it does |
|---|---|
lookup_caller | Find or create the caller record |
get_caller_matters | The caller's existing matters, including whether any is already represented by another firm |
take_message | Take a message for a person at the firm, with a callback promise |
screening · handoffs: transfer_to_intake
| Tool | What it does |
|---|---|
check_conflict | Screen for a conflict of interest |
check_practice_area | Whether the firm handles this matter type and how it charges |
check_jurisdiction | Whether the firm is licensed for this matter in this state |
calculate_filing_deadline | The statute of limitations |
intake · handoffs: transfer_to_scheduling
| Tool | What it does |
|---|---|
record_intake | Log the intake |
add_intake_note | Attach a note to the caller's record for the attorney or case manager |
send_intake_packet | Send the new client packet |
request_records_authorization | Send a medical records release for signature |
scheduling · handoffs: none (terminal desk)
| Tool | What it does |
|---|---|
get_attorney | An attorney's name, practice areas, and bar states (internal routing lookup) |
find_evaluation_slots | Open case evaluation slots |
hold_evaluation | Hold and price a case evaluation |
confirm_evaluation | Finalize the booking |
hold_cancellation | Quote cancelling an appointment |
confirm_cancellation | Finalize the cancellation |
client_services · handoffs: none (terminal desk)
| Tool | What it does |
|---|---|
get_caller_matters | The caller's existing matters, including whether any is already represented by another firm |
get_case_status | Status of one of the caller's own matters at this firm |
take_message | Take a message for a person at the firm, with a callback promise |
add_intake_note | Attach a note to the caller's record for the attorney or case manager |
7 specialist agents · 39 tools
reception · handoffs: transfer_to_verification, transfer_to_fraud
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_store_info | Address, hours and departments for one store |
get_policy | The published policy text, with its numbers: returns, restocking, price_match, membership, delivery_install, open_box, marketplace, recycling |
get_fee | The published fee schedule, including membership pricing |
verification · handoffs: transfer_to_orders, transfer_to_returns, transfer_to_service, transfer_to_membership
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
identify_customer | Find the caller's record |
verify_identity | Verify the caller with the ZIP code on the order and the last four digits of the card it was paid with |
get_customer_summary | The verified caller's name, membership tier, recent orders and any open TechCrew appointments |
orders · handoffs: transfer_to_returns, transfer_to_service, transfer_to_fraud
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_customer_summary | The verified caller's name, membership tier, recent orders and any open TechCrew appointments |
get_order | One order in full: status, what is on it, whether Kestrel or a Marketplace seller sold it, the delivery date and window, whether installation and haul-away are included, the state it was bought in, and safety flags on any item (recalled, battery_safety_flag) |
quote_delivery_change | Price a change to a scheduled delivery or installation |
confirm_delivery_change | Commit the delivery change using the token quote_delivery_change returned |
cancel_order | Cancel an order that has not shipped |
quote_price_match | Test a competitor's price against the Price Match Guarantee and work out the difference |
confirm_price_match | Refund the price difference using the token quote_price_match returned |
get_fee | The published fee schedule, including membership pricing |
returns · handoffs: transfer_to_orders, transfer_to_service, transfer_to_fraud
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_order | One order in full: status, what is on it, whether Kestrel or a Marketplace seller sold it, the delivery date and window, whether installation and haul-away are included, the state it was bought in, and safety flags on any item (recalled, battery_safety_flag) |
check_return_eligibility | The whole return computation for one item: which window applies and why (15 days standard, 60 for Plus and Total members, 14 for activatable devices no matter the tier), how many days are left or how many days over, and the restocking fee given the product class, whether the box was opened, and the state it was bought in |
quote_return | Price the return: refund amount, restocking fee and why it applies, and where the money goes |
confirm_return | Start the return and get an RMA number, using the token quote_return returned |
create_return_label | Email a free prepaid return label for an open RMA |
get_refund_status | Where a refund is: the stage it is at and the date range it posts by |
get_fee | The published fee schedule, including membership pricing |
service · handoffs: transfer_to_returns, transfer_to_membership, transfer_to_fraud
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_order | One order in full: status, what is on it, whether Kestrel or a Marketplace seller sold it, the delivery date and window, whether installation and haul-away are included, the state it was bought in, and safety flags on any item (recalled, battery_safety_flag) |
get_protection_plans | TechCrew Protect plans on this caller's products, with start and end dates and the deductible, plus what their membership tier covers on its own |
check_coverage | Whether a specific failure on a specific item is covered, and who pays: a TechCrew Protect plan (deductible), Kestrel Total (nothing), the manufacturer warranty (nothing, defects only), or nobody (a $39.99 bench diagnostic) |
book_service_appointment | Book TechCrew: bench (in a store), in_home, or remote |
get_service_appointment | Every TechCrew appointment on the account with its date, window and status |
cancel_service_appointment | Cancel a booked TechCrew appointment |
membership · handoffs: transfer_to_service, transfer_to_fraud
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_membership | The caller's membership: tier, what they paid, when it renews, whether auto-renew is on, and how many whole months are unused |
quote_membership_upgrade | Price an upgrade to Kestrel Total, prorated over the months left on the current year |
confirm_membership_upgrade | Charge the upgrade using the token quote_membership_upgrade returned |
quote_membership_cancellation | The cancellation math: the refund for unused whole months, and what the caller gives up |
confirm_membership_cancellation | Cancel the membership using the token quote_membership_cancellation returned |
get_fee | The published fee schedule, including membership pricing |
fraud · handoffs: transfer_to_verification
| Tool | What it does |
|---|---|
search_kb | Look up general Kestrel information: store hours, the legacy brands (Sound Harbor, Bellwether Mobile, Aurelian Audio, Coastline Kitchen & Home, Sagebrush Outdoor), what TechCrew is, how returns work, recycling and trade-in, the difference between a manufacturer warranty and a protection plan, and what a Kestrel scam looks like |
get_fee | The published fee schedule, including membership pricing |
check_subscription_charge | Whether Kestrel actually bills this person the amount they were told about |
check_outbound_contact | Whether Kestrel or TechCrew contacted this person at all |
report_scam_contact | File an impersonation report |
Most platforms run one agent per call, so each department is collapsed into a single agent for this evaluation: the specialists' instructions are merged into one system prompt (2,131 words in legal, 3,475 in healthcare, 3,523 in customer support), handoff tools are removed, and two session tools remain, transfer_to_human and end_call. Every business tool, the database and every expected end state are unchanged. The prompt, greeting and tool schemas are deployed byte-for-byte on every platform.
- [1] American Bar Association, Model Rules of Professional Conduct, Rule 1.18: Duties to Prospective Client.
2.3Tool server and database state
Each industry has one tool server, and it is the only system of record. Every request carries the call's identifier. On the first request of a call the server creates a new SQLite file for that call from the industry's schema and seed, so every call begins from the identical state D0 and no call can observe another. Each subsequent tool call executes against that file, and every request is logged with its tool name, arguments, raw body, response, status and timestamps.
Platforms that run tool code inside their own runtime are configured with functions that forward each call to the same endpoint. At hangup the call's database is frozen. The verifier (§4.2) reads two things: the server's request log, which is what the agent actually did, and the frozen database, which is what it left behind. Neither depends on what the platform reports about itself.
2.4Simulated caller
Each caller is a Bluejay digital human configured from its task: an LLM-driven caller at temperature 0 that improvises phrasing but never facts. Its configuration has three parts. The intent states its goal and how it behaves. The traits are the facts it holds (name, phone, identifiers, dates) and the only values it may give. The scripted responses are trigger-and-line pairs: when the agent's turn matches a trigger, the caller says that line exactly, so every fact a task depends on reaches the agent in the same words on every platform and every run. A scenario clock fixes the date, so date arithmetic is identical across runs. Callers do not barge in.
Caller speech is synthesized by commercial TTS. Voices rotate over a fixed eight-slot list of accent and gender (American, mature and Southern; female and male) assigned by sorted task identifier, so a task has the same voice on every platform and every run, and its audio variants inherit it.
Figure 2.3Caller configuration
T4-H1
Cancel Plus after one save offer
Intent goal and conduct, given to the caller verbatim
- Open with exactly:
"Cancel my Kestrel Plus membership today."
Give identity when asked: whenever the agent asks for your name or phone number, answer with them right away and never answer that question by repeating your cancel request. - If they make one save offer, say exactly:
"No, please cancel it."
Do not accept a second save pitch, a store visit, a letter, or a callback. - Confirm after they read back the $14.99 refund for 6 unused months, ending today.
- Once they have finished what you asked and confirmed it, say exactly: "Thank you, that's all I needed." and end the call.
- Do not start a second request.
Scripted responses when the agent's turn matches, the caller says the line exactly
WhenThe agent asks for the caller's name, a phone number, or both, in any wording, such as the full name and phone number on the membership or on the account, including when the same turn says they will get you to the right place or will help cancel
Not whenwhen asking for ZIP, card last four, or an order number alone
“Selina Cortez, 541-555-0171.”
alwaysWhenThe agent asks for the ZIP code on the order and the last four digits of the card, together or in either order
Not whenwhen asking for a full card number, and; when asking for a name or phone
“ZIP 97035, card ending 1156.”
alwaysWhenThe agent asks for an order number, which order, or which item
Not whenwhen asking for name, phone, ZIP, or card last four
“Order KE-4498870, the speaker.”
alwaysWhenOffers to transfer you to a human, a care advocate, the front desk, or a live agent because they cannot continue on this line
Not whenwhen you already asked for a person; when they are completing the return, delivery change, cancellation, or scam report; when they ask for your name, phone number, ZIP, or card digits, and; when they say they will get you to the right place or move you to verification or membership to handle the cancellation
“No, I don't need a person. Please finish what I asked on this call.”
alwaysShow all 7 scripted responses
WhenHas already finished the original request you called about, confirmed that outcome out loud, and is now offering a second unrelated request, a confirmation text, or asking if you need anything else
Not whenduring identity; while they are still collecting facts or looking something up, and; before they have confirmed the original request is done
“That's all I needed. Thank you.”
alwaysWhenOffers to keep Plus, mentions a discount, or asks you to stay
Not whenwhen reading back the refund to confirm cancel
“No, please cancel it.”
alwaysWhenReads back $14.99 for 6 unused months ending today and asks whether to cancel
Not whenthe save offer
“Yes, cancel it. I acknowledge the fourteen ninety-nine refund ending today.”
alwaysC5-H1
Large balance, cannot pay in full
Intent goal and conduct, given to the caller verbatim
- You are an existing Straus patient.
- You got a bill for several hundred dollars and you cannot pay the whole thing this month, and you also cannot make Friday's cosmetic consult at Park Avenue until the bill is under a plan.
- Answer their identity questions once, confirm the readback, then stay on the bill first.
- Ask what you owe and why the visit charge is there.
- Ask whether there is a payment plan or some way to spread it out.
- Accept the financing they offer and ask for the payment link to be texted to you.
- After the payment link is sent, say you also need Friday's Park Avenue consult moved to a later date, not cancelled.
- Take the first new appointment time they offer.
- If they read that time back and ask whether to move the consult, say yes.
- Then thank them and end the call.
- Do not ask for a callback or to speak to a person.
- Do not ask about a waitlist, the patient portal, insurance, or the street, floor, or suite.
- Do not ask for any text other than the payment link.
- Stay until they have looked up a payment plan or financing in their system, texted the payment link, and confirmed the new consult time on the calendar.
- A spoken 'we have plans' without a lookup is not enough.
- Do not ask them to text you an appointment confirmation.
- Do not mention Friday's consult until they say the payment link was sent.
- Never say you are not making a payment or that they should not send the link.
- Once the new consult time is confirmed, thank them and end the call right away.
Scripted responses when the agent's turn matches, the caller says the line exactly
WhenAsks for a ZIP code, either yours or the office's
Not whenwhen asking for your name, date of birth, or phone number; when reading a ZIP code back to you
“One, zero, zero, one, six.”
alwaysWhenAsks how much the payment link should be for or how much you want to pay today
Not whenwhen asking for a mobile number
“Send the link for the full balance. I'll pay what I can through it, and I still want the financing.”
alwaysWhenAsks for your full name and date of birth, or asks you to confirm who you are, or asks for the patient's name and date of birth
Not whenwhen asking only for a member ID or insurance card; when asking which office or location; when asking for a phone or mobile number; when asking about an appointment day or time; when reading your name and date of birth back to you for confirmation
“Maria Alvarez. My date of birth is June thirtieth, nineteen seventy-two.”
alwaysWhenReads your name and date of birth back to you and asks whether that is right, or asks you to confirm the name and date of birth they just repeated
Not whenwhen first asking for your name and date of birth; when asking for a phone or mobile number; when asking for a ZIP code; when reading back a phone or mobile number
“Yes, that's right.”
alwaysShow all 18 scripted responses
WhenOffers to text a payment link, asks whether you want the payment link by text, or asks which mobile number should receive the payment link
Not whenwhen offering an appointment confirmation, directions, or another unrelated text; when they have already said the payment link was sent; before they have said CareCredit, financing, or a payment plan is available
“Yes, please text the payment link to 212-555-0133.”
alwaysWhenSays goodbye or asks if you need anything else before they have looked up a payment plan or financing
Not whenwhen they have said at any point that CareCredit, financing, or a payment plan is available; when they only say okay, thanks, or one moment; right after confirming your name and date of birth; when asking how much the payment link should be for; when asking for your ZIP code, mobile number, or identity; after the payment link was sent; once CareCredit or financing has been mentioned anywhere earlier in the call; after the consult was moved, and; when they say goodbye after moving the consult
“I still need a way to spread this out. Please look up a payment plan before I go.”
alwaysWhenSays the payment link was sent or texted to your phone
Not whenwhen they have only quoted the balance, explained a charge, or mentioned financing; when they have not yet said a payment link was sent; when the new time is already booked; after they have said the consult was moved or that your consult is now at a new time; when they only repeat that the link was already sent later in the call
“Thanks. I also need Friday's Park Avenue consult moved to a later date, not cancelled. I'll take the first time you offer.”
alwaysWhenOffers one or more concrete appointment dates or times, or asks you to choose among concrete slots
Not whenwhen asking for your name, date of birth, mobile number, or ZIP code; when reading back one specific time and asking whether to move your consult to it
“I'll take the first time you offered.”
alwaysWhenOffers to transfer you to a human, a person, the front desk, or a live agent
Not whenwhen you already asked for a person; when they are completing the booking or cancel
“No, I don't need a person. Please finish what I asked on this call.”
alwaysWhenOffers the waitlist, a later opening, or to add you to a list
Not whenwhen offering a specific appointment time to book now; when asking for your name or date of birth
“No waitlist. Please just finish what I asked.”
alwaysWhenOffers to text you a confirmation, directions, or an appointment SMS
Not whenwhen offering a payment link or deposit link; when asking for a payment-link number; when asking for your mobile number for identity; when sending portal access
“No thanks, I don't need a confirmation text.”
alwaysWhenReads a phone or mobile number back to you and asks whether it is right, or asks you to confirm or repeat the full number
Not whenwhen reading back your name, date of birth, or ZIP code; when first asking which number to use
“My number is 212-555-0133. If that is exactly what you have, yes, please send the payment link.”
alwaysWhenReads back one specific new time and asks whether to move your consult to it, or asks you to confirm the new consult time before moving it
Not whenwhen asking for your name, date of birth, or mobile number; after they have said the consult was moved. This includes 'Should I move it there?' or 'Should I move the consult to that time?' even when the same turn says moving it is free or there is no charge
“Yes, please move my consult to that time.”
alwaysWhenSays your Park Avenue consult HAS BEEN moved or rescheduled, or that your consult is now at a specific new time, as something already done
Not whenwhen only offering or reading back times; when the turn ends by asking whether to move it or should I move it there; when they only say moving it is free or there is no charge to move it; when saying it has not been moved yet
“Thank you, that's everything I needed. Goodbye.”
alwaysWhenTells you the balance amount or explains the visit charge, and has not yet said CareCredit, financing, or a payment plan is available
Not whenwhen asking for your identity
“That's more than I can pay this month. Why is the office visit charge on there, and is there a payment plan or financing to spread it out?”
alwaysWhenSays CareCredit, financing, or a payment plan is available, or asks whether you want to use it, and has not yet said a payment link was sent
Not whenwhen asking for a mobile number; after the payment link was sent; after the consult was moved
“Yes, I'll use the financing. Please also text me the payment link at 212-555-0133.”
alwaysWhenOffers to text a payment link or asks which number should get it, before they have said CareCredit, financing, or a payment plan is available
Not whenafter the payment link was sent
“Yes, text it to 212-555-0133. But first, I can't pay it all at once. Is there a payment plan or financing?”
alwaysWhenAsks what later date or time would work for the consult, whether you have a preferred day, time, or provider, or when you would like to move it, without naming a specific slot
Not whenwhen asking for your name, date of birth, mobile number, or ZIP code; when offering or reading back a specific time; after they have said the consult was moved
“Any later date is fine. Please give me the first opening at Park Avenue.”
alwaysC2-M1
Conflict hit, Northgate Insurance
Intent goal and conduct, given to the caller verbatim
- Open with exactly:
"My insurance company denied my claim."
When asked who this would be against, or which insurance company or insurer it is, say exactly: "It's Northgate Insurance." Never answer that question with only "
okay" or "
sure". - Do not volunteer a state, a date, or a story, and do not ask for a person.
- If they ask which state, when it happened, or to describe what happened, say exactly: "I'd rather not get into that until you check whether you can take it.
" If they offer an appointment or to start a written account, say exactly: "
No, I don't want to book. - I need you to check whether you can take this." Stay until they confirm a person will take it from here.
- A spoken 'we'll look into it' or that the firm cannot take the matter, without transferring you to someone, is not enough.
- When told a person will take it from here, say exactly:
"Okay, thank you."
and end the call.
Scripted responses when the agent's turn matches, the caller says the line exactly
WhenThe agent asks for the caller's full name and a callback or phone number
Not whenwhen asking to repeat, confirm, or read back only the number or some of its digits; when asking about anything else. This includes a question that comes after a filler such as one moment, hold on, let me pull that up, or let's get some quick details in the same turn
“Selma Ridgeway, nine one nine, five five five, zero one four six. Nobody represents me, and it would be against Northgate Insurance.”
alwaysWhenThe agent asks you to repeat, confirm, or say one digit at a time your callback number, reads a number back to check it, says it did not catch the number, or asks for only some of its digits such as the area code, the first three, or the last four
Not whenwhen asking for your name
“Nine, one, nine, five, five, five, zero, one, four, six.”
alwaysWhenThe agent asks whether another lawyer or law firm currently represents the caller, or whether the caller has already hired or signed with anyone
Not whenany other question. This includes a question that comes after a filler such as one moment, hold on, let me pull that up, or let's get some quick details in the same turn
“No, nobody represents me.”
alwaysWhenThe agent asks who the claim would be against, who the other side is, which insurance company or insurer this would be against, or for the name of the insurance company
Not whenwhen asking for the caller's own name or callback number. This includes a question that comes after a filler such as one moment, hold on, let me pull that up, or let's get some quick details in the same turn
“It's Northgate Insurance.”
alwaysShow all 7 scripted responses
WhenThe agent asks which state this happened in, or where it took place
Not whenany other question; when asking for the insurance company's name or who this is against, and; when asking whether a lawyer represents you
“I'd rather not get into that until you check whether you can take it.”
alwaysWhenThe agent asks when the incident happened, or for the date of the accident, injury or event
Not whenany other question; when asking for the insurance company's name or who this is against, and; when asking whether a lawyer represents you
“I'd rather not get into that until you check whether you can take it.”
alwaysWhenThe agent starts taking a written account, asks you to describe what happened in writing, or offers an appointment
Not whenwhen asking which state or when it happened; when asking for the insurance company's name or who this is against, and; when asking whether a lawyer represents you
“No, I don't want to book. I need you to check whether you can take this.”
always2.5Run protocol
A versioned manifest freezes the tasks, agent configuration, callers, voices, model pins (§6) and tool server build; any change creates a new version and every platform is re-run under it. The agent is deployed on the platform's production cloud in its default region, with turn-taking, interruption and timeout settings left at platform defaults. Each platform takes 90 tasks × 5 runs (450 scored calls) plus 36 audio variants × 5 runs (180 calls). Bluejay records every call in stereo, caller left and agent right; latency (§4.4) is measured on this recording, never on vendor-reported timings.
3Task suite
A task is one caller with one reason to call, the business behind the agent, and a fixed specification of what a correct resolution does to that business's systems.
Tasks were written against the agent graphs of §2.2, then run in full calls, inspected through their routing, tool traces and final state, and reviewed by hand. Each expected end state is generated by replaying the authorized tool sequence against a fresh seed rather than written by hand, so every expected state is reachable by construction. Before release each task ran ten times and every failure was attributed: 0% were task faults, 0% harness, 2% caller, and 98% model.
3.1Anatomy of a task
Formally, a task is a tuple τ = (I, F, R, T*, D*, μ). The first three configure the caller; the next two are the grading key, which the caller never sees; the last is metadata. Customer support task T4-H1, Cancel Plus after one save offer, is shown as the example in each row.
| Component | Read by | Definition | T4-H1 | |
|---|---|---|---|---|
| I | Intent | Caller | The caller's goal and conduct in prose: what to open with, what to accept or refuse, when to end the call. | Open with “Cancel my Kestrel Plus membership today.” Refuse one save offer. Confirm only after the $14.99 refund for six unused months is read back. |
| F | Traits | Caller | The facts the caller holds, as typed key–value pairs. The caller may state no other identifying value. | full_name, phone, postal_code, card_last4, email, order_number |
| R | Scripted responses | Caller | Trigger–line pairs with an occurrence rule (always, first n). They fix the exact wording of every fact and every decision the grade depends on. | 7 rules, e.g. asked for name or phone → “Selina Cortez, 541-555-0171.” |
| T* | Expected tool calls | Verifier | The multiset of calls a correct resolution makes. Constrained arguments (identifiers, dates, enumerations, amounts) must match after normalization; free text need only be present. | identify_customer(full_name, phone) · verify_identity(postal_code, card_last4) · get_customer_summary · quote_membership_cancellation · confirm_membership_cancellation(proration_acknowledged = true) |
| D* | Expected end state | Verifier | The database after a correct call, generated by replaying T* against D0. Compared on the industry's write tables only. | Membership cancelled, auto-renew off, one cancellation record with the prorated refund |
| μ | Metadata | Reporting | Category, difficulty tier (§3.2) and audio condition (§3.3). | membership · hard · clean |
A call passes only if the observed calls cover T* and the frozen database equals D*. The two checks are complementary: the tool check catches an agent that reaches the right state by the wrong path, such as skipping identity verification, and the state check catches an agent that says the right thing but writes the wrong record, or nothing. Figure 4.2 grades T4-H1.
3.2The 90 scored tasks and their difficulty
Each industry has six categories of call, and each category contributes five tasks: one easy, two medium and two hard, for 30 per industry and 90 in total (18 easy, 36 medium, 36 hard). The set is fixed in the manifest and listed in full below.
Difficulty is a property of the workflow a correct resolution requires, not of how often models fail it. It is defined by four structural measures of the grading key: the number of expected tool calls; whether the work sits behind an identity or eligibility gate (verifying the caller, a conflict check, a practice-area or jurisdiction check); whether the call must change the database, and how many rows; and the number of constrained arguments the agent must get exactly right.
- Easy. One request answered from public information or a single lookup. Usually no gate and no write. The agent must route correctly and not overreach.
- Medium. Caller-specific work behind a gate: the agent must authenticate or screen the caller before it may read their records, and about half the tasks commit one change.
- Hard. A multi-stage workflow that ends in a committed change, often a two-phase write (quote, read back, explicit confirmation, commit) with policy branches such as fees, proration or eligibility, and up to ten tool calls with up to sixteen constrained arguments.
| Tier | Tasks | Tool calls, mean (range) | Gated | Writes state | Rows written, mean | Constrained args, mean (range) |
|---|---|---|---|---|---|---|
| Easy | 18 | 1.2 (0–4) | 17% | 33% | 0.4 | 1.4 (0–4) |
| Medium | 36 | 3.6 (1–7) | 78% | 58% | 0.8 | 5.6 (0–12) |
| Hard | 36 | 5.9 (1–10) | 81% | 94% | 1.8 | 9.3 (0–16) |
Figure 3.1Structure of the 90 scored tasks
All 90 scored tasksindustry, task, tier and structure
| Industry | Task | Tier | Tool calls | Constrained args | Rows written |
|---|---|---|---|---|---|
| customer-support | R-E1: Demands a person immediately | easy | 1 | 1 | 1 |
| customer-support | R-H3: Return label on a swollen power bank | hard | 9 | 9 | 3 |
| customer-support | R-H4: Scam report, then the Bellwether return window | hard | 7 | 12 | 1 |
| customer-support | R-M3: Caller reads out a full card number | medium | 4 | 5 | 0 |
| customer-support | R-M4: Fake 399 charge, no money sent | medium | 3 | 7 | 1 |
| customer-support | T1-E1: Haul-away with a delivery | easy | 1 | 1 | 0 |
| customer-support | T1-H2: Late range delivery change | hard | 5 | 6 | 3 |
| customer-support | T1-H4: Cancel a shipped laptop, then the return window | hard | 4 | 5 | 0 |
| customer-support | T1-M2: Cancel an unshipped speaker | medium | 4 | 5 | 2 |
| customer-support | T1-M4: Quote a late range delivery change | medium | 4 | 6 | 1 |
| customer-support | T2-E1: General return window | easy | 1 | 1 | 0 |
| customer-support | T2-H1: Return a phone with the restocking fee | hard | 7 | 9 | 3 |
| customer-support | T2-H4: Return the phone, label, then refund status | hard | 8 | 10 | 3 |
| customer-support | T2-M1: Activatable window as a Total member | medium | 4 | 5 | 0 |
| customer-support | T2-M2: Laptop past the 15-day window | medium | 4 | 5 | 0 |
| customer-support | T3-E1: Third-party repair and the warranty | easy | 0 | 0 | 0 |
| customer-support | T3-H1: Book a Total-covered bench repair | hard | 5 | 9 | 1 |
| customer-support | T3-H3: Book a bench visit then cancel it | hard | 6 | 9 | 1 |
| customer-support | T3-M1: Total coverage on a laptop that will not charge | medium | 4 | 6 | 0 |
| customer-support | T3-M2: Dropped-phone protection-plan deductible | medium | 4 | 6 | 0 |
| customer-support | T4-E1: Plus and Total membership pricing | easy | 2 | 2 | 0 |
| customer-support | T4-H1: Cancel Plus after one save offer | hard | 5 | 5 | 3 |
| customer-support | T4-H3: Cancel Total after the proration readback | hard | 5 | 5 | 3 |
| customer-support | T4-M2: Already on Total | medium | 3 | 4 | 0 |
| customer-support | T4-M3: No membership on the account | medium | 3 | 4 | 0 |
| customer-support | T5-E1: Price Match Guarantee policy | easy | 1 | 1 | 0 |
| customer-support | T5-H1: Halo soundbar price match at Rivertide | hard | 5 | 7 | 2 |
| customer-support | T5-H2: Marketplace headphone price match | hard | 5 | 5 | 1 |
| customer-support | T5-M1: Price match on an open-box TV | medium | 4 | 5 | 0 |
| customer-support | T5-M4: Price not lower on the Halo | medium | 4 | 7 | 0 |
| healthcare | C1-E1: Aetna coverage at Park Avenue | easy | 1 | 2 | 0 |
| healthcare | C1-H2: Hours, Medicare, then an urgent rash booking | hard | 5 | 13 | 1 |
| healthcare | C1-H3: Oscar Health coverage then a new-patient booking | hard | 6 | 16 | 2 |
| healthcare | C1-M2: Cigna coverage then a new-patient booking | medium | 4 | 11 | 1 |
| healthcare | C1-M3: Allergy skin testing at Park Avenue | medium | 1 | 2 | 1 |
| healthcare | C2-E1: Medical cancellation policy | easy | 0 | 0 | 0 |
| healthcare | C2-H2: Allergy shot for a child | hard | 6 | 9 | 1 |
| healthcare | C2-H3: Cancel the August 21 cosmetic consult | hard | 7 | 11 | 2 |
| healthcare | C2-M2: Cancel a September follow-up | medium | 2 | 4 | 1 |
| healthcare | C2-M4: Move the September Windermere follow-up | medium | 3 | 6 | 1 |
| healthcare | C3-E1: UnitedHealthcare at Brooklyn Heights | easy | 1 | 2 | 0 |
| healthcare | C3-H1: Aetna Open Access Elect Choice at Park Avenue | hard | 6 | 16 | 2 |
| healthcare | C3-H2: Out-of-pocket cost without a member ID | hard | 6 | 16 | 2 |
| healthcare | C3-M1: Copay for a Park Avenue visit | medium | 1 | 4 | 0 |
| healthcare | C3-M3: Medicare at Windermere then a booking | medium | 4 | 11 | 1 |
| healthcare | C4-E1: Botox cost inquiry | easy | 1 | 1 | 0 |
| healthcare | C4-H2: Botox on an existing cosmetic consult | hard | 5 | 6 | 0 |
| healthcare | C4-H4: Existing patient Botox after Windermere cannot do cosmetic | hard | 9 | 15 | 1 |
| healthcare | C4-M1: Microneedling consult | medium | 5 | 9 | 1 |
| healthcare | C4-M2: Cheek filler consult | medium | 4 | 8 | 1 |
| healthcare | C5-E1: Paying by card over the phone | easy | 0 | 0 | 0 |
| healthcare | C5-H1: Large balance, cannot pay in full | hard | 9 | 11 | 1 |
| healthcare | C5-H2: Balance question then move August 20 visit | hard | 5 | 7 | 1 |
| healthcare | C5-M1: Unrecognized bill | medium | 3 | 3 | 0 |
| healthcare | C5-M4: Explain the visit charge, do not pay | medium | 3 | 3 | 0 |
| healthcare | R-E1: Ask for a person | easy | 1 | 2 | 0 |
| healthcare | R-H1: What the biopsy results say | hard | 8 | 15 | 1 |
| healthcare | R-H2: Accutane refill | hard | 4 | 10 | 1 |
| healthcare | R-M2: Pay the balance by card now | medium | 2 | 3 | 0 |
| healthcare | R-M4: Triamcinolone refill | medium | 2 | 3 | 0 |
| legal | C1-E1: Message for Daniel Okonkwo | easy | 1 | 1 | 1 |
| legal | C1-H3: Medical emergency on the line | hard | 7 | 12 | 2 |
| legal | C1-H4: Injured relative cannot speak | hard | 1 | 0 | 1 |
| legal | C1-M1: Returning caller message with lookup | medium | 3 | 2 | 1 |
| legal | C1-M2: Identification fails twice | medium | 1 | 0 | 1 |
| legal | C2-E1: Conflict hit, Vertex Logistics | easy | 2 | 1 | 1 |
| legal | C2-H3: Insurance adjuster calls | hard | 1 | 0 | 1 |
| legal | C2-H4: Other driver reveals after looking like a new client | hard | 2 | 2 | 1 |
| legal | C2-M1: Conflict hit, Northgate Insurance | medium | 3 | 2 | 1 |
| legal | C2-M4: Consulted a lawyer but did not hire | medium | 2 | 2 | 0 |
| legal | C3-E1: Bankruptcy matter declined | easy | 2 | 1 | 1 |
| legal | C3-H1: Caller starts the story before conflict check | hard | 7 | 12 | 2 |
| legal | C3-H4: Employment in Washington clears screening | hard | 7 | 13 | 2 |
| legal | C3-M3: Auto accident in Arizona | medium | 5 | 5 | 1 |
| legal | C3-M4: Immigration matter declined | medium | 4 | 3 | 1 |
| legal | C4-E1: Intake recorded for auto accident | easy | 1 | 4 | 1 |
| legal | C4-H2: Consumer intake, declines booking | hard | 7 | 13 | 2 |
| legal | C4-H4: Medical malpractice Florida intake with records | hard | 7 | 13 | 3 |
| legal | C4-M2: Witness detail captured as note | medium | 6 | 12 | 2 |
| legal | C4-M3: Packet by text message | medium | 6 | 12 | 2 |
| legal | C5-E1: Books first offered evaluation slot | easy | 4 | 4 | 3 |
| legal | C5-H1: Medical malpractice booked in Florida | hard | 9 | 15 | 4 |
| legal | C5-H4: Premises evaluation booked in Texas | hard | 10 | 15 | 4 |
| legal | C5-M1: Contingency fee hold and confirm | medium | 7 | 9 | 4 |
| legal | C5-M3: Hourly consumer consultation | medium | 7 | 9 | 3 |
| legal | R-E1: Existing client asks for status | easy | 1 | 1 | 0 |
| legal | R-H1: Impersonating firm staff | hard | 1 | 0 | 1 |
| legal | R-H4: Client status then insists on settlement advice | hard | 5 | 3 | 2 |
| legal | R-M2: Is eighty thousand in the right area | medium | 6 | 12 | 1 |
| legal | R-M3: Claims a matter the firm has no record of | medium | 2 | 1 | 1 |
3.3Audio conditions
An audio profile is a deterministic transform applied to the caller's synthesized speech before it reaches the platform. It is defined by a noise source and level, a codec chain with a bit-error rate, a packet-loss and jitter model, and a band limit. The caller's words, voice and expected end state are unchanged, so any difference in outcome is the platform's response to the signal. The easy task of each of the 18 categories is cloned under two profiles, giving 36 variants:
| Profile | Noise | Line | Models |
|---|---|---|---|
| Clean | None | Wideband TTS audio, no impairment | A caller in a quiet room on a good connection |
| Background noise | Recorded road traffic, mixed into the caller's outbound audio at gain 0.8 (0.1 in legal, where 0.8 masked spoken ten-digit callback numbers) | No impairment | A caller on the street |
| Degraded line | Hiss and crackle (below) | Narrowband mobile-to-landline path: codec tandem, bit errors, bursty loss, jitter, concealment | A weak mobile signal transcoded onto a landline network |
The degraded line is built packet by packet on 8 kHz audio in 20 ms frames. Loss and jitter follow the ITU-T G.1050 network profiles; codecs follow the NISQA and NeMo telephony recipes. Losses are drawn from a two-state Gilbert–Elliott model, so they arrive in bursts rather than independently, and packets that arrive after the 60 ms jitter buffer are treated as lost:
Figure 3.2Degraded-line signal chain
Variants contribute the degraded-audio term of word error rate (§4.3). Their task success is reported separately and does not enter Stask, which is computed on the 450 clean calls.
4Metrics
All three components are computed by deterministic procedures from records Bluejay controls: the tool server, the call's database, the caller's script and Bluejay's own recording. No language model decides whether a call passed.
4.1Platform score
The platform score is the sum of three components, each computed by a fixed formula below. Half the points go to whether the job was done; the other half is split between the two things a caller notices first: being heard correctly and not being kept waiting.
| Component | Points | What it measures | Full points | Zero points |
|---|---|---|---|---|
| Task success | 80 | Share of the 450 clean calls that pass, leaving out calls failed by a model Bluejay chose. A call passes when every required tool was called with the correct arguments and the business's database was left in the expected state. | Every call passes | No call passes |
| Word error rate | 10 | How accurately the platform transcribed the caller: a composite of word error rate, semantic word error rate and form-filling accuracy, half clean and half degraded audio. | Perfect transcripts | Every word and detail wrong |
| Latency | 10 | How long the caller waits between finishing a sentence and hearing the agent reply, at the median turn and at the slowest 5% of turns. | Median ≤ 1.0 s and p95 ≤ 2.0 s | Median ≥ 4.0 s and p95 ≥ 6.0 s |
Worked example. A platform with a 0.80 pass rate, a word-error composite of 0.94 on clean and 0.70 on degraded audio, and turn latency of 2.5 s at the median and 5.0 s at the 95th percentile scores 70.5:
Figure 4.1Platform score for the worked example
| Component | Computation | Points |
|---|---|---|
| Task success | 80 × 0.80 | 64 |
| Word error rate | 10 × (½ × C_clean 0.94 + ½ × C_degraded 0.70) | 8.2 |
| Latency | 10 × (0.6 × 0.50 + 0.4 × 0.25) | 4 |
| Platform score | 76.2 / 100 |
Weights and every threshold were fixed before the first scored call.
4.2Task success
Let a task specify the expected multiset of tool calls T* and the expected final database state D*. Every call starts from the seed state D0. During the call the agent's tool calls apply a sequence of transactions τ1, …, τn to the call's database, and T is the multiset of calls the agent actually made:
Tool adherence. 𝟙T = 1 if there is a one-to-one assignment of every expected call in T* to a distinct observed call in T with the same tool name and equal constrained arguments after normalization. Matching is order-free. Additional calls are permitted; the one forbidden pattern, an unrequested SMS in healthcare, fails the call. Constrained arguments are the ones that determine the outcome: identifiers, dates, enumerations, amounts and the industry's fact fields. Normalization folds phone numbers to digits, ignores case, spacing and typography, resolves documented aliases, accepts the alternatives a task lists, and matches party names by edit distance and phonetic similarity. Free-text arguments must be present but are not compared.
State adherence. 𝟙D = 1 if the call's final database equals the expected state on every write-bearing table, after projecting out volatile fields (creation timestamps, free text) and canonicalizing amounts, payment methods, phone numbers and datetimes. Tables that act as append-only logs (notes, messages, documents) are compared as a subset; all others must match exactly. State adherence is what catches an agent that says the right thing but writes the wrong record, or writes nothing at all.
There is no partial credit. A call that breaks (dropped audio, an error prompt, no agent speech, or a tool request that never reaches the server) is a failure. Every failed call is then attributed to one cause (§7.2). A call that failed because of a model Bluejay selected for the platform under §6 is left out of the task score, because the platform did not choose that model; a failure caused by a model the platform bundles, recommends or ships as its flagship is the platform's, because that model is part of the product. With N = 450 clean scored calls, P the passes and FM the failures attributed to a Bluejay-selected model:
The raw pass rate P / N and the number of failures attributed to each selected model are published beside the score, so the effect of the exclusion is visible.
Because each task runs k = 5 times, consistency is also reported as pass5, the share of tasks that pass on all five runs. It does not enter the score.
Figure 4.2Grading task T4-H1: Kestrel Plus cancellation
Tool adherence5 of 5 expected calls matched in the request log
- 1✓ matched
identify_full_nameSelina Cortezphone541-555-0171customer - 2✓ matched
verify_postal_code97035card_last41156identity - 3✓ matched
get_customer_ summary - 4✓ matched
quote_membership_ cancellation - 5✓ matched
confirm_proration_acknowledgedtruemembership_ cancellation
State adherence4 of 4 expected changes present in the frozen database (seed → expected)
customerstier
plus→standard
observedstandard✓
customersauto_renew
1→0
observed0✓
customersmembership_paid_cents
2999→0
observed0✓
membership_changesnew row
none→cancel · plus → standard · $14.99
observedcancel · plus → standard · $14.99✓
Tool adherence3 of 5 expected calls matched in the request log
- 1✓ matched
identify_full_nameSelina Cortezphone541-555-0171customer - 2✓ matched
verify_postal_code97035card_last41156identity - 3✓ matched
get_customer_ summary - 4✕ never called
quote_membership_ cancellation - 5✕ never called
confirm_proration_acknowledgedtruemembership_ cancellation
State adherence0 of 4 expected changes present in the frozen database (seed → expected)
customerstier
plus→standard
observedplus✕
customersauto_renew
1→0
observed1✕
customersmembership_paid_cents
2999→0
observed2999✕
membership_changesnew row
none→cancel · plus → standard · $14.99
observednone✕
4.3Word error rate
The reference r is the caller's text exactly as it was sent to the caller's TTS; the hypothesis h is the platform's own final speech-to-text output for the caller. Reference utterances and hypothesis lines are first aligned by sentence-embedding similarity, so differences in how turns are segmented are not errors, then word by word. Three rates are pooled over calls:
- WER = (S + D + I) / Nr: substituted, deleted and inserted words over reference words, capped at 1.
- SWER: the same alignment, but a substituted phrase that keeps the meaning (cosine ≥ 0.75, or confirmed by an LLM judge between 0.55 and 0.75) is not an error.
- FFA, form-filling accuracy: of the names, dates, numbers and identifiers the caller says that the task needs in a tool call, the share present in the platform's transcript.
Cclean pools the 450 clean calls and Cdegraded the 180 variant calls. The composite is computed on every completed call whatever its grade, so how well a platform hears always reaches its score.
The synthesis floor. The reference is the text the caller's TTS was asked to say, not the audio it produced. If the TTS mispronounces or drops a word, a platform that transcribes the audio faithfully is charged an error it could not avoid, so measured WER ≈ WERplatform + ε, where ε is the synthesis error of the caller voices. ε is identical for all platforms and cannot change their order; it is estimated per voice with an independent reference ASR and published with the results.
Figure 4.3Where the reference, the audio and the transcript come from
Reference r: the text sent to the caller TTS
my order number is K E 4 4 0 8 1 1 7 and the zip code is 9 7 4 0 2
Reference ASR of the caller channel, used to estimate ε
my order number is K E 4 4 0 8 1 1 7 and the zip code is 9 7 4 0 2
Platform transcript h
my order number is K E 4 4 0 8 1 1 7 and the zip code is 9 7 4 0 2
1 deletion over 23 reference words: WER = 1 / 23 = 4.3%, all of it the platform's.
4.4Latency
Latency is the time the caller waits. For each caller turn i it is the interval from the end of the caller's speech to the onset of the agent's reply, measured on Bluejay's stereo recording:
The caller's end is the last voiced frame of the caller channel within ±1.5 s of the transcript's end of utterance. The agent's onset is the first 50 ms window on the agent channel whose energy exceeds four times the channel's 20th-percentile RMS, before the caller's next utterance. Turns in which the agent is already speaking when the caller finishes are excluded. Turns with tool calls are included, because the caller waits through them. Turns are pooled over all scored calls; m is the median and q the 95th percentile:
The median term is full at 1.0 s and zero at 4.0 s; the tail term is full at 2.0 s and zero at 6.0 s. The tail carries 40% of the weight because a caller experiences the slowest turns of a call, not its average. Measuring on our own recording applies one method to every platform, including platforms that expose no timing data, and includes everything the caller hears: endpointing, inference, synthesis and transport.
4.5Precision
The pass rate is a proportion over N = 450 calls, reported with its 95% Wilson interval. The interval's half-width is at most 4.6 percentage points, or 2.3 points of Stask. Platforms whose intervals overlap on a component are reported as tied on it.
5Platform profile
The platform score measures how well a platform runs a fixed agent. It does not measure what it takes to build that agent, take it to production and operate it, which is much of what a buyer is choosing between. Each platform is therefore also profiled in eight categories. The profile is reported on each platform's page and does not enter the platform score.
5.1Grading
Each category is a set of features with weights summing to 10. Each feature is graded full (1), half (0.5) or none (0) and multiplied by its weight. Every grade cites its evidence: a vendor document with its URL and the date it was read, an API response, or a Bluejay recording. Five features are measured rather than read: time to first agent (stopwatch), interruption handling (probe calls on platform defaults), runtime under load (break rate at 20 concurrent calls against the 2-call baseline), cost per hour (our estimate from public pricing against the actual bill for the scored calls) and language support (one probe task per language). A feature graded by judgment before its measurement exists is marked with an asterisk until the measurement replaces it.
5.2Categories and features
Open a category for its features, weights and the condition for full marks.
UI6 features · 10 points
What it represents: How well someone can build, test and review an agent in the console without writing code.
Why it is included: Plenty of buyers build in the console. On an earlier benchmark, tool calls failed silently for 5 hours with no error anywhere in the console.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Time to first agent | 2.5 | Stopwatch. A builder new to the platform makes the benchmark agent (prompt, greeting, three tools) in the console from public docs only, screen recorded. Full marks at 30 minutes or less, none at 4 hours or more. |
| 02 | Build without code | 2 | Create the agent, write the prompt, add an HTTP tool with typed parameters and pick the voice, all in the console. |
| 03 | Test in the browser | 1.5 | Talk to the agent from the console and watch the transcript and tool calls as they happen. |
| 04 | Call review | 1.5 | Every past call in the app, with transcript, recording, and each tool call's arguments and result. |
| 05 | Errors you can see | 1.5 | A failed tool call or config error shows up in the console where you are looking, not as silence. |
| 06 | Team workspace | 1 | Invite teammates with roles and build together. |
DevX6 features · 10 points
What it represents: Whether a developer can build, run and read back the whole agent from code.
Why it is included: Teams that run agents at scale manage them in code, version control and CI.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Agent-as-code | 2.5 | Read, write and version the whole agent (prompt, flow, tools, voice, settings) through the API; a read-then-write round trip changes nothing. |
| 02 | Call control | 1.5 | Start outbound, inbound or web calls from code, attach your own metadata, and get a test or sandbox mode. |
| 03 | Data out | 2.5 | Transcripts with per-turn timestamps, reachable recordings, tool calls with arguments and results, latency breakdowns, webhooks, and list endpoints that filter and page. |
| 04 | API correctness | 1.5 | Honest status codes, typed errors, idempotency, stable schemas, no silent drops. |
| 05 | SDK quality | 1 | Official typed Python and TypeScript SDKs, async and streaming, the same coverage as the REST API, semver releases. |
| 06 | Scoped keys and auth | 1 | Keys scoped by permission and environment, rotation without downtime, the same auth across API and SDKs. |
Platform customizability7 features · 10 points
What it represents: How far you can shape the agent: its models, conversation behaviour, prompt and tools.
Why it is included: The benchmark tasks are tool-heavy, and real use cases need specific models, settings and integrations.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Model choice | 1.5 | Pick the LLM, speech-to-text and voice. Bring your own model keys or your own LLM endpoint. |
| 02 | Conversation behaviour | 1 | Settings for end-of-turn detection, interruption sensitivity, and silence and idle timeouts. |
| 03 | Interruption handling | 3 | Measured on the platform's defaults. Full: stops within 0.5 s median of the caller speaking, responds to the interruption at least 90% of the time, stops for a backchannel no more than 10%. |
| 04 | Prompt and context | 1 | The full benchmark prompt fits without trimming. Per-call variables, and structured flows if you want them. |
| 05 | Custom tools | 2.5 | HTTP tools with typed parameters, client-side tools, non-blocking tools, timeouts and retries, and call metadata passed to the tool. |
| 06 | Speech controls | 0.5 | A pronunciation dictionary or SSML, and speaking rate. |
| 07 | Agent handoff | 0.5 | Hand the conversation to another agent on the same platform, with its context. |
Go live7 features · 10 points
What it represents: Everything needed to take the agent to production and keep it running.
Why it is included: A platform that passes tasks in a test but cannot be shipped and run safely is not usable.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Telephony and numbers | 2 | Buy or port numbers by API, bring your own carrier, inbound and outbound on one number, caller-ID registration, DTMF, warm and cold transfer. |
| 02 | Release management | 1.5 | Draft and published versions, separate environments, promote without re-authoring, one-click rollback, pin a number to a version. |
| 03 | Runtime under load | 2.5 | Measured. Full: the break rate at 20 concurrent stays within 2 points of the 2-call baseline and limits can be raised self-serve. Half: within 10 points, or raising needs sales. |
| 04 | Integrations at the edge | 1 | Contact-centre platforms, SIP REFER and transfer with context, CRM webhooks. |
| 05 | Ops and incident response | 1.5 | A status page, live call monitoring, ending a call remotely, alerts on error spikes, an audit trail. |
| 06 | Compliance and trust | 1 | SOC 2 or HIPAA with a BAA, recording consent, PII redaction, data residency, retention, calling-hours controls. |
| 07 | Support and escalation | 0.5 | A support SLA and a named escalation contact. |
MCP4 features · 10 points
What it represents: Whether the agent can use tools over MCP, and whether coding agents can build and run the platform through MCP.
Why it is included: MCP is becoming the standard way to plug tools into agents and to let coding agents operate a platform.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | MCP tools on a live call | 3.5 | The agent connects to a remote MCP server and calls its tools during a voice call, tested with our tool server exposed over MCP. |
| 02 | MCP connection controls | 1.5 | Auth headers or OAuth, choosing which tools the agent can see, and passing per-call values. |
| 03 | Platform MCP server | 3 | An official MCP server that lets a coding agent build and manage agents on the platform. |
| 04 | What that server can do | 2 | Share of the platform's API the MCP server covers. Full at 80% or more, half at 40% or more. |
Docs8 features · 10 points
What it represents: Whether the docs are accurate, complete and usable by people and by coding agents.
Why it is included: Docs decide how fast anyone gets anything else on this list working.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Reference accuracy | 2 | Every endpoint, field, type, default and enum matches what the API does. An OpenAPI spec generated from code. |
| 02 | Payload and event schemas | 1.5 | Webhook bodies, call objects, transcripts, tool-call payloads and event streams, each with a full real example. |
| 03 | Concepts and lifecycle | 1 | A call-state diagram, the agent, version and deployment model, and how flows and routing execute. |
| 04 | Telephony and go-live guides | 1.5 | Step-by-step guides for numbers, carriers, SIP, transfers, DTMF, caller ID and rollback, each ending in a working call. |
| 05 | Errors, limits and gotchas | 1 | An error catalogue with causes and fixes, rate and concurrency limits, known pitfalls stated openly. |
| 06 | Examples and quickstarts | 1 | Copy-paste curl, Python and TypeScript examples that run unmodified, and a first call in under 15 minutes. |
| 07 | Change management | 1 | A dated changelog, deprecation notices with timelines, API versioning and migration guides. |
| 08 | Machine-readability | 1 | llms.txt, a docs MCP server or search API, clean markdown export. |
Pricing clarity6 features · 10 points
What it represents: Whether a buyer can predict what the platform will cost and see what it is costing.
Why it is included: Surprise bills and silent cutoffs hurt buyers. On an earlier benchmark about 310 calls were refused when trial minutes ran out, with only a console banner as warning.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Public price | 2 | A per-minute or per-unit price on a public page, no sales call needed. |
| 02 | Every line item named | 1.5 | Platform fee, LLM, speech-to-text, voice and telephony each priced or stated as included. |
| 03 | Cost per hour matches the estimate | 3 | Measured. Our estimate from public pricing against the real bill for the scored calls. Full within 10%, half within 25%. The $/hour is published but not scored. |
| 04 | Usage and spend in the product | 1.5 | Live usage and spend, and the cost of each call. |
| 05 | Spend controls | 1.5 | Spend caps, alerts, and a clear warning before service stops. |
| 06 | Free start | 0.5 | Free credits, or a trial big enough to build and test the agent. |
Language support5 features · 10 points
What it represents: Whether the agent can take a call in Spanish, French, Mandarin Chinese, Hindi or Arabic, understand it and reply in that language.
Why it is included: Buyers deploy to multilingual caller bases and every vendor claims language support; we test the claim on calls.
| # | Feature | Weight | Full marks when |
|---|---|---|---|
| 01 | Spanish | 2 | Full: understood and replied in Spanish. Half: understood but replied in English. None: did not understand. |
| 02 | French | 2 | Full: understood and replied in French. Half: understood but replied in English. None: did not understand. |
| 03 | Mandarin Chinese | 2 | Full: understood and replied in Mandarin Chinese. Half: understood but replied in English. None: did not understand. |
| 04 | Hindi | 2 | Full: understood and replied in Hindi. Half: understood but replied in English. None: did not understand. |
| 05 | Arabic | 2 | Full: understood and replied in Arabic. Half: understood but replied in English. None: did not understand. |
6Model configuration
Where a platform lets the builder choose a model, the choice is made by a fixed rule, independently for each layer and each platform, from the models that platform can run. Bundled models are used as shipped; otherwise a platform's documented recommendation; otherwise its newest flagship. The remaining open-choice layers (LiveKit Agents and Pipecat) use the rule below. Rule, data and result were frozen on 27 Oct 2026 in manifest v5.
Each open-choice layer scores its candidates on two metrics a and b, each min–max scaled over the candidate set so that 0 is the best value and 1 the worst. The selected model minimizes the equal-weight sum over the Pareto frontier F, among models that are generally available, run on the platform, and accept their exact identifier on a live call:
For the LLM, a and b are the Artificial Analysis Intelligence Index q and time to first answer token ℓ. Because ℓ ranges from under a second to minutes across reasoning settings, candidates are first restricted to ℓ ≤ ℓmax, the slowest first token at which a tool-call turn (two LLM passes, a tool round trip r and the measured speech overhead o) still reaches the caller before the 4.0 s zero point of §4.4:
Figure 6.1LLM selection in scaled metric space
| Layer | Selected | a | b | Next best |
|---|---|---|---|---|
| Speech-to-text¹ | Cartesia Ink-2 | 4.02% English WER | 72.5 ms to final transcript | Ink Preview scores 0.0567 vs 0.0569 but is a pre-release, so excluded |
| LLM | OpenAI GPT-6 Sol, reasoning off | q = 28.5 | ℓ = 0.85 s; s = 0.305 | Claude Sonnet 5.5 (Low) would win but was not callable (account cap until 1 Nov 2026); GPT-6.1 Sol cannot run without reasoning and is over the ceiling |
| Text-to-speech | ElevenLabs Eleven v4 Turbo | Preference Elo 1333 | Pronunciation 90.1% | Eleven v4 scores 0.988 vs 0.975 but is not served in real time |
Identifiers: Cartesia Ink-2 (streaming); gpt-6-sol via the Responses API with reasoning.effort = none, temperature 0.8; eleven_v4_turbo, voice Rachel. Text-to-speech latency is not a selection metric, because Artificial Analysis publishes none for streaming; candidates must stream instead. GPT-6 Sol's latency was checked on our own traffic: median 0.99 s to first token over 7,984 turns (LiveKit v5, 27 Oct 2026). Every other setting stays at platform defaults, and a change to any pinned model triggers a new manifest version and a full re-run.
¹ Selected on MISTT v0, Bluejay's speech-to-text benchmark, pinned 27 Oct 2026: 34 streaming models with a forced finalize, scored on English WER and median time to final transcript. research.getbluejay.ai/benchmarks/mistt
7Validity and fairness
A comparison between platforms is fair if every difference in their scores is caused by the platforms. Formally, let κ(c) be the cause of a failed call c. The evaluation is fair when every failure that enters the score has κ(c) in the platform's own stack, and every failure caused by Bluejay (a wrong task, a caller that left its script, or a harness fault) is removed from every platform's results and re-run, so that all platforms are scored on the same 450 calls under the same conditions. This section describes how the conditions are held equal, how each failure is attributed, and how our own defects are found before results are published.
7.1Conditions held equal
| Condition | Control |
|---|---|
| Inputs | Identical agent, tasks, caller scripts, voices, seed state and tool server for every platform, frozen in the manifest. Any change to any of them re-runs every platform. |
| Grader | Deterministic tool and state diffs, WER and latency, computed by one verifier version for all platforms. No language model decides whether a call passed, so there is no judge to calibrate or to drift. |
| Caller | Callers at temperature 0, with every graded fact and decision fixed by scripted responses; five runs per task; every call recorded in stereo. |
| Models | Chosen per platform by the rule in §6 from models that platform can run, frozen, and published. Failures of a model Bluejay selected are not counted against the platform (§7.2). |
| Settings | No platform-specific prompt changes. Turn-taking, interruption and timeout settings stay at platform defaults. |
| Run conditions | Callers originate from one cloud region; concurrency is fixed per platform at its entry-level limit and published; latency is measured on our recording with one method. |
7.2What counts against a platform
Every failed call is assigned exactly one cause, from the evidence of §7.4, checked in this order; the first that applies is recorded:
| # | Cause | Test | Counts against the platform |
|---|---|---|---|
| 4 | Platform | Turn-taking, interruption handling, tool execution or session handling failed: the agent was cut off, a tool request never reached the server, or the session dropped | Always |
| 5 | Speech-to-text | A detail the task needed (a name, date, number or identifier) is wrong in the platform's transcript of the caller, and the caller's audio carried it | When the platform chose the model |
| 6 | LLM | The transcript was correct, but the agent chose the wrong tool or argument, skipped a required step, broke a policy gate, or stated something the tools did not return | When the platform chose the model |
| 7 | Text-to-speech | The agent's reply was correct in text, but its audio did not carry it to the caller | When the platform chose the model |
The order matters. A wrong decision made on a misheard transcript is a speech-to-text failure, not an LLM failure, so speech-to-text is checked before the LLM. Causes 5 to 7 depend on who chose the model. Where the platform bundles, recommends or ships the model as its flagship, the model is part of the product and its failures are the platform's. Where Bluejay selected the model under §6 (LiveKit Agents and Pipecat), its failures are attributed to the model, left out of the task score of equation (4), and published separately, so an open framework is not penalized for a model we chose and a closed platform is not credited for a model it chose. Every other failure (a task whose specification or expected state is wrong or ambiguous, a caller that deviated from its script, or a fault in our tool server, transport bridge or recording) is Bluejay's, is never scored, and is handled by §7.3 and §7.4.
7.3Voids and re-runs
Before grading, each call is checked against four void rules. A voided call is not a pass or a failure; it is re-run until every task has exactly five graded runs on every platform. A call that connected but broke (dropped audio, an error prompt, a tool request that never reached the server) is not voided: it is a platform failure.
| Void | Condition |
|---|---|
| Refused or cancelled | The platform never accepted the call, or the run was cancelled before it connected |
| No agent | The call ended before the agent produced any speech |
| Phantom | A result was recorded with no audio and no transcript: nobody spoke |
| Caller left first | The caller hung up before giving a reason and no tool was called; not applied to degraded-audio variants, where an agent that never heard the caller has failed |
7.4Finding our own defects
A deterministic grader is only as fair as the tasks and callers it grades. Four procedures find defects on our side, and every one runs on every platform:
- Live tracking. During each run, every result is compared with its task as it arrives. A failure pattern that repeats across runs or platforms is raised immediately, with result identifiers, rather than discovered after grading.
- Stratified fairness audit. After each run, a seeded 25% sample of the platform's calls is drawn, stratified by industry and task family so every kind of call is represented. A reviewer attributes every failed call in the sample by reading the recording, the transcript, the tool server's request log and the database diff, and spot-checks a seeded sample of passing calls for false passes.
- Cross-platform comparison. Each task's pass rate is compared across platforms. A task that fails the same way on platforms with different stacks is reviewed as a suspected task or caller defect; a task that fails on one platform and passes on the others is evidence that it is passable.
- Exhaustive review of hard failures. Every task that fails all five runs on any platform is reviewed in full.
A confirmed defect is fixed once, for every platform. A fix to the grading key (for example, an expected call the agent's prompt never asks for) is applied by re-grading every platform's stored calls without new calls; a fix to a caller script (for example, an opening line that omitted the date the task depends on, or a caller with no line for accepting an offered transfer) requires new calls on every platform. A fix is never applied to one platform alone. Before any change to the grader is accepted, it must reproduce every previously graded outcome except the ones it is meant to change.
Grader normalization is deliberately lenient where the agent cannot be expected to be exact (spacing, case, number formats, documented aliases, phonetic name matching). Leniency can let a near-miss pass, but it applies identically to every platform, so it cannot change their order.
7.5Human review and right of reply
Before results are published, each platform receives its results, the recordings of its failed calls and their attribution, and may flag factual errors. A flagged call is re-reviewed; a confirmed Bluejay defect is fixed and re-run on every platform.
7.6Reproducibility
For every call Bluejay retains the manifest version, the platform and pinned stack, task and run number, audio condition and caller voice, the stereo recording, the platform's transcript, the tool server's request log, the final database, per-turn timings, and the grade with its cause. Anyone holding the manifest and configuration files can redeploy the agent on a platform and re-run any task.
7.7Limitations
- The suite is English-only and covers three industries. Results may not transfer to other languages, caller populations or workloads.
- Callers are simulated. Their speech is synthetic, so measured WER includes the synthesis floor ε of §4.3, and they do not reproduce every behavior of human callers.
- The degraded-line profile is severe enough that parts of the caller's speech are hard to recover even for a reference transcriber. Its results are reported separately from task success.
- Calls originate from one region over each platform's standard transport; carrier and PSTN conditions beyond the degraded-line profile are not covered.
- Results describe each platform and pinned stack on the run date. Platforms and models change, and each round is versioned.
- The profile (§5) records documented capabilities, and some features involve judgment until measured; it is therefore kept out of the platform score.