Skip to content

We ran the same 90 Voice AI tasks five times. Here’s what we learned about reliability.

Laura BurgessInsights

Updated

Six voice AI platform logos on separate tiles, each branching into blue and orange audio waveforms, with Retell above ElevenLabs in the center.

“Voice agents are customers’ first impressions of your business. Interacting with an effective, reliable voice agent is how customers build trust with your brand.”

Faraz Siddiqi
Chief Technology Officer, Bluejay

G2 worked with Bluejay, a company that specializes in testing voice AI systems, to put six Voice AI Agent platforms through the same 90 simulated business tasks. Bluejay ran the calls across healthcare, legal and customer support environments, with 30 tasks in each designed to reflect the types of interactions these agents may be expected to handle.

A call counted as successful only when the required actions were completed correctly in the simulated business system. Success was verified using the system logs and the final state of its records, so a conversation that sounded convincing but failed to complete the required action did not pass.

To assess reliability, we ran each task five times on every platform and compared single-call success with five-run consistency.

Voice AI evaluation at a glance

The six Voice AI platforms tested were Bland, ElevenLabs Agents, LiveKit Agents, Pipecat, Retell AI and Vapi. More platforms are expected to join the G2 Voice AI Leaderboard over time.

Platforms evaluatedTasksGraded runs per taskScored clean calls per platformSimulated business environments
69054503
The 90 tasks were split evenly across a dermatology practice, a law firm, and an electronics retailer (30 each).

Single-call success ran ahead of five-run consistency on every platform

The evaluation looked at performance in two ways: success across individual calls, and consistency when the same task was repeated five times.

  • Single-call success is the share of scored calls that passed. It shows how often a platform completed the required task correctly.
  • Consistency across five runs is the share of tasks that passed on all five runs. A task that passed four times and failed once adds four passes to the single-call tally but does not count toward consistency.

Single-call success vs. five-run consistency, by platform

Single-call successFive-run consistency

LiveKit
89.1%
73.3%
Pipecat
86.0%
61.1%
Retell
85.6%
70.0%
ElevenLabs
81.1%
61.1%
Vapi
63.8%
35.6%
Bland
43.3%
20.0%
View exact results and call counts
PlatformSingle-call successFive-run consistency
LiveKit89.1% (401 of 450 calls)73.3% (66 of 90 tasks)
Pipecat86.0% (387 of 450 calls)61.1% (55 of 90 tasks)
Retell85.6% (385 of 450 calls)70.0% (63 of 90 tasks)
ElevenLabs81.1% (365 of 450 calls)61.1% (55 of 90 tasks)
Vapi63.8% (287 of 450 calls)35.6% (32 of 90 tasks)
Bland43.3% (195 of 450 calls)20.0% (18 of 90 tasks)
Platforms are sorted by single-call success.

The consistency gap varies across Voice AI platforms

Across the six platforms, five-run consistency was 15.6 to 28.2 percentage points lower than single-call success. The gap comes from tasks a platform completed on some attempts but missed on others.

LiveKit had the highest task-success figures on both measures, and 66 of its 90 tasks (73.3%) passed on all five runs. Vapi’s gap was the largest, from 63.8% to 35.6%. Pipecat and Retell were almost level on single-call success, at 86.0% and 85.6%, but their five-run consistency differed: 61.1% for Pipecat and 70.0% for Retell.

Many tasks passed on some runs, but failed on others

Each run started from the same conditions: the simulated business system was reset to an identical state, and the caller supplied the same scripted information. A different result on a later run wasn’t explained by a different task or a different starting point.

Number of tasks by how many of the five runs they passed (each row totals 90 tasks):

Platform5 of 54 of 53 of 52 of 51 of 50 of 5
LiveKit66116412
Retell63671211
ElevenLabs55129635
Pipecat55206540
Vapi321511121010
Bland181588536
Platforms are sorted by five-run consistency.

For LiveKit, Retell, ElevenLabs, and Pipecat, five or fewer tasks failed on all five runs. On those platforms, most tasks that didn’t pass every time instead passed on some runs and failed on others.

Bland showed a different pattern, as 36 of its 90 tasks never passed, while another 36 passed on some runs but not others.

“On a poor phone line, most platforms fail at least 50% of the easy calls. I’d guess a human would not have done much better in similar conditions: parts of the caller’s speech were very hard to understand even with the best transcription tools. Voice agents let a business pick up the phone at any time, serve customers in any language, and take more calls as the business grows. Performance is improving much faster than we’d expect, and the best platform already passed 89% of calls and 73% of tasks on all five runs of the evaluation.”

Cyril Delattre
Agents Evals Product at G2

Voice AI performance varied on the same task

The pass-count data shows how results varied across repeated runs. One Vapi law firm task shows what the variation looked like in practice. The task passed on one run and failed on another, despite both runs starting from the same state and using the same caller script.

VapiLaw firm: conflict of interest found

Run that passed

The caller wanted to sue a delivery company, and the firm’s conflict check flagged it. The agent then handed him to firm staff, as its instructions required for a conflict.

Conflict check — passing run

Run that failed

The same conflict check flagged the same company. The agent told him the firm could not take the matter, then ended the call without handing him to staff, so the required hand-off never happened.

Conflict check — failing run

Run that passedRun that failed
Key actionHanded the caller to staff after the conflict check (reason: conflict)Ended the call after the conflict check, with no hand-off
Business system outcomeCaller record and a hand-off for conflict, as expectedCaller record but no hand-off; a hand-off for conflict was expected
ResultPassFail

ElevenLabs showed a different split on the same task. In the passing run, the agent took the caller’s name and number before handing him to staff. In the failing run, it never asked for them, so the hand-off reached staff with no caller attached.

These examples illustrate how outcomes differed between runs, but they don’t indicate how frequently these failures occurred.

Breaking down Voice AI call failures

Bluejay classified each failed call into one of four categories:

  • Model decisions: the transcript was accurate, but the AI model behind the agent took the wrong action or gave the wrong answer, such as skipping a required step.
  • Speech recognition: the platform misheard the caller, such as a name or number.
  • Platform behavior: the platform itself mishandled the call or did not carry out the action, such as a booking request that never reached the business system.
  • Speech output: the agent’s words were right, but the audio did not carry them.

Failed calls by cause

CauseLiveKitRetellElevenLabsPipecatVapiBlandAll
Model decisions1741691498137376 (55%)
Speech recognition25238495233190 (28%)
Platform behavior71801383112 (16%)
Speech output0000022 (0.3%)
Failed calls49658563163255680

LiveKit

Model decisions
17
Speech recognition
25
Platform behavior
7
Speech output
0
Failed calls
49

Retell

Model decisions
41
Speech recognition
23
Platform behavior
1
Speech output
0
Failed calls
65

ElevenLabs

Model decisions
69
Speech recognition
8
Platform behavior
8
Speech output
0
Failed calls
85

Pipecat

Model decisions
14
Speech recognition
49
Platform behavior
0
Speech output
0
Failed calls
63

Vapi

Model decisions
98
Speech recognition
52
Platform behavior
13
Speech output
0
Failed calls
163

Bland

Model decisions
137
Speech recognition
33
Platform behavior
83
Speech output
2
Failed calls
255

All platforms

Model decisions
376 (55%)
Speech recognition
190 (28%)
Platform behavior
112 (16%)
Speech output
2 (0.3%)
Failed calls
680
Each failed call was reviewed against its transcript, tool logs and final business-system records, then assigned one cause. The platform’s transcript was also used to distinguish speech recognition errors from incorrect model decisions.

What Voice AI call failures reveal

Most failures came from model decisions rather than speech recognition. Of the 680 failed calls, 376 (55%) involved an incorrect action or answer despite an accurate transcript, such as skipping a required step or choosing the wrong option. Speech recognition accounted for 190 failures (28%), platform behavior for 112 (16%) and speech output for just two (0.3%).

In one failed Retell law firm intake call, the agent recorded the caller’s phone number incorrectly, leaving the firm unable to call the client back.

The causes also varied considerably between platforms. Model decisions were the largest category for Retell, ElevenLabs, Vapi and Bland. On ElevenLabs, for example, 69 of its 85 failed calls fell into this category. Speech recognition was the leading cause on LiveKit and Pipecat, accounting for 49 of Pipecat's 63 failed calls. On both platforms, Bluejay selected the speech, language and voice models used in testing, so their results may reflect those choices as well as the platforms themselves.

Platform behavior failures were particularly concentrated on Bland, which accounted for 83 of the 112 failures in this category. Of those 83, 77 involved appointment bookings that never reached the business system.

Law firm calls had lower pass rates across all six Voice AI platforms

Pass rates also varied by task type. Law firm calls had lower pass rates than the same platform’s other 300 calls (healthcare and customer support) on all six platforms. Each platform handled 150 law firm calls (30 tasks repeated five times).

PlatformLaw firm calls passed (of 150)
LiveKit124 (82.7%)
Retell120 (80.0%)
ElevenLabs118 (78.7%)
Pipecat*107 (71.3%)
Vapi85 (56.7%)
Bland51 (34.0%)
*Pipecat’s law firm prompt gave the current date at the end, while the other platforms’ prompts gave it at the start. Bluejay found that none of Pipecat’s failed law firm calls were caused by the agent not knowing the date.

The gap is specific to the 30 law firm tasks used here. The evaluation did not test whether legal work is harder for voice agents in general.

How background noise and poor phone lines affected Voice AI performance

In a separate test of audio conditions, Bluejay repeated 18 easy tasks, one from each of the evaluation’s task categories, with recorded road traffic added and, separately, over a weak, distorted phone connection. That gave 90 calls per platform in each condition, reported separately from the 450 scored calls.

For the background-noise tests, Bluejay used a lower noise level for the six law firm tasks (0.1) than for the healthcare and customer support tasks (0.8), because the higher setting obscured the numbers required for legal intake.

Background noise changed the number of successful calls by five or fewer out of 90 on five platforms. Bland dropped from 63 to 38 successful calls out of 90. Under the poor-phone-line condition, every platform passed between 28 and 47 of the 90 calls. The simulated connection included compression, packet loss and audio interruptions, making parts of the caller’s speech difficult to understand even for an independent transcription tool.

What to look for when evaluating Voice AI platforms

Three checks for a demo, trial or pilot:

  • Run each key workflow several times: Across the six platforms, five-run consistency was 15.6 to 28.2 percentage points lower than single-call success, so one successful test call does not show whether the same task will succeed again.
  • Check the record in the business system after each call: In this evaluation, a call passed only when the right actions were logged and the records ended in the expected state. In a pilot, that could mean confirming that an appointment was booked correctly, a cancellation was recorded or a customer record was updated as expected. In the Retell call described earlier, the caller's number was saved incorrectly, preventing the firm from calling her back.
  • Test with realistic details and audio: Speech recognition caused 190 of the 680 failed calls (28%), so test calls should include realistic names, phone numbers and order numbers. The evaluation also examined different workflows and audio conditions. A pilot can include both.

Methodology and limitations

Bluejay ran the calls across three simulated business environments (a dermatology practice, a law firm and an electronics retailer), building on its MIVAS benchmark. Six platforms handled the same 90 tasks, 30 per environment, each run five times. Each platform ran at its default settings, with the same tasks and starting states. One difference applied to Pipecat’s law firm calls, whose prompt gave the current date at the end rather than at the start, as noted in the law firm results above.

The results reflect performance under these controlled conditions. The callers were simulated, spoke English and did not interrupt, so the results may not carry over to other languages, accents or workloads. They should not be read as a prediction of production performance. They also describe complete Voice AI Agent platforms as they ran on the test dates, and do not rank the AI models inside them. This article looks at task success and repeated-run consistency only. The G2 Voice AI Leaderboard also scores word accuracy and response speed, so platforms rank differently there.

For more detail on how the evaluation was designed, run and scored, see the Voice AI evaluation methodology and the full technical report.

Comments

Questions about how this works? Ideas for what comes next? Join the discussion.

Loading comments…