We ran the same 90 Voice AI tasks five times. Here’s what we learned about reliability.
Laura BurgessInsights
Updated
“Voice agents are customers’ first impressions of your business. Interacting with an effective, reliable voice agent is how customers build trust with your brand.”
G2 worked with Bluejay, a company that specializes in testing voice AI systems, to put six Voice AI Agent platforms through the same 90 simulated business tasks. Bluejay ran the calls across healthcare, legal and customer support environments, with 30 tasks in each designed to reflect the types of interactions these agents may be expected to handle.
A call counted as successful only when the required actions were completed correctly in the simulated business system. Success was verified using the system logs and the final state of its records, so a conversation that sounded convincing but failed to complete the required action did not pass.
To assess reliability, we ran each task five times on every platform and compared single-call success with five-run consistency.
Voice AI evaluation at a glance
The six Voice AI platforms tested were Bland, ElevenLabs Agents, LiveKit Agents, Pipecat, Retell AI and Vapi. More platforms are expected to join the G2 Voice AI Leaderboard over time.
| Platforms evaluated | Tasks | Graded runs per task | Scored clean calls per platform | Simulated business environments |
|---|---|---|---|---|
| 6 | 90 | 5 | 450 | 3 |
Single-call success ran ahead of five-run consistency on every platform
The evaluation looked at performance in two ways: success across individual calls, and consistency when the same task was repeated five times.
- Single-call success is the share of scored calls that passed. It shows how often a platform completed the required task correctly.
- Consistency across five runs is the share of tasks that passed on all five runs. A task that passed four times and failed once adds four passes to the single-call tally but does not count toward consistency.
Single-call success vs. five-run consistency, by platform
Single-call successFive-run consistency
View exact results and call counts
| Platform | Single-call success | Five-run consistency |
|---|---|---|
| LiveKit | 89.1% (401 of 450 calls) | 73.3% (66 of 90 tasks) |
| Pipecat | 86.0% (387 of 450 calls) | 61.1% (55 of 90 tasks) |
| Retell | 85.6% (385 of 450 calls) | 70.0% (63 of 90 tasks) |
| ElevenLabs | 81.1% (365 of 450 calls) | 61.1% (55 of 90 tasks) |
| Vapi | 63.8% (287 of 450 calls) | 35.6% (32 of 90 tasks) |
| Bland | 43.3% (195 of 450 calls) | 20.0% (18 of 90 tasks) |
The consistency gap varies across Voice AI platforms
Across the six platforms, five-run consistency was 15.6 to 28.2 percentage points lower than single-call success. The gap comes from tasks a platform completed on some attempts but missed on others.
LiveKit had the highest task-success figures on both measures, and 66 of its 90 tasks (73.3%) passed on all five runs. Vapi’s gap was the largest, from 63.8% to 35.6%. Pipecat and Retell were almost level on single-call success, at 86.0% and 85.6%, but their five-run consistency differed: 61.1% for Pipecat and 70.0% for Retell.
Many tasks passed on some runs, but failed on others
Each run started from the same conditions: the simulated business system was reset to an identical state, and the caller supplied the same scripted information. A different result on a later run wasn’t explained by a different task or a different starting point.
Number of tasks by how many of the five runs they passed (each row totals 90 tasks):
| Platform | 5 of 5 | 4 of 5 | 3 of 5 | 2 of 5 | 1 of 5 | 0 of 5 |
|---|---|---|---|---|---|---|
| LiveKit | 66 | 11 | 6 | 4 | 1 | 2 |
| Retell | 63 | 6 | 7 | 12 | 1 | 1 |
| ElevenLabs | 55 | 12 | 9 | 6 | 3 | 5 |
| Pipecat | 55 | 20 | 6 | 5 | 4 | 0 |
| Vapi | 32 | 15 | 11 | 12 | 10 | 10 |
| Bland | 18 | 15 | 8 | 8 | 5 | 36 |
For LiveKit, Retell, ElevenLabs, and Pipecat, five or fewer tasks failed on all five runs. On those platforms, most tasks that didn’t pass every time instead passed on some runs and failed on others.
Bland showed a different pattern, as 36 of its 90 tasks never passed, while another 36 passed on some runs but not others.
“On a poor phone line, most platforms fail at least 50% of the easy calls. I’d guess a human would not have done much better in similar conditions: parts of the caller’s speech were very hard to understand even with the best transcription tools. Voice agents let a business pick up the phone at any time, serve customers in any language, and take more calls as the business grows. Performance is improving much faster than we’d expect, and the best platform already passed 89% of calls and 73% of tasks on all five runs of the evaluation.”
Voice AI performance varied on the same task
The pass-count data shows how results varied across repeated runs. One Vapi law firm task shows what the variation looked like in practice. The task passed on one run and failed on another, despite both runs starting from the same state and using the same caller script.
VapiLaw firm: conflict of interest found
Run that passed
The caller wanted to sue a delivery company, and the firm’s conflict check flagged it. The agent then handed him to firm staff, as its instructions required for a conflict.
Conflict check — passing run
Run that failed
The same conflict check flagged the same company. The agent told him the firm could not take the matter, then ended the call without handing him to staff, so the required hand-off never happened.
Conflict check — failing run
| Run that passed | Run that failed | |
|---|---|---|
| Key action | Handed the caller to staff after the conflict check (reason: conflict) | Ended the call after the conflict check, with no hand-off |
| Business system outcome | Caller record and a hand-off for conflict, as expected | Caller record but no hand-off; a hand-off for conflict was expected |
| Result | Pass | Fail |
ElevenLabs showed a different split on the same task. In the passing run, the agent took the caller’s name and number before handing him to staff. In the failing run, it never asked for them, so the hand-off reached staff with no caller attached.
These examples illustrate how outcomes differed between runs, but they don’t indicate how frequently these failures occurred.
Breaking down Voice AI call failures
Bluejay classified each failed call into one of four categories:
- Model decisions: the transcript was accurate, but the AI model behind the agent took the wrong action or gave the wrong answer, such as skipping a required step.
- Speech recognition: the platform misheard the caller, such as a name or number.
- Platform behavior: the platform itself mishandled the call or did not carry out the action, such as a booking request that never reached the business system.
- Speech output: the agent’s words were right, but the audio did not carry them.
Failed calls by cause
| Cause | LiveKit | Retell | ElevenLabs | Pipecat | Vapi | Bland | All |
|---|---|---|---|---|---|---|---|
| Model decisions | 17 | 41 | 69 | 14 | 98 | 137 | 376 (55%) |
| Speech recognition | 25 | 23 | 8 | 49 | 52 | 33 | 190 (28%) |
| Platform behavior | 7 | 1 | 8 | 0 | 13 | 83 | 112 (16%) |
| Speech output | 0 | 0 | 0 | 0 | 0 | 2 | 2 (0.3%) |
| Failed calls | 49 | 65 | 85 | 63 | 163 | 255 | 680 |
LiveKit
- Model decisions
- 17
- Speech recognition
- 25
- Platform behavior
- 7
- Speech output
- 0
- Failed calls
- 49
Retell
- Model decisions
- 41
- Speech recognition
- 23
- Platform behavior
- 1
- Speech output
- 0
- Failed calls
- 65
ElevenLabs
- Model decisions
- 69
- Speech recognition
- 8
- Platform behavior
- 8
- Speech output
- 0
- Failed calls
- 85
Pipecat
- Model decisions
- 14
- Speech recognition
- 49
- Platform behavior
- 0
- Speech output
- 0
- Failed calls
- 63
Vapi
- Model decisions
- 98
- Speech recognition
- 52
- Platform behavior
- 13
- Speech output
- 0
- Failed calls
- 163
Bland
- Model decisions
- 137
- Speech recognition
- 33
- Platform behavior
- 83
- Speech output
- 2
- Failed calls
- 255
All platforms
- Model decisions
- 376 (55%)
- Speech recognition
- 190 (28%)
- Platform behavior
- 112 (16%)
- Speech output
- 2 (0.3%)
- Failed calls
- 680
What Voice AI call failures reveal
Most failures came from model decisions rather than speech recognition. Of the 680 failed calls, 376 (55%) involved an incorrect action or answer despite an accurate transcript, such as skipping a required step or choosing the wrong option. Speech recognition accounted for 190 failures (28%), platform behavior for 112 (16%) and speech output for just two (0.3%).
In one failed Retell law firm intake call, the agent recorded the caller’s phone number incorrectly, leaving the firm unable to call the client back.
The causes also varied considerably between platforms. Model decisions were the largest category for Retell, ElevenLabs, Vapi and Bland. On ElevenLabs, for example, 69 of its 85 failed calls fell into this category. Speech recognition was the leading cause on LiveKit and Pipecat, accounting for 49 of Pipecat's 63 failed calls. On both platforms, Bluejay selected the speech, language and voice models used in testing, so their results may reflect those choices as well as the platforms themselves.
Platform behavior failures were particularly concentrated on Bland, which accounted for 83 of the 112 failures in this category. Of those 83, 77 involved appointment bookings that never reached the business system.
Law firm calls had lower pass rates across all six Voice AI platforms
Pass rates also varied by task type. Law firm calls had lower pass rates than the same platform’s other 300 calls (healthcare and customer support) on all six platforms. Each platform handled 150 law firm calls (30 tasks repeated five times).
| Platform | Law firm calls passed (of 150) |
|---|---|
| LiveKit | 124 (82.7%) |
| Retell | 120 (80.0%) |
| ElevenLabs | 118 (78.7%) |
| Pipecat* | 107 (71.3%) |
| Vapi | 85 (56.7%) |
| Bland | 51 (34.0%) |
The gap is specific to the 30 law firm tasks used here. The evaluation did not test whether legal work is harder for voice agents in general.
How background noise and poor phone lines affected Voice AI performance
In a separate test of audio conditions, Bluejay repeated 18 easy tasks, one from each of the evaluation’s task categories, with recorded road traffic added and, separately, over a weak, distorted phone connection. That gave 90 calls per platform in each condition, reported separately from the 450 scored calls.
For the background-noise tests, Bluejay used a lower noise level for the six law firm tasks (0.1) than for the healthcare and customer support tasks (0.8), because the higher setting obscured the numbers required for legal intake.
Background noise changed the number of successful calls by five or fewer out of 90 on five platforms. Bland dropped from 63 to 38 successful calls out of 90. Under the poor-phone-line condition, every platform passed between 28 and 47 of the 90 calls. The simulated connection included compression, packet loss and audio interruptions, making parts of the caller’s speech difficult to understand even for an independent transcription tool.
What to look for when evaluating Voice AI platforms
Three checks for a demo, trial or pilot:
- Run each key workflow several times: Across the six platforms, five-run consistency was 15.6 to 28.2 percentage points lower than single-call success, so one successful test call does not show whether the same task will succeed again.
- Check the record in the business system after each call: In this evaluation, a call passed only when the right actions were logged and the records ended in the expected state. In a pilot, that could mean confirming that an appointment was booked correctly, a cancellation was recorded or a customer record was updated as expected. In the Retell call described earlier, the caller's number was saved incorrectly, preventing the firm from calling her back.
- Test with realistic details and audio: Speech recognition caused 190 of the 680 failed calls (28%), so test calls should include realistic names, phone numbers and order numbers. The evaluation also examined different workflows and audio conditions. A pilot can include both.
Methodology and limitations
Bluejay ran the calls across three simulated business environments (a dermatology practice, a law firm and an electronics retailer), building on its MIVAS benchmark. Six platforms handled the same 90 tasks, 30 per environment, each run five times. Each platform ran at its default settings, with the same tasks and starting states. One difference applied to Pipecat’s law firm calls, whose prompt gave the current date at the end rather than at the start, as noted in the law firm results above.
The results reflect performance under these controlled conditions. The callers were simulated, spoke English and did not interrupt, so the results may not carry over to other languages, accents or workloads. They should not be read as a prediction of production performance. They also describe complete Voice AI Agent platforms as they ran on the test dates, and do not rank the AI models inside them. This article looks at task success and repeated-run consistency only. The G2 Voice AI Leaderboard also scores word accuracy and response speed, so platforms rank differently there.
For more detail on how the evaluation was designed, run and scored, see the Voice AI evaluation methodology and the full technical report.
Comments
Questions about how this works? Ideas for what comes next? Join the discussion.
Loading comments…