LiveKit Agents
Run on self-hosted LiveKit workers. A LiveKit Cloud re-run is planned.
Open-source framework for real-time voice agents built on WebRTC rooms. Agents run as workers you host yourself or on LiveKit Cloud, and every model layer is configurable.
Score breakdown
Task success is worth up to 80 points, word error rate and latency up to 10 each.
Testing implementation
One of the calls LiveKit Agents took, end to end: the simulated caller, the agent as deployed on LiveKit Agents, the tool server it called and the grade. Callers, tools and grading are identical on every platform; only the platform and its model stack change.
Caller
C2-M1 · Conflict hit, Northgate Insurance
“My insurance company denied my claim.”
- full name
- Selma Ridgeway
- opposing party
- Northgate Insurance
- phone
- 919-555-0146
LiveKit Agents agent
Legal front desk
- STT
- Cartesia ink-2
- LLM
- gpt-6-sol (reasoning off)
- TTS
- ElevenLabs Eleven v4 Turbo (HTTP stream)
- Transport
- WebRTC
- Prompt and tools
- Shared legal spec, 16 tools
Tool server
Bluejay legal systems
- Tools
- 16 HTTP tools
- State
- Seeded database per call
- Log
- Every request, keyed to the call
- Same build
- Every platform
Grade
Pass
- ✓
lookup_callerfull_name=Selma Ridgeway - ✓
check_conflictopposing_party=Northgate Insurance - ✓
escalate_to_human
Tool-call diffempty
Database-state diffmatch
Task success by industry
Share of calls where the agent completed the caller's request. Click an industry to see what its callers ask for.
Most fails: the speech-to-text dropped spoken phone numbers after a name.
Word error rate
How well the platform heard the caller: the share of the caller's words its transcript got wrong. Lower is better. Variants replay a task with background noise or a degraded line.
Latency
How fast the agent answered: time from the caller's last word to the agent's first audio, over every turn. Full marks inside the shaded band; no points in the hatched end.
Findings
Setup review
The authors' assessment of building, deploying and running the agent on LiveKit Agents: interface, developer experience, customization, going live, MCP, documentation, pricing and languages. Bluejay's voice AI engineers score it during setup from documented capabilities and hands-on use. Expand a category for what we check and how each item is scored.
UIHow well someone can build, test and review an agent in the console without writing code.6.8* / 10
How it is scored. Six weighted line items, each graded full, half or none: time to first agent, build without code, test in the browser, call review, errors you can see, team workspace.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Time to first agent | 2.5 | half* 1.25 | Quickstarts claim minutes; no stopwatch run on either |
| Build without code | 2.0 | full 2 | LiveKit's browser Builder needs no code; PolyAI tools are Python functions |
| Test in the browser | 1.5 | full 1.5 | |
| Call review | 1.5 | half 0.75 | LiveKit insights are admin-only and kept 30 days |
| Errors you can see | 1.5 | half 0.75 | |
| Team workspace | 1.0 | half 0.5 |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
DevXWhether a developer can build, run and read back the whole agent from code.7.5 / 10
How it is scored. Six weighted line items: agent-as-code, call control, data out, API correctness, SDK quality, scoped keys and auth.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Agent-as-code | 2.5 | full 2.5 | |
| Call control | 1.5 | full 1.5 | PolyAI web calls need a provisioned connector token |
| Data out | 2.5 | half 1.25 | LiveKit transcripts are dashboard-only; PolyAI's Conversations API exports turns, latency, tool events, audio |
| API correctness | 1.5 | half 0.75 | No idempotency on either |
| SDK quality | 1.0 | full 1 | PolyAI has no client SDK |
| Scoped keys and auth | 1.0 | half 0.5 |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
Platform customizabilityHow far you can shape the agent: its models, conversation behaviour, prompt and tools.7.2* / 10
How it is scored. Seven weighted line items, including interruption handling measured on the platform's defaults during end-to-end validation.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Model choice | 1.5 | full 1.5 | |
| Conversation behaviour | 1.0 | full 1 | PolyAI silence and idle timeouts undocumented |
| Interruption handling | 3.0 | half* 1.5 | Barge-in on by default on both; stop time and backchannel rate not measured |
| Prompt and context | 1.0 | full 1 | Full benchmark prompts fit on both |
| Custom tools | 2.5 | half 1.25 | LiveKit: no built-in retry. PolyAI: Python-only, no HTTP tool |
| Speech controls | 0.5 | full 0.5 | PolyAI has no speaking-rate setting |
| Agent handoff | 0.5 | full 0.5 | PolyAI handoffs are SIP transfers only |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
Go liveEverything needed to take the agent to production and keep it running.5.8 / 10
How it is scored. Seven weighted line items, including runtime under load: the break rate at 20 concurrent calls against the 2-call baseline.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Telephony and numbers | 2.0 | half 1 | LiveKit numbers inbound-only; neither documents porting or STIR/SHAKEN |
| Release management | 1.5 | full 1.5 | |
| Runtime under load | 2.5 | half 1.25 | LiveKit Cloud 20-session project cap: NO_ANSWER above it; cut 12->10, 6->4 (concurrency_change.jsonl) |
| Integrations at the edge | 1.0 | half 0.5 | PolyAI has guided CCaaS setups |
| Ops and incident response | 1.5 | half 0.75 | |
| Compliance and trust | 1.0 | half 0.5 | |
| Support and escalation | 0.5 | half 0.25 | LiveKit publishes no numeric SLA; very responsive to us in practice |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
MCPWhether the agent can use tools over MCP, and whether coding agents can build and run the platform through MCP.4.2 / 10
How it is scored. Four weighted line items: MCP tools on a live call, connection controls, a platform MCP server, and how much of the API that server covers.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| MCP tools on a live call | 3.5 | full 3.5 | Both docs-graded |
| MCP connection controls | 1.5 | half 0.75 | |
| Platform MCP server | 3.0 | none 0 | PolyAI ships Builder and Data MCP servers; LiveKit's is docs-only |
| What that server can do | 2.0 | none 0 | PolyAI about 74% of documented REST endpoints |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
DocsWhether the docs are accurate, complete and usable by people and by coding agents.6 / 10
How it is scored. Eight weighted line items, from reference accuracy and payload schemas to change management and machine-readability.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Reference accuracy | 2.0 | half 1 | PolyAI publishes OpenAPI specs; LiveKit is protobuf-only |
| Payload and event schemas | 1.5 | half 0.75 | |
| Concepts and lifecycle | 1.0 | half 0.5 | |
| Telephony and go-live guides | 1.5 | half 0.75 | |
| Errors, limits and gotchas | 1.0 | half 0.5 | |
| Examples and quickstarts | 1.0 | full 1 | PolyAI has no TypeScript examples |
| Change management | 1.0 | half 0.5 | |
| Machine-readability | 1.0 | full 1 |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
Pricing clarityWhether a buyer can predict what the platform will cost and see what it is costing.6.2* / 10
How it is scored. Six weighted line items, including cost per hour: our estimate from public pricing against the real bill for the scored calls.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Public price | 2.0 | full 2 | PolyAI publishes no price |
| Every line item named | 1.5 | full 1.5 | |
| Cost per hour matches the estimate | 3.0 | half* 1.5 | LiveKit estimate possible, bill not reconciled; PolyAI has no estimate to check against |
| Usage and spend in the product | 1.5 | half 0.75 | |
| Spend controls | 1.5 | none 0 | |
| Free start | 0.5 | full 0.5 |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
Language supportWhether the agent can take a call in Spanish, French, Mandarin Chinese, Hindi or Arabic, understand it and reply in that language.10* / 10
How it is scored. One translated probe task per language, two runs each. Full when the agent understood and replied in the language, half when it understood but replied in English.
| What we checked | Weight | Grade | Note |
|---|---|---|---|
| Spanish | 2.0 | full* 2 | Graded by configuration; live probes not run |
| French | 2.0 | full* 2 | Graded by configuration; live probes not run |
| Mandarin Chinese | 2.0 | full* 2 | Graded by configuration; live probes not run |
| Hindi | 2.0 | full* 2 | Graded by configuration; live probes not run |
| Arabic | 2.0 | full* 2 | Graded by configuration; live probes not run |
Initial grades from the 28–29 Oct 2026 run and public docs; * = judged from docs, not yet verified on a call.
Run details
Model configuration
- STT
- Cartesia ink-2
- LLM
- gpt-6-sol (reasoning off)
- TTS
- ElevenLabs Eleven v4 Turbo (HTTP stream)
- Transport
- WebRTC
Run
- Scored calls
- 630
- Runs per task
- 5
- Run dates
- 21–22 Oct 2026
- Concurrency held
- up to 16 calls
- Turns measured
- 3,334 turns