Guide

How to Evaluate an AI Voice Agent: The SayVo Standard

How to evaluate an AI voice agent before it calls your leads: test every call outcome on a real phone, trace it into your CRM and prove the stop button works.

TCPA Compliance
RESPA Guidelines
Fair Lending
State Regulations
A
Arman Zolmajd
Founder, SayVo.ai · November 12, 2026 · 10 min read
how to evaluate an ai voice agenthow to test an ai voice agentai voice agent testing checklistai voice agent qaquestions to ask an ai voice agent vendordoes ai calling sound robotic

To evaluate an AI voice agent, judge it on real phone calls, not a demo: every possible call outcome, tested end to end, with each result checked inside your own CRM and a way to stop the AI on any lead in one step. That is the bar SayVo sets for itself. Before one of our agents calls a real lead, all 9 call outcomes it can produce are tested on a real phone, from booked to wrong number, and every one is traced into the client's CRM. This page is that standard written down, built from 200,000+ real calls, so you can hold any vendor to it, including us.

TL;DR
  • A demo shows the best call. Ask to see the busy, voicemail and “not interested” calls too.
  • Judge the voice on a real phone. People expect a reply within a fraction of a second.
  • Every call outcome should land in your CRM: stage, tag, note, recording, owner.
  • You need a one-step stop button, proven before launch, and a staged rollout.
9
call outcomes SayVo tests end to end, on a real phone, before any agent goes live
Source: SayVo, 2026
0 to 200 ms
the most common gap before a person replies in conversation, across 10 languages
Source: PNAS, 2009
95%
of companies in MIT NANDA's dataset saw their generative AI efforts fall short
Source: Fortune, 2025

Why does a demo call tell you so little?

A demo is one call, chosen by the vendor, to a person who wants it to go well. Your leads are not that person. They are busy, half-listening, driving, or sure it's spam. The hard part of AI calling is everything that happens on those calls, and a demo never shows it.

You are probably the sixth AI company we have interviewed, and most of them sound robotic.

Owner of a real estate and mortgage business, on a SayVo discovery call

The wider data says the same thing. MIT's NANDA initiative found that for 95% of companies in its dataset, generative AI was falling short.[1] The same research found that tools bought from specialized vendors succeeded about 67% of the time, while internal builds succeeded about a third as often.[1]Buying beats building, but only if you can tell a tested product from a good demo. (If you're weighing that choice, see enterprise voice AI vs done-for-you.)

So test it like you would test a new hire: on the calls that go wrong.

Does it sound like a person on a real phone?

Start with timing, because people notice it before they notice words. A study of conversation in 10 languages found the most common gap before someone replies is 0 to 200 milliseconds, with a median of about a tenth of a second.[2]The same study found replies that don't answer the question are often delayed by up to a full second.[2]So a long pause doesn't just feel slow. It sounds like an evasive answer.

Test on a real phone, not a browser widget. We see this often: a test widget can sound worse than the phone line, with pauses that never happen on a real call, and it can also hide problems. Call it from your own cell, in your car, with the radio on. Then listen for these:

What a natural call sounds like
It answers fast and lets you interrupt it without talking over you.
It doesn't repeat back what you just said before every question.
It never uses help-desk lines like "I'd be happy to help" or "great question."
It reads names, addresses and numbers the way a person would say them.
When you say "I'm busy," it books a time instead of promising to call back later.

Those last three are rules we enforce in every script. Each SayVo prompt carries a banned list of customer-service phrases that make a caller sound like a bot, and every line the AI says in testing is scanned for them by script. The no-repeat rule exists because parroting the caller is one of the fastest tells. For how the words are trained, see what training on 200,000+ calls means.

What happens on the calls that go wrong?

Most calls are not bookings. They are voicemails, busy signals, wrong numbers and people who say no. Each one needs a different next step, and a mistake on any of them costs you the lead or annoys someone who asked to be left alone. That is why our go-live check covers every outcome, not just the happy path:

1
Booked

The appointment lands on the right person's calendar and the alerts reach the owner and the rep.

2
Qualified, didn't book

The lead is flagged for a human and the follow-up starts.

3
Not interested

The AI stops calling and the CRM says why.

4
Unqualified

The lead is moved out of the calling list, with the reason in the notes.

5
Busy

A callback is timed, not a vague "we'll try later."

6
Voicemail and AI voicemail

The lead enters the follow-up sequence and stays in it. Both kinds are tested, because call screeners are common now.

7
Call failed

The system retries instead of marking the lead as worked.

8
Wrong contact

The number is flagged so nobody keeps dialing it.

Voicemail and AI voicemail count as two outcomes, which makes 9 in all. We repeat test calls until each one has been seen end to end. It works the same way in any business: an estimate follow-up for a home services company and an intake call for a law firm have their own versions of “booked” and “not interested,” and each needs the same proof.

Before those real-phone tests, we stress-test the script itself. The agent calls a second agent that plays the lead, with full, partial and empty lead details, each with an objection. One build went through six prompt versions in a single evening this way. It tests the script, not the voice, so we always finish on a real phone.

How does it handle “Is this a robot?”

Someone will ask. How the AI answers decides whether the call continues. Our default is simple: when a caller asks if it's AI, the agent says yes the first time, never denies it, and moves straight to the next step. It also treats “Are you real?” as the same question. We learned that one the hard way, when an agent answered “yeah, I'm real” and admitted it was an AI a minute later.

As soon as they realize AI, they're hanging up.

Owner of an independent mortgage brokerage, describing another AI caller

Ask every vendor to show you this moment on a real call. Ask too what disclosure options you get, because some businesses want the AI to say it up front. Rules for AI calls are also stricter than many owners expect. The FCC ruled in February 2024 that AI-generated voices count as “artificial” under the Telephone Consumer Protection Act.[3]Consent and calling rules are your business's responsibility, not the vendor's; our guide to TCPA rules for AI calling explains what to check with your counsel.

Does it sound like your business, not a generic bot?

A good agent should pass a test your own best salesperson would set. It should know what you sell, which questions to ask before it books, and what it must never say. Generic voice tools often get the first part right and miss the other two. For more on that gap, see industry-trained vs generic voice AI.

Every business has its own words to avoid. We keep a ban list for each client in its script, on top of the house list. One owner banned a casual filler word because, in their market, it “sounds like you're lying.” Another banned slang that felt too familiar for their callers. These sound small. On a call, they are the difference between “that was your assistant” and “that was a robot.”

Test it on your own business
Ask it three questions your callers ask most. Are the answers right, and short?
Give it a lead with missing details. Does it ask for them without sounding like a form?
Ask it for something only your team can give, like a price or a quote. It should hand that to a person.
Say no twice. Does it respect it, or keep pushing?
Ask for a human. It should honour that by the second ask at the latest.

Does every result land in your CRM?

A call that isn't recorded in your CRM didn't happen, as far as your team is concerned. For each of the 9 outcomes, check that the right stage, tag, note, recording link and owner show up on the lead record your team already opens. Then check it in the CRM itself, not the vendor's dashboard. A system can report “saved” and change nothing.

Two more checks catch problems late. First, the trigger: a new lead should start a call within 60 seconds. Second, the live switch: test connections should move to production last, only after every outcome works. Our CRM integration guide covers each system in detail, and the full list of tools we connect is on our integrations page.

Can you stop it on one lead, right now?

Every business eventually needs the AI to stop calling one person immediately, without emailing the vendor. If there is no clear way, your team will invent one, and it may silently do nothing. We have seen it: a team kept using a stop command the system didn't recognize, and the AI kept calling a lead they believed was stopped.

So the stop button is a launch step for every SayVo client, not an extra. We build it into the CRM the team already uses, prove it on a test lead through the live system, confirm the lead actually left every workflow, and walk the team through it on the launch call.

How should the launch be staged?

Never switch on your whole list at once. The safest launches start with older leads, where a mistake costs least, and grow in steps. That also trains the agent on real conversations before your newest, most expensive leads reach it.

1
A small sample

10 to 20 older leads. Read every transcript.

2
A real batch

100 to 200 leads. Read the results with the owner and fix what showed up.

3
Full volume

New leads, called within 60 seconds, around the clock, with the stop button in place.

Before the go-live, we also audit the live systems, not our notes. When one build was copied from an earlier one, it carried six small flaws, and all six had passed earlier reviews. Reading the live settings line by line is what caught them.

I don't want to just turn this on and say, hey, it sucks and cut ties.

Owner of an independent mortgage brokerage, on a SayVo discovery call

What should you watch in the first two weeks?

Testing before launch is half the job. The other half is watching the first real calls closely, because real leads say things no test thought of. Here is what we watch on every new agent, and what you should expect any vendor to watch with you:

The first two weeks
Read the transcripts of the first batch, all of them, not a sample.
Listen to every booking. Was the lead really qualified, and did the time suit them?
Check that every appointment alert reached the owner and the rep, in the place they actually look.
Compare the CRM notes with the recordings. Does the summary match what was said?
Look for leads that got stuck in a stage, or stopped being called with no reason in the notes.
Write down every new objection, and fix the script the same week.

Fixes after launch should be small and surgical: change the one line that caused the problem, re-test the path it affects with a real call, and leave the rest alone. Rewriting a working script to fix one moment is how new problems get in.

What should you ask to see, beyond the demo?

CheckWhat a demo showsWhat to ask for
VoiceOne polished call in a browserA live call to your own cell, where you interrupt and push back
OutcomesA bookingRecordings of busy, voicemail, not interested and wrong-number calls
"Is this AI?"Rarely comes upThe exact answer the agent gives, and your disclosure options
CRMA dashboardOne test lead traced into your own CRM for each outcome
Stop buttonNot shownA stop you trigger yourself, proven before launch
Launch"Turn it on"A staged plan: small sample, a real batch, then full volume

Questions to ask any AI voice agent vendor

Ask these before you sign
Which call outcomes do you test before launch, and can I hear each one?
Do you test on a real phone, or only in a browser?
What does the agent say when someone asks if it's AI, or if it's real?
Which phrases is the agent banned from saying, and how do you check?
Can I trace one test lead into my own CRM for every outcome?
How does my team stop the AI on one lead, and will you prove it works?
How is the launch staged, and who reads the first transcripts?
Who fixes it when something breaks after launch?

For how a well-built agent runs a call from first ring to booking, see how the AI qualifies and books a lead. New to the category? Start with what an AI voice agent does. Comparing vendors side by side? Our platform comparison covers the field.

Where this breaks

Testing finds most problems, not all of them. Real leads say things no test script thought of, which is why the first batches get read line by line. Agent-to-agent tests check the script but not the voice. A browser test can make a good agent sound bad, and a scripted demo can make a weak one sound great. Even a stop button only works if the team knows it exists. None of this means AI calling is risky. It means the testing is the product, and you should know exactly how a vendor does it before you sign.

Hear a real call, then decide

Get a live call from a SayVo agent on your own phone. Interrupt it, push back, ask if it's AI. Then book a demo and we'll walk you through how we'd test it on your leads.

Book a demo

SayVo is the premium, done-for-you AI voice agent for independent mortgage brokerages. Built on 200,000+ real mortgage calls, it calls every new lead within 60 seconds, works inside your CRM and goes live in 2 to 4 weeks.

Frequently Asked Questions

Can I test an AI voice agent before it calls my leads?

You should. Ask for a live call to your own phone, recordings of the calls that don't book, and a test lead traced into your own CRM. At SayVo, every call outcome is tested end to end before an agent reaches a real lead.

Will people hang up when they realize it's AI?

Some will, as they do with human callers. What matters most is how the agent answers the question. Ours says yes the first time it's asked, never denies it, and moves straight to the next step, so the call keeps its purpose.

How do I stop the AI from calling one person?

With a stop button inside the CRM your team already uses, such as a set task or tag. It should be proven on a test lead before launch. SayVo builds and proves one for every client and shows the team how to use it on the launch call.

Does an AI voice agent sound robotic?

Many do. Judge it on a real phone, not a browser widget: listen for fast replies, no repeating what you said, no help-desk phrases, and natural handling of names and numbers. A long pause before a reply is the most common giveaway.

How long does testing take before launch?

At SayVo, testing is part of the 2 to 4 week build. The launch is then staged: a small sample of older leads, then a larger batch, then full volume once the results have been read with you.

Who is responsible for consent and calling rules?

Your business is. The FCC treats AI-generated voices as artificial under the TCPA, so consent rules apply. A good vendor will explain the rules and build to your choices, but confirm your approach with your own counsel.

Sources & References

Related Articles

AI Voice Agent for Mortgage Brokers: Enterprise or Done-for-You?
Guide

AI Voice Agent for Mortgage Brokers: Enterprise or Done-for-You?

AI Voice Agents That Work Inside Your Mortgage CRM
Guide

AI Voice Agents That Work Inside Your Mortgage CRM

What Is an AI Voice Agent for Mortgage Brokers?
Guide

What Is an AI Voice Agent for Mortgage Brokers?

Ready to convert more leads?

Join mortgage teams already using SayVo to respond to every lead in seconds.

Book a Demo