Voice AI agents answer and make phone calls using speech recognition, a language model and synthetic speech, fast enough that callers can talk naturally and interrupt. In 2026 they are a practical option for appointment booking, order and status enquiries, reminders and first-line call triage, especially where call volumes spike and staff spend hours on repetitive calls.
What changed is speed and naturalness. Real-time speech models and faster speech synthesis cut the awkward pauses that made older IVR bots feel robotic, and Indian-language speech models improved enough to handle code-mixed Hindi-English or Malayalam-English callers in many cases.
This guide explains how voice agents work, the architecture choices, what to expect on Indian languages, the TRAI, FCC and DPDP rules that apply, what drives cost, and how to run a pilot.
What a voice AI agent actually does
A voice agent is a phone-capable AI agent. It answers inbound calls or places outbound ones, understands what the caller wants, takes actions in your systems (check a slot, book it, look up an order), and speaks the answer back.
It differs from a classic IVR in three ways: callers speak naturally instead of pressing keys, the agent can handle follow-up questions and changes of mind, and it can act on live data rather than reading a fixed script. It differs from a text chatbot mainly in timing: on a call, a two-second silence feels broken.
If you are weighing voice against chat, our post on the AI chatbot for hospitals covers the text side, and WhatsApp automation for hospitals and clinics covers the channel most Indian patients prefer for confirmations.
Two architectures: cascaded vs speech-to-speech
| Design | How it works | Strengths | Weaknesses |
|---|---|---|---|
| Cascaded | Speech-to-text, then an LLM, then text-to-speech | Pick the best model for each step; easy to log and audit text; strong language choice | More hops add latency; tone of voice is lost in transcription |
| Speech-to-speech | One real-time model listens and speaks directly | Lowest latency; natural interruptions and tone | Fewer language options; harder to audit; vendor lock-in |
| Hybrid | Real-time model for conversation, separate transcription for records and analytics | Natural feel plus an audit trail | More moving parts |
OpenAI made its Realtime API generally available on 28 August 2025 with the gpt-realtime model and added SIP support, so a phone system can connect directly. Its model page lists audio at $32 per million input tokens and $64 per million output tokens, and connections over WebRTC, WebSocket or SIP. That original snapshot is now marked deprecated as newer realtime models replace it, which is a reminder to build so the model can be swapped.
For Indian languages, cascaded designs with Indian speech models are often the stronger choice today. Sarvam's Saaras speech recognition covers 22 Indian languages plus English with code-mixed audio, and in October 2026 the company launched Bulbul V4 Flash, a text-to-speech model for 11 languages that it says delivers under 250 milliseconds to first audio in production, priced at ₹20 per 10,000 characters.
Latency: the number that decides the experience
Callers judge a voice agent mostly by how quickly it responds after they stop talking. Every hop adds delay: telephony, voice activity detection, recognition, model reasoning, any tool call, speech synthesis and the trip back.
Practical rules that matter more than model choice:
- Stream everything. Start speaking the first sentence while the rest is still being generated.
- Keep tool calls fast. A slow appointment API ruins a fast model. Cache doctor schedules and slot lists.
- Host near your callers. Round trips from India to a distant region add noticeable delay.
- Handle interruptions ("barge-in"). The agent must stop talking the moment the caller speaks.
- Use short filler phrases ("Let me check that for you") only when a lookup genuinely takes time.
Indian languages and code-mixing
Indian callers switch languages mid-sentence, use English words for medical or banking terms, and speak with strong regional accents over noisy lines. This is where many voice projects stumble.
- Test with real call recordings (with consent) from your own callers, not studio audio.
- Check names, addresses, dates and numbers separately; they fail more often than general conversation.
- Confirm critical details back to the caller ("That is Tuesday the 14th at 10:30 with Dr. Menon, correct?").
- Offer a quick route to a human in the caller's language.
- For Kerala, test Malayalam and Malayalam-English mixing explicitly before launch rather than assuming coverage.
Where voice agents work well
| Use case | Why it fits | Keep a human for |
|---|---|---|
| Appointment booking and rescheduling | Structured, high volume, clear success | Complex multi-department bookings |
| Reminders and confirmations | Outbound, short, scripted outcome | Callers with clinical questions |
| Order, delivery or report status | Lookup and read-back | Disputes and complaints |
| After-hours call answering | Captures intent, books callbacks | Emergencies (route immediately) |
| Lead qualification | Short question set | Pricing negotiations |
| Call overflow during peaks | Absorbs queues, books callbacks | Anything sensitive |
For clinics and hospitals, voice booking works best when it writes directly into the scheduling system, which is why it belongs alongside an online appointment booking system rather than replacing it. A voice agent should never give medical advice; anything clinical or urgent goes to staff immediately.
Rules you must follow
India. TRAI's amendments to the telecom commercial communication regulations, notified in February 2025, split commercial calling into the 140 series for promotional calls and the 1600 series for service and transactional calls, and require senders to inform their telecom provider in advance about auto-dialer or robocall use. The TRAI press release also describes higher penalties. Outbound promotional calls must respect DND preferences. Under the DPDP Act and Rules, call recordings and transcripts are personal data: tell callers they are being recorded, state the purpose, and set retention limits.
United States. In February 2024 the FCC ruled that AI-generated voices count as "artificial" voices under the TCPA, so outbound calls using them need the same prior express consent as prerecorded calls, plus identification and an opt-out. Healthcare calls also bring HIPAA into play.
UAE and elsewhere. Check local telecom and health data rules before outbound campaigns.
What drives cost
Indicative costs depend on these factors more than on any headline price:
- Minutes and talk ratio. Speech-to-speech models bill for both listening and speaking. Long calls with a lot of caller speech cost more.
- Model choice. Smaller real-time models cost a fraction of the largest ones and are often good enough for booking flows.
- Telephony. SIP trunking, Indian numbers in the right series, and call recording storage.
- Integration. Connecting to your booking, HMS or CRM system is usually the largest one-time cost.
- Languages. Each additional language needs testing, prompt tuning and often a different voice.
For many booking and status use cases, the running cost per call is small compared with staff time, but integration and testing make up most of the first-year spend.
Designing the conversation and the hand-off
The model is only part of a good voice agent. The conversation design decides whether callers finish the call satisfied or hang up.
- Open with purpose and disclosure. A short greeting that names the organisation, says the caller is speaking with an AI assistant, and asks how it can help.
- Collect one thing at a time. Ask for the department, then the preferred day, then the time. Long compound questions confuse callers and recognition alike.
- Confirm before committing. Read back every detail that will be written into a system, and wait for a clear yes.
- Spell out tricky items. For names and reference numbers, ask the caller to spell or repeat, or send a WhatsApp or SMS link to complete the detail.
- Know when to stop. After two failed attempts at the same step, transfer to a person instead of looping.
- Transfer with context. When a call moves to staff, pass the transcript summary and collected details so the caller does not repeat everything.
Write down the rules for what the agent must never do: give clinical or financial advice, promise refunds, discuss another person's information, or handle an emergency beyond routing it. Test those rules with deliberately tricky calls before launch, and keep testing after every prompt or model change.
Metrics that show whether it works
| Metric | What it tells you |
|---|---|
| Containment rate | Share of calls fully handled without staff |
| Task accuracy | Bookings or updates that were correct when checked |
| Transfer reasons | Where the agent fails or is not allowed to act |
| Average handling time | Whether calls are faster than with staff |
| Caller hang-ups mid-call | Frustration, latency or recognition problems |
| Repeat calls within 24 hours | Problems not truly resolved |
Containment alone is misleading. An agent that keeps callers on the line but books the wrong slot looks good on containment and terrible on accuracy.
How to run a pilot
- Pull a month of call logs and categorise calls by intent. Pick the one or two intents with the most volume and the simplest outcome.
- Map the exact steps a good agent follows for those calls, including the questions and the system lookups.
- Choose an architecture: speech-to-speech for English-heavy, latency-sensitive flows; cascaded with Indian speech models for multilingual callers.
- Integrate with the booking or order system through a narrow, well-tested API.
- Test with 100 or more real recorded scenarios, including noisy lines and code-mixed speech.
- Launch on overflow or after-hours calls first, with instant transfer to staff.
- Review transcripts weekly and track containment rate, booking accuracy and transfer reasons.
Algon builds voice agents as part of patient engagement and AI agents in healthcare projects, where the hard part is clean integration with the hospital's scheduling data rather than the voice itself.
Conclusion
Voice AI agents crossed the line from robotic to usable in 2025-2026. Real-time speech models made English conversations feel natural, and Indian speech models made multilingual calls workable for structured jobs like bookings and status checks.
Success still depends on unglamorous things: fast integrations, careful testing on real calls, clear disclosure, compliance with TRAI or FCC rules, and a quick route to a human. Start with one high-volume call type, measure it honestly, and grow from there.
Frequently asked questions
Can an AI voice agent book appointments over the phone?
Yes. Booking and rescheduling are among the best use cases, provided the agent connects directly to your scheduling system and confirms date, time and doctor back to the caller.
What is a speech-to-speech AI model?
It is a real-time model that listens to audio and replies with audio directly, without converting to text in between. This lowers latency and keeps tone, but offers fewer language options and is harder to audit.
Do voice AI agents work in Indian languages like Hindi or Malayalam?
Indian speech models now cover many Indian languages and code-mixed speech, but accuracy varies by language, accent and line quality. Test with your own callers' recordings before launch.
Are AI voice calls legal in India?
Yes, within TRAI rules. Promotional calls use the 140 series and service or transactional calls the 1600 series, DND preferences apply, and auto-dialer or robocall use must be disclosed to your telecom provider. DPDP rules apply to recordings.
Do I need consent for AI voice calls in the US?
Yes. Since the FCC's February 2024 ruling, AI-generated voices are treated as artificial voices under the TCPA, so outbound calls need prior express consent, identification and an opt-out.
Should a voice agent tell callers it is an AI?
Yes. Disclosing that the caller is speaking with an AI assistant, and offering a quick route to a human, builds trust and aligns with regulatory expectations in India, the US and the EU.
