Build a Voice Agent That Tells You What to Fix

01What you'll need

Set these up before the session, so you spend it building instead of filling in signup forms.

On your laptop you'll need Node.js 20 or newer, git, and ngrok, logged in with ngrok config add-authtoken <your-token>. If you use a coding agent like Claude Code, it can do the setup for you:

02How it works

A voice agent has two very different jobs, and the trick is keeping them apart.

The live call has to be fast. Someone is waiting on the line. So the agent gets one job, one tool, a hard 1.5-second limit on that tool, and a fixed line to say when anything goes wrong. When a caller asks for something else, it hands them to a person instead of improvising.

Improving the agent happens later, offline. After every call, the agent writes a record of what was said, what the tool returned, how long it took, and how the call ended. Afterwards, two small programs read those records.

The review loop never changes the agent, and it doesn't use an AI model. It counts. When the same thing happens enough times, it writes a question, and you decide.

03The build

Every step tells you what you should see. If you don't see it, jump to If you get stuck.

1Clone and install

$ git clone https://github.com/Vonage-Community/workshop-voice-node_deepgram-agent_loops.git
$ cd workshop-voice-node_deepgram-agent_loops
$ npm install

2Create your Vonage application

1. Open the Vonage dashboard → Applications and create a new application. Name it something like voice-agent-workshop.
2. Click Generate public and private key. A file called private.key downloads. Move it into the workshop-voice-node_deepgram-agent_loops folder, keeping that exact name.
3. Turn on Voice. For the Answer URL and Event URL, enter https://example.com/answer and https://example.com/event. This app sends its own URLs with every call, so these are never used.
4. Create the application with the button at the bottom of the page, then copy the Application ID.

3Get your Deepgram key

In the Deepgram console, go to API Keys and create a key. Copy it now; Deepgram only shows it once.

4Start the tunnel

In a new terminal window:

$ ngrok http 3000

5Fill in your settings

$ cp .env.example .env

Open .env and set these four:

Leave everything else as it is. VONAGE_NUMBER and SUPPORT_PHONE_NUMBER stay blank on a trial account.

6Start the agent

$ npm run dev

7Call yourself

In a third terminal window:

$ npm run call-me

Answer it. The agent introduces itself. Say:

"Where is order A1001?"

It looks the order up and tells you it's in transit. In the npm run dev window you'll see the whole conversation go by: what you said, the tool call, how long it took, and finally outcome=completed.

8Open the dashboard

$ npm run dashboard

Open localhost:4000. Your call is under Recent Calls. The other panels are empty, and each one tells you the command that fills it.

9Run the loop

Run these in order, and watch the dashboard after each one.

$ npm run eval

$ npm run seed-calls

$ npm run review

"3 callers asked about returns and got transferred. Is transferring the right behavior, or should this agent handle returns?"

Click Add to Suite on one. It moves to Decisions → Working as intended. Then run the tests again:

$ npm run eval

10Make a pattern of your own

Now give the loop something real to find. Run npm run call-me twice more, and both times say:

"Hi, I want to cancel my order."

The agent says it'll connect you to support. On a trial account there's no one to transfer to, so just hang up. The handoff is still recorded.

After the first cancellation, run npm run review. Nothing new: one call isn't a pattern. After the second, run it again.

04Make it yours

Change the voice

Open .env and set:

DEEPGRAM_SPEAK_MODEL=aura-2-draco-en
AGENT_VERSION=order-status-v2

aura-2-draco-en and aura-2-pandora-en are British, aura-2-hyperion-en is Australian, and there are about forty more. Changing AGENT_VERSION labels every call from now on, so you can tell the new agent from the old one.

Settings are read when the server starts, so stop npm run dev (Ctrl-C), start it again, and run npm run call-me. The new call shows up on the dashboard as order-status-v2.

Try to trip it up

Call enough times and run npm run review again. Patterns you created start showing up as questions.

Break it on purpose

This is what the test suite is for. Open src/agent/tool-policy.ts and find the line that starts with ["billing", …. Delete it, save, and run:

$ npm run eval

Now put it back:

$ git checkout src/agent/tool-policy.ts

05Point it at a real API

The agent's one tool is a fake order database. Swap it for something real, and keep the rules that make it safe on a phone call: one lookup per call, a 1.5-second limit, and the fixed fallback line when it fails. These need no API key, and their answers are short enough to say out loud:

• Open-Meteo — weather anywhere"Do I need an umbrella in Brooklyn tomorrow?"
• PokéAPI — every Pokémon, type, and move"What's Pikachu weak against?"
• Open Library — books and authors"Who wrote The Left Hand of Darkness, and when?"
• REST Countries — capitals, currencies, populations"What currency do they use in Peru?"
• TheMealDB — recipes by name or ingredient"Give me a dinner idea with chickpeas."
• icanhazdadjoke — exactly what it sounds like"Tell me a joke."

It's a bigger change than tonight's steps, so let a coding agent do the work, and let the tests tell you if it went right:

06Quick reference

All commands run from the project folder.

07If you get stuck

Or hand it to your coding agent: