An open-source prototype that turns a phone call into a draft clinical note.
Place a call from your browser to any ordinary phone. When you hang up, the call is transcribed, the speakers are separated, and a language model drafts a structured encounter note — plain text, ready to paste into an EHR.
⚠️ This is a prototype, not a product. It is not HIPAA compliant. There are no BAAs with Twilio, ElevenLabs, or Anthropic on a personal account. Use it with role-played scenarios only. Never call a real patient or family, and never put real PHI through it. See Before you go anywhere near real patients.
Independent personal project. This is my own side project, built on personal equipment with personal accounts. It is not affiliated with, endorsed by, funded by, or connected to the Children's Hospital of Philadelphia (CHOP) or any other employer or institution, and it does not represent their views, systems, or practices. No patient data, protected health information, institutional systems, or proprietary information were used in building or testing it — every sample in this repository is synthetic. Opinions here are mine alone.
Telephone work is some of the most under-documented labor in medicine. A triage nurse calls a family about a febrile toddler, spends nine minutes gathering a history, giving red-flag precautions, and agreeing on a plan — then spends another four minutes typing it into the chart from memory. The documentation is always thinner than the conversation, and the parts that get lost first are exactly the parts that matter in retrospect: what precautions were actually given, and what the parent actually said.
Commercial products solve this. Doximity Dialer AI does it well. But when a tool is a black box, it's hard to answer the questions that decide whether it belongs in a clinical workflow:
- How good does the transcript have to be before the note is trustworthy?
- What happens when the model hears something ambiguous — does it hedge, or invent?
- Where does speaker attribution break down, and how much does that matter?
- What does this actually cost per call at scale?
This project exists to make those questions answerable with open components. It is small enough to read in an afternoon and cheap enough to run on a personal account (well under $0.50 for a ten-minute call, end to end). If you're evaluating ambient documentation for your organization, this is a way to develop intuition using something you fully control.
The design bet: most "AI phone" projects chase a real-time conversational agent — streaming speech-to-text into an LLM into text-to-speech, fighting latency at every hop. That's a genuinely hard engineering problem, and it's the wrong problem here. A nurse doesn't need an AI to talk to the family. They need the conversation documented. Moving all the AI work to after the call removes the latency budget entirely, which means better models, better accuracy, and far less code. Everything here is batch.
The call detail view puts the diarized transcript beside the generated note. You edit the note if needed, then copy it into the chart.
Everything above came from a synthetic call — no real patient audio. Notes are generated
as plain text, not markdown, because EHR text fields render ## and ** literally;
producing markdown and converting later just moves the problem around.
The dialer offers two modes. Simulated has Twilio call you and play a scripted nurse voice, which produces a real two-sided recording without needing a working computer microphone — the fastest way to see the whole pipeline work.
A Demo mode toggle in the header masks every phone number, so you can screen-share or record a walkthrough without exposing them.
Browser softphone ──WebRTC──► Twilio ──PSTN──► any ordinary phone
│
│ dual-channel recording
▼
POST /api/recording-done (signature-verified webhook)
│
▼
ElevenLabs Scribe v2 → diarized transcript
│
▼
Claude Sonnet → plain-text encounter note
│
▼
SQLite + local disk
Why dual-channel recording matters. Twilio's record-from-answer-dual puts each
party on a separate stereo channel. That means speaker attribution is a fact about the
audio, not a guess from voice similarity — the nurse is channel 0 because they were on
channel 0. Diarization models are good and getting better, but they still stumble on
crosstalk, similar voices, and poor phone audio. Free ground truth beats a good model.
The one wrinkle: which channel is "the nurse" depends on how the call was placed. On a
browser softphone call, channel 0 is the nurse's browser leg. On a simulated call
(where Twilio dials you and plays scripted audio), channel 0 is the human who answers.
The app tracks this per call rather than assuming — see is_simulated in
src/lib/pipeline.ts.
| Layer | Choice | Why |
|---|---|---|
| Telephony | Twilio Programmable Voice | Browser→PSTN, dual-channel recording, nothing to install on the far end |
| Transcription | ElevenLabs Scribe v2 | Strong accuracy on phone-quality audio, built-in diarization, ~$0.22/hr |
| Summarization | Claude Sonnet | Good clinical phrasing, follows "don't invent things" reliably |
| App | Next.js (App Router) + SQLite | One process, no infrastructure, runs on a laptop |
| Item | Rate | 10-min call |
|---|---|---|
| Twilio voice (both legs) | ~$0.028/min | ~$0.28 |
| Twilio recording | $0.0025/min | ~$0.03 |
| ElevenLabs Scribe v2 | $0.22/hr | ~$0.04 |
| Claude Sonnet note | ~$0.01–0.03 | ~$0.02 |
| Total | ~$0.37 |
Plus ~$1.15/month for the phone number.
You need about 30 minutes, three accounts, and roughly $2 to try this end to end.
Requires Node 20+. The optional sample-call generator also needs
ffmpeg (brew install ffmpeg).
git clone https://github.com/YOUR_USERNAME/chipper-dialer.git
cd chipper-dialer
npm install
cp .env.example .env.localElevenLabs (elevenlabs.io) — free tier is enough to start.
Profile → API Keys → create one with speech_to_text and text_to_speech permissions.
Paste it into ELEVENLABS_API_KEY.
Anthropic (console.anthropic.com) → API Keys.
Paste into ANTHROPIC_API_KEY.
npm run devOpen http://localhost:3000, click Upload recording, and pick any two-person audio file. You'll watch it move through Transcribing → Summarizing → Ready, then get a diarized transcript and a draft note.
Don't have a recording handy? Generate a synthetic one — this script has two ElevenLabs voices perform a scripted pediatric triage call, so there's no real patient audio involved anywhere:
node scripts/generate-sample-call.mjsIt writes data/sample-call.mp3. Upload that. This is the fastest way to evaluate
whether the summarization quality is good enough for your purposes, and it costs
pennies. If the answer is no, you can stop here without ever touching Twilio.
Buy a number. Sign up at twilio.com, complete the Trust Hub compliance profile (required before a new account can buy US numbers), then Phone Numbers → Buy a number with Voice capability. ~$1.15/month.
Collect four values into .env.local:
| Variable | Where |
|---|---|
TWILIO_ACCOUNT_SID |
Console dashboard, starts with AC |
TWILIO_AUTH_TOKEN |
Console dashboard, next to the SID |
TWILIO_API_KEY / TWILIO_API_SECRET |
Account → API keys & tokens → Create API key |
TWILIO_PHONE_NUMBER |
The number you bought, E.164 (+13125551234) |
Expose your machine so Twilio's webhooks can reach it:
ngrok http 3000Copy the https://….ngrok-free.app URL into PUBLIC_BASE_URL in .env.local.
Create a TwiML App: Console → Voice → TwiML Apps → Create. Set the Voice Request
URL to <your ngrok URL>/api/voice. Copy the App SID (starts with AP) into
TWILIO_TWIML_APP_SID.
Restart npm run dev. The tunnel URL changes every time ngrok restarts — when it does,
update both PUBLIC_BASE_URL and the TwiML App's Voice Request URL.
Open http://localhost:3000/dialer. Two modes:
Simulated (recommended first) — Twilio calls you and plays a scripted nurse
voice with pauses. You answer your phone and improvise as the parent. This produces a
genuine two-sided recording and needs no working computer microphone, which
sidesteps the most common setup problem. The audio ships with the repo, so this works
out of the box; to change what the nurse says, edit
src/lib/sim-script.json and re-run
node scripts/generate-sim-lines.mjs.
Note the nurse is reading a fixed script and cannot react to you. If you report alarming symptoms, she'll still deliver the next reassuring line — the note documents that faithfully, which is itself a useful illustration of where a scripted demo ends.
Live call — the browser becomes a softphone and you dial out. Grant Chrome microphone access when prompted; if you skip it, the call connects but your side of the recording will be silent.
Either way: hang up, open the call from the Calls list, and watch it process.
This prototype deliberately stops short of clinical use. To go further you would need, at minimum:
- BAAs with every vendor. Twilio and ElevenLabs offer them on enterprise plans; Anthropic offers them via commercial agreements. Consumer-tier accounts are not covered, full stop.
- Authentication and audit logging. There is no login. Every call is visible to anyone who opens the app.
- Encryption at rest for recordings and transcripts. Right now they're plain files
and an unencrypted SQLite database in
data/. - A real retention policy. You can delete an individual call (removing its recording, transcript, and note) or wipe everything from the Calls page, but deletion is manual — there is no automatic expiry, scheduled purge, or enforced retention window, and nothing is deleted unless you ask for it.
- Clinical validation. Someone has to answer how often the note misses or distorts something clinically meaningful, on your patient population and your call types. That question is the whole ballgame and no amount of good engineering answers it.
- Two-party consent. Call recording law varies by state. Many require all-party consent.
Running a dev server behind a public tunnel is the risky part of this setup: the tunnel URL is guessable, and everything behind it would otherwise be world-readable. The default posture is therefore deny by default.
src/proxy.ts allows exactly four path prefixes to be reachable from
the internet — the three Twilio genuinely needs plus the audio it plays:
/api/voice /api/recording-done (both signature-verified)
/api/simulate/twiml /sim/*.mp3
Everything else — the UI, /api/calls, /api/token, /api/ingest, /api/simulate —
returns 403 unless the request originated on your machine. Without this, a stranger with
the tunnel URL could read every transcript and note, rewrite a note before it's pasted
into a chart, or pull a Voice token and place calls billed to you.
Also in place:
- Twilio webhooks are signature-verified (
X-Twilio-Signature), so callbacks can't be fabricated. Mismatches log the URL they were validated against — the usual cause is a stalePUBLIC_BASE_URLafter the tunnel restarted. - The locality check doesn't rely on the
Hostheader alone, which is client-supplied. It inspects forwarding headers, which a remote caller cannot forge past the tunnel edge. Seesrc/lib/localOnly.ts. - Recording downloads are restricted to
twilio.comover HTTPS with redirects disabled — the URL arrives in webhook data and is treated as untrusted. - Uploads are capped at 200 MB with an audio-extension allowlist.
- Processing is claimed atomically, so a webhook retry and a manual reprocess can't both transcribe the same recording.
- Secrets live only in
.env.local, gitignored along withdata/. - Calls can be deleted individually or all at once, which removes the audio file from disk as well as the database row. Worth doing after a demo.
What is not here: there is no authentication. Anyone with access to your machine has access to every call. That's acceptable for a single-operator prototype and unacceptable for anything else.
If you fork this, rotate any credential that has ever been pasted somewhere shared.
src/
app/
page.tsx Calls list
dialer/page.tsx Softphone + simulated-call UI
calls/[id]/page.tsx Transcript beside the editable note
api/
token/ Twilio Voice access token for the browser SDK
voice/ TwiML for outbound softphone calls
simulate/ Places a scripted call (local-only)
recording-done/ Twilio webhook → kicks off processing
ingest/ Upload a recording, skip telephony entirely
calls/ Read/update call records
lib/
pipeline.ts transcribe → label speakers → summarize
transcribe.ts ElevenLabs Scribe v2 + channel handling
summarize.ts Clinical note prompt + markdown stripping
guard.ts Webhook signature + local-only checks
db.ts SQLite schema and queries
scripts/
generate-sample-call.mjs Synthetic two-voice call for offline testing
generate-sim-lines.mjs Renders the simulated nurse's lines
- Verify the note against the transcript with a second model pass and flag unsupported claims
- Compare summarization models head-to-head on the same transcripts
- Structured output (FHIR
DocumentReference) instead of a text blob - Let clinicians correct speaker labels and feed that back into evaluation
MIT — see LICENSE.
An independent personal project by Stephon Proctor, developed on personal time and equipment using personal accounts. Not affiliated with, endorsed by, or representing the Children's Hospital of Philadelphia (CHOP) or any other employer or institution. Nothing here reflects the views, systems, policies, or practices of any organization I am associated with. No patient data, PHI, institutional systems, or proprietary information were used — all examples are synthetic.
Also not affiliated with or endorsed by Doximity, Twilio, ElevenLabs, or Anthropic.
Not a medical device. Not validated for clinical use. Provided as-is for demonstration and research purposes only.