voice-engine

Package @27colab/voice-engine · Source on GitHub · MIT · Node 20+ · ESM

A small, tested TypeScript reference for the turn-taking part of a real-time phone agent: knowing what the caller actually heard when they interrupt, and not talking over them.

Not for you if…

• You want a complete voice-agent platform. Vapi, Retell, LiveKit Agents and Pipecat already ship barge-in; this is a compact implementation of the few mechanisms that are easy to get subtly wrong, not a replacement.
• You need latency or quality numbers. Nothing here has been benchmarked, and the offline fakes answer instantly.
• You need multi-tenant routing, persistence, metering, prompt assembly or telephony provisioning. None of that is included.

What it does

Mechanism	The problem it solves
Mark-confirmed playback	Sending audio is not playing audio. Every chunk is followed by a Twilio mark; only echoed marks count as heard.
Unspoken text from TTS alignment	Maps “confirmed N ms played” to a text prefix using per-character timings (multi-chunk, Spanish accents, NFC/NFD, emoji, word-boundary snapping).
History rewrite on barge-in	The model’s next prompt shows "We have openings Monday [interrupted]", not a sentence it never finished saying.
Backchannel vs interruption	“uh-huh” and “sí claro” do not cut the agent off; a real utterance does. The classifier is pluggable.
Speculative turn start	Start the LLM on Deepgram Flux EagerEndOfTurn; cancel on TurnResumed (restoring history); keep the reply if the final transcript matches.
Latency timeline	Per-call events and per-turn distributions (p50/p90) in the CallRecord.
Twilio Media Streams transport	start/media/mark/stop in, media/mark/clear out, over a structural socket type (no ws import in core).

Install

npm install @27colab/voice-engine

@anthropic-ai/sdk and ws are optional peer dependencies: install them only if you use the Anthropic adapter or run a WebSocket server.

Bring your own keys

The engine itself needs nothing. The live provider adapters need your own accounts:

Variable	Used by
DEEPGRAM_API_KEY	Deepgram Flux speech-to-text adapter
ELEVENLABS_API_KEY, a voice id	ElevenLabs text-to-speech adapter (any voice id; per-language voice selection)
ANTHROPIC_API_KEY	Anthropic Messages chat adapter
A Twilio number with TwiML streaming to your wss:// URL	Transport. Twilio credentials are not read by this package.

Run it without keys

The loopback script plays one scripted call against the fakes: a greeting, a question, a long answer, and a caller who talks over it after about two seconds of audio were confirmed played.

samples/src/voice-loopback.ts

import { createCallSession } from '@27colab/voice-engine';
import { FakeChat, FakeStt, FakeTransport, FakeTts } from '@27colab/voice-engine/testing';

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

const transport = new FakeTransport();
const stt = new FakeStt();
const tts = new FakeTts({ msPerChar: 60 });
const chat = new FakeChat()
  .pushText('We have openings Monday morning, Tuesday afternoon, and Wednesday evening if any of those work.')
  .pushText('No problem, what day works better for you?');

const session = createCallSession(
  { transport, sttFactory: stt, ttsFactory: tts, chat },
  { callId: 'demo-call', startedAt: new Date() },
  { onEnd: (record, reason) => console.log(`call ended (${reason}); barge-ins: ${record.stats.bargeIns}`) },
  {
    languages: ['en'],
    greeting: 'Hi, thanks for calling Acme. How can I help?',
    system: 'You are a concise front-desk assistant.',
    llm: { model: 'scripted', maxTokens: 200 },
    voice: { provider: 'fake', voices: {}, defaultVoiceId: 'any-voice-id', model: 'fake', outputFormat: 'ulaw_8000' },
    stt: { model: 'fake' },
  },
);

await session.start();
await sleep(20);
await sleep(20);
transport.confirmAllMarks(); // the greeting finished playing
await sleep(20);

stt.last.emitEagerEndOfTurn('Do you have any openings this week?');
await sleep(5);
stt.last.emitEndOfTurn('Do you have any openings this week?');
await sleep(20);

// The caller "hears" about 2 s of the agent's answer, then talks over it.
const base = transport.totalQueuedMs - tts.last.totalMs;
transport.confirmPlayedMs(base + 2000);
await sleep(5);
stt.last.emitEndOfTurn('Actually, never mind, thanks');
await sleep(20);

const next = JSON.parse(chat.seenMessages[1]!) as Array<{ role: string; content: unknown }>;
console.log('history sent to the model on the next turn:');
for (const m of next) console.log(`  ${m.role.padEnd(9)} ${JSON.stringify(m.content)}`);

transport.confirmAllMarks();
await sleep(20);
await session.end('caller_hangup');

Output (recorded from a real run of this file):

output

history sent to the model on the next turn:
  user      "Do you have any openings this week?"
  assistant "We have openings Monday morning, [interrupted]"
  user      "Actually, never mind, thanks"
call ended (caller_hangup); barge-ins: 1

The assistant turn is stored as "We have openings Monday morning, [interrupted]": only the part the caller heard, with a marker, rather than the full sentence the model had planned.

Quickstart with real providers

Needs keys. Typechecked against the package; not executed in CI.

samples/src/voice-live.ts

// Needs real keys; typechecked here, not run. Bring your own wss server (ws shown).
import { WebSocketServer } from 'ws';
import { createCallSession, TwilioMediaTransport } from '@27colab/voice-engine';
import { createAnthropicChat, createDeepgramFluxFactory, createElevenLabsFactory } from '@27colab/voice-engine/providers';

const env = process.env;
const sttFactory = createDeepgramFluxFactory(env.DEEPGRAM_API_KEY!);
const ttsFactory = createElevenLabsFactory(env.ELEVENLABS_API_KEY!);
const chat = createAnthropicChat(env.ANTHROPIC_API_KEY!);

const wss = new WebSocketServer({ port: 8080, path: '/media' });

wss.on('connection', async (ws) => {
  const transport = new TwilioMediaTransport(ws);
  const start = await transport.started; // streamSid, callSid, custom <Parameter>s

  const session = createCallSession(
    { transport, sttFactory, ttsFactory, chat },
    { callId: start.callSid, startedAt: new Date() },
    { onEnd: (record) => console.log(`[${record.callId}]`, JSON.stringify(record.stats)) },
    {
      languages: ['en', 'es'],
      languagePolicy: 'follow-caller',
      greeting: { en: 'Hi, how can I help?', es: 'Hola, ¿en qué puedo ayudarle?' },
      system: (lang) => `You are a concise phone assistant. Reply in ${lang === 'es' ? 'Spanish' : 'English'}.`,
      llm: { model: 'claude-haiku-4-5', maxTokens: 300 },
      voice: {
        provider: 'elevenlabs',
        voices: { en: env.ELEVENLABS_VOICE_EN, es: env.ELEVENLABS_VOICE_ES },
        defaultVoiceId: env.ELEVENLABS_VOICE_EN!,
        model: 'eleven_flash_v2_5',
        outputFormat: 'ulaw_8000',
      },
      stt: { model: 'flux-general-multi', eagerEotThreshold: 0.6 },
    },
  );
  await session.start();
});

Point a Twilio number at TwiML that streams to this server (the URL must be wss://; use a tunnel for local development):

<Response><Connect><Stream url="wss://YOUR-HOST/media" /></Connect></Response>

To pass the caller’s number to your tools, add <Parameter name="from" value="{{From}}"/> inside <Stream>; it arrives as start.customParameters. The repository’s own examples/live-twilio.ts is a demo, not a deployable server: no auth, no signature validation, no concurrency limits.

API overview

Entry points (@27colab/voice-engine)

• createCallSession(deps, info, hooks, config) returns a session with start() and end(reason). deps is { transport, sttFactory, ttsFactory, chat }; hooks.onEnd(record, reason) receives a CallRecord (transcript, stats, events).
• TwilioMediaTransport(ws) with await transport.started giving streamSid, callSid, customParameters.
• CallSessionConfig: languages, languagePolicy ('fixed' or 'follow-caller'), greeting, system (string or function of language), llm, tools and executeTool (a tool result may set endCall: true), voice, stt, maxCallSeconds, turnTaking (speculative, bargeInMinWords, classify, playbackPaddingMs).

Interfaces (all in interfaces.ts): MediaTransport, SttFactory/SttStream, TtsFactory/TtsStream, ChatEngine. Implement them to swap any provider.

Building blocks, usable on their own: splitAtPlayedMs, computeSpokenText, AlignmentAccumulator, PlaybackTracker, rewriteOnBargeIn, classifyUtterance, detectLang, CallTimeline, selectVoice.

Adapters (@27colab/voice-engine/providers): createDeepgramFluxFactory, createElevenLabsFactory, createAnthropicChat.

Fakes (@27colab/voice-engine/testing): FakeTransport (confirmPlayedMs(ms), Twilio-style mark echo on clear), FakeStt (emitEagerEndOfTurn, emitTurnResumed, emitEndOfTurn), FakeTts (deterministic alignment), FakeChat (scripted turns and tool calls, optional gate to simulate latency).

Limits

• playedMs is a lower bound. It advances only at echoed marks, so resolution is the TTS chunk size. The rewrite errs toward “heard less”, never more.
• Alignment assumptions are unverified against the live API. Fields are read as ElevenLabs chars / charStartTimesMs / charDurationsMs; chunk timestamps may be chunk-relative or cumulative and a heuristic handles both. Verify against your account. Providers that rewrite text inside the alignment (such as number expansion) will mis-map.
• Without alignment the session falls back to a crude fixed ~15 chars/s estimate.
• Buffer-then-speak. Each agent turn is synthesized as one utterance; sentence-by-sentence streaming TTS is not implemented.
• Speculation trade-off. A speculative reply can begin speaking before the final end-of-turn; if cancelled, the caller may hear a clipped fragment. Disable with turnTaking.speculative: false.
• Heuristics, not models. The backchannel list (EN/ES) and detectLang are small word lists. Replace classify for anything serious.
• Providers are not tested in CI. Message shapes follow the vendors’ docs at the time of writing and can drift.
• Single process, in-memory. No reconnect, no resumption, no tenancy, persistence or metering.

Next

Assemble it with the other packages around your own agent.