Web Voice Activity Detection (VAD)

Adaption of @ricky0123's vad library that slightly shifts the API to only support passing a media stream, addresses some Typescript issues and reduces the codebase where possible. The primary purpose of this adaption is to support realtime voice agents, such as those provided by Pipecat.

Getting started

npm install onnxruntime-web web-vad

Copy Silero model somewhere accessible

Ensure silero_vad.onnx (included in this repo here) is hosted somewhere accessible (e.g. a public / static path.)

Ensure audio worker is available globally

Browsers ensure worklets cannot be imported as modules for safety reasons. Either import it with your framework specific syntax (e.g. import AudioWorkletURL from "web-vad/dist/worklet.js?worker&url";) or include it manually in a <script> declaration (at a higher order.)

Example project

An barebones example is included in this repo:

1cd test-site
2yarn
3yarn run build # Copies onnx wasm to dist directory
4yarn run dev

Navigate to the URL shown in your terminal

Usage

1import { VAD } from "web-vad";
2import AudioWorkletURL from "web-vad/dist/worklet.js?worker&url";
3
4
5const localAudioTrack = ... // Get mic or other audio track
6const stream = new MediaStream([localAudioTrack!]);
7
8const vad = new VAD({
9    workletURL: AudioWorkletURL,
10    modelUrl: "path-to-silero.onnx",
11    stream,
12    onSpeechStart: () => {
13        console.log("speaking start");
14    },
15    onVADMisfire: () => {
16        console.log("misfire");
17    },
18    onSpeechEnd: () => {
19        console.log("speaking end");
20    },
21});
22
23// Initalize and load models
24await vad.init();
25
26// Start when ready
27vad.start();
28
29console.log(vad.state); 
30// > VADState.listening

Next / Vite support

Web VAD uses WASM files provided by ONNX. Whilst these can be loaded at runtime, it is recommended to copy these files to your build / deployment. Here is an example vite.config.js that copies these files across at build time:

1// vite.config.js
2
3export default defineConfig({
4  assetsInclude: ["**/*.onnx"],
5  server: {
6    headers: {
7      "Cross-Origin-Embedder-Policy": "require-corp",
8      "Cross-Origin-Opener-Policy": "same-origin",
9    },
10  },
11  resolve: {
12    alias: {
13      "@": path.resolve(__dirname, "./src"),
14    },
15  },
16  plugins: [
17    viteStaticCopy({
18      targets: [
19        {
20          src: "node_modules/onnxruntime-web/dist/*.wasm",
21          dest: "./",
22        },
23      ],
24    }),
25  ],
26});
27

Precaching models

Both the Silero.onnx and ONNX runtime wasms are quite large in size (~10mb). The VAD class exposes a static method for precaching these:

1import {VAD} from "web-vad";
2
3async function run() {
4  console.log("Precaching models");
5  await VAD.precacheModels("/silero-vad.onnx");
6  console.log("Download complete!");
7  
8  //...start()
9}

References

[1] Silero Team. (2021). Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier. GitHub, GitHub repository, https://github.com/snakers4/silero-vad, hello@silero.ai.

[2] Ricky Samore. Original code, https://github.com/ricky0123/vad, rickycontact9@gmail.com