npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

augnitorecorder

v1.0.34

Published

Audio recorder and streamer compatible with any browser

Downloads

10,245

Readme

Async JS Client for Voice Backend

Contents

Intent

The rationale behind an async implementation of the JS voice client is to decouple the following sets of tasks...

  • generating an audio stream
  • processing this audio stream (resample, conversion, etc.)
  • streaming to server over a websocket connection
  • handling ASR returned from server

...such that each of the above run independently in their own thread insulated from each other and communicate via message passing.

Components

The async JS client is composed of the following components, each portraying well-defined and exclusive roles.

Streamer

Streamer is the entrypoint of the async JS client. It's role is two-fold:

  • initialise the Worklet and the Executor components
  • manage communication and control between all components

When a new dictation session is started by the end-user, following steps are executed:

  1. Streamer is initialised along with the Worklet and the Executor
  2. It initialises the audio context post which audio generation begins
  3. It sends the audio packets to the Worklet for further processing
  4. It receives the processed and buffered packets from the Worklet and sends it to the Executor for ASR reception, processing and presentation.
  5. It repeats again from (3.)

When a dictation session is stopped by the end-user, following steps are executed:

  1. Streamer stops the audio context
  2. It sends DONE messages to Worklet and Executor asking them to close gracefully

NOTE: Audio source in the existing implementation is in the form of a recorded audio played in a loop. To support audio generation through a microphone, please uncomment pertinent code in Streamer.

Worklet

Worklet is employed mainly for audio processing, employing AudioWorklets. In this context, it can be used to resample the incoming audio stream before sending the same to the server. It also keeps the processed audio packets buffered for a short period of time before sending it back to the Streamer to decrease overhead of message passing between OS-level threads running itself and the Streamer.

Executor

This component deals with the following tasks:

  • obtain processed audio packets from the Streamer and stream it to the server over a websocket connection
  • obtain ASR from the server, process it well before pasting it to an editor screen

Executor manages a websocket connection with the server in a Web Worker. Audio packets received from the Streamer are buffered in a read-queue to be sent over to the server. It houses 3 daemons running periodically:

  1. Consumer: It consumes audio packets from the read-queue and streams it to the server over a websocket connection
  2. Healthcheck: It oversees the websocket connection and closes it forcefully when the server is unreachable.
  3. IdleThread: It keeps track of the fact that data is sent regularly over the websocket connection. If the connection remains idle for a given period of time, this daemon closes the connection gracefully thus freeing up network and server resources.

Usage

Initialise Audio Stream

This can be employed when a new dictation session begins by the user with an intent of creating a clinical document.

const recorderInstance = new AugnitoRecorder(
  {
    serverURL: WS_URL,
    enableLogs: false,
    isDebug: false,
    bufferInterval: 1,
    EOS_Message: "EOS",
    socketTimeoutInterval: 10000,
    shouldSendAudioDataSequence: false,
  },
  heavyOp
);
recorderInstance.toggleStartStopAudioStream(); //if you want to start/stop audio on button click
or;
recorderInstance.togglePauseResumeAudioStream(); //if you want to pause/resume audio on button click
  • WS_URL is the server websocket endpoint to which the client connects to, to stream audio.
  • heavyOp is a CPU-intensive operation which is run on the received ASR before displaying it on the editor. A dummy example is as follows:
  • enableLogs is set to true if you want to see logs in console else set to false
  • isDebug is set to true if you want to save recorded audio to a file else set to false
  • bufferInterval is set to any non negative integer, indicates the buffer size interval in seconds, default value is 1 sec
  • EOS_Message is set to any string that indicates EOS or can be undefined, default value is EOS
  • socketTimeoutInterval is set to any socket timeout interval as desired, default value is 10000
  • shouldSendAudioDataSequence is set to true if you want the audio packet sequence sent with every packet as header else set to false
const heavyOp = (text) => {
  let num = -1;
  let iters = 0;
  let sum = 0;
  for (let i = 0; i < text.length; i++) {
    sum += text.charCodeAt(i);
  }
  console.debug(`Running Heavy Operation for "${text}" with sum: ${sum}`);
  while (num != sum) {
    num = Math.floor(Math.random() * 100000);
    iters++;
  }
  console.log(`Iterations completed for sum(${sum}): ${iters}`);
  return text;
};

Audio Stream Pause and Resume

This can be employed when the user pauses and resumes the dictation session emulating a toggle of the microphone button.

recorderInstance.pauseAudio();

/*
Do something interesting here...
*/

recorderInstance.resumeAudio();

Stop Audio Stream

This can be employed when the user has completed their dictation session after finishing off compiling the medical document.

recorderInstance.stopAudio();

Library Callbacks

These callbacks can be employed to read socket connection state change event, session event, speech text output event or any other error event from the library.

recorderInstance.onStateChanged = (connected) => {
  //connected is true when socket is opened and false when socket is closed
};
recorderInstance.onSessionEvent = (response) => {
  //handle all meta events
};
recorderInstance.onError = (error) => {
  //handle any error message
};
recorderIns.onPartialResult = (text) => {
  // hypothesis text output generated as the user speaks
};
recorderInstance.onFinalResult = (textResponse) => {
  //text reponse can be parsed to read the speech output json
};

Compile library

Below steps to compile library and the output is generated in dist folder

cd AugnitoRecorderJS
npm run build

Running an Example

cd AugnitoRecorderJS
python3 -m http.server

Once done, fire up the webpage at: http://localhost:8000. The webpage has following controls for speech:

  • Start Stream: This begins playing the recorded audio which is streamed to the server and the ASR obtained pertaining to it is pasted on screen live. This control marks the start of a new session with the server and so a new WS connection is created for audio streaming and ASR reception. The recording and the ensuing ASR can be paused and resumed using the same control.
  • Stop Stream: This stops the recorded audio from playing marking the end of the client session. The connection with the server is severed when this control is engaged.
  • Go Crazy...: This control randomly starts, pauses, resumes and stops the audio streaming session in a loop. It employs the above two controls as it's building block and is a hands-free approach to communicate with the server in the form of a virtual user.