orca/mobile/packages/expo-two-way-audio

Orca Two Way Audio

Vendored Expo module for capturing and playing PCM audio data in the Orca mobile app (iOS and Android).

The aim of the module is to facilitate creating real-time conversational apps. The following features are provided:

• Request audio recording permissions
• Get clean (applying Acoustic Echo Cancelling) microphone samples in PCM format (1 channel 16 bit at 16kHz)
• Play audio samples in PCM format (1 channel 16 bit at 16kHz). Playback happens through main speaker unless external audio sources are connected.
• Provide volume level both for the input and output samples. Float between 0 and 1.
• [iOS only] Get microphone mode and prompt user to select a microphone mode.

Installation

npm i @orca/expo-two-way-audio

Usage

1. Request permissions for recording audio

   import {useMicrophonePermissions} from "@orca/expo-two-way-audio";
   
   const [micPermission, requestMicPermission] = useMicrophonePermissions();
   console.log(micPermission);
   

2. Initialize the module before calling any audio functionality.

   useEffect(() => {
       const initializeAudio = async () => {
           await initialize();
       };
       initializeAudio();
   }, []);
   
   

3. Play audio

   [!NOTE] The sample below uses the buffer module: npm install buffer

    import { Buffer } from "buffer";
   
    // As an example, play pcm data hardcoded in a variable.
    const audioChunk = "SOME PCM DATA BASE64 ENCODED HERE"
    const buffer = Buffer.from(audioChunk, "base64");
    const pcmData = new Uint8Array(buffer);
    playPCMData(pcmData);
   

4. Get microphone samples

   // Set up a function to deal with microphone sample events.
   // In this case just print the data in the console.
   useExpoTwoWayAudioEventListener(
       "onMicrophoneData",
       useCallback<MicrophoneDataCallback>((event) => {
           console.log(`MIC DATA: ${event.data}`);
       }, []),
   );
   
   // Unmute the microphone to get microphone data events
   toggleRecording(true);
   

Notes

Some audio features of expo-two-way-audio like Acoustic Echo Cancelling, noise reduction or microphone modes (iOS) don't work on simulator. Run the Orca mobile app on a real device to test these features.

# iOS
npx expo run:ios --device --configuration Release

# Android
npx expo run:android --device --variant release

For Android, the following permissions are needed: RECORD_AUDIO, MODIFY_AUDIO_SETTINGS. In Expo apps they can bee added in your app.json file:

expo.android.permissions: ["RECORD_AUDIO", "MODIFY_AUDIO_SETTINGS"]