Banuba WebAR Agora extension

@banuba/agora-extension - Agora extension and video processor based on @banuba/webar.

Take a look at the @banuba/webar npm package and Banuba WebAR docs for a deep dive.

Quickstart

CDN

Via <script> tag:

<div id="video-container" style="width: 640px; height: 640px"></div>

<!-- Exposes AgoraRTC variable on window -->
<script src="https://cdn.jsdelivr.net/npm/[email protected]/AgoraRTC_N-production.min.js"></script>
<!-- Exposes BanubaSDK variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.js"></script>
<!-- Exposes BanubaAgoraExtension variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/index.browser.js"></script>

<script>
  const banuba = new BanubaAgoraExtension.Extension({
    clientToken: "xxx-xxx-xxx",
    locateFile: BanubaSDK.locateFile
  })
  AgoraRTC.registerExtensions([banuba])

  const processor = banuba.createProcessor()
  await processor.addModule(
    ...["face_tracker", "background"].map((m) => "https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/" + m + ".zip"),
  )
  const video = await AgoraRTC.createCameraVideoTrack()

  video.pipe(processor).pipe(video.processorDestination)
  video.play(document.querySelector("#video-container"))
</script>

Via <script type="module"> tag:

<div id="video-container" style="width: 640px; height: 640px"></div>

<!-- Exposes AgoraRTC variable on window -->
<script src="https://cdn.jsdelivr.net/npm/[email protected]/AgoraRTC_N-production.min.js"></script>
<!-- Exposes BanubaSDK variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.js"></script>

<script type="module">
  import * as BanubaAgoraExtension from "https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/index.js"

  const banuba = new BanubaAgoraExtension.Extension({
    clientToken: "xxx-xxx-xxx",
    locateFile: BanubaSDK.locateFile
  })
  AgoraRTC.registerExtensions([banuba])

  const processor = banuba.createProcessor()
  await processor.addModule(
    ...["face_tracker", "background"].map(
      (m) => "https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/modules/" + m + ".zip",
    ),
  )
  const video = await AgoraRTC.createCameraVideoTrack()

  video.pipe(processor).pipe(video.processorDestination)
  video.play(document.querySelector("#video-container"))
</script>

NPM

npm i --save agora-rtc-sdk-ng @banuba/agora-extension

import AgoraRTC from "agora-rtc-sdk-ng"
import { Extension } from "@banuba/agora-extension"
import data from "@banuba/webar/dist/BanubaSDK.data"
import wasm from "@banuba/webar/dist/BanubaSDK.wasm"
import simd from "@banuba/webar/dist/BanubaSDK.simd.wasm"
import FaceTracker from "@banuba/webar/dist/modules/face_tracker.zip"
import Background from "@banuba/webar/dist/modules/background.zip"

const banuba = new Extension({
  clientToken: "xxx-xxx-xxx",
  locateFile: {
    "BanubaSDK.data": data,
    "BanubaSDK.wasm": wasm,
    "BanubaSDK.simd.wasm": simd,
  },
})
AgoraRTC.registerExtensions([banuba])

const processor = banuba.createProcessor()
await processor.addModule(FaceTracker, Background)
const video = await AgoraRTC.createCameraVideoTrack()

video.pipe(processor).pipe(video.processorDestination)
// assume the page has a div element with id="video-container" and `width` and `height` set
video.play(document.querySelector("#video-container"))

Read the Integration tutorials intro for the explanation of the locateFile option.

Read the Bundlers section of the Integrations guides for more receipts for different bundlers.

API Reference

Extension

The Extension constructor accepts all the same options as the Player.create() method from Banuba WebAR SDK, most noticeable options are:

• clientToken - Banuba Client Token, required. See Obtaining Banuba Client token for details
• locateFile - specifies where to look for the Banuba WebAR’s .wasm and .data files, optional. Check the Integration tutorials intro for explanation.
• devicePixelRatio - image scaling factor for HiDPI devices, optional. If not specified it has a value of 1 as a reasonable default for WebRTC AR.

Extension.checkCompatibility()

Checks if the browser is capable to run the Extension and Processor:

const banuba = new Extension({
  clientToken: "xxx-xxx-xxx",
  locateFile: BanubaSDK.locateFile
})

if (!banuba.checkCompatibility()) alert("The browser does not support the extension :(")

Processor

The Processor instance created by the Extension.createProcessor() method has the following methods:

Processor.addModule(url | WebArModule)

Adds AR modules (neural networks) to the processor, so effects can consume them:

// This is a syntax sugar over
// import { Module } from "@banuba/agora-extension"
// const face_tracker = new Module("/path/to/face_tracker.zip")
// await processor.addModule(face_tracker)

await processor.addModule("/path/to/face_tracker.zip")

See the WebAR API Reference for the list of available AR modules and usage code examples.

Also you may want to check the docs of the Player.addModule()from Banuba WebAR SDK as the processor’s method is a simple wrapper around the Player.addModule().

Processor.applyEffect(url | WebArEffect)

Applies an AR effect to the processor:

// This is a syntax sugar over
// import { Effect } from "@banuba/agora-extension"
// const octopus = new Effect("/path/to/Octopus.zip")
// await processor.applyEffect(octopus)

await processor.applyEffect("/path/to/Octopus.zip")

You can download and use test effect from the Demo face filters page.

Also you may want to check the docs of the Player.applyEffect() from Banuba WebAR SDK as the processor’s method is a simple wrapper around the Player.applyEffect().

Processor.clearEffect()

Clears the applied effect if any:

await processor.applyEffect("/path/to/Octopus.zip")

// ... a few moments later ...

processor.clearEffect()

Processor.destroy()

Destroys the processor instance, frees up resources and allocated memory:

processor.destroy()

Try to not to forget to destroy the processor instance once it’s not needed to avoid accidental memory leaks.

The processor instance con not be re-used once it’s destroyed.

Obtaining Banuba Client token

Banuba Client token is required to get Banuba SDK Web AR working.

To receive a new trial client token please fill in the form on banuba.com website, or contact us via [email protected].

Resources and links

• Banuba WebAR overview

• Banuba WebAR demo applications

• Demo face filters

• Banuba WebAR JS API docs

• Changelog