@astronautlabs/wdi
v0.0.13
Published
A web-native media transport protocol to replace RTMP
Downloads
54
Maintainers
Readme
@/wdi » Reference Implementation
A web-native media transport protocol to replace RTMP
Home | Reference Implementation | Example
Alpha Quality
This software is very new and unstable. Use with caution, and avoid use in production without careful consideration. Major API changes may be made frequently.
WDI (the Web Device Interface) is a client/server media transport scheme based on WebRTC and WebSockets, this is the reference implementation for that specification. It can be used to easily add WDI support to any Javascript or Typescript app, both in the browser and on the server using Node.js.
Usage in the Browser
For the browser, it is sufficient to just install the reference implementation:
npm install @astronautlabs/wdi
Within the package you will find a class called WDIClient
that lets you easily
establish connections to WDI servers (via WebSockets and WebRTC).
let client = new WDIClient(`wss://mywdiserver.example.com:1234/path`);
await client.addStream(someMediaStream);
await client.connect();
You can also pass arbitrary metadata with your stream.
await client.addStream(someMediaStream, {
yourDataHere: 123
});
For convenience, you can pass a URL as the identity parameter instead.
await client.addStream(someMediaStream, 'https://example.com/');
WDI will expand this into a StreamIdentity
object like so:
{
"url": "https://example.com/"
}
Note that WDI itself assigns no special meaning to the identity data passed alongside a stream. The meaning needs to be agreed upon between the client and server applications.
WDI servers can also forward streams to the client. You can listen for incoming streams:
client.remoteStreamAdded.subscribe(remoteStream => {
let stream : MediaStream = remoteStream.stream;
// do something with the stream
});
The API for sending and receiving streams is isomorphic; it works the same on the server as on the client. Both sides can also request that a new stream be delivered.
client.acquireStream(identityMetadata)
You can receive acquired streams by listening to remoteStreamAdded
, or by awaiting
the promise returned by acquireStream
let remoteStream = await client.acquireStream(identityMetadata);
Conceptually, acquireStream
just sends a simple message to the server. It is
then up to the server to produce and forward a stream to send back to the client
for that request.
Usage on the Server
In addition to WDI itself, you will need implementations for WebRTC and
WebSockets. We recommend wrtc
(based on Google's libwebrtc
) and
ws
, but you can use any standards compliant implementations.
npm install @astronautlabs/wdi wrtc ws
You'll need to ensure that the RTCPeerConnection
class from your chosen
implementation is available in the global scope before trying to use WDI.
import { RTCPeerConnection } from 'wrtc';
global['RTCPeerConnection'] = RTCPeerConnection;
Your server application will be responsible for accepting incoming WebSocket
sessions and passing them to an instance of WDIServer
.
import { WDIServer } from '@astronautlabs/wdi';
import { WebSocket } from 'ws';
// Construct a WDI server instance and subscribe
// to incoming remote streams.
const wdiServer = new WDIServer();
wdiServer.remoteStreamAdded.subscribe(async remoteStream => {
let stream : MediaStream = remoteStream.stream;
// do something with the stream (perhaps guided by remoteStream.identity)
});
// Start a WebSocket server and pass clients to the WDI server
new WebSocket.Server({ port: 3000 })
.addListener('connection', socket => wdiServer.accept(<any>socket))
;
For metadata sent alongside the stream from the client,
see remoteStream.identity
.
The types of things you can do with the MediaStream depends on the WebRTC
implementation you have selected. The wrtc
package offers a set of classes
that let you send/receive raw audio/video data from streams sent over WebSockets.
For that, see RTCVideoSink
, RTCVideoSource
, RTCAudioSink
, RTCAudioSource
.
Note that operations on video frames (especially 1080p and up) are expensive, you should take care to minimize unnecessary copies or frame transformations. For more information see Performance.
Handling acquireStream
requests
You can handle acquireStream
requests (see above) on both clients and servers
by subclassing WDIClient
and/or WDIServer
and implementing provideStream
:
protected async provideStream(identity : StreamIdentity): Promise<NonNullable<MediaStream>>;
If your application can provide a stream that matches the given identity (however you decide to interpret it), return it and the stream will be automatically returned across WebRTC to the requestor.
If your application cannot service the request, throw an error. WDI will catch
the error and send it back to the client, where it will be cause the promise
returned by acquireStream()
to reject/throw.
Performance
The reference implementation is very small and does none of the heavy lifting. The performance you observe will be heavily dependent on what you do with the media streams being sent/received.
Transforming Video Frames
Raw video frames at modern resolutions are very large. Copying, converting, scaling and rotating video frames efficiently requires use of the host CPU's SIMD (single input, multple destination) instructions, or at least a highly optimized naive implementation. The easiest way to achieve this is to offload frame manipulation to WebAssembly (in the browser) and native C/C++/Rust addons (on Node.js).
The included server example utilizes
Astronaut Labs' libyuv
NPM package,
a native add-on for Node.js that exposes Chromium's libyuv
library.
Chromium's libyuv
is written in C++ and implements SIMD on x64, ARM and MIPS.
libyuv
provides color space conversions, scaling, and rotation routines for
raw video frames, like the kind produced by wrtc
.