MFX

In-browser video editing using WebCodecs, WebStreams, and WebGL → mfxlib.com

Usage

Decode MP4 Video -> zoom out -> encode to WebM keeping original Audio (vp8):

import {
  shaders,
  decode,
  encode,
  writeToFile,
  GLEffect,
} from "mfx";

// Files can be fetched locally
const file = await fetch("https://example.com/myvideo.mp4");

// Decode video container, returns each track as a WebStream
const { video, audio } = await decode(file, "video/mp4", {
  // Addresses Chromium WebCodecs bug, Set to true for HEVC or if "Can't readback frame textures" is thrown. Has ~10% performance impact.
  forceDecodeToSoftware: false,
});

// Create video pipeline taking raw frames through Web Streams
const videoOutput = video.pipeThrough(new GLEffect([ // Apply zoom out effect
  shaders.zoom({ factor: 0.5, x: 0.5, y: 0.25 }),
]));

// Readable WebStream
const outputStream = encode({
  mimeType: `video/webm; codecs="vp8,opus"`, // Transcode to WebM VP8 (video) and Opus (audio)
  video: {
    ...video.track.config, // Inherit configuration from input video
    stream: videoOutput,
    width: 640,
    height: 360,
    bitrate: 1e6,
    framerate: 30,
  },
  audio: {
    ...audio.track.config, // Inherit configuration from input audio
    stream: audio
  }
});

// Opens a save dialog in the browser
await writeToFile(outputStream, "output.webm");

// Alternatively you can pipeTo a fetch POST request
await fetch("example.com/save", {
  method: "POST",
  body: outputStream
});

Support Table

While codec support heavily depends on the browser, mfx aims to provide support for the following container / codec pairs:

| Container | Codec | Encode / Decode | | -------- | --------- | --------------- | MP4 | H.264/AVC | Both | | MP4 | H.265/HEVC | Decode | | MP4 | VP8 | Both | | MP4 | VP9 | Both | | WebM | VP8 | Both | | WebM | VP9 | Both | | | Audio | | MP4 | Opus | Both | | MP4 | AAC | Both | | WebM | Opus | Both | | WebM | Vorbis | Both |

Roadmap

Soon

• Fix Audio trimming to include subframes
• Functional APIs as abstractions to WebStreams
• Compositor texture alpha masks
  • Blend mode and opacity as compositor functions
  • Dynamic layer counts using GLSL generation / Or Canvas based
  • compose function to quickly merge
• Deprecate ExtendedVideoFrame
• API Documentation
  • Add note on VP9 probe
• GIF codec
  • decode: https://github.com/mattdesl/gifenc (or ImageDecoder)
  • encode: https://github.com/jnordberg/gif.js
• Testing: Source videos with frame duration > fps to showcase frameRate
• Run tests on Github actions
• Contribution Guide

Later

• Decode WebM via Matroska decoder to resolve issues of jswebm dependency (https://www.npmjs.com/package/ebml-stream), alternatively build libwebm for WebAssembly https://github.com/webmproject/libwebm/tree/main/webm_parser (e.g. https://github.com/ForeverSc/web-demuxer/blob/main/lib/web-demuxer/web_demuxer.cpp)
  • https://github.com/GoogleChromeLabs/webm-wasm/tree/master
• Utilize (https://github.com/dmnsgn/media-codecs?tab=readme-ov-file) for codec string generation
• Canvas frame generator
  • Add threejs demo
• SVG → Image → Frame animated pipeline
• Audio effect support
  • Audio waveform
• Color Grading
  • HSV support
  • Palette detection / Adjustment
  • Mask: Alpha/Green-screen
• Audio Containers (mp3, flac, wav, opus)
• Improve encoding performance by reverting fill behavior for nearly identical frames (high effort)
• Seek
  • Clips view (similar to QuickTime)
• Per frame quantizer
• Log WebCodec bug (https://chromium.googlesource.com/chromium/src/+/7786d34a4e7771725b85f354247ad1bb1902c556/third_party/blink/renderer/modules/webcodecs/video_encoder.cc#939)
• Reduce CPU → GPU → CPU copy times using texture atlas
• Benchmarks (during test) against ffmpeg (AVC https://trac.ffmpeg.org/wiki/Encode/H.264#FAQ and possibly WebM)
• Integrate GLSL debugger using Spector

Contributing

Install git-lfs to pull sample files:

brew install git-lfs

npm install
npm start

License

MIT License

Disclaimer on Sample Videos

Some test videos are sourced from coverr.co yet they are only used for testing and will not be built into the MFX package. These videos are under a permissive license (https://coverr.co/license).