Browser Gym

Browser Gym is a library for visualization and comparison of reinforcement learning algorithms. It gives you access to a set of standardized environments that you can control using JavaScript.

The core features of the library are

• [x] Access to multiple environments through a unified API
• [x] Procedurally generated environments
• [x] Rendering of environments using WebGL
• [x] Web worker support (and rendering communication done for you)
• [x] Implementation of common RL data structures and logic, i.e. replay buffers and samplers
• [x] Built on top of WebAssembly and Box2D

Environments

The library currently implements the following environments.

| Environment | Name | Agent count | Status | Example | | ------------ | ---------------- | --------------------- | :------------: | :----------------------------------------------------------------: | | Mountain car | MountainCarEnv | 1 | Fully featured | Link | | Pong | PongEnv | 2 | Fully featured | Link | | Cheetah | CheetahEnv | 1 | Reward missing | Link | | Maze | MazeEnv | 1 (eventually more) | Reward missing | Link |

Install

npm install browser-gym

Setup for web

We recommend to use Webpack for bundling. Setup Webpack using . To load the WebAssembly files, it's best to use file-loader.

{
   test: /\.wasm$/i,
   type: "javascript/auto",
   use: [
      {
         loader: "file-loader",
         options: { esModule: false }
      }
   ]
}

Setup for Node

To run Browser Gym with Node.js, import it and load the backend as follows.

const gym = require("browser-gym")
gym.loadBackend().then(() => {
   console.log("Gym backend loaded 🎉")
})

Now you are ready to go.

Build

First, make sure you have emscripten installed. You will need it to compile the WebAssembly binaries. To build browser-gym run the following commands.

git submodule update --init --remote
npm install

Example

Here is a quick example of how to use browser-gym using a web worker. Rendering is facilitated for you. All you need to worry about is supplying an actor. In this example we are using the keyboard as the "actor". The following is the code for the worker script.

index.worker.js:

import * as Gym from "browser-gym"
async function main() {
   // Load WebAssembly file
   await Gym.loadBackend({ locateFile: () => require("browser-gym/backend.wasm") })
   // Create environment and worker instance
   let env = new Gym.Environment({ name: "MountainCarEnv" })
   let app = new Gym.Worker(1.0 / 50.0, env) // world operates at 50Hz
   let action = 0
   // Listen for keystrokes and set action appropriately
   app.addEventListener("keyboard", (e) => {
      if (e.code === 65 || e.code === 37) action = 100 // Left and `A` key
      if (e.code === 68 || e.code === 39) action = -100 // Right and `D` key
   })
   // Main loop
   app.run(() => {
      env.agent.action[0] = action // Update action
      app.step() // Take world step
      app.render() // Render to screen by sending to host
   })
}
console.log("Hi from worker!")
main()

To run the worker and render the environment to screen, you need to create a host which runs on the main thread. The RenderHost class helps you with this. Have a look at the following script for that.

index.js:

import { RenderHost } from "browser-gym/renderer"
import Worker from "./index.worker.js"

const renderer = new RenderHost({ worker: new Worker() }) // Create render host and worker instance
renderer.setup(document.getElementById("renderer")) // Pass in the canvas element
renderer.hydrate({
   // Start the worker
   keyPropagation: [37, 39, 65, 68], // Forward key codes for left and right keys to worker
})

And add the following html:

<div>
   <canvas id="renderer" />
</div>

For deep reinforcement learning you would likely want to create a replay buffer which stores experiences of an agent. For that we provide a Gym.ReplayBuffer class that encapsulates this logic directly in WebAssembly. Simply run the following snippet after you created an environment:

const replayBuffer = new Gym.ReplayBuffer(stateSize, actionSize, bufferCapacity)
replayBuffer.attach(env) // where _env_ is the environment instance

Now you can sample batches using the replayBuffer.sample() method and retrieve the size of the buffer using replayBuffer.size.

For more information and an example how to setup Webpack, take a look at the example folder.