p5-holoplay

Allows to create holographic p5js sketches and sends them to Looking Glass holographic displays.

See a quick demonstration here.

Prerequisites

• A Looking Glass holographic display
• Install Looking Glass Bridge to communicate with the device
• Plug the device to your computer both by USB and HDMI
• Install node on your computer

Getting started

You can create your own project by modifying the included projects sample-p2d and sample-webgl. See instructions on how to get them running in their own README.md.

Or you can integrate this in your own projects by installing the module

npm i p5-holoplay

and including it in your javascript as a CommonsJS module (it will need bundling to run on the browser)

const { sketchP2d, sketchWebgl } = require('p5-holoplay');

Then prepare your setup, draw, etc. functions as explained below and pass them to the chosen mode, P2D or WEBGL.

How to create p5 holograms

This project integrates p5js in instance mode. This means the usual p5 methods and properties are not in the global namespace (available everywhere), but bundled in a variable.

You have to choose one of two drawing modes.

• sketchP2d: Uses p5's P2D renderer. It draws flat (2D) shapes to 3D space by specifying a depth value for each block of drawing code.
• sketchWebgl: Uses p5's WEBGL renderer. It draws 3D shapes and uses lights to show volumes.

The basic usage of both modes is similar. The sketchP2d or sketchWebgl function accepts an object with functions named after the typical p5js functions (preload, setup, draw) and an options object.

sketchp2d({ preload, setup, draw, options });

These functions will pass the p variable so you can use the p5 methods and properties. So instead of

function setup() {
  background(255);
}

You can do

const setup = p => {
  p.background(255);
};

The functions you provide to each mode are significantly different:

sketchP2d

• setup provides p, error, meta (additional data for advanced work). Creates a canvas of the necessary dimensions (you don't need to create it yourself). Similar to setup(). Runs automatically once at the beginning of the life cycle.
• draw provides p, add (essential function to add layers), meta. Runs every frame. Allows to add layers in the form of a function and its depth. P5 work commonly done in p5 draw() should work here. Layer functions are run multiple times (one for every camera view, 48 in the case of the Looking Glass Portrait), while the rest of the code is only ran once per frame. So take that into account when deciding what to put inside of an "add" function (probably not an incrementing counter). Similar to draw().

"Add" functions must receive a drawing function and a depth value (positive means further from the viewer, negative is closer to them).

const draw = (p, add) => {
  add(() => {
    p.fill(255, 0, 0);
    p.ellipse(0, 0, 100);
  }, 100);
};

Depth values between 100 and -100 seem to draw layers with noticeable depth but more or less within the frame of the device. Larger values will produce more impressive effects, but also blurrier graphics (which might be fine, creatively). If depth is omitted, Infinity will be assumed, which is meant for functions that don't rely on depth, like p.background().

Layers are not necessarily drawn in the order your add them. They are drawn from farther to nearer, so don't expect changes you make to things like stroke, fill and other to persist between added layers. Set all you need for each layer within its own function. Each added function can be thought of like a mini p5 draw function.

This syntax can get complicated quickly, but is an alternative to thoroughly modifying p5 to make it generate the 'quilt'.

sketchWebgl

• setup provides p, error, meta. Runs automatically once at the beginning of the life cycle, but p here refers to the preview window, not the canvases where the actual shapes are drawn. This means you can include here things that need to happen only once (like lifecycle events, mouse/keyboard interactions...), but not those that affect the drawings directly (like colors, strokes, fills, fonts...). Similar to setup().
• setupEach provides p, error, meta. Creates all the necessary canvases to build the 3D images from multiple camera perspectives. Runs multiple times due to the complex underlying structure. Specify permanent drawing settings here, like potentially colors, strokes, fills, fonts... Similar to setup().
• preDraw provides p, meta. Runs once per every new frame of the preview window. It does not affect the final drawn image. You can include code that should run each frame but is not related to each camera view. For example, store mouse positions only once per frame to use in the draw function. Similar to draw().
• draw provides p, meta. Runs multiple times per frame (once for each virtual camera perspective). Place here the geometries you want to draw, camera work, and style data (stroke, colors...) that changes based on other inputs. The frame and lights reset every frame, so don't rely on them being preserved. Similar to draw().

let ballX, ballY;

const preDraw = p => {
  // Store the same mouse positions for all the camera views
  ballX = p.mouseX - p.width / 2;
  ballY = p.mouseY - p.height / 2;
};

const draw = p => {
  normalMaterial();
  sphere(ballX, ballY, 50);
};

Common

Additionally, a preload function can be passed to both modes.

• preload receives p. Useful for loading things like images or fonts before running the p5 sketch. Similar to preload().

meta

A meta object is passed with some functions to enable advanced work. The object can include the following (but not always does, so check the data before using it):

• device: Hardware data from holoplay-core.
• viewerFrame: Frame number of the Looking Glass device. Potential replacement of p5's frameCount Useful because it does not update at p5's normal frame rate.
• previewFrame: (WEBGL only) Frame number of the preview canvas. Potential replacement of p5's frameCount Useful because it does not update at p5's normal frame rate, nor at the Looking Glass rate.
• cam: (WEBGL only) p5.Camera of the view that is currently being drawn. It allows camera transformations like position and field of view.
• millis: (WEBGL only) Time since the stetch started running, in milliseconds. Potential replacement of millis(), which would change while the multiple views are being drawn, this providing unexpected results.
• quilt: Reference to the p5.Graphics holding the 'quilt' being sent to the Looking Glass device. Useful for doing things with that specific canvas, like saving it to a file with [].save()](https://p5js.org/reference/#/p5/save).
• preview: Reference to the p5.Renderer holding the preview visualization.
• p: Reference to the p5 sketch. In WEBGL mode, it can be more representative than the main p of the draw function, as that refers to the camera perspective being drawn at any given time. So use it for things like sketch width, mouse position...

Other p5 functions

Other p5 functions that would be normally set globally should be set in the setup once the p variable is available.

For example, instead of

function mouseClicked() {
  console.log(mouseX, mouseY);
}

you would do

setup = p => {
  p.mouseClicked = () => {
    console.log(p.mouseX, p.mouseY);
  };
};

For more p5 methods and properties, see the reference.

Options

An options object can also be passed to the main functions.

const options = {
  wigglePreview: false,
  previewQuilt: true,
  adaptSize: false,
  depth: 120
};

• wigglePreview default true: The preview in the browser window switches between displaying two of the camera perspectives to convey a sense of depth even if you are not looking at the holographic display.
• previewQuilt deafult false: See the entire 'quilt' with multiple camera views in your browser, instead of the simplified preview.
• adaptSize (P2D only) default true: Adapts the size of layers to increase perceived depth with some conic perspective. This is not geometrically accurate. Disabling it increases precision when drawing elements. For example, if adaptSize is enabled and you draw a dot at the coordinates 0,0 of a layer with negative depth, the dot will be outside of the screen due to the conic perspective.
• depth (WEBGL only) default 100: Increases or decreases the depth perception by drawing a wider or narrower range of virtual camera positions.

How does this work?

The code generates a Quilt for each frame. That is, a series of camera perspectives (48 in the case of the Looking Glass Portrait) to be sent to the holographic display.

Notes

• Currently frame rates are very low. The canvas (which is large due to all the hidden camera views) takes time to convert to PNG so it can be sent to the device.
• Some p5 methods and properties like frameRate, frameCount and probably more do not work as expected.
• Multiple hacky and experimental JavaScript and P5 features are used. This will not work on all browsers.

TODO

• Simplify WEBGL mode to a single canvas with p5.EasyCam Multiview: https://github.com/freshfork/p5.EasyCam/blob/master/examples/MultiView/MultiView.js
• Optimise fps somehow? Converting the canvas takes time, and HoloPlay Core takes time to receive it and confirm https://github.com/Looking-Glass/HoloPlayJS_Issues/issues/19
• See if holoplay-core can be updated so "message" is not always printed https://github.com/Looking-Glass/HoloPlayJS_Issues/issues/18
• Test on multiple devices, browsers, etc.
• Use mathematically correct formula for adaptSize

About

If you liked this you might like some of my other projects.