react-colorizer

The colorizer allows a "2.5D" representation of images generated in 3D software (Rhino and SketchUp). Elements of the image can be colored or have their visibility switched on/off. This solution allows 3D stacking to occur by supporting overlapping pixels through a secondary data image.

Multiple layers may be overlaid to form the final composite image.

Motivation

As a design firm we rely on a combination of 3D modelling tools and graphical editing tools to communicate effectively with high quality graphics. While comparable graphics may be possible directly with 3D scenes in the browser, we do not have the skill sets in house for optimizing these, nor the time to customize them on each project to match the look and feel of our other graphics. We need a solution that can leverage the same 2D graphics pipelines to generate high quality renderings, but that can also be manipulated in the browser to enable different combinations of elements and colors.

Features

• Setting element color (while maintaining 3D shading effects)
• Setting element visibility (allowing elements behind to show through)
• Setting element alpha
• Setting element patterns (e.g. used to show selection state)
• Click events on elements (based on top-most visible element at clicked pixel position)

Encoding the images

Scripts for encoding are available in Node.js and C# (used in Rhino). The Node.js scripts are currently here: react-fiddles/_maintain/colorizer-encodings

Example

https://github.com/sasakiassociates/react-colorizer-example

Installation

Run the following command to install react-colorizer in your project:

yarn add @strategies/react-colorizer

OR

npm install @strategies/react-colorizer

Usage

Versions

Version 1.x.x

This version has been replaced by version 2.x.x, but is maintained on a maintenance branch in order support early versions of Colorizer used in Dashi. This version uses a pixel encoding that includes lightness, idx and pattern. It proved not to be sufficiently extensible when looking to add more realistic rendering effects or outlines. This version used separate layers for graphical elements that needed to come in front-of or behind the encoded canvas as a separate canvas element (the LayerIdCanvas).

Version 2.x.x

Version 2.x.x uses a similar lookup, but instead of encoding the color information directly, we instead encode a reference to a position in a sprite sheet. This allows more extensibility including support for fully rendered graphics. A separate graphical canvas is no longer required.

Overlapping Tiles

In order to achieve a 2.5D image, we represent all objects as fully visible whether or not they are obstructed by other objects. This means we need to store more pixels than are visible at any given time.

We could store a full image for each object, but this is not feasible for large numbers of objects given limits on shader texture counts. Full images would also require loading a large number of empty pixels in most cases.

A more efficient solution is to break the input images into tiles and then store only those tiles that contain content. By using regular sized tiles, we can also pack these tightly into the resulting image. All that's needed is a way to relate those individual tiles back to its location in the original image.

ColorizerStateImages

The colorizer state image serves as the primary communication channel for sending instructions from the CPU to the GPU. It is designed to send data in a compact form that can be easily read by the shader. Since the number of images passed to a shader is limited, we combine multiple representations into a single image.

Bands + Channels

The left side of the canvas is reserved for bands and channels.

Each row can contain a number of bands. Each band communicates with one graphical tiled sprite sheet. Within each band, 2 default channels are provided

• Color (with alpha) - applied directly as a graphical color + alpha
• Encoded pattern data

In addition, any number of additional channels can be included.

Position/Id Lookup

The right side of the image is reserved for a lookup that relates the unique index of each object to the positions it occupies in the sprite sheet.

The sprite-sheet position is specified in the encoded images (either directly in the key.png or via a reference in the data.png stacks). It is referenced as a start position with a length of 1 (for consistency with the stack lookups). Once we have this position within the sprite-sheets, we can render the sprite graphics, however we need a way to relate the position within the sprite-sheet back to an actual object index. The position/id lookup provides the shader with that information.

See pseudo-code below for details on how the position/id lookup is used.

Pseudo-code for generating the graphics

This pseudo-code shows how the encoded images are used along with the colorizerStateImage to derive a final graphical state.

The code shows the case where the more complex stack lookup is used, but direct lookup is also supported:

read start, length encoding from key image to get unique reference to stack of tiles
for each stack pixel get tile offset (x,y)
grab the pixel in the position/id lookup at position x,y and decode the index value (corresponds to row in band info)

calc innerX + innerY (e.g. innerX = xPos % 64)

for each band
    if alpha > 0
        grab corresponding pixels at tile offset + innerX, innerY from sprite-sheet image
        apply logic based on channels and graphical properties
        store these pixels in the pixel stack

blend the pixel stack

Getting the Bands Back Together

For the graphics to appear correctly, stacking order is very important. This is similar to the order of layers in tools like PhotoShop. However, unlike typical graphics software where layer order is consistent across the entire image, the order of the layers is per-pixel. This allows objects to intersect or wrap each other while preserving the correct 3D illusion and is a key element in the ability of the Colorizer to achieve fake 3D over pure object or layer-based solutions.

Stacking respects the z-ordering of the objects at the given pixel position. The object stacking order (represented in the data png) is the primary sorting mechanism.

Multiple pixels may be returned for the same object (obtained from each of the sprite sheets) These pixels will be drawn in the order of the bands (leftmost first). So, for example, stroke elements should be placed after solid elements in order to be rendered on top.

Direct or Stacked

In a typical use case, the majority of pixels will be occupied by a single object. To avoid flooding the data image with unnecessary lookups, we use a flag to indicate whether we should refer to the lookup stack image - or go directly to the sprite sheets. Both lookups use the same format of a start position and a length.

Layer Tags

In some cases, the same object will be required to be represented at multiple points within the pixel stack - for example near a foreground object where another object comes between that object and its shadow. A 2 bit encoding is used for the "layer" tag:

0: No split
1: Foreground
2: Background

Extensible by Design

Any number of sprite sheets can be supported Any number of custom colors can be passed through to a shader In future additional graphical instructions such as hatch patterns could be passed within the colorizer state image.

License

MIT

Notes on package use

The package on npm is currently simply raw ts source code (not rolled up)

Previous notes on package use

Currently this package is designed to be used by either a Vite environment (directly from source) or as a package (can be locally linked). The package JSON needs to be modified for either use case! Very much a hacky, temporary solution.

local links (Vite)

  "types": "./src",
  "main": "./src"

package dist

run yarn start to start file watcher and use these:

    "types": "dist/index.d.ts",
    "main": "dist/index.js",
    "module": "dist/index.es.js",
    "jsnext:main": "dist/index.es.js"

Use publish-npm.js to publish the package to npm. It will automatically apply the correct settings for publishing... (NOTE: not working fully - see notes in file)