unicode-shaper-rust

A Rust port of a subset of the functionality of International Components for Unicode (ICU). Supports right-to-left langauges like Arabic and Hebrew.

Purpose

This library is intended to be used in the browser, and can be compiled to WebAssembly. Mapbox uses emscripten but as you can see, the bundle size is 186.21kB [44.08kB (gzip)]. Because of it's size, it's imported as a separate module. Due to Rusts's first class WASM support the bundle size is 12kB bytes of wasm + 3.257 kB JS (example case). This is a 91.8065624832% reduction in size.

Using the unicode shaper

The unicode shaper exposes three functions for modules:

pub fn shape_arabic(dest: &mut [u16], tashkeel_flag: i8, shapeVars: UShapeVariables)

Takes an input string in "logical order" (i.e. characters in the order they are typed, not the order they will be displayed) and replaces unicodes like Arabic characters with the "presentation form" of the character that represents the appropriate glyph based on the character's location within a word.

pub fn process_bidi_text(input: &[u16]) -> Vec<u16>

Takes an input string with characters in "logical order", along with a set of chosen line break points, and applies the Unicode Bidirectional Algorithm to the string. Returns a new line in "visual order" (i.e. characters in the order they are displayed, left-to-right).

pub fn shape_unicode(source: &[u16], options: &u32) -> Vec<u16>

Combines shape_arabic and process_bidi_text to process a string with both shaping and bidirectional processing. Takes an input string with characters in "logical order", and applies the Unicode Bidirectional Algorithm to the string. Returns a new line with characters in "visual order" (i.e. characters in the order they are displayed, left-to-right) and replaces Arabic characters with the "presentation form" of the character that represents the appropriate glyph based on the character's location within a word.

isRTL(input: u16) bool

Check if the unicode character is right to left.

isCJK(input: u16) bool

Check if the unicode character is Chinese, Japanese, or Korean. Useful if you want to draw text as a vertical line for CJK characters.

Install

npm install unicode-shaper-rust
pnpm add unicode-shaper-rust
yarn add unicode-shaper-rust
bun add unicode-shaper-rust

Example

    import WASM from 'unicode-shaper-rust'
    const wasm = new WASM()
    const text = 'سلام۳۹' // [1587, 1604, 1575, 1605, 1779, 1785]
    const output = wasm.processString(input)
    console.log(output) // [1779, 1785, 65249, 65276, 65203]
    // => '۳۹ﻡﻼﺳ'

Prerequisites for Contributing

Bun

Follow the installation guides for Bun.

Rust

Follow the installation guides for Rust.

WABT

Follow the installation guides for WABT. Be sure to expose the tools in your path.

CPP TESTS: Build

1 Download and build ICU

# feel free to check https://github.com/unicode-org/icu/releases for latest version
# just be sure to use the RC `release-XX-rc/icu4c-XXrc-src.tgz`
wget https://github.com/unicode-org/icu/releases/download/release-75-rc/icu4c-75rc-src.tgz
tar xzf icu4c-75rc-src.tgz
rm icu4c-75rc-src.tgz

cd icu/source

./runConfigureICU --help
# MACOSX
CXXFLAGS=-std=c++20 ./runConfigureICU MacOSX --disable-renaming
# Linux
CXXFLAGS=-std=c++20 ./runConfigureICU Linux/gcc --disable-renaming

make clean
make -j4
sudo make install

2 Run comparison experiments via test.cpp

g++ -stdlib=libc++ -std=c++20 -w -fPIC -I/usr/local/include -L/usr/local/lib -licuuc test.cpp -o test

Supported Lanaguages

[x] Standard (Latin, Cyrillic, Greek, etc.) [x] Arabic [ ] Buginese [x] Hangul [x] Hebrew [ ] Indic: Bengali [ ] Indic: Devanagari [ ] Indic: Gujarati [ ] Indic: Gurmukhi [ ] Indic: Kannada [ ] Indic: Malayalam [ ] Indic: Odia [ ] Indic: Tamil [ ] Indic: Telugu [ ] Javanese [ ] Khmer [x] Lao [x] Myanmar [ ] Sinhala [x] Syric [x] Thaana [x] Thai [x] Tibetan