BiReader / BiWriter

A feature rich binary reader and writer that keeps track of your position to quickly create file structures. Includes shared naming conventions, programmable inputs and advanced math for easy data conversions on low level parsing. Accepts Uint8Array, Buffer or a filePath with Streams.

Supported data types:

• Bitfields ([u]bit{1-32}{le|be}) 1-32 bit signed or unsigned value in big or little endian order
• Bytes ([u]int8, byte) 8 bit signed or unsigned value
• Shorts ([u]int16, word, short{le|be}) 16 bit signed or unsigned value in big or little endian order
• Half Floats (halffloat, half{le|be}) 16 bit decimal value in big or little endian order
• Integers ([u]int32, long, int, double{le|be}) 32 bit signed or unsigned value in big or little endian order
• Floats (float{le|be}) 32 bit decimal value in big or little endian
• Quadwords ([u]int64, quad, bigint{le|be}) 64 bit signed or unsigned in big or little endian
• Double Floats (doublefloat, dfloat{le|be}) 64 bit decimal value in big or little endian
• Strings (string) Fixed and non-fixed length, UTF, pascal, wide pascal. Includes all TextEncoder types

What's New?

v3

• Added enforceBigInt option for always returning a BigInt type on 64 bit reads, otherwise will return a number if integer safe.
• Added Browser, Node CommonJS and Node ESM modules.
• Added new BiReaderStream and BiWriterStream (Node only) that works without loading the whole file into memory. See documention for how to use.
• Added .deleteFile() and .renameFile(filePath).
• Added setter .strSettings for use with .str for easier coding.
• Added better options for extending array buffer when writing data with extendBufferSize.
• Consolidated all options argument into single object when creating class.
• Removed deprecated bireader and biwriter classes.
• Fixed standalone hexdump function.

v2

• Created new BiReader and BiWriter classes with get and set functions for easier coding.
• Marked bireader and biwriter as deprecated. Set to be removed next update.

v1

• Included math functions and value searches.
• Many bug fixes.

Installation

npm install bireader

Provides both CommonJS and ES modules for Node and Browser.

Example

Import the reader or writer. Create a new parser with the data and start parsing.

Includes presents for quick parsing or programmable functions (examples below).

import {BiReader, BiWriter} from 'bireader';

// read example - parse a webp file
function parse_webp(data){
  const br = new BiReader(data);
  br.strSettings = {length: 4};
  br.hexdump({suppressUnicode:true}); // console.log data as hex

  //         0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F  0123456789ABCDEF
  // 00000  52 49 46 46 98 3a 00 00 57 45 42 50 56 50 38 58  RIFF.:..WEBPVP8X
  // 00010  0a 00 00 00 10 00 00 00 ff 00 00 ff 00 00 41 4c  ..............AL
  // 00020  50 48 26 10 00 00 01 19 45 6d 1b 49 4a 3b cf 0c  PH&.....Em.IJ;..
  // 00030  7f c0 7b 60 88 e8 ff 04 80 a2 82 65 56 d2 d2 86  ..{`.......eV...
  // 00040  24 54 61 d0 83 8f 7f 0e 82 b6 6d e3 f0 a7 bd ed  $Ta.......m.....
  // 00050  87 10 11 13 40 3b 86 8f 26 4b d6 2a b7 6d 24 39  ....@;..&K.*.m$9
  // 00060  52 4f fe 39 7f 3b 62 4e cc ec 9b 17 31 01 0c 24  RO.9.;bN....1..$
  // 00070  49 89 23 e0 01 ab 52 64 e3 23 fc 61 db 76 cc 91  I.#...Rd.#.a.v..
  // 00080  b6 7d fb 51 48 c5 69 db 4c 1b 63 db b6 ed b9 6d  .}.QH.i.L.c....m
  // 00090  db be 87 8d b1 6d db 9e b6 cd a4 d3 ee 24 95 54  .....m.......$.T
  // 000a0  52 b8 8e 65 a9 eb 38 ce ab 52 75 9d 67 ff 75 2f  R..e..8..Ru.g.u/
  // 000b0  77 44 40 94 6d 25 6c 74 91 a8 88 86 58 9b da 6e  [email protected]%lt....X..n

  const header = {};
  header.magic = br.str;    // RIFF
  header.size = br.uint32le;               // 15000
  header.fileSize = header.size + 8;       // 15008
  header.payload = br.str;  // WEBP
  header.format = br.str;   // VP8X
  header.formatChunkSize = br.uint32le;    // 10
  switch (header.format){
    case "VP8 ":
        header.formatType = "Lossy";
        var read_size = 0;
        header.frame_tag = br.ubit24;
        read_size += 3;
        header.key_frame = header.frame_tag & 0x1;
        header.version = (header.frame_tag >> 1) & 0x7;
        header.show_frame = (header.frame_tag >> 4) & 0x1;
        header.first_part_size = (header.frame_tag >> 5) & 0x7FFFF;
        header.start_code = br.ubit24; // should be 2752925
        header.horizontal_size_code = br.ubit16;
        header.width = header.horizontal_size_code & 0x3FFF;
        header.horizontal_scale = header.horizontal_size_code >> 14;
        header.vertical_size_code = br.ubit16;
        header.height = header.vertical_size_code & 0x3FFF;
        header.vertical_scale = header.vertical_size_code >> 14;
        read_size += 7;
        header.VP8data = br.extract(header.formatChunkSize - read_size, true);
        break;
    case "VP8L":
        header.formatType = "Lossless";
        var read_size = 0;
        header.signature = br.ubyte; // should be 47
        read_size += 1;
        header.readWidth =  br.ubit14;
        header.width = header.readWidth+1;
        header.readHeight =  br.ubit14;
        header.height = header.readHeight+1;
        header.alpha_is_used =  br.bit1;
        header.version_number =  br.ubit3;
        read_size += 4;
        header.VP8Ldata = br.extract(header.formatChunkSize - read_size, true);
        break;
    case "VP8X":
        header.formatType = "Extended";
        br.big();              // switch to Big Endian bit read
        header.rsv = br.bit2;  // Reserved
        header.I = br.bit1;    // ICC profile
        header.L = br.bit1;    // Alpha
        header.E = br.bit1;    // Exif
        header.X = br.bit1;    // XMP
        header.A = br.bit1;    // Animation
        header.R = br.bit1;    // Reserved
        br.little();           // return to little
        header.rsv2 = br.ubit24;
        header.widthMinus1 = br.ubit24;
        header.width = header.widthMinus1 + 1
        header.heightMinus1 = br.ubit24;
        header.height = header.heightMinus1 + 1
        if(header.I)
        {
          header.ICCP = br.str;  // Should be ICCP
          header.ICCPChunkSize = br.uint32;
          header.ICCPData = br.extract(header.ICCPChunkSize, true);
        }
        if(header.L)
        {
          header.ALPH = br.str;  // Should be ALPH
          header.ALPHChunkSize = br.uint32;     // 4134
          header.ALPHData = br.extract(header.ALPHChunkSize, true);
        }
        if(header.A)
        {
          header.ANI = br.str;  // Should be ANIM or ANIF
          header.ANIChunkSize = br.uint32;
          if(header.ANI == "ANIM")
          {
            header.BGColor = br.uint32;
            header.loopCount = br.ushort;
            header.ANIMData = br.extract(header.ANIChunkSize, true);
          } else
          if (header.ANI == "ANIF")
          {
            header.FrameX = br.ubit24;
            header.FrameY = br.ubit24;
            header.readFrameWidth = br.ubit24;
            header.readFrameHeight = br.ubit24;
            header.frameWidth = readFrameWidth + 1;
            header.frameHeight = readFrameHeight + 1;
            header.duration = br.ubit24;
            header.rsv3 = br.ubit6;
            header.byte.B = br.bit1; // Blending
            header.byte.D = br.bit1; // Disposal
            header.frameData = br.extract(16, true);
            header.ANIFData = br.extract(header.ANIChunkSize, true);
          }
        }
        header.extFormatStr = br.str;
        header.extChunkSize = br.uint32;
        header.extData = br.extract(header.extChunkSize, true);
        if(header.E)
        {
          header.EXIF = br.str;  // Should be EXIF
          header.EXIFChunkSize = br.uint32;
          header.EXIFData = br.extract(header.EXIFChunkSize, true);
        }
        if(header.X)
        {
          header.XMP = br.str;  // Should be XMP
          header.XMPChunkSize = br.uint32;
          header.XMPMetaData = br.extract(header.XMPChunkSize, true);
        }
        break;
    default:
        header.data = br.extract(header.formatChunkSize, true);
        break;
  }
  br.finished();
  return header;
}

// write example - write a webp file from read data
function write_webp(data){
  const bw = new BiWriter(new Uint8Arry(0x100000)); // Will extends array as we 
  // write if needed by default
  bw.strSettings = {length: 4};
  bw.str = "RIFF";
  bw.uint32le = 0; // dummy for now, will be final size - 8
  bw.str = "WEBP";
  switch(data.format){
    case "VP8 ":
      bw.str = "VP8 ";
      bw.uint32le = data.VP8data.length;
      bw.ubit24 = data.key_frame;
      bw.ubit24 = data.start_code;
      bw.ubit16 = data.horizontal_size_code;
      bw.ubit16 = data.vertical_size_code;
      bw.overwrite(data.VP8data ,true);
      break;
    case "VP8L":
      bw.str = "VP8L";
      bw.uint32le = data.VP8Ldata.length - 4;
      bw.ubyte = 47;
      bw.ubit14 = data.width - 1;
      bw.ubit14 = data.heigth - 1;
      bw.ubit1 = data.alpha_is_used;
      bw.bit3 = data.version_number;
      bw.overwrite(data.VP8Ldata,true);
      break;
    case "VP8X":
      bw.str = "VP8X";
      bw.uint32le = 10;
      bw.big();
      bw.bit2 = 0;
      bw.bit1 = data.I;
      bw.bit1 = data.L;
      bw.bit1 = data.E;
      bw.bit1 = data.X;
      bw.bit1 = data.A;
      bw.bit1 = 0;
      bw.little();
      bw.ubit24 = data.rsv2;
      bw.ubit24 = data.width - 1;
      bw.ubit24 = data.height - 1;
      if(data.I)
      {
        bw.str = data.ICCP;
        bw.uint32 = data.ICCPData.length;;
        bw.replace(data.ICCPData, true);
      }
      if(data.L)
      {
        bw.str = data.ALPH;
        bw.uint32 = data.ALPHData.length;
        bw.replace(data.ALPHData);
      }
      if(data.A)
      {
        bw.str = data.ANI;
        bw.uint32 = data.ANIChunkSize;
        if(data.ANI == "ANIM")
        {
          bw.uint32 = data.BGColor;
          bw.ushort = data.loopCount;
          bw.replace(data.ANIMData);
        } else
        if (data.ANI == "ANIF")
        {
          bw.ubit24 = data.FrameX;
          bw.ubit24 = data.FrameY;
          bw.ubit24 = data.frameWidth - 1;
          bw.ubit24 = data.frameHeigh - 1;
          bw.ubit24 = data.duration;
          bw.ubit6  data.rsv3;
          bw.bit1 = data.byte.B;
          bw.bit1 = data.byte.D;
          bw.replace(data.frameData, true);
          bw.replace(data.ANIFData, true);
        }
      }
      bw.str = data.extFormatStr;
      bw.uint32 = data.extData.length;
      bw.replace(data.extData, true);
      if(data.E)
      {
        bw.str = data.EXIF;
        bw.uint32 = data.EXIFData.length;
        bw.replace( data.EXIFData, true);
      }
      if(data.X)
      {
        bw.str = data.XMP;
        bw.uint32 = data.XMPMetaData.length;
        bw.replace(data.XMPMetaData, true);
      }
      break;
    default:
      break;
  }
  bw.trim(); // remove any remaining bytes
  bw.goto(4);
  bw.uint32le = bw.size - 8; // write file size
  return bw.return();
}

Common Functions

Common functions for setup, movement, manipulation and math shared by both.

BiStreams

With 3.1 you can now use BiReaderStream and BiWriterStream (Node.js only) designed for larger files (or if you don't want to or need to load the whole file loaded to memory all at once).

Programmers Note: These are NOT data streams like fs.createReadStream where the whole file is streamed over time. It uses fs.openSync and fs.readSync and fs.writeSync to read and write just the data requested at the position requested at the time of operation, saving memory but costing processing power. This is mostly for legacy uses and niche cases as the newer versions of the Node.js have a much larger Buffer size over the older 4gig limit.

Streaming Caveats

• Naming: Same function naming applies to streamers as Common Functions section but the file is saved after every operation. BiReaderStream / BiReader and BiWriterStream / BiWriter are interchangeable when it comes to all functions and class objects names for easy use with type script.
• Writing: Unlike the other BiReader, all write functions will throw an error unless you switch to .writeMode(true) The file is read only until you do. Any write functions inside the reader will error beforehand.
• Large removal: When using any function that removes data from the file and would return a Buffer, if the size of the returned Buffer is outside of the Node max size for a Buffer, a new file will be made with the location and size concat to the name with a .removed file extention instead.

Bit field

Parse value as a bit field. There are 32 functions from bit1 to bit32 and can be signed or unsigned (with a u at the start) and in little or big endian order (be or le at the end).

Note: Remaining bits are dropped when returning to a byte read. Example, after using bit4 then ubyte, the read locations drops the remaining 4 bits after bit4 when reading ubyte. Any bit reading under 8 will always be unsigned.

Byte

Parse value as a byte (aka int8). Can be signed or unsigned (with a u at the start).

Short

Parse value as a int16 (aka short or word). Can be signed or unsigned (with a u at the start) and in little or big endian order (be or le at the end).

Half Float

Parse value as a half float (aka half). Can be in little or big endian order (be or le at the end).

Integer

Parse value as a int32 (aka int, long or double). Can be signed or unsigned (with a u at the start) and in little or big endian order (be or le at the end).

Float

Parse value as a float. Can be in little or big endian order (be or le at the end).

Quadword

Parse value as a int64 (aka quad or bigint). Can be signed or unsigned (with a u at the start) and in little or big endian order (be or le at the end).

Double Float

Parse value as a double float (aka dfloat). Can be in little or big endian order (be or le at the end).

Strings

Parse a string in any format. Be sure to use options object for formatting unless using a preset. Strings with larger than 1 byte character reads can use be or le at the end for little or big endian.

Presents include C or Unicode, Ansi and multiple pascals.

Acknowledgements

This project was born from the desire to have a single library that could both read and write in binary with common named functions. Having been using tools like Binary-parser, QuickBMS and 010 Editor in the past, I wanted something I could translate quickly to a Node app and then use in a web site without having to redo work.

I'm happy to connect and grow this library if others find it useful. Pull requests or bug reports are welcome!

License

MIT