Hash Worker

Introduce

中文文档

Hash-worker is a library for fast calculation of file chunk hashes.

It is based on hash-wasm and utilizes WebWorkers for parallel computation, which speeds up computation when processing file blocks.

Hash-worker supports two hash computation algorithms: MD5 and CRC32.

Both browser and Node.js are supported.

[!WARNING] The merkleHash computed by the Hash-worker is the root hash of a MerkleTree constructed based on file block hashes. Note that this is not directly equivalent to a hash of the file itself.

Install

$ pnpm install hash-worker

Usage

Global

<script src="./global.js"></script>
<script src="./worker/hash.worker.mjs"></script>
<script>
  HashWorker.getFileHashChunks()
</script>

The global.js and hash.worker.mjs are the build artifacts resulting from executing build:core in package.json.

The build artifacts are located in the packages/core/dist directory.

ESM

import { getFileHashChunks, destroyWorkerPool, HashChksRes, HashChksParam } from 'hash-worker'

function handleGetHash(file: File) {
  const param: HashChksParam = {
    file: file,
    config: {
      workerCount: 8,
      strategy: Strategy.md5
    }
  }

  getFileHashChunks(param).then((data: HashChksRes) => {
    console.log('chunksHash', data.chunksHash)
  })
}

/**
 * Destroy Worker Thread
 */
function handleDestroyWorkerPool() {
  destroyWorkerPool()
}

[!WARNING] If you are using Vite as a build tool and are experiencing dependency optimization issues with the hash-worker package, you can exclude the hash-worker package from dependency optimization in the vite.config.js file.

Attention: Old version of vite may not emit errors.

// vite.config.js
import { defineConfig } from 'vite';
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [ vue() ],
  optimizeDeps: {
    exclude: [ 'hash-worker' ] // new added
  }
})

Options

HashChksParam

HashChksParam is used to configure the parameters needed to calculate the hash.

| filed | type | default | description | |----------|--------|---------|--------------------------------------------------------------------------------------| | file | File | / | Files that need to calculate the hash (required for browser environments) | | filePath | string | / | Path to the file where the hash is to be calculated (required for Node environments) | | config | Config | Config | Parameters for calculating the Hash |

Config

| filed | type | default | description | |--------------------------|----------|----------------|-----------------------------------------------------------------------------------| | chunkSize | number | 10 (MB) | Size of the file slice | | workerCount | number | 8 | Number of workers turned on at the same time as the hash is calculated | | strategy | Strategy | Strategy.mixed | Hash computation strategy | | borderCount | number | 100 | The cutoff for the hash calculation rule in 'mixed' mode | | isCloseWorkerImmediately | boolean | true | Whether to destroy the worker thread immediately when the calculation is complete |

// strategy.ts
export enum Strategy {
  md5 = 'md5',
  crc32 = 'crc32',
  mixed = 'mixed',
}

When Strategy.mixed strategy is used, if the number of file fragments is less than borderCount, the md5 algorithm will be used to calculate the hash value to build the MerkleTree. Otherwise, it switches to using the crc32 algorithm for MerkleTree construction.

HashChksRes

HashChksRes is the returned result after calculating the hash value.

| filed | type | description | |------------|--------------|-------------------------------------------------------------------------| | chunksBlob | Blob[] | In a browser environment only, the Blob[] of the file slice is returned | | chunksHash | string[] | Hash[] for file slicing | | merkleHash | string | The merkleHash of the file | | metadata | FileMetaInfo | The metadata of the file |

FileMetaInfo

| filed | type | description | |--------------|--------|-------------------------------------------------| | name | string | The name of the file used to calculate the hash | | size | number | File size in KB | | lastModified | number | Timestamp of the last modification of the file | | type | string | file extension |

Benchmark (MD5)

| Worker Count | Speed | |--------------|-----------| | 1 | 234 MB/s | | 4 | 610 MB/s | | 8 | 851 MB/s | | 12 | 1011 MB/s |

• These measurements were made with Chrome v126 on a Zen3 Desktop CPU

LICENSE

MIT

Contributions

Contributions are welcome! If you find a bug or want to add a new feature, please open an issue or submit a pull request.

Author and contributors