Hash Worker

Introduce

中文文档

Hash-worker is a library for fast calculation of file chunk hashes.

It is based on hash-wasm and utilizes WebWorkers for parallel computation, which speeds up computation when processing file blocks.

Hash-worker supports two hash computation algorithms: MD5 and CRC32.

Both browser and Node.js are supported.

Warning

The merkleHash computed by the Hash-worker is the root hash of a MerkleTree constructed based on file block hashes. Note that this is not directly equivalent to a hash of the file itself.

Install

$ pnpm install hash-worker

Usage

Global

<script src="./global.js"></script>
<script src="./worker/hash.worker.mjs"></script>
<script>
  HashWorker.getFileHashChunks()
</script>

The global.js and hash.worker.mjs are the build artifacts resulting from executing build:core in package.json.

The build artifacts are located in the packages/core/dist directory.

ESM

import { getFileHashChunks, destroyWorkerPool, HashChksRes, HashChksParam } from 'hash-worker'

function handleGetHash(file: File) {
  const param: HashChksParam = {
    file: file,
    config: {
      workerCount: 8,
      strategy: Strategy.md5
    }
  }

  getFileHashChunks(param).then((data: HashChksRes) => {
    console.log('chunksHash', data.chunksHash)
  })
}

/**
 * Destroy Worker Thread
 */
function handleDestroyWorkerPool() {
  destroyWorkerPool()
}

Warning

If you are using Vite as a build tool and are experiencing dependency optimization issues with the hash-worker package, you can exclude the hash-worker package from dependency optimization in the vite.config.js file.

Attention: Old version of vite may not emit errors.

// vite.config.js
import { defineConfig } from 'vite';
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [ vue() ],
  optimizeDeps: {
    exclude: [ 'hash-worker' ] // new added
  }
})

Options

HashChksParam

HashChksParam is used to configure the parameters needed to calculate the hash.

filed	type	default	description
file	File	/	Files that need to calculate the hash (required for browser environments)
filePath	string	/	Path to the file where the hash is to be calculated (required for Node environments)
config	Config	Config	Parameters for calculating the Hash

Config

filed	type	default	description
chunkSize	number	10 (MB)	Size of the file slice
workerCount	number	8	Number of workers turned on at the same time as the hash is calculated
strategy	Strategy	Strategy.mixed	Hash computation strategy
borderCount	number	100	The cutoff for the hash calculation rule in 'mixed' mode
isCloseWorkerImmediately	boolean	true	Whether to destroy the worker thread immediately when the calculation is complete

// strategy.ts
export enum Strategy {
  md5 = 'md5',
  crc32 = 'crc32',
  mixed = 'mixed',
}

When Strategy.mixed strategy is used, if the number of file fragments is less than borderCount, the md5 algorithm will be used to calculate the hash value to build the MerkleTree. Otherwise, it switches to using the crc32 algorithm for MerkleTree construction.

HashChksRes

HashChksRes is the returned result after calculating the hash value.

filed	type	description
chunksBlob	Blob[]	In a browser environment only, the Blob[] of the file slice is returned
chunksHash	string[]	Hash[] for file slicing
merkleHash	string	The merkleHash of the file
metadata	FileMetaInfo	The metadata of the file

FileMetaInfo

filed	type	description
name	string	The name of the file used to calculate the hash
size	number	File size in KB
lastModified	number	Timestamp of the last modification of the file
type	string	file extension

Benchmark (MD5)

Worker Count	Speed
1	234 MB/s
4	610 MB/s
8	851 MB/s
12	1011 MB/s

• These measurements were made with Chrome v126 on a Zen3 Desktop CPU

LICENSE

MIT

Contributions

Contributions are welcome! If you find a bug or want to add a new feature, please open an issue or submit a pull request.

Author and contributors