How to block until unzip is done?

[YasharF]

I am trying to unzip a deflate file that is on disk and then do stuff with the unzipped file and need to have a way to block the calling code from running until the unzipping is done and the uncompressed file is written back to the disk. The zipped file is 20+MB and the unzipped file is 150+ MB so they can take a tad bit to finish. I have written the following and it creates the uncompressed data file on disk. What is the best way to block the response from the unzipFile function so when doing an await unzipFile it is resolved when the file unzipping is finished, and the file is written to the disk? Also, is there a better way to return how many files are in the zipped file?

Thanks

const fs = require('fs');
const { Unzip, UnzipInflate } = require('fflate');

async function unzipFile({ input, output }) {
	let count = 0;
	console.log(`Unzipping: ${input}`);
	const zipped = fs.readFileSync(input);
	const unzipper = new Unzip(stream => {
		const fileWriteStream = fs.createWriteStream(output);
		stream.ondata = (err, chunk, final) => {
			if (err !== null) throw err;
			fileWriteStream.write(chunk);
			if (final) fileWriteStream.end();
		};
		console.log(`Extracting: ${stream.name}`);
		stream.start();
		count++;
	});
	unzipper.register(UnzipInflate);
	unzipper.push(zipped, true);
	return count;
}

(async () => {
    console.log(`Number of files in the zip: ${await unzipFile({ input: './dbbackup.zip', output: 'live.db' })}`);
    //error if there was more than one file in the zip
    //then do stuff with the live.db like load it with sqlite3; but wait, got an error that the live.db binary doesn't exist
  })();

[101arrowz]

You can make your current code work by manually resolving a returned Promise; I can send code to do so if you'd like. However, since you're already loading all the data of the ZIP file into memory, I'd probably not bother with streaming the unzip process and instead use the buffer-based API.

const { promisify } = require('util');
const { unzip } = require('fflate');

const unzipAsync = promisify(unzip);

async function unzipFile({ input, output }) {
  const compressed = await fs.promises.readFile(input);
  const unzipped = await unzip(compressed);
  if (!unzipped[output]) {
    // file not found in archive
    throw new Error(`could not find ${output} in zip file ${input}`);
  }
  await fs.promises.writeFile(output, unzipped[output]);
  // Number of files in archive is number of keys in returned object
  return Object.keys(unzipped.length);
}

[101arrowz]

Also note that for this specific task, you might be better off using a purpose-built library like yauzl.

[Chargeuk]

Hi @101arrowz , I also have this question but with streaming in a browser. I can't work out how to know when all files in the zip file have been processed. Can you tell me how to do this please?

async function extractJsonFromZip(zipFileIn: File): Promise<File[]> {
  const extractedFilesLookup = new Map<string, Uint8Array[]>();
  const jsonFiles: File[] = [];

  const unzip = new Unzip((unzippedFile) => {
    if (unzippedFile.name.toLowerCase().endsWith(".json")) {
      unzippedFile.ondata = (err, chunk, final) => {
        console.log(
          `read ${chunk.byteLength / 1024}kb chunk of: ${unzippedFile.name}. Final=${final}\
        );
        if (extractedFilesLookup.has(unzippedFile.name)) {
          extractedFilesLookup.get(unzippedFile.name)?.push(chunk);
        } else {
          extractedFilesLookup.set(unzippedFile.name, [chunk]);
        }
        if (final) {
          // this is the final chunk of this extracted file
          const fileData = extractedFilesLookup.get(unzippedFile.name);
          if (fileData) {
            const morphFile = new File(fileData, unzippedFile.name);
            jsonFiles.push(morphFile);
            console.log(`extracted: ${unzippedFile.name}`);
          }
          extractedFilesLookup.delete(unzippedFile.name);
        }
      };

      console.log(
        `Reading: ${unzippedFile.name}. Compressed size: ${unzippedFile.size}. Decompressed size: ${unzippedFile.originalSize}`
      );

      if (unzippedFile.size !== undefined) {
        unzippedFile.start();
      }
    }
  });
  unzip.register(AsyncUnzipInflate);
  const zipFileSize = zipFileIn.size;
  let totalBytesRead = 0;
  const chunkSize = 1024 * 100; // 100k chunks;
  while (totalBytesRead < zipFileSize) {
    // eslint-disable-next-line no-await-in-loop
    const chunkBuffer = await zipFileIn
      .slice(totalBytesRead, totalBytesRead + chunkSize)
      .arrayBuffer();
    const chunk = new Uint8Array(chunkBuffer);
    unzip.push(chunk, totalBytesRead + chunkSize >= zipFileSize);
    totalBytesRead += chunkSize;
  }

  // \todo - don't return until verything is complete
  return jsonFiles;
}

[101arrowz]

You'll want to create a Promise chain that resolves when your files are done. You can do that by checking the final flag at each file.

const fileReads = [];
const unzip = new Unzip((unzippedFile) => {
  if (unzippedFile.name.toLowerCase().endsWith(".json")) {
    let onComplete;
    let onErr;

    unzipped.ondata = (err, chunk, final) => {
      if (err) onErr(err);
      // ...
      if (final) {
        // finish handling your file here
        // ...
        onComplete();
      }
    }

    // ...

    if (unzippedFile.size !== undefined) {
      fileReads.push(new Promise((resolve, reject) => { onComplete = resolve; onErr = reject; }));
      unzippedFile.start();
    }
  }
}

// ...

// Now wait for all the files to finish reading
await Promise.all(fileReads);
return jsonFiles;

[Chargeuk]

thank you!

[Di9]

if (unzippedFile.size !== undefined) {
      fileReads.push(new Promise((resolve, reject) => { onComplete = resolve; onErr = reject; }));
      unzippedFile.start();
    }

thank you, @101arrowz ! I'd like to see something like that in package's readme, because I've read all the docs several times and couldn't catch the finish trigger, event or function for an async unzip...