Can we have an option for gatsby-source-filesystem where it doesn't compute MD5 hashes since this can be slow with big files?

[FraserThompson]

I have a gatsby site with many large files (like almost a 1000 MP3s which are up to 120mb or so and 16,000 JPGs which are like 2-3mb before processing) and currently gatsby-source-filesystem chokes computing MD5 hashes for all these big files. This results in issues with Node open file limits (presumably because when each file takes a long time to hash, it means more will be opened simultaneously) and extremely long build times: 100 minutes currently, with 87 of those being in the source and transform nodes build step.

As far as I can tell the only purpose of the MD5 hashing is to determine if the file has changed. So I forked gatsby-source-filesystem and made a variant where instead of hashing the file for the contentDigest it just concatenates the file size in bytes and the modification time in milliseconds into a string. I figured this would be a pretty damn accurate "estimate" of whether the file has changed.

This brought my source and transform nodes time in a smaller test environment with only 3663 JPGs and 284 MP3s down from 781 to 10 seconds.

Does this seem like something which could be added into gatsby-source-filesystem , maybe as an option? I understand MD5 hashing is more robust, but in the vast majority of cases it seems like this approach wouldn't cause any issues and would significantly lower build times for people building sites with large files.

I'm not sure how many sites would benefit tbh since mine is pretty extreme, and I'm happy to maintain my fork for myself, but maybe it could be useful as an option for those edge cases.

[LekoArts]

Hi!

Thanks for the suggestion 👍 We currently use md5-file (https://github.com/kodie/md5-file/blob/master/index.js) which uses the standard Node.js crypto to create the hash. It might not be the fastest tho

Coming from https://stackoverflow.com/questions/1655769/fastest-md5-implementation-in-javascript?answertab=modifieddesc#tab-top, maybe give https://github.com/Daninet/hash-wasm at try in your fork (or one of the accepted code snippets - but unsure about the LICENSE there)?

Generally speaking I'd think an option would be fine (especially since you pointed out that the methodology is used in production elsewhere) but my preference would be to optimize this for everyone. If a faster md5 is not sufficient, then we can add the option :)

[KyleAMathews]

so quick code:

const key = statObj.mtime + statObj.inode
let hash = await cache.get(key)
if (!hash) {
   hash = generateHash()
   await cache.set(key, hash)
}

const contentDigest = key + hash + pathToFile

[LekoArts]

@FraserThompson Would you be willing to work on this or should I take this? Don't want to step on your toes if you want to contribute :)

[FraserThompson]

@LekoArts Sure, I'd be happy to help. I'll start working on a pull request!

[FraserThompson]

Actually, here's an alternative idea for the "noHash" (working title) mode... I just noticed what gatsby-source-filesystem is doing for the contentDigest of directories:

    const contentDigest = createContentDigest({
      stats: stats,
      absolutePath: slashedFile.absolutePath,
    })

It passes an object containing the entire output of fs.stat and the absolute path to createContentDigest, which uses node-object-hash to return an MD5 hash of that object. This same approach could be probably be used for files?

I tested it and it's a tad slower than just using inode + mtime (14 seconds vs 10 seconds), but still a lot faster than hashing each entire file (781 seconds), so it still solves the problem of large files causing massive build times.

Thought it was worth bringing up in case we'd prefer to use an already implemented and well-tested method for making contentDigest's instead of adding another one (even if it isn't the absolute fastest)?

[LekoArts]

Thanks for bringing that up! Let's first use inode + mtime and if that causes problems in some way we could try the object hashing :) I'll also think that my PR #37433 gets merged today so you can then use hash-wasm's hashing function in your PR

And let's not use a negated option, we made that mistake in the past. We'd prefer "enable" options, so working title could be "fastHashing"

[LekoArts]

The PR #37464 will be available in Gatsby 5.5 🎉