Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gatsby-source-filesystem): Only generate hashes when a file has changed, and add an option for skipping hashing #37464

Conversation

FraserThompson
Copy link
Contributor

Description

Generating an MD5 hash in gatsby-source-filesystem is quite an expensive operation for large files, and can increase build times significantly, as well as introduce other issues with open file handles.

To mitigate this:

  • Cache the inode and modification time of the file and only re-generate a hash if this has changed
  • Add a "fastHash" option which skips MD5 hashing altogether, instead just using the inode and modification time as the contentDigest, which can be used on sites with many large files which still experience build time issues

Documentation

I've added a section to the README explaining the new option. Not sure if what I wrote goes into sufficient detail (or too much detail) though.

Related Issues

Addresses #12011
Fixes #37425

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jan 13, 2023
@LekoArts LekoArts added topic: source-plugins Relates to the Gatsby source plugins (e.g. -filesystem) and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Jan 16, 2023
Copy link
Contributor

@LekoArts LekoArts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I'll change the README a little bit more later, so code wise I only have some small requests :)

Copy link
Contributor

@LekoArts LekoArts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! A lot of people will benefit from that ❤️

@LekoArts LekoArts merged commit df58891 into gatsbyjs:master Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: source-plugins Relates to the Gatsby source plugins (e.g. -filesystem)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants