Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(gatsby): add two experimental flags to control cache chunk size #29039

Closed
wants to merge 1 commit into from

Conversation

pvdz
Copy link
Contributor

@pvdz pvdz commented Jan 15, 2021

This adds two experimental env variables through which we can control and/or get more insight into chunk size during cache persistence. This is an attempt to unblock people stuck by #23627 without sacrificing performance on a general scale.

If the default heuristic is causing you to see the "Assertion `length <= kMaxLength" Buffer assertion error during build then it's possible your site is very large and the heuristic for storing the cache is failing to pick an adequate enough chunk size. We cannot trap this error as it's a nodejs internal assertion error so hopefully it finds your way through this PR or linked issue.

The GATSBY_EXPERIMENTAL_CACHE_FORCE_CHUNK_SIZE flag sets the chunk size to the given non-zero positive integer value. What this means is that the nodes array, which is the biggest piece of memory to be stored, is stored in arbitrary chunks of this many nodes.

The GATSBY_EXPERIMENTAL_CACHE_CHUNK_GUESS_STEP flag does two things. Firstly, it allows you to control the heuristic by explicitly setting the step size for guessing how big the largest node is. Measuring the size of a node is a matter of v8.serialize(node).length and is very expensive at scale. By default we only scan every (nodes.length/step)-th node, a very arbitrary number. You can set the step all the way to 1 if you want for a completely accurate step size, at the price of time.

Additionally, if you set GATSBY_EXPERIMENTAL_CACHE_CHUNK_GUESS_STEP to any value it will print the chunk size stats that it gathers so you can use that to take further action.

@gatsbot gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Jan 15, 2021
@pvdz pvdz added topic: scaling builds and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Jan 15, 2021
@pvdz pvdz closed this Jan 28, 2021
@LekoArts LekoArts deleted the chunky branch April 23, 2021 11:06
@and-who and-who mentioned this pull request Jul 5, 2022
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant