Gatsby 4 cannot handle csv sourcing that was handled in Gatsby 3 (out-of-memory)

[witcradg]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

This issue is preventing me from upgrading a Gatsby 3 site to Gatsby 4.

I'm working on a site with something around 9k pages built with sourcing from a csv (tsv) file containing over 8k records.

I was able to build this with the default memory allocation when using Gatsby 3. (As I understand it, the default memory configuration for Gatsby 3 is 2GB). The site is in production using Netlify and Gatsby 3.

When I tried to upgrade to Gatsby 4, I discovered the data source I used could not be handled. This seems to be an out-of-memory condition resulting in the "Killed" termination even when increasing the memory allocated using NODE_OPTIONS=--max-old-space-size=8192

I've created a public repository demonstating the problem.
I started with the default Gatsby (4) starter. Then I only added the tsv file, installed gatsby-transformer-csv and modified gatsby-config.js to load the tsv.

The results:

gatsby develop --verbose
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "develop"
verbose loading local command from:
/home/dean/projects/gatsby/gatsby4-issue-demo/node_modules/gatsby/dist/commands/develop.js
verbose running command: develop

verbose Transition to "initializing"
success open and validate gatsby-configs, load plugins - 0.716s
success onPreInit - 0.006s
success initialize cache - 0.111s
success copy gatsby files - 0.086s
verbose Attaching functions to development server
success Compiling Gatsby Functions - 0.393s
success onPreBootstrap - 0.420s
verbose Creating 1 worker
verbose Transition to "initializingData"
success createSchemaCustomization - 0.027s
verbose Transition to "initializingData" > "sourcingNodes"
⠴ source and transform nodes
Killed

Reproduction Link

https://gitlab.com/witcradg/gatsby4-issue-demo

Steps to Reproduce

1. clone and npm install the repo
2. run gatsby develop

...

Expected Result

I expect the build to complete.

Actual Result

The build fails.

Environment

System:
    OS: Linux 5.4 Ubuntu 20.04.3 LTS (Focal Fossa)
    CPU: (4) x64 AMD Athlon 200GE with Radeon Vega Graphics
    Shell: 5.0.17 - /bin/bash
  Binaries:
    Node: 17.2.0 - ~/.nvm/versions/node/v17.2.0/bin/node
    Yarn: 1.22.4 - /usr/bin/yarn
    npm: 8.1.4 - ~/.nvm/versions/node/v17.2.0/bin/npm
  Languages:
    Python: 2.7.18 - /usr/bin/python
  Browsers:
    Chrome: 96.0.4664.110
    Firefox: 95.0.1
  npmPackages:
    gatsby: ^4.4.0 => 4.4.0
    gatsby-plugin-gatsby-cloud: ^4.4.0 => 4.4.0
    gatsby-plugin-image: ^2.4.0 => 2.4.0
    gatsby-plugin-manifest: ^4.4.0 => 4.4.0
    gatsby-plugin-offline: ^5.4.0 => 5.4.0
    gatsby-plugin-react-helmet: ^5.4.0 => 5.4.0
    gatsby-plugin-sharp: ^4.4.0 => 4.4.0
    gatsby-source-filesystem: ^4.4.0 => 4.4.0
    gatsby-transformer-csv: ^4.4.0 => 4.4.0
    gatsby-transformer-sharp: ^4.4.0 => 4.4.0
  npmGlobalPackages:
    gatsby: 4.4.0

Config Flags

No response

[joernroeder]

could be related to #34081

[witcradg]

could be related to #34081

Could the solution be related to the solution provided for #34336 ?

[joernroeder]

looks like the same issue as there is also no await on createNode https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-transformer-csv/src/gatsby-node.js#L66

see the comment by @KyleAMathews why this breaks in v4 here #34081 (comment)

[joernroeder]

@witcradg If you follow the structure of my fix PR for the json transformer you should be able to fix this one as well. https://github.com/gatsbyjs/gatsby/pull/34084/files

[witcradg]

That worked! I modified the source directly in the node_modules/gatsby-transformer-csv/gatsby-node.js and it worked. Now I just have to figure out how to make that work when I deploy to Netlify.

[KyleAMathews]

https://www.npmjs.com/package/patch-package 👍

[KyleAMathews]

Would be great if you could pr the improvement to the plugin as well!

[joernroeder]

@witcradg awesome! let me know if you need help with patching. would love to see the fix in the plugin and can help if needed.

[LekoArts]

Closing as duplicate of #33868 - we'd be happy to review a PR fixing this. Thanks!

[witcradg]

@witcradg awesome! let me know if you need help with patching. would love to see the fix in the plugin and can help if needed.
Thank you for your help on this @joernroeder. I'll try to get on it in the next day or two. I'm having other issues with deployment of the client's site so that has priority :-D

[witcradg]

Is it normal to close an issue before the change to the codebase has been merged when such a change is needed?

[joernroeder]

This issue is closed bc it's a duplicste of the mentioned issue #33868. Just head over and we'll consolidate the conversation to prep the fix over there.