Gatsby 4 can't build large amount of CSV files, other Github solutions not working

[smf706fish]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

Have 50 CSV files for each state with location data and 1 big ~10K row CSV with page slugs to create pages. Gatsby can build locally on very small instances (like using just 1-4 state CSV file), Netlify can only build using 1 state CSV file.

Some of the things I have tried:

• Gatsby 4 cannot handle csv sourcing that was handled in Gatsby 3 (out-of-memory) #34336 making change to csv transformer with async/await
• NODE_OPTIONS=--max-old-space-size=8192 gatsby build
• Making sure case sensitivity checks out for netlify
• Incremental build using yarn run build and .github/workflows (which actually seemed to work if I just added 4 files at a time locally, not on netlify so worth looking more into)

All I care about is being able to get this thing deployed to Netlify, super frustrating, first Gatsby and React project and I want to wrap it up. Please clone and test different variations of this and see if you can deploy to Netlify. If you want to change the amount of state CSV files used to build do this ...

1. Add / delete both .csv and .js template for a state
2. Go into testing-slugs.csv and make sure slugs are only the states you are using (ex. if using alabama and alaska only, the testing-slugs.csv should contain lines 9119 - 9424 only)

Any help is very much appreciated, thanks

Reproduction Link

https://github.com/smf706fish/covid-resources-near-me-site

Steps to Reproduce

1. clone and npm install the repo
2. gatsby build / yarn run build (for incremental builds)

...

Expected Result

Build should complete, only around 10k total pages

Actual Result

Build fails pretty much all the time on Netlify

Local build fails a lot too, either Javascript heap out of memory OR Killed:9 (Seg fault I believe)

Environment

System:
    OS: macOS 12.1
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 16.13.0 - /usr/local/bin/node
    Yarn: 1.22.15 - /usr/local/bin/yarn
    npm: 8.5.1 - /usr/local/bin/npm
  Languages:
    Python: 2.7.18 - /usr/bin/python
  Browsers:
    Chrome: 98.0.4758.102
    Safari: 15.2
  npmPackages:
    gatsby: ^4.7.2 => 4.7.2
    gatsby-plugin-feed: ^4.7.0 => 4.7.0
    gatsby-plugin-gatsby-cloud: ^4.7.0 => 4.7.0
    gatsby-plugin-google-analytics: ^4.7.0 => 4.7.0
    gatsby-plugin-google-gtag: ^4.7.0 => 4.7.0
    gatsby-plugin-image: ^2.7.0 => 2.7.0
    gatsby-plugin-manifest: ^4.7.0 => 4.7.0
    gatsby-plugin-netlify: ^4.1.0 => 4.1.0
    gatsby-plugin-offline: ^5.7.0 => 5.7.0
    gatsby-plugin-react-helmet: ^5.7.0 => 5.7.0
    gatsby-plugin-sharp: ^4.7.0 => 4.7.0
    gatsby-plugin-sitemap: ^5.7.0 => 5.7.0
    gatsby-remark-copy-linked-files: ^5.7.0 => 5.7.0
    gatsby-remark-images: ^6.7.0 => 6.7.0
    gatsby-remark-prismjs: ^6.7.0 => 6.7.0
    gatsby-remark-responsive-iframe: ^5.7.0 => 5.7.0
    gatsby-remark-smartypants: ^5.7.0 => 5.7.0
    gatsby-source-filesystem: ^4.7.0 => 4.7.0
    gatsby-transformer-csv: ^4.7.0 => 4.7.0
    gatsby-transformer-remark: ^5.7.0 => 5.7.0
    gatsby-transformer-sharp: ^4.7.0 => 4.7.0
  npmGlobalPackages:
    gatsby-cli: 4.7.0

Config Flags

No response

[tyhopp]

Hi @smf706fish!

Sorry to hear you're running into an issue. I cloned your reproduction locally and was able to build it on my machine:

It's hard to say but looking at your env description above my guess is your local machine may not have enough memory to handle processing the number and size of csv files.

As a workaround I would recommend either trying different configuration options to load only what you need, pre-processing these files into a different format prior to building the site, or switching machines.

[smf706fish]

@tyhopp Hi, thanks for the reply and trying it locally on your machine. Just curious what machine/specs are you building with? Really appreciate the help and it is nice to know another build succeeded after so many of my failed attempts lol.