Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: @redwoodjs/testing/config/jest/api causes a memory leak on api side tests #6322

Open
1 task done
jmackie opened this issue Aug 30, 2022 · 9 comments
Open
1 task done

[Bug]: @redwoodjs/testing/config/jest/api causes a memory leak on api side tests #6322

jmackie opened this issue Aug 30, 2022 · 9 comments
Labels
bug/needs-info More information is needed for reproduction

Comments

@jmackie
Copy link
Contributor

jmackie commented Aug 30, 2022 •
edited
Loading

What's not working?

At Nous our Redwood test suite has become incredibly resource hungry, to the point that we can't actually run the entire suite on our local machines. We can just about run it on CI by splitting the work between multiple GHA jobs.

After some investigation it's clear that we have a huge memory leak. And one of the contributors seems to be @redwoodjs/testing/config/jest/api, which leaks memory even with the most trivial test suite.

Apologies if this is something you're already aware of - I searched for similar issues but no luck 🙈

Otherwise our experience with Redwood remains very positive, thank you team for all your hard work 🙏

How do we reproduce the bug?

https://github.com/project-nous/redwood-jest-memory-leak-repro

What's your environment? (If it applies)

No response

Are you interested in working on this?

  • I'm interested in working on this
@jmackie jmackie added the bug/needs-info More information is needed for reproduction label Aug 30, 2022
@redwoodjs-bot redwoodjs-bot bot added this to Main Aug 30, 2022
@redwoodjs-bot redwoodjs-bot bot moved this to Triage in Main Aug 30, 2022
@zygopleural
Copy link
Contributor

zygopleural commented Aug 30, 2022 •
edited
Loading

From my own debugging, with the standard repro I get

 PASS   api  api/src/7.test.ts (158 MB heap size)
 PASS   api  api/src/14.test.ts (204 MB heap size)
 PASS   api  api/src/6.test.ts (286 MB heap size)
 PASS   api  api/src/15.test.ts (364 MB heap size)
 PASS   api  api/src/17.test.ts (444 MB heap size)
 PASS   api  api/src/4.test.ts (523 MB heap size)
 PASS   api  api/src/20.test.ts (354 MB heap size)
 PASS   api  api/src/16.test.ts (431 MB heap size)
 PASS   api  api/src/5.test.ts (510 MB heap size)
 PASS   api  api/src/13.test.ts (591 MB heap size)
 PASS   api  api/src/0.test.ts (670 MB heap size)
 PASS   api  api/src/9.test.ts (508 MB heap size)
 PASS   api  api/src/8.test.ts (589 MB heap size)
 PASS   api  api/src/12.test.ts (668 MB heap size)
 PASS   api  api/src/1.test.ts (747 MB heap size)
 PASS   api  api/src/19.test.ts (630 MB heap size)
 PASS   api  api/src/3.test.ts (703 MB heap size)
 PASS   api  api/src/10.test.ts (781 MB heap size)
 PASS   api  api/src/2.test.ts (862 MB heap size)
 PASS   api  api/src/11.test.ts (940 MB heap size)
 PASS   api  api/src/18.test.ts (709 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        12.48 s

If I comment everything out in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:

 PASS   api  api/src/7.test.ts (75 MB heap size)
 PASS   api  api/src/18.test.ts (84 MB heap size)
 PASS   api  api/src/20.test.ts (83 MB heap size)
 PASS   api  api/src/19.test.ts (91 MB heap size)
 PASS   api  api/src/2.test.ts (92 MB heap size)
 PASS   api  api/src/15.test.ts (85 MB heap size)
 PASS   api  api/src/9.test.ts (92 MB heap size)
 PASS   api  api/src/14.test.ts (92 MB heap size)
 PASS   api  api/src/13.test.ts (100 MB heap size)
 PASS   api  api/src/12.test.ts (90 MB heap size)
 PASS   api  api/src/3.test.ts (98 MB heap size)
 PASS   api  api/src/11.test.ts (99 MB heap size)
 PASS   api  api/src/1.test.ts (106 MB heap size)
 PASS   api  api/src/10.test.ts (108 MB heap size)
 PASS   api  api/src/4.test.ts (115 MB heap size)
 PASS   api  api/src/0.test.ts (98 MB heap size)
 PASS   api  api/src/16.test.ts (106 MB heap size)
 PASS   api  api/src/8.test.ts (105 MB heap size)
 PASS   api  api/src/17.test.ts (113 MB heap size)
 PASS   api  api/src/6.test.ts (115 MB heap size)
 PASS   api  api/src/5.test.ts (123 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        1.089 s, estimated 13 s

If I just comment out the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:

 PASS   api  api/src/7.test.ts (131 MB heap size)
 PASS   api  api/src/1.test.ts (196 MB heap size)
 PASS   api  api/src/19.test.ts (261 MB heap size)
 PASS   api  api/src/6.test.ts (321 MB heap size)
 PASS   api  api/src/4.test.ts (380 MB heap size)
 PASS   api  api/src/3.test.ts (446 MB heap size)
 PASS   api  api/src/8.test.ts (297 MB heap size)
 PASS   api  api/src/9.test.ts (356 MB heap size)
 PASS   api  api/src/13.test.ts (421 MB heap size)
 PASS   api  api/src/12.test.ts (481 MB heap size)
 PASS   api  api/src/16.test.ts (547 MB heap size)
 PASS   api  api/src/5.test.ts (410 MB heap size)
 PASS   api  api/src/20.test.ts (471 MB heap size)
 PASS   api  api/src/14.test.ts (535 MB heap size)
 PASS   api  api/src/10.test.ts (594 MB heap size)
 PASS   api  api/src/17.test.ts (659 MB heap size)
 PASS   api  api/src/15.test.ts (520 MB heap size)
 PASS   api  api/src/18.test.ts (583 MB heap size)
 PASS   api  api/src/11.test.ts (643 MB heap size)
 PASS   api  api/src/2.test.ts (707 MB heap size)
 PASS   api  api/src/0.test.ts (767 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        9.762 s, estimated 11 s

If I keep the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js commented out and update some of the imports, namely:

-const { getConfig, getDMMF } = require('@prisma/sdk')
+const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands')

-const { setContext } = require('@redwoodjs/graphql-server')
+const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext')
 const { getPaths } = require('@redwoodjs/internal/dist/paths')
-const { defineScenario } = require('@redwoodjs/testing/dist/api')
+
+const { defineScenario } = require('../../../dist/api/scenario')

I get:

 PASS   api  api/src/7.test.ts (94 MB heap size)
 PASS   api  api/src/8.test.ts (113 MB heap size)
 PASS   api  api/src/19.test.ts (114 MB heap size)
 PASS   api  api/src/16.test.ts (134 MB heap size)
 PASS   api  api/src/5.test.ts (153 MB heap size)
 PASS   api  api/src/12.test.ts (137 MB heap size)
 PASS   api  api/src/2.test.ts (157 MB heap size)
 PASS   api  api/src/4.test.ts (174 MB heap size)
 PASS   api  api/src/10.test.ts (192 MB heap size)
 PASS   api  api/src/14.test.ts (210 MB heap size)
 PASS   api  api/src/6.test.ts (228 MB heap size)
 PASS   api  api/src/9.test.ts (246 MB heap size)
 PASS   api  api/src/1.test.ts (264 MB heap size)
 PASS   api  api/src/15.test.ts (282 MB heap size)
 PASS   api  api/src/11.test.ts (300 MB heap size)
 PASS   api  api/src/17.test.ts (318 MB heap size)
 PASS   api  api/src/20.test.ts (336 MB heap size)
 PASS   api  api/src/13.test.ts (354 MB heap size)
 PASS   api  api/src/18.test.ts (372 MB heap size)
 PASS   api  api/src/0.test.ts (390 MB heap size)
 PASS   api  api/src/3.test.ts (408 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        3.174 s, estimated 4 s

i.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.

@jmackie
Copy link
Contributor Author

jmackie commented Aug 30, 2022

Not sure if it's related, but it's also slightly alarming that this trivial test suite is taking >10s 😅

@zygopleural
Copy link
Contributor

zygopleural commented Aug 30, 2022 •
edited
Loading

If I keep the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js commented out and update some of the imports, namely:

-const { getConfig, getDMMF } = require('@prisma/sdk')
+const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands')

-const { setContext } = require('@redwoodjs/graphql-server')
+const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext')
 const { getPaths } = require('@redwoodjs/internal/dist/paths')
-const { defineScenario } = require('@redwoodjs/testing/dist/api')
+
+const { defineScenario } = require('../../../dist/api/scenario')

I get:

 PASS   api  api/src/7.test.ts (94 MB heap size)
 PASS   api  api/src/8.test.ts (113 MB heap size)
 PASS   api  api/src/19.test.ts (114 MB heap size)
 PASS   api  api/src/16.test.ts (134 MB heap size)
 PASS   api  api/src/5.test.ts (153 MB heap size)
 PASS   api  api/src/12.test.ts (137 MB heap size)
 PASS   api  api/src/2.test.ts (157 MB heap size)
 PASS   api  api/src/4.test.ts (174 MB heap size)
 PASS   api  api/src/10.test.ts (192 MB heap size)
 PASS   api  api/src/14.test.ts (210 MB heap size)
 PASS   api  api/src/6.test.ts (228 MB heap size)
 PASS   api  api/src/9.test.ts (246 MB heap size)
 PASS   api  api/src/1.test.ts (264 MB heap size)
 PASS   api  api/src/15.test.ts (282 MB heap size)
 PASS   api  api/src/11.test.ts (300 MB heap size)
 PASS   api  api/src/17.test.ts (318 MB heap size)
 PASS   api  api/src/20.test.ts (336 MB heap size)
 PASS   api  api/src/13.test.ts (354 MB heap size)
 PASS   api  api/src/18.test.ts (372 MB heap size)
 PASS   api  api/src/0.test.ts (390 MB heap size)
 PASS   api  api/src/3.test.ts (408 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        3.174 s, estimated 4 s

i.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.

FYI, if I put the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js back in but keep the import updates, I get:

 PASS   api  api/src/7.test.ts (114 MB heap size)
 PASS   api  api/src/8.test.ts (161 MB heap size)
 PASS   api  api/src/19.test.ts (196 MB heap size)
 PASS   api  api/src/12.test.ts (240 MB heap size)
 PASS   api  api/src/9.test.ts (287 MB heap size)
 PASS   api  api/src/18.test.ts (333 MB heap size)
 PASS   api  api/src/16.test.ts (380 MB heap size)
 PASS   api  api/src/14.test.ts (426 MB heap size)
 PASS   api  api/src/4.test.ts (472 MB heap size)
 PASS   api  api/src/5.test.ts (307 MB heap size)
 PASS   api  api/src/10.test.ts (353 MB heap size)
 PASS   api  api/src/20.test.ts (399 MB heap size)
 PASS   api  api/src/0.test.ts (445 MB heap size)
 PASS   api  api/src/1.test.ts (492 MB heap size)
 PASS   api  api/src/3.test.ts (539 MB heap size)
 PASS   api  api/src/2.test.ts (585 MB heap size)
 PASS   api  api/src/15.test.ts (631 MB heap size)
 PASS   api  api/src/13.test.ts (472 MB heap size)
 PASS   api  api/src/6.test.ts (519 MB heap size)
 PASS   api  api/src/11.test.ts (565 MB heap size)
 PASS   api  api/src/17.test.ts (611 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        5.917 s

i.e. heap still out of control, but tests are significantly quicker

@dac09 dac09 changed the title [Bug]: @redwoodjs/testing/config/jest/api causes a memory leak [Bug]: @redwoodjs/testing/config/jest/api causes a memory leak on api side tests Aug 30, 2022
@dac09
Copy link
Collaborator

dac09 commented Aug 30, 2022

Hello both - thank you for reporting this and the reproduction.

I just merged in a few changes (mainly targetting the web side) in #6281 - would you like to try this out in your real project to see if it improves things for you?

yarn rw upgrade -t canary - we're very close to a v3 release and the canary isn't quite ready for prime time yet, but would be a great if you could give it a try.

I'll try it in your reproduction in the mean time.

@realStandal
Copy link
Collaborator

Possibly related to #4360

@dac09
Copy link
Collaborator

dac09 commented Aug 31, 2022

Hey so spent some time trying to improve speed/memory here - but I want to point out something important - the reproduction doesn't actually have any scenarios, or real usecases.

It's a tradeoff between being able to write scenarios, and seed the db without more boilerplate in each test vs keeping jest as vanilla as possible.

@redwoodjs-bot redwoodjs-bot bot moved this from Triage to In progress in Main Oct 4, 2022
@jtoar
Copy link
Contributor

jtoar commented Oct 4, 2022

Adding this one to the current cycle just so we keep seeing it and thinking about next steps.

@jtoar jtoar moved this from In progress to Backlog in Main Dec 8, 2022
@jan-stehlik
Copy link

jan-stehlik commented Mar 3, 2023 •
edited
Loading

hey @dac09 @jtoar is this being looked into please? memory leak is killing our apps test and our CI bills. We have to introduce excessive (and expensive) parallelism in order to have small enough test suites, so that they don't eat away all the memory. This is a major issue for us and causes so much pain that we don't really consider redwood test helpers production ready and are considering moving away from it unless this gets resolved

@jtoar
Copy link
Contributor

jtoar commented Mar 7, 2023

@jan-stehlik it's still on our radar, but progress is sporadic right now. Preparing for v5 (react 18) is taking most of our attention; at first it seemed like it would take a long time but things are looking optimistic, so our attention may return to this fairly soon.

We've heard many complaints about the speed but not as many about the memory. I know those are related to some extent. Unless you've already connected with one of us, maybe it's worth connecting to see what you're doing in your project and if we can offer any tips to alleviate things for the time being?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug/needs-info More information is needed for reproduction
Projects
Main
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dac09 @callingmedic911 @jmackie @realStandal @zygopleural @jtoar @jan-stehlik