Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

same unit test failing consistently, inconsistently on CircleCI #213

Closed
1 of 5 tasks
thescientist13 opened this issue Sep 27, 2019 · 3 comments
Closed
1 of 5 tasks

same unit test failing consistently, inconsistently on CircleCI #213

thescientist13 opened this issue Sep 27, 2019 · 3 comments
Assignees
@thescientist13
Labels
bug Something isn't working help wanted Extra attention is needed P0 Critical issue that should get addressed ASAP v0.4.0 Plugins and Packages v0.5.0 Data w/ GraphQL
Milestone
1.0

Comments

@thescientist13
Copy link
Member

thescientist13 commented Sep 27, 2019 •
edited
Loading

Type of Change

  • New Feature Request
  • Documentation / Website
  • Improvement / Suggestion
  • Bug
  • Other (please clarify below)

Summary

The tests have started failing consistently inconsistently recently (at least within the release/0.4.0 branch).

First it was this one
#195 (review)
https://circleci.com/gh/ProjectEvergreen/greenwood/644

 5 failing

  1) Build Greenwood With: 
       Custom Webpack Plugin and Default Workspace
         Running Smoke Tests: Custom Webpack Plugin and Default Workspace
           Index (Home) page
             should have a <title> tag in the <head>:

      AssertionError: expected 'My App' to equal 'Greenwood App'
      + expected - actual

      -My App
      +Greenwood App
      
      at Context.<anonymous> (test/smoke-test.js:86:29)

  2) Build Greenwood With: 
       Custom Webpack Plugin and Default Workspace
         Running Smoke Tests: Custom Webpack Plugin and Default Workspace
           Index (Home) page
             should have the correct route tags in the <body>:

      AssertionError: expected 0 to equal 3
      + expected - actual

      -0
      +3
      
      at Context.<anonymous> (test/smoke-test.js:109:37)

  3) Build Greenwood With: 
       Custom Webpack Plugin and Default Workspace
         Running Smoke Tests: Custom Webpack Plugin and Default Workspace
           Index (Home) page
             should have the expected heading text within the index page in the public directory:
     TypeError: Cannot read property 'textContent' of null
      at Context.<anonymous> (test/smoke-test.js:113:64)

  4) Build Greenwood With: 
       Custom Webpack Plugin and Default Workspace
         Running Smoke Tests: Custom Webpack Plugin and Default Workspace
           Index (Home) page
             should have the expected paragraph text within the index page in the public directory:
     TypeError: Cannot read property 'textContent' of null
      at Context.<anonymous> (test/smoke-test.js:119:65)

  5) Build Greenwood With: 
       Custom Webpack Plugin and Default Workspace
         Running Smoke Tests: Custom Webpack Plugin and Default Workspace
           Hello World (dummy) page
             "before each" hook for "should output a hello page directory":
     Error: ENOENT: no such file or directory, open '/root/repo/packages/cli/test/cases/build.plugins-webpack/public/hello/index.html'

And then more failures, but all different test, but the same ones every time (there is some overlap in failure between the last ones)

Screen Shot 2019-09-26 at 9 47 13 PM

Details

  332 passing (3m)
  2 failing

  1) Build Greenwood With: 
       Default Greenwood Configuration and Default Workspace w/ Nested Directories
         Custom blog page directory
           should have the expected heading text within the hello example page in the hello directory:

      AssertionError: expected 'Greenwood' to equal 'Blog Page'
      + expected - actual

      -Greenwood
      +Blog Page
      
      at Context.<anonymous> (packages/cli/test/cases/build.default.workspace-nested/build.default.workspace-nested.spec.js:61:28)

  2) Build Greenwood With: 
       Default Greenwood Configuration and Default Workspace w/ Nested Directories
         Custom blog page directory
           should have the expected paragraph text within the hello example page in the hello directory:

      AssertionError: expected 'This is the home page built by Greenwood. Make your own pages in src/pages/index.js!' to equal 'This is the test blog page built by Greenwood.'
      + expected - actual

      -This is the home page built by Greenwood. Make your own pages in src/pages/index.js!
      +This is the test blog page built by Greenwood.
      
      at Context.<anonymous> (packages/cli/test/cases/build.default.workspace-nested/build.default.workspace-nested.spec.js:67:30)
@thescientist13 thescientist13 added help wanted Extra attention is needed chore unit testing, maintenance, etc P0 Critical issue that should get addressed ASAP labels Sep 27, 2019
@thescientist13 thescientist13 self-assigned this Sep 27, 2019
@thescientist13 thescientist13 mentioned this issue Sep 27, 2019
Closed
@thescientist13
Copy link
Member Author

phew, that was easy.
Screen Shot 2019-09-26 at 10 17 15 PM

we can't use async in our mocha tests

@thescientist13
Copy link
Member Author

thescientist13 commented Sep 27, 2019 •
edited
Loading

@hutchgrant
Hmm, now can't remember, is there a rule around async usage in our tests? I see the usage in our it blocks inconsistently. Wondering if that could be contributing to the issue?

edit: no, it was arrow functions we can't use in mocha tests

@thescientist13 thescientist13 changed the title same unit test failing consistently, inconsistently same unit test failing consistently, inconsistently on CircleCI Sep 27, 2019
This was referenced Sep 27, 2019
Closed
Closed
Closed
@thescientist13
Copy link
Member Author

thescientist13 commented Oct 2, 2019 •
edited
Loading

Well, good news is that its reproducible in the Docker container following these steps
https://github.com/thegreenhouseio/docker-nodejs-dev#workflow

Ran yarn test twice against release/0.4.0 branch, and both times it failed with the exact same failure as the very first build failure observed 🚫 🚫
https://circleci.com/gh/ProjectEvergreen/greenwood/644

Started testing some of these PRs

  1. Enhancement/issue 16 parallel serialization #216 : Ran this twice and it passed both times ✅ ✅
  2. remove resolve inside recursive walk directory call #215 : Ran this twice and it passed both times ✅ ✅
  3. release/0.4.0: Ran this twice again and it passed twice ✅ ✅
  4. 😳
  5. Killed the container, started it again
  6. release/0.4.0: Ran this twice again and it passed one, failed once ✅ 🚫
326 passing (4m)
  8 failing

🤷‍♂

@hutchgrant hutchgrant mentioned this issue Oct 2, 2019
Merged
@thescientist13 thescientist13 mentioned this issue Oct 10, 2019
Closed
@thescientist13 thescientist13 added bug Something isn't working help wanted Extra attention is needed P0 Critical issue that should get addressed ASAP v0.4.0 Plugins and Packages and removed P0 Critical issue that should get addressed ASAP help wanted Extra attention is needed chore unit testing, maintenance, etc bug Something isn't working labels Oct 10, 2019
@thescientist13 thescientist13 added P0 Critical issue that should get addressed ASAP v0.4.0 Plugins and Packages bug Something isn't working help wanted Extra attention is needed and removed v0.4.0 Plugins and Packages P0 Critical issue that should get addressed ASAP bug Something isn't working labels Oct 12, 2019
This was referenced Oct 12, 2019
Merged
Closed
Closed
Closed
@thescientist13 thescientist13 mentioned this issue Nov 15, 2019
Merged
@thescientist13 thescientist13 added this to the MVP milestone Nov 26, 2019
@thescientist13 thescientist13 added the v0.5.0 Data w/ GraphQL label Jan 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thescientist13 thescientist13

Labels
bug Something isn't working help wanted Extra attention is needed P0 Critical issue that should get addressed ASAP v0.4.0 Plugins and Packages v0.5.0 Data w/ GraphQL
Projects
None yet
Milestone
1.0
Development

No branches or pull requests
1 participant
@thescientist13