-
-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix flaky tests #7180
Comments
@dplewis Do you have any idea how we can investigate this? Maybe add NewRelic to it to get the call stack when a test fails? |
The way I do is basically creating a loop around the test and making it to execute let's say 100 times. By this way you can see it failing and you can debug. I am tackling the second one now. |
#7196 probably solves items 2 and 3 |
Few more flaky tests that I've observed:
|
I added them to the list above. I'll mark tests as solved by #7196 if they don't occur anymore after some time. |
"ParseServerRESTController transactions should handle a batch request with transaction = true" was actually also solved by #7187 so let's observe this one as well. |
Thanks, updated. |
One more:
|
Added! |
For "Parse.GeoPoint testing creating geo point exception two fields" I've run all geo point related tests 300 times for Postgres and Mongo in a row with no error. I remember in that time there was other strange messages happening for the Postgres database, so I guess the Postgres was in an inconsistent state maybe because of anther test but it does not look to have a problem with this test itself. I suggest we observe it more and investigate it again in the case it happens to fail. |
This one is for Redis cache:
|
I think I may have a fix for this one in another branch. Or it fixes the
|
"rest create test specified session length" is now possibly fixed by #7208 |
Cool! Could you please open a PR? |
I've also couldn't reproduce "ParseGraphQLServer Auto API Schema Objects Queries Find read preferences should support readPreference argument" yet. I suggest now we observe more as most of the flaky tests were tackled. I will continue working in other stuff and come back to here when seeing something else failing. |
Added the 2 above mentioned tests and one of them as possibly fixed by #7212. |
|
Updated list. |
Added new flaky test: PostgresStorageAdapter should use index for caseInsensitive query using default indexname |
One more: 1) ParseGraphQLServer Auto API Schema Files Mutations Create should return File object Reference: https://github.com/parse-community/parse-server/runs/1972047490?check_suite_focus=true |
Added! |
https://github.com/parse-community/parse-server/runs/2041751762#step:8:30
|
Added! |
https://github.com/parse-community/parse-server/runs/2048364609#step:8:48
https://github.com/parse-community/parse-server/pull/7242/checks?check_run_id=2048468586#step:8:48
|
|
Updated, added 2, fixed 1 |
Jobs and Push flaky tests are addressed by #7267
|
Nice, updated! |
Added ParseGraphQLServer Auto API Schema Objects Queries Get read preferences should support includeReadPreference argument
|
🎉 This change has been released in version 6.1.0-alpha.8 |
🎉 This change has been released in version 6.1.0-beta.2 |
🎉 This change has been released in version 6.1.0 |
Closing via #8468 |
New Issue Checklist
Issue Description
Flaky tests impede the CI process. With the increasing number of test scenarios to ensure Parse Server compatibility with recent versions of MongoDB, Postgres and Node.js, the likeliness of flaky tests disrupting the CI process also increases. In addition, GitHub Actions currently does not support to re-run individual jobs which makes the manual re-run a tedious effort.
Expected outcome
Identify root cause of flakiness and fix for tests to be stable.
The following lists flaky tests to analyze and possibly find a pattern to determine the root cause. The list should also be a helper for maintainers to identify flaky tests when reviewing PRs. Feel free to update the list if you encounter a flaky test that is not in the list yet.
Last list update: 2021-02-24
Flaky tests
Tests which still have to be analyzed to fix the flakiness.
Schema Performance test schema update class
Installations update ios fails with new deviceToken and no installationId
Parse.User testing only creates a single session for an installation / user pair (#2885)
ParseLiveQuery expect afterEvent delete
Apple Game Center Auth adapter validateAuthData should validate
Cloud Code test beforeSave with invalid field
rest create test specified session length
ParseGraphQLServer Auto API Schema Objects Queries Get read preferences should support includeReadPreference argument
ParseGraphQLServer Auto API Schema Objects Queries Find read preferences should support readPreference argument
ParseGraphQLServer Auto API Schema Files Mutations Create should return File object
Possibly fixed
Tests for which a PR has been merged to fix the flakiness. These are currently under observation to confirm the fix. If you observe such a test to fail, please post a comment for further evaluation.
by #7187:
ParseServerRESTController transactions should handle a batch request with transaction = true
by #7196:
Cloud Code cloud jobs should define a job
Cloud Code cloud jobs should define a job
Cloud Code cloud jobs should run with master key basic auth
by #7212:
RedisCacheAdapter should find un-expired records
bv #7213:
PushController properly creates _PushStatus
by #7228:
PostgresStorageAdapter should use index for caseInsensitive query using default indexname
by #7265:
Parse.GeoPoint testing creating geo point exception two fields
by #7267:
Cloud Code cloud jobs should set the message / success on the job
PushController should not enqueue push when device token is not set
Parse.Push does not get stuck with _PushStatus 'running' on 1 installation added
Fixed tests
Tests that have not been observed anymore for a while and are therefore considered fixed.
The text was updated successfully, but these errors were encountered: