Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rearranged service_domainn_db_SUITE to run more tests in parallel #4176

Merged
merged 2 commits into from
Dec 18, 2023

Conversation

pawlooss1
Copy link
Member

@pawlooss1 pawlooss1 commented Nov 21, 2023

This PR refactors service_domainn_db_SUITE. New grouping of test cases is introduced. Some of the tests are using randomised domain names so they don't interfere each other.

For MSSQL the test are being run sequentially because of the random fails (due to poor dev container with MSSQL that we use on the CI). To finally check if we don't introduce new flaky tests I ran CI multiple times and none of the time this suite failed - https://app.circleci.com/pipelines/github/esl/MongooseIM/11803.
There are flaky tests in other suites though. I'd be nice if we addressed it in the future.

@mongoose-im

This comment was marked as outdated.

Copy link

codecov bot commented Nov 21, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0342139) 84.27% compared to head (71a2523) 84.31%.
Report is 7 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4176      +/-   ##
==========================================
+ Coverage   84.27%   84.31%   +0.04%     
==========================================
  Files         551      551              
  Lines       33467    33467              
==========================================
+ Hits        28204    28219      +15     
+ Misses       5263     5248      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@mongoose-im

This comment was marked as outdated.

@mongoose-im

This comment was marked as outdated.

@mongoose-im

This comment was marked as outdated.

@pawlooss1 pawlooss1 marked this pull request as ready for review December 13, 2023 16:22
Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's some nice rework, groups/0 is now so much more readable, and tests are nicely separated in better classified groups. Also, nice we can parallelise so many things!

Only two small comments in the random domain generation below :)

big_tests/tests/service_domain_db_SUITE.erl Outdated Show resolved Hide resolved
big_tests/tests/service_domain_db_SUITE.erl Outdated Show resolved Hide resolved
@mongoose-im
Copy link
Collaborator

mongoose-im commented Dec 18, 2023

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 71a2523
Reports root/ big
OK: 368 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0


small_tests_25 / small_tests / 71a2523
Reports root / small


small_tests_26 / small_tests / 71a2523
Reports root / small


small_tests_26_arm64 / small_tests / 71a2523
Reports root / small


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 71a2523
Reports root/ big
OK: 4225 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0


ldap_mnesia_25 / ldap_mnesia / 71a2523
Reports root/ big
OK: 2268 / Failed: 0 / User-skipped: 847 / Auto-skipped: 0


ldap_mnesia_26 / ldap_mnesia / 71a2523
Reports root/ big
OK: 2268 / Failed: 0 / User-skipped: 847 / Auto-skipped: 0


dynamic_domains_mysql_redis_26 / mysql_redis / 71a2523
Reports root/ big
OK: 4192 / Failed: 0 / User-skipped: 135 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 71a2523
Reports root/ big
OK: 4225 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0


internal_mnesia_26 / internal_mnesia / 71a2523
Reports root/ big
OK: 2408 / Failed: 0 / User-skipped: 707 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 71a2523
Reports root/ big
OK: 4222 / Failed: 0 / User-skipped: 105 / Auto-skipped: 0

graphql_muc_light_SUITE:admin_cli:admin_muc_light:end_per_group
{error,
 {{unregistering_failed,
   {amount,2},
   {unregistered_items,
  [{{<<"_admin_invite_user_1064">>,
     [{escalus_event_mgr,<0.24144.0>},
    {tc_name,admin_invite_user},
    {escalus_cleaner,<0.24143.0>},
    {watchdog,<0.24142.0>},
    {muc_light_host,<<"muclight.domain.example.com">>},
    {secondary_muc_light_host,<<"muclight.domain.example.org">>},
    {protocol,cli},
    {schema_endpoint,admin},
    {{ejabberd_cwd,mongooseim@localhost},
     "/home/circleci/project/_build/mim1/rel/mongooseim"},
    {preset,"odbc_mssql_mnesia"},
    {mim_data_dir,
     "/home/circleci/project/big_tests/tests/graphql_muc_light_SUITE_data"},
    {tc_logfile,
     "https://circleci-mim-results.s3.eu-central-1.amazonaws.com/PR/4176/203834/odbc_mssql_mnesia.26.1.2-amd64/big/ct_run.test%40aba79982b5e2.2023-12-18_10.56.55/big_tests.tests.graphql_muc_light_SUITE.logs/run.2023-12-18_11.01.04/graphql_muc_light_suite.admin_invite_user.50178.html"},
    {tc_group_properties,[{name,admin_muc_light}]},
    {tc_group_path,[[{name,admin_cli}]]},
    {data_dir,
     "/home/circleci/project/big_tests/_build/default/lib/mongoose_tests/ebin/graphql_muc_light_SUITE_data/"},
    {priv_dir,
     "https://circleci-mim-results.s3.eu-central-1.amazonaws.com/PR/4176/203834/odbc_mssql_mnesia.26.1.2-amd64/big/ct_run.test%40aba79982b5e2.2023-12-18_10.56.55/big_tests.tests.graphql_muc_light_SUITE.logs/run.2023-12-18_11.01.04/log_private/"},
    {{saved_modules,mongooseim@localhost,<<"test type">>},
     #{mod_vcard =>
      #{matches => 30,
        host => {prefix,<<"vjud.">>},
        search => true,iqdisc => parallel,backend => rdbms},
    ...

Report log


mysql_redis_26 / mysql_redis / 71a2523
Reports root/ big
OK: 4593 / Failed: 0 / User-skipped: 130 / Auto-skipped: 0


pgsql_cets_26 / pgsql_cets / 71a2523
Reports root/ big
OK: 4273 / Failed: 0 / User-skipped: 110 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / 71a2523
Reports root/ big
OK: 4614 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0


pgsql_mnesia_26 / pgsql_mnesia / 71a2523
Reports root/ big
OK: 4614 / Failed: 0 / User-skipped: 109 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / 71a2523
Reports root/ big
OK: 4611 / Failed: 0 / User-skipped: 112 / Auto-skipped: 0

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌🏽

@NelsonVides NelsonVides merged commit b29ab06 into master Dec 18, 2023
4 checks passed
@NelsonVides NelsonVides deleted the parallelise-tests branch December 18, 2023 13:07
@jacekwegr jacekwegr added this to the 6.2.1 milestone Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants