Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky init_per_group in the MUC suite #4241

Merged
merged 1 commit into from
Mar 13, 2024

Conversation

jacekwegr
Copy link
Contributor

@jacekwegr jacekwegr commented Mar 12, 2024

This PR fixes a problem where tests might fail if an HTTP listener has already been started in the init_per_group function of the MUC suite.

Sometimes, if other tests that need an HTTP listener don't work right, they can leave the listener running. This causes a problem when trying to start it again in the init_per_group.

@jacekwegr jacekwegr changed the title Fix flaky init_per_group in MUC suite Fix flaky init_per_group in the MUC suite Mar 12, 2024
@mongoose-im

This comment was marked as outdated.

Copy link

codecov bot commented Mar 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.39%. Comparing base (468b1df) to head (a75eaf1).
Report is 10 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4241      +/-   ##
==========================================
- Coverage   84.40%   84.39%   -0.01%     
==========================================
  Files         552      552              
  Lines       33538    33538              
==========================================
- Hits        28308    28305       -3     
- Misses       5230     5233       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jacekwegr jacekwegr force-pushed the muc-suite-fix-init-per-group branch from 7adbbef to ed2d95a Compare March 12, 2024 08:01
@mongoose-im

This comment was marked as outdated.

@jacekwegr jacekwegr marked this pull request as ready for review March 12, 2024 09:06
@NelsonVides
Copy link
Collaborator

Do we know which suite left the state dirty with this helper still running? It'd be good to clean that up too 🤔

@jacekwegr
Copy link
Contributor Author

Do we know which suite left the state dirty with this helper still running? It'd be good to clean that up too 🤔

Yeah, it was mod_event_pusher_http_SUITE. I will add cleanup for that too. Thanks for pointing that out!

Copy link
Member

@chrzaszcz chrzaszcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that this is fixed. I am only thinking of a more generic solution, because this might appear again in a different place. Maybe we could amend http_helper:start/3 to have this check instead? If it's already running, it could log such information, then stop and start just like in this PR.

Otherwise, fixing the testcase that actually causes the problem might be the best option as @NelsonVides suggested.

@jacekwegr jacekwegr force-pushed the muc-suite-fix-init-per-group branch from ed2d95a to a75eaf1 Compare March 13, 2024 08:15
@mongoose-im
Copy link
Collaborator

mongoose-im commented Mar 13, 2024

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / a75eaf1
Reports root/ big
OK: 429 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0


small_tests_25 / small_tests / a75eaf1
Reports root / small


small_tests_26 / small_tests / a75eaf1
Reports root / small


small_tests_26_arm64 / small_tests / a75eaf1
Reports root / small


ldap_mnesia_25 / ldap_mnesia / a75eaf1
Reports root/ big
OK: 2275 / Failed: 0 / User-skipped: 895 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / a75eaf1
Reports root/ big
OK: 4502 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


dynamic_domains_mysql_redis_26 / mysql_redis / a75eaf1
Reports root/ big
OK: 4469 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0


ldap_mnesia_26 / ldap_mnesia / a75eaf1
Reports root/ big
OK: 2275 / Failed: 0 / User-skipped: 895 / Auto-skipped: 0


internal_mnesia_26 / internal_mnesia / a75eaf1
Reports root/ big
OK: 2415 / Failed: 0 / User-skipped: 755 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / a75eaf1
Reports root/ big
OK: 4502 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / a75eaf1
Reports root/ big
OK: 4891 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


pgsql_cets_26 / pgsql_cets / a75eaf1
Reports root/ big
OK: 4425 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0


pgsql_mnesia_26 / pgsql_mnesia / a75eaf1
Reports root/ big
OK: 4891 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


mysql_redis_26 / mysql_redis / a75eaf1
Reports root/ big
OK: 4870 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / a75eaf1
Reports root/ big
OK: 4887 / Failed: 1 / User-skipped: 121 / Auto-skipped: 0

muc_SUITE:register:user_submits_registration_form
{error,
  {{assertion_failed,assert,is_iq_result,
     [{xmlel,<<"iq">>,
        [{<<"type">>,<<"set">>},
         {<<"id">>,<<"328b69ef1f57d12be46556e3e578fdf5">>},
         {<<"to">>,<<"muc.localhost">>}],
        [{xmlel,<<"query">>,
           [{<<"xmlns">>,<<"jabber:iq:register">>}],
           [{xmlel,<<"x">>,
            [{<<"xmlns">>,<<"jabber:x:data">>},
             {<<"type">>,<<"submit">>}],
            [{xmlel,<<"field">>,
               [{<<"type">>,<<"hidden">>},
                {<<"var">>,<<"FORM_TYPE">>}],
               [{xmlel,<<"value">>,[],
                  [{xmlcdata,<<"jabber:iq:register">>}]}]},
             {xmlel,<<"field">>,
               [{<<"type">>,<<"text-single">>},
                {<<"var">>,<<"nick">>}],
               [{xmlel,<<"value">>,[],
                  [{xmlcdata,
                     <<"thirdwitchroom-965aabd12c">>}]}]}]}]}]}],
     {xmlel,<<"iq">>,
       [{<<"from">>,<<"muc.localhost">>},
        {<<"to">>,
         <<"alice_user_submits_registration_form_2886@localhost/res1">>},
        {<<"type">>,<<"error">>},
        {<<"id">>,<<"328b69ef1f57d12be46556e3e578fdf5">>}],
       [{xmlel,<<"query">>,
          [{<<"xmlns">>,<<"jabber:iq:register">>}],
          [{xmlel,<<"x">>,
             [{<<"xmlns">>,<<"jabber:x:data">>},
            {<<"type">>,<<"submit">>}],
             [{xmlel,<<"field">>,
              [{<<"type">>,<<"hidden">>},
               {<<"var">>,<<"FORM_TYPE">>}],
              [{xmlel,<<"value">>,[],
         ...

Report log


dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / a75eaf1
Reports root/ big
OK: 4499 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / a75eaf1
Reports root/ big
OK: 4886 / Failed: 2 / User-skipped: 121 / Auto-skipped: 0

graphql_muc_light_SUITE:admin_cli:admin_muc_light:admin_create_room_with_unprepped_id
{error,
  {#{what => invalid_response_code,expected_type => ok,
     response_code => {exit_status,1}},
   [{graphql_helper,assert_response_code,2,
      [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
       {line,256}]},
    {graphql_helper,get_ok_value,2,
      [{file,"/home/circleci/project/big_tests/tests/graphql_helper.erl"},
       {line,239}]},
    {graphql_muc_light_SUITE,admin_create_room_with_unprepped_id,1,
      [{file,
         "/home/circleci/project/big_tests/tests/graphql_muc_light_SUITE.erl"},
       {line,1156}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1234}]}]}}

Report log

graphql_muc_light_SUITE:domain_admin:domain_admin_muc_light:admin_create_room_with_unprepped_id
{error,
  {{badmatch,null},
   [{graphql_muc_light_SUITE,admin_create_room_with_unprepped_id,1,
      [{file,
         "/home/circleci/project/big_tests/tests/graphql_muc_light_SUITE.erl"},
       {line,1155}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1302}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1234}]}]}}

Report log

graphql_muc_light_SUITE:domain_admin:domain_admin_muc_light:end_per_group
{error,
 {{unregistering_failed,
   {amount,1},
   {unregistered_items,
  [{{<<"_domain_admin_invite_user_no_permission_1111">>,
     [{escalus_event_mgr,<0.25122.0>},
    {tc_name,domain_admin_invite_user_no_permission},
    {escalus_cleaner,<0.25121.0>},
    {watchdog,<0.25120.0>},
    {muc_light_host,<<"muclight.localhost">>},
    {secondary_muc_light_host,<<"muclight.localhost.bis">>},
    {protocol,http},
    {domain_admin,{<<"admin@localhost">>,<<"3714b64ac7a829f2">>}},
    {schema_endpoint,domain_admin},
    {{ejabberd_cwd,mongooseim@localhost},
     "/home/circleci/project/_build/mim1/rel/mongooseim"},
    {preset,"odbc_mssql_mnesia"},
    {mim_data_dir,
     "/home/circleci/project/big_tests/tests/graphql_muc_light_SUITE_data"},
    {tc_logfile,
     "https://circleci-mim-results.s3.eu-central-1.amazonaws.com/PR/4241/212028/odbc_mssql_mnesia.26.1.2-amd64/big/ct_run.test%4065a864254808.2024-03-13_08.44.46/big_tests.tests.graphql_muc_light_SUITE.logs/run.2024-03-13_08.49.13/graphql_muc_light_suite.domain_admin_invite_user_no_permission.html"},
    {tc_group_properties,[{name,domain_admin_muc_light}]},
    {tc_group_path,[[{name,domain_admin}]]},
    {data_dir,
     "/home/circleci/project/big_tests/_build/default/lib/mongoose_tests/ebin/graphql_muc_light_SUITE_data/"},
    {priv_dir,
     "https://circleci-mim-results.s3.eu-central-1.amazonaws.com/PR/4241/212028/odbc_mssql_mnesia.26.1.2-amd64/big/ct_run.test%4065a864254808.2024-03-13_08.44.46/big_tests.tests.graphql_muc_light_SUITE.logs/run.2024-03-13_08.49.13/log_private/"},
    {{saved_modules,mongooseim@localhost,<<"localhost">>},
     #{mod_vcard =...

Report log


mssql_mnesia_26 / odbc_mssql_mnesia / a75eaf1
Reports root/ big
OK: 4888 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

Copy link
Member

@chrzaszcz chrzaszcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's ok for me. @NelsonVides do you think we need to fix the root cause in this PR, or shall we merge this?

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as it is, but in a separate story it'd be good to find out why the other suite is leaving the state dirty too :)

@NelsonVides NelsonVides merged commit 7ae0848 into master Mar 13, 2024
4 checks passed
@NelsonVides NelsonVides deleted the muc-suite-fix-init-per-group branch March 13, 2024 11:42
@jacekwegr jacekwegr added this to the 6.2.1 milestone Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants