Fix randomly failing delayed_receive test in PEP suite #3961

jacekwegr · 2023-02-13T15:39:53Z

This PR changes the make_friends function in pep_SUITE to be less prone to the race conditions that might cause the delayed_receive and delayed_receive_with_sm tests to fail. It also changes end_per_group function so that it always restores dynamic modules. The tests were repeated 100 times on the CI for additional certainty.

codecov · 2023-02-13T15:50:08Z

Codecov Report

Base: 83.54% // Head: 83.57% // Increases project coverage by +0.02% 🎉

Coverage data is based on head (0c23f34) compared to base (25757b1).
Patch has no changes to coverable lines.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3961      +/-   ##
==========================================
+ Coverage   83.54%   83.57%   +0.02%     
==========================================
  Files         538      538              
  Lines       34003    34003              
==========================================
+ Hits        28408    28418      +10     
+ Misses       5595     5585      -10

Impacted Files	Coverage Δ
src/ejabberd.erl	`45.00% <0.00%> (-10.00%)`	⬇️
src/domain/service_domain_db.erl	`83.33% <0.00%> (-2.09%)`	⬇️
src/wpool/mongoose_wpool.erl	`82.24% <0.00%> (-0.94%)`	⬇️
src/rdbms/mongoose_rdbms.erl	`66.31% <0.00%> (-0.71%)`	⬇️
src/pubsub/mod_pubsub_db_mnesia.erl	`93.24% <0.00%> (-0.43%)`	⬇️
src/mod_roster.erl	`78.57% <0.00%> (-0.24%)`	⬇️
src/mod_muc_log.erl	`62.82% <0.00%> (ø)`
src/pubsub/mod_pubsub.erl	`73.66% <0.00%> (+0.18%)`	⬆️
src/pubsub/mod_pubsub_db_rdbms.erl	`95.34% <0.00%> (+0.25%)`	⬆️
src/mod_caps.erl	`78.01% <0.00%> (+0.86%)`	⬆️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

chrzaszcz

Looks good 👍

big_tests/tests/pep_SUITE.erl

mongoose-im · 2023-02-15T08:29:53Z

small_tests_24 / small_tests / 0c23f34
Reports root / small

small_tests_25 / small_tests / 0c23f34
Reports root / small

ldap_mnesia_24 / ldap_mnesia / 0c23f34
Reports root/ big
OK: 2219 / Failed: 0 / User-skipped: 825 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 0c23f34
Reports root/ big
OK: 4169 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / 0c23f34
Reports root/ big
OK: 2237 / Failed: 1 / User-skipped: 825 / Auto-skipped: 0

pubsub_SUITE:dag+basic:publish_test

{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,491}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,481}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,471}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

dynamic_domains_mysql_redis_25 / mysql_redis / 0c23f34
Reports root/ big
OK: 4143 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 0c23f34
Reports root/ big
OK: 4169 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 0c23f34
Reports root/ big
OK: 4166 / Failed: 0 / User-skipped: 91 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 0c23f34
Reports root/ big
OK: 4549 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 0c23f34
Reports root/ big
OK: 2721 / Failed: 0 / User-skipped: 662 / Auto-skipped: 0

riak_mnesia_24 / riak_mnesia / 0c23f34
Reports root/ big
OK: 2559 / Failed: 0 / User-skipped: 654 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 0c23f34
Reports root/ big
OK: 2361 / Failed: 0 / User-skipped: 683 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 0c23f34
Reports root/ big
OK: 4549 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 0c23f34
Reports root/ big
OK: 4543 / Failed: 3 / User-skipped: 100 / Auto-skipped: 0

inbox_extensions_SUITE:async_pools:one_to_one:mute_muted_entry_gets_unmuted

{error,
  {{assert,
     [{module,inbox_extensions_SUITE},
      {line,882},
      {expression,"escalus_pred : is_message ( Message )"},
      {expected,true},
      {value,false}]},
   [{inbox_extensions_SUITE,check_message_with_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,882}]},
    {inbox_extensions_SUITE,set_inbox_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,876}]},
    {inbox_extensions_SUITE,'-mute_muted_entry_gets_unmuted/1-fun-2-',2,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,575}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

inbox_extensions_SUITE:async_pools:one_to_one:mute_muted_conv_restarts_timestamp

{error,
  {{assert,
     [{module,inbox_extensions_SUITE},
      {line,882},
      {expression,"escalus_pred : is_message ( Message )"},
      {expected,true},
      {value,false}]},
   [{inbox_extensions_SUITE,check_message_with_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,882}]},
    {inbox_extensions_SUITE,set_inbox_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,876}]},
    {inbox_extensions_SUITE,'-mute_muted_conv_restarts_timestamp/1-fun-2-',
      2,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,615}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

muc_SUITE:hibernation:hibernated_room_can_be_queried_for_archive

{error,{{assertion_failed,assert,is_groupchat_message,
              [<<"Restorable message">>],
              undefined,"undefined"},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {muc_SUITE,wait_for_mam_result,3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4394}]},
     {muc_SUITE,'-hibernated_room_can_be_queried_for_archive/1-fun-0-',3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4130}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {muc_SUITE,hibernated_room_can_be_queried_for_archive,1,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4126}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mysql_redis_25 / mysql_redis / 0c23f34
Reports root/ big
OK: 4534 / Failed: 1 / User-skipped: 111 / Auto-skipped: 0

graphql_session_SUITE:admin_session:admin_session_cli:admin_kick_user_session

{error,
  {function_clause,
    [{graphql_helper,get_error,
       [1,
        {{<<"200">>,<<"OK">>},
         #{<<"data">> =>
           #{<<"session">> =>
               #{<<"kickUserSession">> =>
                 #{<<"code">> => null,
                   <<"jid">> =>
                     <<"alice_admin_kick_user_session_1157@localhost/res1">>,
                   <<"kicked">> => true,
                   <<"message">> => <<"Session kicked">>}}}}}],
       []},
     {graphql_helper,get_err_msg,2,
       [{file,
          "/home/circleci/project/big_tests/tests/graphql_helper.erl"},
        {line,227}]},
     {graphql_session_SUITE,admin_kick_user_session_story,3,
       [{file,
          "/home/circleci/project/big_tests/tests/graphql_session_SUITE.erl"},
        {line,441}]},
     {escalus_story,story,4,
       [{file,
          "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
       [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
       [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mysql_redis_25 / mysql_redis / 0c23f34
Reports root/ big
OK: 4535 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 0c23f34
Reports root/ big
OK: 4546 / Failed: 0 / User-skipped: 100 / Auto-skipped: 0

This comment was marked as outdated.

Sign in to view

jacekwegr force-pushed the fix-randomly-failing-pep-tests branch from aa3f667 to 0cfbf7c Compare February 14, 2023 08:40

This comment was marked as outdated.

Sign in to view

jacekwegr force-pushed the fix-randomly-failing-pep-tests branch from 0cfbf7c to 5feaf84 Compare February 14, 2023 11:37

This comment was marked as outdated.

Sign in to view

jacekwegr marked this pull request as ready for review February 14, 2023 12:53

chrzaszcz approved these changes Feb 14, 2023

View reviewed changes

big_tests/tests/pep_SUITE.erl Outdated Show resolved Hide resolved

big_tests/tests/pep_SUITE.erl Outdated Show resolved Hide resolved

Change delayed_receive test

0c23f34

jacekwegr force-pushed the fix-randomly-failing-pep-tests branch from 5feaf84 to 0c23f34 Compare February 15, 2023 08:21

chrzaszcz approved these changes Feb 15, 2023

View reviewed changes

chrzaszcz merged commit 56f756c into master Feb 15, 2023

chrzaszcz deleted the fix-randomly-failing-pep-tests branch February 15, 2023 09:37

jacekwegr added this to the 6.1.0 milestone Apr 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix randomly failing delayed_receive test in PEP suite #3961

Fix randomly failing delayed_receive test in PEP suite #3961

jacekwegr commented Feb 13, 2023 •

edited

Loading

This comment was marked as outdated.

codecov bot commented Feb 13, 2023 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

chrzaszcz left a comment

mongoose-im commented Feb 15, 2023 •

edited

Loading

Fix randomly failing delayed_receive test in PEP suite #3961

Fix randomly failing delayed_receive test in PEP suite #3961

Conversation

jacekwegr commented Feb 13, 2023 • edited Loading

This comment was marked as outdated.

codecov bot commented Feb 13, 2023 • edited Loading

Codecov Report

This comment was marked as outdated.

This comment was marked as outdated.

chrzaszcz left a comment

Choose a reason for hiding this comment

mongoose-im commented Feb 15, 2023 • edited Loading

jacekwegr commented Feb 13, 2023 •

edited

Loading

codecov bot commented Feb 13, 2023 •

edited

Loading

mongoose-im commented Feb 15, 2023 •

edited

Loading