Reporting to ga4 #4040

DenysGonchar · 2023-06-17T22:50:35Z

This PR introduces the following changes:

reporting of the failed to GA4:
- GH Actions reporting
- CircleCI reporting
- see test cases counters.
- some minor improvements were required to ct_markdown_errors_hook to cover failures in init_per_testcase/end_per_testcases.
improved a bit ct_groups_summary_hook:
- covered situation when repeat_until_all_ok flag is missing (on the last attempt to repeat the test)
- it's still not enough to cover correctly all the edge cases of eventually passing tests, this hook should be completely replaced/reworked in the upcoming PR.
introduced test hook and suit to investigate corner cases (these 2 to be removed in the upcoming PR, once ct_groups_summary_hook is replaced).
minor clean-up of the existing hooks:
- remote logging is moved into ct_mongoose_log_hook. Previously, there were 2 redundant implementations in ct_mongoose_hook and ct_tty_hook hooks.
- ct_mim_config_hook is removed completely, mim_data_dir config is now set by ct_mongoose_hook.
- dummy ct hooks callbacks are removed from all hooks.
- halting logic is moved from ct_tty_hook:on_tc_skip/3 to ct_groups_summary_hook:on_tc_skip/3. I didn't test that code, but ct_tty_hook is definitely an improper place for it.
some minor fixes test execution scripts.

UPDATE:

reporting to GA4 is done for master branch only.

chrzaszcz

Please make the tests pass. Could you hide the stale reports with failed tests as well? It's a lot to scroll down. Thanks!

chrzaszcz

Looks good in general, I added some comments.

big_tests/failure_repoting_testing.spec

big_tests/run_common_test.erl

big_tests/src/ct_groups_summary_hook.erl

tools/summarise-ct-results

tools/test.sh

big_tests/src/ct_groups_summary_hook.erl

DenysGonchar · 2023-07-11T08:38:25Z

oh my god, so many typos in this PR.

for testing use the following commands: . tools/test-runner-complete.sh test-runner.sh --skip-preset --skip-build-mim --skip-small-tests --skip-start-nodes --skip-stop-nodes --db --skip-cover --skip-preset --dev-nodes --spec failure_reporting_testing.spec

ct_mongoose_log_hook already does it, and does it in a better way.

it's quite an overhead to have an extra hook just to set the mim_data_dir parameter, so the hook is merged with ct_mongoose_hook

…ry_hook:on_tc_skip/3

testing package building for ubuntu_xenial as it's not tested by CircleCI

chrzaszcz

It looks good. I only suggest changing the target to a feature branch and following with a fix for these errors that appear multiple times in the comments to avoid affecting other PR's:

codecov · 2023-07-12T09:24:49Z

Codecov Report

Patch coverage has no change and project coverage change: +0.04 🎉

Comparison is base (563f592) 83.82% compared to head (15d30b5) 83.87%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4040      +/-   ##
==========================================
+ Coverage   83.82%   83.87%   +0.04%     
==========================================
  Files         526      526              
  Lines       33133    33133              
==========================================
+ Hits        27773    27789      +16     
+ Misses       5360     5344      -16

see 15 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

mongoose-im · 2023-07-12T09:32:20Z

small_tests_24 / small_tests / 15d30b5
Reports root / small

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 15d30b5
Reports root/ big
OK: 369 / Failed: 0 / User-skipped: 38 / Auto-skipped: 0

small_tests_25_arm64 / small_tests / 15d30b5
Reports root / small

small_tests_25 / small_tests / 15d30b5
Reports root / small

ldap_mnesia_24 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2250 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4212 / Failed: 0 / User-skipped: 82 / Auto-skipped: 0

mam_SUITE:rdbms_async_cache_muc_all:muc_configurable_archiveid:muc_only_stanzaid

{failed,
  {mam_SUITE,end_per_testcase,
    {'EXIT',
      {{room_archive_size,0,[{times,200,1}],ok},
       [{mongoose_helper,do_wait_until,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
           {line,357}]},
        {mam_helper,wait_for_room_archive_size,3,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,789}]},
        {mam_helper,clean_room_archive,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,733}]},
        {mam_helper,destroy_room,1,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_helper.erl"},
           {line,726}]},
        {mam_SUITE,end_per_testcase,2,
          [{file,
             "/home/circleci/project/big_tests/tests/mam_SUITE.erl"},
           {line,965}]},
        {test_server,do_end_per_testcase,4,
          [{file,"test_server.erl"},{line,1627}]},
        {test_server,run_test_case_eval1,6,
          [{file,"test_server.erl"},{line,1335}]},
        {test_server,run_test_case_eval,9,
          [{file,"test_server.erl"},{line,1224}]}]}}}}

Report log

ldap_mnesia_25 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2249 / Failed: 1 / User-skipped: 826 / Auto-skipped: 0

mod_global_distrib_SUITE:hosts_refresher:test_host_refreshing

{error,
  {{trees_for_connections_present,true,[{times,50,false}],ok},
   [{mongoose_helper,do_wait_until,2,
      [{file,"/home/circleci/project/big_tests/tests/mongoose_helper.erl"},
       {line,357}]},
    {mod_global_distrib_SUITE,test_host_refreshing,1,
      [{file,
         "/home/circleci/project/big_tests/tests/mod_global_distrib_SUITE.erl"},
       {line,384}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

dynamic_domains_mysql_redis_25 / mysql_redis / 15d30b5
Reports root/ big
OK: 4186 / Failed: 0 / User-skipped: 108 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4595 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4209 / Failed: 0 / User-skipped: 85 / Auto-skipped: 0

internal_mnesia_25 / internal_mnesia / 15d30b5
Reports root/ big
OK: 2396 / Failed: 0 / User-skipped: 680 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / 15d30b5
Reports root/ big
OK: 4581 / Failed: 0 / User-skipped: 103 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / 15d30b5
Reports root/ big
OK: 4595 / Failed: 0 / User-skipped: 89 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4586 / Failed: 1 / User-skipped: 92 / Auto-skipped: 5

bosh_SUITE:essential_https:accept_higher_hold_value

{error,
  {{assertEqual,
     [{module,bosh_SUITE},
      {line,260},
      {expression,"get_bosh_sessions ( )"},
      {expected,[]},
      {value,
        [{bosh_session,<<"227142cad1242272923402949f73dca646676058">>,
           <9322.10168.0>}]}]},
   [{bosh_SUITE,accept_higher_hold_value,1,
      [{file,"/home/circleci/project/big_tests/tests/bosh_SUITE.erl"},
       {line,260}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

ldap_mnesia_25 / ldap_mnesia / 15d30b5
Reports root/ big
OK: 2250 / Failed: 0 / User-skipped: 826 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / 15d30b5
Reports root/ big
OK: 4592 / Failed: 0 / User-skipped: 92 / Auto-skipped: 0

DenysGonchar · 2023-07-12T10:49:17Z

It looks good. I only suggest changing the target to a feature branch and following with a fix for these errors that appear multiple times in the comments to avoid affecting other PR's:

I have fixed metrics_roster_SUITE:end_per_testcase/2 and it won't spam other PRs, but I would rather want to merge the changes into the master, so we can finally start collecting TC failure statistics for GH Actions.

chrzaszcz

Thanks for fixing it!

DenysGonchar force-pushed the reporting-to-ga4 branch 2 times, most recently from 95aed9f to cc6768c Compare June 17, 2023 23:28

DenysGonchar force-pushed the reporting-to-ga4 branch from cc6768c to 8a234f2 Compare June 26, 2023 22:35

DenysGonchar force-pushed the reporting-to-ga4 branch 4 times, most recently from c832c43 to 791c3d5 Compare July 7, 2023 13:20

DenysGonchar marked this pull request as ready for review July 7, 2023 13:41

DenysGonchar requested a review from chrzaszcz July 7, 2023 14:52

chrzaszcz reviewed Jul 10, 2023

View reviewed changes

esl deleted a comment from mongoose-im Jul 10, 2023

esl deleted a comment from codecov bot Jul 10, 2023

esl deleted a comment from mongoose-im Jul 10, 2023

esl deleted a comment from codecov bot Jul 10, 2023

DenysGonchar force-pushed the reporting-to-ga4 branch from 27d9a2e to 2332d47 Compare July 10, 2023 20:38

esl deleted a comment from codecov bot Jul 10, 2023

esl deleted a comment from mongoose-im Jul 10, 2023

chrzaszcz reviewed Jul 11, 2023

View reviewed changes

DenysGonchar added 9 commits July 11, 2023 12:10

improving ct_groups_summary_hook

b6c35a4

improving truncated_counter_file reporting at ct_markdown_errors_hook

6c0f28f

removing redundant logging from ct_mongoose_hook and ct_tty_hook.

4b31075

ct_mongoose_log_hook already does it, and does it in a better way.

removing ct_mim_config_hook

61140cd

it's quite an overhead to have an extra hook just to set the mim_data_dir parameter, so the hook is merged with ct_mongoose_hook

removing dummy optional CT hook callbacks from big_tests hooks

30b5798

minor improvements to ct_tty_hook

d6b3420

moving halting logic from ct_tty_hook:on_tc_skip/3 to ct_groups_summa…

adec35a

…ry_hook:on_tc_skip/3

adding optional CT suite/group/testcase logging at MIM node

1547306

updating pkg GH Actions job

5bbe8dd

testing package building for ubuntu_xenial as it's not tested by CircleCI

DenysGonchar force-pushed the reporting-to-ga4 branch 3 times, most recently from f460f45 to d38b2fc Compare July 11, 2023 12:26

report failed test cases to GA4 only for master branch

ea1c44d

DenysGonchar force-pushed the reporting-to-ga4 branch from d38b2fc to ea1c44d Compare July 11, 2023 13:43

esl deleted a comment from mongoose-im Jul 11, 2023

esl deleted a comment from codecov bot Jul 11, 2023

esl deleted a comment from mongoose-im Jul 11, 2023

esl deleted a comment from codecov bot Jul 11, 2023

esl deleted a comment from mongoose-im Jul 11, 2023

chrzaszcz reviewed Jul 12, 2023

View reviewed changes

fixing failures in metrics_roster_SUITE:end_per_testcase/2

15d30b5

esl deleted a comment from mongoose-im Jul 12, 2023

chrzaszcz approved these changes Jul 12, 2023

View reviewed changes

chrzaszcz merged commit d87eb4c into master Jul 12, 2023

chrzaszcz deleted the reporting-to-ga4 branch July 12, 2023 10:57

chrzaszcz added this to the 6.2.0 milestone Dec 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reporting to ga4 #4040

Reporting to ga4 #4040

DenysGonchar commented Jun 17, 2023 •

edited

Loading

chrzaszcz left a comment

chrzaszcz left a comment

DenysGonchar commented Jul 11, 2023

chrzaszcz left a comment

codecov bot commented Jul 12, 2023 •

edited

Loading

mongoose-im commented Jul 12, 2023 •

edited

Loading

DenysGonchar commented Jul 12, 2023

chrzaszcz left a comment

Reporting to ga4 #4040

Reporting to ga4 #4040

Conversation

DenysGonchar commented Jun 17, 2023 • edited Loading

chrzaszcz left a comment

Choose a reason for hiding this comment

chrzaszcz left a comment

Choose a reason for hiding this comment

DenysGonchar commented Jul 11, 2023

chrzaszcz left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 12, 2023 • edited Loading

Codecov Report

mongoose-im commented Jul 12, 2023 • edited Loading

DenysGonchar commented Jul 12, 2023

chrzaszcz left a comment

Choose a reason for hiding this comment

DenysGonchar commented Jun 17, 2023 •

edited

Loading

codecov bot commented Jul 12, 2023 •

edited

Loading

mongoose-im commented Jul 12, 2023 •

edited

Loading