Domain service test fix #4362

arcusfelis · 2024-08-20T18:31:32Z

This PR addresses "MIM-2272 Fix flaky "service_domain_db" tests"

Example of the broken test timeout https://circleci-mim-results.s3.eu-central-1.amazonaws.com/PR/4357/235426/odbc_mssql_mnesia.27.0.1-amd64/big/ct_run.test%404f8408ead5c6.2024-08-14_09.06.15/big_tests.tests.service_domain_db_SUITE.logs/run.2024-08-14_09.27.28/service_domain_db_suite.rest_delete_domain_cleans_data_from_mam.283010.html

Proposed changes include:

Higher timeouts when pinging domain service gen_server. It affects only the test code. 15 seconds for ping timeout, 30 seconds for the RPC timeout, so we can see what the server is doing, when the ping times out.
Add debug logging for SQL queries which are done by the domain service.
Do some stuff in parallel.
There was slow code in fixing gaps because duplicate key error was causing the insert_dummy_event transaction to get restarted (i.e. 3100 milliseconds for a gap). Sometimes test could create 20 gaps, so 20300 = 6 seconds, and our timeout is default 5 seconds. So, we added catch there.
Add debug_ dictionary entries. Useful to figure out what the server is doing from the outside.
Added info how to debug it.
Repeated 30 times - no errors locally or on CI.

mongoose-im · 2024-08-20T18:35:37Z

ldap_mnesia_26 / ldap_mnesia / b94a8cf
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

ldap_mnesia_27 / ldap_mnesia / b94a8cf
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

internal_mnesia_27 / internal_mnesia / b94a8cf
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / b94a8cf
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / b94a8cf
Reports root/ big
OK: 439 / Failed: 0 / User-skipped: 47 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / b94a8cf
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

mysql_redis_27 / mysql_redis / b94a8cf
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

pgsql_cets_27 / pgsql_cets / b94a8cf
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / b94a8cf
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / b94a8cf
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

dynamic_domains_mysql_redis_27 / mysql_redis / b94a8cf
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

small_tests_26 / small_tests / b94a8cf
Reports root / small

small_tests_27 / small_tests / b94a8cf
Reports root / small

mssql_mnesia_27 / odbc_mssql_mnesia / b94a8cf
Reports root/ big
OK: 3375 / Failed: 6 / User-skipped: 0 / Auto-skipped: 0

service_domain_db_SUITE:db:rest_with_auth:rest_with_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,timeout},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1303}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1235}]}]}}

Report log

service_domain_db_SUITE:db:rest_with_auth:rest_with_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,timeout},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1303}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1235}]}]}}

Report log

service_domain_db_SUITE:db:rest_without_auth:rest_without_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,timeout},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1303}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1235}]}]}}

Report log

service_domain_db_SUITE:db:rest_without_auth:rest_without_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,timeout},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1303}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1235}]}]}}

Report log

service_domain_db_SUITE:db:rest_without_auth:rest_without_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,
     {'EXIT',
       {{badmatch,
          {badrpc,
            {'EXIT',
              {timeout,
                {gen_server,call,[<10349.62465.0>,ping]}}}}},
        [{service_domain_db,'-sync_local/0-lc$^1/1-1-',1,
           [{file,
            "/home/circleci/project/src/domain/service_domain_db.erl"},
          {line,99}]},
         {service_domain_db,sync_local,0,
           [{file,
            "/home/circleci/project/src/domain/service_domain_db.erl"},
          {line,99}]}]}}},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_c...

Report log

service_domain_db_SUITE:db:rest_without_auth:rest_without_auth_parallel:rest_delete_domain_cleans_data_from_mam

{error,
  {{badrpc,timeout},
   [{distributed_helper,rpc,
      [#{node => mongooseim@localhost},service_domain_db,sync_local,[]],
      [{file,
         "/home/circleci/project/big_tests/../test/common/distributed_helper.erl"},
       {line,140}]},
    {service_domain_db_SUITE,sync_local,1,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1214}]},
    {service_domain_db_SUITE,sync,0,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,1193}]},
    {service_domain_db_SUITE,
      '-rest_delete_domain_cleans_data_from_mam/1-fun-0-',5,
      [{file,
         "/home/circleci/project/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,965}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1794}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1303}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1235}]}]}}

Report log

dynamic_domains_mssql_mnesia_27 / odbc_mssql_mnesia / b94a8cf
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

small_tests_27_arm64 / small_tests / b94a8cf
Reports root / small

codecov · 2024-08-20T18:42:00Z

Codecov Report

Attention: Patch coverage is 84.61538% with 4 lines in your changes missing coverage. Please review.

Project coverage is 84.98%. Comparing base (87ce067) to head (85eb17d).
Report is 9 commits behind head on master.

Files	Patch %	Lines
src/domain/mongoose_domain_sql.erl	86.66%	2 Missing ⚠️
src/domain/service_domain_db.erl	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4362      +/-   ##
==========================================
+ Coverage   84.96%   84.98%   +0.01%     
==========================================
  Files         552      552              
  Lines       33697    33714      +17     
==========================================
+ Hits        28630    28651      +21     
+ Misses       5067     5063       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mongoose-im · 2024-08-21T11:53:01Z

ldap_mnesia_26 / ldap_mnesia / 70b06f2
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

internal_mnesia_27 / internal_mnesia / 70b06f2
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

ldap_mnesia_27 / ldap_mnesia / 70b06f2
Reports root/ big
OK: 211 / Failed: 0 / User-skipped: 125 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 70b06f2
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 70b06f2
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / 70b06f2
Reports root/ big
OK: 439 / Failed: 0 / User-skipped: 47 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 70b06f2
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / 70b06f2
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

mysql_redis_27 / mysql_redis / 70b06f2
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

pgsql_cets_27 / pgsql_cets / 70b06f2
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

dynamic_domains_mysql_redis_27 / mysql_redis / 70b06f2
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_27 / odbc_mssql_mnesia / 70b06f2
Reports root/ big
OK: 3380 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

small_tests_27 / small_tests / 70b06f2
Reports root / small

mssql_mnesia_27 / odbc_mssql_mnesia / 70b06f2
Reports root/ big
OK: 3381 / Failed: 0 / User-skipped: 0 / Auto-skipped: 0

small_tests_27_arm64 / small_tests / 70b06f2
Reports root / small

small_tests_26 / small_tests / 70b06f2
Reports root / small

mongoose-im · 2024-08-21T13:59:42Z

elasticsearch_and_cassandra_27 / elasticsearch_and_cassandra_mnesia / 85eb17d
Reports root/ big
OK: 439 / Failed: 0 / User-skipped: 47 / Auto-skipped: 0

small_tests_26 / small_tests / 85eb17d
Reports root / small

small_tests_27_arm64 / small_tests / 85eb17d
Reports root / small

small_tests_27 / small_tests / 85eb17d
Reports root / small

ldap_mnesia_27 / ldap_mnesia / 85eb17d
Reports root/ big
OK: 2338 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

ldap_mnesia_26 / ldap_mnesia / 85eb17d
Reports root/ big
OK: 2338 / Failed: 0 / User-skipped: 905 / Auto-skipped: 0

dynamic_domains_mysql_redis_27 / mysql_redis / 85eb17d
Reports root/ big
OK: 4588 / Failed: 0 / User-skipped: 145 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_27 / pgsql_mnesia / 85eb17d
Reports root/ big
OK: 4621 / Failed: 0 / User-skipped: 112 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 85eb17d
Reports root/ big
OK: 4621 / Failed: 0 / User-skipped: 112 / Auto-skipped: 0

internal_mnesia_27 / internal_mnesia / 85eb17d
Reports root/ big
OK: 2478 / Failed: 0 / User-skipped: 765 / Auto-skipped: 0

pgsql_cets_27 / pgsql_cets / 85eb17d
Reports root/ big
OK: 4532 / Failed: 0 / User-skipped: 179 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_27 / odbc_mssql_mnesia / 85eb17d
Reports root/ big
OK: 4618 / Failed: 0 / User-skipped: 115 / Auto-skipped: 0

pgsql_mnesia_27 / pgsql_mnesia / 85eb17d
Reports root/ big
OK: 5012 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

mysql_redis_27 / mysql_redis / 85eb17d
Reports root/ big
OK: 4991 / Failed: 0 / User-skipped: 140 / Auto-skipped: 0

pgsql_mnesia_26 / pgsql_mnesia / 85eb17d
Reports root/ big
OK: 5012 / Failed: 0 / User-skipped: 119 / Auto-skipped: 0

mssql_mnesia_27 / odbc_mssql_mnesia / 85eb17d
Reports root/ big
OK: 5009 / Failed: 0 / User-skipped: 122 / Auto-skipped: 0

jacekwegr

Looks good! Thank you for making those changes. This should help the CI run more smoothly.

arcusfelis added 2 commits August 20, 2024 18:58

Do service_domain_db:sync_local/1 in parallel

ffc4d4e

Add SQL logging for mongoose_domain_sql

b1312b9

arcusfelis added 6 commits August 21, 2024 15:50

Log time duration for the SQL calls in mongoose_domain_sql

3e90edf

Debug gaps in the domains

790bdd2

Do not fail on insert_full_event_mssql (ignore duplicate keys)

2b09f48

Pass multiple ids into insert_dummy_events

b950863

Increase sync_local ping timeout

5748de7

Disable debug logging

85eb17d

arcusfelis force-pushed the domain-service-test-fix branch from 70b06f2 to 85eb17d Compare August 21, 2024 13:52

arcusfelis marked this pull request as ready for review August 21, 2024 14:29

jacekwegr approved these changes Aug 22, 2024

View reviewed changes

jacekwegr merged commit f7a862c into master Aug 22, 2024
4 checks passed

jacekwegr deleted the domain-service-test-fix branch August 22, 2024 11:36

jacekwegr added this to the 6.3.0 milestone Oct 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Domain service test fix #4362

Domain service test fix #4362

arcusfelis commented Aug 20, 2024 •

edited

Loading

mongoose-im commented Aug 20, 2024 •

edited

Loading

codecov bot commented Aug 20, 2024 •

edited

Loading

mongoose-im commented Aug 21, 2024 •

edited

Loading

mongoose-im commented Aug 21, 2024 •

edited

Loading

jacekwegr left a comment

Domain service test fix #4362

Domain service test fix #4362

Conversation

arcusfelis commented Aug 20, 2024 • edited Loading

mongoose-im commented Aug 20, 2024 • edited Loading

codecov bot commented Aug 20, 2024 • edited Loading

Codecov Report

mongoose-im commented Aug 21, 2024 • edited Loading

mongoose-im commented Aug 21, 2024 • edited Loading

jacekwegr left a comment

Choose a reason for hiding this comment

arcusfelis commented Aug 20, 2024 •

edited

Loading

mongoose-im commented Aug 20, 2024 •

edited

Loading

codecov bot commented Aug 20, 2024 •

edited

Loading

mongoose-im commented Aug 21, 2024 •

edited

Loading

mongoose-im commented Aug 21, 2024 •

edited

Loading