Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove domain/incremental db deletes #3889

Merged
merged 5 commits into from
Dec 2, 2022

Conversation

NelsonVides
Copy link
Collaborator

This PR implements for inbox what was implemented already in #3777

Quoting and adapting to the inbox context, the current query, looking like:

DELETE FROM inbox WHERE domain = ?

is, while technically correct, surely going to fail because of timeouts, once it is actually removing any big data.

This PR introduces a configuration flag that allows to set a batch size, and then a delete work will commit only deleting that many records, and iterates until nothing more is deleted.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Dec 2, 2022

small_tests_24 / small_tests / 8e484f9
Reports root / small


small_tests_25 / small_tests / 8e484f9
Reports root / small


ldap_mnesia_24 / ldap_mnesia / 8e484f9
Reports root/ big
OK: 2169 / Failed: 0 / User-skipped: 819 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 8e484f9
Reports root/ big
OK: 4119 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


ldap_mnesia_25 / ldap_mnesia / 8e484f9
Reports root/ big
OK: 2169 / Failed: 0 / User-skipped: 819 / Auto-skipped: 0


dynamic_domains_mysql_redis_25 / mysql_redis / 8e484f9
Reports root/ big
OK: 4092 / Failed: 1 / User-skipped: 114 / Auto-skipped: 0

muc_SUITE:hibernation:hibernated_room_can_be_queried_for_archive
{error,{{assertion_failed,assert,is_groupchat_message,
              [<<"Restorable message">>],
              undefined,"undefined"},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {muc_SUITE,wait_for_mam_result,3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4394}]},
     {muc_SUITE,'-hibernated_room_can_be_queried_for_archive/1-fun-0-',3,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4130}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {muc_SUITE,hibernated_room_can_be_queried_for_archive,1,
          [{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
           {line,4126}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 8e484f9
Reports root/ big
OK: 4119 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 8e484f9
Reports root/ big
OK: 4119 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0


internal_mnesia_25 / internal_mnesia / 8e484f9
Reports root/ big
OK: 2311 / Failed: 0 / User-skipped: 677 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / 8e484f9
Reports root/ big
OK: 4493 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 8e484f9
Reports root/ big
OK: 2664 / Failed: 0 / User-skipped: 663 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 8e484f9
Reports root/ big
OK: 2505 / Failed: 0 / User-skipped: 652 / Auto-skipped: 0


mysql_redis_25 / mysql_redis / 8e484f9
Reports root/ big
OK: 4479 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / 8e484f9
Reports root/ big
OK: 4493 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


mssql_mnesia_25 / odbc_mssql_mnesia / 8e484f9
Reports root/ big
OK: 4493 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0


dynamic_domains_mysql_redis_25 / mysql_redis / 8e484f9
Reports root/ big
OK: 4093 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0

@codecov
Copy link

codecov bot commented Dec 2, 2022

Codecov Report

Base: 83.13% // Head: 83.13% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (8e484f9) compared to base (83ed0f6).
Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3889   +/-   ##
=======================================
  Coverage   83.13%   83.13%           
=======================================
  Files         535      535           
  Lines       34200    34203    +3     
=======================================
+ Hits        28431    28434    +3     
  Misses       5769     5769           
Impacted Files Coverage Δ
src/inbox/mod_inbox.erl 87.44% <ø> (ø)
src/inbox/mod_inbox_backend.erl 100.00% <ø> (ø)
src/inbox/mod_inbox_rdbms_async.erl 70.58% <ø> (-2.95%) ⬇️
src/mam/mod_mam_utils.erl 89.93% <ø> (ø)
src/inbox/mod_inbox_rdbms.erl 92.34% <100.00%> (+0.12%) ⬆️
src/smart_markers/mod_smart_markers_rdbms.erl 91.66% <0.00%> (-3.34%) ⬇️
src/mod_muc_log.erl 62.82% <0.00%> (ø)
src/ejabberd_c2s.erl 89.02% <0.00%> (+0.07%) ⬆️
src/domain/mongoose_domain_loader.erl 90.17% <0.00%> (+0.89%) ⬆️
... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Member

@chrzaszcz chrzaszcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍

@chrzaszcz chrzaszcz merged commit 0d0c822 into master Dec 2, 2022
@chrzaszcz chrzaszcz deleted the remove_domain/incremental_db_deletes branch December 2, 2022 15:11
@chrzaszcz chrzaszcz added this to the 6.0.0 milestone Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants