Extend aggregator requests and fix stall-after-error bug #3757

NelsonVides · 2022-09-12T17:31:15Z

Maybe one task requests to remove data a previous task added, and in the resulting task after aggregation there is no data to either remove nor request, so we might want to have the request been skipped and the next one been attempted, instead of wasting time and message passing on an empty request.

codecov · 2022-09-12T17:44:25Z

Codecov Report

Merging #3757 (b198cb0) into master (7e19c4f) will increase coverage by 0.00%.
The diff coverage is 75.00%.

@@           Coverage Diff           @@
##           master    #3757   +/-   ##
=======================================
  Coverage   82.72%   82.73%           
=======================================
  Files         529      529           
  Lines       33970    33975    +5     
=======================================
+ Hits        28102    28108    +6     
+ Misses       5868     5867    -1

Impacted Files	Coverage Δ
src/rdbms/mongoose_rdbms.erl	`66.31% <ø> (ø)`
src/async_pools/mongoose_aggregator_worker.erl	`63.33% <75.00%> (+1.51%)`	⬆️
src/inbox/mod_inbox_rdbms_async.erl	`72.05% <0.00%> (-1.48%)`	⬇️
src/mod_roster.erl	`79.42% <0.00%> (ø)`
src/mod_muc_log.erl	`62.82% <0.00%> (ø)`
src/pubsub/mod_pubsub_db_mnesia.erl	`92.82% <0.00%> (+0.42%)`	⬆️
src/muc_light/mod_muc_light_db_mnesia.erl	`92.63% <0.00%> (+1.05%)`	⬆️
src/pubsub/node_pep.erl	`79.62% <0.00%> (+1.85%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

mongoose-im · 2022-09-12T21:44:10Z

small_tests_24 / small_tests / b198cb0
Reports root / small

small_tests_25 / small_tests / b198cb0
Reports root / small

dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / b198cb0
Reports root/ big
OK: 3583 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0

ldap_mnesia_24 / ldap_mnesia / b198cb0
Reports root/ big
OK: 1948 / Failed: 0 / User-skipped: 551 / Auto-skipped: 0

dynamic_domains_mysql_redis_25 / mysql_redis / b198cb0
Reports root/ big
OK: 3566 / Failed: 0 / User-skipped: 105 / Auto-skipped: 0

dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / b198cb0
Reports root/ big
OK: 3583 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0

ldap_mnesia_25 / ldap_mnesia / b198cb0
Reports root/ big
OK: 1948 / Failed: 0 / User-skipped: 551 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / b198cb0
Reports root/ big
OK: 3582 / Failed: 1 / User-skipped: 88 / Auto-skipped: 0

muc_SUITE:register:user_unregisters_nick_twice

{error,
  {{assertion_failed,assert,is_iq_result,
     [{xmlel,<<"iq">>,
        [{<<"type">>,<<"set">>},
         {<<"id">>,<<"1711d7075861febca8e9c38810619a3a">>},
         {<<"to">>,<<"groupchats.domain.example.com">>}],
        [{xmlel,<<"query">>,
           [{<<"xmlns">>,<<"jabber:iq:register">>}],
           [{xmlel,<<"x">>,
            [{<<"xmlns">>,<<"jabber:x:data">>},
             {<<"type">>,<<"submit">>}],
            [{xmlel,<<"field">>,
               [{<<"type">>,<<"hidden">>},
                {<<"var">>,<<"FORM_TYPE">>}],
               [{xmlel,<<"value">>,[],
                  [{xmlcdata,<<"jabber:iq:register">>}]}]},
             {xmlel,<<"field">>,
               [{<<"type">>,<<"text-single">>},
                {<<"var">>,<<"nick">>}],
               [{xmlel,<<"value">>,[],
                  [{xmlcdata,
                     <<"thirdwitch1room-4a6da7c02e">>}]}]}]}]}]}],
     {xmlel,<<"iq">>,
       [{<<"from">>,<<"groupchats.domain.example.com">>},
        {<<"to">>,
         <<"[email protected]/res1">>},
        {<<"type">>,<<"error">>},
        {<<"xml:lang">>,<<"en">>},
        {<<"id">>,<<"1711d7075861febca8e9c38810619a3a">>}],
       [{xmlel,<<"query">>,
          [{<<"xmlns">>,<<"jabber:iq:register">>}],
          [{xmlel,<<"x">>,
             [{<<"xmlns">>,<<"jabber:x:data">>},
            {<<"type">>,<<"submit">>}],
             [{xmlel,<<"field">>,
              [{<<"type">>,<<"hidden">>},
               {<<...

Report log

internal_mnesia_25 / internal_mnesia / b198cb0
Reports root/ big
OK: 2069 / Failed: 0 / User-skipped: 430 / Auto-skipped: 0

pgsql_mnesia_24 / pgsql_mnesia / b198cb0
Reports root/ big
OK: 3957 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0

elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / b198cb0
Reports root/ big
OK: 2409 / Failed: 0 / User-skipped: 425 / Auto-skipped: 0

mysql_redis_25 / mysql_redis / b198cb0
Reports root/ big
OK: 3952 / Failed: 0 / User-skipped: 102 / Auto-skipped: 0

pgsql_mnesia_25 / pgsql_mnesia / b198cb0
Reports root/ big
OK: 3957 / Failed: 0 / User-skipped: 97 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / b198cb0
Reports root/ big
OK: 3979 / Failed: 2 / User-skipped: 97 / Auto-skipped: 0

pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription

{error,
  {{badmatch,
     [{xmlel,<<"message">>,
        [{<<"from">>,
        <<"alice_unsubscribe_after_presence_unsubscription_2319@localhost">>},
         {<<"to">>,
        <<"bob_unsubscribe_after_presence_unsubscription_2319@localhost/res1">>},
         {<<"type">>,<<"headline">>}],
        [{xmlel,<<"event">>,
           [{<<"xmlns">>,
           <<"http://jabber.org/protocol/pubsub#event">>}],
           [{xmlel,<<"items">>,
            [{<<"node">>,<<"lvD6WvdsPnXMYLcEX1ccYg==">>}],
            [{xmlel,<<"item">>,
               [{<<"id">>,<<"salmon">>}],
               [{xmlel,<<"entry">>,
                  [{<<"xmlns">>,
                  <<"http://www.w3.org/2005/Atom">>}],
                  []}]}]}]},
         {xmlel,<<"headers">>,
           [{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
           []}]}]},
   [{pep_SUITE,'-unsubscribe_after_presence_unsubscription/1-fun-0-',2,
      [{file,"/home/circleci/project/big_tests/tests/pep_SUITE.erl"},
       {line,384}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription

{error,
  {{badmatch,
     [{xmlel,<<"message">>,
        [{<<"from">>,
        <<"alice_unsubscribe_after_presence_unsubscription_2331@localhost">>},
         {<<"to">>,
        <<"bob_unsubscribe_after_presence_unsubscription_2331@localhost/res1">>},
         {<<"type">>,<<"headline">>}],
        [{xmlel,<<"event">>,
           [{<<"xmlns">>,
           <<"http://jabber.org/protocol/pubsub#event">>}],
           [{xmlel,<<"items">>,
            [{<<"node">>,<<"kud/WeM4YbKj11z0h7Kwhg==">>}],
            [{xmlel,<<"item">>,
               [{<<"id">>,<<"salmon">>}],
               [{xmlel,<<"entry">>,
                  [{<<"xmlns">>,
                  <<"http://www.w3.org/2005/Atom">>}],
                  []}]}]}]},
         {xmlel,<<"headers">>,
           [{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
           []}]}]},
   [{pep_SUITE,'-unsubscribe_after_presence_unsubscription/1-fun-0-',2,
      [{file,"/home/circleci/project/big_tests/tests/pep_SUITE.erl"},
       {line,384}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

riak_mnesia_24 / riak_mnesia / b198cb0
Reports root/ big
OK: 2248 / Failed: 0 / User-skipped: 418 / Auto-skipped: 0

dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / b198cb0
Reports root/ big
OK: 3583 / Failed: 0 / User-skipped: 88 / Auto-skipped: 0

mssql_mnesia_25 / odbc_mssql_mnesia / b198cb0
Reports root/ big
OK: 3956 / Failed: 1 / User-skipped: 97 / Auto-skipped: 0

inbox_extensions_SUITE:async_pools:one_to_one:mute_muted_entry_gets_unmuted

{error,
  {{assert,
     [{module,inbox_extensions_SUITE},
      {line,798},
      {expression,"escalus_pred : is_message ( Message )"},
      {expected,true},
      {value,false}]},
   [{inbox_extensions_SUITE,check_message_with_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,798}]},
    {inbox_extensions_SUITE,set_inbox_properties,4,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,792}]},
    {inbox_extensions_SUITE,'-mute_muted_entry_gets_unmuted/1-fun-2-',2,
      [{file,
         "/home/circleci/project/big_tests/tests/inbox_extensions_SUITE.erl"},
       {line,568}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1291}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1223}]}]}}

Report log

mssql_mnesia_25 / odbc_mssql_mnesia / b198cb0
Reports root/ big
OK: 3968 / Failed: 1 / User-skipped: 97 / Auto-skipped: 0

pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription

{error,{{assertion_failed,assert_many,false,[is_roster_set],[],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {escalus_story,'-make_all_clients_friends/1-fun-0-',2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,108}]},
     {escalus_utils,'-each_with_index/3-fun-0-',3,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_utils.erl"},
             {line,87}]},
     {lists,foldl_1,3,[{file,"lists.erl"},{line,1355}]},
     {escalus_utils,'-each_with_index/3-fun-0-',3,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_utils.erl"},
             {line,87}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1350}]},
     {escalus_utils,distinct_pairs,2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_utils.erl"},
             {line,60}]},
     {escalus_story,make_all_clients_friends,1,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,106}]}]}}

Report log

arcusfelis

ok

NelsonVides requested a review from arcusfelis September 12, 2022 17:31

This comment was marked as outdated.

Sign in to view

mongoose-im mentioned this pull request Sep 12, 2022

CI build failures summary - part 2 #3685

Closed

NelsonVides added 2 commits September 12, 2022 22:23

OS extend aggregator requests

9c2c138

OS fix return type for transaction request

552a713

NelsonVides force-pushed the async/drops branch from 4380919 to 552a713 Compare September 12, 2022 20:24

This comment was marked as outdated.

Sign in to view

NelsonVides added 4 commits September 12, 2022 23:25

Test the aggregation

2864996

Cleanup async tests

353186e

Test for bug where an error stops the aggregator!

b43756b

Ensure to keep flushing until a request does not drop

b198cb0

NelsonVides force-pushed the async/drops branch from 352fb95 to b198cb0 Compare September 12, 2022 21:35

NelsonVides changed the title ~~Extend aggregator requests~~ Extend aggregator requests and fix stall-after-error bug Sep 12, 2022

arcusfelis approved these changes Sep 13, 2022

View reviewed changes

arcusfelis merged commit adc1136 into master Sep 13, 2022

arcusfelis deleted the async/drops branch September 13, 2022 21:26

chrzaszcz added this to the 6.0.0 milestone Dec 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend aggregator requests and fix stall-after-error bug #3757

Extend aggregator requests and fix stall-after-error bug #3757

NelsonVides commented Sep 12, 2022

This comment was marked as outdated.

codecov bot commented Sep 12, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Sep 12, 2022 •

edited

Loading

arcusfelis left a comment

Extend aggregator requests and fix stall-after-error bug #3757

Extend aggregator requests and fix stall-after-error bug #3757

Conversation

NelsonVides commented Sep 12, 2022

This comment was marked as outdated.

codecov bot commented Sep 12, 2022 • edited Loading

Codecov Report

This comment was marked as outdated.

This comment was marked as outdated.

mongoose-im commented Sep 12, 2022 • edited Loading

arcusfelis left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 12, 2022 •

edited

Loading

mongoose-im commented Sep 12, 2022 •

edited

Loading