Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MIM-1539 Allow global to be passed into mongoose_backend as an arg #3386

Conversation

arcusfelis
Copy link
Contributor

@arcusfelis arcusfelis commented Nov 4, 2021

This PR addresses MIM-1539

Proposed changes include:

  • Removed behaviour_info, idk why it is commented out.
  • Renamed init_per_host_type (so many files to change).

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 4, 2021

small_tests_24 / small_tests / 6c47fe4
Reports root / small


internal_mnesia_24 / internal_mnesia / 6c47fe4
Reports root/ big
OK: 1589 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 6c47fe4
Reports root / small


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 6c47fe4
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / 6c47fe4
Reports root/ big
OK: 2685 / Failed: 0 / User-skipped: 201 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / 6c47fe4
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / 6c47fe4
Reports root/ big
OK: 1489 / Failed: 1 / User-skipped: 400 / Auto-skipped: 0

mod_event_pusher_rabbit_SUITE:group_chat_message_publish:group_chat_message_received_event_properly_formatted
{error,
  {{assertMatch,
     [{module,mod_event_pusher_rabbit_SUITE},
      {line,435},
      {expression,
        "get_decoded_message_from_rabbit ( AliceGroupChatMsgRecvRK )"},
      {pattern,
        "# { << \"from_user_id\" >> := BobRoomJID , << \"to_user_id\" >> := AliceFullJID , << \"message\" >> := Message }"},
      {value,
        #{<<"from_user_id">> =>
          <<"[email protected]/bOb_unnamed_92.559415">>,
        <<"message">> => <<"Hi there!">>,
        <<"to_user_id">> =>
          <<"alice_unnamed_92.677872@localhost/res1">>}}]},
   [{mod_event_pusher_rabbit_SUITE,
      '-group_chat_message_received_event_properly_formatted/1-fun-0-',3,
      [{file,
         "/home/circleci/app/big_tests/tests/mod_event_pusher_rabbit_SUITE.erl"},
       {line,435}]},
    {escalus_story,story,4,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
       {line,72}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1784}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1293}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1225}]}]}}

Report log


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 6c47fe4
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / 6c47fe4
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 6c47fe4
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 6c47fe4
Reports root/ big
OK: 1862 / Failed: 0 / User-skipped: 323 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / 6c47fe4
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / 6c47fe4
Reports root/ big
OK: 3054 / Failed: 0 / User-skipped: 228 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 6c47fe4
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 6c47fe4
Reports root/ big
OK: 1709 / Failed: 0 / User-skipped: 326 / Auto-skipped: 0

@codecov
Copy link

codecov bot commented Nov 4, 2021

Codecov Report

Merging #3386 (9299add) into without-dynamic-backend-modules (dd9c278) will increase coverage by 0.01%.
The diff coverage is 93.93%.

Impacted file tree graph

@@                         Coverage Diff                         @@
##           without-dynamic-backend-modules    #3386      +/-   ##
===================================================================
+ Coverage                            80.88%   80.89%   +0.01%     
===================================================================
  Files                                  413      414       +1     
  Lines                                32631    32640       +9     
===================================================================
+ Hits                                 26393    26405      +12     
+ Misses                                6238     6235       -3     
Impacted Files Coverage Δ
src/ejabberd_sm_mnesia.erl 96.55% <ø> (ø)
src/ejabberd_sm_redis.erl 94.64% <ø> (ø)
src/mod_keystore_mnesia.erl 78.57% <ø> (ø)
src/mongoose_backend.erl 100.00% <ø> (ø)
src/mongooseim.erl 0.00% <ø> (ø)
src/wpool/mongoose_wpool.erl 85.43% <ø> (ø)
src/wpool/mongoose_wpool_http.erl 92.00% <ø> (ø)
src/ejabberd_sm_backend.erl 90.00% <90.00%> (ø)
src/ejabberd_sm.erl 84.91% <90.47%> (+0.03%) ⬆️
src/event_pusher/mod_event_pusher_push_backend.erl 100.00% <100.00%> (ø)
... and 28 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ed6b62...9299add. Read the comment docs.

@@ -1074,9 +1068,4 @@ get_cached_unique_count() ->

-spec sm_backend() -> backend().
sm_backend() ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the new changes, is this function even used? If it's not, I don't see a point keeping it, as HostTypes (or, until it's changed, global) should be enough to determine which backend is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is used in tests

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly I find this a mess, what are we trying to do? Where is the place where we acknowledge this is not host_type ready and it only accepts the global key? I think that that should be well defined somewhere in a unique place. Right now some places in ejabberd_sm are ignoring the HostType vars and passing globaland then again that is done here. Then this is also duplicating knowledge of which one is the backend both in ejabberd_sm and in ejabberd_gen_sm, that should also happen in only one place.

src/ejabberd_gen_sm.erl Outdated Show resolved Hide resolved
src/mongoose_backend.erl Outdated Show resolved Hide resolved
src/mongoose_backend.erl Outdated Show resolved Hide resolved
src/mongoose_backend.erl Show resolved Hide resolved
src/ejabberd_gen_sm.erl Outdated Show resolved Hide resolved
@gustawlippa
Copy link
Contributor

To answer @NelsonVides , what we are trying to do, is to allow mongoose_backend to use global and not just HostTypes, as there are some modules which need that. The first module, that could use this option is ejabberd_sm, and it is changed here to use it. It was noted in a previous PR, that it should not keep the backend module in some ad-hoc created persistent term and follow suit with other MIM modules and use mongoose_backend.
I also don't like, that ejabberd_gen_sm discards HostTypes, but I'm not sure if other comments should come into the scope of this PR?

@NelsonVides
Copy link
Collaborator

To answer @NelsonVides , what we are trying to do, is to allow mongoose_backend to use global and not just HostTypes, as there are some modules which need that. The first module, that could use this option is ejabberd_sm, and it is changed here to use it. It was noted in a previous PR, that it should not keep the backend module in some ad-hoc created persistent term and follow suit with other MIM modules and use mongoose_backend. I also don't like, that ejabberd_gen_sm discards HostTypes, but I'm not sure if other comments should come into the scope of this PR?

Yeah, I know, I'm aware of all of that excatly, the thing is that introducing host_type_or_global() to mongoose_backend is well done here, except that that type definition could be placed in mongooseim.erl instead of here (as it is used in many other places, we can reuse later), but how's that's all used by ejabberd_sm seems very messy to me, where both ejabberd_sm and ejabberd_gen_sm know about the fact that this is all a global, and mix actual HostTypes with globals everywhere.

Also, ejabberd_gen_sm is the same proxy-module we establish in many other places as being named with the _backend suffix. So I think it wouldn't hurt to actually rename the module to ejabberd_sm_backend and follow exactly the same code patterns we follow everywhere else. It's boring, but very predictable, if we just follow so everywhere.

@arcusfelis
Copy link
Contributor Author

Not renaming HostType argument in mongoose_backend.
It is done similar in ACL and in Wpool.
Just look at spec.
Could be a Tag, but you would have to do it consistently in this case.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 5, 2021

small_tests_24 / small_tests / 34fc43b
Reports root / small


internal_mnesia_24 / internal_mnesia / 34fc43b
Reports root/ big
OK: 1589 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 34fc43b
Reports root / small


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 34fc43b
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / 34fc43b
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / 34fc43b
Reports root/ big
OK: 2685 / Failed: 0 / User-skipped: 201 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 34fc43b
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / 34fc43b
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 34fc43b
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 34fc43b
Reports root/ big
OK: 1862 / Failed: 0 / User-skipped: 323 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 34fc43b
Reports root/ big
OK: 3089 / Failed: 1 / User-skipped: 211 / Auto-skipped: 0

jingle_SUITE:all:jingle_session_is_established_when_calling_a_number
{error,
  {{assertion_failed,assert,is_iq_result,
     {xmlel,<<"iq">>,
       [{<<"from">>,<<"[email protected]">>},
        {<<"to">>,
         <<"alice_jingle_session_is_established_when_calling_a_number_12.360541@localhost/res1">>},
        {<<"id">>,<<"180d19d5-6608-47fb-9a86-fc502e95b78e">>},
        {<<"type">>,<<"set">>}],
       [{xmlel,<<"jingle">>,
          [{<<"xmlns">>,<<"urn:xmpp:jingle:1">>},
           {<<"action">>,<<"session-info">>},
           {<<"sid">>,<<"6a28a420-de33-411e-840b-9981c8bea83a">>}],
          [{xmlel,<<"ringing">>,
             [{<<"xmlns">>,<<"urn:xmpp:jingle:apps:rtp:info:1">>}],
             []}]}]},
     "<iq from='[email protected]' to='alice_jingle_session_is_established_when_calling_a_number_12.360541@localhost/res1' id='180d19d5-6608-47fb-9a86-fc502e95b78e' type='set'><jingle xmlns='urn:xmpp:jingle:1' action='session-info' sid='6a28a420-de33-411e-840b-9981c8bea83a'><ringing xmlns='urn:xmpp:jingle:apps:rtp:info:1'/></jingle></iq>"},
   [{escalus_new_assert,assert_true,2,
      [{file,
         "/home/circleci/app/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
       {line,84}]},
    {jingle_SUITE,send_initiate_and_wait_for_first_iq_set,2,
      [{file,"/home/circleci/app/big_tests/tests/jingle_SUITE.erl"},
       {line,395}]},
    {jingle_SUITE,
      '-jingle_session_is_established_when_calling_a_number/1-fun-0-',1,
      [{file,"/home/circleci/app/big_tests/tests/jingle_SUITE.erl"},
       {lin...

Report log


pgsql_mnesia_24 / pgsql_mnesia / 34fc43b
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / 34fc43b
Reports root/ big
OK: 3069 / Failed: 2 / User-skipped: 211 / Auto-skipped: 0

service_domain_db_SUITE:db:db_reinserted_from_one_node_while_service_disabled_on_another
{error,
  {{badmatch,{ok,<<"dbgroup2">>}},
   [{service_domain_db_SUITE,
      db_reinserted_from_one_node_while_service_disabled_on_another,1,
      [{file,
         "/home/circleci/app/big_tests/tests/service_domain_db_SUITE.erl"},
       {line,547}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test_server.erl"},{line,1292}]},
    {test_server,run_test_case_eval,9,
      [{file,"test_server.erl"},{line,1224}]}]}}

Report log

service_domain_db_SUITE:db:db_keeps_syncing_after_cluster_join
{error,{test_case_failed,{[<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>],
              [<<"example1.com">>,<<"example2.com">>,
               <<"example3.com">>,<<"example4.com">>]}}}

Report log


mysql_redis_24 / mysql_redis / 34fc43b
Reports root/ big
OK: 3054 / Failed: 0 / User-skipped: 228 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 34fc43b
Reports root/ big
OK: 1709 / Failed: 0 / User-skipped: 326 / Auto-skipped: 0

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now this seems more organized, and it's clear that this is simply a global module, and if we want to make it per-host-type in the future, all the necessary changes are clear and straightforward (albeit long and mechanical, yeah, big diff, but very predictable and organised diff).

Just have one more comment 👌🏽

src/ejabberd_sm.erl Outdated Show resolved Hide resolved
@NelsonVides NelsonVides dismissed their stale review November 5, 2021 11:06

Not opposed to it anymore, just a minor comment away from approval :)

Copy link
Collaborator

@NelsonVides NelsonVides left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent for me, thanks for all the applied requests :)

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 5, 2021

small_tests_24 / small_tests / 9299add
Reports root / small


internal_mnesia_24 / internal_mnesia / 9299add
Reports root/ big
OK: 1589 / Failed: 0 / User-skipped: 297 / Auto-skipped: 0


small_tests_23 / small_tests / 9299add
Reports root / small


dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 9299add
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


ldap_mnesia_24 / ldap_mnesia / 9299add
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


dynamic_domains_mysql_redis_24 / mysql_redis / 9299add
Reports root/ big
OK: 2685 / Failed: 0 / User-skipped: 201 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_23 / pgsql_mnesia / 9299add
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


ldap_mnesia_23 / ldap_mnesia / 9299add
Reports root/ big
OK: 1486 / Failed: 0 / User-skipped: 400 / Auto-skipped: 0


pgsql_mnesia_24 / pgsql_mnesia / 9299add
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_24 / odbc_mssql_mnesia / 9299add
Reports root/ big
OK: 2702 / Failed: 0 / User-skipped: 184 / Auto-skipped: 0


elasticsearch_and_cassandra_24 / elasticsearch_and_cassandra_mnesia / 9299add
Reports root/ big
OK: 1862 / Failed: 0 / User-skipped: 323 / Auto-skipped: 0


mysql_redis_24 / mysql_redis / 9299add
Reports root/ big
OK: 3054 / Failed: 0 / User-skipped: 228 / Auto-skipped: 0


pgsql_mnesia_23 / pgsql_mnesia / 9299add
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


mssql_mnesia_24 / odbc_mssql_mnesia / 9299add
Reports root/ big
OK: 3071 / Failed: 0 / User-skipped: 211 / Auto-skipped: 0


riak_mnesia_24 / riak_mnesia / 9299add
Reports root/ big
OK: 1709 / Failed: 0 / User-skipped: 326 / Auto-skipped: 0

Copy link
Member

@chrzaszcz chrzaszcz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@gustawlippa gustawlippa merged commit d21eb5d into without-dynamic-backend-modules Nov 5, 2021
@gustawlippa gustawlippa deleted the mu-allow-global-in-mongoose_backend branch November 5, 2021 15:38
NelsonVides added a commit that referenced this pull request Nov 9, 2021
With the changes done in #3386, this turned out to be really easy. Before that, I tried to make mongoose_rdbms_backend expose its API with HostTypes, but it was a bit messy. I think it's better to do it in a separate PR (after the whole without-dynamic-backend-modules branch is merged), to keep this already big diff smaller and more focused.

Other issue is that mongoose_rdbms has many functions exported, which seem to be not used, mainly for escaping different types. This could be revisited, but, in similar fashion, I left this as is for now.
What's left to do is remove dynamic mongoose_rdbms_type, but it will be done in a separate PR.
@Premwoik Premwoik modified the milestones: 5.1.0, 5.0.0 May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants