Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pubsub node mnesia backend #2141

Merged
merged 13 commits into from
Nov 28, 2018
Merged

Pubsub node mnesia backend #2141

merged 13 commits into from
Nov 28, 2018

Conversation

michalwski
Copy link
Contributor

This PR extracts mnesia operations around pubsub_node table to the backend module. Fully covered is the nodetree_dag module implementing Pub Sub Collection Nodes. The default nodetree_tree module (implementing custom nodes structure) is covered to some extend.

@mongoose-im

This comment has been minimized.

@codecov
Copy link

codecov bot commented Nov 23, 2018

Codecov Report

Merging #2141 into master will decrease coverage by 0.14%.
The diff coverage is 70.93%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2141      +/-   ##
==========================================
- Coverage    75.5%   75.35%   -0.15%     
==========================================
  Files         327      327              
  Lines       28296    28310      +14     
==========================================
- Hits        21364    21334      -30     
- Misses       6932     6976      +44
Impacted Files Coverage Δ
src/pubsub/mod_pubsub_db.erl 100% <ø> (ø) ⬆️
src/pubsub/mod_pubsub.erl 67.09% <100%> (+0.02%) ⬆️
src/pubsub/nodetree_tree.erl 49.01% <60%> (+0.63%) ⬆️
src/pubsub/nodetree_dag.erl 59.32% <68.75%> (-5.8%) ⬇️
src/pubsub/mod_pubsub_db_mnesia.erl 87.24% <71.42%> (-6.21%) ⬇️
src/pubsub/mod_pubsub_db_rdbms.erl 93.14% <77.77%> (-0.84%) ⬇️
src/jingle_sip/jingle_sip_callbacks.erl 0% <0%> (-89.93%) ⬇️
src/jingle_sip/sip_to_jingle.erl 0% <0%> (-88.38%) ⬇️
src/jingle_sip/jingle_sip_helper.erl 0% <0%> (-86.67%) ⬇️
src/jingle_sip/mod_jingle_sip_backend.erl 0% <0%> (-80%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b7819f...809851f. Read the comment docs.

@@ -8,6 +8,18 @@
]},
{lager, [
{colored, true},
%% Alternate colors for white background
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wat? Is it related to PR? :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not relevant but it started being annoying for me so I decided to add it to the repo so that I can just uncomment whatever I need :)

mod_pubsub_db_mnesia:find_nodes_by_key(Key).

-spec find_nodes_by_id_and_pred(Key :: mod_pubsub:hostPubsub() | jid:ljid(),
Nodes :: [mod_pubsub:nodeId()],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broken indentation.

-callback find_node_by_id(Nidx :: mod_pubsub:nodeIdx()) ->
{error, not_found} | {ok, mod_pubsub:pubsubNode()}.

-callback find_node(Key :: mod_pubsub:hostPubsub() | jid:ljid(), Node :: mod_pubsub:nodeId()) ->
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering that find_node_by_name could be more intuitive.

mnesia:match_object(#pubsub_node{nodeid = {Key, '_'}, _ = '_'}).

-spec find_nodes_by_id_and_pred(Key :: mod_pubsub:hostPubsub() | jid:ljid(),
Nodes :: [mod_pubsub:nodeId()],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent.

@@ -10,6 +10,7 @@

-include("pubsub.hrl").
-include("jlib.hrl").
-include_lib("stdlib/include/qlc.hrl").
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this compiler complains about qlc:q/1 function with the following warning:

qlc:q/1 called, but "qlc.hrl" not included

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. :)

@michalwski
Copy link
Contributor Author

@fenek I addressed your comments and also pushed additional commit: 437e6ef

@mongoose-im

This comment has been minimized.

@mongoose-im
Copy link
Collaborator

mongoose-im commented Nov 27, 2018

5923.1 / Erlang 19.3 / small_tests / 4bc58f4
Reports root / small


5923.3 / Erlang 19.3 / mysql_redis / 4bc58f4
Reports root/ big
OK: 2883 / Failed: 0 / User-skipped: 222 / Auto-skipped: 0


5923.2 / Erlang 19.3 / internal_mnesia / 4bc58f4
Reports root/ big
OK: 1106 / Failed: 1 / User-skipped: 51 / Auto-skipped: 19

jingle_SUITE:init_per_suite
{'EXIT',
  {{badrpc,
     {'EXIT',
       {{error,
          {transport_error,
            {could_not_start,{udp,could_not_open_tcp}}}},
        [{mod_jingle_sip,start_nksip_service_or_error,1,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,71}]},
         {mod_jingle_sip,start,2,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,53}]},
         {gen_mod,start_module_for_host,3,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/gen_mod.erl"},
          {line,122}]},
         {rpc,'-handle_call_call/6-fun-0-',5,
           [{file,"rpc.erl"},{line,197}]}]}}},
   [{escalus_rpc,call_with_cookie_match,
      [mongooseim@localhost,gen_mod,start_module,
       [<<"localhost">>,mod_jingle_sip,
      [{proxy_host,"localhost"},
       {proxy_port,12345},
       {username_to_phone,[{<<"2000006168">>,<<"+919177074440">>}]}]],
       5000,ejabberd],
      [{file,
         "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_rpc.erl"},
       {line,34}]},
    {dynamic_modules,start,4,[{file,"dynamic_modules.erl"},{line,66}]},
    {jingle_SUITE,init_per_suite,1,[{file,"jingle_SUITE.erl"},{line,62}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test...

Report log

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve40.767498@localhost/res1">>,escalus_tcp,
          <0.12994.1>,
          [{event_manager,<0.12985.1>},
           {server,<<"localhost">>},
           {username,<<"eve40.767498">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.12985.1>},
            {server,<<"localhost">>},
            {username,<<"eve40.767498">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve40.767498">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve40.767498">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"0AA176E125EA5A5D">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,607}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
    ...

Report log


5923.4 / Erlang 19.3 / odbc_mssql_mnesia / 4bc58f4
Reports root/ big
OK: 2897 / Failed: 0 / User-skipped: 208 / Auto-skipped: 0


5923.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 4bc58f4
Reports root/ big
OK: 445 / Failed: 0 / User-skipped: 8 / Auto-skipped: 0


5923.5 / Erlang 19.3 / ldap_mnesia / 4bc58f4
Reports root/ big
OK: 1069 / Failed: 3 / User-skipped: 88 / Auto-skipped: 21

vcard_simple_SUITE:all:retrieve_own_card
{error,{test_case_failed,"Expected <<\"alice\">> got undefined\n"}}

Report log

vcard_simple_SUITE:all:retrieve_own_card
{error,{test_case_failed,"Expected <<\"alice\">> got undefined\n"}}

Report log

vcard_simple_SUITE:all:retrieve_own_card
{error,{test_case_failed,"Expected <<\"alice\">> got undefined\n"}}

Report log


5923.8 / Erlang 20.0 / pgsql_mnesia / 4bc58f4
Reports root/ big / small
OK: 2910 / Failed: 0 / User-skipped: 176 / Auto-skipped: 19

jingle_SUITE:init_per_suite
{'EXIT',
  {{badrpc,
     {'EXIT',
       {{error,
          {transport_error,
            {could_not_start,{udp,could_not_open_tcp}}}},
        [{mod_jingle_sip,start_nksip_service_or_error,1,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,71}]},
         {mod_jingle_sip,start,2,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,53}]},
         {gen_mod,start_module_for_host,3,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/gen_mod.erl"},
          {line,122}]},
         {rpc,'-handle_call_call/6-fun-0-',5,
           [{file,"rpc.erl"},{line,197}]}]}}},
   [{escalus_rpc,call_with_cookie_match,
      [mongooseim@localhost,gen_mod,start_module,
       [<<"localhost">>,mod_jingle_sip,
      [{proxy_host,"localhost"},
       {proxy_port,12345},
       {username_to_phone,[{<<"2000006168">>,<<"+919177074440">>}]}]],
       5000,ejabberd],
      [{file,
         "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_rpc.erl"},
       {line,34}]},
    {dynamic_modules,start,4,[{file,"dynamic_modules.erl"},{line,66}]},
    {jingle_SUITE,init_per_suite,1,[{file,"jingle_SUITE.erl"},{line,62}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test...

Report log


5923.9 / Erlang 21.0 / riak_mnesia / 4bc58f4
Reports root/ big / small
OK: 1329 / Failed: 1 / User-skipped: 49 / Auto-skipped: 19

jingle_SUITE:init_per_suite
{'EXIT',
  {{badrpc,
     {'EXIT',
       {{error,
          {transport_error,
            {could_not_start,{udp,could_not_open_tcp}}}},
        [{mod_jingle_sip,start_nksip_service_or_error,1,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,71}]},
         {mod_jingle_sip,start,2,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/jingle_sip/mod_jingle_sip.erl"},
          {line,53}]},
         {gen_mod,start_module_for_host,3,
           [{file,
            "/home/travis/build/esl/MongooseIM/_build/mim1/lib/mongooseim/src/gen_mod.erl"},
          {line,122}]},
         {rpc,'-handle_call_call/6-fun-0-',5,
           [{file,"rpc.erl"},{line,197}]}]}}},
   [{escalus_rpc,call_with_cookie_match,
      [mongooseim@localhost,gen_mod,start_module,
       [<<"localhost">>,mod_jingle_sip,
      [{proxy_host,"localhost"},
       {proxy_port,12345},
       {username_to_phone,[{<<"2000006168">>,<<"+919177074440">>}]}]],
       5000,ejabberd],
      [{file,
         "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_rpc.erl"},
       {line,34}]},
    {dynamic_modules,start,4,[{file,"dynamic_modules.erl"},{line,66}]},
    {jingle_SUITE,init_per_suite,1,[{file,"jingle_SUITE.erl"},{line,62}]},
    {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1545}]},
    {test_server,run_test_case_eval1,6,
      [{file,"test...

Report log

mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server
{error,
  {timeout_when_waiting_for_stanza,
    [{escalus_client,wait_for_stanza,
       [{client,<<"eve89.715298@localhost/res1">>,escalus_tcp,
          <0.22142.1>,
          [{event_manager,<0.22133.1>},
           {server,<<"localhost">>},
           {username,<<"eve89.715298">>},
           {resource,<<"res1">>}],
          [{event_client,
             [{event_manager,<0.22133.1>},
            {server,<<"localhost">>},
            {username,<<"eve89.715298">>},
            {resource,<<"res1">>}]},
           {resource,<<"res1">>},
           {username,<<"eve89.715298">>},
           {server,<<"localhost">>},
           {host,<<"localhost">>},
           {port,5222},
           {auth,{escalus_auth,auth_plain}},
           {wspath,undefined},
           {username,<<"eve89.715298">>},
           {server,<<"localhost">>},
           {password,<<"password">>},
           {port,5222},
           {stream_management,true},
           {stream_id,<<"BC367B04F01E29D3">>}]},
        10000],
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
        {line,138}]},
     {mod_global_distrib_SUITE,
       '-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
       4,
       [{file,"mod_global_distrib_SUITE.erl"},{line,607}]},
     {escalus_story,story,4,
       [{file,
          "/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
        {line,72}]},
    ...

Report log


5923.5 / Erlang 19.3 / ldap_mnesia / 4bc58f4
Reports root/ big
OK: 1077 / Failed: 0 / User-skipped: 86 / Auto-skipped: 0

@fenek fenek merged commit afa472a into master Nov 28, 2018
@fenek fenek deleted the pubsub-node-mnesia-backend branch November 28, 2018 08:51
@fenek fenek added this to the 3.2.0++ milestone Jan 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants