Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cets/prevent crash on upgrade #4256

Merged
merged 3 commits into from
Apr 9, 2024
Merged

Conversation

chrzaszcz
Copy link
Member

@chrzaszcz chrzaszcz commented Apr 9, 2024

Add CETS major.minor version as a suffix to cluster name in the discovery_nodes table, preventing CETS from crashing if a rolling upgrade is performed, and the CETS versions are incompatible.

This would occur when upgrading MIM from 6.2.0 to the upcoming 6.2.1, which translates to CETS upgrade from 0.1.0 to 0.2.0. If only the patch version of CETS is changed, e.g. 0.2.0 to 0.2.1, they are still considered compatible. From now on, we only need to ensure that CETS versions are marked accordingly before MIM releases.

This change was tested manually by upgrading from 6.2.0 to this PR. The test was performed twice:

  1. In local environment: 3-node cluster created with test-helper,
  2. With helm upgrade.

Note: The version is not a separate DB column, because the nodes running version 6.2.0 would still try to connect to the upgraded nodes. Extending the name seems like a sane option, and similar solutions are used in well-known tools and platforms like k8s.

When performing a rolling upgrade e.g. from 6.2.0 to the upcoming 6.2.1,
CETS would crash on all nodes due to version incompatibility.

This can be prevented by putting cets 'Major.Minor' version in the
cluster name, making the two cluster components disjoint during
upgrade.

The only thing to keep in mind is to bump the Major or Minor version
number when there is any incompatibility in the CETS inter-node API.
@chrzaszcz chrzaszcz changed the base branch from master to migration-6.2.1 April 9, 2024 10:04
@mongoose-im
Copy link
Collaborator

mongoose-im commented Apr 9, 2024

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / e4ef412
Reports root/ big
OK: 437 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0


small_tests_25 / small_tests / e4ef412
Reports root / small


small_tests_26 / small_tests / e4ef412
Reports root / small


small_tests_26_arm64 / small_tests / e4ef412
Reports root / small


ldap_mnesia_25 / ldap_mnesia / e4ef412
Reports root/ big
OK: 2283 / Failed: 1 / User-skipped: 897 / Auto-skipped: 0

pubsub_SUITE:dag+basic:request_all_items_test
{error,{{badmatch,false},
    [{pubsub_tools,check_response,2,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,444}]},
     {pubsub_tools,receive_response,3,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,434}]},
     {pubsub_tools,receive_and_check_response,4,
             [{file,"/home/circleci/project/big_tests/tests/pubsub_tools.erl"},
            {line,424}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


ldap_mnesia_26 / ldap_mnesia / e4ef412
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0


dynamic_domains_mysql_redis_26 / mysql_redis / e4ef412
Reports root/ big
OK: 4508 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0


pgsql_cets_26 / pgsql_cets / e4ef412
Reports root/ big
OK: 4428 / Failed: 0 / User-skipped: 176 / Auto-skipped: 30

graphql_cets_SUITE:init_per_suite
{'EXIT',{{timeout,true,[{times,50,false}],ok},
     [{mongoose_helper,do_wait_until,2,
               [{file,"/home/circleci/project/big_tests/../test/common/mongoose_helper.erl"},
              {line,359}]},
      {graphql_cets_SUITE,init_per_suite,1,
                [{file,"/home/circleci/project/big_tests/tests/graphql_cets_SUITE.erl"},
                 {line,51}]},
      {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
      {test_server,run_test_case_eval1,6,
             [{file,"test_server.erl"},{line,1390}]},
      {test_server,run_test_case_eval,9,
             [{file,"test_server.erl"},{line,1234}]}]}}

Report log


internal_mnesia_26 / internal_mnesia / e4ef412
Reports root/ big
OK: 2424 / Failed: 0 / User-skipped: 757 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412
Reports root/ big
OK: 4538 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0


mysql_redis_26 / mysql_redis / e4ef412
Reports root/ big
OK: 4909 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


pgsql_mnesia_26 / pgsql_mnesia / e4ef412
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / e4ef412
Reports root/ big
OK: 4927 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

Copy link

codecov bot commented Apr 9, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.41%. Comparing base (cb92b6a) to head (082f1e0).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4256      +/-   ##
==========================================
- Coverage   84.43%   84.41%   -0.03%     
==========================================
  Files         552      552              
  Lines       33547    33551       +4     
==========================================
- Hits        28327    28322       -5     
- Misses       5220     5229       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Base automatically changed from migration-6.2.1 to master April 9, 2024 10:33
@mongoose-im
Copy link
Collaborator

mongoose-im commented Apr 9, 2024

elasticsearch_and_cassandra_26 / elasticsearch_and_cassandra_mnesia / 082f1e0
Reports root/ big
OK: 437 / Failed: 0 / User-skipped: 43 / Auto-skipped: 0


small_tests_25 / small_tests / 082f1e0
Reports root / small


small_tests_26 / small_tests / 082f1e0
Reports root / small


small_tests_26_arm64 / small_tests / 082f1e0
Reports root / small


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4540 / Failed: 1 / User-skipped: 111 / Auto-skipped: 0

graphql_SUITE:tls_enabled:tls_connect_admin_unknown_certificate
{error,{{assertMatch,[{module,graphql_SUITE},
            {line,253},
            {expression,"Result"},
            {pattern,"{ error , { tls_alert , { unknown_ca , _ } } }"},
            {value,{error,connection_closed}}]},
    [{graphql_SUITE,tls_connect_admin_unknown_certificate,1,
            [{file,"/home/circleci/project/big_tests/tests/graphql_SUITE.erl"},
             {line,253}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1291}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1223}]}]}}

Report log


ldap_mnesia_25 / ldap_mnesia / 082f1e0
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0


ldap_mnesia_26 / ldap_mnesia / 082f1e0
Reports root/ big
OK: 2284 / Failed: 0 / User-skipped: 897 / Auto-skipped: 0


dynamic_domains_mysql_redis_26 / mysql_redis / 082f1e0
Reports root/ big
OK: 4508 / Failed: 0 / User-skipped: 144 / Auto-skipped: 0


dynamic_domains_pgsql_mnesia_26 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


internal_mnesia_26 / internal_mnesia / 082f1e0
Reports root/ big
OK: 2424 / Failed: 0 / User-skipped: 757 / Auto-skipped: 0


dynamic_domains_mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4538 / Failed: 0 / User-skipped: 114 / Auto-skipped: 0


pgsql_cets_26 / pgsql_cets / 082f1e0
Reports root/ big
OK: 4456 / Failed: 0 / User-skipped: 178 / Auto-skipped: 0


pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


pgsql_mnesia_26 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4930 / Failed: 0 / User-skipped: 118 / Auto-skipped: 0


mysql_redis_26 / mysql_redis / 082f1e0
Reports root/ big
OK: 4909 / Failed: 0 / User-skipped: 139 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4922 / Failed: 5 / User-skipped: 121 / Auto-skipped: 0

mod_blocking_SUITE:manage:remove_user_from_blocklist
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-remove_user_from_blocklist/1-fun-0-',2,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,196}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mod_blocking_SUITE:manage:add_another_user_to_blocklist
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-add_another_user_to_blocklist/1-fun-0-',2,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,176}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log

mod_blocking_SUITE:manage:add_many_users_to_blocklist
{error,{{badmatch,0},
    [{escalus_story,drop_presences,2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,191}]},
     {escalus_story,'-start_ready_clients/2-fun-0-',3,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,135}]},
     {lists,foldl,3,[{file,"lists.erl"},{line,1594}]},
     {escalus_story,start_ready_clients,2,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,128}]},
     {lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
     {lists,flatmap_1,2,[{file,"lists.erl"},{line,1579}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,69}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]}]}}

Report log

mod_blocking_SUITE:manage:remove_many_user_from_blocklist
{error,
  {{badmatch,
     {error,
       {connection_step_failed,
         {{escalus_session,session},
          {client,
            <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635@localhost/res1">>,
            escalus_tcp,<0.64064.0>,undefined,
            [{event_client,
               [{event_manager,<0.64036.0>},
              {server,<<"localhost">>},
              {username,
                <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
              {resource,<<"res1">>}]},
             {resource,<<"res1">>},
             {username,
               <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
             {server,<<"localhost">>},
             {host,<<"localhost">>},
             {port,5222},
             {auth,{escalus_auth,auth_plain}},
             {wspath,undefined},
             {username,
               <<"kate_unnamed_2625_remove_many_user_from_blocklist_2635">>},
             {server,<<"localhost">>},
             {password,<<"makrowe;p">>},
             {stream_id,<<"65f6807913ba58f2">>}]},
          [{compression,false},
           {starttls,true},
           {stream_management,true},
           {advanced_message_processing,true},
           {client_state_indication,false},
           {sasl_mechanisms,[<<"SCRAM-SHA-256">>,<<"PLAIN">>]},
           {caps,undefined}]},
         {timeout,session_reply}}}},
   [{escalus_story,'-start_ready_clients/2-fun-0-',3,
      [{file,
         "/home/circleci/project/big_tests/_...

Report log

mod_blocking_SUITE:manage:clear_blocklist
{error,{{assertion_failed,assert_many,false,
              [is_iq_result,#Fun<mod_blocking_SUITE.32.129816315>],
              [],[]},
    [{escalus_new_assert,assert_true,2,
               [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
                {line,84}]},
     {mod_blocking_SUITE,'-clear_blocklist/1-fun-0-',3,
               [{file,"/home/circleci/project/big_tests/tests/mod_blocking_SUITE.erl"},
                {line,217}]},
     {escalus_story,story,4,
            [{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
             {line,72}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1793}]},
     {test_server,run_test_case_eval1,6,
            [{file,"test_server.erl"},{line,1302}]},
     {test_server,run_test_case_eval,9,
            [{file,"test_server.erl"},{line,1234}]}]}}

Report log


dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 082f1e0
Reports root/ big
OK: 4541 / Failed: 0 / User-skipped: 111 / Auto-skipped: 0


mssql_mnesia_26 / odbc_mssql_mnesia / 082f1e0
Reports root/ big
OK: 4927 / Failed: 0 / User-skipped: 121 / Auto-skipped: 0

@chrzaszcz chrzaszcz marked this pull request as ready for review April 9, 2024 12:34
Copy link
Contributor

@JanuszJakubiec JanuszJakubiec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@JanuszJakubiec JanuszJakubiec merged commit e798ab4 into master Apr 9, 2024
4 checks passed
@JanuszJakubiec JanuszJakubiec deleted the cets/prevent-crash-on-upgrade branch April 9, 2024 14:27
@chrzaszcz chrzaszcz added this to the 6.2.1 milestone Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants