-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix for RFE Bug68782 #6
Conversation
Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment: |
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it. |
Hi, thank you for your contribution. Your code has been assigned to an internal queue. Please follow |
…FOR DATABASE Patch #6 To repeat: ./mtr --mem --charset-for-testdb=utf8mb4 join_cache_bka join_cache_bka_nixbnl join_cache_bkaunique join_cache_bnl join_cache_nojb binary join Fix: re-record with explicit charset for queries/tables with mismatch in explain output. Change-Id: I3e40a721f21dba14da362e2aba7a3935f7fc7627
In initData(), a new array receivedProcessInfo is dynamically allocated to hold one ProcessInfo struct for each configured API and MGM node.
THREE BYTES ON X86 Analysis: ========= The macro uint3korr reads 4 bytes of data instead of 3 on on x86 machines. Multiple definitions were created for this macro for optimization in WIN32. The idea was to optimize reading of 3 byte ints by reading an ordinary int and masking away the unused byte. However this is an undefined behavior. It will be an issue unless users are aware of allocating an extra byte for using this macro. Fix: ==== Removing the definition which reads 4 bytes of data. The only definition of this macro would now read just 3 bytes of data thus prohibiting the usage of an extra byte. Note: ===== This is a backport of Patches #5 and #6 for Bug#17922198.
In WL-included builds ASAN run witnessed missed ~Query_log_event invocation. The destruct-or was not called due to the WL's changes in the error propagation that specifically affect LC MTS. The failure is exposed in particular by rpl_trigger as the following stack: #0 0x9ecd98 in __interceptor_malloc (/export/home/pb2/test/sb_2-22611026-1489061390.32/mysql-commercial-8.0.1-dmr-linux-x86_64-asan/bin/mysqld+0x9ecd98) mysql#1 0x2b1a245 in my_raw_malloc(unsigned long, int) obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:209:12 mysql#2 0x2b1a245 in my_malloc obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:72 mysql#3 0x2940590 in Query_log_event::Query_log_event(char const*, unsigned int, binary_log::Format_description_event const*, binary_log::Log_event_type) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:4343:46 mysql#4 0x293d235 in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, bool) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:1686:17 mysql#5 0x293b96f in Log_event::read_log_event() mysql#6 0x2a2a1c9 in next_event(Relay_log_info*) Previously before the WL Mts_submode_logical_clock::wait_for_workers_to_finish() had not returned any error even when Coordinator thread is killed. The WL patch needed to refine such behavior, but at doing so it also had to attend log_event.cc::schedule_next_event() to register an error to follow an existing pattern. While my_error() does not take place the killed Coordinator continued scheduling, ineffectively though - no Worker gets engaged (legal case of deferred scheduling), and without noticing its killed status up to a point when it resets the event pointer in apply_event_and_update_pos(): *ptr_ev= NULL; // announcing the event is passed to w-worker The reset was intended for an assigned Worker to perform the event destruction or by Coordinator itself when the event is deferred. As neither is the current case the event gets unattended for its termination. In contrast in the pre-WL sources the killed Coordinator does find a Worker. However such Worker could be already down (errored out and exited), in which case apply_event_and_update_pos() reasonably returns an error and executes delete ev in exec_relay_log_event() error branch. **Fixed** with deploying my_error() call in log_event.cc::schedule_next_event() error branch which fits to the existing pattern. THD::is_error() has been always checked by Coordinator before any attempt to reset *ptr_ev= NULL. In the errored case Coordinator does not reset and destroys the event itself in the exec_relay_log_event() error branch pretty similarly to how the pre-WL sources do. Tested against rpl_trigger and rpl suites to pass. Approved on rb#15667.
Patch #6: Use depth-first search in Json_dom::seek(), like in Json_wrapper::seek(). This allows the search to stop earlier than the original breadth-first search when the need_only_one flag is given. It also avoids building up a vector of candidate results for each path leg. Microbenchmarks (64-bit, Intel Core i7-4770 3.4 GHz, GCC 6.3): BM_JsonDomSearchEllipsis 21666 ns/iter [ +0.5%] BM_JsonDomSearchEllipsis_OnlyOne 135 ns/iter [+11371.1%] BM_JsonDomSearchKey 120 ns/iter [ +5.0%] BM_JsonBinarySearchEllipsis 71022 ns/iter [ -0.1%] BM_JsonBinarySearchEllipsis_OnlyOne 98 ns/iter [ -1.0%] BM_JsonBinarySearchKey 80 ns/iter [ -1.3%] Change-Id: Ice7476f6e087a80d69474a38ab4fca918667b332
…TABLE_UPGRADE_GUARD To repeat: cmake -DWITH_ASAN=1 -DWITH_ASAN_SCOPE=1 ./mtr --mem --sanitize main.dd_upgrade_error A few dd tests fail with: ==26861==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7000063bf5e8 at pc 0x00010d4dbe8b bp 0x7000063bda40 sp 0x7000063bda38 READ of size 8 at 0x7000063bf5e8 thread T2 #0 0x10d4dbe8a in Prealloced_array<st_plugin_int**, 16ul>::empty() const prealloced_array.h:186 #1 0x10d406a8b in lex_end(LEX*) sql_lex.cc:560 #2 0x10dae4b6d in dd::upgrade::Table_upgrade_guard::~Table_upgrade_guard() (mysqld:x86_64+0x100f87b6d) #3 0x10dadc557 in dd::upgrade::migrate_table_to_dd(THD*, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, bool) (mysqld:x86_64+0x100f7f557) #4 0x10dad7e85 in dd::upgrade::migrate_plugin_table_to_dd(THD*) (mysqld:x86_64+0x100f7ae85) #5 0x10daec6a1 in dd::upgrade::do_pre_checks_and_initialize_dd(THD*) upgrade.cc:1216 #6 0x10cd0a5c0 in bootstrap::handle_bootstrap(void*) bootstrap.cc:336 Change-Id: I265ec6dd97ee8076aaf03763840c0cdf9e20325b Fix: increase lifetime of 'LEX lex;' which is used by 'table_guard'
…E TO A SERVER Problem ======================================================================== Running the GCS tests with ASAN seldomly reports a user-after-free of the server reference that the acceptor_learner_task uses. Here is an excerpt of ASAN's output: ==43936==ERROR: AddressSanitizer: heap-use-after-free on address 0x63100021c840 at pc 0x000000530ff8 bp 0x7fc0427e8530 sp 0x7fc0427e8520 WRITE of size 8 at 0x63100021c840 thread T3 #0 0x530ff7 in server_detected /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:962 #1 0x533814 in buffered_read_bytes /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1249 #2 0x5481af in buffered_read_msg /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1399 #3 0x51e171 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4690 #4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) #9 0x7fc047ff2bfc in __clone (/lib64/libc.so.6+0xfebfc) 0x63100021c840 is located 64 bytes inside of 65688-byte region [0x63100021c800,0x63100022c898) freed by thread T3 here: #0 0x7fc04a5d7508 in __interceptor_free (/lib64/libasan.so.4+0xde508) #1 0x52cf86 in freesrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:836 #2 0x52ea78 in srv_unref /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:868 #3 0x524c30 in reply_handler_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4914 #4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) previously allocated by thread T3 here: #0 0x7fc04a5d7a88 in __interceptor_calloc (/lib64/libasan.so.4+0xdea88) #1 0x543604 in mksrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:721 #2 0x543b4c in addsrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:755 #3 0x54af61 in update_servers /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1747 #4 0x501082 in site_install_action /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1572 #5 0x55447c in import_config /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/site_def.c:486 #6 0x506dfc in handle_x_snapshot /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5257 #7 0x50c444 in xcom_fsm /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5325 #8 0x516c36 in dispatch_op /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4510 #9 0x521997 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4772 #10 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140 #11 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324 #12 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164 #13 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107 #14 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4) Analysis ======================================================================== The server structure is reference counted by the associated sender_task and reply_handler_task. When they finish, they unreference the server, which leads to its memory being freed. However, the acceptor_learner_task keeps a "naked" reference to the server structure. Under the right ordering of operations, i.e. the sender_task and reply_handler_task terminating after the acceptor_learner_task acquires, but before it uses, the reference to the server structure, leads to the acceptor_learner_task accessing the server structure after it has been freed. Solution ======================================================================== Let the acceptor_learner_task also reference count the server structure so it is not freed while still in use. Reviewed-by: André Negrão <[email protected]> Reviewed-by: Venkatesh Venugopal <[email protected]> RB: 21209
…TH VS 2019 [#6] [noclose] storage\ndb\src\kernel\blocks\ndbfs\Ndbfs.cpp(1816,40): warning C4311: 'type cast': pointer truncation from 'AsyncIoThread *' to 'long' Change-Id: Ia9d05864ae600b83788a01f16dd6bb2e47202dda
Add more instructions for the ECR command
The NdbDictionary::Event returned from NdbDictionar::Dictionary::getEvent() need to be released. Fix by: - Improve function description of NdbDictionary:Dictionary::getEvent() to indicate that the returned Event need to be released. - Add new NdbDictionary:Dictionary::releaseEvent() for releasing the event returned. - Add RAII support to NdbApi for managing the lifetime NdbDictionary::Event using std::unique_ptr, this is enabled when compiling with at least C++11 support. This patch fixes potential memory leaks in various places as well as the one in NdbInfoScanVirtual that was found by valgrind. Change-Id: Ib52133abff1cd28eef17aeb6cb0527de9fab9574
# This is the 1st commit message: WL#15280: HEATWAVE SUPPORT FOR MDS HA Problem Statement ----------------- Currently customers cannot enable heatwave analytics service to their HA DBSystem or enable HA if they are using Heatwave enabled DBSystem. In this change, we attempt to remove this limitation and provide failover support of heatwave in an HA enabled DBSystem. High Level Overview ------------------- To support heatwave with HA, we extended the existing feature of auto- reloading of tables to heatwave on MySQL server restart (WL-14396). To provide seamless failover functionality to tables loaded to heatwave, each node in the HA cluster (group replication) must have the latest view of tables which are currently loaded to heatwave cluster attached to the primary, i.e., the secondary_load flag should be in-sync always. To achieve this, we made following changes - 1. replicate secondary load/unload DDL statements to all the active secondary nodes by writing the DDL into the binlog, and 2. Control how secondary load/unload is executed when heatwave cluster is not attached to node executing the command Implementation Details ---------------------- Current implementation depends on two key assumptions - 1. All MDS DBSystems will have RAPID plugin installed. 2. No non-MDS system will have the RAPID plugin installed. Based on these assumptions, we made certain changes w.r.t. how server handles execution of secondary load/unload statements. 1. If secondary load/unload command is executed from a mysql client session on a system without RAPID plugin installed (i.e., non-MDS), instead of an error, a warning message will be shown to the user, and the DDL is allowed to commit. 2. If secondary load/unload command is executed from a replication connection on an MDS system without heatwave cluster attached, instead of throwing an error, the DDL is allowed to commit. 3. If no error is thrown from secondary engine, then the DDL will update the secondary_load metadata and write a binlog entry. Writing to binlog implies that all the consumer of binlog now need to handle this DDL gracefully. This has an adverse effect on Point-in-time Recovery. If the PITR backup is taken from a DBSystem with heatwave, it may contain traces of secondary load/unload statements in its binlog. If such a backup is used to restore a new DBSystem, it will cause failure while trying to execute statements from its binlog because a) DBSystem will not heatwave cluster attached at this time, and b) Statements from binlog are executed from standard mysql client connection, thus making them indistinguishable from user executed command. Customers will be prevented (by control plane) from using PITR functionality on a heatwave enabled DBSystem until there is a solution for this. Testing ------- This commit changes the behavior of secondary load/unload statements, so it - adjusts existing tests' expectations, and - adds a new test validating new DDL behavior under different scenarios Change-Id: Ief7e9b3d4878748b832c366da02892917dc47d83 # This is the commit message #2: WL#15280: HEATWAVE SUPPORT FOR MDS HA (PITR SUPPORT) Problem ------- A PITR backup taken from a heatwave enabled system could have traces of secondary load or unload statements in binlog. When such a backup is used to restore another system, it can cause failure because of following two reasons: 1. Currently, even if the target system is heatwave enabled, heatwave cluster is attached only after PITR restore phase completes. 2. When entries from binlogs are applied, a standard mysql client connection is used. This makes it indistinguishable from other user session. Since secondary load (or unload) statements are meant to throw error when they are executed by user in the absence of a healthy heatwave cluster, PITR restore workflow will fail if binlogs from the backup have any secondary load (or unload) statements in them. Solution -------- To avoid PITR failure, we are introducing a new system variable rapid_enable_delayed_secondary_ops. It controls how load or unload commands are to be processed by rapid plugin. - When turned ON, the plugin silently skips the secondary engine operation (load/unload) and returns success to the caller. This allows secondary load (or unload) statements to be executed by the server in the absence of any heatwave cluster. - When turned OFF, it follows the existing behavior. - The default value is OFF. - The value can only be changed when rapid_bootstrap is IDLE or OFF. - This variable cannot be persisted. In PITR workflow, Control Plane would set the variable at the start of PITR restore and then reset it at the end of workflow. This allows the workflow to complete without failure even when heatwave cluster is not attached. Since metadata is always updated when secondary load/unload DDLs are executed, when heatwave cluster is attached at a later point in time, the respective tables get reloaded to heatwave automatically. Change-Id: I42e984910da23a0e416edb09d3949989159ef707 # This is the commit message #3: WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST CHANGES) This commit adds new functional tests for the MDS HA + HW integration. Change-Id: Ic818331a4ca04b16998155efd77ac95da08deaa1 # This is the commit message #4: WL#15280: HEATWAVE SUPPORT FOR MDS HA BUG#34776485: RESTRICT DEFAULT VALUE FOR rapid_enable_delayed_secondary_ops This commit does two things: 1. Add a basic test for newly introduced system variable rapid_enable_delayed_secondary_ops, which controls the behavior of alter table secondary load/unload ddl statements when rapid cluster is not available. 2. It also restricts the DEFAULT value setting for the system variable So, following is not allowed: SET GLOBAL rapid_enable_delayed_secondary_ops = default This variable is to be used in restricted scenarios and control plane only sets it to ON/OFF before and after PITR apply. Allowing set to default has no practical use. Change-Id: I85c84dfaa0f868dbfc7b1a88792a89ffd2e81da2 # This is the commit message #5: Bug#34726490: ADD DIAGNOSTICS FOR SECONDARY LOAD / UNLOAD DDL Problem: -------- If secondary load or unload DDL gets rolled back due to some error after it had loaded / unloaded the table in heatwave cluster, there is no undo of the secondary engine action. Only secondary_load flag update is reverted and binlog is not written. From User's perspective, the table is loaded and can be seen on performance_schema. There are also no error messages printed to notify that the ddl didn't commit. This creates a problem to debug any issue in this area. Solution: --------- The partial undo of secondary load/unload ddl will be handled in bug#34592922. In this commit, we add diagnostics to reveal if the ddl failed to commit, and from what stage. Change-Id: I46c04dd5dbc07fc17beb8aa2a8d0b15ddfa171af # This is the commit message #6: WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST FIX) Since ALTER TABLE SECONDARY LOAD / UNLOAD DDL statements now write to binlog, from Heatwave's perspective, SCN is bumped up. In this commit, we are adjusting expected SCN values in certain tests which does secondary load/unload and expects SCN to match. Change-Id: I9635b3cd588d01148d763d703c72cf50a0c0bb98 # This is the commit message mysql#7: Adding MTR tests for ML in rapid group_replication suite Added MTR tests with Heatwave ML queries with in an HA setup. Change-Id: I386a3530b5bbe6aea551610b6e739ab1cf366439 # This is the commit message mysql#8: WL#15280: HEATWAVE SUPPORT FOR MDS HA (MTR TEST ADJUSTMENT) In this commit we have adjusted the existing test to work with the new MTR test infrastructure which extends the functionalities to HA landscape. With this change, a lot of mannual settings have now become redundant and thus removed in this commit. Change-Id: Ie1f4fcfdf047bfe8638feaa9f54313d509cbad7e # This is the commit message mysql#9: WL#15280: HEATWAVE SUPPORT FOR MDS HA (CLANG-TIDY FIX) Fix clang-tidy warnings found in previous change#16530, patch#20 Change-Id: I15d25df135694c2f6a3a9146feebe2b981637662 Change-Id: I3f3223a85bb52343a4619b0c2387856b09438265
…rters The socket in parent class Transporter becomes an NdbSocket here. In the SHM and Loopback transporters, use the NdbSocket without expecting to ever find SSL enabled on it. In the TCP transporter: * doSend() returns true in the TLS_BUSY_TRY_AGAIN case. The calling function is TransporterRegistry::performSend(), and true here means "true if anything remains to be sent. Will require another performSend()." * doReceive() handles TLS_BUSY_TRY_AGAIN by returning it up to the caller, TransporterRegistry::performReceive(), where no special handling is required. Change-Id: I97a0cd390dd837bec207e6f169dbd9d71871eff7
…ting_sharing ASAN reports a use-after-free in the exit-handlers of the routing-sharing integration tests: AddressSanitizer: heap-use-after-free on address ... ... mysql#4 0x708d05 in std::default_delete<SharedRestartableRouter>:: operator()(SharedRestartableRouter*) const .../include/c++/9/bits/unique_ptr.h:81:2 mysql#5 0x708c98 in std::unique_ptr<SharedRestartableRouter, std::default_delete<SharedRestartableRouter> >::~unique_ptr() .../include/c++/9/bits/unique_ptr.h:292:4 mysql#6 0x6a7448 in std::array<std::unique_ptr<SharedRestartableRouter, std::default_delete<SharedRestartableRouter> >, 3ul>::~array() .../include/c++/9/array:94:12 mysql#7 0x7f160a5e68d6 in __run_exit_handlers .../glibc-2.31/stdlib/exit.c:108:8 ... as the RestartedRouters are destructed after their process-manager is destructed. - The process manager is owned by the test - The RestartableRouters are 'static' and outlive the test. Change ------ - explicitely free the intermediate routers at test-suite teardown to ensure the proper sequence. Change-Id: Id405fb26b0519c49820a58b5f70d0ee59d2cb83f
…etwork https://bugs.mysql.com/bug.php?id=109668 Description ----------- GR suffered from problems caused by the security probes and network scanner processes connecting to the group replication communication port. This usually is not a problem, but poses a serious threat when another member tries to join the cluster by initialting a connection to the member which is affected by external processes using the port dedicated for group communication for longer durations. On such activites by external processes, the SSL enabled server stalled forever on the SSL_accept() call waiting for handshake data. Below is the stacktrace: Thread 55 (Thread 0x7f7bb77ff700 (LWP 2198598)): #0 in read () mysql#1 in sock_read () mysql#2 in BIO_read () mysql#3 in ssl23_read_bytes () mysql#4 in ssl23_get_client_hello () mysql#5 in ssl23_accept () mysql#6 in xcom_tcp_server_startup(Xcom_network_provider*) () When the server stalled in the above path forever, it prohibited other members to join the cluster resulting in the following messages on the joiner server's logs. [ERROR] [MY-011640] [Repl] Plugin group_replication reported: 'Timeout on wait for view after joining group' [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member is already leaving or joining a group.' Solution -------- This patch adds two new variables 1. group_replication_xcom_ssl_socket_timeout It is a file-descriptor level timeout in seconds for both accept() and SSL_accept() calls when group replication is listening on the xcom port. When set to a valid value, say for example 5 seconds, both accept() and SSL_accept() return after 5 seconds. The default value has been set to 0 (waits infinitely) for backward compatibility. This variable is effective only when GR is configred with SSL. 2. group_replication_xcom_ssl_accept_retries It defines the number of retries to be performed before closing the socket. For each retry the server thread calls SSL_accept() with timeout defined by the group_replication_xcom_ssl_socket_timeout for the SSL handshake process once the connection has been accepted by the first accept() call. The default value has been set to 10. This variable is effective only when GR is configred with SSL. Note: - Both of the above variables are dynamically configurable, but will become effective only on START GROUP_REPLICATION. - This patch is only for the Linux systems.
Part of WL#15135 Certificate Architecture Add Jtie mappings for Ndb_cluster_connection.configure_tls(). Add native method configure_tls() to Java class Ndb_cluster_connection. Add public Java method configureTls to interface ClusterConnection in clusterj-core. Map the TLS search path to a new Cluster/J connection property, com.mysql.clusterj.tls.path Cluster/J test cases will come as part of later worklogs. Change-Id: Id193d9a33a47eb7638abc58184fb990dca95db75
This patch exposes the transporter-level encrypted flag for publication in ndbinfo.transporters. Change-Id: I7bab7d437aaae001c4d4298ef9f827632401785f
This completes the authorization framework introduced in the previous patch, adding the client side and a test case. It adds three new error codes to the MGM API: the generic code NDB_MGM_NOT_AUTHORIZED, and two more specific codes, NDB_MGM_AUTH_REQUIRES_TLS and NDB_MGM_AUTH_REQUIRES_CLIENT_CERT. A new class RewindInputStream helps the MGM client to reset its protocol parsing context whenever it recieves an authorization failure from the server. The test case is testMgmd -n RequireTls Change-Id: I9cc8bbbad5c1131f6de2fb4b3a6f4b11e82df7e3
…and a local DDL executed https://bugs.mysql.com/bug.php?id=113727 Problem ------- In high concurrency scenarios, MySQL replica can enter into a deadlock due to a race condition between the replica applier thread and the client thread performing a binlog group commit. Analysis -------- It needs at least 3 threads for this deadlock to happen 1. One client thread 2. Two replica applier threads How this deadlock happens? -------------------------- 0. Binlog is enabled on replica, but log_replica_updates is disabled. 1. Initially, both "Commit Order" and "Binlog Flush" queues are empty. 2. Replica applier thread 1 enters the group commit pipeline to register in the "Commit Order" queue since `log-replica-updates` is disabled on the replica node. 3. Since both "Commit Order" and "Binlog Flush" queues are empty, the applier thread 1 3.1. Becomes leader (In Commit_stage_manager::enroll_for()). 3.2. Registers in the commit order queue. 3.3. Acquires the lock MYSQL_BIN_LOG::LOCK_log. 3.4. Commit Order queue is emptied, but the lock MYSQL_BIN_LOG::LOCK_log is not yet released. NOTE: SE commit for applier thread is already done by the time it reaches here. 4. Replica applier thread 2 enters the group commit pipeline to register in the "Commit Order" queue since `log-replica-updates` is disabled on the replica node. 5. Since the "Commit Order" queue is empty (emptied by applier thread 1 in 3.4), the applier thread 2 5.1. Becomes leader (In Commit_stage_manager::enroll_for()) 5.2. Registers in the commit order queue. 5.3. Tries to acquire the lock MYSQL_BIN_LOG::LOCK_log. Since it is held by applier thread 1 it will wait until the lock is released. 6. Client thread enters the group commit pipeline to register in the "Binlog Flush" queue. 7. Since "Commit Order" queue is not empty (there is applier thread 2 in the queue), it enters the conditional wait `m_stage_cond_leader` with an intention to become the leader for both the "Binlog Flush" and "Commit Order" queues. 8. Applier thread 1 releases the lock MYSQL_BIN_LOG::LOCK_log and proceeds to update the GTID by calling gtid_state->update_commit_group() from Commit_order_manager::flush_engine_and_signal_threads(). 9. Applier thread 2 acquires the lock MYSQL_BIN_LOG::LOCK_log. 9.1. It checks if there is any thread waiting in the "Binlog Flush" queue to become the leader. Here it finds the client thread waiting to be the leader. 9.2. It releases the lock MYSQL_BIN_LOG::LOCK_log and signals on the cond_var `m_stage_cond_leader` and enters a conditional wait until the thread's `tx_commit_pending` is set to false by the client thread (will be done in the Commit_stage_manager::process_final_stage_for_ordered_commit_group() called by client thread from fetch_and_process_flush_stage_queue()). 10. The client thread wakes up from the cond_var `m_stage_cond_leader`. The thread has now become a leader and it is its responsibility to update GTID of applier thread 2. 10.1. It acquires the lock MYSQL_BIN_LOG::LOCK_log. 10.2. Returns from `enroll_for()` and proceeds to process the "Commit Order" and "Binlog Flush" queues. 10.3. Fetches the "Commit Order" and "Binlog Flush" queues. 10.4. Performs the storage engine flush by calling ha_flush_logs() from fetch_and_process_flush_stage_queue(). 10.5. Proceeds to update the GTID of threads in "Commit Order" queue by calling gtid_state->update_commit_group() from Commit_stage_manager::process_final_stage_for_ordered_commit_group(). 11. At this point, we will have - Client thread performing GTID update on behalf if applier thread 2 (from step 10.5), and - Applier thread 1 performing GTID update for itself (from step 8). Due to the lack of proper synchronization between the above two threads, there exists a time window where both threads can call gtid_state->update_commit_group() concurrently. In subsequent steps, both threads simultaneously try to modify the contents of the array `commit_group_sidnos` which is used to track the lock status of sidnos. This concurrent access to `update_commit_group()` can cause a lock-leak resulting in one thread acquiring the sidno lock and not releasing at all. ----------------------------------------------------------------------------------------------------------- Client thread Applier Thread 1 ----------------------------------------------------------------------------------------------------------- update_commit_group() => global_sid_lock->rdlock(); update_commit_group() => global_sid_lock->rdlock(); calls update_gtids_impl_lock_sidnos() calls update_gtids_impl_lock_sidnos() set commit_group_sidno[2] = true set commit_group_sidno[2] = true lock_sidno(2) -> successful lock_sidno(2) -> waits update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()` if (commit_group_sidnos[2]) { unlock_sidno(2); commit_group_sidnos[2] = false; } Applier thread continues.. lock_sidno(2) -> successful update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()` if (commit_group_sidnos[2]) { <=== this check fails and lock is not released. unlock_sidno(2); commit_group_sidnos[2] = false; } Client thread continues without releasing the lock ----------------------------------------------------------------------------------------------------------- 12. As the above lock-leak can also happen the other way i.e, the applier thread fails to unlock, there can be different consequences hereafter. 13. If the client thread continues without releasing the lock, then at a later stage, it can enter into a deadlock with the applier thread performing a GTID update with stack trace. Client_thread ------------- mysql#1 __GI___lll_lock_wait mysql#2 ___pthread_mutex_lock mysql#3 native_mutex_lock <= waits for commit lock while holding sidno lock mysql#4 Commit_stage_manager::enroll_for mysql#5 MYSQL_BIN_LOG::change_stage mysql#6 MYSQL_BIN_LOG::ordered_commit mysql#7 MYSQL_BIN_LOG::commit mysql#8 ha_commit_trans mysql#9 trans_commit_implicit mysql#10 mysql_create_like_table mysql#11 Sql_cmd_create_table::execute mysql#12 mysql_execute_command mysql#13 dispatch_sql_command Applier thread -------------- mysql#1 ___pthread_mutex_lock mysql#2 native_mutex_lock mysql#3 safe_mutex_lock mysql#4 Gtid_state::update_gtids_impl_lock_sidnos <= waits for sidno lock mysql#5 Gtid_state::update_commit_group mysql#6 Commit_order_manager::flush_engine_and_signal_threads <= acquires commit lock here mysql#7 Commit_order_manager::finish mysql#8 Commit_order_manager::wait_and_finish mysql#9 ha_commit_low mysql#10 trx_coordinator::commit_in_engines mysql#11 MYSQL_BIN_LOG::commit mysql#12 ha_commit_trans mysql#13 trans_commit mysql#14 Xid_log_event::do_commit mysql#15 Xid_apply_log_event::do_apply_event_worker mysql#16 Slave_worker::slave_worker_exec_event mysql#17 slave_worker_exec_job_group mysql#18 handle_slave_worker 14. If the applier thread continues without releasing the lock, then at a later stage, it can perform recursive locking while setting the GTID for the next transaction (in set_gtid_next()). In debug builds the above case hits the assertion `safe_mutex_assert_not_owner()` meaning the lock is already acquired by the replica applier thread when it tries to re-acquire the lock. Solution -------- In the above problematic example, when seen from each thread individually, we can conclude that there is no problem in the order of lock acquisition, thus there is no need to change the lock order. However, the root cause for this problem is that multiple threads can concurrently access to the array `Gtid_state::commit_group_sidnos`. In its initial implementation, it was expected that threads should hold the `MYSQL_BIN_LOG::LOCK_commit` before modifying its contents. But it was not considered when upstream implemented WL#7846 (MTS: slave-preserve-commit-order when log-slave-updates/binlog is disabled). With this patch, we now ensure that `MYSQL_BIN_LOG::LOCK_commit` is acquired when the client thread (binlog flush leader) when it tries to perform GTID update on behalf of threads waiting in "Commit Order" queue, thus providing a guarantee that `Gtid_state::commit_group_sidnos` array is never accessed without the protection of `MYSQL_BIN_LOG::LOCK_commit`.
Problem: Starting ´ndb_mgmd --bind-address´ may potentially cause abnormal program termination in MgmtSrvr destructor when ndb_mgmd restart itself. Core was generated by `ndb_mgmd --defa'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f8ce4066b8f in raise () from /lib64/libc.so.6 #1 0x00007f8ce4039ea5 in abort () from /lib64/libc.so.6 #2 0x00007f8ce40a7d97 in __libc_message () from /lib64/libc.so.6 #3 0x00007f8ce40af08c in malloc_printerr () from /lib64/libc.so.6 #4 0x00007f8ce40b132d in _int_free () from /lib64/libc.so.6 #5 0x00000000006e9ffe in MgmtSrvr::~MgmtSrvr (this=0x28de4b0) at mysql/8.0/storage/ndb/src/mgmsrv/MgmtSrvr.cpp: 890 #6 0x00000000006ea09e in MgmtSrvr::~MgmtSrvr (this=0x2) at mysql/8.0/ storage/ndb/src/mgmsrv/MgmtSrvr.cpp:849 #7 0x0000000000700d94 in mgmd_run () at mysql/8.0/storage/ndb/src/mgmsrv/main.cpp:260 #8 0x0000000000700775 in mgmd_main (argc=<optimized out>, argv=0x28041d0) at mysql/8.0/storage/ndb/src/ mgmsrv/main.cpp:479 Analysis: While starting up, the ndb_mgmd will allocate memory for bind_address in order to potentially rewrite the parameter. When ndb_mgmd restart itself the memory will be released and dangling pointer causing double free. Fix: Drop support for bind_address=[::], it is not documented anywhere, is not useful and doesn't work. This means the need to rewrite bind_address is gone and bind_address argument need neither alloc or free. Change-Id: I7797109b9d8391394587188d64d4b1f398887e94
… for connection xxx'. The new iterator based explains are not impacted. The issue here is a race condition. More than one thread is using the query term iterator at the same time (whoch is neithe threas safe nor reantrant), and part of its state is in the query terms being visited which leads to interference/race conditions. a) the explain thread uses an iterator here: Sql_cmd_explain_other_thread::execute is inspecting the Query_expression of the running query calling master_query_expression()->find_blocks_query_term which uses an iterator over the query terms in the query expression: for (auto qt : query_terms<>()) { if (qt->query_block() == qb) { return qt; } } the above search fails to find qb due to the interference of the thread b), see below, and then tries to access a nullpointer: * thread #36, name = ‘connection’, stop reason = EXC_BAD_ACCESS (code=1, address=0x0) frame #0: 0x000000010bb3cf0d mysqld`Query_block::type(this=0x00007f8f82719088) const at sql_lex.cc:4441:11 frame #1: 0x000000010b83763e mysqld`(anonymous namespace)::Explain::explain_select_type(this=0x00007000020611b8) at opt_explain.cc:792:50 frame #2: 0x000000010b83cc4d mysqld`(anonymous namespace)::Explain_join::explain_select_type(this=0x00007000020611b8) at opt_explain.cc:1487:21 frame #3: 0x000000010b837c34 mysqld`(anonymous namespace)::Explain::prepare_columns(this=0x00007000020611b8) at opt_explain.cc:744:26 frame #4: 0x000000010b83ea0e mysqld`(anonymous namespace)::Explain_join::explain_qep_tab(this=0x00007000020611b8, tabnum=0) at opt_explain.cc:1415:32 frame #5: 0x000000010b83ca0a mysqld`(anonymous namespace)::Explain_join::shallow_explain(this=0x00007000020611b8) at opt_explain.cc:1364:9 frame #6: 0x000000010b83379b mysqld`(anonymous namespace)::Explain::send(this=0x00007000020611b8) at opt_explain.cc:770:14 frame #7: 0x000000010b834147 mysqld`explain_query_specification(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, query_term=0x00007f8f82719088, ctx=CTX_JOIN) at opt_explain.cc:2088:20 frame #8: 0x000000010bd36b91 mysqld`Query_expression::explain_query_term(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, qt=0x00007f8f82719088) at sql_union.cc:1519:11 frame #9: 0x000000010bd36c68 mysqld`Query_expression::explain_query_term(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, qt=0x00007f8f8271d748) at sql_union.cc:1526:13 frame #10: 0x000000010bd373f7 mysqld`Query_expression::explain(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00) at sql_union.cc:1591:7 frame #11: 0x000000010b835820 mysqld`mysql_explain_query_expression(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, unit=0x00007f8f7a090360) at opt_explain.cc:2392:17 frame #12: 0x000000010b835400 mysqld`explain_query(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, unit=0x00007f8f7a090360) at opt_explain.cc:2353:13 * frame #13: 0x000000010b8363e4 mysqld`Sql_cmd_explain_other_thread::execute(this=0x00007f8fba585b68, thd=0x00007f8fbb111e00) at opt_explain.cc:2531:11 frame #14: 0x000000010bba7d8b mysqld`mysql_execute_command(thd=0x00007f8fbb111e00, first_level=true) at sql_parse.cc:4648:29 frame #15: 0x000000010bb9e230 mysqld`dispatch_sql_command(thd=0x00007f8fbb111e00, parser_state=0x0000700002065de8) at sql_parse.cc:5303:19 frame #16: 0x000000010bb9a4cb mysqld`dispatch_command(thd=0x00007f8fbb111e00, com_data=0x0000700002066e38, command=COM_QUERY) at sql_parse.cc:2135:7 frame #17: 0x000000010bb9c846 mysqld`do_command(thd=0x00007f8fbb111e00) at sql_parse.cc:1464:18 frame #18: 0x000000010b2f2574 mysqld`handle_connection(arg=0x0000600000e34200) at connection_handler_per_thread.cc:304:13 frame #19: 0x000000010e072fc4 mysqld`pfs_spawn_thread(arg=0x00007f8fba8160b0) at pfs.cc:3051:3 frame #20: 0x00007ff806c2b202 libsystem_pthread.dylib`_pthread_start + 99 frame #21: 0x00007ff806c26bab libsystem_pthread.dylib`thread_start + 15 b) the query thread being explained is itself performing LEX::cleanup and as part of the iterates over the query terms, but still allows EXPLAIN of the query plan since thd->query_plan.set_query_plan(SQLCOM_END, ...) hasn't been called yet. 20:frame: Query_terms<(Visit_order)1, (Visit_leaves)0>::Query_term_iterator::operator++() (in mysqld) (query_term.h:613) 21:frame: Query_expression::cleanup(bool) (in mysqld) (sql_union.cc:1861) 22:frame: LEX::cleanup(bool) (in mysqld) (sql_lex.h:4286) 30:frame: Sql_cmd_dml::execute(THD*) (in mysqld) (sql_select.cc:799) 31:frame: mysql_execute_command(THD*, bool) (in mysqld) (sql_parse.cc:4648) 32:frame: dispatch_sql_command(THD*, Parser_state*) (in mysqld) (sql_parse.cc:5303) 33:frame: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in mysqld) (sql_parse.cc:2135) 34:frame: do_command(THD*) (in mysqld) (sql_parse.cc:1464) 57:frame: handle_connection(void*) (in mysqld) (connection_handler_per_thread.cc:304) 58:frame: pfs_spawn_thread(void*) (in mysqld) (pfs.cc:3053) 65:frame: _pthread_start (in libsystem_pthread.dylib) + 99 66:frame: thread_start (in libsystem_pthread.dylib) + 15 Solution: This patch solves the issue by removing iterator state from Query_term, making the query_term iterators thread safe. This solution labels every child query_term with its index in its parent's m_children vector. The iterator can therefore easily compute the next child to visit based on Query_term::m_sibling_idx. A unit test case is added to check reentrancy. One can also manually verify that we have no remaining race condition by running two client connections files (with \. <file>) with a big number of copies of the repro query in one connection and a big number of EXPLAIN format=json FOR <connection>, e.g. EXPLAIN FORMAT=json FOR CONNECTION 8\G in the other. The actual connection number would need to verified in connection one, of course. Change-Id: Ie7d56610914738ccbbecf399ccc4f465f7d26ea7
This adds the channel_name and program_name for the Slave I/O Thread.
https://bugs.mysql.com/bug.php?id=68782
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.