… by server

When server closes the router-server connection while the router waits for the
client to send the next command, router will not close its side of the
router-server connection.

The server side may be closed by:

- opening a 2nd connection and KILL <first-connection-id>
- wait-timeout expiring and closing the connection

Change
------

- when idling wait for a read-event on both client and server
  connection.

- If the server sends something first or closes the
  connection, forward the error to the client and close the client and
  server connection

- If client sends first, stop waiting for input from the server and
  handle the client command.

Change-Id: Ie743d907ea45ae8fa501feca355a983fd685e1c1

…es [postfix]

After the rename of the replication terms for source and replica in the
error-msgs, the routertests started to fail as the expected errormsgs
didn't match anymore.

Change
------

In the binlog related tests, changed the expected error-msg texts to
contain the updated terms:

- allow 'master' and 'source'
- allow 'slave' and 'replica'

Change-Id: I6784fd8fccc287e5321330d5c6fa9611c6b336e8

…d on PB2 weekly 8.0

Test gr_acf_group_member_maintenance was failing due to a
error log message `failed registering replica on source`.
Though the message is expected since the test is triggering source
failures and the respective asynchronous replication channel
reconnection and failover.

To solve the above issue, now we do supress the error log message.

Change-Id: I44906c06a84c7a2ad313a0af015832a4f665b84c

… Signal (get_store_key at sql/sql_select.cc:2383)

These are two related but distinct problems manifested in the shrinkage
of key definitions for derived tables or common table expressions,
implemented in JOIN::finalize_derived_keys().

The problem in Bug#34572040 is that we have two references to one CTE,
each with a valid key definition. The function will first loop over
the first reference (cte_a) and move its used key from position 0 to
position 1. Next, it will attempt to move the key for the second
reference (cte_b) from position 4 to position 2.
However, for each iteration, the function will calculate used key
information. On the first iteration, the values are correct, but since
key value #1 has been moved into position #0, the old information is
invalid and provides wrong information. The problem is thus that for
subsequent iterations we read data that has been invalidated by earlier
key moves.

The best solution to the problem is to move the keys for all references
to the CTE in one operation. This way, we can calculate used keys
information safely, before any move operation has been performed.

The problem in Bug#34634469 is also related to having more than one
reference to a CTE, but in this case the first reference (ref_3) has
a key in position 5 which is moved to position 0, and the second
reference (ref_4) has a key in position 3 that is moved to position 1.

However, the key parts of the first key will overlap with the key parts
of the second key after the first move, thus invalidating the key
structure during the copy. The actual problem is that we move
a higher-numbered key (5) before a lower-numbered key (3), which in
this case makes it impossible to find an empty space for the moved key.

The solution to this problem is to ensure that keys are moved in
increasing key order.

The patch changes the algorithm as follows:

- When identifying a derived table/common table expression, ensure
  to move all its keys in one operation (at least those references
  from the same query block).

- First, collect information about all key uses: hash key,
  unique index keys and actual key references.
  For the key references, also populate a mapping array that enumerates
  table references with key references in order of increasing
  key number.
  Also clear used key information for references that do not use keys.

- For each table reference with a key reference in increasing key order,
  move the used key into the lowest available position. This will ensure
  that used entries are never overwritten.

- When all table references have been processed, remove unused key
  definitions.

Change-Id: I938099284e34a81886621f6a389f34abc51e78ba

The GSS plugin for SASL appears to have leaks. This causes the LDAP
SASL client plugin to fail with ASAN and valgrind.
Fixed by:
1. making sure sasl_client_done is called by the client's deinit method.
2. add vg and asan suppressions to cover the library leaks.

Change-Id: Iceb6fbb2d9483b2fcc51c2a0f004735b288bb4f0

…g on PB2 - Windows

Disable test gr_primary_mode_group_operations_net_partition_4
on Windows until the bug is fixed.

Change-Id: I32e247363eefab08372989c24670e5238c720f2d

The failing queries are 'semi-joins', which semantically are expected
to eliminate join duplicates after the first match has been found - Contrary
to normal joins where all matching rows should be returned (in any order).

Thus the differens semi-join iterators in the mysql server does some kind
of skip-read after the first matching set of row(s) has been found for a
semi-join nest of tables. This may also skip over result rows from
other tables depending on the table(s) being skip-read. I.e tables
being in the same query tree branch as the rows being skipped.

That is fine when these tables are a part of the same semi-join as
being skip-then - Then this is intended behavior.

However, we sometimes ends up with query plans where the semi-join'ed
tables are evaluated first, and the inner joined tables ends up
depending in the semi-join'ed parts. Usually (only?) seen when the
'duplicate eliminate' iterators are use din the query plan.

Note that this effectively turns the table order in the originating
SQL query upside down. E.g. the pseudo SQL query:

   select ... from t1 where <column1> in (select <column2> from t2 where <pred>)

Might get the query plan

   duplicate eliminate (select <column2> from t2) join t1 on <pred>

Thus, we have a plan where t1 depends on a semi-joined t2, without being
part of the semi-join itself. However it will have t2 as an ancestor in the
SPJ query tree if the query is pushed -> t1 becomes a subject of
the t2 duplicate elimination, effectively a skip-read operation

Due to the finite size of the batch row buffers when returning SPJ results
to the API, we might need to return t1 result rows over multiple batches,
with the t2 result rows being reused/repeated. Thus they will appear as
dupliacted to the iterators, and be skipped over, together with the t1 rows
which should not have been skipped.

Patch identifies when we have such query plans where non-semi-joined
tables are depending on semi-joined tables, _and_ both tables are
scan operation subject to such batching mechanisms. We will then reject
pushing of depending scan-tables not being an intended part of the
semi-joins itself.

Note that such query plans seems to be a rare corner case.

Patch also changes some test cases where:
 - Added two variants of existing test cases where coverage of
   duplicate eliminating iterators were not sufficient
 - Added SEMIJOIN(LOOSESCAN) hint to enisure that intended planes where produced.
 - Added two test cases for bug itself.

That ^ smoked out a query plan which returned incorrect results after modification.
With the patch pushability was reduced, and result became correct.

Change-Id: Iae890ef702cac8a50564d5fb0e493a4715c4dafd

Windows specific: Replaced use of jemalloc for memory management
within OpenSSL (on Windows) via the call to
CRYPTO_set_mem_functions in mysqld.cc. The OpenSSL memory management
functions used on Windows now use std::malloc, std::free and
std::realloc instead.

The memory management code in my_malloc.cc is refactored using function
templates to avoid duplicating the performance schema instrumentation
and debugging code.

Change-Id: I4df2d3974f215f3a8a9a7bd0fd82dd54c96fecb7

…n PB2

Test gr_member_actions_error_on_read_on_mpm_to_spm does test how
a group mode switch handles a failure during the update of the
member actions table, causing the member to leave the group.
That is achieved by enabling a debug flag that returns a error
when we close the member actions table. Though that flag, which is
set on a common code path can affect other steps of the group mode
switch, which will continue to fail and leave the group.
The test was failing because the expected error message was not
logged into the error log, which means that the group mode switch
did error out before reaching the member actions table error.

Given that the point on which group mode switch fails is not
deterministic, we remove the error log message assert from the test.

Change-Id: I42c9e3564f79c15b80ae99a1c2edee634be0f524

…d on weekly-trunk

Test gr_acf_start_failover_channels_error_on_bootstrap was failing
due to a error log message
```
  [ERROR] [MY-013211] [Repl] Plugin
  group_replication reported: 'Error while sending message. Context: primary
  election process.'
```
though the message is expected since the test is triggering group
bootstrap errors, which does include a primary election.

To solve the above issue, now we do suppress the error log message.

Change-Id: I0eb504fec68189191dc0591effd56ba26f8b3283

gr_parallel_start_uninstall forces a race condition between
  `UNINSTALL PLUGIN group_replication;`
and
  `START GROUP_REPLICATION;`
Despite the test first asynchronously executes the `UNINSTALL`,
there is the possibility that the `START` is executed first.
`START` does enabled `super_read_only`, disabling it after the
member joins the group and it is a primary.
When the `UNINSTALL` is allowed to execute once the `START`
is complete, that may happen before the `super_read_only` is
disabled.
If that happens the `UNINSTALL` will hit the error:
  ERROR 1290 (HY000): The MySQL server is running with the
  --super-read-only option so it cannot execute this statement

Since the above error is possible, we added to one of possible error
status of `UNINSTALL PLUGIN group_replication;`.

Change-Id: I9847def076ec1236a2e273befbef52d3fcdf1376

gr_parallel_stop_dml forces the execution of
  `INSERT INTO t1 VALUES(1)`
while
  `STOP GROUP_REPLICATION`
is ongoing.
The `INSERT` must fail, throwing one of the errors:
 1) `Error on observer while running replication hook
    'before_commit'`
     when the plugin is stopping.
 2) `The MySQL server is running with the
     --super-read-only option so it cannot execute this statement`
     when the plugin already stopped and enabled `super_read_only`.

The test was not considering the second error, thence we added it.

Change-Id: I1d4e539cea1a37c11c9e133f92add3615f7aabf0

               corruption if both are set.

Issue :
  Check table shall check if both the version and instant bit are set
  for a compact/dynamic row. This is a corruption scenario and check
  table shall report it.

Fix :
  check table checks the INSTANT/VERSION bits in the records and
  report if both are set.

Change-Id: I551d6d6296d8df052bcca9450e7856a24a2c5416

When a table is first created with a reference to a non-existing
variable, the derived type is text. The second time an identical
table is created, the derived type is mediumblob.
This is due to an actual variable being created on the first table
creation, and this variable is then used on the second table creation.

The main problem with this is that the variable is created with
a binary character set, whereas the first table creation is given
the correct default character set. This problem is fixed by assigning
the correct default character set to the source item when creating
the user variable.

Even after this fix, there is still a minor difference between the
two table creations: the first table gets a column with maximum length
262140 bytes, wheras the second table gets a column with maximum length
4294967295 bytes. This is because the first creation utilizes a default
character type, whereas the second utilizes the created user variable,
and those instances use different maximum lengths. Fixing this will
require a large rewrite and is not deemed worthwhile at the time being.

Change-Id: I8cd1f946dbf87047c261bfeca9d8ba7d23a9629c

Post push fix, re-recorded spj_rqg_hyeprgraph.results

Change-Id: Ifcb0cfabef31004b5aa2af32f24736810cc2ffec

Post push fix: static inline functions std_realloc and redirecting_realloc are
only used when USE_MALLOC_WRAPPER is not defined, so make these functions
conditionally compiled to avoid build breakage when compiling in maintainer
mode (-Werror).

Change-Id: If98ef4bba95289fbdd92c9cf9808ab83e4fe1d42

               DEFUALTS During UPDATE

Issue :
 This is a followup issue of 34558510 which fixes the cases for which,
 During UPDATE, we shall not materialize INSTANT ADD columns added in
 earlier implementation.

 If a table has versions, it indicates it has INSTANT ADD/DROP
 columns in new implementation. And in new implementation it is made
 sure that the maximum possible row is within the permissible limit,
 otherwise INSTANT ADD is rejected.

Fix:
 While deciding to materialize, check if table has An INSTANT ADD columns
 with added in a row versions. If it does, then we can be assured that if
 INSTANT DEFAULT are materialized, row will be within permissible limit.

Change-Id: Ia22ab7a5aa96966741ee1b95833a5eb6705448d7

…40243300361984

  Issue:
    When user keeps adding and dropping columns instantly, n_def increases.
    When n_def is increased beyond REC_MAX_N_FIELDS, it rotates back to 0
    causing the assertion.

  Fix:
    Alter handler must know if INSTANT is possible. Hence we must check the
    value of n_def and number of columns being added before proceeding with
    ALGORITHM=INSTANT. Further we must ensure that if we cannot use INSTANT;
    we must:
    1. Fall-back to INPLACE if algorithm=DEFAULT, or not specified.
    2. Error out with ER_TOO_MANY_FIELDS(Too many columns) if algorithm=INSTANT;

  Note:
    Current patch will not allow n_def to cross 1022. This is because when we
    add even 1 more column, n_def could become 1023 (which is equal to
    REC_MAX_N_FIELDS). Furthermore, this patch will error with ER_TOO_MANY_FIELDS
    only when ADDing a new column with INSTANT. We can still drop any number of
    columns instantly

  Thanks to Marcelo Altmann ([email protected]) and Percona for the contribution

Change-Id: Iff5c7d6e45c294548d515458cddfb35c00aff43e

… ONE

Post push fix : Adding a wait to fsync.

Reviewed by: Mauritz Sundell <[email protected]>

Change-Id: I26a19b9c653fd9a46849a2a3af20b9d815fcccdc

Change-Id: I78a4c09a1790d8843b6ca14ba8856c88425966a4

Change-Id: I67c36b3afcc0c1fea40efbea8c8a0b283ccbabd1

- We have much use of sprintf, which is now flagged by clang as unsafe.
  Silence this, since we have too many uses to rewrite easily.

- This version if Xcode also flags loss of precision from 64 to 32
  bits integer, silence this also. Typically when x of type size_t is
  assigned to an int.

Change-Id: I3e5f829c7fdb8ddb08c56149bc0db1a5dc277f34

This commit fixes the above bug by making better row estimates for "GROUP BY".
We now use (non-hash) indexes and histograms to make row estimates where possible.
Otherwise, we use rules-of-thumb based on table sizes and input set sizes.

Change-Id: Ibfdd246f7251c29bb6a8b3a641ea067d65b72dbc

…ation OFF

Post-push fix. Make test ndb.ndb_reorganize_partition deterministic.

Failed on some runs:

CURRENT_TEST: ndb.ndb_reorganize_partition
@@ -176,12 +176,12 @@
 1	test/def/t1	Unique hash index	2
 2	test/def/t1	Unique hash index	1
 2	test/def/t1	Unique hash index	2
-3	test/def/t1	Unique hash index	2
 3	test/def/t1	Unique hash index	1
+3	test/def/t1	Unique hash index	2
 4	test/def/t1	Unique hash index	1
 4	test/def/t1	Unique hash index	2
-5	test/def/t1	Unique hash index	2
 5	test/def/t1	Unique hash index	1
+5	test/def/t1	Unique hash index	2
 6	test/def/t1	Unique hash index	1
 6	test/def/t1	Unique hash index	2
 7	test/def/t1	Unique hash index	1

Change-Id: I85d58c7b77104cc73fc03bf20675a4b85c5aaff2

ndb_config --diff-default should show which configured parameter differ
from the default value and could.

Since 8.0.29 it did not not work neither for system, node, or,
connection sections.

  $ ndb_config --diff-default --system
  Segmentation fault (core dumped)

  $ ndb_config --diff-default --nodes
  config of node id 1 that is different from default
CONFIG_PARAMETER,ACTUAL_VALUE,DEFAULT_VALUE .../ndb_config.cpp:463:
require((InitConfigFileParser::convertStringToUint64(def_str, def_value)))
failed
  Aborted (core dumped)

  $ ndb_config --diff-default --connnections
  <nothing>

Even before 8.0.29 there were some issues with --diff-default that
all are now fixed:

Mandatory parameters or parameters with no default value were never
printed.

Furthermore the option --type only worked for --nodes it was not
possible to select TCP or SHM for --connections.

For --nodes, one could specify ndbd or NDB for --type, but not DB as is
a valid section name in config.ini, now DB is also a valid node type.

Enumerated values in configuration were shown as numbers, these numbers
were neither documented for user nor were it possible to use them in
config.ini. Now the enumerated values are shown by name not number.

Change-Id: I5aca113174bca431269dd517623dfbc33e36b396

Adding ndb_config option --config-binary-file to read configuration from
binary (cached) configuration file.

Change-Id: Ia89acb338f9ec25e244d96960f9f4a777831d8d4

Followup patch to fix RG alter changes when RG information is
stored with a system thread.

Original patch for this bug allows system thread to bind to
user(THD) RG.  But if a system thread remembers THD RG on bind,
then altering RG attributes during will not be applied to a
system thread.

To fix this issue, introduced a version number in the RG in-memory
instance. RG version is incremented on alter operation. System
thread should check version number before deciding to re-use
stored RG. If version number mismatches, then altered THD RG should
be re-applied to a system thread.

Change-Id: I19af513e6cb49893b93cbce4143e565197f36b17

…read Pool

MySQL Resource Groups(RG) provides ability to manage and assign
resources to the threads. But at-present, RG feature is not supported
when thread pool(TP) is enabled. This WL integrates RG feature
with the TP.

With RG, TP works as below,

*) By default "USR_default" resource group is assigned to the
   user connections, "SYS_internal" resource group is assigned
   to the TP query worker threads and "SYS_default" to other
   TP background threads.

*) RG of type SYSTEM can be assigned to TP background threads
   except TP query worker threads and type USER to user connections.

   TP query worker threads are assigned to resource group
   "SYS_internal". User is not allowed to assign other resource
   group to these threads.

*) While attaching tp_worker thread to a user connection to execute
   connection's statement, user connection's RG is bind to the
   tp_worker thread.
   tp_worker thread's RG is unbind in the thread detachment
   stage.

Change-Id: I45d4a6c205b9aa0aaa6ee8f53eaef1bcdeafd76f

When resource group assigned to a thread is dropped, USR_default
resource group is assigned to it. USR_default resource group
has "0" CPU priority and no CPU affinity. So thread should be
able to use any CPU in this case. But thread is bound to CPU's of
a dropped resource group. This issue is observed on Linux and
FreeBSD only.

While applying default resource group, CPU affinity is dropped
by setting bits for all available CPUs in the system. To get
number of CPUs in Linux and FreeBSD, pthread_getaffinity_np() is
invoked incorrectly. This method actually returns CPU_SET with
bits set for CPUs to which invoking thread is bound. If thread
dropping resource group is assigned to the same resource group
then all threads assigned to resource groups keeps on using same
the set of CPUs instead of using all CPUs.

To fix this issue, changed method to get correct CPU count.

Change-Id: I5def0582ff0a3c07f886b2a18e2c95c8a0c1e2bc

post-push fix: Result of ndb.join_pushdown_default had to be updated to
reflect the new algorithm for calculating GROUP BY row estimates.

Change-Id: Ic5f061a61f25bad70df6e2f75c85c7804393db74

When connection pooling is used and a connection to a destination is
picked from the pool, it is ignored if the connection is belongs to
another io-thread.

- By default, as many io-threads are starts as there are CPU-threads.
- Each new connection is assigned another io-thread (round-robin)

E.g. a 24 core machine, with 2-threads per core, leads to 48 io-threads.

If a server-connection is taken from the pool for the current
client-connection then it must match the right:

- destination AND
- io-thread

With 3 possible destinations and 48 io-threads, the connection pool
has to contain 3 * 48 connections to have a good chance that there is 1
matching connection for this client-connection.

The higher the io-thread count, the lower the cache-hit rate.

Change
------

- ignore the io-thread when taking connections from the pool
- move pooled-connections to the current client connection's io-thread
  when taken from the pool.

Change-Id: I586e7b5b1904a3a0020d6c8784928d549a1675e4

Connecting through mysql-router with an 'caching-sha2-password' account
over unix-socket fails:

  $ mysql --socket=/tmp/router.sock
  HY000 (2013) Lost connection to MySQL server at 'reading final connect
  information', system error: 95

Background
----------

As part of the authentication handshake of caching-sha2-password, the
router asks the client for the plaintext password and expects the client
to respond with a 'get-public-key' request as the connection to the
client is not using TLS.

But the client sends the plaintext password instead, as it treats the
unix-socket as a secure transport.

Change
------

- treat unix-sockets as 'secure-transport'
- added tests

Change-Id: I72e01c604f4616ec998174a6aa6e2ae97d07088b

ASAN reports use-after-free around async_send_tls() when running
the routertest_component_mock_server.

async_send_tls() calls the completion function with net::defer()
without a execution-context, which leads to net::system_executor being
called which starts a new thread to run the completion function.

That's not intended and leads to a data-race.

Change
------

- net::defer() the execution of the completion function in the socket's
  execution context instead to properly synchronize.

Change-Id: I512c195adaa2a73210b541d18804b7d15a8d7f4e

Problem
-------

Commit ordering is not preserved for NDB transactions, even if
replica_preserve_commit_order is turned on.

Analysis / Root-cause analysis
------------------------------

Commit ordering is implemented in several places:

1. binlog.cc, in the `ordered_commit` function
2. handler.cc, in the `ha_commit_low` function

Ordering is enabled in case binlogging is turned on and binlog has
transaction information in its local caches or binlogging option is
disabled. NDB uses MYSQL_BINLOG as transaction coordinator however
it does not cache data locally. Therefore, commit ordering is never
taken into account.

Solution
--------

Ordering condition in the `ha_commit_low` function is extended to
take into account engines that do not use binlog cache manager -
newly introduced flag 'is_low_level_ordering_enabled' is checked.
The flag is set up in the 'binlog::commit' function to true in
case binlog caches are disabled or empty and binlogging is
enabled in the current call to the function. This condition is
checked by the implemented
'is_current_stmt_binlog_enabled_and_caches_empty' function
which may be safely called in case binlogging is disabled
or binlog cache manager is not initialized. The check is done
in the 'binlog::commit' because there are cases in which binlog
caches were emptied after the thread entered the ordered_commit
function in the MYSQL_BIN_LOG.
Please also note that it is important to check
opt_bin_log in is_current_stmt_binlog_log_replica_updates_disabled,
because of statements such as ALTER TABLE OPTIMIZE PARTITION,
where the last call to trans_commit_stmt in the
mysql_inplace_alter_table (Implicit_substatement_guard disabled)
is not the last call.

Signed-off-by: Karolina Szczepankiewicz <[email protected]>
Change-Id: I942f0a42aa9b3a033279a58deb8d37345b887b90

…s mysqld's tempdir

When creating the file INFO_BIN: invoke mysqld with --no-defaults
to avoid interfering with possible other installations of MySQL.

Change-Id: I25072ec674b5040f20d57da3a6ac70d7bebc9b1f
(cherry picked from commit 9b2f3d4dab7b1ca07e6a4f8b51b60e3de9270147)

Use full source path for Bison and Flex source files.
This simplifies debugging, and gcov reports.

Change-Id: I10bd9966056a63c5f32b5ae2a5bd1a850bfeb8e1
(cherry picked from commit 4ec5dc61b099da1dbbf1473c82ea3be53f8ed18a)

Some InnoDB tests stop all log activity to allow tests to create deterministic redo log changes. For this to work, all redo producing threads must be stopped. But GTID Persistor thread was not stopped. It was producing redo records and was interfering with the tests assumptions.
A new debug sync switch is added to effectively disable this thread. This new switch is used in the log_disable_page_cleaners.inc and transitively in many tests that use it, including the log_first_rec_group.test which uses the above include file instead of manual, incomplete snippets.

Change-Id: I1a0b2d727e9ef13a6fed6dcb5783f050ea81bad8

…count (weekly-trunk)

Issue: Mismatch in archive file count while loading the archiver.

Background: When an empty archive file was found during recovery, we would set the variable m_new_empty_file to true and delete that file before calling load_archiver and the file count for that group would be decremented. The m_new_empty_file variable was set from false (default value) to true if the file size was 0.

Cause: After changes in 5cad80198a2b77068b3806d13b162e4a32f7cbc4, we started creating new archive files with headers. Which meant that the file size would not be zero for a newly created empty archive file but the header size. Hence m_new_empty_file remained false and the file count was not decremented and thus the assertion failure for file count mismatch.

Fix: Modify the condition for setting m_new_empty_file from '!=0' to '> header_len'.

Change-Id: Ie989ae67c249281ab29802818fedfc6d17fd87b2

Add Result::Suspend state and resume() method to Routers Processor
and MysqlRoutingClassicConnection classes.

Change-Id: I8bb517172010e31835f8501e65a9354e95accd59

…ndows

WL #15336 added support for using MIT Kerberos to build
authentication_kerberos_client on Windows.
The cmake code assumed 64bit libraries.

Fix cmake code to look for 32bit libraries for 32bit builds.
Also fix some build warnings, and a build break (redefined typedef) for
   typedef SSIZE_T ssize_t;

Change-Id: I30d0b9c434202c3fb51ce8d1c9329abd737038e2

Problem
-------
To be consistent with other native functions, GTID functions GTID_SUBSET and GTID_SUBTRACT should return
NULL when being passed NULL or erroneous arguments. WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS and WAIT_FOR_EXECUTED_GTID_SET function return error when NULL is passed.

Analysis / Root-cause analysis
------------------------------
GTID_SUBSET and GTID_SUBTRACT function need to first check for their parameters if they are NULL and in that case make the functions return NULL. WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS and WAIT_FOR_EXECUTED_GTID_SET function return error when NULL is passed.

Solution
--------
Added quard code to check if parameters are NULL and if so return NULL or report error.

Change-Id: Ib984ba04447fbbf3f390447516399aa9fe58f7c0

The copying ALTER safety check uses the sum of per-fragment commit
count values to determine whether any writes have committed to
a table over a period of time.

Different replicas of a fragment will not necessarily have the
same commit count over time, as a fragment replica's commit
count is reset during node restart.

ReadPrimary (RP) tables always route read requests to a table's
primary fragment replicas.

ReadBackup(RB) and FullyReplicated (FR) tables optimise reads by
allowing CommittedRead operations to be routed to backup fragment
replicas.

This results in the set of commit counts read not always being
stable for RB and FR tables, which can cause false positive
failures for the copying ALTER TABLE safety check.

This is solved by performing the copying ALTER TABLE safety check
using a locking scan.  Locked reads are routed to the same set
of primary/main fragments every time, giving stable counts.

Testcase ndb_alter_table_copy_check is extended to test this
functionality.

Approved by : Magnus Blaudd <[email protected]>

In some cases with innodb-fast-shutdown=2 page cleaner will shut down
with buffer pool IO writes pending. Waiting for writes to  complete
ensures clean exit.

Similarly, in the startup process where the code waits for pending IO
reads, it is changed to also wait for pending writes.

Change-Id: I40120071371157a7ec2399dbaa65da163d15e5a6

…_start || trx->state.load(std::memory_order_relaxed) == TRX_STATE_ACTIVE

Innodb detects a deadlock with just two transactions,
choosing one as a victim to be rolled back.
This transaction is stopped in the middle of a subquery, but
would continue the query anyways, resulting in a failed
assertion down the line.

This patch forces UpdateRowsIterator::DoImmediateUpdatesAndBufferRowIds
to return return true, thus setting the local_error variable in the reading
loop in UpdateRowsIterator::Read, which stops it from continuing the query.

The query ends with an error message about the deadlock instead.

Note: the main issue is, the code ignored the return value of fill_record,
which can signal an error.

Change-Id: I3278a419d813eb1dcf5c5be1a451436afb2bc7a6

Emit a deprecation warning if an SQL statement has an unquoted
identifier that starts  with a '$'. This is in preparation for future
support for dollar quoted strings, which would conflict with such
unquoted identifiers. Quoted identifiers will continue to have a
starting dollar with no warning.

class Admin_command_collection_handler defined in plugin code used
such an identifier while generating an internal SQL. Quoted that
identifier.

Add the 'starting $' rule in require_quotes() so that internally
generated SQLs correctly quote identifiers starting with $.

Some mtr tests used identifiers with dollar, which had to be adjusted.

router/src/routing/src/sql_lexer.cc has a copy of lex_one_token()
function; so copied the changes into that file.

Change-Id: I68cc581a7f443a1a36629d5ef51a9e6fce23192d

incorrect 'Subquery returns more than 1 row'

Issue:
When the LOAD DATA INFILE command is used with a subquery having
a JOIN condition, the subquery returns more than one row where
it is expected to return only one row.

Analysis:
In case of LOAD DATA the local transformations are skipped and
hence the join condition is not set. So during the execution,
the hash iterator doesn't have the join condition and results in
returning multiple rows.

Fix:
Make sure the local transformations are applied even in the case
of LOAD command.

Change-Id: If4295ef60573eb380837ff7043497dad07645337

Post-push fix: Fix for EL6 build

Change-Id: I9700aca9601a1486efd75daf73a6ccfe1bd2b818

There is a function EstimateFieldSelectivity() that estimates the
selectivity of a given field (when there is an equi-join with some
other field.) As it was, this function looked for (non-hash) indexes
starting with that field, and used KEY::records_per_key() to estimate
the selectivity. If no suitable index was found, a default selectivity
of 10% was used.

This patch ensures that if there is no suitable index, we will look
for a histogram. If there is one, we use
Histogram::get_num_distinct_values() to estimate the selectivity.

Change-Id: I83af46e04142aa0260e701cd07e03c246696b3be

Post-push fix for broken build on windows:
%lu was used for argument of type size_t.

Change-Id: I35635db84b095c17f52db27300460b889c990f0e
(cherry picked from commit 33e2f68365200f79b3db6d7526e8178ea62710f9)

Problem: MySQL misses evaluation of constant filter condition with
masking functions, thus we see access path with impossible or always
true filter condition.

Solution: If we have filter with Constant condition, handle it.
Following possibilities are handled based on value of filter condition:
- If false, we replace by zero_rows.
- If  true, we ignore this accesspath.

Change-Id: Ic32c24c2da32430acfbc96d3a0f719ecadcb95c1

We have two targets: parser-t and Parser-t.
This is problematic on systems with case-insensitive file systems,
notably Windows.

Rename all the test targets in unittest/gunit/components/keyring_common/CMakeLists.txt
(Note these renamed targets are not built by default)

Change-Id: Ied01f5233a3960089b8c3b03c32939b6e2705b2f
(cherry picked from commit b643e71e1fe9dddd997d1b8b634940576dab4bf9)

If the router is slower than the server (like running under strace,
valgrind, nice-level, ...) it may succeed to read() and write() without
ever switching to another connection.

async_read() executes

- 'read()->EGAIN->poll()->read()' in case there is no data, where poll()
  may lead to other connections being handled
- 'read()->...' in case there is data already.

Change
------

- always 'poll()->read()' to guarantee other connections are not
  starving.

Change-Id: I805a4311fded1361ebf488512fdec6987575fc0d

After an event fired for epoll_wait(), the managed set of events is
updated according to the ONESHOT rules:

1. if multiple events were registered and one fired, the remaining ones
   must be added again with epoll_ctl(.., MOD, ...);
2. if only one event was registered and that one fired, the epoll_ctl()
   can be skipped.

Checking with strace, the one-event-case still issues a

   epoll_ctl(..., MOD, {ONESHOT|ET})

which is bogus, even though a check for that case is in place.

Change
------

- fixed the check for "there are still tracked events" before calling
  epoll_ctl() in after_events_fired()

Change-Id: Ibb66d61e5278e00c53d154aeb1f4218a8391d45d

strace shows that lots of read()/write() on a eventfd after a
fd-event-add/remove even though there is no need for that.

   write(6, "\1\0\0\0\0\0\0\0", 8) = 8
   <... epoll_wait resumed>[{events=EPOLLIN,
	data={u32=6, u64=6}}], 8192, -1) = 1
   <... read resumed>"\1\0\0\0\0\0\0\0", 8) = 8

After a (fd-event, timer) is added, modified or removed the io-service
must be interrupted to become aware of that change.

That 'notify' is supposed to interrupt blocking poll()/epoll_wait()/...
running in other io-threads, but aren't needed when running in the same
thread.

Change
------

- added a per-thread Callstack to check if the add/remove event is
  running the same thread as the io-context where it is added.
- don't interrupt (notify) the io-service if the add/remove is coming
  for the same thread.
- updated running_in_this_thread() to work according to spec.
- fixed the Timer::wait() to not call dispatch() as it would defer the
  execution.

Change-Id: I5a48a2af00cb522c4de5a702293a7bfa73c36026

Windows supports AF_UNIX streamsockets since Windows 10 (April 2018
Update).

The low-level functions should allow to use it.

Change
------

- updated the socketpair() emulation on windows to allow AF_UNIX
- added tests for autobind and abstract unix-sockets

Change-Id: I0e564f3d7790cc8eb89bc8c795b4c9cbb0cb48b9

… by server [postfix]

Fixed test-failure if the "Error 4031 disconnect due to inactivity"
wasn't received in time before the socket was closed, which results in
the "Error 2013 Lost connection".

Change-Id: I1745d24692159509929e25ebc445d29d5f3a432f

Create new THD instances for worker threads in ddl0fts.cc
to avoid some problems caused by more than one thread
using the same THD instance.
Error messages from worker threads must be propragated
to the parent thread.

Change-Id: I8fe4cfafa9c697591cc662250c7d691846b83247

Issue:
If the subquery of an order by clause has a view reference, it is
not cleaned up. This is because the subquery might be referred to
again during the execution.

In the case of execution of such a prepared statement, the server
exits as the query expression from the order by clause is present
but not marked optimized.

Fix:
Cleanup query expression of subquery if it is marked as not used
by the resolver.

Change-Id: I7ce2c6243e132339ad69c2c353f64c8d1c0485a2

…al data

Change checks for array access to handle incorrect signal data that
could cause Uint32 wrap around.

Approved by: Mauritz Sundell <[email protected]>

Change-Id: I04e9990953d14ed208c7c0568dc8a6e3ca311a33

Change-Id: Iae607c54f14c5a56316f1c8d529b2c3743b4008b

Change-Id: I5390e8fdd25ba017b28d6e371aed39fd57bf1d84

Change-Id: Ie9accae7c10ee6fde9dc546fba1df2a32d1f2705

…tables)' failed.

A query hit an assertion in debug builds and returned wrong results in
release builds when running with the hypergraph optimizer, because it
tried to sort the results on a column that was not available.

The problem happened when doing a sort with duplicate removal for a
DISTINCT clause, and there was a streaming step somewhere below the
sort access path. The sort path sorted the rows on a column that was
not mentioned in the SELECT list, and the column was therefore not
materialized in the streaming step before the sort. The sort used the
column value that happened to be available from the base table scan,
instead of getting it from the temporary table, and this caused wrong
results.

The reason why the sort access path sorts on a column not in the
SELECT list, is that the hypergraph optimizer tries all available
sort-ahead orderings for satisfying DISTINCT in the hope that one of
them might also satisfy the ORDER BY clause that is processed just
after DISTINCT. It picks a sort-ahead ordering that satisfies the
ordering required for DISTINCT (via functional dependencies) even
though it doesn't reference the column in the SELECT list. This
ordering is picked only because it is seen first; not because it is in
any way better than sorting on the original column.

This patch changes how the sort for DISTINCT is chosen. Instead of
proposing all sort-ahead orderings that would satisfy DISTINCT, it now
proposes:

1) The sort-ahead ordering that is directly derived from the DISTINCT
clause in the query.

2) If there is an ORDER BY clause, all sort-ahead orderings that
satisfy both DISTINCT and ORDER BY.

This makes it prefer orderings that reference columns in the SELECT
list or in ORDER BY, and such columns are materialized, should there
be a streaming step below the sort access path.

Change-Id: Iadd9cf7a43f95aaf6a650c1c400166a4adf8f198

Description:

Test ndb_opt.index_stat fails on Solaris sparc.

This since since primary key hash value is calculated differently on
little endian and big endian system which causes slightly different
distribution.

The partition that is scanned for index statistics in this test happen
to have 21 records on little endian system, and 27 on big endian
(sparc).

Adjust test to allow both 21 and 27 records.

Change-Id: Ic1168e83a88dbfdd7dc403cb58e8660ad9a16a3e

…er endian

Post push fix. Test ndb.ndb_print_backup_file failed with old perl.

CURRENT_TEST: ndb.ndb_print_backup_file
Bareword found where operator expected at ... line 47, near "s|$ENV{MYSQL_TEST_DIR}/||r"
syntax error at ... line 47, near "s|$ENV{MYSQL_TEST_DIR}/||r"

Replaced use of return value option /r for regex that is not supported
for old perl versions before 5.14.

Change-Id: I736120b9bb0c0b8e7bc80acc245494a0b9c134c6

…e optimization lead to reject_multiple_rows being ineffective

Pushing down the condition into the derived table made from the
subquery leads to a wrong result here. If we rely run-time checking of
the cardinality, we should not push down the condition, since it will
potentially change the cardinality to zero or one, which is
semantically wrong:

This query

  SELECT 1 AS one FROM t WHERE 1=(SELECT 1 UNION SELECT 2)

is transformed to

SELECT 1 AS one FROM
       t JOIN
       ( SELECT 1 AS col1 UNION
         SELECT 2) derived
where 1 = derived.col1

and pushing down '1 = derived.col1' into the unio removes the
contribution from SELECT 2, which is wrong.

Solution: don't push down in this case.

Change-Id: If214f5fa88ed79c186ca4b9806445524e8f37e5e

…QL.CC:2928:PREBUILT->TEMP

A DELETE statement which is executed using an index skip scan fails
an assertion on the row template type. The index skip scan
initialisation sets a keyread flag to indicate that it is sufficient
to read only the indexed fields. Based on the keyread flag, a
template is built which specifies a subset of the fields instead of
the whole row. While performing the row delete, this hits an assert
since the template type is required to specify the whole row for
deletes.

This is fixed in index skip scan initialisation by checking if the
query requires fields which are not included in the index and
setting the keyread flag accordingly. A test case is added for a
skip-scan delete query.

Change-Id: Iaa0cbd380ae12519b3dadc7924b538432bf6a77d

optimizer for point selects [1/7, benchmark, noclose]

The point select microbenchmark in hypergraph_optimizer-t does not set
up multiple equalities. Since a fair amount of time is spent on the
multiple equalities in a point select query, the microbenchmark is
changed to set up multiple equalities too, so that it more accurately
simulates the work done in a real query.

Change-Id: I844d71422c4c495fc9be537fbebd0975fd642593

optimizer for point selects [2/7, refactor fast path, noclose]

Factor out the single-table fast path in MakeJoinHypergraph() as a
separate function.

Change-Id: If16e95b6af246a93dd1165f338b6210e2102c980

optimizer for point selects [3/7, refactor conjunctions, noclose]

Factor out the code for visiting all terms of a conjunction in
ExtractConditions() as a function template called WalkConjunction().
The template will be reused in later patches in this series.

Change-Id: I34c7d51b31e67487b406d626c414d1b3bf311226

optimizer for point selects [4/7, skip normalization, noclose]

Some non-trivial amount of time is spent doing unnecessary
normalization of the WHERE clause in point select queries. This
includes common subexpression elimination and constant propagation,
and a rerun of constant folding in case the previous steps made
something foldable.

In the point select case, the WHERE clause is a simple equality, which
has already gone through constant folding, so none of these steps can
make it any simpler. This patch skips the normalization when a
single-table query has a WHERE clause that consists of a multiple
equality only, or a conjunction of multiple equalities, as these will
all be expanded to table filters that cannot be simplified any
further.

Also, the call to CanonicalizeConditions() is removed entirely when a
query accesses a single table. CanonicalizeConditions()'s only purpose
is to expand remaining multiple equalities that couldn't be entirely
pushed down as join conditions or table filters. In the case of
single-table queries, the multiple equalities can always be fully
converted to table filters, so there's nothing for it to do.

Microbenchmark results:

BM_FindBestQueryPlanPointSelect    1906 ns/iter -> 1545 ns/iter [+23%]

Change-Id: Ib3137a3603861966339c3beb6049c884aada1818

optimizer for point selects [5/7, mem_root_unordered_map, noclose]

CostingReceiver stores the plan candidates in a std::unordered_map.
This leads to dynamic memory allocation when adding candidates to it.
This patch replaces it with a mem_root_unordered_map, which allocates
memory on the execution MEM_ROOT instead. The AccessPath objects that
are stored in the map, and which consume more space than the map
itself, are already stored on the execution MEM_ROOT.

Microbenchmark results:

BM_FindBestQueryPlanPointSelect    1545 ns/iter -> 1471 ns/iter [+5.0%]

Change-Id: I38a14d8faefa9618e3b4ab9b51587e0f6aa31145

optimizer for point selects [6/7, sargable, noclose]

When analyzing equality predicates to see if they are sargable, some
time is spent checking that the arguments have compatible types so
that the predicate may be used for an index lookup. When the equality
predicate comes from a multiple equality, we already know that the
arguments have compatible types, since multiple equalities are
constructed only if the arguments have the exact same type.

We now skip the check for compatible types if the equality comes from
a multiple equality.

One bug in the multiple equality construction had to be fixed in order
to get this to work: A multiple equality was created for a string
column compared to a constant JSON expression, if the string column
had the same collation as the JSON expression (utf8mb4_bin). The
predicate would therefore be considered sargable, and a lookup was
performed on the index for the string column. This made the predicate
use string comparison semantics instead of JSON comparison semantics,
and wrong results were returned. This was fixed by adding an extra
check to check_simple_equality() to prevent creation of multiple
equalities mixing JSON and string types.

Microbenchmark results:

BM_FindBestQueryPlanPointSelect    1471 ns/iter -> 1432 ns/iter [+2.7%]

Change-Id: I21788c44668ef9c5d2ca2c7d7c6d0a7163be3c35

optimizer for point selects [7/7, final predicates]

After join enumeration, the hypergraph optimizer applies the final
predicates (those that reference no tables or are non-deterministic)
on top of all candidate plans, and creates FILTER nodes for all (both
final and non-final) predicates in the access path tree below.

None of this is necessary in a point select query, since it has no
final predicates to apply, and no FILTER nodes to create (since its
predicate is pushed to an index lookup). This step unnecessarily
copies the EQ_REF access path from MEM_ROOT to stack, and from stack
back to MEM_ROOT, and adds it to a freshly allocated Mem_root_array.

While it's not terribly expensive, the cost is noticeable for the
cheapest queries, such as point select. This patch therefore skips
this step when it sees it has a point select plan.

Microbenchmark results:

BM_FindBestQueryPlanPointSelect    1432 ns/iter -> 1366 ns/iter [+4.8%]

Change-Id: I834091382499c43b8b8de27cfa7e0b6057063c7f

…e optimization lead to reject_multiple_rows being ineffective [follow-up]

Skip explain for hypergraph due to different plan from old optimizer.

Change-Id: Ia07c20689473afc3ff486b362957b69173c37254