*: Add Vector data type

[EricZequan]

What problem does this PR solve?

Issue Number: ref #54245

Problem Summary: Support Vector data type

What changed and how does it work?

• New variable: @@GLOBAL.TIDB_ENABLE_VECTOR_TYPE

1. When SEM is enabled (normal / strict mode), users without RESTRICTED_ priv cannot modify this variable.

• Defining table using new data type: CREATE TABLE foo(val VECTOR)
• Supported aggregation functions:

1. Count
2. Count Distinct
3. Min
4. Max

• Supported scalar functions:

1. Cast (Cast into / from non-string will throw error)
2. Compare
3. IsTrue/IsFalse/IsNull
4. vec_dims, vec_from_text, vec_as_text

Other scalar functions (like vector distances, vector arithmetics) will be added in future PRs.

The VectorFloat32's Datum / Chunk / Memory layouts are identical, as follows:

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

mysql> SET GLOBAL TIDB_ENABLE_VECTOR_TYPE = ON;
Query OK, 0 rows affected (0.01 sec)


mysql> CREATE TABLE foo(val VECTOR);
Query OK, 0 rows affected (0.05 sec)


mysql> insert into foo values ('[1.1,2.2,3.3]');
Query OK, 1 row affected (0.01 sec)


mysql> select count(val) from foo;
+------------+
| count(val) |
+------------+
|          1 |
+------------+
1 row in set (0.00 sec)


mysql> insert into foo values ('[1.1,2.2,1.1]');
Query OK, 1 row affected (0.01 sec)


mysql> select Min(val) from foo;
+---------------+
| Min(val)      |
+---------------+
| [1.1,2.2,1.1] |
+---------------+
1 row in set (0.00 sec)


mysql> select Max(val) from foo;
+---------------+
| Max(val)      |
+---------------+
| [1.1,2.2,3.3] |
+---------------+
1 row in set (0.01 sec)


mysql> SELECT COUNT(DISTINCT val) FROM foo;
+---------------------+
| COUNT(DISTINCT val) |
+---------------------+
|                   2 |
+---------------------+
1 row in set (0.00 sec)


mysql> SELECT CAST(val AS CHAR) FROM foo;
+-------------------+
| CAST(val AS CHAR) |
+-------------------+
| [1.1,2.2,3.3]     |
| [1.1,2.2,1.1]     |
+-------------------+
2 rows in set (0.01 sec)


mysql> SELECT * FROM foo WHERE val > '[1.0, 2.0, 3.0]';
+---------------+
| val           |
+---------------+
| [1.1,2.2,3.3] |
| [1.1,2.2,1.1] |
+---------------+
2 rows in set (0.00 sec)


mysql> SELECT * FROM foo WHERE val < '[1.0, 2.0, 3.0]';
Empty set (0.00 sec)


mysql> SELECT * FROM foo WHERE val IS NULL;
Empty set (0.00 sec)


mysql> SELECT vec_dims(val) AS dims FROM foo;
+------+
| dims |
+------+
|    3 |
|    3 |
+------+
2 rows in set (0.01 sec)


mysql> INSERT INTO foo (val) VALUES (vec_from_text('[4.4, 5.5, 6.6]'));
Query OK, 1 row affected (0.01 sec)


mysql> SELECT vec_as_text(val) AS text_val FROM foo;
+---------------+
| text_val      |
+---------------+
| [1.1,2.2,3.3] |
| [1.1,2.2,1.1] |
| [4.4,5.5,6.6] |
+---------------+
3 rows in set (0.00 sec)

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[sre-bot]

All committers have signed the CLA.

[tiprow]

Hi @EricZequan. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hawkingrei]

/ok-to-test

[codecov]

Codecov Report

Attention: Patch coverage is 51.01843% with 505 lines in your changes missing coverage. Please review.

Project coverage is 75.4360%. Comparing base (1acb8f7) to head (f132098).

❗ Current head f132098 differs from pull request most recent head 6542f06

Please upload reports for the commit 6542f06 to get more accurate results.

Additional details and impacted files

@@                              Coverage Diff                               @@
##           feature/vector-search/vector-data-type     #54635        +/-   ##
==============================================================================
+ Coverage                                 72.8511%   75.4360%   +2.5848%     
==============================================================================
  Files                                        1558       1561         +3     
  Lines                                      438406     439440      +1034     
==============================================================================
+ Hits                                       319384     331496     +12112     
+ Misses                                      99322      87406     -11916     
- Partials                                    19700      20538       +838     

Flag	Coverage Δ
integration	50.8653% <2.5267%> (?)
unit	71.7228% <50.7274%> (-0.1356%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9656% <ø> (ø)
parser	∅ <ø> (∅)
br	63.1556% <ø> (+17.2813%)	⬆️

[tangenta]

Is there a plan to support these in the future?

[breezewish]

It depends. In most cases user is writing a constant vector string so that it is const-folded in the planning stage and does not need cast pushdown in the execution stage.

[tangenta]

Rest LGTM

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-24 15:30:05.172802951 +0000 UTC m=+1059027.163744421: ☑️ agreed by breezewish.
• 2024-07-31 07:56:18.717476666 +0000 UTC m=+342494.997524738: ☑️ agreed by tangenta.

[EricZequan]

/retest

mis-operation

[wuhuizuo]

/retest

[XuHuaiyu]

Do we need to support vecEvalXXX for vector type?

[breezewish]

Yes. Currently it is not implemented though. @EricZequan will find some time to provide a vectorized version for them.

[EricZequan]

@XuHuaiyu Vectorized version has been merged in cse. I will later merge the changes in this branch.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: breezewish, hawkingrei, tangenta, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [XuHuaiyu,breezewish,hawkingrei,tangenta]
• pkg/ddl/OWNERS [tangenta]
• pkg/expression/OWNERS [XuHuaiyu]
• pkg/parser/OWNERS [tangenta]
• pkg/planner/OWNERS [hawkingrei]
• pkg/table/OWNERS [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment