executor, session: refine insert unsigned bigint autoIncreID #8181

XuHuaiyu · 2018-11-05T10:04:36Z

What problem does this PR solve?

Before this commit, the behavior of the following sqls in TiDB is wrong.

unsigned auto_increment integer column:

-- We cannot insert an unsigned integer value which is larger than MaxInt64 into a signed auto_increment column.
tidb> create table unsigned_autoid(`auto_inc_id` bigint unsigned not null unique key auto_increment);
tidb> insert into unsigned_autoid values(9223372036854775808);
ERROR 1690 (22003): constant 9223372036854775808 overflows bigint

signed auto_increment integer column:

-- The value inserted is wrong if the auto_increment col is MaxInt64 already.
tidb> create table signed_auto_inc(auto_inc_col bigint signed auto_increment, key(au
to_inc_col));
Query OK, 0 rows affected (0.01 sec)
tidb> insert into signed_auto_inc value(cast((1<<63)-1 as signed));
Query OK, 1 row affected (0.00 sec)
tidb> insert into signed_auto_inc values();
Query OK, 1 row affected (0.00 sec)
tidb> insert into signed_auto_inc values();
Query OK, 1 row affected (0.00 sec)
tidb> select * from signed_auto_inc;
+----------------------+
| auto_inc_col         |
+----------------------+
|  9223372036854775807 |
| -9223372036854775807 |
| -9223372036854775805 |
+----------------------+
3 rows in set (0.00 sec)

What is changed and how it works?

Add check for the unsigned flag for allocator.
Add check for MaxUint64 and MaxInt64.

Note: After this commit, we can promise the correctness for the behavior when inserting a specific value or an auto-generated value into the auto-increment column.
BUT it is not 100% consistent with MySQL now, we :

unsigned auto_increment id

create table unsigned_autoid(`auto_inc_id` bigint unsigned not null unique key auto_increment);

-- MySQL:
mysql> insert into unsigned_autoid values(18446744073709551614);
Query OK, 1 row affected (0.00 sec)
mysql> insert into unsigned_autoid values();
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine
mysql> insert into unsigned_autoid values(18446744073709551615); # MySQL performs well.
Query OK, 1 row affected (0.00 sec)

--TiDB:
tidb> insert into unsigned_autoid values(18446744073709551614);
Query OK, 1 row affected (0.00 sec)
tidb> insert into unsigned_autoid values();
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine
tidb> insert into unsigned_autoid values(18446744073709551615);
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine

signed auto_increment id

create table signed_autoid(`auto_inc_id` bigint signed not null unique key auto_increment);

-- MySQL:
mysql> insert into signed_autoid values(9223372036854775806);
Query OK, 1 row affected (0.00 sec)
mysql> insert into signed_autoid values(); # MySQL performs well.
Query OK, 1 row affected (0.00 sec)
mysql> insert into signed_autoid values(); # MySQL returns DupEntry error
ERROR 1062 (23000): Duplicate entry '9223372036854775807' for key 'auto_inc_id'

--TiDB:
tidb> insert into signed_autoid values(9223372036854775806);
Query OK, 1 row affected (0.00 sec)
tidb> insert into signed_autoid values();
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine
tidb> insert into signed_autoid values();
ERROR 1467 (HY000): Failed to read auto-increment value from storage engine

Check List

Tests

Integration test

Code changes

Has exported function/method change

Side effects

none

Related changes

Need to cherry-pick to the release branch

release-2.0 release-2.1

This change is

XuHuaiyu · 2018-11-05T10:09:22Z

/run-all-tests

zz-jason

~~LGTM~~

zz-jason · 2018-11-05T10:14:05Z

/run-all-tests

XuHuaiyu · 2018-11-05T10:14:39Z

@zimulala @jackysp PTAL

jackysp · 2018-11-05T10:20:27Z

session/session_test.go

+	tk.MustExec("drop table if exists autoid")
+	tk.MustExec("create table autoid(`auto_inc_id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`))")
+	tk.MustExec("insert into autoid values(9223372036854775808);")
+	tk.MustQuery("select * from autoid").Check(testkit.Rows("9223372036854775808"))


How about insert auto values after this? e.g. insert into autid values ();

mysql> create table autoid(`auto_inc_id` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`)); Query OK, 0 rows affected (0.02 sec) mysql> insert into autoid values(9223372036854775808); Query OK, 1 row affected (0.00 sec) mysql> select * from autoid; +---------------------+ | auto_inc_id | +---------------------+ | 9223372036854775808 | +---------------------+ 1 row in set (0.00 sec) mysql> insert into autoid values(); Query OK, 1 row affected (0.00 sec) mysql> select * from autoid; +---------------------+ | auto_inc_id | +---------------------+ | 9223372036854775808 | | 2 | +---------------------+ 2 rows in set (0.00 sec)

zimulala

The ID generated on the TiKV side is int64. We see the function of "func (m *Meta) GenAutoTableID(dbID, tableID, step int64) (int64, error)".

eurekaka · 2018-11-06T02:49:03Z

executor/insert_common.go

+			recordID, err = d.ToInt64(sc)
+			if e.filterErr(err) != nil {
+				return types.Datum{}, errors.Trace(err)
+			}


ConvertTo checks mysql.HasUnsignedFlag, how about:

datum, err := d.ConvertTo(sc, &c.FieldType) if e.filterErr(err) != nil { return types.Datum{}, err } recordID = datum.GetInt64()

XuHuaiyu · 2018-11-08T15:54:51Z

PTAL @jackysp @zimulala @winkyao

tiancaiamao · 2018-11-12T05:54:39Z

pingcap/parser#34 is merged.
Please update the go.mod file to use the latest parser

crazycs520

I got a problem:

create table autoid(`auto_inc_id` bigint unsigned not null unique key auto_increment);
insert into autoid values(9223372036854775808);
insert into autoid values();
insert into autoid values(18446744073709551615);
insert into autoid values();
select * from autoid
+----------------------+
| auto_inc_id          |
+----------------------+
| 9223372036854775808  |
| 9223372036854805810  |
| 18446744073709551615 |
| 30001                |
+----------------------+
4 rows in set
Time: 0.006s

This may cause by func (t *TxStructure) HInc(key []byte, field []byte, step int64) (int64, error)

…d_autoid

crazycs520 · 2018-11-12T17:40:23Z

session/session_test.go

+	tk.MustExec("drop table if exists autoid")
+	tk.MustExec("create table autoid(`auto_inc_id` bigint(20) NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`))")
+	tk.MustExec("insert into autoid values(9223372036854775806);")
+	tk.MustExec("insert into autoid values();")


the test has a problem:

mysql root@127.0.0.1:test> create table autoid(`auto_inc_id` bigint(20) NOT NULL AUTO_INCREMENT,UNIQUE KEY `auto_inc_id` (`auto_inc_id`)); Query OK, 0 rows affected mysql root@127.0.0.1:test> insert into autoid values(9223372036854775806); Query OK, 1 row affected mysql root@127.0.0.1:test> select auto_inc_id,_tidb_rowid from autoid +---------------------+---------------------+ | auto_inc_id | _tidb_rowid | +---------------------+---------------------+ | 9223372036854775806 | 9223372036854775807 | +---------------------+---------------------+ 1 row in set mysql root@127.0.0.1:test> insert into autoid values(); # auto_id = rowid= 9223372036854775807, cause below admin check error. Query OK, 1 row affected mysql root@127.0.0.1:test> select auto_inc_id,_tidb_rowid from autoid +---------------------+---------------------+ | auto_inc_id | _tidb_rowid | +---------------------+---------------------+ | 9223372036854775807 | 9223372036854775807 | +---------------------+---------------------+ mysql root@127.0.0.1:test> admin check table autoid; (8003, u'autoid err:[admin:1]index:&admin.RecordData{Handle:9223372036854775807, Values:[]types.Datum{types.Datum{k:0x1, collation:0x0, decimal:0x0, length:0x0, i:9223372036854775806, b:[]uint8(nil), x:interface {}(nil)}}} != record:&admin.RecordData{Handle:9223372036854775807, Values:[]types.Datum{types.Datum{k:0x1, collation:0x0, decimal:0x0, length:0x0, i:9223372036854775807, b:[]uint8(nil), x:interface {}(nil)}}}') mysql root@127.0.0.1:test> select count(*) from autoid; +----------+ | count(*) | +----------+ | 1 | +----------+ 1 row in set mysql root@127.0.0.1:test> select count(*) from autoid use index(auto_inc_id); +----------+ | count(*) | +----------+ | 2 | +----------+ 1 row in set

Please address this comment @XuHuaiyu.

crazycs520 · 2018-11-12T17:53:52Z

meta/autoid/autoid.go

-	alloc.base++
-	log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", alloc.base, tableID, alloc, alloc.dbID)
+	if alloc.isUnsigned {
+		if uint64(alloc.base) == math.MaxUint64 {


!alloc.isUnsigned need this kind of check( if alloc.base == math.MaxInt64 ) too.

Could we put this check to line204? Then we needn't do check each time Alloc is called.

crazycs520 · 2018-11-12T17:57:30Z

I have a question: Why auto_increment and _tidb_rowid both use the same allocator?
@zimulala

tiancaiamao · 2018-11-13T03:22:38Z

meta/autoid/autoid.go

@@ -137,13 +137,12 @@ func (alloc *allocator) Rebase(tableID, requiredBase int64, allocIDs bool) error
 		if allocIDs {
 			if alloc.isUnsigned {


You want to prevent overflow?
Should we

if uNewBase > (math.Uint64 - step) { panic }

Yes,
but why we panic here? @tiancaiamao

Well, maybe error or warning...
I mean, if the ID overflow, it should not be changed to math.Uint64

This check is used to prevent the alloc.end overflow.
alloc.base is checked here.

It may be reasonable to change alloc.end to math.Uint64 if it will overflow?
@tiancaiamao

If alloc.end overflow, what will happen?

@tiancaiamao
The result of TiDB master:

tidb> create table t(a bigint signed not null auto_increment, unique key(a)); Query OK, 0 rows affected (0.02 sec) tidb> insert into t value(9223372036854775806); Query OK, 1 row affected (0.00 sec) tidb> insert into t value(); Query OK, 1 row affected (0.00 sec) tidb> insert into t value(); Query OK, 1 row affected (0.00 sec) tidb> insert into t value(); Query OK, 1 row affected (0.00 sec) tidb> insert into t value(); Query OK, 1 row affected (0.00 sec) tidb> select * from t; +----------------------+ | a | +----------------------+ | -9223372036854775808 | | -9223372036854775806 | | -9223372036854775804 | | -9223372036854775802 | | 9223372036854775806 | +----------------------+ 5 rows in set (0.00 sec)

Overwrite the old row id is dangerous for data consistency.
Is it acceptable to change the behavior to throw error?

As a tradeoff, #8181 (review) may be better?

I think throw error is simple, easier, and more robust.

meta/autoid/autoid.go

XuHuaiyu · 2018-12-13T10:38:06Z

should be merged with pingcap/parser#79

jackysp

LGTM

zimulala · 2018-12-14T03:20:21Z

meta/autoid/autoid.go

 		}
 		alloc.base, alloc.end = newBase, newEnd
 	}

 	alloc.base++
-	log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", alloc.base, tableID, alloc, alloc.dbID)
+	log.Debugf("[kv] Alloc id %d, table ID:%d, from %p, database ID:%d", uint64(alloc.base), tableID, alloc, alloc.dbID)


Could we remove "uint64"?

zimulala · 2018-12-14T03:23:26Z

meta/autoid/autoid_test.go

@@ -57,7 +57,7 @@ func (*testSuite) TestT(c *C) {
 	})
 	c.Assert(err, IsNil)

-	alloc := autoid.NewAllocator(store, 1)
+	alloc := autoid.NewAllocator(store, 1, false)


Please add some tests for NewAllocator when isUnsigned is true. Otherwise, the test coverage of this package will decrease.

crazycs520

LGTM

winkyao

LGTM

winkyao · 2018-12-15T15:11:23Z

@XuHuaiyu CI failed.

…d_autoid

…to unsigned_autoid

XuHuaiyu · 2018-12-18T11:07:34Z

/run-all-tests tidb-test=pr/651

zimulala

LGTM

XuHuaiyu · 2018-12-18T12:52:27Z

/run-all-tests tidb-test=pr/651

XuHuaiyu · 2018-12-18T13:15:45Z

/run-all-tests tidb-test=pr/651

XuHuaiyu · 2018-12-18T13:34:48Z

/run-common-tests
/run-integration-common-test

XuHuaiyu · 2018-12-18T13:44:58Z

/run-common-tests tidb-test=pr/651
/run-integration-common-test tidb-test=pr/651

zz-jason

LGTM

…#8181)

…9339)

executor, session: refine insert unsigned bigint autoIncreID

a7df24d

XuHuaiyu added type/bugfix This PR fixes a bug. component/DDL-need-LGT3 labels Nov 5, 2018

XuHuaiyu added sig/execution SIG execution and removed component/DDL-need-LGT3 labels Nov 5, 2018

zz-jason reviewed Nov 5, 2018

View reviewed changes

zz-jason added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 5, 2018

jackysp reviewed Nov 5, 2018

View reviewed changes

XuHuaiyu added the status/DNM label Nov 5, 2018

zimulala reviewed Nov 5, 2018

View reviewed changes

eurekaka reviewed Nov 6, 2018

View reviewed changes

address comment

9d21807

XuHuaiyu mentioned this pull request Nov 8, 2018

model: add GetAutoIncrementColInfo for TableInfo pingcap/parser#34

Merged

XuHuaiyu removed the status/DNM label Nov 8, 2018

update parser

2db694e

crazycs520 reviewed Nov 12, 2018

View reviewed changes

XuHuaiyu added 2 commits November 12, 2018 14:27

Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…

1eb9325

…d_autoid

address comment

53fd0a6

crazycs520 reviewed Nov 12, 2018

View reviewed changes

tiancaiamao reviewed Nov 13, 2018

View reviewed changes

meta/autoid/autoid.go Outdated Show resolved Hide resolved

XuHuaiyu added 2 commits November 13, 2018 13:26

address comment

758fd31

make golint happy

a68a08d

jackysp reviewed Dec 13, 2018

View reviewed changes

zimulala reviewed Dec 14, 2018

View reviewed changes

crazycs520 reviewed Dec 14, 2018

View reviewed changes

winkyao reviewed Dec 14, 2018

View reviewed changes

jackysp and others added 5 commits December 17, 2018 10:47

Merge branch 'master' into unsigned_autoid

d0616ac

add unit tests for Allocator

6792306

Merge branch 'master' of https://github.com/pingcap/tidb into unsigne…

46d20a4

…d_autoid

Merge branch 'unsigned_autoid' of https://github.com/XuHuaiyu/tidb in…

548ea41

…to unsigned_autoid

update go.mod

61bfec6

XuHuaiyu added 2 commits December 18, 2018 19:11

address comment

6c4419c

make tidy

3f1fcb4

zimulala previously approved these changes Dec 18, 2018

View reviewed changes

go mod tidy

71ea318

XuHuaiyu dismissed zimulala’s stale review via 71ea318 December 18, 2018 12:50

Merge branch 'master' into unsigned_autoid

18c62fe

zz-jason approved these changes Dec 18, 2018

View reviewed changes

Merge branch 'master' into unsigned_autoid

d688346

XuHuaiyu merged commit 7056bb0 into pingcap:master Dec 18, 2018

XuHuaiyu deleted the unsigned_autoid branch December 18, 2018 15:01

XuHuaiyu added a commit to XuHuaiyu/tidb that referenced this pull request Feb 18, 2019

executor, session: refine insert unsigned bigint autoIncreID (pingcap…

720715e

…#8181)

XuHuaiyu mentioned this pull request Feb 18, 2019

executor, session: refine insert unsigned bigint autoIncreID (#8181) #9339

Merged

zz-jason pushed a commit that referenced this pull request Feb 18, 2019

executor, session: refine insert unsigned bigint autoIncreID (#8181) (#…

bf68dd4

…9339)

wjhuang2016 mentioned this pull request May 10, 2019

insert value 2^64-1(18446744073709551615) in field failed which has bigint type and AUTO_INCREMENT property #10400

Closed

		@@ -137,13 +137,12 @@ func (alloc *allocator) Rebase(tableID, requiredBase int64, allocIDs bool) error
		if allocIDs {
		if alloc.isUnsigned {

executor, session: refine insert unsigned bigint autoIncreID #8181

executor, session: refine insert unsigned bigint autoIncreID #8181

Conversation

XuHuaiyu commented Nov 5, 2018 • edited by c4pt0r Loading

What problem does this PR solve?

What is changed and how it works?

Check List

XuHuaiyu commented Nov 5, 2018

zz-jason left a comment • edited Loading

Choose a reason for hiding this comment

zz-jason commented Nov 5, 2018

XuHuaiyu commented Nov 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zimulala left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu commented Nov 8, 2018

tiancaiamao commented Nov 12, 2018

crazycs520 left a comment • edited Loading

Choose a reason for hiding this comment

crazycs520 Nov 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazycs520 Nov 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazycs520 commented Nov 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XuHuaiyu commented Dec 13, 2018

jackysp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crazycs520 left a comment

Choose a reason for hiding this comment

winkyao left a comment

Choose a reason for hiding this comment

winkyao commented Dec 15, 2018

XuHuaiyu commented Dec 18, 2018

zimulala left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Dec 18, 2018

XuHuaiyu commented Dec 18, 2018

XuHuaiyu commented Dec 18, 2018

XuHuaiyu commented Dec 18, 2018

zz-jason left a comment

Choose a reason for hiding this comment

XuHuaiyu commented Nov 5, 2018 •

edited by c4pt0r

Loading

zz-jason left a comment •

edited

Loading

zimulala left a comment •

edited

Loading

crazycs520 left a comment •

edited

Loading

crazycs520 Nov 12, 2018 •

edited

Loading

crazycs520 Nov 12, 2018 •

edited

Loading