Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

plan: fix a bug when 'only_full_group_by' is set in sql_mode #6734

Merged
merged 35 commits into from
Jun 25, 2018
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
977b7cb
plan: when 'only_full_group_by' is set in sql_mode, raise error for n…
spongedu Jun 2, 2018
d13d11e
add more tests
spongedu Jun 2, 2018
00f4ef3
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 3, 2018
3d71d75
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 4, 2018
a0c7e94
code refine
spongedu Jun 4, 2018
f96eeba
Refine code format
spongedu Jun 5, 2018
62fe5fc
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 5, 2018
6515cb4
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 5, 2018
2e0fa4c
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 5, 2018
baea970
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 6, 2018
4a04f0d
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 6, 2018
1224876
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 6, 2018
4921233
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 7, 2018
e6b8d7f
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 8, 2018
e0d9b9c
code refine
spongedu Jun 8, 2018
8b4e13d
fix merge error
spongedu Jun 8, 2018
eb23718
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 8, 2018
375aeab
Merge branch 'master' into fix_only_full_group_by
zz-jason Jun 12, 2018
26af262
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 12, 2018
35c0ef3
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 12, 2018
b2122f3
Merge branch 'fix_only_full_group_by' of github.com:spongedu/tidb int…
spongedu Jun 12, 2018
7f9a3ef
Merge remote-tracking branch 'upstream/master' into fix_only_full_gro…
spongedu Jun 13, 2018
b5589a4
refine tests
spongedu Jun 13, 2018
2e081e0
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 14, 2018
a11a30c
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 16, 2018
ac564c1
Merge branch 'fix_only_full_group_by' of github.com:spongedu/tidb int…
spongedu Jun 16, 2018
35ac5bf
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 19, 2018
93274e0
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 19, 2018
c87e4b9
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 19, 2018
38c0858
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 21, 2018
2c9317d
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 22, 2018
e46c082
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 23, 2018
d961c65
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 24, 2018
3ba0eeb
Merge branch 'fix_only_full_group_by' of github.com:spongedu/tidb int…
spongedu Jun 24, 2018
883ba84
Merge branch 'master' into fix_only_full_group_by
spongedu Jun 25, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions executor/aggregate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -425,6 +425,14 @@ func (s *testSuite) TestOnlyFullGroupBy(c *C) {
c.Assert(terror.ErrorEqual(err, plan.ErrFieldNotInGroupBy), IsTrue)
_, err = tk.Exec("select -b from t group by c")
c.Assert(terror.ErrorEqual(err, plan.ErrFieldNotInGroupBy), IsTrue)
_, err = tk.Exec("select a, max(b) from t")
c.Assert(terror.ErrorEqual(err, plan.ErrMixOfGroupFuncAndFields), IsTrue)
_, err = tk.Exec("select sum(a)+b from t")
c.Assert(terror.ErrorEqual(err, plan.ErrMixOfGroupFuncAndFields), IsTrue)
_, err = tk.Exec("select count(b), c from t")
c.Assert(terror.ErrorEqual(err, plan.ErrMixOfGroupFuncAndFields), IsTrue)
_, err = tk.Exec("select distinct a, b, count(a) from t")
c.Assert(terror.ErrorEqual(err, plan.ErrMixOfGroupFuncAndFields), IsTrue)
// test compatible with sql_mode = ONLY_FULL_GROUP_BY
tk.MustQuery("select a from t group by a,b,c")
tk.MustQuery("select b from t group by b")
Expand All @@ -444,6 +452,9 @@ func (s *testSuite) TestOnlyFullGroupBy(c *C) {
tk.MustQuery("select b like '%a' from t group by b")
tk.MustQuery("select c REGEXP '1.*' from t group by c")
tk.MustQuery("select -b from t group by b")
tk.MustQuery("select max(a+b) from t")
tk.MustQuery("select avg(a)+1 from t")
_, err = tk.Exec("select count(c), 5 from t")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check this error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. @zimulala PTAL :)

// test functinal depend on primary key
tk.MustQuery("select * from t group by a")
// test functional depend on unique not null columns
Expand Down
117 changes: 60 additions & 57 deletions plan/errors.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,25 +26,26 @@ const (
codeWrongParamCount = 5
codeSchemaChanged = 6

codeWrongUsage = mysql.ErrWrongUsage
codeAmbiguous = mysql.ErrNonUniq
codeUnknownColumn = mysql.ErrBadField
codeUnknownTable = mysql.ErrUnknownTable
codeWrongArguments = mysql.ErrWrongArguments
codeBadGeneratedColumn = mysql.ErrBadGeneratedColumn
codeFieldNotInGroupBy = mysql.ErrFieldNotInGroupBy
codeBadTable = mysql.ErrBadTable
codeKeyDoesNotExist = mysql.ErrKeyDoesNotExist
codeOperandColumns = mysql.ErrOperandColumns
codeInvalidWildCard = mysql.ErrParse
codeInvalidGroupFuncUse = mysql.ErrInvalidGroupFuncUse
codeIllegalReference = mysql.ErrIllegalReference
codeNoDB = mysql.ErrNoDB
codeUnknownExplainFormat = mysql.ErrUnknownExplainFormat
codeWrongGroupField = mysql.ErrWrongGroupField
codeDupFieldName = mysql.ErrDupFieldName
codeNonUpdatableTable = mysql.ErrNonUpdatableTable
codeInternal = mysql.ErrInternal
codeWrongUsage = mysql.ErrWrongUsage
codeAmbiguous = mysql.ErrNonUniq
codeUnknownColumn = mysql.ErrBadField
codeUnknownTable = mysql.ErrUnknownTable
codeWrongArguments = mysql.ErrWrongArguments
codeBadGeneratedColumn = mysql.ErrBadGeneratedColumn
codeFieldNotInGroupBy = mysql.ErrFieldNotInGroupBy
codeBadTable = mysql.ErrBadTable
codeKeyDoesNotExist = mysql.ErrKeyDoesNotExist
codeOperandColumns = mysql.ErrOperandColumns
codeInvalidWildCard = mysql.ErrParse
codeInvalidGroupFuncUse = mysql.ErrInvalidGroupFuncUse
codeIllegalReference = mysql.ErrIllegalReference
codeNoDB = mysql.ErrNoDB
codeUnknownExplainFormat = mysql.ErrUnknownExplainFormat
codeWrongGroupField = mysql.ErrWrongGroupField
codeDupFieldName = mysql.ErrDupFieldName
codeNonUpdatableTable = mysql.ErrNonUpdatableTable
codeInternal = mysql.ErrInternal
codeMixOfGroupFuncAndFields = mysql.ErrMixOfGroupFuncAndFields
)

// error definitions.
Expand All @@ -56,48 +57,50 @@ var (
ErrWrongParamCount = terror.ClassOptimizer.New(codeWrongParamCount, "Wrong parameter count")
ErrSchemaChanged = terror.ClassOptimizer.New(codeSchemaChanged, "Schema has changed")

ErrWrongUsage = terror.ClassOptimizer.New(codeWrongUsage, mysql.MySQLErrName[mysql.ErrWrongUsage])
ErrAmbiguous = terror.ClassOptimizer.New(codeAmbiguous, mysql.MySQLErrName[mysql.ErrNonUniq])
ErrUnknownColumn = terror.ClassOptimizer.New(codeUnknownColumn, mysql.MySQLErrName[mysql.ErrBadField])
ErrUnknownTable = terror.ClassOptimizer.New(codeUnknownTable, mysql.MySQLErrName[mysql.ErrUnknownTable])
ErrWrongArguments = terror.ClassOptimizer.New(codeWrongArguments, mysql.MySQLErrName[mysql.ErrWrongArguments])
ErrBadGeneratedColumn = terror.ClassOptimizer.New(codeBadGeneratedColumn, mysql.MySQLErrName[mysql.ErrBadGeneratedColumn])
ErrFieldNotInGroupBy = terror.ClassOptimizer.New(codeFieldNotInGroupBy, mysql.MySQLErrName[mysql.ErrFieldNotInGroupBy])
ErrBadTable = terror.ClassOptimizer.New(codeBadTable, mysql.MySQLErrName[mysql.ErrBadTable])
ErrKeyDoesNotExist = terror.ClassOptimizer.New(codeKeyDoesNotExist, mysql.MySQLErrName[mysql.ErrKeyDoesNotExist])
ErrOperandColumns = terror.ClassOptimizer.New(codeOperandColumns, mysql.MySQLErrName[mysql.ErrOperandColumns])
ErrInvalidWildCard = terror.ClassOptimizer.New(codeInvalidWildCard, "Wildcard fields without any table name appears in wrong place")
ErrInvalidGroupFuncUse = terror.ClassOptimizer.New(codeInvalidGroupFuncUse, mysql.MySQLErrName[mysql.ErrInvalidGroupFuncUse])
ErrIllegalReference = terror.ClassOptimizer.New(codeIllegalReference, mysql.MySQLErrName[mysql.ErrIllegalReference])
ErrNoDB = terror.ClassOptimizer.New(codeNoDB, mysql.MySQLErrName[mysql.ErrNoDB])
ErrUnknownExplainFormat = terror.ClassOptimizer.New(codeUnknownExplainFormat, mysql.MySQLErrName[mysql.ErrUnknownExplainFormat])
ErrWrongGroupField = terror.ClassOptimizer.New(codeWrongGroupField, mysql.MySQLErrName[mysql.ErrWrongGroupField])
ErrDupFieldName = terror.ClassOptimizer.New(codeDupFieldName, mysql.MySQLErrName[mysql.ErrDupFieldName])
ErrNonUpdatableTable = terror.ClassOptimizer.New(codeNonUpdatableTable, mysql.MySQLErrName[mysql.ErrNonUpdatableTable])
ErrInternal = terror.ClassOptimizer.New(codeInternal, mysql.MySQLErrName[mysql.ErrInternal])
ErrWrongUsage = terror.ClassOptimizer.New(codeWrongUsage, mysql.MySQLErrName[mysql.ErrWrongUsage])
ErrAmbiguous = terror.ClassOptimizer.New(codeAmbiguous, mysql.MySQLErrName[mysql.ErrNonUniq])
ErrUnknownColumn = terror.ClassOptimizer.New(codeUnknownColumn, mysql.MySQLErrName[mysql.ErrBadField])
ErrUnknownTable = terror.ClassOptimizer.New(codeUnknownTable, mysql.MySQLErrName[mysql.ErrUnknownTable])
ErrWrongArguments = terror.ClassOptimizer.New(codeWrongArguments, mysql.MySQLErrName[mysql.ErrWrongArguments])
ErrBadGeneratedColumn = terror.ClassOptimizer.New(codeBadGeneratedColumn, mysql.MySQLErrName[mysql.ErrBadGeneratedColumn])
ErrFieldNotInGroupBy = terror.ClassOptimizer.New(codeFieldNotInGroupBy, mysql.MySQLErrName[mysql.ErrFieldNotInGroupBy])
ErrBadTable = terror.ClassOptimizer.New(codeBadTable, mysql.MySQLErrName[mysql.ErrBadTable])
ErrKeyDoesNotExist = terror.ClassOptimizer.New(codeKeyDoesNotExist, mysql.MySQLErrName[mysql.ErrKeyDoesNotExist])
ErrOperandColumns = terror.ClassOptimizer.New(codeOperandColumns, mysql.MySQLErrName[mysql.ErrOperandColumns])
ErrInvalidWildCard = terror.ClassOptimizer.New(codeInvalidWildCard, "Wildcard fields without any table name appears in wrong place")
ErrInvalidGroupFuncUse = terror.ClassOptimizer.New(codeInvalidGroupFuncUse, mysql.MySQLErrName[mysql.ErrInvalidGroupFuncUse])
ErrIllegalReference = terror.ClassOptimizer.New(codeIllegalReference, mysql.MySQLErrName[mysql.ErrIllegalReference])
ErrNoDB = terror.ClassOptimizer.New(codeNoDB, mysql.MySQLErrName[mysql.ErrNoDB])
ErrUnknownExplainFormat = terror.ClassOptimizer.New(codeUnknownExplainFormat, mysql.MySQLErrName[mysql.ErrUnknownExplainFormat])
ErrWrongGroupField = terror.ClassOptimizer.New(codeWrongGroupField, mysql.MySQLErrName[mysql.ErrWrongGroupField])
ErrDupFieldName = terror.ClassOptimizer.New(codeDupFieldName, mysql.MySQLErrName[mysql.ErrDupFieldName])
ErrNonUpdatableTable = terror.ClassOptimizer.New(codeNonUpdatableTable, mysql.MySQLErrName[mysql.ErrNonUpdatableTable])
ErrInternal = terror.ClassOptimizer.New(codeInternal, mysql.MySQLErrName[mysql.ErrInternal])
ErrMixOfGroupFuncAndFields = terror.ClassOptimizer.New(codeMixOfGroupFuncAndFields, "In aggregated query without GROUP BY, expression #%d of SELECT list contains nonaggregated column '%s'; this is incompatible with sql_mode=only_full_group_by")
)

func init() {
mysqlErrCodeMap := map[terror.ErrCode]uint16{
codeWrongUsage: mysql.ErrWrongUsage,
codeAmbiguous: mysql.ErrNonUniq,
codeUnknownColumn: mysql.ErrBadField,
codeUnknownTable: mysql.ErrBadTable,
codeWrongArguments: mysql.ErrWrongArguments,
codeBadGeneratedColumn: mysql.ErrBadGeneratedColumn,
codeFieldNotInGroupBy: mysql.ErrFieldNotInGroupBy,
codeBadTable: mysql.ErrBadTable,
codeKeyDoesNotExist: mysql.ErrKeyDoesNotExist,
codeOperandColumns: mysql.ErrOperandColumns,
codeInvalidWildCard: mysql.ErrParse,
codeInvalidGroupFuncUse: mysql.ErrInvalidGroupFuncUse,
codeIllegalReference: mysql.ErrIllegalReference,
codeNoDB: mysql.ErrNoDB,
codeUnknownExplainFormat: mysql.ErrUnknownExplainFormat,
codeWrongGroupField: mysql.ErrWrongGroupField,
codeDupFieldName: mysql.ErrDupFieldName,
codeNonUpdatableTable: mysql.ErrUnknownTable,
codeInternal: mysql.ErrInternal,
codeWrongUsage: mysql.ErrWrongUsage,
codeAmbiguous: mysql.ErrNonUniq,
codeUnknownColumn: mysql.ErrBadField,
codeUnknownTable: mysql.ErrBadTable,
codeWrongArguments: mysql.ErrWrongArguments,
codeBadGeneratedColumn: mysql.ErrBadGeneratedColumn,
codeFieldNotInGroupBy: mysql.ErrFieldNotInGroupBy,
codeBadTable: mysql.ErrBadTable,
codeKeyDoesNotExist: mysql.ErrKeyDoesNotExist,
codeOperandColumns: mysql.ErrOperandColumns,
codeInvalidWildCard: mysql.ErrParse,
codeInvalidGroupFuncUse: mysql.ErrInvalidGroupFuncUse,
codeIllegalReference: mysql.ErrIllegalReference,
codeNoDB: mysql.ErrNoDB,
codeUnknownExplainFormat: mysql.ErrUnknownExplainFormat,
codeWrongGroupField: mysql.ErrWrongGroupField,
codeDupFieldName: mysql.ErrDupFieldName,
codeNonUpdatableTable: mysql.ErrUnknownTable,
codeInternal: mysql.ErrInternal,
codeMixOfGroupFuncAndFields: mysql.ErrMixOfGroupFuncAndFields,
}
terror.ErrClassToMySQLCodes[terror.ClassOptimizer] = mysqlErrCodeMap
}
60 changes: 57 additions & 3 deletions plan/logical_plan_builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -1286,6 +1286,14 @@ func checkExprInGroupBy(p LogicalPlan, expr ast.ExprNode, offset int, loc string
}

func (b *planBuilder) checkOnlyFullGroupBy(p LogicalPlan, fields []*ast.SelectField, orderBy *ast.OrderByClause, gby *ast.GroupByClause, from ast.ResultSetNode, where ast.ExprNode) {
if gby != nil {
b.checkOnlyFullGroupByWithGroupClause(p, fields, orderBy, gby, from, where)
return
}
b.checkOnlyFullGroupByWithOutGroupClause(p, fields)
}

func (b *planBuilder) checkOnlyFullGroupByWithGroupClause(p LogicalPlan, fields []*ast.SelectField, orderBy *ast.OrderByClause, gby *ast.GroupByClause, from ast.ResultSetNode, where ast.ExprNode) {
gbyCols := make(map[*expression.Column]struct{}, len(fields))
gbyExprs := make([]ast.ExprNode, 0, len(fields))
schema := p.Schema()
Expand Down Expand Up @@ -1342,6 +1350,51 @@ func (b *planBuilder) checkOnlyFullGroupBy(p LogicalPlan, fields []*ast.SelectFi
}
}

func (b *planBuilder) checkOnlyFullGroupByWithOutGroupClause(p LogicalPlan, fields []*ast.SelectField) {
resolver := colResolverForOnlyFullGroupBy{}
//for idx, field := range fields {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the useless comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

for idx, field := range fields {
resolver.exprIdx = idx
field.Accept(&resolver)
if err := resolver.Check(); err != nil {
b.err = err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b.err = errors.Trace(err)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

return
}
}
}

type colResolverForOnlyFullGroupBy struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add comments for it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

firstNonAggCol *ast.ColumnName
exprIdx int
firstNonAggColIdex int
hasAggFunc bool
}

func (c *colResolverForOnlyFullGroupBy) Enter(node ast.Node) (ast.Node, bool) {
switch node.(type) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use x := node.(type), so that we don't need to do the type assertion again at line 1381.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

case *ast.AggregateFuncExpr:
c.hasAggFunc = true
return node, true
case *ast.ColumnNameExpr:
if c.firstNonAggCol == nil {
c.firstNonAggCol, c.firstNonAggColIdex = node.(*ast.ColumnNameExpr).Name, c.exprIdx
}
return node, true
}
return node, false
}

func (c *colResolverForOnlyFullGroupBy) Leave(node ast.Node) (ast.Node, bool) {
return node, true
}

func (c *colResolverForOnlyFullGroupBy) Check() error {
if c.hasAggFunc && c.firstNonAggCol != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about select min(c1) from t where c2 in (select c2 from t where c3 > 10); Could this checker handle the situation of subquery?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this situation is not handled by this checker. ColumnRefs in predicates is not affected under this circumstance because predicates are processed prior to aggregations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least, this PR is an improvement. @shenli

return ErrMixOfGroupFuncAndFields.GenByArgs(c.firstNonAggColIdex+1, c.firstNonAggCol.Name.O)
}
return nil
}

type colResolver struct {
p LogicalPlan
cols map[*expression.Column]struct{}
Expand Down Expand Up @@ -1512,10 +1565,11 @@ func (b *planBuilder) buildSelect(sel *ast.SelectStmt) LogicalPlan {
if b.err != nil {
return nil
}
if b.ctx.GetSessionVars().SQLMode.HasOnlyFullGroupBy() {
b.checkOnlyFullGroupBy(p, sel.Fields.Fields, sel.OrderBy, sel.GroupBy, sel.From.TableRefs, sel.Where)
}
}
if b.ctx.GetSessionVars().SQLMode.HasOnlyFullGroupBy() && sel.From != nil {
b.checkOnlyFullGroupBy(p, sel.Fields.Fields, sel.OrderBy, sel.GroupBy, sel.From.TableRefs, sel.Where)
}

// We must resolve having and order by clause before build projection,
// because when the query is "select a+1 as b from t having sum(b) < 0", we must replace sum(b) to sum(a+1),
// which only can be done before building projection and extracting Agg functions.
Expand Down