Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: introduce a new execution framework for aggregate functions #6852

Merged
merged 24 commits into from
Jun 29, 2018
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
182658b
executor: refactor aggregate functions
zz-jason Jun 17, 2018
01ec369
change type name
zz-jason Jun 20, 2018
faa3f7d
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 21, 2018
1ebf4d5
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 21, 2018
038135a
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 21, 2018
7f1fd3a
handle distinct in partial1 and complete mode
zz-jason Jun 21, 2018
e5acefb
handle 0 count
zz-jason Jun 21, 2018
e25d273
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 22, 2018
dca132a
add implementation of SUM
zz-jason Jun 23, 2018
81c812b
Merge branch 'dev/refactor-agg' of https://github.com/zz-jason/tidb i…
zz-jason Jun 23, 2018
cacad7c
add missing file
zz-jason Jun 23, 2018
d3a1ced
only introduce the framework in this PR
zz-jason Jun 24, 2018
cf34a16
remove useless code
zz-jason Jun 24, 2018
b1335c3
remove debug log
zz-jason Jun 24, 2018
ce821f9
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 24, 2018
dbbef24
fix ci
zz-jason Jun 24, 2018
57f57b8
Merge branch 'dev/refactor-agg' of https://github.com/zz-jason/tidb i…
zz-jason Jun 24, 2018
5f17822
address comment
zz-jason Jun 25, 2018
aa71782
address comment
zz-jason Jun 25, 2018
f3b005f
address comment
zz-jason Jun 26, 2018
c7f7144
Merge branch 'master' into dev/refactor-agg
lysu Jun 28, 2018
66801be
Merge branch 'master' into dev/refactor-agg
zz-jason Jun 28, 2018
0d859db
addres comment
zz-jason Jun 29, 2018
7f8134c
Merge branch 'master' into dev/refactor-agg
coocood Jun 29, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions executor/aggfuncs/aggfuncs.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
// Copyright 2018 PingCAP, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// See the License for the specific language governing permissions and
// limitations under the License.

package aggfuncs

import (
"github.com/pingcap/tidb/expression"
"github.com/pingcap/tidb/sessionctx"
"github.com/pingcap/tidb/util/chunk"
)

// All the AggFunc implementations are listed here for navigation.
var (
// All the AggFunc implementations for "COUNT".
// All the AggFunc implementations for "SUM".
// All the AggFunc implementations for "AVG".
_ AggFunc = (*avgDedup4Decimal)(nil)
_ AggFunc = (*avgOriginal4Decimal)(nil)
_ AggFunc = (*avgPartial4Decimal)(nil)

_ AggFunc = (*avgDedup4Float64)(nil)
_ AggFunc = (*avgOriginal4Float64)(nil)
_ AggFunc = (*avgPartial4Float64)(nil)

_ AggFunc = (*avgDedup4Float32)(nil)
_ AggFunc = (*avgOriginal4Float32)(nil)
_ AggFunc = (*avgPartial4Float32)(nil)

// All the AggFunc implementations for "FIRSTROW".
// All the AggFunc implementations for "MAX".
// All the AggFunc implementations for "MIN".
// All the AggFunc implementations for "GROUP_CONCAT".
// All the AggFunc implementations for "BIT_OR".
// All the AggFunc implementations for "BIT_XOR".
// All the AggFunc implementations for "BIT_AND".
)

// AggFunc is the interface to evaluate aggregate functions.
type AggFunc interface {
AllocPartialResult() []byte
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that we need another struct which contains partialResultBytes to handle the hashagg evaluation and aggfunc with distinct?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for now, we can just add a map field in a specific aggregate function implementation, during the execution of UpdatePartialResult we use that map to deduplicate the input, when ResetPartialResult, we reset that map.

ResetPartialResult(partialBytes []byte)
UpdatePartialResult(sctx sessionctx.Context, rowsInGroup []chunk.Row, partialBytes []byte) error

AppendPartialResult2Chunk(sctx sessionctx.Context, partialBytes []byte, chk *chunk.Chunk) error
AppendFinalResult2Chunk(sctx sessionctx.Context, partialBytes []byte, chk *chunk.Chunk) error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/ AppendFinalResult2Chunk/ GetFinalResult

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the original name, which indicates the result is appended to the output chunk

}

type baseAggFunc struct {
input []expression.Expression
output []int
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comments for these two args.

}
141 changes: 141 additions & 0 deletions executor/aggfuncs/builder.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
// Copyright 2018 PingCAP, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// See the License for the specific language governing permissions and
// limitations under the License.

package aggfuncs

import (
"github.com/pingcap/tidb/ast"
"github.com/pingcap/tidb/expression/aggregation"
"github.com/pingcap/tidb/mysql"
"github.com/pingcap/tidb/types"
)

// Build is used to build AggFunc according to the aggFuncDesc
func Build(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
switch aggFuncDesc.Name {
case ast.AggFuncCount:
return buildCount(aggFuncDesc, output)
case ast.AggFuncSum:
return buildSum(aggFuncDesc, output)
case ast.AggFuncAvg:
return buildAvg(aggFuncDesc, output)
case ast.AggFuncFirstRow:
return buildFirstRow(aggFuncDesc, output)
case ast.AggFuncMax:
return buildMax(aggFuncDesc, output)
case ast.AggFuncMin:
return buildMin(aggFuncDesc, output)
case ast.AggFuncGroupConcat:
return buildGroupConcat(aggFuncDesc, output)
case ast.AggFuncBitOr:
return buildBitOr(aggFuncDesc, output)
case ast.AggFuncBitXor:
return buildBitXor(aggFuncDesc, output)
case ast.AggFuncBitAnd:
return buildBitAnd(aggFuncDesc, output)
}
return nil
}

func buildCount(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildSum(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildAvg(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
base := baseAggFunc{
input: aggFuncDesc.Args,
output: output,
}

switch aggFuncDesc.Mode {
// Build avg functions which consume the original data and remove the
// duplicated input of the same group.
case aggregation.DedupMode:
return nil
// TODO: implement UpdatePartialResult for the following structs.
// switch aggFuncDesc.Args[0].GetType().Tp {
// case mysql.TypeDecimal:
// return &avgDedup4Decimal{baseAvgDecimal{base}}
// case mysql.TypeFloat:
// return &avgDedup4Float32{baseAvgFloat32{base}}
// case mysql.TypeDouble:
// return &avgDedup4Float64{baseAvgFloat64{base}}
// }

// Build avg functions which consume the original data and update their
// partial results.
case aggregation.CompleteMode, aggregation.Partial1Mode:
switch aggFuncDesc.Args[0].GetType().Tp {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should consider all the input types,
use EvalType here may be better?

case mysql.TypeNewDecimal:
if aggFuncDesc.HasDistinct {
return &avgOriginal4Decimal{baseAvgDecimal{base}, make(map[types.MyDecimal]bool)}
}
return &avgOriginal4Decimal{baseAvgDecimal{base}, nil}
case mysql.TypeFloat:
if aggFuncDesc.HasDistinct {
return &avgOriginal4Float32{baseAvgFloat32{base}, make(map[float32]bool)}
}
return &avgOriginal4Float32{baseAvgFloat32{base}, nil}
case mysql.TypeDouble:
if aggFuncDesc.HasDistinct {
return &avgOriginal4Float64{baseAvgFloat64{base}, make(map[float64]bool)}
}
return &avgOriginal4Float64{baseAvgFloat64{base}, nil}
}

// Build avg functions which consume the partial result of other agg
// functions and update their partial results.
case aggregation.Partial2Mode, aggregation.FinalMode:
switch aggFuncDesc.Args[1].GetType().Tp {
case mysql.TypeNewDecimal:
return &avgPartial4Decimal{baseAvgDecimal{base}}
case mysql.TypeFloat:
return &avgPartial4Float32{baseAvgFloat32{base}}
case mysql.TypeDouble:
return &avgPartial4Float64{baseAvgFloat64{base}}
}
}
return nil
}

func buildFirstRow(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildMax(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildMin(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildGroupConcat(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildBitOr(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildBitXor(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}

func buildBitAnd(aggFuncDesc *aggregation.AggFuncDesc, output []int) AggFunc {
return nil
}
Loading