Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat:add mysql table meta func && sql addEscape, delEscape func issue… #294

Merged
merged 11 commits into from
Oct 13, 2022
3 changes: 3 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ require (
vimagination.zapto.org/byteio v0.0.0-20200222190125-d27cba0f0b10
)

require github.com/google/go-cmp v0.5.8 // indirect
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个咋放括号外面了

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个一般是工具生成的,也可以手动移进去,我移动一下。


require (
cloud.google.com/go v0.93.3 // indirect
contrib.go.opencensus.io/exporter/prometheus v0.4.1 // indirect
Expand Down Expand Up @@ -145,6 +147,7 @@ require (
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gotest.tools v2.2.0+incompatible
moul.io/http2curl v1.0.0 // indirect
vimagination.zapto.org/memio v0.0.0-20200222190306-588ebc67b97d // indirect
)
5 changes: 5 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ github.com/go-playground/locales v0.14.0 h1:u50s323jtVGugKlcYeyzC0etD1HifMjqmJqb
github.com/go-playground/locales v0.14.0/go.mod h1:sawfccIbzZTqEDETgFXqTho0QybSa7l++s0DH+LDiLs=
github.com/go-playground/universal-translator v0.18.0 h1:82dyy6p4OuJq4/CByFNOn/jYrnRPArHwAcmLoJZxyho=
github.com/go-playground/universal-translator v0.18.0/go.mod h1:UvRDBj+xPUEGrFYl+lu/H90nyDXpg0fqeB/AQUGNTVA=
github.com/go-playground/validator/v10 v10.10.0/go.mod h1:74x4gJWsvQexRdW8Pn3dXSGrTK4nAUsbPlLADvpJkos=
github.com/go-playground/validator/v10 v10.11.0/go.mod h1:i+3WkQ1FvaUjjxh1kSvIA4dMGDBiPU55YFDl0WbKdWU=
github.com/go-playground/validator/v10 v10.11.1 h1:prmOlTVv+YjZjmRmNSF3VmspqJIxJWXmqUsHwfTRRkQ=
github.com/go-playground/validator/v10 v10.11.1/go.mod h1:i+3WkQ1FvaUjjxh1kSvIA4dMGDBiPU55YFDl0WbKdWU=
Expand Down Expand Up @@ -743,6 +744,7 @@ github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0
github.com/satori/go.uuid v1.2.1-0.20181028125025-b2ce2384e17b h1:gQZ0qzfKHQIybLANtM3mBXNUtOfsCFXeTsnBqCsx1KM=
github.com/satori/go.uuid v1.2.1-0.20181028125025-b2ce2384e17b/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/shirou/gopsutil v3.20.11+incompatible h1:LJr4ZQK4mPpIV5gOa4jCOKOGb4ty4DZO54I4FGqIpto=
github.com/shirou/gopsutil v3.20.11+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=
github.com/shirou/gopsutil/v3 v3.21.6/go.mod h1:JfVbDpIBLVzT8oKbvMg9P3wEIMDDpVn+LwHTKj0ST88=
github.com/shirou/gopsutil/v3 v3.22.2 h1:wCrArWFkHYIdDxx/FSfF5RB4dpJYW6t7rcp3+zL8uks=
Expand Down Expand Up @@ -1130,10 +1132,12 @@ golang.org/x/sys v0.0.0-20220111092808-5a964db01320/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220915200043-7b5979e65e41 h1:ohgcoMbSofXygzo6AD2I1kz3BFmW1QArPYTtwEM3UXc=
golang.org/x/sys v0.0.0-20220915200043-7b5979e65e41/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down Expand Up @@ -1414,6 +1418,7 @@ gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gotest.tools v2.2.0+incompatible h1:VsBPFP1AI068pPrMxtb/S8Zkgf9xEmTLJjfM+P5UIEo=
gotest.tools v2.2.0+incompatible/go.mod h1:DsYFclhRJ6vuDpmuTbkuFWG+y2sxOXAzmJt81HFBacw=
honnef.co/go/tools v0.0.0-20180728063816-88497007e858/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
Expand Down
29 changes: 15 additions & 14 deletions pkg/datasource/sql/datasource/base/meta_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package base

import (
"context"
"database/sql"
"errors"
"sync"
"time"
Expand All @@ -29,7 +30,7 @@ import (
type (
// trigger
trigger interface {
LoadOne(table string) (types.TableMeta, error)
LoadOne(ctx context.Context, dbName string, table string, conn *sql.Conn) (*types.TableMeta, error)

LoadAll() ([]types.TableMeta, error)
}
Expand All @@ -46,13 +47,14 @@ type BaseTableMetaCache struct {
expireDuration time.Duration
capity int32
size int32
dbName string
cache map[string]*entry
cancel context.CancelFunc
trigger trigger
}

// NewBaseCache
func NewBaseCache(capity int32, expireDuration time.Duration, trigger trigger) (*BaseTableMetaCache, error) {
func NewBaseCache(capity int32, dbName string, expireDuration time.Duration, trigger trigger) *BaseTableMetaCache {
ctx, cancel := context.WithCancel(context.Background())

c := &BaseTableMetaCache{
Expand All @@ -61,15 +63,14 @@ func NewBaseCache(capity int32, expireDuration time.Duration, trigger trigger) (
size: 0,
expireDuration: expireDuration,
cache: map[string]*entry{},
dbName: dbName,
cancel: cancel,
trigger: trigger,
}

if err := c.Init(ctx); err != nil {
return nil, err
}
c.Init(ctx)

return c, nil
return c
}

// init
Expand Down Expand Up @@ -135,32 +136,32 @@ func (c *BaseTableMetaCache) scanExpire(ctx context.Context) {
}

// GetTableMeta
func (c *BaseTableMetaCache) GetTableMeta(table string) (types.TableMeta, error) {
func (c *BaseTableMetaCache) GetTableMeta(ctx context.Context, tableName string, conn *sql.Conn) (types.TableMeta, error) {
c.lock.Lock()
defer c.lock.Unlock()

v, ok := c.cache[table]
v, ok := c.cache[tableName]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

删掉空行

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if !ok {
meta, err := c.trigger.LoadOne(table)
meta, err := c.trigger.LoadOne(ctx, c.dbName, tableName, conn)
if err != nil {
return types.TableMeta{}, err
}

if !meta.IsEmpty() {
c.cache[table] = &entry{
value: meta,
if meta != nil && !meta.IsEmpty() {
c.cache[tableName] = &entry{
value: *meta,
lastAccess: time.Now(),
}

return meta, nil
return *meta, nil
}

return types.TableMeta{}, errors.New("not found table metadata")
}

v.lastAccess = time.Now()
c.cache[table] = v
c.cache[tableName] = v

return v.value, nil
}
Expand Down
2 changes: 1 addition & 1 deletion pkg/datasource/sql/datasource/datasource_manager.go
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ type TableMetaCache interface {
// Init
Init(ctx context.Context, conn *sql.DB) error
// GetTableMeta
GetTableMeta(table string) (types.TableMeta, error)
GetTableMeta(ctx context.Context, table string, conn *sql.Conn) (*types.TableMeta, error)
// Destroy
Destroy() error
}
Expand Down
40 changes: 36 additions & 4 deletions pkg/datasource/sql/datasource/mysql/meta_cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,55 @@ package mysql
import (
"context"
"database/sql"
"sync"
"time"

"github.com/pkg/errors"

"github.com/seata/seata-go/pkg/datasource/sql/datasource/base"
"github.com/seata/seata-go/pkg/datasource/sql/types"
)

var (
capacity int32 = 1024
EexpireTime = 15 * time.Minute
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是已经格式化后的样子,看着确实有点奇怪。

tableMetaInstance *tableMetaCache
tableMetaOnce sync.Once
DBName = "seata"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里为啥写死呢?我理解应该是动态传的参数哈

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是指 DBName 吗,原本设想是 config 功能实现后放到配置里面,所以暂时写死了。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

加上 todo,然后说明下这个值以后是从 config 中读取

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todo 已加上

)

type tableMetaCache struct {
cache *base.BaseTableMetaCache
tableMetaCache *base.BaseTableMetaCache
}

func GetTableMetaInstance() *tableMetaCache {
// Todo constant.DBName get from config
tableMetaOnce.Do(func() {
tableMetaInstance = &tableMetaCache{
tableMetaCache: base.NewBaseCache(capacity, DBName, EexpireTime, NewMysqlTrigger()),
}
})

return tableMetaInstance
}

// Init
func (c *tableMetaCache) Init(ctx context.Context, conn *sql.DB) error {
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里不需要进行初始化吗?我看你在启动的时候调用过这个方法

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个可以删掉。他这个Init没有用

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

也可以,目前确实没什么用。

}

// GetTableMeta
func (c *tableMetaCache) GetTableMeta(table string) (types.TableMeta, error) {
return types.TableMeta{}, nil
// GetTableMeta get table info from cache or information schema
func (c *tableMetaCache) GetTableMeta(ctx context.Context, tableName string, conn *sql.Conn) (*types.TableMeta, error) {
if tableName == "" {
return nil, errors.New("TableMeta cannot be fetched without tableName")
}

tableMeta, err := c.tableMetaCache.GetTableMeta(ctx, tableName, conn)
if err != nil {
return nil, err
}

return &tableMeta, nil
}

// Destroy
Expand Down
55 changes: 55 additions & 0 deletions pkg/datasource/sql/datasource/mysql/meta_cache_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package mysql

import (
"context"
"database/sql"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

format

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里也是格式化之后的样子。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"testing" 不应该放上面吗

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

格式化工具的区别吗,我格式化完之后是根据字母排序的,所以跑到最后了。

_ "github.com/go-sql-driver/mysql"
"gotest.tools/assert"
"testing"
)

// TestGetTableMeta
func TestGetTableMeta(t *testing.T) {
// local test can annotation t.SkipNow()
t.SkipNow()

testTableMeta := func() {
metaInstance := GetTableMetaInstance()

db, err := sql.Open("mysql", "root:123456@tcp(127.0.0.1:3306)/seata?multiStatements=true")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以的,我看看。

if err != nil {
t.Fatal(err)
}

defer db.Close()

ctx := context.Background()
conn, _ := db.Conn(ctx)

tableMeta, err := metaInstance.GetTableMeta(ctx, "undo_log", conn)
assert.NilError(t, err)

t.Logf("%+v", tableMeta)
}

t.Run("testTableMeta", func(t *testing.T) {
testTableMeta()
})
}
Loading