Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: dag support mysql store #21

Open
wants to merge 71 commits into
base: master
Choose a base branch
from
Open

Conversation

Wenne
Copy link

@Wenne Wenne commented May 10, 2023

支持 mysql 存储

Wenne added 2 commits May 9, 2023 11:52
* add gorm define to entity
* add store for mysql
* add keeper for mysql
@Wenne
Copy link
Author

Wenne commented May 10, 2023

@ShiningRush 大佬有空可以 review 一下

* add store for mysql
* add keeper for mysql
@ShiningRush
Copy link
Owner

好的,我周末看下,测试用例看起来有部分失败了,可以先修复下

@Wenne
Copy link
Author

Wenne commented May 11, 2023

好的,我周末看下,测试用例看起来有部分失败了,可以先修复下

已修复~

@ShiningRush
Copy link
Owner

@Wenne hi,还麻烦补充下集成测试用例,参考之前的 store 和 keeper 的用例集即可,CI配置可以参考 #24

go.mod Outdated Show resolved Hide resolved

const LeaderKey = "leader"

// Keeper mongo implement
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

pkg/entity/dag.go Outdated Show resolved Hide resolved
@@ -188,9 +188,12 @@ func (c *LeaderCollector) Collect(ch chan<- prometheus.Metric) {
)
}

// todo: just add metrics, register used by existed http server
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个todo是?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已剔除

keeper/mysql/entity.go Outdated Show resolved Hide resolved

type TaskInstanceParams map[string]interface{}

type TaskInstanceDependOn []string
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

以上三个结构的定义均是通用的,不用加上TaskInstance的前缀吧

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这边之前是为了适配 gorm 序列化必须要 struct 并且实现 json 解析的一些方法才行,但是新办法不需要了,加上 serializer:json 即可。。我这边去掉

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

type Election struct {
ID string `gorm:"primaryKey;type:VARCHAR(256);not null"`
WorkerKey string `gorm:"type:VARCHAR(256);not null"`
UpdatedAt time.Time `gorm:"autoUpdateTime;type:timestamp;index;"`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

与 L10 相同

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改


mutex sync.Mutex
mutex sync.Mutex `json:"-" bson:"-"`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

私有化字段不会被序列化,这个tag是无用的

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

@@ -105,9 +107,9 @@ var (
// ExecuteContext's Context
type ShareData struct {
Dict map[string]string
Save func(data *ShareData) error
Save func(data *ShareData) error `json:"-" bson:"-"`
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

由于ShareData实现了 JsonMarshalBsonMarshal 方法,因此不需要tag来控制序列化

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

@@ -269,9 +271,9 @@ func (dagIns *DagInstance) CanModifyStatus() bool {
}

// Render variables
func (vars DagInstanceVars) Render(p map[string]interface{}) (map[string]interface{}, error) {
func (vars *DagInstanceVars) Render(p map[string]interface{}) (map[string]interface{}, error) {
Copy link
Owner

@ShiningRush ShiningRush May 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里调整成引用类型的目的是什么
DagInstanceVars 是一个 map, map本身即为引用类型,因此定义一个指向引用的指针没有太大价值,反而导致 L276 必须要显式去做寻址后才能进行 for range

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修改

@Wenne
Copy link
Author

Wenne commented May 22, 2023

工作时候事情有点多。。我周末再提交 pr 一个个修复下 comments

@Wenne Wenne closed this May 22, 2023
@Wenne Wenne reopened this May 22, 2023
@ShiningRush
Copy link
Owner

ShiningRush commented May 23, 2023

工作时候事情有点多。。我周末再提交 pr 一个个修复下 comments

不急,慢慢弄,我最近事情比较多,也都是只有周末才能仔细瞅瞅代码 😄

ch <- prometheus.MustNewConstMetric(
failedTaskCountDesc,
prometheus.CounterValue,
float64(c.FailedTaskCount),
mod.GetKeeper().WorkerKey(),
dagInsIdStr,
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

正常来说不建议把ID的类的字段作为Tag 加入到时序曲线中,因为会导致时序图数量激增,对底层的存储也有较大压力。
可以评估下是否必要,非必要的话可以移除。
如果必要的话,建议针对FailedTask和SuccessTask都独立加入DagInstanceID的Tag即可,而不是用,来连接他们,这会导致原时序不断变化,不是好的实践

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是也是当下我们公司的告警系统建设是围绕 metics 直接建设的,加上这个的意义是为了方便告警信息可以有 dag ins id。社区版本我先废除掉吧

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants