Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

全文检索数据同步改造 #7561

Draft
wants to merge 1 commit into
base: v3.12.x
Choose a base branch
from

Conversation

wcy00000000000000
Copy link
Collaborator

优化的功能:

  • 全文检索数据同步由使用Monstache调整为引入sync_server服务进行同步

从Monstache迁移到sync-server的迁移指引

背景

CMDB老版本的全文检索数据是通过Monstache插件和Elasticsearch进行同步的,存在以下问题:

  • CMDB使用的是Monstache提供的全量同步和MongoDB的oplog增量同步能力,性能较差
  • 需要额外引入Monstache组件,增加系统的复杂度和编译部署难度
  • CMDB使用的Monstache是单点部署的,稳定性较差

迁移指引

CMDB新版本中新增了sync server进行全文检索数据同步。从老版本迁移到新版本时流程如下:

  1. 停止Monstache服务
  2. 删除Monstache组件和插件
  3. 在common.yaml配置文件中新增全文检索同步相关的配置,其中indexShardNum和indexReplicaNum对应原Monstache插件的 extra.toml配置的elasticsearch-shard-num和elasticsearch-replica-num:
# syncServer相关配置
syncServer:
  # 全文检索同步相关配置
  fullTextSearch:
    # 是否开启全文检索同步, 默认为false
    enableSync: true
    # ES索引拥有的主分片数量,详情请参见:https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html
    indexShardNum: 1
    # ES索引每个主分片拥有的副本数量,详情请参见:https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html
    indexReplicaNum: 1
  1. 部署cmdb_syncserver服务。服务启动时会进行Elasticsearch索引的初始化和全量数据同步,耗时较长,如果数据同步中途服务退出,则需要通过数据同步小工具按需进行数据同步

# @param common.syncServer.fullTextSearch full-text search synchronization common config parameters
fullTextSearch:
# @param common.syncServer.fullTextSearch.enableSync if full-text search synchronization is enabled
enableSync: true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

直接用es.fullTextSearch这个配置就好了吧

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

目前的设计是当做可选项,可以通过syncserver同步或者其他方式同步或者不同步

# 全文检索同步相关配置
fullTextSearch:
# 是否开启全文检索同步, 默认为false
enableSync: false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

直接用es那个配置的开关就好了吧

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同上

docs/support-file/helm/values.yaml Show resolved Hide resolved
src/scene_server/sync_server/service/sync.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants