Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mongoshake向kafka同步数据速率可以优化吗？ #553

Closed
donkeyForward opened this issue Mar 5, 2021 · 4 comments
Closed

mongoshake向kafka同步数据速率可以优化吗？ #553

donkeyForward opened this issue Mar 5, 2021 · 4 comments
Labels
enhancement New feature or request

Comments

@donkeyForward
Copy link

请问用mongoshake 向kafka 同步数据，设置哪些参数可以提高写入速率啊，目前我的写入速率为2.47M/s 左右，kafka 设置一个topic，一个partition 和一个副本。
@donkeyForward
Copy link
Author

补充下，mongoshake 是增量同步模式。

@vinllen vinllen added the enhancement New feature or request label Mar 5, 2021
@vinllen
Copy link
Collaborator

vinllen commented Mar 5, 2021

这个优化有几个方面：

  1. 优化写入到多个partition，hash的方式按照配置文件中指定的incr_sync.shard_key决定。考虑向上兼容，需要多开放一个参数，否则默认还是写入1个partition。
  2. 目前序列号是单线程的操作，序列化后再发送到kafka，这个可以优化为序列化和发送分离，序列化采用多线程保序，以解决序列化的瓶颈。开放多线程序列化参数，这里默认为1。

vinllen added a commit that referenced this issue Mar 5, 2021
@vinllen
1. improve the performance of writing kafka #553. 2.support format ca…
038c33a 
…nonical extented JSON format #542
vinllen added a commit that referenced this issue Mar 9, 2021
@vinllen
1. improve the performance of writing kafka #553. 2.support format ca… (
cdf52cb 
#554)

* 1. improve the performance of writing kafka #553. 2.support format canonical extented JSON format #542

* add kafka tunnel write debug switch: incr_sync.tunnel.kafka.debug

* release v2.4.21
@vinllen
Copy link
Collaborator

vinllen commented Mar 9, 2021

2.4.21版本已经发布

@vinllen
Copy link
Collaborator

vinllen commented Mar 11, 2021

问题先关闭，后面如果还有问题可以重新回复或者打开

@vinllen vinllen closed this as completed Mar 11, 2021
vinllen added a commit that referenced this issue Mar 19, 2021
@vinllen
Improve 2.4.21 (#561)
6b353f3 
* 1. improve the performance of writing kafka #553. 2.support format canonical extented JSON format #542

* add kafka tunnel write debug switch: incr_sync.tunnel.kafka.debug

* release v2.4.21
vinllen added a commit that referenced this issue Mar 19, 2021
@vinllen
Improve 2.4.22 (#562)
876169a 
* 1. improve the performance of writing kafka #553. 2.support format canonical extented JSON format #542

* add kafka tunnel write debug switch: incr_sync.tunnel.kafka.debug

* release v2.4.21

* IMPROVE: add 'tunnel.json.format' option to let user chose json format. #542, #559, #558
vinllen added a commit that referenced this issue May 27, 2021
@vinllen
1. improve the performance of writing kafka #553. 2.support format ca… (
1b5201a 
#554)

* 1. improve the performance of writing kafka #553. 2.support format canonical extented JSON format #542

* add kafka tunnel write debug switch: incr_sync.tunnel.kafka.debug

* release v2.4.21
vinllen added a commit that referenced this issue May 27, 2021
@vinllen
Improve 2.4.22 (#562)
6d62743 
* 1. improve the performance of writing kafka #553. 2.support format canonical extented JSON format #542

* add kafka tunnel write debug switch: incr_sync.tunnel.kafka.debug

* release v2.4.21

* IMPROVE: add 'tunnel.json.format' option to let user chose json format. #542, #559, #558
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vinllen @donkeyForward