Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ticdc to pulsar doc #14913

Merged
merged 43 commits into from
Sep 27, 2023
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
540ec55
add ticdc-sink-to-pulsar.md && fix ticdc-changefeed-config.md
yk133 Aug 30, 2023
eb36238
Merge branch 'pingcap:master' into master
yk133 Aug 30, 2023
b44fc1d
Update ticdc/ticdc-changefeed-config.md
yk133 Sep 13, 2023
f2af82c
Update ticdc/ticdc-changefeed-config.md
yk133 Sep 13, 2023
6a416d4
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
56b7b91
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
a6cf4bf
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
f69db8e
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
d9d0b76
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
0c2d4d0
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
50d69c5
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
345e025
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
f305d08
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
d1ad84c
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
ff2c69d
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
087cb26
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
ec79f51
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
3a79f07
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
feeb191
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
f171648
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
5035b1c
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
8ed7145
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
19d532f
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
964088a
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
03eefbb
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
929fdcb
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 13, 2023
3aaf46a
Update ticdc-sink-to-pulsar.md
asddongmen Sep 13, 2023
e5ea690
Merge remote-tracking branch 'upstream/master' into pr/14913
hfxsd Sep 13, 2023
28038ca
Update ticdc-changefeed-config.md
hfxsd Sep 13, 2023
31010f3
Update ticdc-sink-to-pulsar.md
hfxsd Sep 13, 2023
900ec8b
Update TOC.md
hfxsd Sep 13, 2023
3badd0b
Update ticdc/ticdc-changefeed-config.md
asddongmen Sep 18, 2023
c8fb8d7
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 18, 2023
23eb55b
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 18, 2023
eaf7e30
Apply suggestions from code review
hfxsd Sep 18, 2023
28ce61d
Update ticdc/ticdc-sink-to-pulsar.md
hfxsd Sep 18, 2023
9a7ba30
Apply suggestions from code review
hfxsd Sep 19, 2023
a53e046
Update ticdc/ticdc-sink-to-pulsar.md
asddongmen Sep 19, 2023
629a231
Update ticdc/ticdc-sink-to-pulsar.md
hfxsd Sep 19, 2023
16e613f
Apply suggestions from code review
hfxsd Sep 21, 2023
a80258e
Apply suggestions from code review
hfxsd Sep 21, 2023
65464a0
Apply suggestions from code review
hfxsd Sep 21, 2023
724f3d9
Update ticdc/ticdc-sink-to-pulsar.md
hfxsd Sep 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -544,6 +544,7 @@
- 创建 Changefeed
- [同步数据到 MySQL 兼容的数据库](/ticdc/ticdc-sink-to-mysql.md)
- [同步数据到 Kafka](/ticdc/ticdc-sink-to-kafka.md)
- [同步数据到 Pulsar](/ticdc/ticdc-sink-to-pulsar.md)
- [同步数据到存储服务](/ticdc/ticdc-sink-to-cloud-storage.md)
- [管理 Changefeed](/ticdc/ticdc-manage-changefeed.md)
- [日志过滤器](/ticdc/ticdc-filter.md)
Expand Down
48 changes: 46 additions & 2 deletions ticdc/ticdc-changefeed-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ enable-table-across-nodes = false
# 支持 partition 及 topic(从 v6.1 开始支持)两种 event 分发器。二者的详细说明见下一节。
# matcher 的匹配语法和过滤器规则语法相同,matcher 匹配规则的详细说明见下一节。
# 注意:该参数只有当下游为消息队列时,才会生效。
# 注意:当下游 MQ 为 Pulsar 时,如果 partition 的路由规则未指定为 'ts', 'index-value', 'table', 'default' 中的任意一个,那么将会使用你设置的字符串作为每一条 Pulsar message 的 key 进行路由。例如,如果你指定的路由规则为 'code' 字符串,那么符合该 matcher 的所有 Pulsar message 都将会以 'code' 作为 key 进行路由。
# dispatchers = [
# {matcher = ['test1.*', 'test2.*'], topic = "Topic 表达式 1", partition = "ts" },
# {matcher = ['test3.*', 'test4.*'], topic = "Topic 表达式 2", partition = "index-value" },
Expand All @@ -125,8 +126,9 @@ enable-table-across-nodes = false

# protocol 用于指定编码消息时使用的格式协议
# 当下游类型是 Kafka 时,支持 canal-json、avro 和 open-protocol。
hfxsd marked this conversation as resolved.
Show resolved Hide resolved
# 当下游类型是 Pulsar 时,仅支持 canal-json 协议。
hfxsd marked this conversation as resolved.
Show resolved Hide resolved
# 当下游类型是存储服务时,目前仅支持 canal-json、csv 两种协议。
# 注意:该参数只有当下游为 Kafka 或存储服务时,才会生效。
# 注意:该参数只有当下游为 Kafka、Pulsar,或存储服务时,才会生效。
# protocol = "canal-json"

# delete-only-output-handle-key-columns 用于指定 Delete 事件的输出内容,只对 canal-json 和 open-protocol 协议有效。从 v7.2.0 开始引入。
Expand Down Expand Up @@ -216,5 +218,47 @@ sasl-oauth-scopes = ["producer.kafka", "consumer.kafka"]
# Kafka SASL OAUTHBEARER 认证机制中的 grant-type。默认值为 "client_credentials"。在使用该认证机制时,该参数可选填。
sasl-oauth-grant-type = "client_credentials"
# Kafka SASL OAUTHBEARER 认证机制中的 audience。默认值为空。在使用该认证机制时,该参数可选填。
sasl-oauth-audience = "kafka"
sasl-oauth-audience="kafka"

# 以下参数仅在下游为 Pulsar 时生效。
[sink.pulsar-config]
# 使用 token 进行 Pulsar 服务端的认证,此处为 token 的值。
authentication-token = "xxxxxxxxxxxxx"
# 指定使用 token 进行 Pulsar 服务端的认证,此处为 token 所在文件的路径。
token-from-file="/data/pulsar/token-file.txt"
# Pulsar 使用 basic 账号密码验证身份。
basic-user-name="root"
# Pulsar 使用 basic 账号密码验证身份,此处为密码。
basic-password="password"
# Pulsar TLS 加密认证证书路径。
auth-tls-certificate-path="/data/pulsar/certificate"
# Pulsar TLS 加密认证私钥路径。
auth-tls-private-key-path="/data/pulsar/certificate.key"
# Pulsar TLS 加密可信证书文件路径。
tls-trust-certs-file-path="/data/pulsar/tls-trust-certs-file"
# Pulsar oauth2 issuer-url 更多详细配置请看 Pulsar 官方介绍:https://pulsar.apache.org/docs/2.10.x/client-libraries-go/#tls-encryption-and-authentication
oauth2.oauth2-issuer-url="https://xxxx.auth0.com"
# Pulsar oauth2 audience
oauth2.oauth2-audience="https://xxxx.auth0.com/api/v2/"
# Pulsar oauth2 private-key
oauth2.oauth2-private-key="/data/pulsar/privateKey"
# Pulsar oauth2 client-id
oauth2.oauth2-client-id="0Xx...Yyxeny"
# Pulsar oauth2 oauth2-scope
oauth2.oauth2-scope="xxxx"

# TiCDC 中缓存 Pulsar Producer 的个数,默认上限为 10240 个。每个 Pulsar Producer 对应一个 topic,如果你需要同步的 topic 数量大于默认值,则需要调大该数量。
pulsar-producer-cache-size=10240
# Pulsar 数据压缩方式,默认不压缩,可选 "lz4"、"zlib"、"zstd"。
compression-type=""
# Pulsar 客户端与服务端建立 TCP 连接的超时时间,默认 5 秒。
connection-timeout=5
# Pulsar 客户端发起创建、订阅等操作的超时时间,默认为 30 秒。
operation-timeout=30
# Pulsar Producer 发送消息时的单个 batch 内的消息数量上限,默认值为 1000。
batching-max-messages=1000
# Pulsar Producer 消息攒批的时间间隔,默认 10 毫秒。
batching-max-publish-delay=10
# Pulsar Producer 发送消息的超时时间,默认 30 秒。
send-timeout=30
```
Loading
Loading