From d98942aa107ce80caae99088ae5ce3f15a8218e2 Mon Sep 17 00:00:00 2001 From: Jia Fan Date: Wed, 11 Sep 2024 16:30:56 +0800 Subject: [PATCH 1/4] [Improve][Document] Refactor job env config document --- docs/en/concept/JobEnvConfig.md | 22 ++++++++++++++-------- docs/zh/concept/JobEnvConfig.md | 22 ++++++++++++++-------- 2 files changed, 28 insertions(+), 16 deletions(-) diff --git a/docs/en/concept/JobEnvConfig.md b/docs/en/concept/JobEnvConfig.md index 77c924b68f2..e21864dcebe 100644 --- a/docs/en/concept/JobEnvConfig.md +++ b/docs/en/concept/JobEnvConfig.md @@ -21,14 +21,26 @@ You can configure whether the task is in batch or stream mode through `job.mode` ### checkpoint.interval -Gets the interval in which checkpoints are periodically scheduled. +Gets the interval (milliseconds) in which checkpoints are periodically scheduled. -In `STREAMING` mode, checkpoints is required, if you do not set it, it will be obtained from the application configuration file `seatunnel.yaml`. In `BATCH` mode, you can disable checkpoints by not setting this parameter. +In `STREAMING` mode, checkpoints is required, if you do not set it, it will be obtained from the application configuration file `seatunnel.yaml`. In `BATCH` mode, you can disable checkpoints by not setting this parameter. In Zeta `STREAMING` mode, the default value is 30000 milliseconds. + +### checkpoint.timeout + +The timeout (in milliseconds) for a checkpoint. If the checkpoint is not completed before the timeout, the job will fail. In Zeta, the default value is 30000 milliseconds. ### parallelism This parameter configures the parallelism of source and sink. +### shade.identifier + +Specify the method of encryption, if you didn't have the requirement for encrypting or decrypting config files, this option can be ignored. + +For more details, you can refer to the documentation [Config Encryption Decryption](../connector-v2/Config-Encryption-Decryption.md) + +## Zeta Engine Parameter + ### job.retry.times Used to control the default retry times when a job fails. The default value is 3, and it only works in the Zeta engine. @@ -43,12 +55,6 @@ This parameter is used to specify the location of the savemode when the job is e The default value is `CLUSTER`, which means that the savemode is executed on the cluster. If you want to execute the savemode on the client, you can set it to `CLIENT`. Please use `CLUSTER` mode as much as possible, because when there are no problems with `CLUSTER` mode, we will remove `CLIENT` mode. -### shade.identifier - -Specify the method of encryption, if you didn't have the requirement for encrypting or decrypting config files, this option can be ignored. - -For more details, you can refer to the documentation [Config Encryption Decryption](../connector-v2/Config-Encryption-Decryption.md) - ## Flink Engine Parameter Here are some SeaTunnel parameter names corresponding to the names in Flink, not all of them. Please refer to the official [Flink Documentation](https://flink.apache.org/). diff --git a/docs/zh/concept/JobEnvConfig.md b/docs/zh/concept/JobEnvConfig.md index c20797604f3..4dddb6e94a0 100644 --- a/docs/zh/concept/JobEnvConfig.md +++ b/docs/zh/concept/JobEnvConfig.md @@ -21,14 +21,26 @@ ### checkpoint.interval -获取定时调度检查点的时间间隔。 +获取定时调度检查点的时间间隔(毫秒)。 -在`STREAMING`模式下，检查点是必须的，如果不设置，将从应用程序配置文件`seatunnel.yaml`中获取。 在`BATCH`模式下，您可以通过不设置此参数来禁用检查点。 +在`STREAMING`模式下，检查点是必须的，如果不设置，将从应用程序配置文件`seatunnel.yaml`中获取。 在`BATCH`模式下，您可以通过不设置此参数来禁用检查点。在Zeta `STREAMING`模式下，默认值为30000毫秒。 + +### checkpoint.timeout + +检查点的超时时间(毫秒)。如果检查点在超时之前没有完成，作业将失败。在Zeta中，默认值为30000毫秒。 ### parallelism 该参数配置source和sink的并行度。 +### shade.identifier + +指定加密方式，如果您没有加密或解密配置文件的需求，此选项可以忽略。 + +更多详细信息，您可以参考文档 [Config Encryption Decryption](../../en/connector-v2/Config-Encryption-Decryption.md) + +## Zeta 引擎参数 + ### job.retry.times 用于控制作业失败时的默认重试次数。默认值为3，并且仅适用于Zeta引擎。 @@ -44,12 +56,6 @@ 当值为`CLIENT`时，SaveMode操作在作业提交的过程中执行，使用shell脚本提交作业时，该过程在提交作业的shell进程中执行。使用rest api提交作业时，该过程在http请求的处理线程中执行。 请尽量使用`CLUSTER`模式，因为当`CLUSTER`模式没有问题时，我们将删除`CLIENT`模式。 -### shade.identifier - -指定加密方式，如果您没有加密或解密配置文件的需求，此选项可以忽略。 - -更多详细信息，您可以参考文档 [Config Encryption Decryption](../../en/connector-v2/Config-Encryption-Decryption.md) - ## Flink 引擎参数 这里列出了一些与 Flink 中名称相对应的 SeaTunnel 参数名称，并非全部，更多内容请参考官方 [Flink Documentation](https://flink.apache.org/) for more. From f7f8cfca7870d65b77f5c78fea8e5a46716a9d27 Mon Sep 17 00:00:00 2001 From: Jia Fan Date: Fri, 13 Sep 2024 11:06:01 +0800 Subject: [PATCH 2/4] update --- docs/en/connector-v2/sink/Assert.md | 2 +- docs/en/start-v2/docker/docker.md | 12 ++++++------ docs/zh/start-v2/docker/docker.md | 12 ++++++------ 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/en/connector-v2/sink/Assert.md b/docs/en/connector-v2/sink/Assert.md index 681cb2de940..e1b93598a43 100644 --- a/docs/en/connector-v2/sink/Assert.md +++ b/docs/en/connector-v2/sink/Assert.md @@ -4,7 +4,7 @@ ## Description -A flink sink plugin which can assert illegal data by user defined rules +A sink plugin which can assert illegal data by user defined rules ## Key Features diff --git a/docs/en/start-v2/docker/docker.md b/docs/en/start-v2/docker/docker.md index 25ac52044bf..d5c4ada4b2c 100644 --- a/docs/en/start-v2/docker/docker.md +++ b/docs/en/start-v2/docker/docker.md @@ -146,14 +146,14 @@ docker run --rm -it apache/seatunnel bash -c '/bin/start-cluste there has 2 ways to create cluster within docker. -### 1. Use Docker Directly +### Use Docker Directly -1. create a network +#### create a network ```shell docker network create seatunnel-network ``` -2. start the nodes +#### start the nodes - start master node ```shell ## start master and export 5801 port @@ -213,7 +213,7 @@ docker run -d --name seatunnel_worker_1 \ ``` -### 2. Use Docker-compose +### Use Docker-compose > docker cluster mode is only support zeta engine. @@ -368,7 +368,7 @@ and run `docker-compose up -d` command, the new worker node will start, and the ### Job Operation on cluster -1. use docker as a client +#### use docker as a client - submit job : ```shell docker run --name seatunnel_client \ @@ -393,7 +393,7 @@ more command please refer [user-command](../../seatunnel-engine/user-command.md) -2. use rest api +### use rest api please refer [Submit A Job](../../seatunnel-engine/rest-api.md#submit-a-job) diff --git a/docs/zh/start-v2/docker/docker.md b/docs/zh/start-v2/docker/docker.md index bccd4b068f1..309417a5993 100644 --- a/docs/zh/start-v2/docker/docker.md +++ b/docs/zh/start-v2/docker/docker.md @@ -149,14 +149,14 @@ docker下的集群模式仅支持Zeta引擎 有两种方式来启动集群 -### 1. 直接使用Docker +### 直接使用Docker -1. 创建一个network +#### 创建一个network ```shell docker network create seatunnel-network ``` -2. 启动节点 +#### 启动节点 - 启动master节点 ```shell ## start master and export 5801 port @@ -214,7 +214,7 @@ docker run -d --name seatunnel_worker_1 \ ./bin/seatunnel-cluster.sh -r worker ``` -### 2. 使用docker-compose +### 使用docker-compose `docker-compose.yaml` 配置文件为： ```yaml version: '3.8' @@ -360,7 +360,7 @@ networks: ### 提交作业到集群 -1. 使用docker container作为客户端 +#### 使用docker container作为客户端 - 提交任务 ```shell docker run --name seatunnel_client \ @@ -383,5 +383,5 @@ docker run --name seatunnel_client \ 更多其他命令请参考[命令行工具](../../seatunnel-engine/user-command.md) -2. 使用RestAPI +#### 使用RestAPI 请参考 [提交作业](../../seatunnel-engine/rest-api.md#提交作业) \ No newline at end of file From ed07cf442a7d79df59df3ca3780ed4e947f65c1d Mon Sep 17 00:00:00 2001 From: Jia Fan Date: Fri, 13 Sep 2024 11:07:37 +0800 Subject: [PATCH 3/4] update --- docs/en/start-v2/docker/docker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/start-v2/docker/docker.md b/docs/en/start-v2/docker/docker.md index d5c4ada4b2c..3dfe3ec8c85 100644 --- a/docs/en/start-v2/docker/docker.md +++ b/docs/en/start-v2/docker/docker.md @@ -393,7 +393,7 @@ more command please refer [user-command](../../seatunnel-engine/user-command.md) -### use rest api +#### use rest api please refer [Submit A Job](../../seatunnel-engine/rest-api.md#submit-a-job) From 765fd8395645c7ee9614348ba88199d71832498c Mon Sep 17 00:00:00 2001 From: Jia Fan Date: Sat, 14 Sep 2024 13:49:19 +0800 Subject: [PATCH 4/4] update --- docs/en/start-v2/locally/deployment.md | 2 +- docs/zh/start-v2/locally/deployment.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/start-v2/locally/deployment.md b/docs/en/start-v2/locally/deployment.md index db5dbdd0fd1..8555c097f36 100644 --- a/docs/en/start-v2/locally/deployment.md +++ b/docs/en/start-v2/locally/deployment.md @@ -69,7 +69,7 @@ You can download the source code from the [download page](https://seatunnel.apac ```shell cd seatunnel -sh ./mvnw clean package -DskipTests -Dskip.spotless=true +sh ./mvnw clean install -DskipTests -Dskip.spotless=true # get the binary package cp seatunnel-dist/target/apache-seatunnel-2.3.8-bin.tar.gz /The-Path-You-Want-To-Copy diff --git a/docs/zh/start-v2/locally/deployment.md b/docs/zh/start-v2/locally/deployment.md index e2ae8dadd03..ce17e773319 100644 --- a/docs/zh/start-v2/locally/deployment.md +++ b/docs/zh/start-v2/locally/deployment.md @@ -69,7 +69,7 @@ connector-console ```shell cd seatunnel -sh ./mvnw clean package -DskipTests -Dskip.spotless=true +sh ./mvnw clean install -DskipTests -Dskip.spotless=true # 获取构建好的二进制包 cp seatunnel-dist/target/apache-seatunnel-2.3.8-bin.tar.gz /The-Path-You-Want-To-Copy