Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][Document] Refactor job env config document #7631

Merged
merged 4 commits into from
Sep 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions docs/en/concept/JobEnvConfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,26 @@ You can configure whether the task is in batch or stream mode through `job.mode`

### checkpoint.interval

Gets the interval in which checkpoints are periodically scheduled.
Gets the interval (milliseconds) in which checkpoints are periodically scheduled.

In `STREAMING` mode, checkpoints is required, if you do not set it, it will be obtained from the application configuration file `seatunnel.yaml`. In `BATCH` mode, you can disable checkpoints by not setting this parameter.
In `STREAMING` mode, checkpoints is required, if you do not set it, it will be obtained from the application configuration file `seatunnel.yaml`. In `BATCH` mode, you can disable checkpoints by not setting this parameter. In Zeta `STREAMING` mode, the default value is 30000 milliseconds.

### checkpoint.timeout

The timeout (in milliseconds) for a checkpoint. If the checkpoint is not completed before the timeout, the job will fail. In Zeta, the default value is 30000 milliseconds.

### parallelism

This parameter configures the parallelism of source and sink.

### shade.identifier

Specify the method of encryption, if you didn't have the requirement for encrypting or decrypting config files, this option can be ignored.

For more details, you can refer to the documentation [Config Encryption Decryption](../connector-v2/Config-Encryption-Decryption.md)

## Zeta Engine Parameter

### job.retry.times

Used to control the default retry times when a job fails. The default value is 3, and it only works in the Zeta engine.
Expand All @@ -43,12 +55,6 @@ This parameter is used to specify the location of the savemode when the job is e
The default value is `CLUSTER`, which means that the savemode is executed on the cluster. If you want to execute the savemode on the client,
you can set it to `CLIENT`. Please use `CLUSTER` mode as much as possible, because when there are no problems with `CLUSTER` mode, we will remove `CLIENT` mode.

### shade.identifier

Specify the method of encryption, if you didn't have the requirement for encrypting or decrypting config files, this option can be ignored.

For more details, you can refer to the documentation [Config Encryption Decryption](../connector-v2/Config-Encryption-Decryption.md)

## Flink Engine Parameter

Here are some SeaTunnel parameter names corresponding to the names in Flink, not all of them. Please refer to the official [Flink Documentation](https://flink.apache.org/).
Expand Down
2 changes: 1 addition & 1 deletion docs/en/connector-v2/sink/Assert.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
## Description

A flink sink plugin which can assert illegal data by user defined rules
A sink plugin which can assert illegal data by user defined rules

## Key Features

Expand Down
12 changes: 6 additions & 6 deletions docs/en/start-v2/docker/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,14 +146,14 @@ docker run --rm -it apache/seatunnel bash -c '<YOUR_FLINK_HOME>/bin/start-cluste

there has 2 ways to create cluster within docker.

### 1. Use Docker Directly
### Use Docker Directly

1. create a network
#### create a network
```shell
docker network create seatunnel-network
```

2. start the nodes
#### start the nodes
- start master node
```shell
## start master and export 5801 port
Expand Down Expand Up @@ -213,7 +213,7 @@ docker run -d --name seatunnel_worker_1 \
```


### 2. Use Docker-compose
### Use Docker-compose

> docker cluster mode is only support zeta engine.

Expand Down Expand Up @@ -368,7 +368,7 @@ and run `docker-compose up -d` command, the new worker node will start, and the

### Job Operation on cluster

1. use docker as a client
#### use docker as a client
- submit job :
```shell
docker run --name seatunnel_client \
Expand All @@ -393,7 +393,7 @@ more command please refer [user-command](../../seatunnel-engine/user-command.md)



2. use rest api
#### use rest api

please refer [Submit A Job](../../seatunnel-engine/rest-api.md#submit-a-job)

2 changes: 1 addition & 1 deletion docs/en/start-v2/locally/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ You can download the source code from the [download page](https://seatunnel.apac

```shell
cd seatunnel
sh ./mvnw clean package -DskipTests -Dskip.spotless=true
sh ./mvnw clean install -DskipTests -Dskip.spotless=true
# get the binary package
cp seatunnel-dist/target/apache-seatunnel-2.3.8-bin.tar.gz /The-Path-You-Want-To-Copy

Expand Down
22 changes: 14 additions & 8 deletions docs/zh/concept/JobEnvConfig.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,26 @@

### checkpoint.interval

获取定时调度检查点的时间间隔。
获取定时调度检查点的时间间隔(毫秒)

`STREAMING`模式下,检查点是必须的,如果不设置,将从应用程序配置文件`seatunnel.yaml`中获取。 在`BATCH`模式下,您可以通过不设置此参数来禁用检查点。
`STREAMING`模式下,检查点是必须的,如果不设置,将从应用程序配置文件`seatunnel.yaml`中获取。 在`BATCH`模式下,您可以通过不设置此参数来禁用检查点。在Zeta `STREAMING`模式下,默认值为30000毫秒。

### checkpoint.timeout

检查点的超时时间(毫秒)。如果检查点在超时之前没有完成,作业将失败。在Zeta中,默认值为30000毫秒。

### parallelism

该参数配置source和sink的并行度。

### shade.identifier

指定加密方式,如果您没有加密或解密配置文件的需求,此选项可以忽略。

更多详细信息,您可以参考文档 [Config Encryption Decryption](../../en/connector-v2/Config-Encryption-Decryption.md)

## Zeta 引擎参数

### job.retry.times

用于控制作业失败时的默认重试次数。默认值为3,并且仅适用于Zeta引擎。
Expand All @@ -44,12 +56,6 @@
当值为`CLIENT`时,SaveMode操作在作业提交的过程中执行,使用shell脚本提交作业时,该过程在提交作业的shell进程中执行。使用rest api提交作业时,该过程在http请求的处理线程中执行。
请尽量使用`CLUSTER`模式,因为当`CLUSTER`模式没有问题时,我们将删除`CLIENT`模式。

### shade.identifier

指定加密方式,如果您没有加密或解密配置文件的需求,此选项可以忽略。

更多详细信息,您可以参考文档 [Config Encryption Decryption](../../en/connector-v2/Config-Encryption-Decryption.md)

## Flink 引擎参数

这里列出了一些与 Flink 中名称相对应的 SeaTunnel 参数名称,并非全部,更多内容请参考官方 [Flink Documentation](https://flink.apache.org/) for more.
Expand Down
12 changes: 6 additions & 6 deletions docs/zh/start-v2/docker/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,14 +149,14 @@ docker下的集群模式仅支持Zeta引擎
有两种方式来启动集群


### 1. 直接使用Docker
### 直接使用Docker

1. 创建一个network
#### 创建一个network
```shell
docker network create seatunnel-network
```

2. 启动节点
#### 启动节点
- 启动master节点
```shell
## start master and export 5801 port
Expand Down Expand Up @@ -214,7 +214,7 @@ docker run -d --name seatunnel_worker_1 \
./bin/seatunnel-cluster.sh -r worker
```

### 2. 使用docker-compose
### 使用docker-compose
`docker-compose.yaml` 配置文件为:
```yaml
version: '3.8'
Expand Down Expand Up @@ -360,7 +360,7 @@ networks:

### 提交作业到集群

1. 使用docker container作为客户端
#### 使用docker container作为客户端
- 提交任务
```shell
docker run --name seatunnel_client \
Expand All @@ -383,5 +383,5 @@ docker run --name seatunnel_client \

更多其他命令请参考[命令行工具](../../seatunnel-engine/user-command.md)

2. 使用RestAPI
#### 使用RestAPI
请参考 [提交作业](../../seatunnel-engine/rest-api.md#提交作业)
2 changes: 1 addition & 1 deletion docs/zh/start-v2/locally/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ connector-console

```shell
cd seatunnel
sh ./mvnw clean package -DskipTests -Dskip.spotless=true
sh ./mvnw clean install -DskipTests -Dskip.spotless=true
# 获取构建好的二进制包
cp seatunnel-dist/target/apache-seatunnel-2.3.8-bin.tar.gz /The-Path-You-Want-To-Copy

Expand Down
Loading