Skip to content

Commit

Permalink
update exchange parameter (#2337)
Browse files Browse the repository at this point in the history
  • Loading branch information
cooper-lzy authored Nov 7, 2023
1 parent ca8c030 commit 3b59a59
Show file tree
Hide file tree
Showing 32 changed files with 62 additions and 62 deletions.
2 changes: 1 addition & 1 deletion docs-2.0-en/import-export/nebula-exchange/ex-ug-FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ Solution:
- batch: The number of data contained in each nGQL statement sent to the NebulaGraph service.
- partition: The number of Spark data partitions, indicating the number of concurrent data imports.
- partition: The number of partitions to be created when the data is written to {{nebula.name}}, indicating the number of concurrent data imports.
- nebula.rate: Get a token from the token bucket before sending a request to NebulaGraph.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ For different data sources, the vertex configurations are different. There are m
|`tags.vertex.prefix`|string|-|No|Add the specified prefix to the VID. For example, if the VID is `12345`, adding the prefix `tag1` will result in `tag1_12345`. The underscore cannot be modified.|
|`tags.vertex.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`tags.batch`|int|`256`|Yes|The maximum number of vertices written into NebulaGraph in a single batch.|
|`tags.partition`|int|`32`|Yes|The number of Spark partitions.|
|`tags.partition`|int|`32`|Yes|The number of partitions to be created when the data is written to {{nebula.name}}. If `tags.partition ≤ 1`, the number of partitions to be created in {{nebula.name}} is the same as that in the data source.|

### Specific parameters of Parquet/JSON/ORC data sources

Expand Down Expand Up @@ -246,7 +246,7 @@ For the specific parameters of different data sources for edge configurations, p
|`edges.target.policy`|string|-|No|Supports only the value `hash`. Performs hashing operations on VIDs of type string.|
|`edges.ranking`|int|-|No|The column of rank values. If not specified, all rank values are `0` by default.|
|`edges.batch`|int|`256`|Yes|The maximum number of edges written into NebulaGraph in a single batch.|
|`edges.partition`|int|`32`|Yes|The number of Spark partitions.|
|`edges.partition`|int|`32`|Yes|The number of partitions to be created when the data is written to {{nebula.name}}. If `edges.partition ≤ 1`, the number of partitions to be created in {{nebula.name}} is the same as that in the data source.|

### Specific parameters for generating SST files

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down Expand Up @@ -277,7 +277,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down Expand Up @@ -310,7 +310,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Number of pieces of data written to NebulaGraph in a single batch.
batch: 256
# Number of Spark partitions
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set Tag Team information.
Expand Down Expand Up @@ -306,7 +306,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -342,7 +342,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -361,7 +361,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down Expand Up @@ -324,7 +324,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256

# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10
# The interval for message reading. Unit: second.
interval.seconds: 10
Expand Down Expand Up @@ -277,7 +277,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# # The number of data written to NebulaGraph in a single batch.
# batch: 10
# # The number of Spark partitions.
# # The number of partitions to be created when the data is written to {{nebula.name}}.
# partition: 10
# # The interval for message reading. Unit: second.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down Expand Up @@ -298,7 +298,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Batch operation types, including INSERT, UPDATE, and DELETE. defaults to INSERT.
#writeMode: INSERT
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition:10
# The number of data written to NebulaGraph in a single batch.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -341,7 +341,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
# Set the information about the Tag Team.
Expand Down Expand Up @@ -334,7 +334,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down Expand Up @@ -289,7 +289,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down Expand Up @@ -289,7 +289,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10
# The interval for message reading. Unit: second.
interval.seconds: 10
Expand Down Expand Up @@ -282,7 +282,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 10
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 10
# The interval for message reading. Unit: second.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
# Whether to repartition data based on the number of partitions of graph spaces in NebulaGraph when generating the SST file.
Expand Down Expand Up @@ -374,7 +374,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The number of data written to NebulaGraph in a single batch.
batch: 256
# The number of Spark partitions.
# The number of partitions to be created when the data is written to {{nebula.name}}.
partition: 32
# Whether to repartition data based on the number of partitions of graph spaces in NebulaGraph when generating the SST file.
Expand Down
2 changes: 1 addition & 1 deletion docs-2.0-zh/import-export/nebula-exchange/ex-ug-FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ Spark 日志提示 `Hive Schema version 1.2.0 does not match metastore's schema
- batch:每次发送给{{nebula.name}}服务的 nGQL 语句中包含的数据条数。
- partition:Spark 数据的分区数,表示数据导入的并发数。
- partition:数据写入{{nebula.name}}时需要创建的分区数,表示数据导入的并发数。
- nebula.rate:向{{nebula.name}}发送请求前先去令牌桶获取令牌。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@
|`tags.vertex.prefix`|string|-||为 VID 增加指定的前缀。例如 VID 为`12345`,增加前缀`tag1`后为`tag1_12345`。下划线无法修改。|
|`tags.vertex.policy`|string|-||仅支持取值`hash`。对 string 类型的 VID 进行哈希化操作。|
|`tags.batch`|int|`256`||单批次写入{{nebula.name}}的最大点数量。|
|`tags.partition`|int|`32`||Spark 分片数量|
|`tags.partition`|int|`32`||数据写入{{nebula.name}}时需要创建的分区数。如果`tags.partition ≤ 1`,在{{nebula.name}}中创建的分区数和数据源的分区数相同|

### Parquet/JSON/ORC 源特有参数

Expand Down Expand Up @@ -249,7 +249,7 @@
|`edges.target.policy`|string|-||仅支持取值`hash`。对 string 类型的 VID 进行哈希化操作。|
|`edges.ranking`|int|-||rank 值的列。没有指定时,默认所有 rank 值为`0`|
|`edges.batch`|int|`256`||单批次写入{{nebula.name}}的最大边数量。|
|`edges.partition`|int|`32`||Spark 分片数量|
|`edges.partition`|int|`32`||数据写入{{nebula.name}}时需要创建的分区数。如果`edges.partition ≤ 1`,在{{nebula.name}}中创建的分区数和数据源的分区数相同|

### 生成 SST 时的特有参数

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@
# 单批次写入{{nebula.name}}的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
Expand Down Expand Up @@ -283,7 +283,7 @@
# 单批次写入{{nebula.name}}的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@
# 指定单批次写入 {{nebula.name}} 的最大点数量。
batch: 256
# 指定 Spark 分片数量
# 数据写入{{nebula.name}}时需要创建的分区数
partition: 32
}
Expand Down Expand Up @@ -308,7 +308,7 @@
# 指定单批次写入 {{nebula.name}} 的最大边数量。
batch: 256
# 指定 Spark 分片数量
# 数据写入{{nebula.name}}时需要创建的分区数
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ ROW COLUMN+CELL
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -304,7 +304,7 @@ ROW COLUMN+CELL
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ scala> sql("select playerid, teamid, start_year, end_year from basketball.serve"
# 单批次写入 {{nebula.name}} 的最大数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -338,7 +338,7 @@ scala> sql("select playerid, teamid, start_year, end_year from basketball.serve"
# 单批次写入 {{nebula.name}} 的最大数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ nebula-exchange_spark_2.2 仅支持单表查询,不支持多表查询。
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
# 设置 Tag team 相关信息。
Expand Down Expand Up @@ -368,7 +368,7 @@ nebula-exchange_spark_2.2 仅支持单表查询,不支持多表查询。
# 单批次写入 {{nebula.name}} 的数据条数。
batch: 256
# Spark 分区数量
# 数据写入{{nebula.name}}时需要创建的分区数。
partition: 32
}
Expand Down
Loading

0 comments on commit 3b59a59

Please sign in to comment.