Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance exchange jdbc #2212

Merged
merged 1 commit into from
Aug 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ Exchange has the following advantages:

## Version compatibility

Exchange supports Spark versions 2.2.x, 2.4.x, and 3.x.x, which are named `nebula-exchange_spark_2.2`, `nebula-exchange_spark_2.4`, and `nebula-exchange_spark_3.0` for different Spark versions.

The correspondence between the NebulaGraph Exchange version (the JAR version), the NebulaGraph core version and the Spark version is as follows.

| Exchange version | NebulaGraph version | Spark version |
Expand Down
30 changes: 25 additions & 5 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ Before importing data, you need to confirm the following information:

- The Hadoop service has been installed and started.

## Precautions

nebula-exchange_spark_2.2 supports only single table queries, not multi-table queries.

## Steps

### Step 1: Create the Schema in NebulaGraph
Expand Down Expand Up @@ -194,11 +198,18 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
driver:"com.mysql.cj.jdbc.Driver"

# Database user name and password
user:root
user:"root"
password:"12345"

table:player
sentence:"select playerid, age, name from player order by playerid"
# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter, and can additionally configure sentence.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.player"

# Use query statement to read data.
# nebula-exchange_spark_2.2 can configure this parameter. Multi-table queries are not supported. Only the table name needs to be written after from. The form `db.table` is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence:"select playerid, age, name from player, team order by playerid"

# (optional)Multiple connections read parameters. See https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
partitionColumn:playerid # optional. Must be a numeric, date, or timestamp column from the table in question.
Expand Down Expand Up @@ -282,8 +293,17 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
driver:"com.mysql.cj.jdbc.Driver"
user:root
password:"12345"
table:follow
sentence:"select src_player,dst_player,degree from follow order by src_player"

# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter, and can additionally configure sentence.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.follow"

# Use query statement to read data.
# nebula-exchange_spark_2.2 can configure this parameter. Multi-table queries are not supported. Only the table name needs to be written after from. The form `db.table` is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence:"select src_player,dst_player,degree from follow order by src_player"

partitionColumn:src_player
lowerBound:1
upperBound:5
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ Before importing data, you need to confirm the following information:

## Precautions

Only client mode is supported when importing Kafka data, i.e. the value of parameters `tags.type.sink` and `edges.type.sink` is `client`.
- Only client mode is supported when importing Kafka data, i.e. the value of parameters `tags.type.sink` and `edges.type.sink` is `client`.

- When importing Kafka data, do not use Exchange version 3.4.0, which adds caching of imported data and does not support streaming data import. Use Exchange versions 3.0.0, 3.3.0, or 3.5.0.

## Steps

Expand Down
32 changes: 26 additions & 6 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ Before importing data, you need to confirm the following information:

- The Hadoop service has been installed and started.

## Precautions

nebula-exchange_spark_2.2 supports only single table queries, not multi-table queries.

## Steps

### Step 1: Create the Schema in NebulaGraph
Expand Down Expand Up @@ -191,11 +195,19 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`

host:192.168.*.*
port:3306
database:"basketball"
table:"player"
user:"test"
password:"123456"
sentence:"select playerid, age, name from player order by playerid;"
database:"basketball"

# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter. Sentence is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.player"

# Use query statement to read data.
# This parameter is not supported by nebula-exchange_spark_2.2.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence: "select * from people, player, team"

# Specify the column names in the player table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
Expand Down Expand Up @@ -264,11 +276,19 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`

host:192.168.*.*
port:3306
database:"basketball"
table:"follow"
user:"test"
password:"123456"
sentence:"select src_player,dst_player,degree from follow order by src_player;"
database:"basketball"

# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter. Sentence is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.follow"

# Use query statement to read data.
# This parameter is not supported by nebula-exchange_spark_2.2.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence: "select * from follow, serve"

# Specify the column names in the follow table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
Expand Down
36 changes: 28 additions & 8 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-oracle.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ Before importing data, you need to confirm the following information:

- The Hadoop service has been installed and started.

## Precautions

nebula-exchange_spark_2.2 supports only single table queries, not multi-table queries.

## Steps

### Step 1: Create the Schema in NebulaGraph
Expand Down Expand Up @@ -189,12 +193,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
sink: client
}

url:"jdbc:oracle:thin:@host:1521:db"
url:"jdbc:oracle:thin:@host:1521:basketball"
driver: "oracle.jdbc.driver.OracleDriver"
user: "root"
password: "123456"
table: "basketball.player"
sentence: "select playerid, name, age from player"

# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter. Sentence is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.player"

# Use query statement to read data.
# This parameter is not supported by nebula-exchange_spark_2.2.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence: "select * from people, player, team"

# Specify the column names in the player table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
Expand Down Expand Up @@ -226,7 +238,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
sink: client
}

url:"jdbc:oracle:thin:@host:1521:db"
url:"jdbc:oracle:thin:@host:1521:basketball"
driver: "oracle.jdbc.driver.OracleDriver"
user: "root"
password: "123456"
Expand Down Expand Up @@ -260,12 +272,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
sink: client
}

url:"jdbc:oracle:thin:@host:1521:db"
url:"jdbc:oracle:thin:@host:1521:basketball"
driver: "oracle.jdbc.driver.OracleDriver"
user: "root"
password: "123456"
table: "basketball.follow"
sentence: "select src_player, dst_player, degree from follow"

# Scanning a single table to read data.
# nebula-exchange_spark_2.2 must configure this parameter. Sentence is not supported.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as sentence.
table:"basketball.follow"

# Use query statement to read data.
# This parameter is not supported by nebula-exchange_spark_2.2.
# nebula-exchange_spark_2.4 and nebula-exchange_spark_3.0 can configure this parameter, but not at the same time as table. Multi-table queries are supported.
# sentence: "select * from follow, serve"

# Specify the column names in the follow table in fields, and their corresponding values are specified as properties in the NebulaGraph.
# The sequence of fields and nebula.fields must correspond to each other.
Expand Down Expand Up @@ -311,7 +331,7 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
sink: client
}

url:"jdbc:oracle:thin:@host:1521:db"
url:"jdbc:oracle:thin:@host:1521:basketball"
driver: "oracle.jdbc.driver.OracleDriver"
user: "root"
password: "123456"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ Before importing data, you need to confirm the following information:

## Precautions

Only client mode is supported when importing Pulsar data, i.e. the value of parameters `tags.type.sink` and `edges.type.sink` is `client`.
- Only client mode is supported when importing Pulsar data, i.e. the value of parameters `tags.type.sink` and `edges.type.sink` is `client`.

- When importing Pulsar data, do not use Exchange version 3.4.0, which adds caching of imported data and does not support streaming data import. Use Exchange versions 3.0.0, 3.3.0, or 3.5.0.

## Steps

Expand Down