Nebula in pyspark:3.3.1

[xs2tarunkukreja]

Hello All,

I am planning to write ETL Pipeline for Nebula. We are using PySpark for ETL Pipeline.

When I added nebula-spark-connector-3.4.0.jar in PySpark, it start to throw some errors like 2 different version of pyspark libraries.
Like - Multiple source found for text (org.apache.spark.sql.execution.datasources.v2.test.TextDataSourceV2, org.apache.spark.sql.execution.datasources.text.TextFileFormat)

It appear as I am using wrong connector version. Would you please guide me to know the correct version.

[wey-gu]

could you please take a look at this? @Nicole00

[xs2tarunkukreja]

@wey-gu and @Nicole00 - Would you please guide me here the correct version of connector?

[Nicole00]

@wey-gu and @Nicole00 - Would you please guide me here the correct version of connector?

@xs2tarunkukreja
the connector you are using is for Spark 2.4, please see https://github.com/vesoft-inc/nebula-spark-connector#how-to-use to get the correct spark-connector version.
ps: nebula-spark-connector_3.0 is compatible for spark 3.0,3.1,3.2,3.3

[xs2tarunkukreja]

ok.. I will check the same.

[xs2tarunkukreja]

Just curious 3.4 is supporting 2.4 version and 3.0 is for 3.3. Versioning is confusing.

[Nicole00]

the 3.4.0 is the version of Nebula Spark Connector, and the 3.0 is the version of Spark.

you can get the version match in README. https://github.com/vesoft-inc/nebula-spark-connector#compatibility-matrix