Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] OpenSearch-Hadoop 1.0.1 RELEASE artifacts are not compatible with Spark 3 #304

Open
nknize opened this issue Jul 7, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@nknize
Copy link
Collaborator

nknize commented Jul 7, 2023

The released OpenSearch-Hadoop artifacts are only compatible w/ Spark 2.x.

Thus builds against Spark 3 dependencies will fail with the following:

[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/...
[error]    org.apache.spark:spark-core _2.13, _2.11

To REPRO use the following example build.sbt

ThisBuild / scalaVersion := "2.13.0"

lazy val root = (project in file("."))
  .settings(
    name := "opensearch-spark-example"
  )

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "3.2.4" exclude("javax", "servlet") exclude("org.apache", "hadoop"),
  "org.opensearch.client" % "opensearch-hadoop" % "1.0.1",
  "org.antlr" % "antlr4-runtime" % "4.8",
  "org.codehaus.janino" % "commons-compiler" % "3.0.8",
  "org.codehaus.janino" % "janino" % "3.0.8"
)

resolvers += Resolver.mavenCentral
resolvers += Resolver.mavenLocal

We need to publish the Spark 3 compatible version which is built and packaged with the artifacts from the spark/sql-30 module

@nknize nknize added bug Something isn't working untriaged labels Jul 7, 2023
@nknize nknize changed the title [BUG] [BUG] OpenSearch-Hadoop 1.0.1 artifacts are not compatible with Spark 3 Jul 7, 2023
@nknize nknize changed the title [BUG] OpenSearch-Hadoop 1.0.1 artifacts are not compatible with Spark 3 [BUG] OpenSearch-Hadoop 1.0.1 RELEASE artifacts are not compatible with Spark 3 Jul 7, 2023
@pajuric pajuric assigned nknize and unassigned nknize Jul 7, 2023
@pajuric
Copy link

pajuric commented Jul 7, 2023

@nknize - Thanks, Nick.

@Xtansia Xtansia removed the untriaged label Oct 31, 2023
@Xtansia
Copy link
Collaborator

Xtansia commented Oct 31, 2023

I believe this is the same issue as #347, and that what's being asked for already exists as "org.opensearch.client" % "opensearch-spark-30" % "1.0.1"?

@wbeckler
Copy link

wbeckler commented Nov 2, 2023

@nknize does @Xtansia's proposal unblock our blog post?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants