Scala 2.13 support. #9099

dotbg · 2023-04-26T14:28:12Z

xgboost4j support of scala 2.13 #8650 and #6596

trivialfis · 2023-04-26T16:59:45Z

Thank you for the work on updating the scala version. I will defer the review to @wbo4958 as I'm not familiar with the ecosystem.

eejbyfeldt

I left a review since I also looked into adding support for scala 2.13. But since I not a maintainer for this project I guess you have to take them for what they are.

Also if I run a compile on this branch

$ mvn compile -Pdefault,scala-2.13
...
[ERROR] Failed to execute goal on project xgboost4j-example_2.13: Could not resolve dependencies for project ml.dmlc:xgboost4j-example_2.13:jar:2.0.0-SNAPSHOT: Could not find artifact ml.dmlc:xgboost4j_2.12:jar:2.0.0-SNAPSHOT in central_maven (https://repo1.maven.org/maven2) -> [Help 1]

it fails looking for some 2.12 version of the xgboost4j jar. Not sure if this just for me or if there is something more that was missed.

eejbyfeldt · 2023-04-27T07:24:45Z

jvm-packages/README.md

+</dependency>
+<dependency>
+    <groupId>ml.dmlc</groupId>
+    <artifactId>xgboost4j-spark_2.12</artifactId>


Suggested change

<artifactId>xgboost4j-spark_2.12</artifactId>

<artifactId>xgboost4j-spark_2.13</artifactId>

?

@eejbyfeldt unfortunately, you have to do mvn install -Pdefault,scala-2.13 instead of simply mvn compile :(

Thanks a lot for your comments. I totally missed few important things :(

eejbyfeldt · 2023-04-27T07:28:58Z

jvm-packages/README.md

@@ -47,7 +60,7 @@ libraryDependencies ++= Seq(

 For the latest release version number, please check [here](https://github.com/dmlc/xgboost/releases).

-To enable the GPU algorithm (`tree_method='gpu_hist'`), use artifacts `xgboost4j-gpu_2.12` and `xgboost4j-spark-gpu_2.12` instead.
+To enable the GPU algorithm (`tree_method='gpu_hist'`), use artifacts `xgboost4j-gpu_2.12`(`xgboost4j-gpu_2.13`) and `xgboost4j-spark-gpu_2.12`(`xgboost4j-spark-gpu_2.13`) instead.


Since xgboost4j-spark-gpu_2.13 depends on rapids-4-spark which only has a 2.12 version https://mvnrepository.com/artifact/com.nvidia/rapids-4-spark I do not think it possible to build the xgboost4j-spark-gpu package for 2.13 yet.

Indeed. NVIDIA/spark-rapids#1525
I will fix the docs, thanks a lot!

jvm-packages/pom.xml

jvm-packages/xgboost4j-gpu/pom.xml

eejbyfeldt · 2023-04-27T07:37:26Z

jvm-packages/xgboost4j-spark/pom.xml

        <version>2.0.0-SNAPSHOT</version>
    </parent>
-    <artifactId>xgboost4j-spark_2.12</artifactId>
+    <name>xgboost4j-spark</name>
+    <artifactId>xgboost4j-spark_${scala.binary.version}</artifactId>


Using a property here leads to maven producing this warning:

[WARNING] Some problems were encountered while building the effective model for ml.dmlc:xgboost4j-spark_2.13:jar:2.0.0-SNAPSHOT [WARNING] 'artifactId' contains an expression but should be a constant. @ ml.dmlc:xgboost4j-spark_${scala.binary.version}:2.0.0-SNAPSHOT, /home/eejbyfeldt/dev/dmlc/xgboost/jvm-packages/xgboost4j-spark/pom.xml, line 12, column 17

is that really save to ignore?

Spark that also uses maven as build system has a dedicated script (https://github.com/apache/spark/blob/master/dev/change-scala-version.sh) for changing the scala version. I am not very familiar with maven but maybe there are some good reasons they chose that approach?

It is safe for now. Sadly, making the script as spark adds more complexity than just using profiles.
I've reused the approach discussed in the davidB/scala-maven-plugin#177
Spark approach originates in 2015 and maven changed since. I can make a script in place of maven profiles, if necessary.

jvm-packages/xgboost4j-tester/generate_pom.py

jvm-packages/xgboost4j/src/main/scala/ml/dmlc/xgboost4j/scala/XGBoost.scala

trivialfis · 2023-04-29T11:37:18Z

But since I not a maintainer for this project I guess you have to take them for what they are.

You are more than welcome to contribute through any means. Also, we are looking for new maintainers for jvm packages and R.

dotbg · 2023-05-02T14:16:00Z

Also, we are looking for new maintainers for jvm packages and R.

@trivialfis I have few things I'd like to do in the jvm-packages and have some spare time for that, so I can lend a hand for a while.

trivialfis · 2023-05-04T19:46:03Z

Thank you for sharing your interest! Let's work through those items first.

trivialfis

Few questions, still trying to make time to learn about the spark ecosystem. ;-)

Otherwise looks fine to me. cc @wbo4958 .

trivialfis · 2023-05-04T19:48:21Z

jvm-packages/README.md

@@ -96,7 +121,9 @@ libraryDependencies ++= Seq(

 For the latest release version number, please check [the repository listing](https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/list.html).

+### GPU algorithm


@wbo4958 Could you please help confirm this is still the case?

sadly, the NVIDIA/spark-rapids#1525 is a blocker for the GPU-enabled modules. I do not think this has to be a showstopper for 2.13.

trivialfis · 2023-05-04T19:49:44Z

jvm-packages/pom.xml

@@ -5,7 +5,7 @@
    <modelVersion>4.0.0</modelVersion>

    <groupId>ml.dmlc</groupId>
-    <artifactId>xgboost-jvm_2.12</artifactId>
+    <artifactId>xgboost-jvm</artifactId>


Apologies for the newbie question. Is this changing the jar name? If so, is it considered a breaking change?

No, it is not changing the jar name, as the relevant project is not published as jar.

<packaging>pom</packaging>

Pom-packaging is a way to specify the container of submodules. It does not lead to a jar.
jvm-packages only produce jars for xgboost4j-spark, xgboost4j, xgboost4j-flink, xgboost4j-example and the gpu-enabled modules

Thank you for the information!

jvm-packages/xgboost4j/src/main/scala/ml/dmlc/xgboost4j/scala/EvalTrait.scala

wbo4958 · 2023-05-08T09:48:07Z

great job. I will review it. Sorry for that

jvm-packages/README.md

wbo4958 · 2023-05-08T10:04:10Z

Really great job. LGTM assuming the CI can pass

dotbg · 2023-05-08T12:02:06Z

Thanks a lot @wbo4958 I've updated the xgboost4j-tester so that it checks both 2.12 and 2.13.

dotbg · 2023-05-09T09:02:46Z

I am not sure whether I can do anything for docs and python tests.
@trivialfis , @wbo4958 are these known issues?

dotbg · 2023-05-10T09:53:03Z

Resolved conflicts.

trivialfis · 2023-05-10T16:31:23Z

Will merge once all tests are green. Thank you for the work on supporting the newer Scala!

dotbg · 2023-05-10T21:20:19Z

@trivialfis had to modify the docker image used for the integration tests. Should be green now.

dotbg · 2023-05-15T14:35:14Z

updated the logic, used to build images for the integration tests + rebased

trivialfis · 2023-05-15T22:02:48Z

Thank you for the update! Are you able to reproduce this error: https://buildkite.com/xgboost/xgboost-ci-multi-gpu/builds/2184#01881fd5-fa29-4540-92f4-eb8861dab13b ?

Feel free to let me know if there's anything I can help.

wbo4958 · 2023-05-16T02:59:31Z

Thank you for the update! Are you able to reproduce this error: https://buildkite.com/xgboost/xgboost-ci-multi-gpu/builds/2184#01881fd5-fa29-4540-92f4-eb8861dab13b ?

Feel free to let me know if there's anything I can help.

Oops, spark-rapids 23.04.0 doesn't support spark3.4. I will fix it.

dotbg · 2023-05-16T07:35:21Z

@wbo4958 thanks a lot! Reproducing some things locally requires patching, so it takes more time than just writing code.
I am not sure whether some of the changes in CI scripts are worth making a PR with these.

dotbg · 2023-05-17T10:58:54Z

@trivialfis, @wbo4958 I have rebased the branch again, so all changes should be included. Do think, I should do anything else before this MR can be merged?

trivialfis · 2023-05-19T06:40:36Z

Restarted the CI due to a network error. Maven seems quite unstable on the CI recently.

jvm-packages/xgboost4j-example/src/test/resources/iris.csv

trivialfis

Overall looks good to me, one minor issue in the comments.

Thank you for the nice work on not only updating the Scala version but also handling the tests!

...boost4j-example/src/test/scala/ml/dmlc/xgboost4j/scala/example/spark/SparkExamplesTest.scala

1. Updated the test logic 2. Added smoke tests for Spark examples. 3. Added integration tests for Spark with Scala 2.13

dotbg · 2023-05-31T09:12:47Z

@trivialfis Thanks a lot !

Are there any release plans?

dotbg marked this pull request as ready for review April 26, 2023 14:30

eejbyfeldt reviewed Apr 28, 2023

View reviewed changes

dotbg force-pushed the scala-2.13-support branch 4 times, most recently from 8004451 to 3cbd533 Compare May 2, 2023 14:13

dotbg requested a review from eejbyfeldt May 2, 2023 14:13

trivialfis reviewed May 4, 2023

View reviewed changes

dotbg requested a review from trivialfis May 8, 2023 08:01

wbo4958 reviewed May 8, 2023

View reviewed changes

jvm-packages/README.md Show resolved Hide resolved

dotbg force-pushed the scala-2.13-support branch from ea51a5d to bbe2111 Compare May 8, 2023 12:00

dotbg requested a review from wbo4958 May 8, 2023 13:55

dotbg force-pushed the scala-2.13-support branch from bbe2111 to 1fd2b3e Compare May 10, 2023 09:52

trivialfis approved these changes May 10, 2023

View reviewed changes

dotbg force-pushed the scala-2.13-support branch 2 times, most recently from 1fd2b3e to 4fa4417 Compare May 10, 2023 21:18

dotbg force-pushed the scala-2.13-support branch 3 times, most recently from a595a85 to 29f2380 Compare May 12, 2023 10:26

dotbg force-pushed the scala-2.13-support branch 3 times, most recently from d8e1088 to 6b3c971 Compare May 15, 2023 14:34

dotbg force-pushed the scala-2.13-support branch from 6b3c971 to 3be1abd Compare May 17, 2023 10:50

trivialfis reviewed May 19, 2023

View reviewed changes

jvm-packages/xgboost4j-example/src/test/resources/iris.csv Outdated Show resolved Hide resolved

dotbg requested a review from trivialfis May 22, 2023 10:05

dotbg force-pushed the scala-2.13-support branch 2 times, most recently from 47d809a to 821b39d Compare May 25, 2023 11:40

trivialfis approved these changes May 25, 2023

View reviewed changes

trivialfis reviewed May 25, 2023

View reviewed changes

...boost4j-example/src/test/scala/ml/dmlc/xgboost4j/scala/example/spark/SparkExamplesTest.scala Outdated Show resolved Hide resolved

dotbg force-pushed the scala-2.13-support branch from 821b39d to 376b576 Compare May 26, 2023 20:50

dotbg added 2 commits May 26, 2023 22:55

Scala 2.13 support.

79671b2

Updated the xgboost4j-tester.

af77259

1. Updated the test logic 2. Added smoke tests for Spark examples. 3. Added integration tests for Spark with Scala 2.13

dotbg force-pushed the scala-2.13-support branch from 376b576 to af77259 Compare May 26, 2023 20:55

trivialfis merged commit a01df10 into dmlc:master May 27, 2023

dotbg deleted the scala-2.13-support branch May 31, 2023 09:12

dotbg mentioned this pull request May 31, 2023

Scala 2.13 and Spark 3.2.1 #8310

Closed

hcho3 mentioned this pull request Jun 1, 2023

XGBoost4J release supporting scala 2.13 #6596

Closed

nicster mentioned this pull request Jun 1, 2023

Will Mleap support Scala 2.13? combust/mleap#811

Open

hcho3 mentioned this pull request Nov 7, 2023

[jvm-package] Upload XGBoost4J with Scala 2.13 to Maven Central #9765

Closed

s0t00524 mentioned this pull request Aug 21, 2024

Compatibility with Spark 3.2 + Scala 2.13 intel-analytics/ipex-llm#7399

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala 2.13 support. #9099

Scala 2.13 support. #9099

dotbg commented Apr 26, 2023 •

edited

Loading

trivialfis commented Apr 26, 2023

eejbyfeldt left a comment

eejbyfeldt Apr 27, 2023

dotbg May 2, 2023 •

edited

Loading

eejbyfeldt Apr 27, 2023 •

edited

Loading

dotbg May 2, 2023 •

edited

Loading

eejbyfeldt Apr 27, 2023

dotbg May 2, 2023

trivialfis commented Apr 29, 2023

dotbg commented May 2, 2023 •

edited

Loading

trivialfis commented May 4, 2023

trivialfis left a comment

trivialfis May 4, 2023

dotbg May 8, 2023

wbo4958 May 8, 2023

trivialfis May 4, 2023

dotbg May 8, 2023

trivialfis May 10, 2023

wbo4958 commented May 8, 2023

wbo4958 commented May 8, 2023

dotbg commented May 8, 2023

dotbg commented May 9, 2023 •

edited

Loading

dotbg commented May 10, 2023

trivialfis commented May 10, 2023

dotbg commented May 10, 2023

dotbg commented May 15, 2023

trivialfis commented May 15, 2023

wbo4958 commented May 16, 2023

dotbg commented May 16, 2023

dotbg commented May 17, 2023

trivialfis commented May 19, 2023

trivialfis left a comment

dotbg commented May 31, 2023 •

edited

Loading

	<artifactId>xgboost4j-spark_2.12</artifactId>
	<artifactId>xgboost4j-spark_2.13</artifactId>

		@@ -96,7 +121,9 @@ libraryDependencies ++= Seq(

		For the latest release version number, please check [the repository listing](https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/list.html).

		### GPU algorithm

Scala 2.13 support. #9099

Scala 2.13 support. #9099

Conversation

dotbg commented Apr 26, 2023 • edited Loading

trivialfis commented Apr 26, 2023

eejbyfeldt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dotbg May 2, 2023 • edited Loading

Choose a reason for hiding this comment

eejbyfeldt Apr 27, 2023 • edited Loading

Choose a reason for hiding this comment

dotbg May 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trivialfis commented Apr 29, 2023

dotbg commented May 2, 2023 • edited Loading

trivialfis commented May 4, 2023

trivialfis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wbo4958 commented May 8, 2023

wbo4958 commented May 8, 2023

dotbg commented May 8, 2023

dotbg commented May 9, 2023 • edited Loading

dotbg commented May 10, 2023

trivialfis commented May 10, 2023

dotbg commented May 10, 2023

dotbg commented May 15, 2023

trivialfis commented May 15, 2023

wbo4958 commented May 16, 2023

dotbg commented May 16, 2023

dotbg commented May 17, 2023

trivialfis commented May 19, 2023

trivialfis left a comment

Choose a reason for hiding this comment

dotbg commented May 31, 2023 • edited Loading

dotbg commented Apr 26, 2023 •

edited

Loading

dotbg May 2, 2023 •

edited

Loading

eejbyfeldt Apr 27, 2023 •

edited

Loading

dotbg May 2, 2023 •

edited

Loading

dotbg commented May 2, 2023 •

edited

Loading

dotbg commented May 9, 2023 •

edited

Loading

dotbg commented May 31, 2023 •

edited

Loading