[SPARK-35180][BUILD] Allow to build SparkR with SBT #32285

sarutak · 2021-04-22T04:32:37Z

What changes were proposed in this pull request?

This PR proposes a change that allows us to build SparkR with SBT.

Why are the changes needed?

In the current master, SparkR can be built only with Maven.
It's helpful if we can built it with SBT.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I confirmed that I can build SparkR on Ubuntu 20.04 with the following command.

build/sbt -Psparkr package

dongjoon-hyun · 2021-04-22T04:53:01Z

cc @srowen and @HyukjinKwon

dongjoon-hyun

It's great. Could you add the build command into R/README.md along with the existing mvn command, @sarutak ?

https://github.com/apache/spark/blame/master/R/README.md#L23

dongjoon-hyun · 2021-04-22T05:03:34Z

BTW, I can do the AS-IS master branch, what is the missing before this PR?

$ build/sbt package -Psparkr
$ R/install-dev.sh
$ bin/sparkR

R version 3.5.2 (2018-12-20) -- "Eggshell Igloo"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

Launching java with spark-submit command /Users/dongjoon/APACHE/spark-merge/bin/spark-submit   "sparkr-shell" /var/folders/p5/tzyffbvj0md4djphs4fym9zw0000gn/T//RtmpV1x8wW/backend_portc786d577092
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/04/21 22:02:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/04/21 22:02:47 WARN SQLUtils: SparkR: enableHiveSupport is requested for SparkSession but Spark is not built with Hive or spark.sql.catalogImplementation is not set to 'hive', falling back to without Hive support.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.0-SNAPSHOT
      /_/


SparkSession Web UI available at http://localhost:4040
SparkSession available as 'spark'(master = local[*], app id = local-1619067767184).
>

sarutak · 2021-04-22T05:08:19Z

It's great. Could you add the build command into R/README.md along with the existing mvn command, @sarutak ?

O.K, I'll do it.

BTW, I can do the AS-IS master branch, what is the missing before this PR?

mvn -Psparkr runs R/install-dev.sh automatically but SBT didn't.

dongjoon-hyun · 2021-04-22T05:10:37Z

Got it!

mvn -Psparkr runs R/install-dev.sh automatically but SBT didn't.

dongjoon-hyun

+1, LGTM.

dongjoon-hyun · 2021-04-22T05:46:08Z

R/README.md

@@ -17,10 +17,14 @@ export R_HOME=/home/username/R

 #### Build Spark

-Build Spark with [Maven](https://spark.apache.org/docs/latest/building-spark.html#buildmvn) and include the `-Psparkr` profile to build the R package. For example to use the default Hadoop versions you can run
+Build Spark with [Maven](https://spark.apache.org/docs/latest/building-spark.html#buildmvn) or [SBT](http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt), and include the `-Psparkr` profile to build the R package. For example to use the default Hadoop versions you can run


Oh, http -> https.

Ah, I just copied from the URL bar. I'll change it. Thanks.

HyukjinKwon · 2021-04-22T05:49:45Z

project/SparkBuild.scala

+      Process(command.toString).!!
+    },
+    (Compile / compile) := {
+      buildRPackage.value


I think we build SparkR after the compilation of core so the order should probably be switched with the next line.

Thanks. I've updated.

HyukjinKwon

LGTM2

SparkQA · 2021-04-22T06:01:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42299/

SparkQA · 2021-04-22T06:01:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42299/

SparkQA · 2021-04-22T06:22:28Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42305/

SparkQA · 2021-04-22T07:21:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42307/

SparkQA · 2021-04-22T07:21:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42307/

SparkQA · 2021-04-22T08:20:49Z

Test build #137771 has finished for PR 32285 at commit 77a8d45.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-22T09:06:34Z

Test build #137778 has finished for PR 32285 at commit 445050b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-22T09:26:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42317/

SparkQA · 2021-04-22T09:26:37Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42317/

SparkQA · 2021-04-22T10:32:41Z

Test build #137788 has finished for PR 32285 at commit 1f9d827.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-04-22T11:20:49Z

retest this please.

HyukjinKwon · 2021-04-22T11:33:28Z

Merged to master.

SparkQA · 2021-04-22T12:23:16Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42333/

SparkQA · 2021-04-22T12:23:17Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42333/

SparkQA · 2021-04-22T14:28:57Z

Test build #137804 has finished for PR 32285 at commit 1f9d827.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Add SparkR build support for SBT.

77a8d45

github-actions bot added the BUILD label Apr 22, 2021

dongjoon-hyun reviewed Apr 22, 2021

View reviewed changes

Update README.

08c0223

github-actions bot added DOCS R labels Apr 22, 2021

dongjoon-hyun approved these changes Apr 22, 2021

View reviewed changes

dongjoon-hyun reviewed Apr 22, 2021

View reviewed changes

http -> https

445050b

HyukjinKwon reviewed Apr 22, 2021

View reviewed changes

HyukjinKwon approved these changes Apr 22, 2021

View reviewed changes

Change the order between compile and building SparkR.

1f9d827

HyukjinKwon approved these changes Apr 22, 2021

View reviewed changes

asfgit closed this in c0972de Apr 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35180][BUILD] Allow to build SparkR with SBT #32285

[SPARK-35180][BUILD] Allow to build SparkR with SBT #32285

sarutak commented Apr 22, 2021

dongjoon-hyun commented Apr 22, 2021

dongjoon-hyun left a comment

dongjoon-hyun commented Apr 22, 2021

sarutak commented Apr 22, 2021

dongjoon-hyun commented Apr 22, 2021

dongjoon-hyun left a comment

dongjoon-hyun Apr 22, 2021

sarutak Apr 22, 2021

HyukjinKwon Apr 22, 2021

sarutak Apr 22, 2021

HyukjinKwon left a comment

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

sarutak commented Apr 22, 2021

HyukjinKwon commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

[SPARK-35180][BUILD] Allow to build SparkR with SBT #32285

[SPARK-35180][BUILD] Allow to build SparkR with SBT #32285

Conversation

sarutak commented Apr 22, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

dongjoon-hyun commented Apr 22, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun commented Apr 22, 2021

sarutak commented Apr 22, 2021

dongjoon-hyun commented Apr 22, 2021

dongjoon-hyun left a comment

Choose a reason for hiding this comment

dongjoon-hyun Apr 22, 2021

Choose a reason for hiding this comment

sarutak Apr 22, 2021

Choose a reason for hiding this comment

HyukjinKwon Apr 22, 2021

Choose a reason for hiding this comment

sarutak Apr 22, 2021

Choose a reason for hiding this comment

HyukjinKwon left a comment

Choose a reason for hiding this comment

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

sarutak commented Apr 22, 2021

HyukjinKwon commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021

SparkQA commented Apr 22, 2021