[SPARK-34887][PYTHON] Port Koalas dependencies into PySpark #32386

xinrong-meng · 2021-04-28T18:10:14Z

What changes were proposed in this pull request?

Port Koalas dependencies appropriately to PySpark dependencies.

Why are the changes needed?

pandas-on-Spark has its own required dependency and optional dependencies.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual test.

Keyword:SPARK-34849

xinrong-meng · 2021-04-28T18:11:19Z

FYI @HyukjinKwon @ueshin @itholic

ueshin · 2021-04-28T18:25:35Z

dev/requirements.txt

+# Optional dependencies in pandas-on-spark.
+mlflow>=1.0
+plotly>=4.8
+matplotlib>=3.0.0,<3.3.0


What is this file used for?

I think it's used for setting up a development environment for PySpark.

I'm not sure we should add them here because even dependencies for SQL or ML modules are not here.

this file is not used anywhere .. it's just for dev setup :-).

ueshin · 2021-04-28T18:27:28Z

python/setup.py

+            'pandas>=0.23.2',
+            'pyarrow>=0.10',
+            'numpy>=1.14,<1.20.0',


I don't think we should install them by default but should be extras.

They are moved to extras_require.

Does it mean when users install pyspark, pandas_on_spark doesn't work by default?

ueshin · 2021-04-28T18:27:58Z

python/setup.py

+            'pandas.mlflow': ['mlflow>=1.0'],
+            'pandas.plotly': ['plotly>=4.8'],
+            'pandas.matplotlib': ['matplotlib>=3.0.0,<3.3.0'],


I'm not sure we should list these here? cc @HyukjinKwon

Maybe we could:

extras_require={ ... 'pandas_on_spark': [ 'pandas>=0.23.2', 'pyarrow>=0.10', 'numpy>=1.14,<1.20.0', ] ...

Yeah, I think we should remove them for now.

We might need to add some description in [1] to add the dependencies info for pandas_on_spark.
[1] https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies

Optional dependencies for pandas-on-spark have been removed.
Dependencies for pandas-on-spark have been moved to extras_require.

Descriptions are added to install.rst.

Shall we call it Pandas-on-spark in the document?

garawalid · 2021-04-28T19:00:17Z

dev/requirements.txt

@@ -6,3 +6,13 @@ pydata_sphinx_theme
 ipython
 nbsphinx
 numpydoc
+
+# dependencies in pandas-on-spark.


@xinrong-databricks
Do you intend to include dev dependencies in this PR?

Yeah, this file is only used when setting up a dev env for now.

Yes, this PR includes dev dependencies.

SparkQA · 2021-04-28T19:23:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42572/

SparkQA · 2021-04-28T19:28:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42572/

SparkQA · 2021-04-28T20:57:21Z

Test build #138053 has finished for PR 32386 at commit 3e4754d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Yikun · 2021-04-29T02:28:28Z

python/setup.py

+            'pandas>=0.23.2',
+            'pyarrow>=0.10',
+            'numpy>=1.14,<1.20.0',
+        ],
        extras_require={
            'ml': ['numpy>=1.7'],
            'mllib': ['numpy>=1.7'],
            'sql': [
                'pandas>=%s' % _minimum_pandas_version,
                'pyarrow>=%s' % _minimum_pyarrow_version,


nit: pyarrow minimun version is introduced in _minimum_pyarrow_version, but if we introduce the pyarrow>=0.10 for pandas_on_spark in here, maybe we could change the _minimum_pyarrow_version to something like _minimum_sql_pyarrow_version

[1] https://github.com/apache/spark/pull/32386/files#diff-eb8b42d9346d0a5d371facf21a8bfa2d16fb49e213ae7c21f03863accebe0fcfR115

Good catch! There is only one usage of _minimum_pyarrow_version. How about removing the variable and using the value instead?

If we can use the same lower bound, I prefer just use _minimum_pyarrow_version directly as suggestion in #32386 (comment)

'pandas_on_spark': [ 'pandas>=%s' % _minimum_pandas_version, 'pyarrow>=%s', % _minimum_pyarrow_version, ]

Sounds good.

Yikun · 2021-04-29T02:30:05Z

python/setup.py

+            'pandas.mlflow': ['mlflow>=1.0'],
+            'pandas.plotly': ['plotly>=4.8'],
+            'pandas.matplotlib': ['matplotlib>=3.0.0,<3.3.0'],


We might need to add some description in [1] to add the dependencies info for pandas_on_spark.
[1] https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies

SparkQA · 2021-04-29T18:20:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42608/

SparkQA · 2021-04-29T18:20:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42608/

SparkQA · 2021-04-29T20:17:52Z

Test build #138088 has finished for PR 32386 at commit 939ca2f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2021-04-30T01:37:32Z

python/setup.py

+            ],
+            'pandas_on_spark': [
+                'pandas>=%s' % _minimum_pandas_version,
+                'pyarrow>=0.10',


The pandas on Spark is based on PySpark SQL with pandas UDF which won't work with pyarrow < 1.0.0. We should use the same lower bound.

Makes sense. The same lower bound is used.

ueshin · 2021-04-30T01:44:05Z

dev/requirements.txt

+# Optional dependencies in pandas-on-spark.
+mlflow>=1.0
+plotly>=4.8
+matplotlib>=3.0.0,<3.3.0


I'm not sure we should add them here because even dependencies for SQL or ML modules are not here.

ueshin · 2021-04-30T01:46:10Z

python/docs/source/getting_started/install.rst

+`pandas`      0.23.2                    Required for Pandas-on-spark
+`pyarrow`     0.10                      Required for Pandas-on-spark
+`Numpy`       1.14(<1.20.0)             Required for Pandas-on-spark


pandas-on-Spark?

or "PySpark APIs on Spark" which will be the official name. Either one should be fine because ML is used over MLlib DataFraem-based API as its official name (see http://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api)

May I confirm the official name is PySpark APIs on Spark? or pandas APIs on Spark?

pandas-on-Spark is used for now to keep consistent with ML for example.

SparkQA · 2021-04-30T18:09:07Z

Test build #138113 has finished for PR 32386 at commit cca7a49.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-04-30T18:23:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42634/

SparkQA · 2021-04-30T18:23:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42634/

ueshin

Otherwise, LGTM.

python/docs/source/getting_started/install.rst

HyukjinKwon · 2021-05-04T00:04:15Z

Merged to master.

xinrong-meng added 2 commits April 27, 2021 16:01

Prototype

d0dbfd6

Fix

3e4754d

github-actions bot added BUILD PYTHON labels Apr 28, 2021

ueshin reviewed Apr 28, 2021

View reviewed changes

garawalid reviewed Apr 28, 2021

View reviewed changes

Yikun reviewed Apr 29, 2021

View reviewed changes

xinrong-meng added 2 commits April 29, 2021 10:00

Adjust setup.py

28d1034

+ description to install.rst

939ca2f

xinrong-meng marked this pull request as ready for review April 29, 2021 17:13

ueshin reviewed Apr 30, 2021

View reviewed changes

pyarrow lower bound; doc

cca7a49

ueshin reviewed Apr 30, 2021

View reviewed changes

python/docs/source/getting_started/install.rst Outdated Show resolved Hide resolved

HyukjinKwon approved these changes May 2, 2021

View reviewed changes

Format

f8a0820

HyukjinKwon closed this in 120c389 May 4, 2021

[SPARK-34887][PYTHON] Port Koalas dependencies into PySpark #32386

[SPARK-34887][PYTHON] Port Koalas dependencies into PySpark #32386

Conversation

xinrong-meng commented Apr 28, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

xinrong-meng commented Apr 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 28, 2021

SparkQA commented Apr 28, 2021

SparkQA commented Apr 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 29, 2021

SparkQA commented Apr 29, 2021

SparkQA commented Apr 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Apr 30, 2021

SparkQA commented Apr 30, 2021

SparkQA commented Apr 30, 2021

ueshin left a comment

Choose a reason for hiding this comment

HyukjinKwon commented May 4, 2021

xinrong-meng commented Apr 28, 2021 •

edited

Loading