-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-34887][PYTHON] Port Koalas dependencies into PySpark #32386
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,3 +6,13 @@ pydata_sphinx_theme | |
ipython | ||
nbsphinx | ||
numpydoc | ||
|
||
# dependencies in pandas-on-spark. | ||
pandas>=0.23.2 | ||
pyarrow>=0.10 | ||
numpy>=1.14,<1.20.0 | ||
|
||
# Optional dependencies in pandas-on-spark. | ||
mlflow>=1.0 | ||
plotly>=4.8 | ||
matplotlib>=3.0.0,<3.3.0 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is this file used for? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's used for setting up a development environment for PySpark. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure we should add them here because even dependencies for SQL or ML modules are not here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this file is not used anywhere .. it's just for dev setup :-). |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -220,6 +220,13 @@ def run(self): | |
'pyspark.bin', | ||
'pyspark.sbin', | ||
'pyspark.jars', | ||
'pyspark.pandas', | ||
'pyspark.pandas.indexes', | ||
'pyspark.pandas.missing', | ||
'pyspark.pandas.plot', | ||
'pyspark.pandas.spark', | ||
'pyspark.pandas.typedef', | ||
'pyspark.pandas.usage_logging', | ||
'pyspark.python.pyspark', | ||
'pyspark.python.lib', | ||
'pyspark.data', | ||
|
@@ -250,14 +257,22 @@ def run(self): | |
license='http://www.apache.org/licenses/LICENSE-2.0', | ||
# Don't forget to update python/docs/source/getting_started/install.rst | ||
# if you're updating the versions or dependencies. | ||
install_requires=['py4j==0.10.9.2'], | ||
install_requires=[ | ||
'py4j==0.10.9.2', | ||
'pandas>=0.23.2', | ||
'pyarrow>=0.10', | ||
'numpy>=1.14,<1.20.0', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we should install them by default but should be extras. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. They are moved to Does it mean |
||
], | ||
extras_require={ | ||
'ml': ['numpy>=1.7'], | ||
'mllib': ['numpy>=1.7'], | ||
'sql': [ | ||
'pandas>=%s' % _minimum_pandas_version, | ||
'pyarrow>=%s' % _minimum_pyarrow_version, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: pyarrow minimun version is introduced in _minimum_pyarrow_version, but if we introduce the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch! There is only one usage of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we can use the same lower bound, I prefer just use 'pandas_on_spark': [
'pandas>=%s' % _minimum_pandas_version,
'pyarrow>=%s', % _minimum_pyarrow_version,
] There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sounds good. |
||
] | ||
], | ||
'pandas.mlflow': ['mlflow>=1.0'], | ||
'pandas.plotly': ['plotly>=4.8'], | ||
'pandas.matplotlib': ['matplotlib>=3.0.0,<3.3.0'], | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure we should list these here? cc @HyukjinKwon There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe we could:
Yeah, I think we should remove them for now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We might need to add some description in [1] to add the dependencies info for pandas_on_spark. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Optional dependencies for pandas-on-spark have been removed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Descriptions are added to Shall we call it |
||
}, | ||
python_requires='>=3.6', | ||
classifiers=[ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xinrong-databricks
Do you intend to include dev dependencies in this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this file is only used when setting up a dev env for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this PR includes dev dependencies.