Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-12617][PySpark]Move Py4jCallbackConnectionCleaner to Streaming #10621

Closed
wants to merge 1 commit into from
Closed

[SPARK-12617][PySpark]Move Py4jCallbackConnectionCleaner to Streaming #10621

wants to merge 1 commit into from

Conversation

zsxwing
Copy link
Member

@zsxwing zsxwing commented Jan 6, 2016

Move Py4jCallbackConnectionCleaner to Streaming because the callback server starts only in StreamingContext.

zsxwing
zsxwing reviewed Jan 6, 2016
View reviewed changes
python/pyspark/streaming/context.py
from py4j.java_gateway import quiet_close, quiet_shutdown

callback_server = self._gateway._callback_server
if callback_server:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a defensive check

@zsxwing zsxwing mentioned this pull request Jan 6, 2016
Closed
@zsxwing
Copy link
Member Author

zsxwing commented Jan 6, 2016

CC @davies

@davies
Copy link
Contributor

davies commented Jan 6, 2016

LGTM

@zsxwing
Copy link
Member Author

zsxwing commented Jan 6, 2016

retest this please

@SparkQA
Copy link

SparkQA commented Jan 6, 2016

Test build #48867 has finished for PR 10621 at commit 329a78b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class Py4jCallbackConnectionCleaner(object):

@zsxwing
Copy link
Member Author

zsxwing commented Jan 6, 2016

Merging to master and 1.6

@zsxwing
Copy link
Member Author

zsxwing commented Jan 6, 2016

and 1.5

asfgit pushed a commit that referenced this pull request Jan 6, 2016
@zsxwing
[SPARK-12617][PYSPARK] Move Py4jCallbackConnectionCleaner to Streaming
d821fae 
Move Py4jCallbackConnectionCleaner to Streaming because the callback server starts only in StreamingContext.

Author: Shixiong Zhu <[email protected]>

Closes #10621 from zsxwing/SPARK-12617-2.

(cherry picked from commit 1e6648d)
Signed-off-by: Shixiong Zhu <[email protected]>
@asfgit asfgit closed this in 1e6648d Jan 6, 2016
asfgit pushed a commit that referenced this pull request Jan 6, 2016
@zsxwing
[SPARK-12617][PYSPARK] Move Py4jCallbackConnectionCleaner to Streaming
598a5c2 
Move Py4jCallbackConnectionCleaner to Streaming because the callback server starts only in StreamingContext.

Author: Shixiong Zhu <[email protected]>

Closes #10621 from zsxwing/SPARK-12617-2.

(cherry picked from commit 1e6648d)
Signed-off-by: Shixiong Zhu <[email protected]>
@zsxwing zsxwing deleted the SPARK-12617-2 branch January 11, 2016 07:02
@sarathjiguru
Copy link

@zsxwing
Could you confirm that the latest version of py4j (0.9.1) got packaged with spark 1.5.2.
The spark that got installed using AWS and the 1.5.2 tag (https://github.com/apache/spark/tree/v1.5.2/python/lib) contains 0.8.2.1.

Let me know, If I have missed anything..

@zsxwing
Copy link
Member Author

zsxwing commented Feb 17, 2016

@zsxwing
Could you confirm that the latest version of py4j (0.9.1) got packaged with spark 1.5.2.
The spark that got installed using AWS and the 1.5.2 tag (https://github.com/apache/spark/tree/v1.5.2/python/lib) contains 0.8.2.1.

Let me know, If I have missed anything..

@sarathjiguru this bug exists in 1.5.2. You need to apply the patches by yourself for now.

@sarathj
Copy link

sarathj commented Feb 22, 2016

@zsxwing
With 1.6 also I ran into the same exception -java.io.IOException: py4j.Py4JException: Cannot obtain a new communication.

I would like to apply patch by using 0.9.1 version of py4j. Could you please let me know, how you applied the fix. Also I could not find py4j-0.9.1-src.zip link directly. Do I need to extract from the https://pypi.python.org/pypi/py4j and zip again?

@zsxwing
Copy link
Member Author

zsxwing commented Feb 22, 2016

@sarathj if you want to upgrade py4j to 0.9.1, you can just check-pick this patch: zsxwing@a3e3e17

If you don't want to upgrade py4j, just check-pick the following two patches should be enough:

f31d0fd
d821fae

@cpalomaressbd
Copy link

Hi,

One question, maybe is stupid question I dont know and I ask sorry in advance. We are workin with hortonworks, in the beginning of the project we were working with HDP 2.4.0 which one has the spark 1.6.0 and wich one has a bug in pyspark, this bug wich one you say resolved in:

zsxwing commented on 22 Feb 2016

The first solution we have done was upgrade to HDP 2.4.3 wich one has spark 1.6.2 and wich one teorically must to have the patch to this problem, but Surprise we have the same bug without the patch.

Ok, Maybe the problem is with hortonworks and I should write in other forum, but I was in the oficial website of spark:

https://spark.apache.org/downloads.html

If you choose the oficial version of spark 1.6.2 and you download this version, wich one was released June 25 2016, if I go to pyspark file, the bug continue in this installation!!!! I am sorry but I dont understadn and We are desperated with this situation.

Should I appply the patch in source and recompile the code? How can I do it step to step?

Thanks in Advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zsxwing @davies @SparkQA @sarathjiguru @sarathj @cpalomaressbd