Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-12678][CORE] MapPartitionsRDD clearDependencies #10623

Closed
wants to merge 1 commit into from
Closed

[SPARK-12678][CORE] MapPartitionsRDD clearDependencies #10623

wants to merge 1 commit into from

Conversation

gpoulin
Copy link
Contributor

@gpoulin gpoulin commented Jan 6, 2016

MapPartitionsRDD was keeping a reference to prev after a call to
clearDependencies which could lead to memory leak.

@gpoulin
[SPARK-12678][CORE] MapPartitionsRDD clearDependencies
59c5b7d 
MapPartitionsRDD was keeping a reference to `prev` after a call to
`clearDependencies` which could lead to memory leak.
@gpoulin
Copy link
Contributor Author

gpoulin commented Jan 6, 2016

I'm not a JVM so I'm not sure how to do a reliable GC unit test apart from calling System.gc() and hope it actually worked. If somebody know how, let me know, I could add a unit test.

@rxin
Copy link
Contributor

rxin commented Jan 6, 2016

cc @zsxwing

@srowen
Copy link
Member

srowen commented Jan 6, 2016

That looks correct, comparing to UnionRDD for example

@zsxwing
Copy link
Member

zsxwing commented Jan 6, 2016

Jenkins, test this please

@zsxwing
Copy link
Member

zsxwing commented Jan 6, 2016

LGTM pending tests

@SparkQA
Copy link

SparkQA commented Jan 6, 2016

Test build #48871 has finished for PR 10623 at commit 59c5b7d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Jan 7, 2016
@gpoulin @rxin
[SPARK-12678][CORE] MapPartitionsRDD clearDependencies
d061b85 
MapPartitionsRDD was keeping a reference to `prev` after a call to
`clearDependencies` which could lead to memory leak.

Author: Guillaume Poulin <[email protected]>

Closes #10623 from gpoulin/map_partition_deps.

(cherry picked from commit b673852)
Signed-off-by: Reynold Xin <[email protected]>
@rxin
Copy link
Contributor

rxin commented Jan 7, 2016

Thanks - I've merged this in master & branch-1.6.

@asfgit asfgit closed this in b673852 Jan 7, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gpoulin @rxin @srowen @zsxwing @SparkQA