[SPARK-6479][Block Manager]Create off-heap block storage API #5430

zhzhan · 2015-04-08T23:33:50Z

This is the classes for creating off-heap block storage API. It also includes the migration for Tachyon. The diff seems to be big, but it mainly just rename tachyon to offheap. New implementation for hdfs will be submit for review in spark-6112.

rxin · 2015-04-09T00:30:35Z

To help me understand this patch, can you also put the Tachyon implementation into this?

SparkQA · 2015-04-09T01:02:36Z

Test build #29903 has finished for PR 5430 at commit b754d2a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- trait OffHeapBlockManager
This patch does not change any dependencies.

zhzhan · 2015-04-09T01:14:45Z

@rxin I have attached the patch with Tachyon migration code to the JIRA.

https://issues.apache.org/jira/secure/attachment/12724088/spark-6479-tachyon.patch

The patch is incomplete on purpose, because most of the diff (not included) is just changing the term from tachyon to offheap. If you think it is better to do tachyon migration with this JIRA, please let me know and I will do it in one shot.

By the way, there is minor change in OffHeapStore and OffHeapBlockManager, which is inconsistent with this PR, please ignore it.

viper-kun · 2015-04-09T02:11:10Z

core/src/main/scala/org/apache/spark/storage/OffHeapBlockManager.scala

+  val subDirsPerDir = 64
+  def create(blockManager: BlockManager,
+             executorId: String): Option[OffHeapBlockManager] = {
+     val sNames = blockManager.conf.getOption("spark.offHeapStore.blockManager")


i think default value is necessary.

User may not want to use offheap, and in this case the OffHeapBlockManager will be None.

rxin · 2015-04-09T03:48:58Z

Thanks - let's put the Tachyon patch with this. In this case, I think it will be easier to review and understand the API semantics.

rxin · 2015-04-09T03:49:30Z

core/src/main/scala/org/apache/spark/storage/OffHeapStore.scala

+  override def getValues(blockId: BlockId): Option[Iterator[Any]] = {
+    getBytes(blockId).map(buffer => blockManager.dataDeserialize(blockId, buffer))
+  }
+


remove the extra blank line here when you update

zhzhan · 2015-04-10T02:28:05Z

@rxin This is the complete change for offheap api, and tachyon migration code. There is no any logical change in tachyong (just move code around), except changing one system.ext to throw exceptions in TachyonBlockManager initialization part.

SparkQA · 2015-04-10T02:39:38Z

Test build #29997 has finished for PR 5430 at commit e4814df.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-10T08:28:17Z

Test build #30017 has finished for PR 5430 at commit bf058bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

rxin · 2015-04-10T20:23:37Z

core/src/main/scala/org/apache/spark/SparkContext.scala

  // Add a timestamp as the suffix here to make it more safe
-  val tachyonFolderName = "spark-" + randomUUID.toString()
-  conf.set("spark.tachyonStore.folderName", tachyonFolderName)
+  val offHeapFolderName = "spark-" + randomUUID.toString()


This is an API breaking change. We need to keep the old one around (as an alias), and deprecate it.

rxin · 2015-04-10T20:49:59Z

@zhzhan I also left a high level comment on JIRA - it'd be better to call this external block store, rather than off-heap store.

SparkQA · 2015-04-14T00:56:47Z

Test build #30203 has finished for PR 5430 at commit 0a49021.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zhzhan · 2015-04-20T22:29:25Z

@rxin Could you help to review the patch and let me know if you have any concern?

SparkQA · 2015-04-20T22:54:14Z

Test build #30609 has finished for PR 5430 at commit d1c9921.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

SparkQA · 2015-04-21T02:07:41Z

Test build #30614 has finished for PR 5430 at commit 3ea0689.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

rxin · 2015-04-22T20:38:04Z

Sorry for the delay - I will look at this again today.

pwendell · 2015-04-22T21:29:13Z

core/src/main/scala/org/apache/spark/util/JsonProtocol.scala

  }

  def blockStatusFromJson(json: JValue): BlockStatus = {
    val storageLevel = storageLevelFromJson(json \ "Storage Level")
    val memorySize = (json \ "Memory Size").extract[Long]
    val diskSize = (json \ "Disk Size").extract[Long]
-    val tachyonSize = (json \ "Tachyon Size").extract[Long]
-    BlockStatus(storageLevel, memorySize, diskSize, tachyonSize)
+    val extBlkStoreSize = (json \ "ExtBlkStore Size").extract[Long]


This needs to handle backwards compatibility - i.e. logs from older versions of Spark where it says Tachyon. If you look there are other backwards compatibility tests relating to this protocol too

If the "Tachyon" version is present, I'd look for that and then just convert it.

rxin · 2015-04-23T00:38:29Z

core/src/main/scala/org/apache/spark/storage/ExtBlockManager.scala

+   * desc for the implementation.
+   *
+   */
+  def desc(): String = {"External Block Store"}


how is desc going to be used? maybe we should just override toString?

zhzhan · 2015-04-25T03:17:52Z

Jenkins, retest this please.

zhzhan · 2015-04-25T17:06:47Z

Jenkins, test it please.

pwendell · 2015-04-25T18:41:26Z

Jenkins, test this please.

zhzhan · 2015-04-27T17:29:49Z

Jenkins, test this please.

SparkQA · 2015-04-28T22:23:46Z

Test build #31175 has finished for PR 5430 at commit a80d50c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zhzhan · 2015-04-29T00:39:42Z

Jenkins, retest this please.

SparkQA · 2015-04-29T02:18:54Z

Test build #31197 has finished for PR 5430 at commit a80d50c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zhzhan · 2015-04-29T20:50:22Z

@rxin @pwendell Can you help review the new patch, and let me know if you have any concern?

pwendell · 2015-05-01T00:04:52Z

I took as pass and this LGTM. However it needs to be brought up to date.

zhzhan · 2015-05-01T02:51:27Z

I think Spark-5213 fail the mina test

#4015

[info] spark-sql: found 1 potential binary incompatibilities (filtered 129)
[error] * method sqlParser()org.apache.spark.sql.SparkSQLParser in class org.apache.spark.sql.SQLContext does not have a correspondent in new version
[error] filter with: ProblemFilters.excludeMissingMethodProblem

SparkQA · 2015-05-01T02:56:26Z

Test build #31502 has finished for PR 5430 at commit 60acd84.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

zhzhan · 2015-05-01T03:12:32Z

@marmbrus master failure with mima test. Probably caused by #4015

pwendell · 2015-05-01T03:33:56Z

Jenkins, retest this please. Thanks @zhzhan I reverted the patch.

SparkQA · 2015-05-01T05:14:36Z

Test build #31506 has finished for PR 5430 at commit 60acd84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

This is the classes for creating off-heap block storage API. It also includes the migration for Tachyon. The diff seems to be big, but it mainly just rename tachyon to offheap. New implementation for hdfs will be submit for review in spark-6112. Author: Zhan Zhang <[email protected]> Closes apache#5430 from zhzhan/SPARK-6479 and squashes the following commits: 60acd84 [Zhan Zhang] minor change to kickoff the test 12f54c9 [Zhan Zhang] solve merge conflicts a54132c [Zhan Zhang] solve review comments ffb8e00 [Zhan Zhang] rebase to sparkcontext change 6e121e0 [Zhan Zhang] resolve review comments and restructure blockmanasger code a7aed6c [Zhan Zhang] add Tachyon migration code 186de31 [Zhan Zhang] initial commit for off-heap block storage api

zhzhan changed the title ~~[SPARK-6479][SPARK-CORE]Create off-heap block storage API~~ [SPARK-6479][Block Manager]Create off-heap block storage API Apr 8, 2015

viper-kun reviewed Apr 9, 2015
View reviewed changes

rxin reviewed Apr 9, 2015
View reviewed changes

zhzhan force-pushed the SPARK-6479 branch from e4814df to bf058bb Compare April 10, 2015 06:58

rxin reviewed Apr 10, 2015
View reviewed changes

zhzhan force-pushed the SPARK-6479 branch from 0a49021 to d1c9921 Compare April 20, 2015 22:28

pwendell reviewed Apr 22, 2015
View reviewed changes

rxin reviewed Apr 23, 2015
View reviewed changes

zhzhan force-pushed the SPARK-6479 branch from ed43aa9 to e6ea2f9 Compare April 25, 2015 02:45

zhzhan force-pushed the SPARK-6479 branch from e6ea2f9 to a80d50c Compare April 28, 2015 20:56

zhzhan added 7 commits April 30, 2015 18:58

initial commit for off-heap block storage api

186de31

add Tachyon migration code

a7aed6c

resolve review comments and restructure blockmanasger code

6e121e0

rebase to sparkcontext change

ffb8e00

solve review comments

a54132c

solve merge conflicts

12f54c9

minor change to kickoff the test

60acd84

zhzhan force-pushed the SPARK-6479 branch from a80d50c to 60acd84 Compare May 1, 2015 02:39

asfgit closed this in 36a7a68 May 1, 2015

JoshRosen mentioned this pull request Jun 30, 2022

[SPARK-39636][CORE][UI] Fix multiple bugs in JsonProtocol, impacting off heap StorageLevels and Task/Executor ResourceRequests #37027

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-6479][Block Manager]Create off-heap block storage API #5430

[SPARK-6479][Block Manager]Create off-heap block storage API #5430

zhzhan commented Apr 8, 2015

rxin commented Apr 9, 2015

SparkQA commented Apr 9, 2015

zhzhan commented Apr 9, 2015

viper-kun Apr 9, 2015

zhzhan Apr 9, 2015

rxin commented Apr 9, 2015

rxin Apr 9, 2015

zhzhan commented Apr 10, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

rxin Apr 10, 2015

rxin commented Apr 10, 2015

SparkQA commented Apr 14, 2015

zhzhan commented Apr 20, 2015

SparkQA commented Apr 20, 2015

SparkQA commented Apr 21, 2015

rxin commented Apr 22, 2015

pwendell Apr 22, 2015

pwendell Apr 22, 2015

rxin Apr 23, 2015

zhzhan commented Apr 25, 2015

zhzhan commented Apr 25, 2015

pwendell commented Apr 25, 2015

zhzhan commented Apr 27, 2015

SparkQA commented Apr 28, 2015

zhzhan commented Apr 29, 2015

SparkQA commented Apr 29, 2015

zhzhan commented Apr 29, 2015

pwendell commented May 1, 2015

zhzhan commented May 1, 2015

SparkQA commented May 1, 2015

zhzhan commented May 1, 2015

pwendell commented May 1, 2015

SparkQA commented May 1, 2015

[SPARK-6479][Block Manager]Create off-heap block storage API #5430

[SPARK-6479][Block Manager]Create off-heap block storage API #5430

Conversation

zhzhan commented Apr 8, 2015

rxin commented Apr 9, 2015

SparkQA commented Apr 9, 2015

zhzhan commented Apr 9, 2015

viper-kun Apr 9, 2015

Choose a reason for hiding this comment

zhzhan Apr 9, 2015

Choose a reason for hiding this comment

rxin commented Apr 9, 2015

rxin Apr 9, 2015

Choose a reason for hiding this comment

zhzhan commented Apr 10, 2015

SparkQA commented Apr 10, 2015

SparkQA commented Apr 10, 2015

rxin Apr 10, 2015

Choose a reason for hiding this comment

rxin commented Apr 10, 2015

SparkQA commented Apr 14, 2015

zhzhan commented Apr 20, 2015

SparkQA commented Apr 20, 2015

SparkQA commented Apr 21, 2015

rxin commented Apr 22, 2015

pwendell Apr 22, 2015

Choose a reason for hiding this comment

pwendell Apr 22, 2015

Choose a reason for hiding this comment

rxin Apr 23, 2015

Choose a reason for hiding this comment

zhzhan commented Apr 25, 2015

zhzhan commented Apr 25, 2015

pwendell commented Apr 25, 2015

zhzhan commented Apr 27, 2015

SparkQA commented Apr 28, 2015

zhzhan commented Apr 29, 2015

SparkQA commented Apr 29, 2015

zhzhan commented Apr 29, 2015

pwendell commented May 1, 2015

zhzhan commented May 1, 2015

SparkQA commented May 1, 2015

zhzhan commented May 1, 2015

pwendell commented May 1, 2015

SparkQA commented May 1, 2015