Skip to content

Commit

Permalink
[SPARK-26082][MESOS][FOLLOWUP] Add UT on fetcher cache option on Meso…
Browse files Browse the repository at this point in the history
…sClusterScheduler

## What changes were proposed in this pull request?

This patch adds UT on testing SPARK-26082 to avoid regression. While apache#23743 reduces the possibility to make a similar mistake, the needed lines of code for adding tests are not that huge, so I guess it might be worth to add them.

## How was this patch tested?

Newly added UTs. Test "supports setting fetcher cache" fails when apache#23743 is not applied and succeeds when apache#23743 is applied.

Closes apache#23744 from HeartSaVioR/SPARK-26082-add-unit-test.

Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit b4e1d14)
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
HeartSaVioR authored and dongjoon-hyun committed Feb 7, 2019
1 parent bb6dbd2 commit 3abf45d
Showing 1 changed file with 48 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,54 @@ class MesosClusterSchedulerSuite extends SparkFunSuite with LocalSparkContext wi
assert(networkInfos.get(0).getLabels.getLabels(1).getValue == "val2")
}

test("supports setting fetcher cache") {
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "true"))

val mem = 1000
val cpu = 1

val response = scheduler.submitDriver(
new MesosDriverDescription("d1", "jar", mem, cpu, true,
command,
Map("spark.mesos.executor.home" -> "test",
"spark.app.name" -> "test"),
"s1",
new Date()))

assert(response.success)

val offer = Utils.createOffer("o1", "s1", mem, cpu)
scheduler.resourceOffers(driver, List(offer).asJava)

val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
val uris = launchedTasks.head.getCommand.getUrisList
assert(uris.stream().allMatch(_.getCache))
}

test("supports disabling fetcher cache") {
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "false"))

val mem = 1000
val cpu = 1

val response = scheduler.submitDriver(
new MesosDriverDescription("d1", "jar", mem, cpu, true,
command,
Map("spark.mesos.executor.home" -> "test",
"spark.app.name" -> "test"),
"s1",
new Date()))

assert(response.success)

val offer = Utils.createOffer("o1", "s1", mem, cpu)
scheduler.resourceOffers(driver, List(offer).asJava)

val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
val uris = launchedTasks.head.getCommand.getUrisList
assert(uris.stream().allMatch(!_.getCache))
}

test("accept/decline offers with driver constraints") {
setScheduler()

Expand Down

0 comments on commit 3abf45d

Please sign in to comment.