Skip to content

Commit

Permalink
[SPARK-46799][CORE][TESTS] Improve MasterSuite to use nanoTime-base…
Browse files Browse the repository at this point in the history
…d appIDs and workerIDs

### What changes were proposed in this pull request?

This PR aims to improve `MasterSuite` to use nanoTime-based appIDs and workerIDs.

### Why are the changes needed?

During testing, I hit a case where two workers have the same ID. This PR will prevent the duplicated IDs in Apps and Workers in `MasterSuite`.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

```
$ build/sbt "core/testOnly *.MasterSuite"
[info] MasterSuite:
[info] - can use a custom recovery mode factory (443 milliseconds)
[info] - SPARK-46664: master should recover quickly in case of zero workers and apps (38 milliseconds)
[info] - master correctly recover the application (41 milliseconds)
[info] - SPARK-46205: Recovery with Kryo Serializer (27 milliseconds)
[info] - SPARK-46216: Recovery without compression (19 milliseconds)
[info] - SPARK-46216: Recovery with compression (20 milliseconds)
[info] - SPARK-46258: Recovery with RocksDB (306 milliseconds)
[info] - master/worker web ui available (197 milliseconds)
[info] - master/worker web ui available with reverseProxy (30 seconds, 123 milliseconds)
[info] - master/worker web ui available behind front-end reverseProxy (30 seconds, 113 milliseconds)
[info] - basic scheduling - spread out (23 milliseconds)
[info] - basic scheduling - no spread out (14 milliseconds)
[info] - basic scheduling with more memory - spread out (10 milliseconds)
[info] - basic scheduling with more memory - no spread out (10 milliseconds)
[info] - scheduling with max cores - spread out (9 milliseconds)
[info] - scheduling with max cores - no spread out (9 milliseconds)
[info] - scheduling with cores per executor - spread out (9 milliseconds)
[info] - scheduling with cores per executor - no spread out (8 milliseconds)
[info] - scheduling with cores per executor AND max cores - spread out (8 milliseconds)
[info] - scheduling with cores per executor AND max cores - no spread out (7 milliseconds)
[info] - scheduling with executor limit - spread out (8 milliseconds)
[info] - scheduling with executor limit - no spread out (7 milliseconds)
[info] - scheduling with executor limit AND max cores - spread out (8 milliseconds)
[info] - scheduling with executor limit AND max cores - no spread out (9 milliseconds)
[info] - scheduling with executor limit AND cores per executor - spread out (8 milliseconds)
[info] - scheduling with executor limit AND cores per executor - no spread out (13 milliseconds)
[info] - scheduling with executor limit AND cores per executor AND max cores - spread out (8 milliseconds)
[info] - scheduling with executor limit AND cores per executor AND max cores - no spread out (7 milliseconds)
[info] - scheduling for app with multiple resource profiles (44 milliseconds)
[info] - scheduling for app with multiple resource profiles with max cores (37 milliseconds)
[info] - SPARK-45174: scheduling with max drivers (9 milliseconds)
[info] - SPARK-13604: Master should ask Worker kill unknown executors and drivers (15 milliseconds)
[info] - SPARK-20529: Master should reply the address received from worker (20 milliseconds)
[info] - SPARK-27510: Master should avoid dead loop while launching executor failed in Worker (34 milliseconds)
[info] - All workers on a host should be decommissioned (28 milliseconds)
[info] - No workers should be decommissioned with invalid host (25 milliseconds)
[info] - Only worker on host should be decommissioned (19 milliseconds)
[info] - SPARK-19900: there should be a corresponding driver for the app after relaunching driver (2 seconds, 60 milliseconds)
[info] - assign/recycle resources to/from driver (33 milliseconds)
[info] - assign/recycle resources to/from executor (27 milliseconds)
[info] - resource description with multiple resource profiles (1 millisecond)
[info] - SPARK-45753: Support driver id pattern (7 milliseconds)
[info] - SPARK-45753: Prevent invalid driver id patterns (6 milliseconds)
[info] - SPARK-45754: Support app id pattern (7 milliseconds)
[info] - SPARK-45754: Prevent invalid app id patterns (7 milliseconds)
[info] - SPARK-45785: Rotate app num with modulo operation (114 milliseconds)
[info] - SPARK-45756: Use appName for appId (7 milliseconds)
[info] - SPARK-46353: handleRegisterWorker in STANDBY mode (10 milliseconds)
[info] - SPARK-46353: handleRegisterWorker in RECOVERING mode without workers (7 milliseconds)
[info] - SPARK-46353: handleRegisterWorker in RECOVERING mode with a unknown worker (33 milliseconds)
[info] Run completed in 1 minute, 5 seconds.
[info] Total number of tests run: 50
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 50, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #44839 from dongjoon-hyun/SPARK-46799.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Jan 23, 2024
1 parent 8d12128 commit f9feddf
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -957,7 +957,7 @@ class MasterSuite extends SparkFunSuite

val desc = new ApplicationDescription(
"test", maxCores, null, "", rp, None, None, initialExecutorLimit)
val appId = System.currentTimeMillis.toString
val appId = System.nanoTime().toString
val endpointRef = mock(classOf[RpcEndpointRef])
val mockAddress = mock(classOf[RpcAddress])
when(endpointRef.address).thenReturn(mockAddress)
Expand All @@ -966,7 +966,7 @@ class MasterSuite extends SparkFunSuite
}

private def makeWorkerInfo(memoryMb: Int, cores: Int): WorkerInfo = {
val workerId = System.currentTimeMillis.toString
val workerId = System.nanoTime().toString
val endpointRef = mock(classOf[RpcEndpointRef])
val mockAddress = mock(classOf[RpcAddress])
when(endpointRef.address).thenReturn(mockAddress)
Expand Down

0 comments on commit f9feddf

Please sign in to comment.