Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-35308][TESTS] Fix bug in SPARK-35266 that creates benchmark files in invalid path with wrong name #32432

Closed
wants to merge 3 commits into from

Conversation

byungsoo-oh
Copy link

What changes were proposed in this pull request?

This PR fixes a bug in SPARK-35266 that creates benchmark files in the invalid path with the wrong name.
e.g. For BLASBenchmark,

  • AS-IS: Creates benchmarksBLASBenchmark-results.txt in {SPARK_HOME}/mllib-local/
  • TO-BE: Creates BLASBenchmark-results.txt in {SPARK_HOME}/mllib-local/benchmarks/

Why are the changes needed?

As you can see in the above example, new benchmark files cannot be created as intended due to this bug.

Does this PR introduce any user-facing change?

No

How was this patch tested?

After building Spark, manually tested with the following command:

SPARK_GENERATE_BENCHMARK_FILES=1 bin/spark-submit --class \
    org.apache.spark.benchmark.Benchmarks --jars \
    "`find . -name '*-SNAPSHOT-tests.jar' -o -name '*avro*-SNAPSHOT.jar' | paste -sd ',' -`" \
    "`find . -name 'spark-core*-SNAPSHOT-tests.jar'`" \
    "org.apache.spark.ml.linalg.BLASBenchmark"

It successfully generated the benchmark files as intended (BLASBenchmark-results.txt in {SPARK_HOME}/mllib-local/benchmarks/).

@github-actions github-actions bot added the CORE label May 4, 2021
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@HyukjinKwon HyukjinKwon changed the title [SPARK-35308][TESTS] Fix bug in SPARK-35266 that creates benchmark files in invalid path with wrong name [SPARK-35308][FOLLOW-UP][TESTS] Fix bug in SPARK-35266 that creates benchmark files in invalid path with wrong name May 4, 2021
@HyukjinKwon
Copy link
Member

I manually tested. Merged to master.

@HyukjinKwon HyukjinKwon changed the title [SPARK-35308][FOLLOW-UP][TESTS] Fix bug in SPARK-35266 that creates benchmark files in invalid path with wrong name [SPARK-35308][TESTS] Fix bug in SPARK-35266 that creates benchmark files in invalid path with wrong name May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants