GH-41952: [R] Turn S3 and ZSTD on by default for macOS (#42210)
conbench-apache-arrow / Conbench performance report
failed
Jun 24, 2024 in 0s
Found 4 regressions
After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 67bbf84.
Benchmarks with performance regressions
There were 4 possible performance regressions, according to the lookback z-score method.
Benchmarks with regressions:
- Commit Run on
ec2-m5-4xlarge-us-east-2at 2024-06-23 23:12:34Zdataframe-to-table(R) with dataset=type_strings, language=Rpartitioned-dataset-filter(R) with dataset=dataset-taxi-parquet, language=R, query=vignettefile-read(Python) with compression=lz4, dataset=nyctaxi_2010-01, file_type=feather, output_type=dataframetpch(R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-03, scale_factor=1
All benchmark runs analyzed:
- Commit Run on
ursa-thinkcentre-m75qat 2024-06-23 22:18:33Z - Commit Run on
arm64-m6g-linux-computeat 2024-06-23 22:16:37Z - Commit Run on
ec2-c6a-4xlarge-us-east-2at 2024-06-23 22:24:16Z - Commit Run on
ec2-r5-4xlarge-us-east-2at 2024-06-23 23:12:47Z - Commit Run on
test-mac-armat 2024-06-23 22:27:19Z - Commit Run on
arm64-t4g-linux-computeat 2024-06-23 22:52:52Z - Commit Run on
ec2-m5-4xlarge-us-east-2at 2024-06-23 23:12:34Z
Unstable benchmarks with performance regressions
The following benchmark results indicate a possible performance regression, but are known to sometimes produce false positives when applying the lookback z-score analysis.
-
Commit Run on
test-mac-armat 2024-06-23 22:27:19Z -
Commit Run on
arm64-t4g-linux-computeat 2024-06-23 22:52:52ZErrorSchemeException(C++) with source=cpp-micro, suite=arrow-type-benchmarkErrorSchemeStatus(C++) with source=cpp-micro, suite=arrow-type-benchmarkErrorSchemeNoError(C++) with source=cpp-micro, suite=arrow-type-benchmarkSchemaEqualsWithMetadata(C++) with source=cpp-micro, suite=arrow-type-benchmarkErrorSchemeResult(C++) with source=cpp-micro, suite=arrow-type-benchmark
-
Commit Run on
ec2-m5-4xlarge-us-east-2at 2024-06-23 23:12:34ZShortVectorInsertAtEnd(C++) with params=<STD_VECTOR(std::shared_ptr)>, source=cpp-micro, suite=arrow-small-vector-benchmarkArrayRankInt64Narrow(C++) with params=1048576/100/tiebreaker:3, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkBenchmarkBitmapVisitBitsetAnd(C++) with params=32768/2, source=cpp-micro, suite=arrow-bit-util-benchmarkShortVectorInsert(C++) with params=<SMALL_VECTOR(std::string)>, source=cpp-micro, suite=arrow-small-vector-benchmarkMemoryBandwidth(C++) with params=/33554432/real_time/threads:1, source=cpp-micro, suite=arrow-io-memory-benchmarkAllocateDeallocate(C++) with params=/size:1048576/real_time/threads:2, source=cpp-micro, suite=arrow-memory-pool-benchmarkBenchmarkBitmapVisitUInt64And(C++) with params=32768/1, source=cpp-micro, suite=arrow-bit-util-benchmarkRecordBatchSortIndicesInt64Wide(C++) with params=1048576/0/8, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkFilterInt64FilterNoNulls(C++) with params=1048576/11, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkListSliceStringListWithStop(C++) with params=32768/1, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkBM_RleEncoding(C++) with params=4096/8, source=cpp-micro, suite=parquet-column-io-benchmarkRecordBatchSortIndicesInt64Wide(C++) with params=1048576/0/16, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkRoundArrayBenchmark(C++) with params=<Round, Int32Type, RoundMode::HALF_TOWARDS_ZERO>/size:1048576/inverse_null_proportion:100, source=cpp-micro, suite=arrow-compute-scalar-round-benchmarkBM_DeltaDecodingByteArray(C++) with params=max-string-length:8/batch-size:512/prefixed-percent:10, source=cpp-micro, suite=parquet-encoding-benchmarkListSliceStringList(C++) with params=32768/100, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkBM_PlainDecodingSpacedBoolean(C++) with params=32768/100, source=cpp-micro, suite=parquet-encoding-benchmarkFilterRecordBatchNoNulls(C++) with params=10/1, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkExecuteScalarExpressionOverhead(C++) with params=complex_expression/rows_per_batch:10000/real_time/threads:1, source=cpp-micro, suite=arrow-acero-expression-benchmarkConstructFromShortStdVector(C++) with params=<STD_VECTOR(std::shared_ptr)>, source=cpp-micro, suite=arrow-small-vector-benchmarkArraySortIndicesBool(C++) with params=32768/1, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkFloatStridedTensorConversionFixture(C++) with params=/ConvertToSparseCOOTensorInt64, source=cpp-micro, suite=arrow-tensor-conversion-benchmarkFilterFixedSizeBinaryFilterWithNulls(C++) with params=1048576/2/8, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkArrayRangeEqualsFixedSizeBinary(C++) with params=32768/0, source=cpp-micro, suite=arrow-compare-benchmarkMemoryBandwidth(C++) with params=/33554432/real_time/threads:8, source=cpp-micro, suite=arrow-io-memory-benchmarkExecuteScalarExpressionOverhead(C++) with params=ref_only_expression/rows_per_batch:10000/real_time/threads:1, source=cpp-micro, suite=arrow-acero-expression-benchmarkBM_ReadOffsetIndex(C++) with params=num_pages:1024, source=cpp-micro, suite=parquet-page-index-benchmarkBufferedOutputStreamLargeWritesToPipe(C++) with params=real_time, source=cpp-micro, suite=arrow-io-file-benchmarkTouchArea(C++) with params=size:65536/real_time/threads:8, source=cpp-micro, suite=arrow-memory-pool-benchmarkListSliceStringListWithStop(C++) with params=32768/2, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkRecordBatchSortIndicesInt64Wide(C++) with params=1048576/4/8, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkArrayRankStringNarrow(C++) with params=1048576/100/tiebreaker:2, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkGrouperWithMultiTypes(C++) with params="{utf8, int32, int64, fixed_size_binary(32), boolean}"/4096/10000, source=cpp-micro, suite=arrow-compute-grouper-benchmarkThreadedTaskGroup(C++) with params=threads:4/task_cost:10000/real_time, source=cpp-micro, suite=arrow-thread-pool-benchmarkRecordBatchSortIndicesInt64Narrow(C++) with params=1048576/100/8, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkDoubleRowMajorMatrixConversionFixture(C++) with params=/ConvertToSparseCSRMatrixInt32, source=cpp-micro, suite=arrow-tensor-conversion-benchmarkIsInStringLargeSet(C++) with source=cpp-micro, suite=arrow-compute-scalar-set-lookup-benchmarkChunkedArraySortIndicesInt64Narrow(C++) with params=1048576/100, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkget Vector(JavaScript) with from=float32Array, source=js-microListSliceStringList(C++) with params=32768/1, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkArrayRankInt64Wide(C++) with params=32768/100/tiebreaker:0, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkBinaryBitBlockCounterSum(C++) with params=4096, source=cpp-micro, suite=arrow-bit-block-counter-benchmarkListSliceStringList(C++) with params=32768/10000, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkBM_RleDecoding(C++) with params=4096/8, source=cpp-micro, suite=parquet-column-io-benchmarkArraySortIndicesInt64WideDict(C++) with params=32768/1, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkFloatColumnMajorTensorConversionFixture(C++) with params=/ConvertToSparseCOOTensorInt64, source=cpp-micro, suite=arrow-tensor-conversion-benchmarkCheckIndexBoundsInt32(C++) with params=32768/0, source=cpp-micro, suite=arrow-int-util-benchmarkRecordReaderReadAndSkipRecords(C++) with params=Repetition:2/BatchSize:100/LevelsPerPage:80000, source=cpp-micro, suite=parquet-column-reader-benchmarkFilterInt64FilterWithNulls(C++) with params=1048576/2, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkParseJSONFields(C++) with params=ordered:1/schema:0/sparsity:0/num_fields:100, source=cpp-micro, suite=arrow-json-parser-benchmarkFilterRecordBatchWithNulls(C++) with params=50/12, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkFilterFixedSizeBinaryFilterWithNulls(C++) with params=1048576/1/8, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkFilterFixedSizeBinaryFilterWithNulls(C++) with params=1048576/6/8, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkTableSortIndicesInt64Narrow(C++) with params=1048576/100/16/4, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkUniqueUInt8(C++) with params=1, source=cpp-micro, suite=arrow-compute-vector-hash-benchmarkListSliceStringListWithStepAndStop(C++) with params=32768/100, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkArraySortIndicesBool(C++) with params=32768/100, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkTouchArea(C++) with params=size:1048576/real_time/threads:8, source=cpp-micro, suite=arrow-memory-pool-benchmarkAllocateTouchDeallocate(C++) with params=/size:16777216/real_time/threads:2, source=cpp-micro, suite=arrow-memory-pool-benchmarkReferenceDecompression(C++) with params=Compression::LZ4, source=cpp-micro, suite=arrow-compression-benchmarkChunkedArrayRankInt64Narrow(C++) with params=1048576/100/tiebreaker:0, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkBM_RleEncodingBoolean(C++) with params=32768, source=cpp-micro, suite=parquet-encoding-benchmarkReverseSetBitRunReader(C++) with params=60, source=cpp-micro, suite=arrow-bit-util-benchmarkFilterRecordBatchWithNulls(C++) with params=100/0, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkParallelMemoryCopy(C++) with params=threads:4/real_time, source=cpp-micro, suite=arrow-io-memory-benchmarkListSliceStringListWithStepAndStop(C++) with params=32768/1, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkTouchArea(C++) with params=size:4096/real_time/threads:8, source=cpp-micro, suite=arrow-memory-pool-benchmarkBenchmarkBitmapVisitBitsetAnd(C++) with params=32768/1, source=cpp-micro, suite=arrow-bit-util-benchmarkAsciiUpper(C++) with source=cpp-micro, suite=arrow-compute-scalar-string-benchmarkArraySortIndicesBool(C++) with params=32768/2, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkFilterFixedSizeBinaryFilterWithNulls(C++) with params=1048576/14/9, source=cpp-micro, suite=arrow-compute-vector-selection-benchmarkBM_RleEncodingBoolean(C++) with params=65536, source=cpp-micro, suite=parquet-encoding-benchmarkFloatRowMajorTensorConversionFixture(C++) with params=/ConvertToSparseCOOTensorInt32, source=cpp-micro, suite=arrow-tensor-conversion-benchmarkChunkJSONPrettyPrinted(C++) with source=cpp-micro, suite=arrow-json-parser-benchmarkArrayRankInt64Narrow(C++) with params=1048576/100/tiebreaker:0, source=cpp-micro, suite=arrow-compute-vector-sort-benchmarkValidateLargeNonAscii(C++) with source=cpp-micro, suite=arrow-utf8-util-benchmarkBenchmarkBitmapVisitUInt64And(C++) with params=32768/2, source=cpp-micro, suite=arrow-bit-util-benchmarkReadMmapUncachedFile(C++) with params=num_cols:8/is_partial:1/real_time, source=cpp-micro, suite=arrow-ipc-read-write-benchmarkThreadPoolSubmit(C++) with params=threads:8/task_cost:1000/real_time, source=cpp-micro, suite=arrow-thread-pool-benchmarkListSliceStringListWithStepAndStop(C++) with params=32768/2, source=cpp-micro, suite=arrow-compute-scalar-list-benchmarkSchemaEqualsWithMetadata(C++) with source=cpp-micro, suite=arrow-type-benchmarkParseJSONFields(C++) with params=ordered:1/schema:1/sparsity:10/num_fields:1000, source=cpp-micro, suite=arrow-json-parser-benchmark
Details
This report was generated using the lookback z-score method with a z-score threshold of 30.0.
Loading