Skip to content

Commit

Permalink
[SPARK-35224][SQL][TESTS] Fix buffer overflow in `MutableProjectionSu…
Browse files Browse the repository at this point in the history
…ite`

### What changes were proposed in this pull request?
In the test `"unsafe buffer with NO_CODEGEN"` of `MutableProjectionSuite`, fix unsafe buffer size calculation to be able to place all input fields without buffer overflow + meta-data.

### Why are the changes needed?
To make the test suite `MutableProjectionSuite` more stable.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
By running the affected test suite:
```
$ build/sbt "test:testOnly *MutableProjectionSuite"
```

Closes apache#32339 from MaxGekk/fix-buffer-overflow-MutableProjectionSuite.

Authored-by: Max Gekk <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
  • Loading branch information
MaxGekk committed Apr 26, 2021
1 parent 38ef477 commit d572a85
Showing 1 changed file with 5 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow}
import org.apache.spark.sql.catalyst.util.IntervalUtils
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.Platform
import org.apache.spark.unsafe.types.UTF8String

class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
Expand Down Expand Up @@ -50,8 +51,10 @@ class MutableProjectionSuite extends SparkFunSuite with ExpressionEvalHelper {
testBothCodegenAndInterpreted("unsafe buffer") {
val inputRow = InternalRow.fromSeq(Seq(
false, 1.toByte, 9.toShort, -18, 53L, 3.2f, 7.8, 4, 9L, Int.MinValue, Long.MaxValue))
val numBytes = UnsafeRow.calculateBitSetWidthInBytes(fixedLengthTypes.length)
val unsafeBuffer = UnsafeRow.createFromByteArray(numBytes, fixedLengthTypes.length)
val numFields = fixedLengthTypes.length
val numBytes = Platform.BYTE_ARRAY_OFFSET + UnsafeRow.calculateBitSetWidthInBytes(numFields) +
UnsafeRow.WORD_SIZE * numFields
val unsafeBuffer = UnsafeRow.createFromByteArray(numBytes, numFields)
val proj = createMutableProjection(fixedLengthTypes)
val projUnsafeRow = proj.target(unsafeBuffer)(inputRow)
assert(SafeProjection.create(fixedLengthTypes)(projUnsafeRow) === inputRow)
Expand Down

0 comments on commit d572a85

Please sign in to comment.