ORC-1065: Fix IndexOutOfBoundsException in ReaderImpl.extractFileTail

[cxzl25]

What changes were proposed in this pull request?

Use buffer limit as readSize to avoid IndexOutOfBoundsException.

main

orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java

Lines 720 to 725 in 3a2cb60

	public static OrcTail extractFileTail(ByteBuffer buffer, long fileLen, long modificationTime)
	throws IOException {
	OrcProto.PostScript ps;
	long readSize = fileLen != -1 ? fileLen : buffer.limit();
	OrcProto.FileTail.Builder fileTailBuilder = OrcProto.FileTail.newBuilder();
	fileTailBuilder.setFileLength(readSize);

branch-1.5

orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java

Lines 487 to 490 in 5f88704

	public static OrcTail extractFileTail(ByteBuffer buffer, long fileLength, long modificationTime)
	throws IOException {
	int readSize = buffer.limit();
	int psLen = buffer.get(readSize - 1) & 0xff;

Why are the changes needed?

ORC-251 remove ReaderImpl.extractFileTail
ORC-685 Add ReaderImpl.extractFileTail back

In ORC-685, file length is used as readsize, which causes that if the buffer is read from the cache, the use of length is incorrect, resulting in IndexOutOfBoundsException.

long readSize = fileLen != -1? fileLen: buffer.limit();
int psLen = buffer.get((int) (readSize-1)) & 0xff; 

Caused by: java.lang.IndexOutOfBoundsException
    at java.nio.Buffer.checkIndex(Buffer.java:540)
    at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
    at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:726)
    at org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:798)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:916)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:885)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1759)
    at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1703) 

How was this patch tested?

local test

[cxzl25]

main branch

orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java

Lines 720 to 725 in 3a2cb60

	public static OrcTail extractFileTail(ByteBuffer buffer, long fileLen, long modificationTime)
	throws IOException {
	OrcProto.PostScript ps;
	long readSize = fileLen != -1 ? fileLen : buffer.limit();
	OrcProto.FileTail.Builder fileTailBuilder = OrcProto.FileTail.newBuilder();
	fileTailBuilder.setFileLength(readSize);

branch-1.5

orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java

Lines 487 to 490 in 5f88704

	public static OrcTail extractFileTail(ByteBuffer buffer, long fileLength, long modificationTime)
	throws IOException {
	int readSize = buffer.limit();
	int psLen = buffer.get(readSize - 1) & 0xff;

[cxzl25]

We used Spark 3.2.0, Hive2.3.9, Orc 1.6.11,
Set spark.sql.hive.convertMetastoreOrc=false in spark, and querying a table triggers this problem for the second time.

The current workaround is to add configuration in hive-site.xml

  <property>
    <name>hive.orc.cache.stripe.details.mem.size</name>
    <value>0</value>
  </property>

    HIVE_ORC_CACHE_STRIPE_DETAILS_MEMORY_SIZE("hive.orc.cache.stripe.details.mem.size", "256Mb",
        new SizeValidator(), "Maximum size of orc splits cached in the client."),

[cxzl25]

cc @pgaref @omalley @dongjoon-hyun

[dongjoon-hyun]

It seems that the PR doesn't pass the UTs. Could you check the UT failures?

Error:  Failures: 
Error:    TestReaderImpl.testOrcTailStripeStats:382 expected: <1980> but was: <-417>

[dongjoon-hyun]

Could you add a test case for your code, @cxzl25 ?

[cxzl25]

Could you add a test case for your code, @cxzl25 ?

ok, let me see how to add a ut to cover this case.

[dongjoon-hyun]

+1, LGTM. Thank you, @cxzl25 .

[dongjoon-hyun]

cc @pgaref and @williamhyun

[dongjoon-hyun]

@cxzl25 . I added you to the Apache ORC contributor group and assigned ORC-1065 to you. Thank you again.