Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

java.nio.BufferOverflowException native jdbc version 2.5.6. #354

Open
gj-zhang opened this issue Jul 13, 2021 · 5 comments
Open

java.nio.BufferOverflowException native jdbc version 2.5.6. #354

gj-zhang opened this issue Jul 13, 2021 · 5 comments
Assignees
@pan3793

Comments

@gj-zhang
Copy link

gj-zhang commented Jul 13, 2021 •
edited
Loading

Environment

  • OS version:
  • JDK version: 1.8
  • ClickHouse Server version: 20.9.3.45
  • ClickHouse Native JDBC version: 2.5.6
  • (Optional) Spark version: N/A
  • (Optional) Other components' version: flink 1.12

Error logs

       java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:521)
	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
	at com.github.housepower.buffer.ByteArrayWriter.writeBinary(ByteArrayWriter.java:37)
	at com.github.housepower.serde.BinarySerializer.writeInt(BinarySerializer.java:76)
	at com.github.housepower.data.type.complex.DataTypeDateTime.serializeBinary(DataTypeDateTime.java:112)
	at com.github.housepower.data.type.complex.DataTypeDateTime.serializeBinary(DataTypeDateTime.java:33)
	at com.github.housepower.data.Column.write(Column.java:31)
	at com.github.housepower.data.Block.appendRow(Block.java:93)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addParameters(ClickHousePreparedInsertStatement.java:190)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addBatch(ClickHousePreparedInsertStatement.java:121)
	at org.apache.flink.connector.clickhouse.internal.executor.ClickHouseBatchExecutor.executeBatch(ClickHouseBatchExecutor.java:134)
	at org.apache.flink.connector.clickhouse.internal.format.ClickHouseBatchOutputFormat.flush(ClickHouseBatchOutputFormat.java:107)
	at org.apache.flink.connector.clickhouse.internal.format.ClickHouseBatchOutputFormat.writeRecord(ClickHouseBatchOutputFormat.java:69)
	at org.apache.flink.connector.clickhouse.internal.format.ClickHouseBatchOutputFormat.writeRecord(ClickHouseBatchOutputFormat.java:24)
	at org.apache.flink.connector.clickhouse.internal.ClickHouseGenericSinkFunction.invoke(ClickHouseGenericSinkFunction.java:38)
	at org.apache.flink.connector.clickhouse.internal.ClickHouseGenericSinkFunction.invoke(ClickHouseGenericSinkFunction.java:18)
	at org.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:72)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:187)
	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:204)
	at org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:174)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:395)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:191)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:609)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: java.nio.BufferOverflowException
		at java.nio.Buffer.nextPutIndex(Buffer.java:521)
		at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
		at com.github.housepower.buffer.ByteArrayWriter.writeBinary(ByteArrayWriter.java:37)
		at com.github.housepower.serde.BinarySerializer.writeInt(BinarySerializer.java:76)
		at com.github.housepower.data.type.complex.DataTypeDateTime.serializeBinary(DataTypeDateTime.java:112)
		at com.github.housepower.data.type.complex.DataTypeDateTime.serializeBinary(DataTypeDateTime.java:33)
		at com.github.housepower.data.Column.write(Column.java:31)
		at com.github.housepower.data.Block.appendRow(Block.java:93)
		at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addParameters(ClickHousePreparedInsertStatement.java:190)
		at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.addBatch(ClickHousePreparedInsertStatement.java:121)
		at org.apache.flink.connector.clickhouse.internal.executor.ClickHouseBatchExecutor.executeBatch(ClickHouseBatchExecutor.java:134)
		at org.apache.flink.connector.clickhouse.internal.format.ClickHouseBatchOutputFormat.flush(ClickHouseBatchOutputFormat.java:107)
		at org.apache.flink.connector.clickhouse.internal.format.ClickHouseBatchOutputFormat.close(ClickHouseBatchOutputFormat.java:83)
		at org.apache.flink.connector.clickhouse.internal.ClickHouseGenericSinkFunction.close(ClickHouseGenericSinkFunction.java:55)
		at org.apache.flink.api.common.functions.util.FunctionUtils.closeFunction(FunctionUtils.java:41)
		at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:117)
		at org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:783)
		at org.apache.flink.streaming.runtime.tasks.StreamTask.runAndSuppressThrowable(StreamTask.java:762)
		at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:681)
		at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:585)

Steps to reproduce

use flink-sql-connector-clickhouse implement by ourself. write batch size 200000.

Other descriptions
@gj-zhang gj-zhang changed the title java.nio.BufferOverflowException java.nio.BufferOverflowException native jdbc version 2.5.6. Jul 14, 2021
@gj-zhang
Copy link
Author

gj-zhang commented Aug 4, 2021

anybody take a look?

@gj-zhang
Copy link
Author

gj-zhang commented Aug 4, 2021 •
edited by pan3793
Loading 

java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:521)
	at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:169)
	at com.github.housepower.buffer.ByteArrayWriter.writeBinary(ByteArrayWriter.java:37)
	at com.github.housepower.serde.BinarySerializer.writeVarInt(BinarySerializer.java:49)
	at com.github.housepower.serde.BinarySerializer.writeBytesBinary(BinarySerializer.java:107)
	at com.github.housepower.serde.BinarySerializer.writeStringBinary(BinarySerializer.java:103)
	at com.github.housepower.data.type.complex.DataTypeString.serializeBinary(DataTypeString.java:77)
	at com.github.housepower.data.Column.write(Column.java:31)
	at com.github.housepower.data.Block.appendRow(Block.java:92)
	at com.github.housepower.statement.ClickHousePreparedInsertStatement.addParameters(ClickHousePreparedInsertStatement.java:162)
	at com.github.housepower.statement.ClickHousePreparedInsertStatement.addBatch(ClickHousePreparedInsertStatement.java:95)

@pan3793 pan3793 added the bug Something isn't working label Aug 5, 2021
@pan3793
Copy link
Member

pan3793 commented Aug 5, 2021 •
edited
Loading

Thanks for the report, it's a bug, we haven't checked the remaining capacity of the buffer.
Have a second look, the buffer should always have remaining capacity before writing because we do the check after every write. And the stack looks weird, are you sure using v2.5.6? The stackstrace and line do not match the codebase.

@pan3793 pan3793 self-assigned this Aug 5, 2021
@pan3793 pan3793 removed the bug Something isn't working label Aug 5, 2021
@gj-zhang
Copy link
Author

gj-zhang commented Aug 5, 2021 •
edited
Loading

are you sure using v2.5.6

yes

The stackstrace and line do not match the codebase.

I changed the package name and wrapped the custom stuff around it. But the core code is not changed

@pan3793
Copy link
Member

pan3793 commented Aug 5, 2021

I changed the package name and wrapped the custom stuff around it.

Not sure if it is related, but I cannot reproduce the issue, please provide the reproducible test case to help dig it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pan3793 pan3793

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gj-zhang @pan3793