`DataException: Invalid type for INT64: class java.lang.Integer` on `connect.default` for int64 Avro type #1740

drnushooz · 2021-08-03T03:41:30Z

There is an issue with Avro 1.9+ where if a field of type long has connect.default property, e.g.

"type": "record"
"name": "sample"
"namespace": "io.apicurio",
"fields": [
  {
    "name": "prop",
    "type": {
      "type": "long",
      "connect.default": 42,
      "connect.version": 1
    }
  }
]

then AvroData.toConnectSchema throws

org.apache.kafka.connect.errors.DataException: Invalid type for INT64: class java.lang.Integer
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1498)
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1230)
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1478)
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1230)
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1226)
	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1205)
	at com.github.drnushooz.kafka.connect.sqs.source.SqsSourceTask.lambda$poll$0(SqsSourceTask.java:138)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.github.drnushooz.kafka.connect.sqs.source.SqsSourceTask.poll(SqsSourceTask.java:149)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.poll(WorkerSourceTask.java:265)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)
	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

The text was updated successfully, but these errors were encountered:

drnushooz · 2021-08-04T21:44:57Z

@EricWittmann would it be possible to triage this and get it prioritized? This is hindering some functionality on our side.

carlesarnal · 2021-08-06T08:06:41Z

Hi @drnushooz, Eric is on PTO, I'll take a look at the linked PR.

drnushooz · 2021-08-06T19:09:04Z

Hi @drnushooz, Eric is on PTO, I'll take a look at the linked PR.

Thank you.

drnushooz · 2021-08-12T00:25:16Z

Hi @drnushooz, Eric is on PTO, I'll take a look at the linked PR.

Is there a tentative schedule for the next maintenance release?

EricWittmann · 2021-08-18T18:35:28Z

Our next upstream release will be 2.1.0. We'll be able to release that soon. Perhaps as early as Friday of this week.

EricWittmann · 2021-08-20T18:14:53Z

Didn't get a chance to plan and execute the release today - I am optimistic it can be done Monday.

EricWittmann · 2021-08-24T17:24:26Z

OK we've done a release of 2.1.0.RC1 - this is a release candidate. We would very much appreciate it if interested parties could give it a try and give us your feedback.

drnushooz · 2021-08-25T16:40:38Z

Thanks, I will pull this release into our dev environment and test it.

EricWittmann · 2021-08-31T17:32:45Z

I will be very interested in your experience, since this is a minor release with some non-trivial changes. Nothing should be breaking, but we may have messed up somewhere along the way. Please report back with any issues and we'll get them fixed before 2.1.0.Final.

HelloSunilSaini · 2022-12-06T10:27:23Z

@EricWittmann @drnushooz
Sill facing this issue have deployed docker image apicurio/apicurio-registry-mem:2.3.1.Final and using io.apicurio.apicurio-registry-distro-connect-converter 2.0.0-SNAPSHOT in kafka-connect sink connector getting error for below field details

"fields": [
            {
              "name": "id",
              "type": {
                "type": "long",
                "connect.default": 0
              },
              "default": 0
            },

ERROR:

 Caused by: org.apache.kafka.connect.errors.DataException: Invalid type for INT64: class java.lang.Integer
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1496)
 	at io.apicurio.registry.utils.converter.avro.AvroData.defaultValueFromAvroWithoutLogical(AvroData.java:1877)
 	at io.apicurio.registry.utils.converter.avro.AvroData.defaultValueFromAvro(AvroData.java:1860)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1793)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchemaWithCycles(AvroData.java:1843)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1691)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1667)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchemaWithCycles(AvroData.java:1843)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1691)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1667)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1523)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1223)
 	at io.apicurio.registry.utils.converter.AvroConverter.toSchemaAndValue(AvroConverter.java:84)
 	at io.apicurio.registry.utils.converter.SchemalessConverter.toConnectData(SchemalessConverter.java:122)
 	at org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)
 	at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$4(WorkerSinkTask.java:516)
 	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
 	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
 	... 13 more

carlesarnal · 2022-12-07T12:35:24Z

That

@EricWittmann @drnushooz Sill facing this issue have deployed docker image apicurio/apicurio-registry-mem:2.3.1.Final and using io.apicurio.apicurio-registry-distro-connect-converter 2.0.0-SNAPSHOT in kafka-connect sink connector getting error for below field details

"fields": [
            {
              "name": "id",
              "type": {
                "type": "long",
                "connect.default": 0
              },
              "default": 0
            },

ERROR:

 Caused by: org.apache.kafka.connect.errors.DataException: Invalid type for INT64: class java.lang.Integer
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1496)
 	at io.apicurio.registry.utils.converter.avro.AvroData.defaultValueFromAvroWithoutLogical(AvroData.java:1877)
 	at io.apicurio.registry.utils.converter.avro.AvroData.defaultValueFromAvro(AvroData.java:1860)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1793)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchemaWithCycles(AvroData.java:1843)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1691)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1667)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchemaWithCycles(AvroData.java:1843)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1691)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1547)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1667)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectSchema(AvroData.java:1523)
 	at io.apicurio.registry.utils.converter.avro.AvroData.toConnectData(AvroData.java:1223)
 	at io.apicurio.registry.utils.converter.AvroConverter.toSchemaAndValue(AvroConverter.java:84)
 	at io.apicurio.registry.utils.converter.SchemalessConverter.toConnectData(SchemalessConverter.java:122)
 	at org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)
 	at org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$4(WorkerSinkTask.java:516)
 	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
 	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
 	... 13 more

This is expected. You're using a converters version without the fix. You need to update from 2.0.0-SNAPSHOT to, at least, 2.1.0.Final.

drnushooz · 2022-12-08T05:57:01Z

@HelloSunilSaini you can find the latest version of the connector on Maven Central https://mvnrepository.com/artifact/io.apicurio/apicurio-registry-distro-connect-converter/2.3.1.Final

drnushooz mentioned this issue Aug 3, 2021

connect.default on long avro field #1738

Merged

drnushooz changed the title ~~org.apache.kafka.connect.errors.DataException: Invalid type for INT64: class java.lang.Integer~~ DataException: Invalid type for INT64: class java.lang.Integer on connect.default for int64 Avro type Aug 3, 2021

carlesarnal closed this as completed in #1738 Aug 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`DataException: Invalid type for INT64: class java.lang.Integer` on `connect.default` for int64 Avro type #1740

`DataException: Invalid type for INT64: class java.lang.Integer` on `connect.default` for int64 Avro type #1740

drnushooz commented Aug 3, 2021 •

edited by EricWittmann

Loading

drnushooz commented Aug 4, 2021

carlesarnal commented Aug 6, 2021 •

edited

Loading

drnushooz commented Aug 6, 2021

drnushooz commented Aug 12, 2021

EricWittmann commented Aug 18, 2021

EricWittmann commented Aug 20, 2021

EricWittmann commented Aug 24, 2021

drnushooz commented Aug 25, 2021 •

edited

Loading

EricWittmann commented Aug 31, 2021

HelloSunilSaini commented Dec 6, 2022 •

edited

Loading

carlesarnal commented Dec 7, 2022

drnushooz commented Dec 8, 2022 •

edited

Loading

DataException: Invalid type for INT64: class java.lang.Integer on connect.default for int64 Avro type #1740

DataException: Invalid type for INT64: class java.lang.Integer on connect.default for int64 Avro type #1740

Comments

drnushooz commented Aug 3, 2021 • edited by EricWittmann Loading

drnushooz commented Aug 4, 2021

carlesarnal commented Aug 6, 2021 • edited Loading

drnushooz commented Aug 6, 2021

drnushooz commented Aug 12, 2021

EricWittmann commented Aug 18, 2021

EricWittmann commented Aug 20, 2021

EricWittmann commented Aug 24, 2021

drnushooz commented Aug 25, 2021 • edited Loading

EricWittmann commented Aug 31, 2021

HelloSunilSaini commented Dec 6, 2022 • edited Loading

carlesarnal commented Dec 7, 2022

drnushooz commented Dec 8, 2022 • edited Loading

`DataException: Invalid type for INT64: class java.lang.Integer` on `connect.default` for int64 Avro type #1740

`DataException: Invalid type for INT64: class java.lang.Integer` on `connect.default` for int64 Avro type #1740

drnushooz commented Aug 3, 2021 •

edited by EricWittmann

Loading

carlesarnal commented Aug 6, 2021 •

edited

Loading

drnushooz commented Aug 25, 2021 •

edited

Loading

HelloSunilSaini commented Dec 6, 2022 •

edited

Loading

drnushooz commented Dec 8, 2022 •

edited

Loading