Avro converter does not work in Kafka Connect

[jpechane]

When converter is used for the first time it throws

connect_1    | WARNING: An illegal reflective access operation has occurred
connect_1    | WARNING: Illegal reflective access by retrofit2.Platform (file:/kafka/external_libs/apicurio/retrofit-2.9.0.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
connect_1    | WARNING: Please consider reporting this to the maintainers of retrofit2.Platform
connect_1    | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
connect_1    | WARNING: All illegal access operations will be denied in a future release
connect_1    | 2020-09-21 08:27:56,001 ERROR  ||  Failed to start task inventory-connector-0   [org.apache.kafka.connect.runtime.Worker]
connect_1    | java.lang.VerifyError: Bad type on operand stack
connect_1    | Exception Details:
connect_1    |   Location:
connect_1    |     org/apache/avro/Schema.<clinit>()V @39: invokevirtual
connect_1    |   Reason:
connect_1    |     Type 'com/fasterxml/jackson/databind/ObjectMapper' (current frame, stack[1]) is not assignable to 'com/fasterxml/jackson/core/ObjectCodec'
connect_1    |   Current Frame:
connect_1    |     bci: @39
connect_1    |     flags: { }
connect_1    |     locals: { }
connect_1    |     stack: { 'com/fasterxml/jackson/core/JsonFactory', 'com/fasterxml/jackson/databind/ObjectMapper' }
connect_1    |   Bytecode:
connect_1    |     0000000: bb01 2359 b701 24b3 005a bb01 2559 b200
connect_1    |     0000010: 5ab7 0126 b300 f6b2 005a b201 27b6 0128
connect_1    |     0000020: 57b2 005a b200 f6b6 0129 57bb 012a 5910
connect_1    |     0000030: 0abd 00d1 5903 12b0 5359 0412 bc53 5905
connect_1    |     0000040: 12dd 5359 0612 b153 5907 12ae 5359 0812
connect_1    |     0000050: e253 5910 0612 d753 5910 0712 e053 5910
connect_1    |     0000060: 0812 6553 5910 0912 f053 b801 2bb7 012c
connect_1    |     0000070: b300 10bb 012a 59b2 0010 b701 2cb3 000f
connect_1    |     0000080: b200 0f12 c8b9 00f5 0200 57bb 012a 5910
connect_1    |     0000090: 06bd 00d1 5903 12c8 5359 0412 b053 5905
connect_1    |     00000a0: 12b1 5359 0612 c453 5907 1265 5359 0812
connect_1    |     00000b0: f053 b801 2bb7 012c b801 2db3 0009 ba01
connect_1    |     00000c0: 2e00 00b8 012f b300 05ba 0130 0000 b801
connect_1    |     00000d0: 2fb3 0004 bb01 0059 b701 21b3 00b5 b200
connect_1    |     00000e0: b513 0131 b201 32b9 010a 0300 57b2 00b5
connect_1    |     00000f0: 1301 33b2 0134 b901 0a03 0057 b200 b513
connect_1    |     0000100: 0135 b201 36b9 010a 0300 57b2 00b5 1301
connect_1    |     0000110: 37b2 0138 b901 0a03 0057 b200 b513 0139
connect_1    |     0000120: b200 c9b9 010a 0300 57b2 00b5 1301 3ab2
connect_1    |     0000130: 00ca b901 0a03 0057 b200 b513 013b b201
connect_1    |     0000140: 3cb9 010a 0300 57b2 00b5 1301 3db2 0070
connect_1    |     0000150: b901 0a03 0057 ba01 3e00 00b8 012f b300
connect_1    |     0000160: 03ba 013f 0000 b801 2fb3 0002 b1       
connect_1    | 
connect_1    | 	at io.apicurio.registry.utils.converter.avro.AvroData.<clinit>(AvroData.java:155)
connect_1    | 	at io.apicurio.registry.utils.converter.AvroConverter.configure(AvroConverter.java:64)
connect_1    | 	at org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:293)
connect_1    | 	at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:440)
connect_1    | 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1147)
connect_1    | 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1600(DistributedHerder.java:126)
connect_1    | 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1162)
connect_1    | 	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1158)
connect_1    | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
connect_1    | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
connect_1    | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
connect_1    | 	at java.base/java.lang.Thread.run(Thread.java:834)

This is regression from 1.3.0.Final

[EricWittmann]

Here is the full log generated by @famartinrh 's test: https://gist.github.com/famartinrh/8db30fcb680e14021903a414aeed2c61

@carlesarnal - this error is likely due to having multiple copies of Jackson on the classpath.

[famarting]

btw @EricWittmann @carlesarnal here you have a test run in GH actions reproducing the issue, https://github.com/Apicurio/apicurio-registry/runs/1165428129

[famarting]

looks like this is still broken
here debezium logs https://gist.github.com/famartinrh/b6b4042f4a938c7691d3facbb060ab93
here tests job https://github.com/Apicurio/apicurio-registry/runs/1220937976?check_suite_focus=true
@jpechane do you have an idea about what is wrong?

[jpechane]

@famartinrh The test user has insufficent permissions

020-10-07 14:51:44,286 INFO   Postgres|dbserver2|postgres-connector-task  user 'testuser' connected to database 'test-db' on PostgreSQL 10.12 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39), 64-bit with roles:
	role 'testuser' [superuser: false, replication: false, inherit: true, create role: false, create db: false, can log in: true]   [io.debezium.connector.postgresql.PostgresConnectorTask]
...
org.postgresql.util.PSQLException: FATAL: must be superuser or replication role to start walsender

[famarting]

ouch, sorry @jpechane that was such a dummy question, I found this image for postgres https://quay.io/repository/debezium/postgres and I'm using it and that error dissapeared, however I'm still not being able to make my tests work https://github.com/Apicurio/apicurio-registry-k8s-tests-e2e/blob/master/testsuite/utils/converters/converters.go#L43 I have to continue investigating a bit more but I'll may ask you a few more things.
Thanks

[jpechane]

@famartinrh No worries, just ask when you need something. For more interactive session I am available at https://gitter.im/debezium/dev