OkHttp client missing content type headers in requests.

[norganna]

Using 1.3.1.Final with the new OkHttp client, some (possibly all?) requests are being made without the Content-Type header being sent which causes certain requests (i.e, ones where it tries to parse the POST data) to time out.

Here's an example of the request from a TCP dump:

POST /api/artifacts?ifExists=RETURN_OR_UPDATE HTTP/1.1
X-Registry-ArtifactType: KCONNECT
X-Registry-Artifactid: ****-value
Content-Length: 3666
Host: apicurio.kafka:8080
Connection: Keep-Alive
Accept-Encoding: gzip
User-Agent: okhttp/3.14.6

We are using it here with Debezium / Connect and are seeing numerous errors like:

Oct 09, 2020 10:51:49 AM io.apicurio.registry.client.request.RequestHandler$ResultCallback handleError
SEVERE: Error getting call result
java.lang.RuntimeException: java.net.SocketTimeoutException: timeout
        at io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:43)
        at io.apicurio.registry.utils.ConcurrentUtil.get(ConcurrentUtil.java:27)
        at io.apicurio.registry.client.request.RequestHandler$ResultCallback.getResult(RequestHandler.java:49)
        at io.apicurio.registry.client.request.RequestHandler.execute(RequestHandler.java:20)
        at io.apicurio.registry.client.RegistryRestClientImpl.createArtifact(RegistryRestClientImpl.java:127)
        at io.apicurio.registry.client.CompatibleClient.createArtifact(CompatibleClient.java:60)
        at io.apicurio.registry.utils.serde.strategy.AbstractCrudIdStrategy.findId(AbstractCrudIdStrategy.java:54)
        at io.apicurio.registry.utils.converter.ExtJsonConverter.fromConnectData(ExtJsonConverter.java:95)
        at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:314)
        at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:146)
        at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:180)
        at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:122)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:314)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:340)
        at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:264)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: timeout
        at okio.Okio$4.newTimeoutException(Okio.java:232)
        at okio.AsyncTimeout.exit(AsyncTimeout.java:286)
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:241)
        at okio.RealBufferedSource.indexOf(RealBufferedSource.java:358)
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:230)
        at okhttp3.internal.http1.Http1ExchangeCodec.readHeaderLine(Http1ExchangeCodec.java:242)
        at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.java:213)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.java:115)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:94)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        ... 3 more
Caused by: java.net.SocketException: Socket closed
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:183)
        at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at okio.Okio$2.read(Okio.java:140)
        at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
        ... 24 more

Here's a list of the versions of the software in our stack:

kubectl -n kafka get po -o yaml | grep image: | sort | uniq
      image: apicurio/apicurio-registry-kafka:1.3.1.Final
      image: banzaicloud/cruise-control:2.5.11
      image: banzaicloud/jmx-javaagent:0.13.0
      image: banzaicloud/kafka-operator:v0.12.4
      image: banzaicloud/kafka:2.13-2.6.0-bzc.1
      image: debezium/connect:1.3.0.Final
      image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
      image: pravega/zookeeper-operator:0.2.8
      image: pravega/zookeeper:latest

[EricWittmann]

@carlesarnal wasn't this already fixed? Perhaps the fix came after 1.3.1.Final?

[carlesarnal]

Hi @EricWittmann, the content-type not being present is supposed to be fixed in 1.3.1.Final as you can see here and @tagarr confirmed that is working. Let me try to reproduce this issue.

[norganna]

@carlesarnal So, I deeply apologize for this, but even though I'm using the 1.3.1.Final apicurio registry server container image, I just checked the Debezium connect container image (debezium/connect:1.3.0.Final) jars and they are packaging the 1.3.0.Final apicurio jars, so that'll be why the headers are not being sent, I think.

[carlesarnal]

No problem @norganna. If that's the case, I guess we can close this issue since this issue has been resolved in 1.3.1.Final.