Unable to start ksqlDB,

[shekp]

Hello,

Could you please help, I am trying to install and start ksqlDB using docker-compose with already installed Apache Kafka version 2.11

Apache Kafka uses SASL PLAIN mechanism.

This is my docker-compose.yml file content:

version: '2'

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.8.1
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: "10.10.10.10:9092"
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_SECURITY_PROTOCOL: "SASL_PLAINTEXT"
      KSQL_SASL_MECHANISM: "PLAIN"
      KSQL_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"admin\" password=\"password\";"
  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.8.1
    container_name: ksqldb-cli
    depends_on:
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

After starting with command - docker-compose up, I am faced with message

ksqldb-server | [2020-04-20 13:51:06,507] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) ksqldb-server | [2020-04-20 13:51:06,507] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig:355) ksqldb-server | [2020-04-20 13:51:07,450] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:63) ksqldb-server | io.confluent.ksql.util.KsqlServerException: Could not get Kafka authorized operations! ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:54) ksqldb-server | at io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:52) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:711) ksqldb-server | at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:629) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:101) ksqldb-server | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59) ksqldb-server | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 7 ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) ksqldb-server | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) ksqldb-server | at io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:52) ksqldb-server | ... 5 more ksqldb-server | Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 7

[ateijelo]

I just ran into that issue. I think it's because KSQL does DescribeClusterOptions().includeAuthorizedOperations() which seems to have been introduced in Kafka 2.3.

ksql/ksqldb-engine/src/main/java/io/confluent/ksql/services/KafkaClusterUtil.java

Line 49 in c49b86c

new DescribeClusterOptions().includeAuthorizedOperations(true)

https://issues.apache.org/jira/browse/KAFKA-7922

[ateijelo]

You mention you're using Kafka 2.11, I think that's the version of Scala that your brokers are using. The most recent version of Kafka as of this comment is 2.5.

[shekp]

You mention you're using Kafka 2.11, I think that's the version of Scala that your brokers are using. The most recent version of Kafka as of this comment is 2.5.

Hello,
Thank you for help.
You are right, we use 2.1.1 Kafka version, and it seems, that we have faced with this issue https://issues.apache.org/jira/browse/KAFKA-7922

[tarekabouzeid]

Hi,
i am using Kafka version 1.1.1 and facing same error as well, i enabled raised the log level to TRACE and below is the detailed error:

[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Metadata is ready to use. (org.apache.kafka.clients.admin.internals.AdminMetadataManager:159)
[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Metadata is ready to use. (org.apache.kafka.clients.admin.internals.AdminMetadataManager:159)
[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Found least loaded node xxxx:9092 (id: 1001 rack: null) connected with no in-flight requests (org.apache.kafka.clients.NetworkClient:677)
[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Found least loaded node xxxx:9092 (id: 1001 rack: null) connected with no in-flight requests (org.apache.kafka.clients.NetworkClient:677)
[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Assigned Call(callName=listNodes, deadlineMs=1588610620826) to node xxxx:9092 (id: 1001 rack: null) (org.apache.kafka.clients.admin.KafkaAdminClient:986)
[2020-05-04 18:42:40,829] TRACE [AdminClient clientId=adminclient-1] Assigned Call(callName=listNodes, deadlineMs=1588610620826) to node xxxx:9092 (id: 1001 rack: null) (org.apache.kafka.clients.admin.KafkaAdminClient:986)
[2020-05-04 18:42:40,829] DEBUG [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=true, includeTopicAuthorizedOperations=false) to xxxx:9092 (id: 1001 rack: null). correlationId=7 (org.apache.kafka.clients.admin.KafkaAdminClient:1037)
[2020-05-04 18:42:40,829] DEBUG [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=true, includeTopicAuthorizedOperations=false) to xxxx:9092 (id: 1001 rack: null). correlationId=7 (org.apache.kafka.clients.admin.KafkaAdminClient:1037)
SLF4J: Failed toString() invocation on an object of type [org.apache.kafka.common.requests.MetadataRequest]
Reported exception:
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
[2020-05-04 18:42:40,830] DEBUG [AdminClient clientId=adminclient-1] Using older server API v5 to send METADATA [FAILED toString()] with correlation id 7 to node 1001 (org.apache.kafka.clients.NetworkClient:510)
[2020-05-04 18:42:40,830] DEBUG [AdminClient clientId=adminclient-1] Using older server API v5 to send METADATA [FAILED toString()] with correlation id 7 to node 1001 (org.apache.kafka.clients.NetworkClient:510)
[2020-05-04 18:42:40,831] DEBUG [AdminClient clientId=adminclient-1] Version mismatch when attempting to send MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=true, includeTopicAuthorizedOperations=false) with correlation id 7 to 1001 (org.apache.kafka.clients.NetworkClient:488)
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
[2020-05-04 18:42:40,831] DEBUG [AdminClient clientId=adminclient-1] Version mismatch when attempting to send MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=true, includeTopicAuthorizedOperations=false) with correlation id 7 to 1001 (org.apache.kafka.clients.NetworkClient:488)
org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=60003) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=60003) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 1 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient:1273)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 1 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient:1273)
[2020-05-04 18:42:40,834] DEBUG [AdminClient clientId=adminclient-1] Call(callName=listNodes, deadlineMs=1588610620826) failed with non-retriable exception after 1 attempt(s) (org.apache.kafka.clients.admin.KafkaAdminClient:734)
java.lang.Exception: UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:735)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1115)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1280)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)
at java.lang.Thread.run(Thread.java:745)
[2020-05-04 18:42:40,834] DEBUG [AdminClient clientId=adminclient-1] Call(callName=listNodes, deadlineMs=1588610620826) failed with non-retriable exception after 1 attempt(s) (org.apache.kafka.clients.admin.KafkaAdminClient:734)
java.lang.Exception: UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:735)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1115)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1280)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1203)
at java.lang.Thread.run(Thread.java:745)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1588610560834 (org.apache.kafka.clients.admin.KafkaAdminClient:961)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1588610560834 (org.apache.kafka.clients.admin.KafkaAdminClient:961)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=299942) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=299942) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient:1273)
[2020-05-04 18:42:40,834] TRACE [AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s) (org.apache.kafka.clients.admin.KafkaAdminClient:1273)
[2020-05-04 18:42:40,835] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1588610560835 (org.apache.kafka.clients.admin.KafkaAdminClient:961)
[2020-05-04 18:42:40,835] TRACE [AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1588610560835 (org.apache.kafka.clients.admin.KafkaAdminClient:961)
[2020-05-04 18:42:40,835] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=299941) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)
[2020-05-04 18:42:40,835] ERROR Failed to start KSQL (io.confluent.ksql.rest.server.KsqlServerMain:63)
io.confluent.ksql.util.KsqlServerException: Could not get Kafka authorized operations!
at io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:51)
at io.confluent.ksql.security.KsqlAuthorizationValidatorFactory.create(KsqlAuthorizationValidatorFactory.java:52)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:639)
at io.confluent.ksql.rest.server.KsqlRestApplication.buildApplication(KsqlRestApplication.java:567)
at io.confluent.ksql.rest.server.KsqlServerMain.createExecutable(KsqlServerMain.java:100)
at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:59)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at io.confluent.ksql.services.KafkaClusterUtil.isAuthorizedOperationsSupported(KafkaClusterUtil.java:49)
... 5 more
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default includeClusterAuthorizedOperations at version 5
[2020-05-04 18:42:40,835] TRACE [AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=299941) (org.apache.kafka.clients.admin.KafkaAdminClient:1271)

so is there any workaround this or the only option to run KSQLDB is having Kafka 2.3+ ?

Best Regards,

[pimpelsang]

Also run into issue, running Kafka brokers 1.1... Can't use ksqlDB?

[stevenpyzhang]

@pimpelsang @tarekabouzeid @shekp can you all try running the ksqlDB server with the following config set to true/false ksql.access.validator.enable=false
That should avoid making that call to includeAuthorizedOperations

[tarekabouzeid]

Hi @stevenpyzhang ,

This configuration successfully started ksqlDB server for me, will continue testing ksqlDB functionalities as well, thanks so much for such important input, i really suggest that this to be added to confluent official documentation in order to make ksqlDB runs against kafka versions older than 2.3.

Best Regards,

[stevenpyzhang]

The server isn't supposed to not start up in these cases, the behavior was fixed in #5617 and the fix should be in starting from ksqlDB 0.10.0

[RafaelGSS]

@pimpelsang @tarekabouzeid @shekp can you all try running the ksqlDB server with the following config set to true/false ksql.access.validator.enable=false
That should avoid making that call to includeAuthorizedOperations

Just a feedback:

The confluentinc/ksqldb-server:0.11.0 shows:

org.apache.kafka.common.config.ConfigException: Invalid value false for configuration ksql.access.validator.enable: String must be one of: on, off, auto

So, change to ksql.access.validator.enable=off works.

[NikolayVaklinov10]

Hi, @RafaelGSS I am trying to find this file in order to change it because I have the same issue for the confluent-6.1.1 platform. Could you please tell me whether you are talking about the "confluent-6.1.1/etc/ksqldb/ksql-server.properties" file where this "ksql.access.validator.enable=off" setting is?