Failed to verify existence of migrations metadata stream 'MIGRATION_EVENTS'

[DmytroMelnyk]

Describe the bug
Unexpected errors when following manual for ksql-migrations

To Reproduce
Steps to reproduce the behavior, include:

1. docker-compose.yml

version: '2'
services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.19.0
    hostname: ksqldb-server
    container_name: ksqldb-server
    depends_on:
      - connect-replicator
    ports:
      - "8088:8088"
    environment:
      KSQL_KSQL_SERVICE_ID: ksql-dev
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "host.docker.internal:9093"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
      KSQL_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
      KSQL_KSQL_CONNECT_URL: "http://connect-replicator:8382"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.19.0
    user: root
    container_name: ksqldb-cli
    depends_on:
      - connect-replicator
      - ksqldb-server
    volumes:
      - "./remove_after_test:/orders/"      
    entrypoint: /bin/sh
    tty: true

2. ksql-migrations.properties:

ksql.server.url=http://host.docker.internal:8088
ksql.migrations.stream.name=MIGRATION_EVENTS
ksql.migrations.table.name=MIGRATION_SCHEMA_VERSIONS
ksql.migrations.topic.replicas=1

3. Follow this manual till: Open this file and add the following command
4. Change recommended script to:

CREATE STREAM ${STREAMNAME} ( orderid STRING, name STRING, items ARRAY<STRING>, cost DECIMAL(6, 2), ordertime TIMESTAMP ) WITH ( kafka_topic='order_event', partitions=1, value_format='json' );

5. run:

ksql-migrations apply -a -c orders/ksql-migrations.properties -d STREAMNAME=ORDER_EVENTS

Expected behavior:
Stream ORDER_EVENTS should be created

Actual behaviour

ksqldb-0.19_ksqldbcli-0.19 (1).txt

[jzaralim]

From the logs:

[2021-11-05 10:18:18,583] INFO Processed unsuccessfully: KsqlRequest{ksql='describe MIGRATION_EVENTS;', configOverrides={}, requestProperties={}, sessionVariables={}, commandSequenceNumber=Optional.empty}, reason:  (io.confluent.ksql.rest.server.resources.KsqlResource:331)
java.lang.NullPointerException
	at io.confluent.ksql.util.QueryMetadataImpl.getState(QueryMetadataImpl.java:230)
	at io.confluent.ksql.rest.server.execution.ListSourceExecutor.lambda$getQueries$11(ListSourceExecutor.java:440)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
	at java.base/java.util.Collections$2.tryAdvance(Collections.java:4747)
	at java.base/java.util.Collections$2.forEachRemaining(Collections.java:4755)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at io.confluent.ksql.rest.server.execution.ListSourceExecutor.getQueries(ListSourceExecutor.java:441)
	at io.confluent.ksql.rest.server.execution.ListSourceExecutor.describeSource(ListSourceExecutor.java:285)
	at io.confluent.ksql.rest.server.execution.ListSourceExecutor.columns(ListSourceExecutor.java:225)
	at io.confluent.ksql.rest.server.validation.CustomValidators.validate(CustomValidators.java:136)
	at io.confluent.ksql.rest.server.validation.RequestValidator.validate(RequestValidator.java:160)
	at io.confluent.ksql.rest.server.validation.RequestValidator.validate(RequestValidator.java:130)
	at io.confluent.ksql.rest.server.resources.KsqlResource.handleKsqlStatements(KsqlResource.java:281)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeKsqlRequest$2(KsqlServerEndpoints.java:191)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeOldApiEndpointOnWorker$23(KsqlServerEndpoints.java:342)
	at io.confluent.ksql.rest.server.KsqlServerEndpoints.lambda$executeOnWorker$22(KsqlServerEndpoints.java:328)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$2(ContextImpl.java:313)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

This is a known bug. It was fixed in #7627 which is in 0.20. I've seen it happen few times and restarting ksqlDB usually fixes it.