Add support for Apache Kafka native image

[eddumelendez]

No description provided.

[davetroiano]

Hi @eddumelendez do you know when a new org.testcontainers:kafka module (v 1.20.1? 1.21.0?) will be released? Is there a pre-release module available anywhere?

[eddumelendez]

I'm planning to release it next week. In the meantime you can use new org.testcontainers.kafka.KafkaContainer(DockerImageName.parse("apache/kafka-native:3.8.0-rc3"). asCompatibleSubstituteFor("apache/kafka"));

[davetroiano]

In the meantime you can use new org.testcontainers.kafka.KafkaContainer(DockerImageName.parse("apache/kafka-native:3.8.0-rc3"));

Won't I need the new release to avoid an exception? I tried this out and hit:

java.lang.IllegalStateException: Failed to verify that image 'apache/kafka-native:3.8.0-rc3' is a compatible substitute for 'apache/kafka'.

[eddumelendez]

I have updated my previous answer. You don't need a new release if you do so. Look at Image Name Substitution

[prabhamanepalli]

Hi Eddu and Dave,

I am directly using without image substitution. However, I am still facing the error Dave shared above.
Can you explain my next steps ?
Do I have to wait for the next module release to make it work without image substitution?

Thanks.

[eddumelendez]

Hi @prabhamanepalli, this has not been released yet. So, the substitution should be done. This change will be part of the next release and could be used without image substitution.

[jabrena]

Hi,

I am testing with Spring boot the new native image and using latest version fo testcontainers 1.20.1:

//private static final String kafkaVersion = "confluentinc/cp-kafka:7.7.0";
private static final String kafkaVersion = "apache/kafka-native:3.8.0";

private static Slf4jLogConsumer logConsumer = new Slf4jLogConsumer(logger);

@Bean
@ServiceConnection
KafkaContainer kafkaContainer() {
        logger.info("Creating Kafka Broker");
        var imageName = DockerImageName.parse(kafkaVersion).asCompatibleSubstituteFor("confluentinc/cp-kafka");
        return new KafkaContainer(imageName).withLogConsumer(logConsumer);
}

but I am receiving the following error:

2024-07-31 16:51:22.430  INFO tc.apache/kafka-native:3.8.0 - Creating container for image: apache/kafka-native:3.8.0
2024-07-31 16:51:22.586  INFO tc.apache/kafka-native:3.8.0 - Container apache/kafka-native:3.8.0 is starting: 32c567b0f548459ff257c88b30dd06d7a60f9cee4ab579b925da8edfdffd9861
2024-07-31 16:51:23.072  INFO i.j.m.c.KafkaContainerConfig - STDERR: /tmp/testcontainers_start.sh: line 3: /etc/confluent/docker/ensure: No such file or directory
2024-07-31 16:51:23.072  INFO i.j.m.c.KafkaContainerConfig - STDERR: /tmp/testcontainers_start.sh: line 7: zookeeper-server-start: command not found
2024-07-31 16:51:23.072  INFO i.j.m.c.KafkaContainerConfig - STDERR: /tmp/testcontainers_start.sh: line 8: /etc/confluent/docker/run: No such file or directory

If I don´t put the compability option:

.asCompatibleSubstituteFor("confluentinc/cp-kafka")

I receive:

Caused by: java.lang.IllegalStateException: Failed to verify that image 'apache/kafka-native:3.8.0' is a compatible substitute for 'confluentinc/cp-kafka'. This generally means that you are trying to use an image that Testcontainers has not been designed to use. If this is deliberate, and if you are confident that the image is compatible, you should declare compatibility in code using the `asCompatibleSubstituteFor` method. For example:
   DockerImageName myImage = DockerImageName.parse("apache/kafka-native:3.8.0").asCompatibleSubstituteFor("confluentinc/cp-kafka");
and then use `myImage` instead.

How to solve this issue?

[eddumelendez]

Hi, you are using org.testcontainers.containers.KafkaContainer but apache/kafka-native is supported at org.testcontainers.kafka.KafkaContainer

[jabrena]

Many thanks @eddumelendez, now it is working the example but not using Service Connection:

Caused by: org.springframework.boot.autoconfigure.service.connection.ConnectionDetailsNotFoundException: No ConnectionDetails found for source '@ServiceConnection source for Bean 'kafkaContainer' defined in class path resource [info/jab/ms/controller/KafkaContainerConfig.class]'

To run the test, I used the previous approach:

public class KafkaContainerInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
...
}

The question that I have is why not use the same annotation for all kafka containers:

• Confluent
• Apache Kafka
• Apache Kafka native
• Red Panda

BTW, the performance improvement is huge :)

Juan Antonio

[eddumelendez]

@jabrena this is expected because org.testcontainers.kafka.KafkaConainer is a new implementation. Support in spring boot has been added for 3.4.0-M1. See spring-projects/spring-boot#40695

The question that I have is why not use the same annotation for all kafka containers

You can take a look at each implementation and those handle different things, env vars or configuration files, in order to set it up correctly.

[jabrena]

@eddumelendez it is clear for me.

Great job 😊