Multi Container Dev Services

[cescoffier]

Hello,

This is based on an email from @lbroudoux asking me about multi-container dev services and links between dev service containers.

---

I wonder how to manage dependencies between Quarkus DevServices and, more generally, between containers managed by DevServices.

I would need to attach/initialize them within the same network to enable communication via aliases and set dependencies to constrain the startup order.

For example, start the Microcks container after the Kafka container to avoid DNS resolution failures.

I know that @ozangunalp and @alesj have recently worked on that subject (or worked around the issue). Can you provide your insights?

[alesj]

I would need to attach/initialize them within the same network to enable communication via aliases and set dependencies to constrain the startup order.

Wrt to "within the same network", this is still a mystery to me -- it works most of the time, but sometimes not.

I do set this in all containers (if shared of course, which they are by default)

        if (config.shared()) {
            withNetwork(Network.SHARED);
        }

and then set aliases to all of them.
But I remember that I had some weird issues where Jager had to be set differently - alias vs. 0.0.0.0 :

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  logging:
    loglevel: debug
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: 0.0.0.0:8428
    namespace: quarkus_observability

Anyway, I'm just about to do a new PR (closed the previous one - as it was too big :-) -- and you can see the concept behind it.
And perhaps we can rewrite some of the existing stuff to this as well.

[lbroudoux]

Thanks for jumping on the conversation!

I agree: sharing the same network can be easily solved thanks to Network.SHARED.

I've gone through a complete demonstration on Microcks capabilities when working with Kafka containers and now have the whole set of requirements. Below is the list of what I'd like to and don't know how to do it with Quarkus at the moment:

• Detect the Kafka (or another broker) DevService is present (configured or running),
• Retrieve the configured (or assigned) hostname + port for this broker container,
  • Another way of doing that would be to have the ability to setup network alias for broker containers and additional listeners (but I'm not sure it is supported atm)
• Delay (or constraint) the creation of my own containers to the availability of broker container (to avoid the above-mentioned DNS issues)

To be transparent, I set up the full demo using Spring Boot at the moment, but I'd like to do exactly the same thing with Quarkus - but highlight that it should be even easier, with less boilerplate, for developers.

IMHO, in an ideal world, everything that lives into my Spring Boot ContainersConfiguration.java must be totally hidden from the user and dynamically instantiated at the Dev Service level. It would be awesome if we achieved this!

If you have any link to the PR you're mentioned, I'd be happy to start reading them and think about it.

[lbroudoux]

For the record, I finally succeeded in doing this using a multiple BuildStep approach as implemented in the new DevServicesMicrocksProcessor

The principle is roughly the next one:

1. A first step checks if DevService is enabled and then it returns a Dev Service Shared Network build item to force other DevServices to register their containers into the shared network (this is required for accessing brokers from host but also from other containers)
2. A second step checks if DevService is enabled and creates/starts/locates the Microcks main container. If we start a new container, we also pre-compute host aliases for potentially other additional containers Microcks would need to start later on. After the container startup, we load artifacts in Microcks and inspect what's being loaded (OpenAPI, Postman Collection, AsyncAPI files, ...)
3. A third step is activated when all other DevServices have been started. At that moment, we can check if brokers have been made available (typically by inspecting contributed properties like kafka.bootstrap.servers) and as we know the artifacts that have been loaded into Microcks, we can decide to start additional containers like the Postman runtime or our Minion dedicated to Async features. Those containers are started in the shared networks with the host aliases that were previously computed so that they are able to connect to Microcks main containers and other broker's containers,
4. The fourth and final step is just using a Recorder to make sure that all our steps chain get invoked in any case.

I got a discussion with Clément today to explain the principles it looks kinda ok to him. Feel free to let me know if you have any other suggestions or concerns regarding this way of doing things.

Thanks!

[maxandersen]

Just so I read it right - this is for a user to customize his own containers in addition to what gets started via dvservices or you are trying to make another devservice orchestrate other devservices?

The former seems fine. The latter would be pushing the limits of what devservices would be good for.

[lbroudoux]

It's mostly the latter case but not really "orchestrating" other devservices. It's much more having the Microcks devservice configuring itself differently (using 1, 2 or 3 containers) depending on other discovered running devsercices.

[lbroudoux]

As explained in an above comment, I finally succeeded in making things work and having a devservice configuring itself differently (using 1, 2 or 3 containers) depending on other discovered running devservices.

For that, I had to force the creation of a shared network for my dev service containers and those related to brokers (eg. Kafka). Kafka dev service manages this well by adding an extra listener when a shared network is detected. So it presents 2 possibilities for connection in the property of the bootstrap server: something like PLAINTEXT://kafka-3g7kd:9092,OUTSIDE://localhost:60790 that allows connecting from both the host network and within the docker network.

However, we discovered side effects... See microcks/microcks-quarkus#30, as that when useShareNetwork is enabled, other dev services like the MongoDB one, configure themself with a single interface on the shared network only that makes them not reachable from the host running the application. I suspect this behavior is the same for most databases-related devservices.

I need help with this as introducing advanced relationships between dev services container breaks basic stuffs...

I'd like to know if we can't introduce a flag that prevents the usage of shared network by devservices where it doesn't make sense.

Something like:

%dev.quarkus.mongodb.devservices.use-shared-network=false

would prevent MongoDB from registering itself in the shared network automatically.

From what I saw (and vaguely understood), a shared network was first introduced for running integration tests and thus using them by default in dev mode is an over-complication.

What do you think? I appreciate your help and ideas on this!

[lbroudoux]

🎉 I got it working! I made the change for both MongoDB and PostgreSQL, and everything is working fine!

So, we're back to the initial situation where:

• When running mvn quarkus:dev, your DB will be accessible using localhost:<port> even if Microcks is using a shared network for its own purposes
• When running mvn test, your DB will be accessible using localhost:<port> even if Microcks is using a shared network for its own purposes
• When running mvn verify -Dquarkus.container-image.build=true (launching QuarkusIntegrationTest on containerized app), your DB will be run on the shared network with mongo-<random>:<port> on the same shared network that all other DevServices.

The condition of using the shared network is now:

boolean useSharedNetwork = globalDevServicesConfig.launchOnSharedNetwork ||
                        (!devServicesSharedNetworkBuildItem.isEmpty()
                                && devServicesSharedNetworkBuildItem.get(0).getSource().equals("io.quarkus.test.junit"));

I duplicate it around the different DevServices, but this could be put into a utility class later on.

What do you think @geoand? How do you think I could go forward and contribute this?

[geoand]

Nice!

I duplicate it around the different DevServices, but this could be put into a utility class later on.

Agreed.

How do you think I could go forward and contribute this?

Sounds like the next step is a Pull Request :)

[lbroudoux]

Ok!

So I'll also push the GlobalDevServicesConfig.launchOnSharedNetwork change. Is it ok for you?

Do you have a specific suggestion regarding the utility class for wrapping this logic? Is there an already existing one that could be related and a good candidate?

Also, is there a specific branch to target the PR?

[geoand]

So I'll also push the GlobalDevServicesConfig.launchOnSharedNetwork change. Is it ok for you?

Sure thing

Do you have a specific suggestion regarding the utility class for wrapping this logic? Is there an already existing one that could be related and a good candidate?

I think the easiest thing would just be to have a static method on DevServicesSharedNetworkBuildItem itself

[lbroudoux]

Hello @geoand and @cescoffier,

The PR is now submitted! See #39899