Active/passive pools of services

[apparentlymart]

I'd like to be able to spin up multiple clusters of the same service, with only one being active at a time. The main reason I'm interested in this is to allow bringing up servers for multiple versions of our app at once but to have only one be active at a time.

One potential design I thought of for this is to allow each service declaration to have an additional "pool" declaration, defaulting to the empty string. The Consul API would then allow a separate process to decide which pool is active for each service, again defaulting to the empty string.

Since the default is the empty string the initial behavior would be the same as today, as if there was no pooling whatsoever.

For my deployment process I would then:

• Bring up the new servers, with them configured to announce a service with the pool set to the version number.
• Once they've settled and the health checks are passing for the new pool, call the API to switch the active pool to the new version number.
• After some period where I'm pretty sure the release has succeeded, shut down the old servers and have them deregister from Consul.

This implies a couple details:

• It's possible to see via the API and UI the healthchecks for servers in non-active pools, so they can be monitored before switching to them.
• When switching pools, Consul would act as if the new services registered and old services deregistered, triggering updates for tools like consul-template that are watching the registry.

Something like the above could be implemented by adding version numbers to the service names, but that is less convenient because it requires every app to be aware of the version numbering and switch versions at the right time. Having Consul handle this itself means that the machinery for managing the pools can be hidden from the applications, which instead just see a bunch of services coming and going like normal.

[highlyunavailable]

Use tags and set the "active" tag maybe? For example, you have a service that's named "myapp" with a tags ["DEADBEEF", "active"] where DEADBEEF is the git commit ID for example.

You deploy a server with version ABAD1DEA and then once it is stable, change the service definition to add the active tag to it.

Your consul template just looks for all instances of myapp with tag active and you can access a random active instance with active.myapp.service.consul.

This would imply #1102 is completed though, or that each server had a local way to update their own tags (e.g. a watcher process that checked a K/V store entry for the "Current active version" and then updated the service definition somehow).

[mdaffin]

You can make use of the maintenance mode for this, just set all nodes that you do not want to be live in maintenance mode and they will be removed from the output of health queries.

Alternatively you could register a unhealthy check manually, or create a script that reports unhealthy on nodes that you want to be inactive (ie if their version does not match one in the key/value store).

[hbagdi]

Is this going to be a thing?
Are there other workarounds to solve this problem with Consul?

[hanshasselberg]

Thanks for reporting! This is possible with Connect now. Please see https://www.consul.io/docs/connect/l7-traffic-management.html.