EnableTagOverride issue

[wyhysj]

Since consul 0.6, a new option called enabletagoverride provide us a gateway to dynamically append tags to an existing service

however, I tried to use it but it only last less than 1 minutes, the tag will sync to old value

Is there something I'm missing?

[root@zabbix-server-10-65-5-102 huangz]# curl -s 10.65.5.110:8500/v1/catalog/service/redistest | jq
[
  {
    "Node": "10.65.5.106-consul",
    "Address": "10.65.5.106",
    "ServiceID": "swarm-nodes-10-65-5-106:redistest_redis_2:6379",
    "ServiceName": "redistest",
    "ServiceTags": [
      "slave"
    ],
    "ServiceAddress": "10.65.5.106",
    "ServicePort": 32808,
    "ServiceEnableTagOverride": true,
    "CreateIndex": 1600171,
    "ModifyIndex": 1602242
  },
  {
    "Node": "10.65.5.113-consul",
    "Address": "10.65.5.113",
    "ServiceID": "localhost:redistest_redis_1:6379",
    "ServiceName": "redistest",
    "ServiceTags": [],
    "ServiceAddress": "10.65.5.113",
    "ServicePort": 32791,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 1600170,
    "ModifyIndex": 1600170
  }
]

[wyhysj]

btw, in my setup there are 3 consul servers

[slackpad]

Hi @wyhysj are you using the Catalog API to update the tags (https://www.consul.io/docs/agent/http/catalog.html)? If you are not using the Catalog API and change them using the Agent API then the stale catalog value will overwrite it, which sounds like what might be happening here.

[wyhysj]

@slackpad yes, I use Catalog API to update tags
One more thing, my consul server already upgraded to 0.6, but my consul client remains at 0.5.2 since lots of clients running

[slackpad]

Ah that explains it - it needs the client and server to both be >=0.6.0 in order to work. The server has to support storing the new field and then the client is the one who honors it and takes the tags from the server instead of sending its own tags up every time.

[wyhysj]

@slackpad Thanks for the clarify
today I upgrade everything >0.6.0 and the problem still exists
First I think it's protocol version
As this page #1531 explains it's irrelevant

root@consul-server-10-65-5-107:~# consul members -rpc-addr=10.65.5.107:8400
Node                       Address           Status  Type    Build  Protocol  DC
.......
10.65.5.111-consul         10.65.5.111:8301  left    client  0.6.1  2         sgsa1
10.65.5.113-consul         10.65.5.113:8301  alive   client  0.6.1  2         sgsa1
10.65.5.114-consul         10.65.5.114:8301  alive   client  0.6.1  2         sgsa1
consul-server-10-65-5-107  10.65.5.107:8301  alive   server  0.6.0  2         sgsa1
consul-server-10-65-5-109  10.65.5.109:8301  alive   server  0.6.0  2         sgsa1
consul-server-10-65-5-110  10.65.5.110:8301  alive   server  0.6.0  2         sgsa1
.......

I use /v1/catalog/register api and my body.json looks like this

{
  "Node": "10.65.5.106-consul",
  "Address": "10.65.5.106",
  "Service": {
    "ID": "swarm-nodes-10-65-5-106:redis_redis_2:6379",
    "Service": "redis",
    "Tags": ["slave"],
    "Address": "10.65.5.106",
    "Port": 32808,
    "EnableTagOverride": true
  }
}

[slackpad]

@wyhysj sorry about this - running through an example now I can see how this is confusing - I think we should beef up our documentation a bit.

Syncing of service registration usually happens from the agents up to the servers. The tag override feature is a rare case that goes the other way. When the agent is syncing, it fetches what's on the server, compares it with what it has, and then pushes up any changes. I think the problem you are seeing is that the agent doesn't have the EnableTagOverride flag set to true in its local configuration, you are setting that in the catalog, so the agent isn't respecting that setting.

Here's an example that shows how it works:

Register the service on the agent using the agent API

Note that we enable tag overrides when we register the service with the agent here.

> curl -X POST -d \
'{
  "Name": "my-service",
  "Tags": [
    "standby"
  ],
  "Port": 8080,
  "EnableTagOverride": true
}' localhost:8500/v1/agent/service/register

Verify that the standby tag is set in the catalog

Note here that the server captured the agent's tag override setting.

> curl localhost:8500/v1/catalog/service/my-service?pretty
[
    {
        "Node": "nyc3-worker-2",
        "Address": "45.55.209.31",
        "ServiceID": "my-service",
        "ServiceName": "my-service",
        "ServiceTags": [
            "standby"
        ],
        "ServiceAddress": "",
        "ServicePort": 8080,
        "ServiceEnableTagOverride": true,
        "CreateIndex": 3653760,
        "ModifyIndex": 3653981
    }
]

Update the tag using the catalog

> curl -X POST -d \
'{
  "Node": "nyc3-worker-2",
  "Address": "45.55.209.31",
  "Service": {
    "Service": "my-service",
    "Tags": [
      "master"
    ],
    "Port": 8080,
    "EnableTagOverride": true
  }
}' localhost:8500/v1/catalog/register

Verify that the agent doesn't overwrite it

> curl localhost:8500/v1/catalog/service/my-service?pretty
[
    {
        "Node": "nyc3-worker-2",
        "Address": "45.55.209.31",
        "ServiceID": "my-service",
        "ServiceName": "my-service",
        "ServiceTags": [
            "master"
        ],
        "ServiceAddress": "",
        "ServicePort": 8080,
        "ServiceEnableTagOverride": true,
        "CreateIndex": 3653760,
        "ModifyIndex": 3654003
    }
]

Verify that the agent took the tags from the master into its local configuration

> curl localhost:8500/v1/agent/services?pretty
{
    "my-service": {
        "ID": "my-service",
        "Service": "my-service",
        "Tags": [
            "master"
        ],
        "Address": "",
        "Port": 8080,
        "EnableTagOverride": true,
        "CreateIndex": 0,
        "ModifyIndex": 0
    }, #...

Please let me know if this makes sense!

[wyhysj]

@slackpad Thanks for such detailed explanation
After changing register service with EnableTagOverride: true
It works great !

[slackpad]

Glad it's working - keeping open to touch up the docs.

[motty-stratoscale]

@slackpad , Thank you from me too.
Can you check if this is correct please?

There are two ways to change the tags of an existing service:

1. By the catalog - v1/catalog/register.
2. By re-registering the service on the agent - v1/agent/service/register.

If you want to use way 1, create the service with EnableTagOverride: true. Because otherwise the tags will be run over.
If you want to use way 2, create the service with EnableTagOverride: false (the default). Why? read on.

Advanced note: It seems that v1/agent/service/register sends the new tags to the catalog regardless of EnableTagOverride. So if you use way 2 yet with EnableTagOverride: true, it will 99% work. The only problem is if you call v1/agent/service/register while the cluster is unavailable; then the registration on the agent will succeed, but it won't send the tag to the server upon re-connection!

[slackpad]

@motty-stratoscale that sounds right, and is a bit subtle, unfortunately. The semantics of EnableTagOverride ended up being more complicated than I would have liked.

[MitchFierro]

I'd like to add one minor caveat here for posterity that had me confused, but ultimately was not a problem.

If you update tags for a service with EnableTagOverride set to true via /v1/catalog/register, and you do not include the "EnableTagOverride" : true in the Service object of that request payload, then a subsequent GET for that service on v1/catalog/service/:service will show "EnableTagOverride" : false. Eventually the catalog service object will pick back up the true value from the agent, but it is confusing and not well documented that this field is only relevant for agents.

If you are using a client library, it's almost certain you'll run into this. The docs don't show that you can in fact pass EnableTagOverride as a Service property in the v1/catalog/register api. Since it's not documented (because it's irrelevant from the catalog's perspective), client library model classes don't usually expose it.

[hanshasselberg]

https://www.consul.io/api/agent/service.html#enabletagoverride has better docs now that are explaining it well imho. Closing because I think this is solved. Let me know if you disagree.