Allow setting of Consul ServiceMeta tags from config file #11084

unRob · 2021-03-11T06:34:02Z

Hi there!

I'd love to be able to set ServiceMeta on the Consul service registered by Vault by specifying the desired tags in the config file, and I'm submitting this PR to that effect.

I'm not happy about unmarshalling json during NewServiceRegistration but considered that a simpler change than the couple of alternatives I came up from a quick glance at the codebase. That is to say, I'd love feedback on the approach (and perhaps suggestions on a proper testing strategy should folks feel this approach works).

fixes #9121
see also #10233

hashicorp-cla · 2021-03-11T06:34:06Z

All committers have signed the CLA.

unRob · 2023-06-24T17:07:45Z

Still very interested in seeing this through, any chance it could be reviewed?

VioletHynes · 2024-05-27T16:32:09Z

Hey there @unRob ! I'm really very sorry this one has waited so long, but I'm pretty eager to get it merged in as long as you are.

There's just a few things I'd like to confirm/address before we do. I'll be leaving a review shortly with some comments, but I'll be watching this closely for updates, and please feel free to tag me just in case :)

VioletHynes

Thanks for bearing with us on this. A few comments and things I'd like to see answered/addressed, but I'm eager to get this merged. I agree that the approach of deserializing the JSON where you have seems to be the easiest place and I don't see it as too problematic.

I know this took a while to be reviewed but I'm committed to helping you see this to the finish line, as long as you're still willing :)

VioletHynes · 2024-05-27T16:33:31Z

serviceregistration/consul/consul_service_registration_test.go

+			pass: false,
+		},
+		{
+			conf: map[string]string{"service_meta": "[{\"key\":\"value\"}]"},


How come the config must be a list? Since we're only taking the first of these values, I'd expect to be able to configure this as a set of JSON strings, e.g.

conf: map[string]string{"service_meta": "{\"key\":\"value\", \"bar\":\"baz\"}"},

VioletHynes · 2024-05-27T16:34:25Z

serviceregistration/consul/consul_service_registration.go

+		metaTagList := []map[string]string{}
+		err := json.Unmarshal([]byte(metaTagsJSON), &metaTagList)
+		if err != nil {
+			return nil, errors.New("service tags must be a dictionary of string keys and values")
+		}
+		metaTags = metaTagList[0]


As mentioned elsewhere, I'm a little confused why we're unmarshaling this into a list first and then taking the first element. Could this be changed so that we don't expect a list?

I don't recall why I made this choice in the context of the codebase I originally submitted this PR to, but agree it does not make any sense at all as in config, these values—as part of storage or service_registration— will be maps. Vaguely remember having trouble understanding if I should be using HCL blocks or objects.

I think my original intention was to allow something like the following config snippet

storage "consul" { service_meta = {some_key = "some value"} }

It's been a while, and I eventually ended up hacking around this shortcoming, so don't really have an informed opinion. Fixing.

VioletHynes · 2024-05-27T16:34:42Z

serviceregistration/consul/consul_service_registration.go

+		if logger.IsDebug() {
+			logger.Debug("config service_meta set", "service_meta", metaTags)
+		}


Suggested change

if logger.IsDebug() {

logger.Debug("config service_meta set", "service_meta", metaTags)

}

c.logger.Debug("config service_meta set", "service_meta", metaTags)

VioletHynes · 2024-05-27T16:35:14Z

website/content/docs/configuration/service-registration/consul.mdx

@@ -85,6 +85,9 @@ at Consul's service discovery layer.
 - `service_tags` `(string: "")` – Specifies a comma-separated list of case-sensitive
  tags to attach to the service registration in Consul.

+- `service_meta` `(map[string]string: {})` – Specifies a key-value list of meta tags to
+  attach to the service registration in Consul.


I think an example here would be great

website/content/docs/configuration/service-registration/consul.mdx

website/content/docs/configuration/storage/consul.mdx

VioletHynes · 2024-05-27T16:38:20Z

serviceregistration/consul/consul_service_registration.go

@@ -229,6 +232,22 @@ func (c *serviceRegistration) merge(conf map[string]string) error {
 		c.logger.Debug("config service_tags set", "service_tags", tags)
 	}

+  // Get user-defined meta tags to attach to the registered service name


The formatting looks a little strange here -- could you give this a make fmt/align this?

Yeah, I used github's UI to resolve the refactor of NewServiceRegistration / serverRegistration.merge and did not really pay attention, while I had hope, I was not really expecting someone to bring this PR back from the dead 😂 . Fixing!

VioletHynes · 2024-05-27T16:40:02Z

serviceregistration/consul/consul_service_registration.go

+		metaTagList := []map[string]string{}
+		err := json.Unmarshal([]byte(metaTagsJSON), &metaTagList)
+		if err != nil {
+			return nil, errors.New("service tags must be a dictionary of string keys and values")


It seems the code has changed around this, and this should just be
return errors.New("service tags must be a dictionary of string keys and values")

VioletHynes · 2024-05-27T16:43:59Z

Oh, one last thing, we would require a changelog for this. I'm happy to do it myself if you don't want to fiddle, but it'd be a file in the changelog directory called 11084.txt with something like the following text:

```release-note:improvement
serviceregistration: Added support for Consul ServiceMeta tags from config file from the new `service_meta` config field.
-```

(Delete the - preceding the three backticks, I couldn't get it to format properly otherwise. The other changelogs in that directory should show you the right formatting though)

VioletHynes

Looks great! Just a few more things before we can get this over the finish line. Thanks for the quick response so far!

VioletHynes · 2024-05-27T19:17:48Z

serviceregistration/consul/consul_service_registration.go

+		if err := json.Unmarshal([]byte(metaTagsJSON), &metaTags); err != nil {
+			return errors.New("service tags must be a dictionary of string keys and values")
+		}
+		metaTags["external-source"] = metaExternalSource


We should do this irrespective of if ok, since otherwise there will be a behaviour change as part of this PR. Previously, the tag would have been set, but now, it won't be set unless I provide my own meta tags.

To put it another way: if I don't provide my own service_meta, it should still be:

map[string]string{ "external-source": metaExternalSource, },

I'd suggest moving this line until after the if ok, that way the behaviour is preserved.

good catch, fixed!

VioletHynes · 2024-05-27T19:19:12Z

serviceregistration/consul/consul_service_registration_test.go

@@ -492,6 +492,43 @@ func TestConsul_serviceTags(t *testing.T) {
 	}
 }

+func TestConsul_serviceMeta(t *testing.T) {


Nit: We should have a godoc describing what this test does and is about

VioletHynes · 2024-05-27T19:19:18Z

serviceregistration/consul/consul_service_registration_test.go

@@ -492,6 +492,43 @@ func TestConsul_serviceTags(t *testing.T) {
 	}
 }

+func TestConsul_serviceMeta(t *testing.T) {


Suggested change

func TestConsul_serviceMeta(t *testing.T) {

func TestConsul_ServiceMeta(t *testing.T) {

VioletHynes · 2024-05-27T19:20:33Z

serviceregistration/consul/consul_service_registration_test.go

I'd love to see a test in some respect here that validates that if I don't provide my own service_meta, that it's still set to what it was before ("external-source": metaExternalSource)

One way we could do this is by improving TestConsul_ServiceMeta to test the resultant map for the three configurations, but I'm open to whatever you feel is easiest :)

probably a bad idea, let's see how it works scaffold tests

Co-authored-by: Violet Hynes <[email protected]>

VioletHynes

Thank you! I appreciate your patience for bearing with us on this, but thanks for being so understanding and great to work with. Approved! I'll look to get this merged as soon as all the CI passes etc.

vercel bot deployed to Preview – vault March 11, 2021 06:34 View deployment

vercel bot temporarily deployed to Preview – vault-storybook March 11, 2021 06:34 Inactive

vercel bot temporarily deployed to Preview – vault-storybook March 12, 2021 02:27 Inactive

vercel bot temporarily deployed to Preview – vault March 12, 2021 02:27 Inactive

sgmiller added the ecosystem label Apr 7, 2021

vercel bot deployed to Preview – vault June 28, 2021 18:06 View deployment

unRob mentioned this pull request Apr 25, 2023

serve "static" service records from kv prefix unRob/coredns-consul#1

Merged

unRob requested a review from yhyakuna as a code owner April 28, 2023 04:06

yhyakuna requested a review from hghaf099 May 1, 2023 17:29

unRob requested a review from a team as a code owner October 6, 2023 04:29

schavis added the content-lgtm Content changes approved. Merge depends on code review label Oct 9, 2023

schavis removed the request for review from yhyakuna October 9, 2023 19:54

schavis added the needs-eng-review Community PR waiting for ENG review label Oct 9, 2023

schavis removed the request for review from a team February 22, 2024 00:30

VioletHynes reviewed May 27, 2024

View reviewed changes

unRob requested a review from VioletHynes May 27, 2024 17:53

VioletHynes reviewed May 27, 2024

View reviewed changes

unRob and others added 6 commits May 27, 2024 13:40

Allow setting of Consul ServiceMeta tags from config file

095d4c9

probably a bad idea, let's see how it works scaffold tests

kick circleci

1ce7bbc

Add links to consul docs

ec9504b

Co-authored-by: Violet Hynes <[email protected]>

add changelog note

1ae7a63

use relative developer docs links

7ed3f7c

address feedback

63b6c5e

VioletHynes approved these changes May 27, 2024

View reviewed changes

please linter

1844eff

VioletHynes merged commit 6e72397 into hashicorp:main May 27, 2024
67 of 68 checks passed

unRob deleted the consul-meta-tags branch May 28, 2024 06:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow setting of Consul ServiceMeta tags from config file #11084

Allow setting of Consul ServiceMeta tags from config file #11084

unRob commented Mar 11, 2021 •

edited

Loading

hashicorp-cla commented Mar 11, 2021 •

edited

Loading

unRob commented Jun 24, 2023

VioletHynes commented May 27, 2024

VioletHynes left a comment

VioletHynes May 27, 2024

VioletHynes May 27, 2024

unRob May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

unRob May 27, 2024

VioletHynes May 27, 2024

VioletHynes commented May 27, 2024 •

edited

Loading

VioletHynes left a comment

VioletHynes May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

unRob May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

VioletHynes May 27, 2024

VioletHynes left a comment

	func TestConsul_serviceMeta(t *testing.T) {
	func TestConsul_ServiceMeta(t *testing.T) {

Allow setting of Consul ServiceMeta tags from config file #11084

Allow setting of Consul ServiceMeta tags from config file #11084

Conversation

unRob commented Mar 11, 2021 • edited Loading

hashicorp-cla commented Mar 11, 2021 • edited Loading

unRob commented Jun 24, 2023

VioletHynes commented May 27, 2024

VioletHynes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VioletHynes commented May 27, 2024 • edited Loading

VioletHynes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VioletHynes left a comment

Choose a reason for hiding this comment

unRob commented Mar 11, 2021 •

edited

Loading

hashicorp-cla commented Mar 11, 2021 •

edited

Loading

VioletHynes commented May 27, 2024 •

edited

Loading