Remove recurse_list from pillar_source_merging_strategy and add pilla… #30062

seanjnkns · 2015-12-29T17:33:34Z

PR for #29601 that adds a new master configuration option of pillar_merge_lists: bool (default True) to readd backwards compatible of merging lists, while still maintaining the option to overwrite them.

…r_merge_list (bool) instead

cachedout · 2015-12-29T17:49:22Z

salt/utils/dictupdate.py

@@ -17,7 +17,7 @@
 log = logging.getLogger(__name__)


-def update(dest, upd, recursive_update=True, merge_lists=False):
+def update(dest, upd, recursive_update=True, merge_lists=True):


The modification of these default values concerns me a little bit because I'm worried that it might regress other functions that use this one. Is this necessary?

I agree with @cachedout's comment.

@s0undt3ch Given that commit 4af5b5c added this option and set the default of merge_lists=False, a non-existent option until that commit, setting it to True by default would in essence, revert things back to the original functionality where lists were merged unless otherwise so chosen via a new configuration option of pillar_merge_lists in the master config. Leaving this at False is much more likely to cause regressions imo, than leaving only some changes set back to True or in other words, the behavior prior to commit 4af5b5c.

Sorry if my comment is a bit confusing.

Nope, that clarified it for me and the reasoning is good. Thanks!

cachedout · 2015-12-29T17:52:16Z

tests/unit/pillar_test.py

@@ -130,38 +130,6 @@ def test_topfile_order(self, Matcher, get_file_client):
        pillar = salt.pillar.Pillar(opts, grains, 'mocked-minion', 'base')
        self.assertEqual(pillar.compile_pillar()['ssh'], 'foo')

-    @patch('salt.pillar.salt.fileclient.get_file_client', autospec=True)


Could you explain why these tests are removed? I'd definitely like to keep them around if possible.

Commit 4af5b5c initially added the merge_lists=bool argument which caused the regression noted in #29601. This simply would set it back to True, as if it wasn't present to begin with and instead adds a new master_config opt of pillar_merge_lists (default True). In this way, it fixes the initially regression while still allowing someone to opt to not merge the lists by setting pillar_merge_lists: False in /etc/salt/master

They were dropped because there'd be no pillar_source_merging_strategy of recurse_list as it would no longer be necessary.

merge_lists=True doesn't seem to be tested here at all, or did I missed something out?

seanjnkns · 2015-12-29T17:56:28Z

Commit 4af5b5c initially added the merge_lists=bool argument which caused the regression noted in #29601. This simply would set it back to True, as if it wasn't present to begin with and instead adds a new master_config opt of pillar_merge_lists (default True). In this way, it fixes the initially regression while still allowing someone to opt to not merge the lists by setting pillar_merge_lists: False in /etc/salt/master

cachedout · 2015-12-29T18:30:45Z

Thanks for the reply here.

So far, this looks pretty good but I definitely want to bring some additional people in for review on this one.

I'm thinking of @ryan-lane, @terminalmage, @basepi and @DmitryKuzmenko as people I'd like to look at this. @thatch45 @jacksontj and @s0undt3ch might also want to weigh in.

If you wouldn't mind being a little patient on this one, @seanjnkns, we'll get this in. I just want to make sure we get a variety of opinions here when we're dealing with merge strategies, since they can have far-reaching consequences.

ryan-lane · 2015-12-29T18:34:11Z

Does this change the default behavior of the current stable release? If so, can we ensure it keeps whatever the current stable behavior is?

seanjnkns · 2015-12-29T18:36:20Z

Absolutely. I'm just glad it's being reviewed and considered. There may be a better solution for this, I just know that the aforementioned commit broke backwards compatibility and although I understand it's merits, ideally users should be able to use any of the merge strategies of their choosing instead of being forced to use the recurse_list.

seanjnkns · 2015-12-29T18:40:04Z

@ryan-lane, this would change the default behavior of the stable release by undoing the aforementioned commit and make it an option if you want to merge_lists or not, but in turn fix the regression that that commit created. For those using the recurse_list merge strategy, it would fall back to recurse so short of a note in their master logs, nothing would be broken or have unintended consequences. Just the few people expecting to NOT merge lists would just need to set pillar_merge_lists: False in the master config.

ryan-lane · 2015-12-29T19:03:49Z

When did the regression appear? Was it in a stable release, or during a point release in stable? What I'd really like to avoid is:

2014.7: lists merge by default
2015.8: lists don't merge by default
2016.1: lists merge by default

For anyone that's upgrading across releases it's going to be a nightmare, because you go to upgrade and some things are subtly broken, then you figure out why, change your code, then upgrade again and now things are subtly broken again.

seanjnkns · 2015-12-29T19:17:57Z

@ryan-lane commit 4af5b5c was introduced in 2015.8.3 so people upgrading from 2015.8.1 to 2015.8.3, like myself, were subtly broken as noted in #29601. This would essentially revert things back to the way they were with 2015.8.1 while still retaining the benefits of commit 4af5b5c. Several issues that requested the initial recurse_list were #27378, #25954, and #28394; possibly others. They'd still get the same benefit from that commit, just would have to set pillar_merge_lists to False.

ryan-lane · 2015-12-29T19:57:11Z

Ouch. That's pretty rough. Yeah, it seems that it should be fixed the way you suggest in the point release.

DmitryKuzmenko · 2015-12-30T08:37:59Z

@cachedout, the PR looks good for me and @seanjnkns arguments sounds reasonable.

DmitryKuzmenko · 2015-12-30T08:42:34Z

@seanjnkns btw, there is a lint error:
salt/modules/config.py:362: [C0303(trailing-whitespace), ] Trailing whitespace
Could you please fix it?
Also I checked the unit tests errors, they look unrelated.

seanjnkns · 2015-12-31T16:21:07Z

Anything further I can do to move this along to ensure it gets in the next release?

basepi · 2016-01-04T20:48:14Z

I was involved in the decision to change merge_lists to False by default. It was decided at that point that while merging dicts was an intuitive behavior, merging lists was not.

See #27890 and #25358 (comment)

I continue to be of the opinion that merging lists is the edge case, not the intuitive default case. However, I misunderstood the original introduction of that feature. I thought it was just adding a recurse_list config value, not a whole new merging strategy. I am of the opinion there should just be one merge strategy for recurse, with a separate config value, recurse_list, which takes a boolean value and decides whether to merge the lists. This should be False by default.

Your pillar_merge_lists value does exactly this. However, I think it should be False by default. I've had many more people state that merging lists should not be the default than I've heard for it.

cachedout · 2016-01-05T18:30:06Z

@seanjnkns Did you see the above comments from @basepi ? We'll want to come to a consensus as much as is possible before we get this merged and the release is coming up quickly. :]

seanjnkns · 2016-01-05T19:26:13Z

@cachedout @basepi, if the decision is to make the merging of lists be false by default, then perhaps there's no need for this PR and just have people use the merge strategy of recurse_list which is just an extension of the recurse merge strategy. If we went that route, I think the only thing that would need to be updated is the /etc/salt/master to mention the recurse_list merging strategy, much like is already mentioned in the docs.

Otherwise, if we want False as the default, I can submit a new commit that sets the default for pillar_merge_lists to False along with the merge_lists in the defs. I can be persuaded either way, just let me know which is the preferred method.

basepi · 2016-01-06T00:08:10Z

The reason this is still a good fix is we want people to be able to use smart, but control whether the lists are merged. I seem to recall someone commenting that they wanted this use case, but I cannot for the life of me find the comment now....

seanjnkns · 2016-01-06T17:00:30Z

Set (pillar_)merge_lists to default to False. Let me know if this satisfies the request and if there's any further changes needed.

basepi · 2016-01-06T17:27:32Z

This looks perfect! Thanks again @seanjnkns !

Remove recurse_list from pillar_source_merging_strategy and add pilla…

bradthurber · 2016-01-06T18:35:04Z

@basepi are there plans to backport this merge into 2015.8 as well?

aboe76 · 2016-01-06T19:25:17Z

@basepi will this also be configured in the minion? some of us run masterless....

basepi · 2016-01-19T20:58:28Z

Ah we should definitely backport this. The change went into 2015.8.

basepi · 2016-01-19T20:59:47Z

@aboe76 This should apply to any entity which compiles pillar, I think it should "just work" in masterless.

rallytime · 2016-01-19T22:08:56Z

Back-port done in #30458

Back-port #30062 to 2015.8

arnisoph · 2016-01-23T19:19:29Z

doc/ref/configuration/master.rst

+
+.. versionadded:: 2015.8.0
+
+Default: ``True``


the default is False, isn't it?

Correct. @seanjnkns would you consider opening a quick PR to fix that mistake in the docs? 2015.8 would be a good target, since this was backported to 2015.8.

@basepi, I did PRs 30602 and 30609 this morning to address this which has already been merged.

Perfect, thank you!

anlutro · 2016-02-02T14:19:33Z

@seanjnkns why is False in opts.get('pillar_merge_lists', 'False') wrapped in quotes?

basepi · 2016-02-02T19:54:04Z

Good catch. I've merged your pull request.

Issue: saltstack#30809 Related PRs: saltstack#30062 saltstack#30458

Remove recurse_list from pillar_source_merging_strategy and add pilla…

41353b7

…r_merge_list (bool) instead

cachedout reviewed Dec 29, 2015
View reviewed changes

seanjnkns mentioned this pull request Dec 29, 2015

pillars not merging properly with 2015.8.3 #29601

Closed

cachedout reviewed Dec 29, 2015
View reviewed changes

Sean Jenkins added 4 commits December 30, 2015 08:21

Merge remote-tracking branch 'origin/develop' into sean

f6a7aec

Fix lint for PR30062

0ffea31

Merge remote-tracking branch 'origin/develop' into sean

1d7ee7b

Merge remote-tracking branch 'upstream/develop' into develop

fb6e895

Set (pillar_)merge_lists to default for PR#30062

8fc656b

basepi added a commit that referenced this pull request Jan 6, 2016

Merge pull request #30062 from seanjnkns/develop

f476c75

Remove recurse_list from pillar_source_merging_strategy and add pilla…

basepi merged commit f476c75 into saltstack:develop Jan 6, 2016

jfindlay added the Release-Notes label Jan 7, 2016

jfindlay mentioned this pull request Jan 11, 2016

pillar data leaks through environments #29421

Closed

basepi added the bugfix-bckport will be be back-ported to an older release branch by creating a PR against that branch label Jan 19, 2016

rallytime mentioned this pull request Jan 19, 2016

Back-port #30062 to 2015.8 #30458

Merged

rallytime added ZZZ[Done]-back-ported-bf RETIRED The pull request has been back-ported to an older branch. and removed bugfix-bckport will be be back-ported to an older release branch by creating a PR against that branch labels Jan 19, 2016

cachedout pushed a commit that referenced this pull request Jan 20, 2016

Merge pull request #30458 from rallytime/bp-30062

73f372d

Back-port #30062 to 2015.8

arnisoph reviewed Jan 23, 2016
View reviewed changes

This was referenced Jan 25, 2016

Fix documentation for pillar_merge_lists which default is False, not … #30602

Merged

Fix documentation for pillar_merge_lists which default is False, not … #30609

Merged

anlutro mentioned this pull request Feb 2, 2016

Master configuration "pillar_merge_lists" has no effect #30809

Closed

anlutro mentioned this pull request Feb 2, 2016

Properly set the default value for pillar_merge_lists #30813

Merged

basepi pushed a commit to basepi/salt that referenced this pull request Feb 6, 2016

Properly set the default value for pillar_merge_lists

2243f25

Issue: saltstack#30809 Related PRs: saltstack#30062 saltstack#30458

Remove recurse_list from pillar_source_merging_strategy and add pilla… #30062

Remove recurse_list from pillar_source_merging_strategy and add pilla… #30062

Conversation

seanjnkns commented Dec 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanjnkns commented Dec 29, 2015

cachedout commented Dec 29, 2015

ryan-lane commented Dec 29, 2015

seanjnkns commented Dec 29, 2015

seanjnkns commented Dec 29, 2015

ryan-lane commented Dec 29, 2015

seanjnkns commented Dec 29, 2015

ryan-lane commented Dec 29, 2015

DmitryKuzmenko commented Dec 30, 2015

DmitryKuzmenko commented Dec 30, 2015

seanjnkns commented Dec 31, 2015

basepi commented Jan 4, 2016

cachedout commented Jan 5, 2016

seanjnkns commented Jan 5, 2016

basepi commented Jan 6, 2016

seanjnkns commented Jan 6, 2016

basepi commented Jan 6, 2016

bradthurber commented Jan 6, 2016

aboe76 commented Jan 6, 2016

basepi commented Jan 19, 2016

basepi commented Jan 19, 2016

rallytime commented Jan 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anlutro commented Feb 2, 2016

basepi commented Feb 2, 2016