Feature request: extend functionality in pillar .sls

[Zogg]

Currently defining defining pillar key the second time will overwrite the contents of the first definition, i.e.:

first.sls:
data:
  love

second.sls:
data:
  hate

top.sls:
base:
 "*":
   - first
   - second

The result pillar:

data:
  hate

Proposition A

As the pillar definitions are pure data, there is no need to have unique IDs, or unique keys at all.

• If there are overlapping keys defined, merge their contents by default:

first.sls:
data:
  love

second.sls:
data:
  hate

resulting pillar:

data:
  love
  hate

• Have keyword reserved to override default behavior and have second definition replace the first one
• Have keyword reserved to have a second definition exclude data from the first one

Proposition B

Pillar is data, data is a set. Implement basic set operations: union, intersection, difference for overlapping data definitions.

[thatch45]

This can get very complicated because pillar data is not structured like state data, but it should be a viable thing to add, I have set this to 0.15.0, but it might get bumped if it proves too complicated

[inthecloud247]

+1*10^3 :-)

[alexy]

Default should be simple, and the current behavior -- overwrite -- is fine for the default. At least you can control it with the order of include's. Union will change arity of the data, which becomes problematic for nested structures.

Let's define a uniform convention and have tests which will demonstrate each case. Then the tests can serve as the docs on include/extend behaviors.

[Mrten]

ran into this today, wanted to extend a network setting pillar with another one derived from the first and was puzzled why the data was missing.
+1 from me!

[eugenea]

+1 Please make some way of mixing data structures. Otherwise it is impossible to manage complicated structures such as access lists without lots of CUT & PASTE.

[ConsoleCatzirl]

Another +1 here. I tend to use the top level pillar object name for scoping the pillar data and would like to be able to extend the object with more data from another pillar file.

[inthecloud247]

pillar of pillars :-)

[awerner]

+1 from me, avoids tedious copying/maintenance of long lists (e.g. users) with only some different entries.

[boltronics]

+1. Forcing us to have unique IDs is making our state configuration messy and difficult to maintain.

[thatch45]

Yes, I am sorry we keep pushing this off, we will get it in!

[l2ol33rt]

This seems to be a duplicate of an older issue I posted #2466. But I am still really looking forward to this being added in 17.0!

[johnnoone]

I posted a proposal about include/extend issues on #2466. Eventually we can use the same logic for top files ?

[anitakrueger]

+1 forced to use unique id's is giving me a long complicated pillar dict for a server. and the state configuration now needs to look for different keys, where it could just loop through one if pillar dicts were extensible.

[luizsilva]

+1 If pillars could have "defaults" then my life would be easier.

[eugenea]

+1

[redbaron]

why don't use Jinja {%- include 'common.sls' -%} ?

[eugenea]

redbaron you can include without jinja. It will still not merge namespaces.

[davidjb]

+1 for Pillar files with some form of shared/extended namespaces.

[kriberg]

+1

[garethgreenaway]

To reiterate what I mentioned in the referenced issue above, what I'm looking to do by extending and replacing Pillar data is this: https://gist.github.com/garethgreenaway/7237050

I want common data available to all hosts but have the ability to overwrite those values on a host by host basis.

[axiom]

Yes exactly, right now I have to do a bunch of copy-and-paste just to "override" one value.

[ruimarinho]

@garethgreenaway @axiom I'm doing something similar by using include with defaults. I think you can do something like this:

common.sls:

openssh:
  keys:
    gareth:
      allow: {{ allow_gareth }}
    root:
      allow: {{ allow_root }}

dns1.sls:

include:
  - common:
      defaults:
        allow_gareth: True
        allow_root: True

This was the workaround I was able to create until we get a proper extend implementation :)

[ConsoleCatzirl]

Oh, that's an interesting work-around, @ruimarinho. That'll definitely help me out until we get extend implemented.

[dara0418]

@ruimarinho
+1 Good work around

[galet]

+1

[cp-richard]

While you are waiting for this to be resolved, you can achieve the same effect (default.yaml, merged with lower down yaml) by setting pillar_source_merging_strategy: aggregate in master config, and then in each yaml, lead with #!yamlex on the top, then on each leaf node you want to support aggregation, add a !aggregate stanza.

This was detailed a bit earlier in the thread, but some extra details were missing (adding !aggregate to each leaf for example).

Works like a charm. It solved the problem that many in this thread are looking for. Obviously there is no way to remove data, just change or append. Having to use !aggregate on each leaf is a problem for some yaml parsers. It would be nice for this to just be a native solution at some point similar to hiera with deep_merge.

ie.. default.yaml

#!yamlex
users: !aggregate
  bob:
    user_data: !aggregate
      fullname: 'Bob Flarn'
      uid: 5001
      gid: 5001
      gid_from_name: True
      optional_groups: !aggregate
        - sudo
      shell: /bin/zsh

Then later in someother.yaml

#!yamlex
users: !aggregate
  bob:
    user_data: !aggregate
      optional_groups: !aggregate
        - someothergroup
  fran:
    user_data: !aggregate
      fullname: 'Fran Foobar'
      uid: 5002
      gid: 5002
      gid_from_name: True
      optional_groups: !aggregate
        - sudo

[danlsgiga]

The yamlex parser is not a good solution for this. I've tried to use it and it has its problems along with the fact that maintaining leaf level tags in the pillars sucks badly. I know the recurse_list merging strategy has been added to the dictupdate util. This will basically provide the features for deep_merging. Don't know if it will be in 2015.8.2 or in the next release though.
https://github.com/saltstack/salt/blob/develop/salt/utils/dictupdate.py

[cp-richard]

I agree having to annotate every leaf is a pain and hard to track down problems if you forget. Again, for me this is a working solution until the proper one is deployed - I couldn't wait, and this request goes back to 2014 already.

[uvsmtid]

Functionality for "deep merge" of dictionaries is already implemented.
Correct me if I'm wrong.

At least, (1) the example in the initial post for this issue and (2) example of this SO question are essentially the same:
http://stackoverflow.com/questions/24631231/how-to-join-two-salt-pillar-files-and-merge-data

So, in case of key-value dictionaries, the solution is in place by default at least since version 2014.7.0 using no keywords like extend/include at all.

Now, merging lists is completely different and less straightforward problem which:

• (A) is not mentioned in the initial post (and, therefore, irrelevant);
• (B) has a separate issue Merging a list in pillars #28394 (which, IMHO, was wrongly closed and I'll comment on it there).

Shouldn't this issue be closed?

[uvsmtid]

Similar problem exists when configured pillars are merged with are overridden by pillar argument supported by some functions on the command line - see #29516.

Merging has to be consistent regardless of pillar data source (configuration or command line).

[developerinlondon]

+1 for a deep merge please!

[Talkless]

I am also missing list mergeing feature. My use case:

# logcheck.sls:
logcheck:
  logfiles:
     - '/some/defaut/log.log'

Now, if I want to deploy apache to some machines, I add apache.sls state and apache/init.sls and apache/logcheck.sls pillars for that machines top files.
apache/logcheck.sls could be like this:

#apache/logcheck.sls
logcheck:
  logfiles:
    - '/var/log/apache/error.log`

And I somehow expected that these two lists would merge. I guess I have to add dummy dicts:

# logcheck.sls:
logcheck:
  logfiles:
    some_log:
      - '/some/defaut/log.log'

#apache/logcheck.sls
logcheck:
  logfiles:
    apache:
      - '/var/log/apache/error.log`

Now logcheck:logfiles dict is merged, just need change template to iterate through these internal lists...

[uvsmtid]

@Talkless, I think you can rely solely on merging dicts.

You already mentioned "dummy dicts".
However, I want to make a slight touch (to simplify it syntactically).

Input

• logcheck.sls:

logcheck:
    logfiles:
        some_log: '/some/defaut/log.log'

• apache/logcheck.sls:

logcheck:
    logfiles:
        apache: '/var/log/apache/error.log'

Output

The result will be:

logcheck:
  logfiles:
    some_log: '/some/defaut/log.log'
    apache: '/var/log/apache/error.log'

Convenience

It is easily loopable list(!) of values():

{% for log_path in pillar['logcheck']['logfiles'].values() %}
...
{% endfor %}

Is it suitable in your case?

The majority of "merge list" use cases are easily implemented through dicts.
That's why I even proposed to close this issue above.

The only exception is when order of items listed is important - see example here. I bet in such cases nobody wants to split lists across files and then merge them somehow in a less deterministic way (instead, it is much clearer to have a single file with one large single list - the problem of merging it does not exist in this case).

[Talkless]

Thank you @uvsmtid , it's nicer to use with values() . Good to know.

[paul-mulvihill]

+1
I'm using Pillar data to set a grain which defines what states are run.
having the ability to inherit this by it combining the pillar entries would be really handy.
I have the following pillar definitions
File1 (server group)
grains:
appliedstates:
- state1
- state2

File2: (server specific)
grains:
appliedstates:
- state3

This way just ends up with the final pillar containing state3. This may be an odd way of using it but using the Pillars to set states gives me a lot more control for automating the configuration of this.
and if i can set things higher and have it filter down would save a lot of duplication..

[genuss]

Guys, the external pillar which provides this functionality is already included in 2016.3 release https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.stack.html !

[mlalpho]

Thanks @genuss! Will take a look at pillarstack. This could save tons of jinja effort :)

[paul-mulvihill]

Thanks @genuss. reading through the page it looks like it would do what i want.. although one thing I'm unsure about is the following line in the PillarStack Configuration Files section
"The path of these yaml files must be relative to the directory of the PillarStack config file."
This would imply that it only works if the pillar yaml files are locally on the salt master with the pillarstack config file?
I'm using the gitfs option to share my state/pillar files accross multiple masters. Any one have PillarStack working with gitfs? or know that it definitely is possible?

[ghost]

PillarStack (and reclass and suchlike) is absolutely overengineered when it comes to simple and intuitive application of pillar data aggregation in relation to merging lists. Using another complex module -- that also requires extra configuration -- just to butter over the lack of a basic future in the default pillar system is just a messy workaround. It's not a good solution.

@uvsmtid 's suggestion to just use dicts is also just a hack because there are actually reasons to lay out data as a list. I.e. templating a list of values in some sort of config file

services:
  - ssh
  - nscd
  - pam

is just $thatlist | join(", ") in Jinja; exchanging that and laying out the same thing as a dictionary with essentially unused keys disturbs both the structured configuration format Salt is all about and adds extra loops to the templating code.

Just merging the list structures in the default pillar system appears to be the most simplest and only reasonable solution.

[mlalpho]

@paul-mulvihill, I'm wondering if you could use salt:// paths that would allow PillarStack to read from git clone? I'd be interested to see the two working together.

[paul-mulvihill]

@mlalpho not sure how well that would work for the setup i am going for.. I'm intending to have the top.sls file become an automatically generated file as well down the line so given the pillarstack lives on the master that wouldn't work. I have a system of records that i'm aiming to create most of the pillar files and then potentially this one as the hierarchy expands.

[mlalpho]

The salt/top.sls or pillar/top.sls ?

From what I've read the PillarStack would have access to the pillar variable, so if you have some things defined in your git pillar, you can operate with them using jinja within PillarStack, which, is something you can't do with normal salt pillars at the moment without using jinja templating and loading yaml, json etc in to a jinja var. Aside from the merging of pillar data as a whole, that's already a big win in my opinion.

[paul-mulvihill]

I'm refering to the pillar top.sls. my setup is gitfs for everything with seperate repos for states and pillars. Eventually i'm goign to have our system of records generate the top.sls file for pillar information,
In my setup the state\top.sls reads the "appliedstate" grain I define and uses it's values to apply states.
I want to avoid any key configuration being installed on the masters except for telling them to look at gitfs. I'd rather changes to the pillar hierarchy or values are purely within the repo.

[mrichar1]

By using PillarStack you do get nice merging strategies, but you lose out on being able to use them with other ext_pillar sources, like git_pillar or pillar_ldap.

It would be nice if the merging strategies that PillarStack offers were part of the 'core' pillar code.

[paul-mulvihill]

:( guessed it would be something like that.. for the scale i'm building a salt estate I need git as the backend, so i'll have to find a work around until we can get this sort of features in the core code.

[danlsgiga]

You can have this very same feature using the slsutil.merge module. Here is a snippet to place in your map.jinja:

# vim: sts=2 ts=2 sw=2 et ai

{# Start with defaults from defaults.yml #}
{%- import_yaml 'nginx/proxy/web/defaults.yml' as default_settings %}

{# Only add key:values here that differ from whats in defaults.yml #}
{%- set os_map = salt['grains.filter_by']({
       'Debian': {},
       'RedHat': {
         'pkg': {
           'name': [
               'nginx',
            ],
         },
         'service': {
           'name': 'nginx',
           'state': 'running',
           'enable': True,
         },
         'user': 'nginx',
         'group': 'nginx',
         'conf': '/etc/nginx/nginx.conf',
         'confd_dir': '/etc/nginx/conf.d',
         'certs_dir': '/etc/nginx/conf.d/certs',
         'vhosts_dir': '/etc/nginx/conf.d/vhosts',
         'docroot': '/usr/share/nginx/html',
         'pool': salt['grains.get']('pool', 'default'),
       },
     },
     grain="os_family"
   )
%}

{# Merge the os_map and the pillar data to the default_settings #}
{%- set nginx = salt['slsutil.merge'](
                  salt['slsutil.merge'](
                      os_map,
                      default_settings.nginx.proxy.web,
                      merge_lists=True
                    ),
                    salt['pillar.get'](
                      'nginx:proxy:web', {}
                    ),
                    merge_lists=True
                )
%}

With that you'll have a full merged dict including lists.

[bbinet]

PillarStack author here: personally I'm also storing my pillar and states data in git (on bitbucket private repositories), and I'm using a docker container to automatically update my pillar and states tree when a new commit is pushed to bitbucket (new commits trigger POST web hook requests to my docker container which update the relevant pillar or states tree).

Here is the docker container I'm using:
https://github.com/bbinet/docker-hooked-git-workdir

[sdemura]

Ran into this today. Would love to see this implemented at some point!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[hramrach]

AFAICT it's still broken, although not deterministically.

Sometimes things are merged, sometimes replaced. There is no obvious pattern.