Impossible to retrieve external pillars with pillar.get when ext_pillar_first is True

[AndreiPashkin]

Description of Issue/Question

ext_pillar_first master settings is supposed to make user able to reference external pillars from regular ones. At least this is my understanding and I'm not sure if it's correct.

The issue is that while this setting makes possible to reference external pillars from regular ones using pillar context variable, it is still impossible to reference them with function pillar.get.

I'm actually not really sure if it's a bug or I'm missing something.

Setup

Master config

ext_pillar_first: True
ext_pillar:
  - cmd_json: 'echo {\"foo\":\"bar\"}'
# Add other necessary settings, like file_roots, pillar_roots, etc.

Pillar top

base:
  '*':
    - main

main.sls

qux: {{ salt['pillar.get']('foo') }}
spam: {{ pillar.get('foo') }}

Steps to Reproduce Issue

1. Execute: salt '*' pillar.items
2. The output would be:

<minion name>:
    ----------
    foo:
        bar
    spam:
        bar
    qux:
        None

3. Observe, that spam variable, value for which was obtained by accessing pillar, is set correctly. But qux variable, for which pillar.get call was used, is None.

Versions Report

Salt Version:
           Salt: 2016.11.0-415-g843ceac
 
Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: 2.0.0
      gitpython: 2.1.0
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul  1 2016, 15:12:24)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 16.0.0
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.5
 
System Versions:
           dist: LinuxMint 18 sarah
        machine: x86_64
        release: 4.4.0-47-generic
         system: Linux
        version: LinuxMint 18 sarah

Related issues

#6955
#37003

[AndreiPashkin]

I guess, that the person, who introduced ext_pillar_first setting, @sylphid8, might be interested in participating.

[gtmanfred]

From the documentation it says that the ext_pillar_first is only used for the targeting pillars in the top file. It does not mention being able to reference them inside other pillars.

[AndreiPashkin]

@gtmanfred, hm, actually you are right. Strange, that I had strong expectation, that it should work with salt['pillar.get']() as well.

[ignatk]

Somehow I got this working: in my setup I'm able to access external pillars from pillars via opts dictionary. So, neither salt['pillar.get']() nor pillar.get('foo') works, but spam: {{ opts['pillar']['foo']['bar'] }} in pillar sls files and even pure jinja templates works OK.

However, it does not work from pillar top file, but top files can match by external pillars anyway, so should not be a big problem.

[seanjnkns]

Actually, and I can spin off another issue if necessary as I'm not intending to hijack this thread, but {{ salt['pillar.get']('foo') }}, or {{ pillar['foo'] }} or any variation thus far I've tried even in the /srv/pillar/top.sls fails. The most I've been able to get is either None, or via a pillar.get('foo', 'meh') get it to say meh cause the pillar.get fails to retrieve anything.

In my case, I'm actually pulling a number of pieces of information from a Galera cluster based on minion_id and then populating some pillar['metadata'] dicts which I then utilize for various states and additional pillar (ex/in)clusions. The only thing I've gotten to work currently is using the {{ opts['pillar']['foo']['bar'] }} solution in a file off of top.sls as @secumod mentioned.

Salt Version:
Salt: 2016.11.3

Dependency Versions:
cffi: 0.8.6
cherrypy: 3.2.2
dateutil: 1.5
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: 1.4.3
M2Crypto: 0.21.1
Mako: 0.8.1
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: 1.2.5
pycparser: 2.14
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.5 (default, Sep 15 2016, 22:37:39)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.2.1511 Core
machine: x86_64
release: 4.9.20-1.el7.centos.x86_64
system: Linux
version: CentOS Linux 7.2.1511 Core

[gtmanfred]

Again, ext_pillar_first is only used for targeting pillars in the top file. Not referencing them in jinja in the top file.

So what you are doing is the correct way to handle this.

Closing this issue.

Thanks,
Daniel

[seanjnkns]

@gtmanfred, I'm not sure you understood me. If I try to use salt.get, pillar[], pillar.get, etc from the top file to call another pillar file, I get various errors, or None. And given the lack of documentation about ext_pillar_first or example usage, I'm not sure that I'm necessarily doing anything wrong and that there isn't a bug in ext_pillar_first itself.

Again, I'm happy to open another bug report on this even if I think the issue is related.

[ignatk]

Also, just to clarify: my option of opts['pillar']['foo']['bar'] is not documented at all. I just arrived at it by try-and-see and a little bit of reading the source.

[AndreiPashkin]

@gtmanfred, hi, look at this comment:
#42383 (comment)

This issue became relevant again, because the documentation changed and the phrase about targeting was removed.