ENH: Read Pillar files into OrderedDict to preserve source order #12161
Comments
|
As far as I know, we are not using OrderedDicts for basic template data, only for the topfile for pillar data. We'll investigate using them for all pillar data to get ordered iteritems. |
westurner
changed the title
|
I got almost the same kind of issues with my own formula of h2o web server: init.sls: {% from "h2o/map.jinja" import h2o with context -%}
h2o:
pkg.installed:
- name: {{ h2o.pkg }}
service.running:
- name: {{ h2o.service }}
file.serialize:
- name: /etc/h2o/{{ h2o.config_file }}
- user: root
- group: root
- mode: 644
- formatter: yaml
- dataset_pillar: h2o:config
- watch_in:
- service: h2opillar: h2o.sls h2o:
lookup:
pkg: h2o-git
config:
listen: 8080
user: http
access-log: /var/log/h2o-access.log
hosts:
"127.0.0.1.xip.io:8080":
paths:
/:
file.dir: /srv/http
file.dirlisting: 'ON'
expires: 1 dayrendered config: /etc/h2o/h2o.conf access-log: /var/log/h2o-access.log
hosts:
127.0.0.1.xip.io:8080:
paths:
/:
expires: 1 day
file.dir: /srv/http
file.dirlisting: 'ON'
listen: 8080
user: httpbut it should be:/etc/h2o/h2o.conf listen: 8080
user: http
access-log: /var/log/h2o-access.log
hosts:
"127.0.0.1.xip.io:8080":
paths:
/:
file.dir: /srv/http
file.dirlisting: 'ON'
expires: 1 day |
|
seeAlso: CPython enhancement 16991 (Python 3.5) https://bugs.python.org/issue16991#msg232825 "Add OrderedDict written in C" |
|
Should this issue be closed because issue #11599 is already closed? |
|
Good catch! This has indeed been implemented. |
|
Thanks!
|
|
I'm terribly sorry to say that, but this issue isn't still fixed. #11599 was addressed to order in pillar top file, not pillar values. Can we move to using OrderedDicts in pillar and states? |
|
I would love to see OrderedDicts everywhere as well. Ordered dicts are useful in a lot of cases where sequence matters, but you want to preserve the ability to override the value. I would think that this would be valuable for formula authors and end-users alike. |
|
Me too. OrderedDicts make the behaviour mor predictive and intuitive. |
|
Can this issue be re-opened? |
|
Someone have any news about this issue? I have the same on our company salt environment and already tried to disable sort_keys on config.sls but doesn't work. |
|
FYI We had problems when we tried to apply nginx state (with passenger) in Ubuntu 18.04 Bionic. The These setup can be used to another part of formula like Thanks for the moment. Regards, Gabriel Miranda |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue. |
|
Unstale. |
|
Thank you for updating this issue. It is no longer marked as stale. |
Before, `!!omap` nodes were incorrectly assumed to be mapping nodes
when `dictclass` was not `dict`. Now they are correctly processed as
sequences of single-entry mappings. The resulting Python object has
type `dictclass` (usually `collections.OrderedDict` or a subclass).
This commit does not change the behavior when `dictclass` is `dict`;
loading an `!!omap` node still returns a list of (key, value)
tuples (PyYAML's default behavior). I consider that to be a bug in
PyYAML, so a future commit may change the behavior in the `dict` case
to match the non-`dict` behavior. (This commit uses `dictclass` for
the non-`dict` case to match what appears to be the original
intention.)
Behavior before:
```pycon
>>> import salt.utils.yaml as y
>>> import collections
>>> y.load("!!omap [{foo: bar}, {baz: bif}]")
[('foo', 'bar'), ('baz', 'bif')]
>>> y.SaltYamlSafeLoader("!!omap [{foo: bar}, {baz: bif}]", dictclass=collections.OrderedDict).get_single_data()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "venv/lib/python3.8/site-packages/yaml/constructor.py", line 51, in get_single_data
return self.construct_document(node)
File "venv/lib/python3.8/site-packages/yaml/constructor.py", line 60, in construct_document
for dummy in generator:
File "salt/utils/yamlloader.py", line 42, in construct_yaml_omap
return (yield from self.construct_yaml_map(node))
File "salt/utils/yamlloader.py", line 36, in construct_yaml_map
value = self.construct_mapping(node)
File "salt/utils/yamlloader.py", line 52, in construct_mapping
raise ConstructorError(
yaml.constructor.ConstructorError: expected a mapping node, but found sequence
in "<unicode string>", line 1, column 1
```
Behavior after:
```pycon
>>> import salt.utils.yaml as y
>>> import collections
>>> y.load("!!omap [{foo: bar}, {baz: bif}]")
[('foo', 'bar'), ('baz', 'bif')]
>>> y.SaltYamlSafeLoader("!!omap [{foo: bar}, {baz: bif}]", dictclass=collections.OrderedDict).get_single_data()
OrderedDict([('foo', 'bar'), ('baz', 'bif')])
```
Relevant bug: saltstack#12161
Now the behavior is consistent regardless of the value of the
Loader's `dictclass` constructor parameter.
Starting with Python 3.6, Python `dict` objects always iterate in
insertion order, so iteration order was already guaranteed when using
an ordinary YAML map. However, `!!omap` provides a stronger guarantee
to users (plain map iteration order can be thought of as a Salt
implementation detail that might change in the future), and it allows
them to make their `.sls` files self-documenting. For example,
instead of:
```yaml
my_pillar_data:
key1: val1
key2: val2
```
users can now do:
```yaml
my_pillar_data: !!omap
- key1: val1
- key2: val2
```
to make it clear to readers that the entries are intended to be
processed in order.
Behavior before:
```pycon
>>> import salt.utils.yaml as y
>>> import collections
>>> y.load("!!omap [{foo: bar}, {baz: bif}]")
[('foo', 'bar'), ('baz', 'bif')]
>>> y.SaltYamlSafeLoader("!!omap [{foo: bar}, {baz: bif}]", dictclass=collections.OrderedDict).get_single_data()
OrderedDict([('foo', 'bar'), ('baz', 'bif')])
```
Behavior after:
```pycon
>>> import salt.utils.yaml as y
>>> import collections
>>> y.load("!!omap [{foo: bar}, {baz: bif}]")
OrderedDict([('foo', 'bar'), ('baz', 'bif')])
>>> y.SaltYamlSafeLoader("!!omap [{foo: bar}, {baz: bif}]", dictclass=collections.OrderedDict).get_single_data()
OrderedDict([('foo', 'bar'), ('baz', 'bif')])
```
Relevant bug: saltstack#12161
|
@westurner Closing this due to age, the old version of Salt and Python 2. Especially given the handling of Ordered Dictionaries in Python 3. |
It would be great if .iteritems of pillar data read out in source order.
Example:
Pillar Input: https://github.com/saltstack-formulas/openssh-formula/blob/master/pillar.example
Template: https://github.com/saltstack-formulas/openssh-formula/blob/master/openssh/files/sshd_config
Output: /etc/ssh/sshd_config
The text was updated successfully, but these errors were encountered: