LDAP state module improvements

changelog/62791.fixed

@@ -0,0 +1,12 @@
+Improvements to `salt.state.ldap` and `salt.modules.ldap3`:
+  * The order of an attribute's values is now preserved when reading from and
+    writing to the LDAP server.
+  * The order of an attribute's values is now ignored when checking value
+    equality, as required by RFC 4511.
+  * Fixed attribute value encoding/decoding corner cases.
+  * Worked around a flaw in the way `python-ldap` adds new attribute values (the
+    flaw only affects entries where attribute value changes can trigger server
+    behavior changes, such as some of OpenLDAP's `cn=config` entries).
+  * Fixed support for `bytes` attribute values that are not in a list.
+  * Documentation improvements.
+  * Miscellaneous code cleanups.

changelog/62932.fixed

@@ -0,0 +1,18 @@
+Improvements to YAML processing (`salt.utils.yaml`):
+  * Loading an `!!omap` node now always returns a `collections.OrderedMap`
+    object.  Before it would sometimes return a list of (key, value) tuples
+    and sometimes raise an exception.
+  * Dumping a `collections.OrderedMap` now consistently produces an `!!omap`
+    node (a sequence of single-entry mappings).
+  * Loading a sequence node tagged with `!!python/tuple` is now supported, and
+    produces a Python `tuple` object.
+  * Dumping a `tuple` object now consistently produces a sequence node
+    explicitly tagged with `!!python/tuple`.
+  * Loading an explicitly tagged `!!timestamp` node now produces a
+    `datetime.datetime` object instead of a string.
+  * Dumping a `datetime.datetime` object now explicitly tags the node with
+    `!!timestamp`.
+  * `salt.utils.yaml.dump()` now defaults to `salt.utils.yaml.OrderedDumper`
+    instead of `yaml.Dumper`.
+  * Fixed indentation in `salt.utils.yaml.IndentedSafeOrderedDumper` output.
+  * Code health clean-ups.

doc/topics/troubleshooting/yaml_idiosyncrasies.rst

@@ -382,50 +382,126 @@ Here's an example:
 Automatic ``datetime`` conversion
 =================================
 
-If there is a value in a YAML file formatted ``2014-01-20 14:23:23`` or
-similar, YAML will automatically convert this to a Python ``datetime`` object.
-These objects are not msgpack serializable, and so may break core salt
-functionality.  If values such as these are needed in a salt YAML file
-(specifically a configuration file), they should be formatted with surrounding
-strings to force YAML to serialize them as strings:
+.. versionchanged:: 2018.3.0
+
+    A YAML scalar node containing a timestamp now always produces a string.
+    Previously, Salt would attempt to create a Python ``datetime.datetime``
+    object, even if the node contained an invalid date (for example,
+    ``4017-16-20``).
+
+.. versionchanged:: 3006.0
+
+    Loading a YAML ``!!timestamp`` node now produces a ``datetime.datetime``
+    object.  Previously, nodes tagged with ``!!timestamp`` produced strings.
+
+.. versionchanged:: 3006.0
+
+    Dumping a ``datetime.datetime`` object to YAML now explicitly tags the node
+    with ``!!timestamp``.  Previously the ``!!timestamp`` tag was omitted.
+
+Salt overrides PyYAML's default behavior and loads YAML nodes that look like
+timestamps as strings:
 
 .. code-block:: pycon
 
-    >>> import yaml
-    >>> yaml.safe_load("2014-01-20 14:23:23")
-    datetime.datetime(2014, 1, 20, 14, 23, 23)
-    >>> yaml.safe_load('"2014-01-20 14:23:23"')
+    >>> import salt.utils.yaml
+    >>> salt.utils.yaml.safe_load("2014-01-20 14:23:23")
     '2014-01-20 14:23:23'
 
-Additionally, numbers formatted like ``XXXX-XX-XX`` will also be converted (or
-YAML will attempt to convert them, and error out if it doesn't think the date
-is a real one).  Thus, for example, if a minion were to have an ID of
-``4017-16-20`` the minion would not start because YAML would complain that the
-date was out of range.  The workaround is the same, surround the offending
-string with quotes:
+To force Salt to produce a ``datetime.datetime`` object instead of a string,
+explicitly tag the node with ``!!timestamp``:
 
 .. code-block:: pycon
 
-    >>> import yaml
-    >>> yaml.safe_load("4017-16-20")
-    Traceback (most recent call last):
-      File "<stdin>", line 1, in <module>
-      File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 93, in safe_load
-        return load(stream, SafeLoader)
-      File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 71, in load
-        return loader.get_single_data()
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 39, in get_single_data
-        return self.construct_document(node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 43, in construct_document
-        data = self.construct_object(node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 88, in construct_object
-        data = constructor(self, node)
-      File "/usr/local/lib/python2.7/site-packages/yaml/constructor.py", line 312, in construct_yaml_timestamp
-        return datetime.date(year, month, day)
-    ValueError: month must be in 1..12
-    >>> yaml.safe_load('"4017-16-20"')
-    '4017-16-20'
+    >>> import salt.utils.yaml
+    >>> salt.utils.yaml.safe_load("!!timestamp 2014-01-20 14:23:23")
+    '2014-01-20 14:23:23'
+
+When dumping a ``datetime.datetime`` object to YAML, Salt tags the node with
+``!!timestamp`` so that it will be loaded back as a ``datetime.datetime``
+object.
+
+Beware that Salt is currently unable to serialize ``datetime.datetime`` objects,
+so ``!!timestamp`` nodes cannot be used in pillar SLS files.
+
+Ordered Dictionaries
+====================
+
+.. versionchanged:: 3006.0
+
+    Loading a YAML ``!!omap`` node now reliably produces a
+    ``collections.OrderedDict`` object.  Previously, an ``!!omap`` node would
+    sometimes produce a ``list`` of (key, value) ``tuple`` objects and sometimes
+    raise an exception.
+
+.. versionchanged:: 3006.0
+
+    Dumping any ``collections.OrderedDict`` object to YAML now reliably produces
+    an ``!!omap`` node.  Previously, only the subtype
+    ``salt.utils.odict.OrderedDict`` was supported, and it produced a plain
+    mapping node.
+
+The YAML specification defines an `ordered mapping type
+<https://yaml.org/type/omap>`_ which is equivalent to a plain mapping except
+iteration order is preserved.  (YAML makes no guarantees about iteration order
+for entries loaded from a plain mapping.)
+
+Ordered mappings are represented as an ``!!omap`` tagged sequence of
+single-entry mappings:
+
+.. code-block:: yaml
+
+    !!omap
+    - key1: value1
+    - key2: value2
+
+Starting with Python 3.6, plain ``dict`` objects iterate in insertion order so
+there is no longer a strong need for the ``!!omap`` type.  However, some users
+may prefer the ``!!omap`` type over the plain ``!!map`` type because (1) it
+makes it obvious that the order of entries is significant, and (2) it provides a
+stronger guarantee of iteration order (plain mapping iteration order can be
+thought of as a Salt implementation detail that may change in the future).
+
+Salt produces a ``collections.OrderedDict`` object when it loads an ``!!omap``
+node.  (Salt's behavior differs from PyYAML's default behavior, which is to
+produce a ``list`` of (key, value) ``tuple`` objects.)  These objects are a
+subtype of ``dict``, so ``!!omap`` is a drop-in replacement for a plain mapping.
+
+When dumping a ``collections.OrderedDict`` object to YAML, Salt produces an
+``!!omap`` node.
+
+Beware that Salt currently serializes ``collections.OrderedDict`` objects the
+same way it serializes plain ``dict`` objects, so they become plain ``dict``
+objects when deserialized by the recipient.
+
+Tuples
+======
+
+.. versionchanged:: 3006.0
+
+    Loading a YAML ``!!python/tuple`` node is now supported.
+
+.. versionchanged:: 3006.0
+
+    Dumping a ``tuple`` object to YAML now always produces a sequence node
+    tagged with ``!!python/tuple``.  Previously, ``salt.utils.yaml.safe_dump()``
+    did not tag the node.
+
+The YAML ``!!python/tuple`` type can be used to produce a Python ``tuple``
+object when loaded:
+
+.. code-block:: yaml
+
+    !!python/tuple
+    - first item
+    - second item
+
+When dumped to YAML, a ``tuple`` object produces a sequence node tagged with
+``!!python/tuple``.
 
+Beware that Salt currently serializes ``tuple`` objects the same way it
+serializes ``list`` objects, so they become ``list`` objects when deserialized
+by the recipient.
 
 Keys Limited to 1024 Characters
 ===============================