Lazy import

[rhansen]

What does this PR do?

• Defines a new lazy_import function that acts like Python's import statement but defers module load until a module attribute is read or deleted. This function is unrelated to and independent of (not a wrapper around) the salt.loader.* stuff.
• Converts some of the existing deferred import statements (import statements inside a function that are evaluated when the function is called) to lazy imports.
• Converts some eager imports to lazy imports to avoid circular imports with planned features (YAML loading and dumping improvements #62932)

The new lazy_import function is a copy of https://docs.python.org/3/library/importlib.html#implementing-lazy-imports except:

• This version checks if the module has already been imported.
• This version supports lazy loading of parent modules (the example given in the Python documentation always eagerly loads any parent modules).
• When importing parent.child, the child attribute of the parent module is set to the parent.child module object (like the import statement does).

What issues does this PR fix or reference?

See #62932 (comment) for context.

Behavior Changes

No behavior changes are expected. The new salt.utils.lazy.lazy_import function is meant to be a drop-in replacement of the Python import statement. In other words, this:

import salt.utils.lazy
_baz = salt.utils.lazy.lazy_import("foo.bar.baz")

behaves the same as:

import foo.bar.baz as _baz

except:

• The actual read of foo/__init__.py, foo/bar/__init__.py, and foo/bar/baz.py—and the execution of their contents—does not happen until an attribute of _baz is read or deleted (assuming the modules are not already loaded).
• Relative imports are not (yet) supported.

Merge requirements satisfied?

• Docs (only docstrings because this only changes internals)
• Changelog - https://docs.saltproject.io/en/master/topics/development/changelog.html
• Tests written/updated

Commits signed with GPG?

No

[rhansen]

(last push was a no-change rebase onto current master)

[rhansen]

@whytewolf said in #62932 (comment):

what is concerning is the magic importing wrapping esp considering you are wrapping the loader which is already a magic import system in another magic import system. your lazy loading at a bare min needs to go through a SEP. see https://github.com/saltstack/salt-enhancement-proposals

I'll start a SEP for this. However, I want to note that the above characterization of this PR is not accurate. The new salt.utils.lazy.lazy_import function is unrelated to and independent of (not a wrapper around) the salt.loader.* stuff. It is meant to be a drop-in replacement of the Python import statement. In other words, this:

import salt.utils.lazy
_baz = salt.utils.lazy.lazy_import("foo.bar.baz")

behaves the same as:

import foo.bar.baz as _baz

except:

• The actual read of foo/__init__.py, foo/bar/__init__.py, and foo/bar/baz.py—and the execution of their contents—does not happen until an attribute of _baz is read or deleted (assuming the modules are not already loaded).
• Relative imports are not (yet) supported.

The new lazy_import function is a copy of https://docs.python.org/3/library/importlib.html#implementing-lazy-imports except:

• This version checks if the module has already been imported.
• This version supports lazy loading of parent modules (the example given in the Python documentation always eagerly loads any parent modules).

[rhansen]

Closing in favor of the more limited PR #63335.