inventory as real plugins

[bcoca]

Proposal: inventory plugins

Author: Brian Coca <@bcoca>

Date: 2016/11/-1

• Status: New
• Proposal type: core design
• Targeted Release: future release
• Estimated time to implement: 2 weeks

Motivation

Current inventory is overtly complex, non modular and mostly still a legacy from 1.x tree. We also want to add a common set of features to most inventory sources but are hampered by the current code base.

The plugin architecture has been very successful with other subsystems and can be leveraged here to make a better experience for users vs the manual copying of inventory scripts and configs they do now.

It was already scheduled for revamp for 2.0 but kept getting delayed due to other priorities.

Problems

The current 'plugins' (directory, ini, yaml and script) are mostly hardcoded to each other and don't allow for flexible behaviour making it very hard to add features.
We currently rely mostly on inventory scripts which re-implement many things and diverge in feature set.

Solution proposal

• Convert the current pseudo plugins into actual plugins (ini, yaml, directory and script)
• keeps backwards compatibility with existing inventory scripts via the 'script' plugin
• roles/plays can now provide their own, just as any other plugin (inventory_plugins/)
• Revamp inventory class to allow use of plugins
• Defer data loading to 'data loader' and variable management to 'variable manager'
• fuse vars_plugins into the new inventory plugins
• implement new features:
  • shared cache: (leverage fact_cache?) which we can reset with meta: refresh_inventory
  • templated groups: use jinja2 template expressions to form groups (see new vmware inventory script)
  • normalized config: single format to rule them all! (might depend on ansible-config)
  • moar?

Dependencies

• possible dependency on ansible-config for the normalized configuration of plugins

Bonus points

start converting inventory scripts into plugins (ec2/gce/vmware/azure/docker/etc)

[mscherer]

So, I fail to see why would a role provides a plugin for inventory, isn't it too late ?

[bcoca]

not at 'compile time' it is too late for include_role

[abadger]

• Dynamic inventory scripts become inventory plugins in this proposal. Current dynamic inventory scripts will continue to be supported via a single inventory plugin.

[bcoca]

• keeps backwards compatibility with existing inventory scripts via the 'script' plugin

just rephrasing that?

[srgvg]

A couple of ideas.

• the internal inventory model should get redesigned, especially how parent child groups behave, how merging variables get precedence; more specific rules are needed here; current (object) model with group.depth and others are too limited; we need more meta information to track things; this needs to happen before we can define how inventory plugins should look like and what interfaces those need
• the dynamic inventory json interface needs to be versioned, and well tested (it currently is not), and coupled with that internal inventory model
• tests, tests, tests

I wrote up a bunch of things on https://serge.vanginderachter.be/2016/current-state-of-the-ansible-inventory-and-how-it-might-evolve/ - I still need to elaborate on that,

[jaddison]

Adding a comment here based on an IRC conversation.

Based on the jargon mentioned in here already, I can't tell if this has already been specified as being included in the refactor. Being able to use inventory/host 'patterns' when defining groups in inventory would be fabulous. ie:

[tag_environment_production]

[tag_class_webapp]

[webservers:children]
tag_class_webapp:&tag_environment_production

This would all one to use the webservers group when rendering a template and have it reflect ONLY production webapp servers. I'm not sure how else to do this in the rendering step.

[bcoca]

You can create a plugin inventory source that supports that.

I'm only trying to create a refactoring of the current framework to allow it to be easier to expand, detailed things like that are no in the scope of this proposal (though will be made easy to add by it).

[srgvg]

FYI Some first design ideas for an Ansible Inventory 2.0
update: and Ansible Inventory 2.0 design rules

[nicolas-g]

+1

[benjamincburns]

Echoing @srvg and @jaddison a bit, but I think it's worth referencing ansible/ansible#9065 here as motivation to accept/approve this proposal.

The thread is a long list of people (including myself most recently) who've encountered the group_vars precedence limitations in the current inventory system and did a bit of a double take. It'd be nice if there was a mechanism which allows users to choose which CM trade offs make the most sense for them.