Provide a mechanism to record and apply pre-staged changes to NetBox objects

[jeremystretch]

NetBox version

v3.3.7

Feature type

New functionality

Proposed functionality

This FR proposes a new staging mechanism that will enable users to stage changes programmatically before actually commiting them to NetBox's database. At a high level, I envision this workflow:

1. User creates a new ChangeGroup, assigning it a name and perhaps a description.

>>> changegroup = ChangeGroup.objects.create(name='my-group')

2. User invokes a change context referencing the newly created group. This initiates a new database transaction, and connects several temporary signal handlers.

>>> with stage(changegroup):
>>>     Site.objects.filter(name='Old Site').delete()
>>>     newsite = Site.objects.create(name='New Site')
>>>     ...

3. Object creations, modifications, and deletions take place normally within this context. All validation, database constraints, etc. apply as they normally do. As changes are made, they are recorded in memory by the context (leveraging the post_save and post_delete signals).
4. When work has been completed, the context is exited, and the transaction is rolled back, returning the database to its original state. The recorded changes are then stored in the database as a set of Change objects assigned to the ChangeGroup.

>>> changegroup = ChangeGroup.objects.get(name='my-group')
>>> changegroup.changes.all()
[Change(type='delete', model='dcim.Site', pk=17), Change(type='create', model='dcim.Site'), ...]

5. Should it be necessary to continue work within the ChangeGroup, the context can be re-entered. Any pre-recorded changes will be replayed within the new transaction, and additional changes will continue to be recorded.

>>> with stage(changegroup):
>>>     # Previous changes automatically replayed within the transaction
>>>     newsite2 = Site.objects.create(name='New Site 2')
>>>     ...

6. Once the ChangeGroup has been finalized and approved (via a means not within the scope of this FR), it can be applied to the NetBox database. The application of a ChangeGroup is an all-or-none operation; any errors at this stage (e.g. due to constraint violations) will revert the application of any changes up to that point.

>>> changegroup = ChangeGroup.objects.get(name='my-group')
>>> changegroup.apply()

7. Upon application, the ChangeGroup is marked as having been applied, but retained for future reference. (In theory, it should be possible to re-apply a ChangeGroup multiple times, though use cases for this behavior seem extremely limited.) Deleting a ChangeGroup deletes its child changes as well.

I've managed to assemble a rough proof of this concept recently, which with a bit more work should be suitable for inclusion in NetBox v3.4.

Use case

In some scenarios it is desirable to stage the creation, modification, or deletion of an object for review prior to effecting the change in production. For example, a custom script may apply a number of proposed changes for human approval, or an ingestion integration may populate newly discovered objects to be created in NetBox. While such cases can be reasonably accommodated via external means, providing this functionality natively within NetBox affords a degree of convenience, as well as provides better efficiency and validation.

The goal of this initial implementation is limited to the introduction of a programmatic interface for enaging a change context, as well as the UI controls to enable the review and application of proposed changes. Although it's possible that we'll iterate on this idea to implement more advanced functionality in the UI for future releases, doing so will likely entail devising our own transaction logic. In the near term, I'd like to focus on the implementation and proof of concept at a low level, which, if successful, can unlock substantial new functionality for NetBox plugins and scripts.

Database changes

extras.ChangeGroup

A set of related proposed changes. Changes are to be applied at the group level and not individually.

• id (PK)
• created = DateTimeField(auto_now_add=True)
• name = CharField()
• status = CharField(choices=[...])
• owner = ForeignKey(User, blank=True)
• comments = TextField(blank=True)

extras.Change

Represents the creation of a new object, or the modification or deletion of an existing object.

• id (PK)
• created = DateTimeField(auto_now_add=True)
• group = ForeignKey(ChangeGroup)
• action = CharField(choices=['add', 'change', 'delete'])
• object_type = ForeignKey(ContentType, blank=True)
• object_id = BigPositiveIntegerField(blank=True)
• object = GenericForeignKey(object_type, object_id)
• data = JSONField(blank=True)

External dependencies

No response

[jeremystretch]

Pretty happy with where we ended up for the initial implementation. We'll proceed to iterate on this concept during the v3.4 beta testing period and may extend the functionality in future releases.

[bluikko]

This feature could be massively important and it would be interesting to know a bit about possible roadmap and if this would be just a small and limited feature or could it perhaps eventually integrate basic change management for those who do not need a complex dedicated change management system bolted to NetBox.
If this feature will live on and grow then where could feedback be given? A "working group" like in cases past could be nice.
It is my opinion that many users could probably use a basic change management and would not need a hugely complex system for it. Already in just 2 weeks this issue got a large number of +1 and other interest!!

For example, could the commit() or apply() require a specific permission so that separation of concerns could be possible.