Backup Json File too many brackets between models

[Hooverdan96]

When extracting the backup json file for analysis, json parser throws an error. This seems to be related to additional ']' set between 2 parts of the file.

In my example, it seems to create another section right after the
"model": "storageadmin.share" and the "model": "smart_manager.service" section, resulting in some json inconsistency.

I assume these extraneous brackets don't impact the import from backup, but it would be good if they weren't in the file, so the contents could be viewed with any json viewer without error.

Here is an excerpt from one of my config backup files:

...
		"pool": 2,
		"size": 10485760
	},
	"model": "storageadmin.share",
	"pk": 13
}]
[{
	"fields": {
		"config": null,
		"display_name": "Replication",
		"name": "replication"
	},
	"model": "smart_manager.service",
	"pk": 1
}, {
... `

[Hooverdan96]

I assume the extraneous brackets come from these two "dumpdata" actions?

rockstor-core/src/rockstor/system/config_backup.py

Line 68 in 81a0d5f

with open(fp, "w") as dfo:

    call_command("dumpdata", *model_list, stdout=dfo)
    dfo.write("\n")
    call_command("dumpdata", database="smart_manager", *model_list, stdout=dfo)

[phillxnet]

@Hooverdan96 Thanks for opening this issue. However we arn't specifically aiming for json format here. We just need a consistent output/input format. So this is not really an issue or bug as such as we are doing what we intend and it working, bar non format related bugs that is such as the pending scheduled task save/restore which is not related to the format as such.

Lets see what @FroggyFlox thinks before we move to close and I'l like to take a close look myself at some point. Can't directly unfortunately. I think once-upon-a-time were were json compatible.

Particularly sensitive as we need to maintain backward compatibility so I'm keen on the 'if it isn't broken then don't fix it' approach. On the other hand if we are just working around this and moving to stick json can be done safely then dandy.

I'll try and have a proper look soon.

Thanks for opening this, good idea to take a closer look.

[FroggyFlox]

I assume the extraneous brackets come from these two "dumpdata" actions?

Correct. As you can see, we simply create a database dump as the config backup. During restore, we parse these to create API requests that will put all this information into the "new system" databases. The key point here is that we have two main databases (smart_manager and storageadmin) and this is why we combine them into a Python list. That makes it easier for us to then parse each of these two lists and extract the information we need from their respective JSON.

As @phillxnet said, we don't really aim at making that a json file so I personally do not see this as an issue. We could discuss the need or benefit for making it easier to alter by the user, maybe.

[Hooverdan96]

sounds good. no need to add additional work to what you're already doing, if you believe it's not causing any potential issues during the restore. I know how to work around it, and some of the editors still display the tree structure correctly when needed (despite throwing an error).

[phillxnet]

@FroggyFlox and @Hooverdan96 Cheers.
I agree then that we are OK with just the more advanced doc entry re viewing the file contents raised in the following issue:
rockstor/rockstor-doc#359

It may be we can provide script to convert the file to a json and then back to the list format. That way folks could pre-process to view nicely and edit/examine if need be, then post process to return to our now well established and mostly working list format. Might end up being a simply python script to sense which way to go at the time. We could then, in time, re-use this pre-post processor to display the config backup file within the Web-UI for instance. That might be nice.

But agreed also that this is a long shot for our current work load and available resources.