Incremental dumping

[egpbos]

Has anybody ever thought about dumping json structures incrementally?

My usecase is a code that runs for a few hours which logs data every few minutes. Currently, I just write out all the collected data at the end, but I'd like to do this immediately after the data is collected. It would be convenient if it were still just one JSON file.

I guess, to do this, one would need a stateful dumper that keeps track of what has already been dumped. Also, inserting things at random places would probably make this really slow. However, for my usecase, I'm just appending things at the end, so all that needs to be done is to remove the closing braces at the end, append new data and put the closing braces back at the end.

[falbrechtskirchinger]

Well, there's nothing readily available to do so. You propose a stateful serializer that keeps track of what information has been written but it might be easier to store that information inside the basic_json type. If I remember correctly, there's a PR that adds a user-defined base class to basic_json.

From there, you could hack something together that suits your use case. A "proper" implementation that can deal with arbitrary insertions could also store the byte offsets in basic_json.

All that being said, is all that effort worth it, or could you just periodically dump data as independent objects (possibly appending to the same file) and merge them later?

[falbrechtskirchinger]

Oh, and there are obviously race conditions to worry about if you're consuming that data as an update is being written. Instead of appending to the file, copy, update, and atomic rename might be the better option.

[egpbos]

Lacking existing implementations, I think your suggestion for just dumping objects separately is a great idea. Thanks!