Could I conceivably use jj for structured data versioning in remote storage locations?

[nicholasjng]

Hello!
I'm very new to jujutsu, but like what I'm seeing so far. In this thread, I wanted to present an idea to someone experienced with the project and get an understanding of whether it's feasible or not.

TL,DR: I want to place potentially large, structured, but not line-based data files (think maybe Parquet or, even simpler, shared object files) under source control, with support for diffs (ideally) and remote storage backends.

I'm specifically looking for a way that gives me data files as a sum of its diffs / snapshots, just like text files, which should arguably be possible for structured file formats with arbitrarily large development effort, I guess. What I like about jujutsu is its promise of storage-independent APIs (which means that to get 3), all I should have to do is implement a backend for my desired storage service, plug it into jj, and that's it (?)), and its conflict modelling and rebasing.

So if this is even possible, what is necessary to add support for structured data in jj's model as described, and will that scale to large file sizes / amounts of files? I'm enticed by the idea that the project is intended to be used with Google's monorepo, so I'm not worried about scaling in general, but more about the fit for jj's data model.

PS: I have searched the docs as well as the discussions for large data file support, and found #59, but I believe that touches on a slightly different angle than what I'm interested in.

- Nico

[joyously]

I'm specifically looking for a way that gives me data files as a sum of its diffs / snapshots, just like text files,[] What I like about jujutsu is its promise of storage-independent APIs [], and its conflict modelling and rebasing.

I don't really know the jj internals, but I think most of the conflict modeling, rebasing, and diffs are text oriented.
You might look into CRDTs with AutoMerge, large files with SnowTrack, remote storage with Grace, or bytewise diffs with Pijul. All of these have their own drawbacks, of course.
Maybe there's a way to combine several.

[martinvonz]

I don't really know the jj internals, but I think most of the conflict modeling, rebasing, and diffs are text oriented.

Yes, that is correct, and I agree that this will probably be a problem for @nicholasjng's use case. @nicholasjng, how would you diff these large, structured files? How would you merge them (if you even need the ability to merge/rebase them)? Can you tell us a bit more about what kinds of files we're talking about? Is it for game development? Bioinformatics?

[nicholasjng]

Thanks for the answers! Here is a basic explanation (I'm a mathematician by trade, so the following might be too idealistic).

I would like to be able to treat these files as a telescopic sum of its diffs, so that at any point in history, $$F = \sum_{i=0}^n D_i,$$ where $D_i$ is the changeset of the file $F$ at a point $n$ in history, and $n$ is the number of changes/commits containing changes of $F$. (I could also set $n$ as the total number of commits, but then diffs $D_i$ of commits not touching $F$ are empty.)

For diffing, say, a Parquet file (again high-level, I'm not sure if this actually works), I have a footer containing the schema, and presumably data buffers containing the columnar data. So I could imagine first parsing the header/footer and checking for schema changes between commits, and only then checking the columns. Merging in this case might be undefined behavior if there is a schema change, or insertion of extra rows/columns if any of the rows/columns were modified.

I'm working in ML Operations, where I'm faced with versioning needs for data regularly, and none of the solutions I know about convince me. Either they are missing diffs, duplicating data every time a new version is made (because everyone is assumed to have infinite storage and cloud credits), or just store plaintext pointers to versions in a repo (think git-lfs), or miss source control semantics that I like. Also, a lot of them are SaaS, hard to self-host, and/or lock in users to varying degrees.

After a few tries, I've arrived at the opinion (and it's just an opinion, I'm happy for contrarian viewpoints) that I would like SCM semantics with something like libgit2, together with support for remote storage options like cloud buckets. Another upside to this is that by virtue of these semantics, devs can be onboarded easily if they work with git/jj on the source code on a daily basis anyway, and you don't need to learn (and pay for!) a new tool.

To me it feels like this is within reach, provided I could "teach" my SCM about structured diffs for files that are not text-based, but I admittedly don't know enough about git/jj internals to supplant that feeling. That was the background of my question.

[PhilipMetzger]

This #2865 (comment) may be of interest to you. I'm in general in favor of having another backend as it has the potential to show errors and missing APIs in the current model.

This quote from @ilyagr may also be useful if you want to keep a Git like model

My ideal outcome (which might be too good to be realistic, there might not be a sane UI possible for it) would be to have that, and then to allow a jj repo to be a "union repo" with different paths handled by different backends.

[nicholasjng]

Thanks for the link! The chunking+checksum calculation for parts was just what I had in mind - it should work well for Parquet files and even JPEGs. The tradeoff is then presumably between chunk size (more chunks->less duplication) and the amount of checksum computation on files.

So I understand correctly that this would all be part of the backend functionality? Then I already have a lead.

[PhilipMetzger]

So I understand correctly that this would all be part of the backend functionality?

Yes, you'd "just" need to provide the correct semantics for it (the "just" is load bearing, as it certainly isn't trivial).

The tradeoff is then presumably between chunk size (more chunks->less duplication) and the amount of checksum computation on files.

This needs a storage side optimization, like block deduplication which is something Dolthub does on their side.