Use Steam manifest format

[NicknineTheEagle]

This migrates DepotDownloader to the original manifest format used by Steam, phasing out the pseudo-format from the old days before Steam itself was using protobuf. The output manifest files stored in download dir are compatible with Steam client and match what you can find in Steam's depotcache dir.

[NicknineTheEagle]

Needs SteamKit with SteamRE/SteamKit#1337 merged since very large depots are slow to serialize without it.

[xPaw]

This would be a breaking change. Can we have a backwards compat for the ProtoManifest.LoadFromFile?

And can you make a separate PR for the error related commits.

[NicknineTheEagle]

This would be a breaking change. Can we have a backwards compat for the ProtoManifest.LoadFromFile?

Not sure how to handle it. Option A is converting from ProtoManifest to DepotManifest which would be tricky since the former doesn't have all the data from the latter. Option B is handling both of these types in tandem which would be a pure nightmare.

[xPaw]

the former doesn't have all the data from the latter.

What data that is needed is missing?

[xPaw]

@azuisleet what do you think? I currently can't update to last SK because it changed the chunk checksum from byte[] to uint, and the breaks the proto contract of the ProtoManifest.

@NicknineTheEagle can you rebase and undo the log message changes please.

[NicknineTheEagle]

@NicknineTheEagle can you rebase and undo the log message changes please.

Done.

What data that is needed is missing?

I don't believe anything that is needed by DepotDownloader is missing but ProtoManifest format is missing a few fields from Steam manifest, for example, filename hashes and symlinks. We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

[xPaw]

We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

We only care about backwards compat on ProtoManifest.LoadFromFile for the cached files, we don't need to save them back.

If .bin exists, load proto, then the proper .manifest, always save .manifest.

[NicknineTheEagle]

We can convert from ProtoManifest to DepotManifest and the result will be valid for DepotDownloader but probably not valid for other tools that work with Steam manifest format.

We only care about backwards compat on ProtoManifest.LoadFromFile for the cached files, we don't need to save them back.

If .bin exists, load proto, then the proper .manifest, always save .manifest.

I don't quite get it. If proto exists, should I load it to check if it's valid then download the proper manifest from Steam servers?

[xPaw]

The proto is loaded from disk to diff it against latest manifest to see which files need to be updated, no?

[NicknineTheEagle]

The proto is loaded from disk to diff it against latest manifest to see which files need to be updated, no?

So are you proposing to use them in tandem then? That would still require internally converting ProtoManifest to DepotManifest (just without re-saving, as you said) but that's probably manageable.
What should be done if the latest manifest id matches user's current ProtoManifest? Just keep ProtoManifest file untouched?

[xPaw]

Keeping it as is instead of resaving sounds simpler.

[NicknineTheEagle]

@xPaw Converting ProtoManifest into SteamKit2.DepotManifest seems to be impossible because DepotManifest.FileData and DepotManifest.ChunkData do not have any public constructors. What should I do?

[xPaw]

One way I can think of is decode ProtoManifest, and re-encode it as ContentManifestPayload, which seems like a silly round trip.

Easier way would be to just make all the properties in FileData public (ChunkData is already public so that's not a problem). But don't you also need DepotManifest properties to have public setters?

[NicknineTheEagle]

One way I can think of is decode ProtoManifest, and re-encode it as ContentManifestPayload, which seems like a silly round trip.

Easier way would be to just make all the properties in FileData public (ChunkData is already public so that's not a problem). But don't you also need DepotManifest properties to have public setters?

Right, but updating properties would require updating SteamKit once again. Should I proceed with that?

[NicknineTheEagle]

Here, this PR is needed before manifest converter can be implemented: SteamRE/SteamKit#1459

[NicknineTheEagle]

Here, ProtoManifest is now converted to DepotManifest at runtime. I can't test this because the downloads seem to be broken, I think DD doesn't properly get licenses list.

[xPaw]

I probably broke that in 9f64bd0

[NicknineTheEagle]

Ok, the conversion code works correctly after a small fix. You can download a depot with old DD and then run validation with PR build and it'll properly load legacy manifest.