How to implement forward compatibility

[dnaka91]

Thank you for this awesome crate! It's a great balance of performance and serialization size.

I was wondering, though, how to properly implement some kind of forward compatibility. What I mean is, to be able to add additional fields to a struct, but still be able to deserialize "old" data that was serialized with the previous version of the struct.

For example, let's say I have this simple struct:

#[derive(Deserialize, Serialize)]
struct Person {
    name: String,
}

And now I'd like to add a new field like:

#[derive(Deserialize, Serialize)]
struct Person {
    name: String,
    #[serde(default)]
    age: Option<u64>,
}

In non-binary formats like JSON, this will just work and a missing field age will deserialize into None. When I try the same with postcard, it'll fail to deserialize. Adding/removing #[serde(default)] doesn't seem to make any difference either.

[jeromegn]

I'm also interested in this!

[iFreilicht]

I don't know if it's possible to do this in the way you want with the wire format of postcard. You could in theory create a flavor that always returns a wire representation of None when the buffer is empty, but that means all new fields you add have to be added to the end of the struct and you can never remove any fields.

What I would recommend to do instead is to use explicit versioning:

#[derive(Deserialize, Serialize)]
struct PersonV1 {
    name: String,
}

#[derive(Deserialize, Serialize)]
struct PersonV2 {
    name: String,
    age: u64
}

#[derive(Deserialize, Serialize)]
enum Person {
    PersonV1(PersonV1),
    PersonV2(PersonV2),
}

It's not pretty, but very robust.

[dnaka91]

@iFreilicht that works, of course. I guess that's the typical approach for any breaking changes. But it would be great to be able to avoid this kind of change, every time we even add a little new field.

Sure, once there are some bigger changes this'd be necessary, but until then would be sad to end up with say 50 different versions that all just add 1-2 new fields to the struct.

I noticed that the same problem exists with bincode. So far, this worked rather well with MessagePack in compact format (rmp-serde), though. I think they do some kind of tagging or so?

[jamesmunns]

@iFreilicht's approach is generally what I suggest. As postcard is not self-describing, it doesn't have "built in" tags like you suggest. This was an explicit design tradeoff. From the spec docs:

Postcard is NOT considered a "Self Describing Format", meaning that users (Serializers and Deserializers) of postcard data are expected to have a mutual understanding of the encoded data.

In practice this requires all systems sending or receiving postcard encoded data share a common schema, often as a common Rust data-type library.

Backwards/forwards compatibility between revisions of a postcard schema are considered outside of the scope of the postcard wire format, and must be considered by the end users, if compatible revisions to an agreed-upon schema are necessary.

In general - there is no way for the binary deserializer to "recover" if it gets unexpected data in the format. After parsing one struct, it expects the next one to be placed immediately after that.

[jamesmunns]

By the way, #92 might be interesting to y'all.