Provide the ability to zero padding bytes and return &Initialized<T>

[joshlf]

Note: We likely won't do this. See the conversation below for the full context, but TL;DR: We'd need for any KnownLayout type to require that all of its fields also implement KnownLayout, which we don't currently require. We expect that the freeze language feature will land soon enough that it's not worth worsening KnownLayout's UX to support this use case.

Progress

• Update this issue description per this comment
• Update KnownLayout to require that fields implement KnownLayout too
• Do one of the following:
  • Decide that the freeze intrinsic (RFC 3605) will land and stabilize soon enough that we can rely on it instead; relax KnownLayout to not be recursive (and watch out for #1162)
  • Use the recursive KnownLayout requirement to implement this design

Details

Issues like this one demonstrate that it is sometimes useful to access the bytes of a type which cannot implement AsBytes. In these cases, it should be sound to:

• Recursively zero any inter-field padding bytes
• Provide access to the bytes of the object as an &Initialized<T> where Initialized: IntoBytes even when T: !IntoBytes

We would need to teach KnownLayout to be able to zero padding, e.g.:

pub unsafe trait KnownLayout {
    fn zero_padding(&mut self) -> &mut Initialized<Self>;
}

#[repr(transparent)]
pub struct Initialized<T> {
    // INVARIANT: Every byte in `inner` is initialized. Note that this implies
    // that an `Initialized` cannot be moved by value unless `T: IntoBytes`
    // since typed copies de-initialize padding bytes.
    inner: T,
}

unsafe impl<T> IntoBytes for Initialized<T> {}

impl<T> Deref for Initialized<T> { ... }

// INVARIANT: Since `T: IntoBytes`, any value that is written via this impl
// has no padding bytes, and so will not invalidate the invariant that all of
// `inner`'s bytes are initialized.
impl<T: IntoBytes> DerefMut for Initialized<T> { ... }

// TODO: Provide field projection

The only requirement for a type supporting this operation is that we know where its padding bytes are. The public API for this type could be in KnownLayout.

As of this writing, KnownLayout does not require that a type's fields also be KnownLayout. We are planning to add that requirement in order to support this design.

Open questions

• What if we want to copy from a &T (which we can't modify) into a buffer while initializing any padding bytes in the destination like musli-zerocopy does? See this discussion.

Related & prior art

• RFC: Add freeze intrinsic and related library functions rust-lang/rfcs#3605

[joshlf]

EDIT: We've decided to require KnownLayout to be recursive to be forwards-compatible with this design. We can always relax that requirement in the future if we decide not to implement this design.

---

I just realized that KnownLayout may not have the necessary context. In particular, we support deriving KnownLayout on all Sized types, even those that are not repr(C), and even those whose field types are not KnownLayout. The "not repr(C)" thing we might be able to work around by querying for field offsets, but without the ability to recursively descend into each field type, we have no way of knowing where padding is inside of individual fields.

[jswrenn]

I suspect default-repr enums are going to render this strategy a non-starter, since we cannot reason about the size of their discriminants. Given that, do we still want to proceed with #1223?

[joshlf]

I suspect default-repr enums are going to render this strategy a non-starter, since we cannot reason about the size of their discriminants. Given that, do we still want to proceed with #1223?

Discussed offline. Given this concern, and given the likelihood of freeze stabilizing in the medium-term, we're going to hold off on this issue for now. We may still do it - I'm not closing it - but we're not going to make any attempt to be forwards-compatible with it.