Check transmute restrictions in the crate where it is used

[brson]

The transmute function is peculiar in that it has extra restrictions on its type parameters that are not checked until translation time. This can result in surprises for downstream users of code that uses transmute. Instead we should type check its instantiation when called or its value is taken. Doing this will require at least one new restriction.

During type check:

• Validate that generic types are only transmuted through pointers
  • this ensures that all uses can be validated in the context of the local trait
  • todo: we have fat and thin pointers, so this would seem insufficient
• Validate that both types are Sized or the same type
• Record all pairs of transmuted types

During translation (of the crate where transmute is used):

• Check that all instantiations used types with the same type
• Warn if any instantiations used types with different alignment

@nikomatsakis Does this sound right? What about the fat/thin pointer issue?

Nominating.

[nikomatsakis]

I don't know why you say "record all pairs of transmuted types".

I don't know whether it makes sense to warn -- certainly this should be lintable.

Regarding the thin/fat pointer issue, we'd have to report an error (during type check) if you transmute from &T to &U and either T or U is not Sized.

[nikomatsakis]

To put another way, I think the idea was to have type check validate that trans will succeed. Sometimes, such as a transmute from T to U (where T and U are typed parameters) or a transmute from &T to &U where T or U are sized, the type checker has to be conservative.

[pnkfelix]

P-backcompat-lang, 1.0.

[nikomatsakis]

It'd probably be useful to survey existing uses of transmute and describe them here. I'd be interested to know how they breakdown!

[nikomatsakis]

I just had an interesting thought regarding const-casts: I'd really like to prevent people from transmuting &T to &mut T (except via Unsafe<T> -- I'm not quite sure how to allow Unsafe<T> to do it while forbidding everyone else). The reason being that this result is undefined behavior. It's easy enough to prevent &T from being transmuted to &mut U as part of this change, and I think we should do that.

However, it seems clear that's not enough. For example, you could always have a transmute from &T to &U and -- somewhere in types T and U -- you might convert a & to &mut. But it occurs to me now that even if you did transmute it, you could never use the &mut value that is within U to mutate, since it itself is still in an aliased location. This is still not an actual guarantee of avoiding undefined behavior -- you could transmute from *mut T to *mut U (with, say, T=&int and U=&mut int) and skin the cat that way -- but it's closer than I thought.

The only way to really check for this sort of const cast would be either (a) add a (presumably special) TransmutableTo trait or (b) keep the current "check during trans" approach.

In the course of writing this message, though, I've started to wonder if preventing trans errors is good enough, or if we shouldn't strive to actually prevent const casts altogether. In particular, I think that we still can't really guarantee that trans will succeed for arbitrary types, at least not once associated types come into play -- I imagine it'd be possible to generate a monomorphization expansion with certain types that never terminates. (Generating foo::<A> then generates foo::<A::T>, where A::T (in some cases) is defined as ~A, such that we wind up monomorphizing foo::<A>, then foo::<~A>, then foo::<~~A>, etc). This is technically different, I guess (nontermination vs error) but to the end user, it's maybe not so different.

[nikomatsakis]

Caveat to last paragraph: we don't, of course, have associated types, so maybe there's some reason something like that last example can't be done.

[emberian]

@nikomatsakis Unsafe<T> doesn't actually do that transmute. It returns a *mut which is the only valid type to return, once we start emitting noalias on &mut T.

[nikomatsakis]

@cmr it starts with an &self, converts to *self, and then transmutes to *mut self. This last transmute is the one I was talking about.