Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DST-ification of libraries #16918

Closed
nrc opened this issue Sep 1, 2014 · 23 comments
Closed

DST-ification of libraries #16918

nrc opened this issue Sep 1, 2014 · 23 comments

Comments

@nrc
Copy link
Member

nrc commented Sep 1, 2014

We should change our impls to a DST style where appropriate. E.g., impl Ord for str .... This requires changing a bunch of traits to have the Sized? bound.

This kind of works. The problem is that we currently do not unify T with unsized types ([T], str, etc.) during subtyping/type inference and thus trait matching. This means that if do the DST-ification, we will fail to type check in places with references (e.g., a == b where a and b both have type &str). However, if we do unify, we will get coherance errors where impls are defined for, e.g., &T and &str. Thus we have a bit of a catch 22. We could cfg our way to victory, but there are literally hundreds of places that need DST-ification, so that will be extremely painful. I'm not sure if trait reform landing can help us here, or perhaps where clauses with != constraints or something equally exotic. Perhaps we could hack coherance with this special case temporarily. Anyway, we need a plan!

cc @nikomatsakis

@nrc nrc mentioned this issue Sep 1, 2014
23 tasks
@nrc
Copy link
Member Author

nrc commented Sep 1, 2014

Here is my very, very early WIP branch: https://github.com/nick29581/rust/tree/dstify

@nikomatsakis
Copy link
Contributor

I should almost have what is needed here.

@nikomatsakis
Copy link
Contributor

In particular, what is needed to enable unification of types.

@japaric
Copy link
Member

japaric commented Oct 23, 2014

@nick29581 Are you working on this atm? I DSTified [T]/str's extension traits like StrSlice, ImmutableSlice, MutableSlice, and both make/make check passed.

Want to coordinate the work in this area?

@japaric
Copy link
Member

japaric commented Oct 23, 2014

I've compiled a table with all the traits that [T]/str implement. Not sure which/if-all traits should be DSTified.

@aturon
Copy link
Member

aturon commented Oct 23, 2014

@japaric I was in the process of assembling a similar patch myself, but I'd be glad to work with you on this. I'm away from the office this week but should be available early next. Please ping me if you post a PR. What you have so far looks like a good start.

@japaric
Copy link
Member

japaric commented Oct 24, 2014

DSTifying PartialEq, PartialOrd causes lots of breakage.

The reason is that, for example, PartialEq is currently implemented for &'a [T], so eq has signature fn eq(&self, other: & &'a [T]). But, once you DSTify it, eq's signature changes to fn eq(&self, other: &[T]) , so the RHS now needs one less level of indirection. And, because of how the operator sugar works, the RHS of the operator now needs to be an unsized array [T] or str.

The following patterns break:

  • a[..] == b[..]: Fixed by changing it to a[..] == *b[..]
  • a[..].partial_cmp(&b[..]): Fixed by changing it to a[..].partial_cmp(b[..])
  • #[deriving(PartialEq]/#[deriving(PartialOrd] on structs that contain either &str/&[T]: error: mismatched types: expected&str, found&&'static str`
  • s == "abc": Fixed by changing it to s == *"abc"
  • s == [b'i', b'n', b'f']: error: mismatched types: expected[u8], found[u8, ..3](expected unsized array, found array)
  • Everything that applies to ==, applies to other operators.

I'll skip those traits for now.

@nikomatsakis
Copy link
Contributor

Some of this (e.g., a[..] == b[..]) seems like it should work. The fact that it doesn't is, I think, because the way we are handling operators like == seems a bit wrong to me.

@japaric
Copy link
Member

japaric commented Oct 27, 2014

@aturon I've got other branches where I've DSTified other 4 traits, all them need a newer snapshot (#18259) that includes #18121. Could you review them? Here are the branches + some remarks:

  • Show
    • This one was rather trivial nothing else to say here
  • Hash
    • This one changed the signature of *_equiv methods from e.g. find_equiv(&"Hello") to find_equiv("Hello"), so some stuff broke.
    • I left a FIXME about adding a test for hashing Box<str>, because there is no way to create that type at the moment (see There is no way to create Box<str> #18283).
    • I also left a FIXME for adding Sized? to the Rc<T> impl, because e.g. Rc<[T]> is not yet allowed (see std::rc::Rc should accept DST #18248).
  • ToCStr
    • There was this FIXME message saying that we should take ToCStr implementors by reference instead of by value, so I went ahead with that.
      • This broke stuff like Command::new(path), which needs to be changed to Command::new(&path).
      • Note that methods like Command::new("foo") still work.
    • I've to kept the impl ToCStr for &'a str because of Command::args, one of the uses is cmd.args(["Hello", "World"]) where args has signature fn args<'a, T: ToCStr>(&'a mut self, args: &[T]) -> &'a mut Command, in that case T has to be &str
  • BytesContainer
    • As with ToCStr I changed some methods from taking the BytesContainer implementors by value to taking them by reference. This broke some stuff.
    • Since BytesContainer has two methods (container_into_owned_bytes and is_str) that take the receiver by value, so I split the trait in two: BytesContainer for Sized? and SizedBytesContainer: BytesContainer + Sized, but see below.
      • It seems that the container_into_owned_bytes method is not used anywhere so we could just remove it, and
      • There is only one use of the is_str method but AFAICT is not necessary, so we could remove this method as well, therefore
      • if those two methods are removed then we can remove the SizedBytesContainer trait as well.

Each branch passes make check locally but I may have forgotten to fix some Windows/*BSD tests since I can't run those.

@Gankra
Copy link
Contributor

Gankra commented Oct 27, 2014

@nikomatsakis Care to elaborate on how we should be handling operators? I personally find it a bummer that they're one of the last places where explicit derefs of an actual reference are used, especially since they then... take a reference.

@japaric
Copy link
Member

japaric commented Oct 27, 2014

@gankro According to #4920, @nikomatsakis idea at that moment was that a OP b should desugar to OpTrait::Op(&a, &b), this means that 5 + 5 would work, whereas &5 + 5 and 5 + &5 shouldn't work (Note that currently, both 5 + 5 and &5 + 5 work, whereas 5 + &5 doesn't). I don't know if he still maintains that position.

@japaric
Copy link
Member

japaric commented Oct 28, 2014

Re: PartialEq and friends, I just realized that I forgot to add Sized? to the &'a T/&'a mut T implementations (e.g. impl<'a, Sized? T: PartialEq> PartialEq for &'a T) in my first attempt and that seems to have caused the large breakage. I've amended that and now the operator sugar seems to work, but #[deriving(PartialEq)] is still broken for structs that contains &str ("error: mismatched types: expected &str, found &&'static str (expected str, found &-ptr)") - I'm looking into it.

bors added a commit that referenced this issue Oct 28, 2014
This PR changes the signature of several methods from `foo(self, ...)` to `foo(&self, ...)`/`foo(&mut self, ...)`, but there is no breakage of the usage of these methods due to the autoref nature of `method.call()`s. This PR also removes the lifetime parameter from some traits (`Trait<'a>` -> `Trait`). These changes break any use of the extension traits for generic programming, but those traits are not meant to be used for generic programming in the first place. In the whole rust distribution there was only one misuse of a extension trait as a bound, which got corrected (the bound was unnecessary and got removed) as part of this PR.

I've kept the commits as small and self-contained as possible for reviewing sake, but I can squash them when the review is over.

See this [table] to get an idea of what's left to be done. I've already DSTified [`Show`][show] and I'm working on `Hash`, but bootstrapping those changes seem to require a more recent snapshot (#18259 does the trick)

r? @aturon 
cc #16918 

[show]: https://github.com/japaric/rust/commits/show
[table]: https://docs.google.com/spreadsheets/d/1MZ_iSNuzsoqeS-mtLXnj9m0hBYaH5jI8k9G_Ud8FT5g/edit?usp=sharing
@nikomatsakis
Copy link
Contributor

@japaric @gankro what I'm most concerned about is that, currently, we do a kind of odd-ball search when looking up operators. e.g., if we have l + r, where the types of l and r are L and R, I just want to search for Add<L,R>, rather than doing a traditional method search (which involves auto-ref, coercion, and other considerations). This does mean that in some cases we might wind up writing "convenience" impls for things like String compared with &str that basically encode common coercions. But I think it's more predictable overall. Today I whippe up a prototype of this approach, though it doesn't quite build yet. One obstacle is the current Eq and Ord traits which do not permit comparison between two distinct types for some reason.

@japaric
Copy link
Member

japaric commented Oct 30, 2014

I got PartialEq and friends passing make now, I had to change the expansion of #[deriving(PartialEq)] (from e.g. true && (*__self_0_0).eq(&(*__self_1_0)) to true && (*__self_0_0) == (*__self_1_0)) to make it work. I'll send a PR once I get make check to pass locally.

@japaric
Copy link
Member

japaric commented Oct 30, 2014

New problem, the == operator is not working as expected for fixed size arrays:

[0u8, 1] == [0u8, 1];
//^~ error: mismatched types: expected `[u8]`, found `[u8, ..2]` (expected unsized array, found array)
[0u8, 1].eq([0u8, 1].as_slice());  // OK
[0u8, 1].eq(&[0u8, 1]);  // OK

@nikomatsakis I think you are working on the operator desugaring at the moment, any clue what could be happening there? I'm not quite sure how the desugaring works for fixed-sized arrays but I think that &[0u8, 1] is being treated as &[u8, ..2] instead of being coerced as &[u8].

@nikomatsakis
Copy link
Contributor

@japaric I am indeed working on cleaning up this code and I actually encountered a similar issue. I think I do know yes what the problem is, though I have to confirm. It has to do with the specifics of when coercions occur and under what conditions. Basically I think what happens (at least in my branch) is that we match the receiver as [u8] but then we type-check the argument, which has type [u8, ..2] and we expect [u8]. We probably ought to be checking the autoref'd type of the argument (&[u8, ..2]) and comparing that to the expected type with ref, &[u8]. This would make the coercion work. I will look into doing that on my branch. The branch is getting very close, shooting for PR today.

bors added a commit that referenced this issue Oct 31, 2014
- The signature of the `*_equiv` methods of `HashMap` and similar structures have changed, and now require one less level of indirection. Change your code from:

``` rust
hashmap.find_equiv(&"Hello");
hashmap.find_equiv(&&[0u8, 1, 2]);
```

to:

``` rust
hashmap.find_equiv("Hello");
hashmap.find_equiv(&[0u8, 1, 2]);
```

- The generic parameter `T` of the `Hasher::hash<T>` method have become `Sized?`. Downstream code must add `Sized?` to that method in their implementations. For example:

``` rust
impl Hasher<FnvState> for FnvHasher {
    fn hash<T: Hash<FnvState>>(&self, t: &T) -> u64 { /* .. */ }
}
```

must be changed to:

``` rust
impl Hasher<FnvState> for FnvHasher {
    fn hash<Sized? T: Hash<FnvState>>(&self, t: &T) -> u64 { /* .. */ }
    //      ^^^^^^
}
```

[breaking-change]

---

After review I'll squash the commits and update the commit message with the above paragraph.

r? @aturon 
cc #16918
bors added a commit that referenced this issue Nov 1, 2014
Methods that used to take `ToCStr` implementors by value, now take them by reference. In particular, this breaks some uses of `Command`:

``` rust
Command::new("foo");  // Still works
Command::new(path) -> Command::new(&path)
cmd.arg(string) -> cmd.arg(&string) or cmd.arg(string.as_slice())
```

[breaking-change]

---

It may be sensible to remove `impl ToCstr for String` since:
- We're getting `impl Deref<str> for String`, so `string.to_cstr()` would still work
- `Command` methods would still be able to use `cmd.arg(string[..])` instead of `cmd.arg(&string)`.

But, I'm leaving that up to the library stabilization process.

r? @aturon 
cc #16918
bors added a commit that referenced this issue Nov 3, 2014
- The `BytesContainer::container_into_owned_bytes` method has been removed

- Methods that used to take `BytesContainer` implementors by value, now take them by reference. In particular, this breaks some uses of Path:

``` rust
Path::new("foo")  // Still works
path.join(another_path) -> path.join(&another_path)
```

[breaking-change]

---

Re: `container_into_owned_bytes`, I've removed it because

- Nothing in the whole repository uses it
- Takes `self` by value, which is incompatible with unsized types (`str`)

The alternative to removing this method is to split `BytesContainer` into `BytesContainer for Sized?` and `SizedBytesContainer: BytesContainer + Sized`, where the second trait only contains the `container_into_owned_bytes` method. I tried this alternative [in another branch](https://github.com/japaric/rust/commits/bytes) and it works, but it seemed better not to create a new trait for an unused method.

Re: Breakage of `Path` methods

We could use the idea that @alexcrichton proposed in #18457 (add blanket `impl BytesContainer for &T where T: BytesContainer` + keep taking `T: BytesContainer` by value in `Path` methods) to avoid breaking any code.

r? @aturon 
cc #16918
@japaric
Copy link
Member

japaric commented Nov 6, 2014

@aturon

Anything left to be done here? (Probably the Index trait, but that's being tracked on #16529)

@aturon
Copy link
Member

aturon commented Nov 6, 2014

@japaric I think it's pretty close, but a few more cases to investigate.

I ran git grep "impl.*for &" and that turned up a few more cases that should be investigated.

The other thing, which is much harder, would be to add Sized? to all generics that can support them (not just the traits that were being implemented for &T). However, in general that's a non-breaking change so we can do this over time as we find the need.

@japaric
Copy link
Member

japaric commented Nov 18, 2014

@aturon Here's the updated table. According to my analysis, this is what's left to be done:

Do look at the table, and let me know if you think I missed anything.

@aturon
Copy link
Member

aturon commented Nov 21, 2014

@japaric Looks good to me! I think the StrAllocating and AsciiCast traits are likely to be revised during API stabilization for their modules, which I'm about to start on; this should allow us to DSTify. (In the long run we shouldn't need StrAllocating at all once str is a library type.)

The couple of places where you ask "shouldn't this be automatically derived?" -- the answer is yes, but not yet: that hasn't been implemented yet. I've just opened an issue about this.

@japaric
Copy link
Member

japaric commented Nov 21, 2014

In the long run we shouldn't need StrAllocating at all once str is a library type.

How is that possible? str is defined in libcore, and String is defined in libcollections, we need an extension trait to glue types that are defined in different crates, right?

@aturon
Copy link
Member

aturon commented Nov 21, 2014

@japaric You're right.

@alexcrichton
Copy link
Member

I'm going to close this as this was basically done for the alpha release and has been an ongoing concern during stabilization. Nice work everyone!

bors added a commit to rust-lang-ci/rust that referenced this issue Mar 31, 2024
fix: Don't assert paths being utf8 when filtering them in the watcher

Closes rust-lang/rust-analyzer#16914
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants