Add flag to disable withFilter pattern desugaring

[puffnfresh]

This adds a flag called -Zirrefutable-generator-patterns which changes
desugaring of code like the following:

for {
  (a, b) <- List(1 -> 'a', 2 -> 'b')
  (c, d) <- List(3 -> 'c', 4 -> 'd')
} yield (a + c, b + d)

Which previously would turn into something like:

List(1 -> 'a', 2 -> 'b').withFilter {
  case (a, b) => true
  case _ => false
}.flatMap {
  case (a, b) =>
    List(3 -> 'c', 4 -> 'd').withFilter {
      case (c, d) => true
      case _ => false
    }.map {
      case (c, d) =>
        (a + c, b + d)
    }
}

With this new flag, it only becomes:

List(1 -> 'a', 2 -> 'b').flatMap {
  case (a, b) =>
    List(3 -> 'c', 4 -> 'd').map {
      case (c, d) =>
        (a + c, b + d)
    }
}

Which creates a few benefits:

1. We don't have to do a useless call to withFilter
2. We can use patterns on things without withFilter
3. Things don't silently disappear when patterns are wrong

[relrod]

Stupidly minor nitpick that probably isn't worth mentioning, but for comprehensions -> for-comprehensions

[non]

I definitely like the -Z convention for Typelevel settings.

[larsrh]

This seems to be a little bit too good to be true. Does the logic for detecting irrefutable patterns actually work and was just not applied properly?

[Blaisorblade]

@larsrh: it appears this is not about detecting irrefutable patterns, it's about expecting them to be.

[puffnfresh]

@Blaisorblade exactly. Happy to clarify this if it's confusing somewhere.

[milessabin]

👍

[milessabin]

Do we want a SIP-18 style language import as well?

[paulp]

Here's why there's a filter:

    for {
      (a, b) <- List(1 -> 'a', 2)
      (c, d) <- List(3 -> 'c', 5)
    } yield List(a, b, c, d)

I realize nobody here would write code like that, but hey, turns out that other people do. I know because I already implemented this (I expect that to be a frequent theme around here) and later we had to revert it.

https://issues.scala-lang.org/browse/SI-6968

Nobody can outrun Any. It's like the Terminator. It absolutely will not stop until it has ruined your plans.

[aloiscochard]

👍

[puffnfresh]

@larsrh there isn't much logic there any more, as @paulp shows, there used to be code to treat things like Tuple2 as irrefutable but that was broken when you did silly things and so was reverted. The motivation for the revert was to maintain the following behaviour:

scala> for { (a, b) <- List(1 -> 'a', 2) } yield a
res0: List[Any] = List(1)

But under this new flag, we get the following:

scala> for { (a, b) <- List(1 -> 'a', 2) } yield a
scala.MatchError: 2 (of class java.lang.Integer)
  at $anonfun$1.apply(<console>:8)
  at scala.collection.immutable.List.map(List.scala:276)
  ... 33 elided

I think most people prefer the latter behaviour.

@milessabin while I would love to turn SIP-18 against itself, the checkFeature part is in Typers and I don't think it can be applied a phase which is this early.

[puffnfresh]

Two things I want to do before we think about accepting this:

• Figure out what changing scala-reflect.jar does to interoperability with Typesafe Scala (particularly macros)
• Write a test for this feature.

[ceedubs]

Completely agree both with preferring the latter behavior when a tuple match isn't irrefutable and with figuring out what changing scala-reflect.jar does particularly with macros.

Thanks @puffnfresh! And thanks for the helpful input @paulp.

[Blaisorblade]

The pull request description describes a case where the change is uncontroversial — one could prove that those withFilter calls are the identity. The discussion and the help text suggests that the change is not limited to such cases, so I'm confused.
Anyway, might I suggest a bit more documentation here before merging?

I think most people prefer the latter behaviour.

You mean you prefer the MatchError? I disagree, and Haskell does as well (see below).

That looks strange... Why is Tuple2 considered irrefutable when matching against values of type Any?

If that's intentional, should I expect a MatchError in the code below?

//Assume the Scala equivalent of Haskell's:
//data T = A x | B | C

for {
  A(x) <- List(A(a), B, C)
} yield x

It seems to me the two cases should be treated in the same way. Do you suggest rejecting the second example as well? GHC accepts the Haskell equivalent:

Prelude> data T = A | B | C
Prelude> [() | A <- [A, B, C]]
[()]

HOWEVER, I now read the linked ticket. People mention parser changes... are you telling me that refutability is so syntactic to be decided by the parser? Oh my.

So @paulp, while Any is certainly bad, here it's not special. It's desugaring things in the parser which is at fault.

[tixxit]

@Blaisorblade Yes, I would expect a MatchError in that case too. Basically, I expect <- to behave like a val assignment.

// data T = A x | B | C
val A(x) = B: T // Fails, because B isn't A

[Blaisorblade]

@tixxit I see, and now I get the argument:

Things don't silently disappear when patterns are wrong

Not going to use this, but it's behind a flag and the user documentation is accurate, so OK.

Also, GHC provides a different syntax for irrefutable patterns. Would something like that in the future make you happier? It shouldn't stop this from merging.

What I would love would be dropping those withFilter when the pattern is in fact irrefutable. But this would require type-aware desugarings — and I realized that's impossible, because for-comprehension's desugaring are completely untyped (and this has some advantages).
Lacking that, it would be cool if an optimizer could remove those withFilter calls — but this is really hard.

[bvenners]

It is not clear to me what the opinion is here from the discussion, so I'll add this even though I may be repeating what others said. What I think would be desirable without the flag (i.e., by default) is that if the compiler can prove there will be no match error possible at runtime, then the withFilter is elided. If it can't prove that, then the withFilter is left in. This would make using Scalactic Or nicer, so I'd be quite happy about a tweak like this. For example, would give no withFilter:

for { (a, b) <- List(1 -> 'a', 2 -> 'b') } yield (foo(a) compose bar(b))

But this would give the withFilter, because the compiler can't prove that Any will always match Tuple2:

for { (a, b) <- List(1, 2 -> 'b') } yield (foo(a) compose bar(b))

The flag idea isn't bad either, though I wonder if there could be a more general flag that adds type errors. This behavior might belong more in a compiler plugin though I'd think, like WartRemover.

[Blaisorblade]

@bvenners That's what I would have hoped for. In particular, the call can be removed if:

• withFilter's receiver type has shape Foo[T] (i.e., a type constructor applied to one parameter), and
• the given patterns are exhaustive for T

I guess this is safe with some assumptions on withFilter, which have never been set in stone.
Also, this assumes that the exhaustiveness checker works correctly, and I've seen a few strange warnings there.

The flag idea isn't bad either, though I wonder if there could be a more general flag that adds type errors.

I'm sorry, but I'm lost here on which type errors you'd want.

[bvenners]

I'm sorry, but I'm lost here on which type errors you'd want.

@Blaisorblade I just meant that there will likely be many situations where typelevel folks would prefer a compiler error, and perhaps they should just all be enabled by one flag instead of 100 different flags. Then on the other hand, perhaps that's better done in a compiler plugin, with a config file to enable and disable the various options. Else the compiler has too many options and modes, which adds complexity.

[Blaisorblade]

@bvenners Ah, that's a fair point. If you trust Typelevel, you might want -Ztypelevel-is-my-guide. And for hard-core purists, -Zscalazzi.

But I'd never enable this particular one, so I'd like some granularity: I'd be happy with -Ztypelevel-is-my-guide -Zno-irrefutable-generator-patterns :-)

[puffnfresh]

@bvenners that has been tried, but here's the problem:

1. The parser desguars to withFilter and patterns
2. The type-checker checks that withFilter exists
3. An optimiser could possibly elide the withFilter calls if the pattern is irrefutable

The problem with step 2 prevents us from using lots of structures which don't support withFilter. The problem with step 3 is that it doesn't happen (try it out with scalac, I haven't observed it working under very simple cases).

This is what scalac (claims to) do but the spec says step 2 shouldn't even happen. I don't have a way to make scalac conform to the spec (and neither did paulp, it seems).

[puffnfresh]

@Blaisorblade I don't want to compare how Haskell desugars do-notation since I think it can be done better (but possibly not by us).

With this PR you get a warning now on refuted patterns. For example, given this:

sealed trait X
case class A(a: Int) extends X
case object B extends X
case object C extends X

object Test {
  for {
    A(x) <- List(A(1), B, C)
  } yield ()
}

We get this result when compiling:

$ ./qbin/scala -nc -Zirrefutable-generator-patterns -Xfatal-warnings /tmp/Test.scala
/tmp/Test.scala:8: warning: match may not be exhaustive.
It would fail on the following inputs: B, C
    A(x) <- List(A(1), B, C)
         ^
error: No warnings can be incurred under -Xfatal-warnings.

[Blaisorblade]

Forgot to mention: happy to see you here, @paulp!

[paulp]

@Blaisorblade I brought up Any as a representative of heterogeneous lists of the List[Whatever] variety. From first principles one could reasonably expect language support for the notion that one's generator will only generate elements of the expected shape, but there is no such facility. It's not dissimilar to the contains(Any) problem, in that generality is a two-way street.

[paulp]

I hope this ticket serves to illustrate that even the smallest changes to the compiler often involve a dizzying array of consequences, and also that there will often be a long history of past efforts orbiting any seemingly simple change which you care to pursue. You will want to hone your facility with git so you can read some punchy commit messages, many of which I expect have never been seen except by the author.

Also the ticket database remains an important source of background, which I have to mention because it is a daily occurrence to see someone who should know better opening a new ticket duplicating one I opened up to five years ago yet which languishes in its meticulous obscurity amongst the ~2000 and climbing open tickets.

[bvenners]

@puffnfresh For my use case, I do have a withFilter on Or, so fixing step 3 and just optimizing the withFilter out would address my need. It wouldn't solve the issue with an optimize-out-able withFilter not being present at type check time, but would help Or users. The reason I care is explained here:

https://groups.google.com/forum/#!msg/scalatest-users/XPtBcsFvmDk/I5fhy1AiKSIJ

Have to scroll down to where I say "Actually that's on purpose." If step 3 worked, then that user would not have encountered a compiler error where he did in the first place. Admittedly I suspect this is a very rare case, but it would be a step-wise improvement to the compiler.

[Blaisorblade]

@Blaisorblade I brought up Any as a representative of heterogeneous lists of the List[Whatever] variety. From first principles one could reasonably expect language support for the notion that one's generator will only generate elements of the expected shape, but there is no such facility.

I'd want a static error for that, and that's possible via a type annotation.

Also, compared to non-OO functional languages with algebraic data types (ADTs), as a built-in, Scala subtyping relates both an ADT with its variants (above, T and C) and a supertype with its type. In Haskell, the difference is rather clear — but in Scala, they're both expressed with subtyping, so you need a solution which supports both.

It's not dissimilar to the contains(Any) problem, in that generality is a two-way street.

I agree at least that typesystems are useful because they rule out (some) bad code while allowing good one.

[puffnfresh]

@paulp in pursuing this change, I find your commits and your issue comments. They were very useful to see why Typesafe didn't include them into scalac but I see the behaviour they desire as broken and so I am happy with my change.

[Blaisorblade]

@milessabin while I would love to turn SIP-18 against itself, the checkFeature part is in Typers and I don't think it can be applied a phase which is this early.

Ignoring the code, for SIP-18 you need to check whether implicits of some type are available, and there's no way to know that before typechecking.

IIUC, Haskell-style flags can be understood by the parser so they wouldn't have this problem. They're more limited, but I've not seen yet anybody using the extra power.
We're going to have also other flags which affect pre-Typer behavior (see #7 for the Unicode stuff), so we might consider a Haskell-like syntax for Typelevel language flags.

[paulp]

I'd want a static error for that, and that's possible via a type annotation.

@Blaisorblade If you have to tell the compiler so the compiler can tell you, I suggest cutting out the middleman.

[Blaisorblade]

@Blaisorblade If you have to tell the compiler so the compiler can tell you, I suggest cutting out the middleman.

To me this boils down to your old "don't infer Any" proposal, for which I've opened #16. Let's continue there.

[bvenners]

@puffnfresh it occurred to me last night on the couch that probably the Or example wouldn't make it pass the type, because even though there's a withFilter on there, the argument doesn't have the expected type. So probably it would still fail there and I'd need the whole enchilada. Would be nice if that could be accomplished someday.

[aloiscochard]

On 4 September 2014 21:47, Paolo G. Giarrusso [email protected]
wrote:

@milessabin https://github.com/milessabin while I would love to turn
SIP-18 against itself, the checkFeature part is in Typers and I don't think
it can be applied a phase which is this early.

Ignoring the code, for SIP-18 you need to check whether implicits of some
type are available, and there's no way to know that before typechecking.

IIUC, Haskell-style flags can be understood by the parser so they wouldn't
have this problem. They're more limited, but I've not seen yet anybody
using the extra power.
We're going to have also other flags which affect pre-Typer behavior, so
we might consider a Haskell-like syntax for Typelevel language flags.

I thought we might have that issue :-/

If we need flags which work pre-typer, we should probably then all make
them (even the non pre-typer) work with a Haskell-like syntax for sake of
consistency.

Λ\ois
http://twitter.com/aloiscochard
http://github.com/aloiscochard

[typelevel-bot]

Can one of the admins verify this patch?

[puffnfresh]

add to whitelist

[aloiscochard]

retest this please