Enabling collection pipelines

[tomasaschan]

I've started looking at what additions would be required to truly enable collection pipeline-style programming in Julia, now that anonymous functions will no longer be a deal-breaker for performance. I'm imagining something like this:

( # wrap in parens to keep it as one expression despite line breaks
countfrom(1)
    |> filter(iseven)
    |> map(square)
    |> drop(5)
    |> take(10)
    |> collect)

# or
(countfrom(1)
    |> filter(iseven)
    |> take(10)
    |> map(sqrt)
    |> sum
    |> inv

This would entail creating methods for functions like map, reduce and filter as well as for the iterator constructors take, skip et al, that would look approximately like this

map(f::Function) = xs -> map(f, xs)
take(n::Int) = xs -> take(xs, n)

Before I embark on a journey to try to find functions where this method addition a) is not in conflict with an already existing method, and b) would make sense semantically, I'd like to probe the community's interest in this kind of construct. Is a PR to this effect likely to get merged? Are there any important considerations that I might be overlooking? What functions make good candidates to have methods like these?

---

I looked around on the issue tracker, but the only relevant issue I found was #554 about (semi-)automatic currying of all functions, and was quickly dismissed as being impossible to combine with multiple dispatch.

I also found this thread on julia-dev, which talks about changing the behavior of |> and <|; I guess a different approach here would be to change the behavior of |> to provide the mechanics of map(f::Function) = xs -> map(f, xs) automatically, but that's a much bigger change with potential to break a lot of stuff.

[eschnett]

I don't think that (semi-mechanically) adding a lot of function definitions is the right way to go. Modifying the behaviour of |> is a much cleaner approach.

If you think there's potential that this might break things, then you can either test it out, or you can define a new operator (different from |>) with your new semantics.

Here is what I would do:

• Define a generic function lift(f, x) = a -> f(x, a)
• Write lift(map, sqrt) instead of map(sqrt); this essentially implements your "rewrite map step"

Here is an example:

julia> [1,2,3] |> lift(map, x->x+1)
3-element Array{Int64,1}:
 2
 3
 4

[johnmyleswhite]

I can't quite tell, but I think this might be a dupe of #5571. I still think the approach that Hack has taken is the way we should eventually redefine |>.

[tomasaschan]

Yes, this is more or less a dupe of #5571 - thanks for finding that for me!

I tried reading through that discussion in its entirety, but it was too long and jumped too much back and forth between different syntax proposals for me to be able to grasp the current stance on this feature. Basically, it seems to me that a lot of people want it, but there isn't a syntax that's clear for it.

I'll close this in favor of #5571, to avoid splitting this discussion.