Nested . syntax

[davidanthoff]

Would it make sense to allow "nested" application of the . syntax for broadcasting? For example

a = rand(3); b = rand(3);
c = [a, b]
log..(c)

Would translate to

broadcast(i->log.(i),c)

and then

broadcast(i->broadcast(j->log(j), i),c)

I imagine this would be especially useful with Nullables, and arrays of Nullables. For example, this would work:

a = [Nullable(3.), Nullable(2.)]
log..(a)

The first dot would apply log. to each array element. And because we have now used log. on the array elements, we get the nice new lifting for these values.

Even crazy things like this would work:

a = Nullable([Nullable(3.), Nullable(2.)])
log...(a)

Here the first dot would lift, the second dot would apply to the elements of the array, and the third dot would lift again.

I think that also relates to the issue that @JeffBezanson had here.

I have to admit that I didn't follow the whole Nullable discussion lately, in which case I'm sorry to bring this up again.

@nalimilan, @davidagold, @johnmyleswhite Not sure who else might be interested.

[johnmyleswhite]

I would support something like this to make working with arrays of nullables easier.

[stevengj]

.. is already a binary operator, so this would be breaking.

There have also been competing proposals for .., e.g. vector..field in #19169 or sum..(f.(g.(x))) for sum(x->f(g(x)), x) in #16285.

[nalimilan]

We indeed need a way to perform 2-level nested broadcast to be able to replace NullableArray with plain Array{Nullable} in the long term (as currently NullableArray automatically lifts when broadcasting, contrary to standard arrays). But I'm not sure I like f..(X).

If the goal is to find a notation with two dots, we could use a pun like f:(X). Though it may be worse than ...

[stevengj]

I also suggested two dots for fusing with non-broadcast calls, like sum:(f.(g.(x)) .+ 3). To me this seems more natural. (Though it would also be breaking.)

Why can't Array{Nullable} automatically lift when broadcasting?

[nalimilan]

Why can't Array{Nullable} automatically lift when broadcasting?

See Jeff's reaction linked to in the issue description.

[bramtayl]

sum:(f.(g.(x)) .+ 3)

If this worked, then presumably

lift:(f.(g.(x)) .+ 3)

could also work?

P.S. I have a new package out called LazyCall.jl which enables doing things like this:

@unweave( f(g(~x)) + 3) |> lift

[StefanKarpinski]

At the point where you'd need this syntax, I think you're better of just calling broadcast explicitly.

[davidanthoff]

At the point where you'd need this syntax, I think you're better of just calling broadcast explicitly.

@StefanKarpinski I'm not following you here. Surely lifted operations on the elements of Array{Nullable} are not a rare corner case at all (especially if that structure replaces NullableArray in the long run), are you suggesting that those cases should always be handled with an explicit broadcast call? That seems a pretty impractical suggestions for an area (data handling) that seems quite important to julia? Or am I misunderstanding what you meant here?

[StefanKarpinski]

I'm still pretty skeptical about broadcasting being the right way to express lifting nullable operations.

[nalimilan]

Well, that works well for NullableArray since for that type broadcasting also implies lifting. But indeed it doesn't work very well with any other structure where nested broadcast is needed.

[davidanthoff]

I'm still pretty skeptical about broadcasting being the right way to express lifting nullable operations.

@StefanKarpinski But isn't that the sanctioned, official approach in base now? Merged, reviewed and accepted on master?

[bramtayl]

Well, that works well for NullableArray since for that type broadcasting also implies lifting. But indeed it doesn't work very well with any other structure where nested broadcast is needed.

Maybe. Though it should be theoretically possible to define a broadcast_recursive or a broadcast(levels = 2) method. Then you could do:

broadcast_recursive:(f.(g.(x)) .+ 3)

or

broadcast:(f.(g.(x)) .+ 3, levels = 2)

[andyferris]

While powerful, how many consecutive dots can we realistically aim to support while having readable code?

While I have felt the lack of a nested broadcasting option for working with vectors of vectors, there are other options for nullables. One thing being Swift-like syntax for lifting to make an explicit but non-verbose approach (perhaps involving ?, ?? or whatever). Another is Jameson's recent work on making union types super fast, which would let the old NA approach work as efficiently as nullable (and there are other suggested changes to Nullable related to this). I feel it would be worth exploring the implications of those, first, and then see if nested broadcasting is really the thing we need to make arrays of nullables more convenient.

[nalimilan]

While I have felt the lack of a nested broadcasting option for working with vectors of vectors, there are other options for nullables. One thing being Swift-like syntax for lifting to make an explicit but non-verbose approach (perhaps involving ?, ?? or whatever).

f?(x) was discussed before making f.(x) lift. I still find it an interesting possibility, but that only works for the scalar case: you still need to find something for lifting broadcast, and f?.(X) isn't really better than f..(X).

Another is Jameson's recent work on making union types super fast, which would let the old NA approach work as efficiently as nullable (and there are other suggested changes to Nullable related to this). I feel it would be worth exploring the implications of those, first, and then see if nested broadcasting is really the thing we need to make arrays of nullables more convenient.

You would still need a way to indicate that you want lifting, i.e. f(nothing) -> nothing. Or we would have to make this the default (i.e. a default fallback for any method involving a missing value), possibly by using a different value than nothing to indicate missingness (e.g. null).

[TotalVerb]

Perhaps, if f..(x) is unavailable/undesirable, there is still (f.).(x). Of course this does not work well for x .(.+) y.

[TotalVerb]

Another idea is to allow dots in addition to the @. macro that has been added recently, so that

@. Nullable{Int}[1, 2, 3] .+ Nullable{Int}[4, 5, 6]

is considered to make the .+ behave like a second order broadcast. There is a nice parallel between

@. Int[1, 2, 3] + Int[4, 5, 6]

and

@. Nullable{Int}[1, 2, 3] .+ Nullable{Int}[4, 5, 6]

[nalimilan]

I'm more and more convinced that the only workable approach would be to define a new Null type for which lifting happens automatically. We would no longer need these hacks to combine broadcasting and lifting, nor would we need NullableArray to do both at the same time: a standard Array would be enough.

This can be done by replacing Nullable with Union{T, Null} and having a fallback function so that for any function f we have f(::Null) = Null() (unless maybe f(::Null) was explicitly overridden, for example for isnull(::Null) = true). IMHO this is an option that should be discussed more seriously in the Nullable Julep, while it currently isn't mentioned at all.

[andyferris]

I happen to agree with @nalimilan - this seems the natural thing to do in a dynamic language like Julia, and if it happens to become fast, then that seems like the most logical choice all around. Not that NA always worked straight out of the box since there are always more methods to define, but IMO that's life (some of the other nullable options seem more extreme/difficult to use in my opinion).

(There is a discussion of Union{T, Null{T}} in the Julep, as well as Union{T, Void})

[nlw0]

I just ran into something like this, and see there's a few similar open issues. Not sure where to comment, but here it goes.

I was working with multidimensional arrays, but felt the need to have non-uniform dimensions. The data can be simply stored as a "list of lists" instead of a structured uniform array but now what were simple map or broadcast operations require a "recursive map".

In my opinion, the good approach to solve this looking from a FP perspective as well, is to have a data-structure where map and broadcasting will reach the leaves. A tree, for instance. So we might even have a "tree view" of an array-of-arrays, and then when you map or broadcast on that object, you get the desired behavior. We just need the different class that offerst the proper catamorphism/isomorphism or whatever.

Is there already any standard or popular library that offers something like this?