pmap slowdown since 0.4

[andreasnoack]

Inspired by a julia-users question, I've timed pmap and it's become quite a bit slower. It has went through various changes during the development of 0.5 and the test case is not of the type that pmap is supposed to do well for but since the slowdown is quite large, I'll report it anyway

Test code

julia> addprocs(10)                                                                
10-element Array{Int64,1}:                                                         
  2                                                                               
  3                                                                                
  4                                                                                
  5                                                                                
  6                                                                                
  7                                                                                
  8                                                                                
  9                                                                                
 10                                                                                
 11                                                                                

julia> x = [bitrand(4,2) for i = 1:100000];                                        

julia> @time pmap(identity, x);

The slowdown didn't start right after 0.4 came out so the first commit below is from January after we switched to 3.7.1.

d4749d2 (Old)

Takes 10.619339 seconds.

853317d (New)

Takes 18.537377 seconds

UPDATE:

This might be a general issue with our parallel infrastructure. With the same version of DistributedArrays.jl I get

d4749d2 (Old)

julia> @time distribute(x);
  2.828939 seconds (4.56 M allocations: 270.816 MB, 2.46% gc time)

julia> @time distribute(x);
  0.560839 seconds (2.54 M allocations: 184.086 MB, 9.37% gc time)

853317d (New)

julia> @time distribute(x);
 14.178851 seconds (44.49 M allocations: 1.438 GB, 17.24% gc time)

julia> @time distribute(x);
 12.930582 seconds (43.28 M allocations: 1.368 GB, 22.75% gc time)

cc: @amitmurthy

[amitmurthy]

The second issue is JuliaParallel/DistributedArrays.jl#72

Will look into the first one.

[amitmurthy]

Just to note that pmap supports batching in 0.5.

For the test case above on my machine:
@time pmap(identity, x); takes 15.3 seconds on master.
@time pmap(identity, x; batch_size=1000); takes 0.9 seconds on master.
@time pmap(identity, x); takes 6.6 seconds on 0.4
@time pmap(identity, x); takes 8.8 seconds with #17331

[multidis]

I am observing a factor of 18 to 20 slowdown with pmap on Julia 0.5. The more cores are made available to pmap, the slower it gets. Please see:
JuliaNLSolvers/Optim.jl#290 (comment)

[StefanKarpinski]

Does using a batch size help at all or is there some other issue?

[amitmurthy]

#17331 addressed a large part of the regression. @multidis 's issue was a little different (see referenced issue in Optim)

[KristofferC]

Seems to be around a 15-20% difference now. Perhaps enough to close?