An example that uses Scatterv

[francispoulin]

I've put together an example (see below) that builds a vector of length N (=10) and then determines how to split this amoungst the different cores and then does just that.

I have a few questions in hopes of learning how to write better code.

1. In Python there is a function numpy.array_split that does the splitting for you. Any chance Julia has something similar?
2. At the end I print out the results and it's difficult to parse because everything printing at the same time. What's a better way to show the output of code that uses MPI.jl?
3. It seems to sometimes store the order of the arrays differently. Is there any way to ensure that the order is done in a sequential fashion?

Below is output that maybe shows what I mean about my output being messy and the ordering being a bit odd. It seems that rank 1 has [7, 8], and rank 3 has [4, 5, 6].

Any advice would be greatly appreciated.

Sample output

rank = 0 k_local = rank = rank = rank = 3 k_local = 2 k_local = 1 k_local = [1.0, 2.0, 3.0] 
[4.0, 5.0, 6.0] 
[9.0, 10.0] 
[7.0, 8.0] 

Code

using MPI
using Printf

MPI.Init()

comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
size = MPI.Comm_size(comm)

N = 10

### Find the split for each rank with a given size
N_local   = zeros(Int64, size)
if rank == 0
    N_guess   = Int64(div(N, size, RoundDown))
    Remainder = Int64(N - N_guess*size)

    for i in collect(1:size)
        if i <= Remainder
            N_local[i] = N_guess + 1
        else
            N_local[i] = N_guess
        end
    end
end

MPI.Bcast!(N_local, 0, comm)

### Define a sample array of size N
if rank == 0
    k = zeros(Float64, N)
    for i in collect(1:N)
        k[i] = i
    end
else
    k = zeros(Float64, 1)
end

### Scatter Array
k_local = MPI.Scatterv(k, N_local, 0, comm)

### Print Array
for i in collect(1:size)
    if rank==(i-1)
        @printf("rank = %s k_local = %s \n", rank, k_local)
    end
end

MPI.Finalize()

[simonbyrne]

What version of MPI.jl are you using? MPI.Scatterv was deprecated in #335 (now you should use MPI.Scatterv! with a VBuffer object). Something like the following should work:

using MPI
using Printf

MPI.Init()

comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
size = MPI.Comm_size(comm)

N = 10

### Find the split for each rank with a given size
if rank == 0
    N_each = cld(N, size)
    N_all  = fill(N_each, size)
    N_all[end] = N - size*N_each
end

RN_local = Ref{Int}()
MPI.Scatter!(rank == 0 ? N_all : nothing, RN_local, 0, comm)
N_local = RN_local[]

### Define a sample array of size N
if rank == 0
    k = Float64[i for i in 1:N]
end
k_local = zeros(N_local)

### Scatter Array
MPI.Scatterv!(rank == 0 ? VBuffer(k, N_all) : nothing, k_local, 0, comm)

### Print Array
for i in 1:size
    if rank==(i-1)
        @printf("rank = %s k_local = %s \n", rank, k_local)
    end
end

MPI.Finalize()

1. In Python there is a function numpy.array_split that does the splitting for you. Any chance Julia has something similar?

Not that I know of, but it is a fairly simple function to write.

2. At the end I print out the results and it's difficult to parse because everything printing at the same time. What's a better way to show the output of code that uses MPI.jl?

Unfortunately there aren't any great solutions here. See #360

3. It seems to sometimes store the order of the arrays differently. Is there any way to ensure that the order is done in a sequential fashion?

I'm not sure what you mean by this, could you expand further, or give an example?

[simonbyrne]

Also, for these sorts of questions, please ask on Discourse in future.

[francispoulin]

Thank you @simonbyrne for the reply, and sorry for not using Discourse for this. I've since discovered where it is and will try that in the future.

First, I am using v0.17.2, which I see is the latest version.

Second, sorry for using the wrong function and will make sure to use the right functions. I asked for help on the function I was using and it showed the following, in case this is helpful.

 [1] Scatterv(sendbuf, counts::Vector{T} where T, root::Integer, comm::MPI.Comm) in MPI at deprecated.jl:70

Third, thanks for the revised code. I admit that I do find the code puzzling but I guess I need to learn how Scatterv! works, since it is the standard function. Unfortunately, when I try running your code on 3 cores I get an error. I think if i go through your code line by line then I should be able to figure out the error.

Fourth, sorry my output was hard to understand. I'll copy it below but parse it differently

rank = 0 rank = 3 rank = 2 rank = 1 
k_local = [1.0, 2.0, 3.0]  k_local = [4.0, 5.0, 6.0]  k_local = [9.0, 10.0]  k_local = [7.0, 8.0] 

If I am reading this correctly then rank 3 has the set of elemnts that follow rank 0, and before rank 1. This is going to be a problem when I gather the elements back together, which I want to do next, since the gathered vector will not be ordered correctly, I think.

[francispoulin]

After doing some tests I'm not sure if I was parsing the output correctly.

I decided to specify the displacements in the VBuffer as that seems like it should force the order to be correct. After taking a lot of suggestions from your code I have the following, which works for all the cases I tried.

Below had both examples on how to scatter and then gather an array.

using MPI
using Printf

MPI.Init()

comm = MPI.COMM_WORLD
rank = MPI.Comm_rank(comm)
size = MPI.Comm_size(comm)

function array_split(N, size, comm)

    counts = zeros(Int64, size)
    displs = zeros(Int64, size)

    if rank == 0
        counts_guess = Int64(div(N, size, RoundDown))
        Remainder    = Int64(N - counts_guess*size)
        counts[:]   .= counts_guess

        for i in 1:Remainder
            counts[i] += 1
        end

        displs[:] = cumsum(append!([0], counts))[1:size]
    end

    MPI.Bcast!(counts, 0, comm)
    MPI.Bcast!(displs, 0, comm)

    return counts, displs
end

N = 10

### Find counts and displacement arrays for scattering
counts, displs = array_split(N, size, comm)

### Define array
rank == 0 ? k = Float64[i for i in 1:N] : k = nothing
rank == 0 ? @printf("original k = %s \n", k) : nothing

### Scatter array 
k_local = zeros(Float64, counts[rank+1])
MPI.Scatterv!(rank == 0 ? VBuffer(k, counts, displs) : nothing, k_local, 0, comm)
@printf("rank = %s  k_local = %s \n", rank, k_local)

### Gatter array
rank == 0 ? knew = zeros(Float64, N) : nothing
MPI.Gatherv!(k_local, rank == 0 ? VBuffer(knew, counts, displs) : nothing, 0, comm)
rank == 0 ? @printf("gathered k = %s\n", knew) : nothing

MPI.Finalize()

[simonbyrne]

If I am reading this correctly then rank 3 has the set of elemnts that follow rank 0, and before rank 1. This is going to be a problem when I gather the elements back together, which I want to do next, since the gathered vector will not be ordered correctly, I think.

I think that is just the printing order getting messed up.