Sparsetimefunctions fail with MPI & DEVITO_FIRST_TOUCH=1

[deckerla]

import devito
import numpy as np

grid = devito.Grid(shape=(25,25,25))
stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
u = devito.TimeFunction(name="u", grid=grid)
inj = stf.inject(field=u.forward, expr=stf)

op = devito.Operator(inj, name="inj")
op.apply()

This runs fine without MPI:

cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ python mfe.py 
Operator `inj` ran in 0.02 s

And with MPI:

cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ DEVITO_MPI=1 mpirun -n 2 python mfe.py 
Operator `inj` ran in 0.18 s
Operator `inj` ran in 0.18 s

Running this with first touch & MPI fails with the following behavior:

cvx@cbox-lukedecker-nocollapse:~/.julia/dev/JetPackDevito/test$ DEVITO_FIRST_TOUCH=1 DEVITO_MPI=1 mpirun -n 2 python 
mfe.py 
Traceback (most recent call last):
  File "/home/cvx/.julia/dev/JetPackDevito/test/mfe.py", line 5, in <module>
Traceback (most recent call last):
  File "/home/cvx/.julia/dev/JetPackDevito/test/mfe.py", line 5, in <module>
    stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
    stf = devito.SparseTimeFunction(name="stf", npoint=1, nt=11, grid=grid, coordinates=np.array([0.5,0.5,0.5]))
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
    newobj.__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 307, in __init_finalize__
    newobj.__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 307, in __init_finalize__
    super(AbstractSparseTimeFunction, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 463, in __init_finalize__
    super(AbstractSparseTimeFunction, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/sparse.py", line 463, in __init_finalize__
    self._coordinates = SubFunction(
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
    self._coordinates = SubFunction(
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/basic.py", line 850, in __new__
    newobj.__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1464, in __init_finalize__
    newobj.__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1464, in __init_finalize__
    super(SubFunction, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1002, in __init_finalize__
    super(SubFunction, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 1002, in __init_finalize__
    super(Function, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 94, in __init_finalize__
    super(Function, self).__init_finalize__(*args, **kwargs)
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/types/dense.py", line 94, in __init_finalize__
    self.data_with_halo[:] = initializer
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/finite_differences/differentiable.py", line 157, in __getattr__
    self.data_with_halo[:] = initializer
  File "/home/cvx/.conda/envs/conda_jl/lib/python3.10/site-packages/devito/finite_differences/differentiable.py", line 157, in __getattr__
    raise AttributeError("%r object has no attribute %r" % (self.__class__, name))
    raise AttributeError("%r object has no attribute %r" % (self.__class__, name))
AttributeError: stf_coords object has no attribute 'data_with_halo'. Did you mean: '_data_with_inhalo'?
AttributeError: stf_coords object has no attribute 'data_with_halo'. Did you mean: '_data_with_inhalo'?
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[21223,1],0]
  Exit code:    1
--------------------------------------------------------------------------

[FabioLuporini]

if DEVITO_FIRST_TOUCH actually useful? do you see benefit from using it?

[deckerla]

Honestly I haven’t used it much. I was exploring the use of it for increasing performance. If you think it won’t do much, then don’t prioritize it.

…

On May 26, 2023, at 2:30 AM, Fabio Luporini ***@***.***> wrote: if DEVITO_FIRST_TOUCH actually useful? do you see benefit from using it? — Reply to this email directly, view it on GitHub <#2132 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADX3SS2LARQ7JKVMOS3V653XIBL2ZANCNFSM6AAAAAAYFRBFQM>. You are receiving this because you authored the thread.