FFTW wisdom is not thread safe?

[jtravs]

See http://www.fftw.org/fftw3_doc/Usage-of-Multi_002dthreaded-FFTW.html
I'm still investigating but it seems that wisdom cannot be used for different numbers of threads. Perhaps have a thread specific wisdom file?

[chrisbrahms]

I think it's only the planning itself that's not threadsafe JuliaMath/FFTW.jl#66

[chrisbrahms]

actually I think even that is safe now JuliaMath/FFTW.jl#105

The other thing to be wary of is that FFTW.jl by default now plans multi-threaded FFTs (JuliaMath/FFTW.jl#121) so concurrently calling multiple FFTs from different threads may not give any speedup

[jtravs]

The other thing to be wary of is that FFTW.jl by default now plans multi-threaded FFTs (JuliaMath/FFTW.jl#121)

This does not appear to be true. In that I get very different results when setting

FFTW.set_num_threads(8)

and if I don't set it the performance monitor shows that only a single thread is used.

[jtravs]

And those PRs did not deal with wisdom.

[jtravs]

E.g. FFTW (the C library) is also thread safe, but does not allow wisdom to be shared for different numbers of threads, see the link I sent.

[chrisbrahms]

Sorry, misunderstood your original point. Yes we probably need one wisdom file for each number of threads. That should be easy to implement.

This does not appear to be true. In that I get very different results when setting

FFTW.set_num_threads(8)

and if I don't set it the performance monitor shows that only a single thread is used.

That's odd, I'm pretty sure I've seen it plan multi-threaded FFTs by default. Is there a difference if you set

FFTW.set_num_threads(4)

instead? AFAIU it defaults to 4 threads (assuming JULIA_NUM_THREADS is 1) so maybe for that number of threads, the single-threaded variant is faster so the planner chooses it?

[jtravs]

So if I do not run FFTW.set_num_threads(4) then both planning and executing an FFT plan take exactly the same time regardless of the number of Julia threads (I tried 1, 4 and 6 Julia threads).

If I do run FFTW.set_num_threads(4) then planning takes much much longer, and executing the FFT is moderately faster (about 30%, as expected for a 1D fft that is not too big, it is not that good at being parallelised). This is regardless of the number of Julia threads.

This is on FFTW 1.2.0 and Julia 1.4.1.

So it seems something is wrong. I might open an issue on this. But note that none of this changes the wisdom story. THE FFTW.jl code directly calls the FFTW c library wisdom code, so I'm sure that must be different amongst threads, and clearing the wisdom cache got rid of a nasty hard to find bug for me.

EDIT: this seems to be JuliaMath/FFTW.jl#138 I'll check on FFTW master

[jtravs]

Nope, on FFTW master this is still not fixed.

[jtravs]

Actually it does work, but only for if the number of Julia threads is strictly > 1, so for 1 thread (default) it does not help. This should give a huge speedup for freespace sims. (Even for 1D sim it gives double speed).

Note that FFTW.PATIENT is even longer now, as for 4 threads it asks for 16 (!) which takes an age to plan (even just for 1D) so wisdom will be really needed.

[chrisbrahms]

So if JULIA_NUM_THREADS is 1, everything is single-threaded unless you set FFTW.set_num_threads explicitly. If JULIA_NUM_THREADS > 1, then the number of FFTW threads is 4x JULIA_NUM_THREADS unless set otherwise by FFTW.set_num_threads.

Did I get that right?

[jtravs]

Yes, on fftw master.

[chrisbrahms]

Note this is now released https://github.com/JuliaMath/FFTW.jl/releases/tag/v1.2.1

[jtravs]

Yes, I asked them to release it!
JuliaMath/FFTW.jl#138 (comment)

[chrisbrahms]

Ah fair enough 😆