Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make some computations in DFTK GPU-compatible #712

Merged
merged 98 commits into from
Nov 22, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
e17fb59
LOBPCG with GPU support (CUDA). Does not yet support preconditionning
GVigne Jul 6, 2022
e80f5b6
Merge branch 'master' into gpu_hpc
GVigne Jul 8, 2022
ed15b32
MWE for self_consistent_field with GPU support (CUDA). Only works wit…
GVigne Jul 18, 2022
19bfa69
Fix package version conflicts while merging
GVigne Jul 18, 2022
f4748ac
Stop using BlockArrays and use a custom BlockVector for GPU compatibi…
GVigne Jul 20, 2022
94f1d2a
GPU support for AtomicLocal term
GVigne Jul 22, 2022
60d8041
First GPU implementation of the non local term + LOBPCG enhancement
GVigne Jul 25, 2022
fb6484a
Merge branch 'master' into gpu_hpc
GVigne Jul 25, 2022
cf1dc3c
add timed examples
mfherbst Jul 25, 2022
11b85f0
Change some code organisation after PR's feedback
GVigne Jul 28, 2022
abb99f4
Code organisation and performance optimisation after PR's feedback
GVigne Jul 28, 2022
a89171a
Code refactoring following PR's feedback
GVigne Aug 2, 2022
44bcb61
PWB is now parametric on the array type: this also fixes type issues
GVigne Aug 9, 2022
646b44c
Update workarounds: remove iszero and isone, add eigen
GVigne Aug 16, 2022
76c697d
Rename block_mul into * + build e on GPU
GVigne Aug 22, 2022
bd684d7
Modify the change of basis functions to be GPU compatible
GVigne Aug 23, 2022
f02c954
Merge branch 'master' into gpu_hpc
GVigne Aug 23, 2022
15d1324
Keep this branch synced with LOBPCG_GPU
GVigne Aug 23, 2022
62d9f79
Add the Hartree term
GVigne Aug 23, 2022
19100cf
Remove CUDA dependency from ortho_qr
GVigne Aug 23, 2022
1184ec1
Bugfix when plotting bandstructure + typo fixes
GVigne Aug 29, 2022
833928b
Make all mixings except Chi0 mixing GPu compatible
GVigne Aug 30, 2022
e12f35b
Prettier way to overload eigen for CuArrays
GVigne Sep 1, 2022
a0c4066
Update comments + remove unnecessary code
GVigne Sep 6, 2022
9cdff93
Update the GPU example
GVigne Sep 6, 2022
20b7b10
Put the Gvectors for each kpoint on the GPU
GVigne Sep 7, 2022
a2d811b
Bugfix after launching the tests
GVigne Sep 7, 2022
8ee55a4
Put the occupation on GPU
GVigne Sep 7, 2022
7909720
Merge branch 'master' into gpu_hpc
GVigne Sep 7, 2022
b700d1c
GPU compatibility for Anderson acceleration
GVigne Oct 5, 2022
981e7a1
Remove unnecessary dependencies
GVigne Oct 6, 2022
272ff85
Merge commit '7bd07e3cbbe847ad71eb2f7f2524d1ff70e47b5b' into gpu_hpc
GVigne Oct 6, 2022
c8af93d
Merge commit 'b83c3cc3c3b3f551970aa8bda1379203f6797669' into gpu_hpc
GVigne Oct 6, 2022
acd4bef
Merge commit '3448b533d3486278bda44689963c72d5168e83fa' into gpu_hpc
GVigne Oct 6, 2022
06b07cc
Merge commit 'a96f551ed78a47f6cbd5280fedf08f6a7443d31e' into gpu_hpc
GVigne Oct 6, 2022
8b294da
Merge commit 'd28391e6ffb366168fc43ea61e5cac9d04b252e3' into gpu_hpc
GVigne Oct 6, 2022
b5b0d09
Merge commit '3ce3b2c5e84f728d12b8e74792ecd31c24ddd5e0' into gpu_hpc
GVigne Oct 6, 2022
9177e8b
Merge commit '353c11f1871adffae8bcbad9aadb95faa929f718' into gpu_hpc
GVigne Oct 6, 2022
5c3b620
Crude bugfix for the nbands algorithm
GVigne Oct 6, 2022
bf77db2
Less CPU-GPU transfers in guess_density + whitespaces
GVigne Oct 6, 2022
ce05d66
Merge branch 'master' into gpu_hpc
GVigne Oct 11, 2022
0c8067e
Whitespaces, typos + stick to the same conventions
GVigne Oct 11, 2022
95fe043
Merge branch 'master' into gpu_hpc
GVigne Oct 13, 2022
b511225
Removed unnecessary imports
GVigne Oct 14, 2022
28e05e7
WIP: solve dependency issues
GVigne Oct 14, 2022
1c5a5b5
Adapt the workarounds for forward differentiation to the new API
GVigne Oct 14, 2022
06eb5b6
WIP: remove TaskLocalRNG for julia versions < 1.6
GVigne Oct 14, 2022
de28ac4
Better fallback for TaskLocalRng for Julia < 1.6
GVigne Oct 14, 2022
866d6bf
Change build_fft_plans in fft_generic.jl to take arrays as arguments
GVigne Oct 14, 2022
2008199
Merge branch 'master' into gpu_hpc
GVigne Oct 17, 2022
7ad1744
Fixes
mfherbst Oct 17, 2022
278a001
Merge commit '175937aaff610da98b5606f06dbaf62029af31e0' into gpu_hpc
GVigne Oct 24, 2022
a3e9acc
Merge commit '9381bf4e250f633673958e9f90c47faffc8a7144' into gpu_hpc
GVigne Oct 24, 2022
682dbde
Merge commit '3b1bffff7f116ad239fbea17cae1acc150213736' into gpu_hpc
GVigne Oct 24, 2022
ca75332
Merge commit '5f24d3dbde5aa71f4c3367f0ba5558fefd22dada' into gpu_hpc
GVigne Oct 24, 2022
f6e45b9
Fix tests
GVigne Oct 24, 2022
bd4e315
Update examples/gpu.jl
mfherbst Oct 26, 2022
a339a4d
Fix CPU performance bug when using FFTs
GVigne Oct 28, 2022
695263a
WIP: remove the AT type in PWB
GVigne Nov 2, 2022
7afc5fd
WIP: copy_like also takes Type arguments
GVigne Nov 2, 2022
2563c58
Remove AT parameter from PWB
GVigne Nov 2, 2022
2d17ebc
Merge commit '6e6d523b4780f43cc534153bd1de929907295bf4' into gpu_hpc
GVigne Nov 2, 2022
8088d67
Merge branch 'master' into gpu_hpc
GVigne Nov 2, 2022
d9d9da1
Add Kpoints's array type as a parametric type in PWB
GVigne Nov 3, 2022
0a69e2c
Fix serialization test
GVigne Nov 3, 2022
b4e6fd5
Merge branch 'master' into gpu_hpc
mfherbst Nov 5, 2022
ecfd306
Fix example
mfherbst Nov 5, 2022
3401d79
Merge branch 'master' into gpu_hpc
mfherbst Nov 5, 2022
4c45977
Simplify coordinate transformation routines
mfherbst Nov 8, 2022
5e4d37a
More refactoring ... barely tested, might fail
mfherbst Nov 8, 2022
6dc6c82
Fixes
mfherbst Nov 8, 2022
a6fe22b
Remove array_type and use AbstractArchitecture structure instead
GVigne Nov 9, 2022
6e12d3a
Merge branch 'master' into gpu_hpc
GVigne Nov 9, 2022
866598c
Fixes
GVigne Nov 9, 2022
0bcd3d1
Fix failing tests
GVigne Nov 9, 2022
d13fa56
Code clarification and better comments
GVigne Nov 15, 2022
085ca99
Refactoring/code cleanup
GVigne Nov 15, 2022
9d72ba0
Merge branch 'master' into gpu_hpc
GVigne Nov 15, 2022
8c6ac54
Replace sum(abs2(...)) by norm2 function
GVigne Nov 16, 2022
a02cd21
Bugfix: extend norm2 to any array
GVigne Nov 16, 2022
0d845fc
Merge branch 'master' into gpu_hpc
GVigne Nov 16, 2022
349b8f1
Remove convert_like and add to_device
GVigne Nov 17, 2022
778712d
Update comments and docstring
GVigne Nov 17, 2022
c39ee19
Enforce the use of to_cpu instead of Array when doing GPU-CPU transfers
GVigne Nov 17, 2022
4501bd9
Small polishing
mfherbst Nov 18, 2022
60506b9
Formatting, updating comments and small nits
GVigne Nov 21, 2022
147c81d
Remove architecture argument from G_vectors + infer type in kinetic_e…
GVigne Nov 21, 2022
da1ac09
Merge branch 'gpu_hpc' of github.com:GVigne/DFTK.jl into gpu_hpc
GVigne Nov 21, 2022
8a40af5
Inline _closure_matmatmul
GVigne Nov 21, 2022
5ee546b
Rename closures and small changes to ortho_qr and build_kpoints
GVigne Nov 21, 2022
b7b611a
Hard-code CPU architecture
mfherbst Nov 21, 2022
fbe675d
Merge branch 'master' into gpu_hpc
mfherbst Nov 22, 2022
d7be3a6
Reformat basis
mfherbst Nov 22, 2022
482d3b2
Docs concurrency
mfherbst Nov 22, 2022
7070487
Value type instead of real type.
mfherbst Nov 22, 2022
cf3fbf8
Minor nits: broadcast norm2, more type inference
GVigne Nov 22, 2022
dec0653
Merge branch 'gpu_hpc' of github.com:GVigne/DFTK.jl into gpu_hpc
GVigne Nov 22, 2022
ff0d9b0
Update xc.jl
mfherbst Nov 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 7 additions & 8 deletions src/PlaneWaveBasis.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Discretization information for ``k``-point-dependent quantities such as orbitals
More generally, a ``k``-point is a block of the Hamiltonian;
eg collinear spin is treated by doubling the number of kpoints.
"""
struct Kpoint{T <: Real, GT <: AbstractArray}
struct Kpoint{T <: Real, GT <: AbstractVector{Vec3{Int}}}
spin::Int # Spin component can be 1 or 2 as index into what is
# # returned by the `spin_components` function
mfherbst marked this conversation as resolved.
Show resolved Hide resolved
coordinate::Vec3{T} # Fractional coordinate of k-point
Expand All @@ -39,10 +39,9 @@ Normalization conventions:

`ifft` and `fft` convert between these representations.
"""
struct PlaneWaveBasis{T, VT, T_G_vectors, T_r_vectors, T_kpt_G_vecs} <: AbstractBasis{
T
} where {VT <: Real, T_G_vectors <: AbstractArray{Vec3{Int}}, T_r_vectors <: AbstractArray,
T_kpt_G_vecs <: AbstractVector{Vec3{Int}}}
struct PlaneWaveBasis{T, VT, T_G_vectors, T_r_vectors, T_kpt_G_vecs} <: AbstractBasis{T} where
{VT <: Real, T_G_vectors <: AbstractArray{Vec3{Int}},
T_r_vectors <: AbstractArray{Vec3}, T_kpt_G_vecs <: AbstractVector{Vec3{Int}}}
mfherbst marked this conversation as resolved.
Show resolved Hide resolved
# T is the default type to express data, VT the corresponding bare value type (i.e. not dual)
model::Model{T, VT}

Expand Down Expand Up @@ -141,7 +140,7 @@ Base.eltype(::PlaneWaveBasis{T}) where {T} = T
mapping_inv = Dict(ifull => iball for (iball, ifull) in enumerate(mapping))
for iσ = 1:model.n_spin_components
push!(kpoints_per_spin[iσ],
Kpoint{T,typeof(Gvecs_k)}(iσ, k, mapping, mapping_inv, Gvecs_k))
Kpoint(iσ, k, mapping, mapping_inv, Gvecs_k))
end
end
vcat(kpoints_per_spin...) # put all spin up first, then all spin down
Expand Down Expand Up @@ -399,7 +398,7 @@ end
The list of ``G + k`` vectors, in reduced coordinates.
"""
function Gplusk_vectors(basis::PlaneWaveBasis, kpt::Kpoint)
coordinate = kpt.coordinate # Avoid closure on kpt (not isbits)
coordinate = kpt.coordinate # Accelerator: avoid closure on kpt (not isbits)
map(G -> G + coordinate, G_vectors(basis, kpt))
end

Expand Down Expand Up @@ -450,7 +449,7 @@ Returns nothing if outside the range of valid wave vectors.
end
end

@inline function index_G_vectors(basis::PlaneWaveBasis, G::AbstractVector{<:Integer})
function index_G_vectors(basis::PlaneWaveBasis, G::AbstractVector{<:Integer})
index_G_vectors(basis.fft_size, G)
end

Expand Down
2 changes: 1 addition & 1 deletion src/scf/chi0models.jl
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ function (χ0::DielectricModel)(basis; kwargs...)
C0 = 1 - εr
iszero(C0) && return nothing # Will yield no contribution

Gsq = [norm2(G) for G in G_vectors_cart(basis)]
Gsq = norm2.(G_vectors_cart(basis))
apply_sqrtL = identity
if χ0.localization != identity
sqrtL = sqrt.(χ0.localization.(r_vectors(basis)))
Expand Down
2 changes: 1 addition & 1 deletion src/scf/mixing.jl
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ end
@timing "KerkerMixing" function mix_density(mixing::KerkerMixing, basis::PlaneWaveBasis,
δF; kwargs...)
T = eltype(δF)
G² = map(norm2, G_vectors_cart(basis))
G² = norm2.(G_vectors_cart(basis))
kTF = T.(mixing.kTF)
ΔDOS_Ω = T.(mixing.ΔDOS_Ω)

Expand Down
7 changes: 3 additions & 4 deletions src/terms/hartree.jl
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,9 @@ function TermHartree(basis::PlaneWaveBasis{T}, scaling_factor) where {T}
# Solving the Poisson equation ΔV = -4π ρ in Fourier space
# is multiplying elementwise by 4π / |G|^2.

GPUArraysCore.allowscalar() do
poisson_green_coeffs = 4T(π) ./ [norm2(G) for G in G_vectors_cart(basis)]
poisson_green_coeffs[1] = 0 # Compensating charge background => Zero DC
end
poisson_green_coeffs = 4T(π) ./ norm2.(G_vectors_cart(basis))
GPUArraysCore.@allowscalar poisson_green_coeffs[1] = 0 # Compensating charge background => Zero DC

enforce_real!(basis, poisson_green_coeffs) # Symmetrize Fourier coeffs to have real iFFT
poisson_green_coeffs = to_device(basis.architecture, poisson_green_coeffs)

Expand Down
4 changes: 2 additions & 2 deletions src/terms/xc.jl
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ end
end
if haskey(terms, :Vl) && any(x -> abs(x) > term.potential_threshold, terms.Vl)
@warn "Meta-GGAs with a Δρ term have not yet been thoroughly tested." maxlog=1
mG² = [-norm2(G) for G in G_vectors_cart(basis)]
mG² = -norm2.(G_vectors_cart(basis))
Vl = reshape(terms.Vl, n_spin, basis.fft_size...)
Vl_fourier = fft(basis, Vl[s, :, :, :])
# TODO: forcing real-valued ifft; should be enforced at creation of array
Expand Down Expand Up @@ -255,7 +255,7 @@ function LibxcDensities(basis, max_derivative::Integer, ρ, τ)
# Compute Δρ
if max_derivative > 1
Δρ_real = similar(ρ_real, n_spin, basis.fft_size...)
mG² = [-norm2(G) for G in G_vectors_cart(basis)]
mG² = -norm2.(G_vectors_cart(basis))
for σ = 1:n_spin
# TODO: forcing real-valued ifft; should be enforced at creation of array
Δρ_real[σ, :, :, :] .= irfft(basis, mG² .* @view ρ_fourier[σ, :, :, :];
Expand Down