Recommend configuration for large scale particles

[Change72]

Hi

I am trying to generate a big dataset with many particles, e.g. billions of particles. For now, I increased particle density but the number of particles didn't have a big difference.

1. Do you have any better idea of how to generate a large-scale particle dataset?
2. Can I configure the number of block number? For example, with the same number particles, I want one dataset with less but big blocks, and another is more but small blocks.

The following is my input file:

#################################
####### GENERAL PARAMETERS ######
#################################
max_step = 2000           # for production, run for longer time, e.g. max_step = 1000
amr.n_cell = 512 512 512   # for production, run with finer mesh, e.g. amr.n_cell = 64 64 512
amr.max_grid_size = 128   # maximum size of each AMReX box, used to decompose the domain
amr.blocking_factor = 32 # minimum size of each AMReX box, used to decompose the domain
geometry.dims = 3
geometry.prob_lo     = -30.e-6   -30.e-6   -56.e-6    # physical domain
geometry.prob_hi     =  30.e-6    30.e-6    12.e-6
amr.max_level = 0 # Maximum level in hierarchy (1 might be unstable, >1 is not supported)
# warpx.fine_tag_lo = -5.e-6   -5.e-6   -50.e-6
# warpx.fine_tag_hi =  5.e-6    5.e-6   -30.e-6

#################################
####### Boundary condition ######
#################################
boundary.field_lo = periodic periodic pec
boundary.field_hi = periodic periodic pec

#################################
############ NUMERICS ###########
#################################
warpx.verbose = 1
warpx.do_dive_cleaning = 0
warpx.use_filter = 1
warpx.cfl = 1. # if 1., the time step is set to its CFL limit
warpx.do_moving_window = 1
warpx.moving_window_dir = z
warpx.moving_window_v = 1.0 # units of speed of light
warpx.do_dynamic_scheduling = 0 # for production, set this to 1 (default)
warpx.serialize_initial_conditions = 1         # for production, set this to 0 (default)

# Order of particle shape factors
algo.particle_shape = 3

#################################
############ PLASMA #############
#################################
particles.species_names = electrons

electrons.charge = -q_e
electrons.mass = m_e
electrons.injection_style = "NUniformPerCell"
electrons.num_particles_per_cell_each_dim = 1 1 1
electrons.xmin = -20.e-6
electrons.xmax =  20.e-6
electrons.ymin = -20.e-6
electrons.ymax =  20.e-6
electrons.zmin =  0
electrons.profile = constant
electrons.density = 2.e30  # number of electrons per m^3
electrons.momentum_distribution_type = "at_rest"
electrons.do_continuous_injection = 1
electrons.addIntegerAttributes = regionofinterest
electrons.attribute.regionofinterest(x,y,z,ux,uy,uz,t) = "(z>10.0e-6) * (z<20.0e-6)"
electrons.addRealAttributes = initialenergy
electrons.attribute.initialenergy(x,y,z,ux,uy,uz,t) = " ux*ux + uy*uy + uz*uz"

#################################
############ LASER  #############
#################################
lasers.names        = laser1
laser1.profile      = Gaussian
laser1.position     = 0. 0. 9.e-6        # This point is on the laser plane
laser1.direction    = 0. 0. 1.           # The plane normal direction
laser1.polarization = 0. 1. 0.           # The main polarization vector
laser1.e_max        = 16.e12             # Maximum amplitude of the laser field (in V/m)
laser1.profile_waist = 5.e-6             # The waist of the laser (in m)
laser1.profile_duration = 15.e-15        # The duration of the laser (in s)
laser1.profile_t_peak = 30.e-15          # Time at which the laser reaches its peak (in s)
laser1.profile_focal_distance = 100.e-6  # Focal distance from the antenna (in m)
laser1.wavelength = 0.8e-6               # The wavelength of the laser (in m)

# Diagnostics
diagnostics.diags_names = diag2
diag2.intervals = 500
diag2.diag_type = Full
diag2.format = openpmd
diag2.openpmd_backend = bp
diag2.openpmd_encoding = g
diag2.adios2_engine.parameters.StatsLevel = 1
diag2.adios2_engine.parameters.NumAggregators = 1024
#diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho

# Reduced Diagnostics
warpx.reduced_diags_names               = FP

FP.type = FieldProbe
FP.intervals = 10
FP.integrate = 0
FP.probe_geometry = Line
FP.x_probe = 0
FP.y_probe = 0
FP.z_probe = -56e-6
FP.x1_probe = 0
FP.y1_probe = 0
FP.z1_probe = 12e-6
FP.resolution = 300
FP.do_moving_window_FP = 1

The following is from bpls:

>bpls -l openpmd.bp
  double    /data/0/fields/B/x                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/B/y                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/B/z                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/E/x                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/E/y                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/E/z                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/j/x                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/j/y                               {512, 512, 512} = 0 / 0
  double    /data/0/fields/j/z                               {512, 512, 512} = 0 / 0
  uint64_t  /data/0/particles/electrons/id                   {10526760} = 0 / 0
  double    /data/0/particles/electrons/initialenergy        {10526760} = 0 / 0
  double    /data/0/particles/electrons/momentum/x           {10526760} = 0 / 0
  double    /data/0/particles/electrons/momentum/y           {10526760} = 0 / 0
  double    /data/0/particles/electrons/momentum/z           {10526760} = 0 / 0
  double    /data/0/particles/electrons/position/x           {10526760} = 0 / 0
  double    /data/0/particles/electrons/position/y           {10526760} = 0 / 0
  double    /data/0/particles/electrons/position/z           {10526760} = 0 / 0
  int32_t   /data/0/particles/electrons/regionofinterest     {10526760} = 0 / 0
  double    /data/0/particles/electrons/weighting            {10526760} = 0 / 0
  double    /data/1000/fields/B/x                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/B/y                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/B/z                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/E/x                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/E/y                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/E/z                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/j/x                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/j/y                            {512, 512, 512} = 0 / 0
  double    /data/1000/fields/j/z                            {512, 512, 512} = 0 / 0
  uint64_t  /data/1000/particles/electrons/id                {52066501} = 0 / 0
  double    /data/1000/particles/electrons/initialenergy     {52066501} = 0 / 0
  double    /data/1000/particles/electrons/momentum/x        {52066501} = 0 / 0
  double    /data/1000/particles/electrons/momentum/y        {52066501} = 0 / 0
  double    /data/1000/particles/electrons/momentum/z        {52066501} = 0 / 0
  double    /data/1000/particles/electrons/position/x        {52066501} = 0 / 0
  double    /data/1000/particles/electrons/position/y        {52066501} = 0 / 0
  double    /data/1000/particles/electrons/position/z        {52066501} = 0 / 0
  int32_t   /data/1000/particles/electrons/regionofinterest  {52066501} = 0 / 0
  double    /data/1000/particles/electrons/weighting         {52066501} = 0 / 0
  double    /data/1500/fields/B/x                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/B/y                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/B/z                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/E/x                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/E/y                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/E/z                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/j/x                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/j/y                            {512, 512, 512} = 0 / 0
  double    /data/1500/fields/j/z                            {512, 512, 512} = 0 / 0
  uint64_t  /data/1500/particles/electrons/id                {57803797} = 0 / 0
  double    /data/1500/particles/electrons/initialenergy     {57803797} = 0 / 0
  double    /data/1500/particles/electrons/momentum/x        {57803797} = 0 / 0
  double    /data/1500/particles/electrons/momentum/y        {57803797} = 0 / 0
  double    /data/1500/particles/electrons/momentum/z        {57803797} = 0 / 0
  double    /data/1500/particles/electrons/position/x        {57803797} = 0 / 0
  double    /data/1500/particles/electrons/position/y        {57803797} = 0 / 0
  double    /data/1500/particles/electrons/position/z        {57803797} = 0 / 0
  int32_t   /data/1500/particles/electrons/regionofinterest  {57803797} = 0 / 0
  double    /data/1500/particles/electrons/weighting         {57803797} = 0 / 0
  double    /data/2000/fields/B/x                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/B/y                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/B/z                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/E/x                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/E/y                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/E/z                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/j/x                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/j/y                            {512, 512, 512} = 0 / 0
  double    /data/2000/fields/j/z                            {512, 512, 512} = 0 / 0
  uint64_t  /data/2000/particles/electrons/id                {59760425} = 0 / 0
  double    /data/2000/particles/electrons/initialenergy     {59760425} = 0 / 0
  double    /data/2000/particles/electrons/momentum/x        {59760425} = 0 / 0
  double    /data/2000/particles/electrons/momentum/y        {59760425} = 0 / 0
  double    /data/2000/particles/electrons/momentum/z        {59760425} = 0 / 0
  double    /data/2000/particles/electrons/position/x        {59760425} = 0 / 0
  double    /data/2000/particles/electrons/position/y        {59760425} = 0 / 0
  double    /data/2000/particles/electrons/position/z        {59760425} = 0 / 0
  int32_t   /data/2000/particles/electrons/regionofinterest  {59760425} = 0 / 0
  double    /data/2000/particles/electrons/weighting         {59760425} = 0 / 0
  double    /data/500/fields/B/x                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/B/y                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/B/z                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/E/x                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/E/y                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/E/z                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/j/x                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/j/y                             {512, 512, 512} = 0 / 0
  double    /data/500/fields/j/z                             {512, 512, 512} = 0 / 0
  uint64_t  /data/500/particles/electrons/id                 {35722398} = 0 / 0
  double    /data/500/particles/electrons/initialenergy      {35722398} = 0 / 0
  double    /data/500/particles/electrons/momentum/x         {35722398} = 0 / 0
  double    /data/500/particles/electrons/momentum/y         {35722398} = 0 / 0
  double    /data/500/particles/electrons/momentum/z         {35722398} = 0 / 0
  double    /data/500/particles/electrons/position/x         {35722398} = 0 / 0
  double    /data/500/particles/electrons/position/y         {35722398} = 0 / 0
  double    /data/500/particles/electrons/position/z         {35722398} = 0 / 0
  int32_t   /data/500/particles/electrons/regionofinterest   {35722398} = 0 / 0
  double    /data/500/particles/electrons/weighting          {35722398} = 0 / 0

[ax3l]

Hi @Change72,

Thank you for the questions.

For now, I increased particle density but the number of particles didn't have a big difference. Do you have any better idea of how to generate a large-scale particle dataset?

The number of numerical particles in WarpX (and PIC simulations in general) is usually not set by the density, but by the number of particles per cell. The reason for that is that particles have a representative "weight" and sampling them as such "macro" particles can save a lot of computational costs.

So, look into the paramters <species_name>.num_particles_per_cell or <species_name>.num_particles_per_cell_each_dim respectively, depending on your <species_name>.injection_style:
https://warpx.readthedocs.io/en/latest/usage/parameters.html#particle-initialization

Can I configure the number of block number? For example, with the same number particles, I want one dataset with less but big blocks, and another is more but small blocks.

Sure, you can change the block size by setting amr.max_grid_size and amr.blocking_factor explicitly to the same numbers, but in practice the block size influences the computational cost and scaling behavior, so we do not set it too arbitrary:
In practice, we use the block size to have somewhere between 4-12 blocks per compute device (e.g., per GPU), and making them large enough to fill the compute device with enough memory, so we can perform load balancing on it and avoid strong-scaling our algorithms. This paper explains more details on this: https://doi.org/10.1145/3468267.3470614

See also the introduction to our domain decomposition here: https://warpx.readthedocs.io/en/latest/usage/workflows/domain_decomposition.html and included links to AMReX.