add support for writing openPMD/ADIOS2 plotfiles

[BenWibking]

Description

This adds support for writing ADIOS2 format plotfiles using openPMD-api (https://github.com/openPMD/openPMD-api). This is the file format used by WarpX for its large-scale runs, and it is the fastest option for large-scale I/O (~Terabytes or larger).

This adds support for cell-centered variables only. (However, it does include cell-centered averages of the face-centered variables, and all of the other derived variables.)

openPMD currently does not support ghost cells, so we exclude them when writing the plotfile in openPMD/ADIOS2 format.

There is no support for checkpoints or restarts with this format (nor is there in WarpX, or any other code that I'm aware of).

Related issues

N/A

Checklist

Before this pull request can be reviewed, all of these tasks should be completed. Denote completed tasks with an x inside the square brackets [ ] in the Markdown source below:

• I have added a description (see above).
• I have added a link to any related issues see (see above).
• I have read the Contributing Guide.
• I have added tests for any new physics that this PR adds to the code.
• I have tested this PR on my local computer and all tests pass.
• I have manually triggered the GPU tests with the magic comment /azp run.
• I have requested a reviewer for this PR.

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[BenWibking]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[BenWibking]

It looks like the clang-tidy complaint about not being able to find the openPMD headers is the last issue to fix.

[BenWibking]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[markkrumholz]

1. It seems like this is only implemented for cell-centered quantities. Did you intend to handle face-centered quantities in a different PR?
2. I see an implementation of writing, but no implementation of reading. So how does this work with restarts? Or is this used only for plot files, with no implementation for checkpoints?
3. I have one minor comment on one of the implementations, which is appended to that file.

[BenWibking]

1. It seems like this is only implemented for cell-centered quantities. Did you intend to handle face-centered quantities in a different PR?
2. I see an implementation of writing, but no implementation of reading. So how does this work with restarts? Or is this used only for plot files, with no implementation for checkpoints?
3. I have one minor comment on one of the implementations, which is appended to that file.

Yes, this is only for cell-centered quantities. I will handle face-centered quantities in another PR as needed.

There is no implementation of reading. This is intended only for plotfiles, which have to be written out with the highest frequency. The main goal of this PR is to enable using openPMD-based analysis tools, and get faster plotfile outputs "for free." Also, getting restarts to work with another format would be very error-prone and likely to result in a non-working restart for the first several attempts, so I'd prefer not do that.

(I've replied to your comment in-line.)

[BenWibking]

I've updated the PR description to note the limitations you pointed out above.

[markkrumholz]

Yes, this is only for cell-centered quantities. I will handle face-centered quantities in another PR as needed.

There is no implementation of reading. This is intended only for plotfiles, which have to be written out with the highest frequency. The main goal of this PR is to enable using openPMD-based analysis tools, and get faster plotfile outputs "for free." Also, getting restarts to work with another format would be very error-prone and likely to result in a non-working restart for the first several attempts, so I'd prefer not do that.

On reading that is fine.

On cc vs fc, that's also fine, but then there should probably be some sort of automation, or at least a warning, that this method is incompatible with fc quantities. For example, you could check if the number of fc quantities is > 0, and issue a warning or throw an error message at compile time if openPMD is on.

[BenWibking]

On cc vs fc, that's also fine, but then there should probably be some sort of automation, or at least a warning, that this method is incompatible with fc quantities. For example, you could check if the number of fc quantities is > 0, and issue a warning or throw an error message at compile time if openPMD is on.

Is outputting the cell-centered averages enough? Otherwise, I could add a warning in this case.

[markkrumholz]

On cc vs fc, that's also fine, but then there should probably be some sort of automation, or at least a warning, that this method is incompatible with fc quantities. For example, you could check if the number of fc quantities is > 0, and issue a warning or throw an error message at compile time if openPMD is on.

Is outputting the cell-centered averages enough? Otherwise, I could add a warning in this case.

I think it would be good to add a warning, since this means that this output option will behave differently than all the other ones, which do support fc quantities. Differences in behaviour should be flagged.

[BenWibking]

Ok, now it should be right.