dials.export format=pets #2014

dagewa · 2022-02-17T14:31:57Z

This enables export of integrated.{expt,refl} for electron diffraction data in the special "*_dyn.cif_pets" format used by PETS 2 in preparation for dynamic diffraction refinement by Jana2020.

The software combination PETS+Jana is currently the only way (to our knowledge) to determine absolute structure from continuous rotation electron diffraction patterns. In addition, dynamical refinement provides much improved R-factors compared to kinematic refinement (see https://chemrxiv.org/engage/chemrxiv/article-details/619971832bf8a9871fe627ed for details). This feature allows DIALS to replace PETS in that combination.

The DIALS output for a quartz crystal has been successfully used for dynamic diffraction refinement in Jana, although the results are not as good as those from PETS. One significant remaining difference is in the intensity error estimates, which are severely underestimated in DIALS, compared to PETS. Partly, this is due to the lack of a proper model for the integrating detector response in DIALS (for reference, with the quartz dataset used in the test, PETS sets a combined gain and cascade factor Gγ=3.5 and pixel noise factor ψ=70). Including a more sophisticated model for the errors is the next step in improving the results of dynamic diffraction refinement in DIALS. This will be addressed in separate PRs.

Draft until dials/data#346 is merged.

Closes #1859

- check only crystal orientation is scan-varying

Turns out this is much more work than is necessary! PETS only outputs the frame orientation, not the reflection orientation. I committed this for reference, but will simplify now, making use of the dials.frame_orientations program

Previously, orientations were returned at the beginning of each frame. This commit calculates the orientation at the centre of each frame. This is closer to expectations, and allows the code to be reused in export_pets

This is still untested. A good test would be to compare predictions before and after reorientation

- Use map_centroids_to_reciprocal_space for simpler way to reconstruct r - This way, we rotate the Ewald sphere rather than the relps - Calculate extinction distances by comparison with the rotated Ewald sphere for the virtual frame centre

Co-authored-by: Richard Gildea <[email protected]>

Name collision between alpha and beta orientation angles and the cell angles.

Clearly reveals our zone axes are inverted compared to PETS. Another indication of geometry issues...

It was a deg/rad mix up! Tidied up and retained the sanity check

…t the full orientation U

- use the excitation error cutoff only to filter reflections - this should include more low resolution reflections

- Makes alpha the rotation angle. Not sure about beta and omega yet...

Otherwise it appears to remove many reflections for the Glycine example (not sure why).

Previously, was using map_centroids_to_reciprocal_space to calculate relps for the excitation error calculation. However, this uses only the scan-static crystal model. We need the calculation using the scan-varying model. Also tidy up, and rename some variables

Also, do not create the last virtual frame if it extends beyond the end of the scan. Then number of VFs matches PETS, and each VF is filled.

(as discussed with Lukas Palatinus)

We are more interested in write_dyn_cif_pets. Non-dynamic diffraction data could be prepared for Jana by exporting in SHELX format.

ndevenish · 2022-03-09T19:32:23Z

🤔 I think these tests should be working now?

dagewa · 2022-03-09T21:08:59Z

It requires dials.data get quartz_processed. Not sure if that works yet in the CI?

ndevenish · 2022-03-10T09:25:35Z

CI has dials-data 2.4.10, but can reproduce on my workstation:

$  dials.data get quartz_processed
Repository location: /dls/science/groups/scisoft/DIALS/dials_data

$ pytest -k test_pets --regression
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-mep23677/pytest-1/test_pets_profile_0/profile.cif_pets'
....
E       FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-mep23677/pytest-1/test_pets_sum_0/sum.cif_pets'

dagewa and others added 30 commits December 13, 2021 16:36

Skeleton for adding PETS2 output option to dials.export

d0116b7

Start work on PETS2Output

ae9f3eb

2.0 is just a version number. Refer to "PETS"

bef1bb9

Moved module to reflect name change

16b7f93

- bug fix for experiment ID selection

5003408

- check only crystal orientation is scan-varying

Remove partial reflections and check required keys

7072b99

Calculate per-reflection orientation.

dd9f0e5

Turns out this is much more work than is necessary! PETS only outputs the frame orientation, not the reflection orientation. I committed this for reference, but will simplify now, making use of the dials.frame_orientations program

More precision in dials.frame_orientations.

abfe01c

Previously, orientations were returned at the beginning of each frame. This commit calculates the orientation at the centre of each frame. This is closer to expectations, and allows the code to be reused in export_pets

Use frame_orientations calculation

40e3524

Reorient experiment and s1 vectors to the PETS frame

5ea6a07

This is still untested. A good test would be to compare predictions before and after reorientation

Calculate extinction distances

91f0adb

- Use map_centroids_to_reciprocal_space for simpler way to reconstruct r - This way, we rotate the Ewald sphere rather than the relps - Calculate extinction distances by comparison with the rotated Ewald sphere for the virtual frame centre

Stub for writing CIF file. Needs filling out

9a55058

completed _write_dyn_cif_pets

b46c559

fixed minor bugs in write_dyn_cif_pets

509d3ac

Check unique space group on MTZ export (dials#1915)

8c087d8

Co-authored-by: Richard Gildea <[email protected]>

Skeleton for adding PETS2 output option to dials.export

c5ce47f

Resolve merge conflict on rebase

fd176cd

Update after merge

191d6dc

fixed bug missing reflections in write_dyn_cif_pets

67e156b

Simplify data structures and fix some bugs

cc7fe8e

We should always have a comment

bd62274

Bug fix

efbe897

Name collision between alpha and beta orientation angles and the cell angles.

Merge branch 'main' into export-pets

0f3da29

Fix bug after merge

e3ae4f6

There is a bug in _reorient_coordinate_frame. Investigating.

8a960b4

Rescale zone axes to match PETS format

54c1200

Clearly reveals our zone axes are inverted compared to PETS. Another indication of geometry issues...

Fixed the geometry problem

031f1e8

It was a deg/rad mix up! Tidied up and retained the sanity check

Include the zone axis inversion when rescaling

5f7a36d

Make format of the reflection loop the same as PETS output

406c28a

Calculation of α, β and ω should be done on the missetting matrix, no…

e5c4b58

…t the full orientation U

dagewa and others added 12 commits January 28, 2022 14:30

Bug fix for excitation error cutoff

6e7b0fa

Don't filter reflections by centroid in the virtual frame

d8933b6

- use the excitation error cutoff only to filter reflections - this should include more low resolution reflections

Change order of axis decomposition

937a469

- Makes alpha the rotation angle. Not sure about beta and omega yet...

Change decomposition of R according to Lukas's email

c213427

Make filtering by integrated flag optional.

92bc7aa

Otherwise it appears to remove many reflections for the Glycine example (not sure why).

Allow optional export of prf instead of sum intensities

8f83fc8

Correct orientation if merging an even number of frames.

c624503

Also, do not create the last virtual frame if it extends beyond the end of the scan. Then number of VFs matches PETS, and each VF is filled.

Set precession_angle to the tilt semi-angle of the virtual frame

3477b49

(as discussed with Lukas Palatinus)

Remove write_cif_pets method.

ba04be8

We are more interested in write_dyn_cif_pets. Non-dynamic diffraction data could be prepared for Jana by exporting in SHELX format.

Merge branch 'dials:main' into export-pets

1e61107

Add basic test for dials.export format=pets

05c633c

dagewa mentioned this pull request Feb 17, 2022

Add processed data for quartz 3DED images dials/data-files#40

Merged

dagewa marked this pull request as ready for review March 7, 2022 17:12

Merge remote-tracking branch 'origin/main' into export-pets

54bc5a2

dagewa and others added 4 commits March 10, 2022 09:46

Bugfix - no suffix in prefix

4fc33a5

Add @ronandrevon to AUTHORS

91fbd05

Remove accidentally re-added newsfragment

a290192

Add news

4a67a5b

ndevenish merged commit 17eecdd into dials:main Mar 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dials.export format=pets #2014

dials.export format=pets #2014

dagewa commented Feb 17, 2022

ndevenish commented Mar 9, 2022

dagewa commented Mar 9, 2022

ndevenish commented Mar 10, 2022

dials.export format=pets #2014

dials.export format=pets #2014

Conversation

dagewa commented Feb 17, 2022

ndevenish commented Mar 9, 2022

dagewa commented Mar 9, 2022

ndevenish commented Mar 10, 2022