About process_datasets.py

We process pretty much all 3DED/microED datasets with the same sequence of DIALS commands:

dials.import /path/to/*.mrc goniometer.axes=1,0,0 distance=$distance panel.pedestal=-64
dials.generate_mask imported.expt untrusted.circle='1031 1023 50' untrusted.polygon='0 1010 600 1010 976 1004 988 998 986 1044 978 1041 650 1051 0 1051'
dials.apply_mask imported.expt mask=pixels.mask
dials.find_spots masked.expt [spot finding options] nproc=$nproc
dials.search_beam_position masked.expt strong.refl
dials.index optimised.expt strong.refl [indexing options] detector.fix=distance
dials.refine indexed.expt indexed.refl detector.fix=distance scan_varying=false output.experiments=refined_static.expt output.reflections=refined_static.refl
dials.refine refined_static.expt refined_static.refl detector.fix=distance scan_varying=true
dials.integrate refined.expt refined.refl nproc=$nproc

In order to make this more reproducible I wrote a Python script process_datasets.py that reads a JSON file with options and image paths and then runs the DIALS commands and reorganises the output by grid and crystal (this is useful when you make a typo in the dataset name for 1 crystal in the middle of a series). The easiest way to run is with Python from a DIALS installation as it imports from libtbx and dxtbx.

The script isn't very user friendly. The idea was to separate dataset specific parameters from the actual commands rather than make it easy to use as there are only 2 users and one of them is the author. If others would find it useful then I can probably make it more user friendly!

Here's a skeleton datasets.json file to edit:

{"parameters":{
                "sample":"XX_nnn",
                "njobs":2,
                "nproc":4,
                "spacegroup":null,
                "import":"goniometer.axes=1,0,0 distance=958.5 panel.pedestal=-64",
                "generate_mask":"untrusted.circle='1031 1023 50' untrusted.polygon='0 1010 600 1010 976 1004 988 998 986 1044 978 1041 650  1051 0 1051'",
                "search_beam":true,
                "find_rotation_axis":"global_search=False",
                "find_spots":"d_min=0.8",
                "index":"detector.fix=distance",
                "refine":"detector.fix=distance",
                "integrate":""
               },
  "datasets":[
            {"template":"XX_nnn_grid1_xtal1/XX_nnn_grid1_xtal1_####.mrc", "grid":1, "xtal":1},
            {"template":"XX_nnn_grid5_xtal99/XX_nnn_grid5_xtal99_####.mrc", "grid":5, "xtal":99}
            ]
}

The goniometer orientation, detector distance and beamstop mask will obviously be different for other setups! We normally only adjust d_min for spotfinding (depending on how well the crystals diffract) and then set the space group once we have indexed a load of datasets in P1.

As of version 0.3.4 you can process data where dials.import requires the path to a single file - e.g. for detectors that produce HDF5 format data - like this:

"datasets":[
            {"file":"XX_nnn_grid1_xtal1/XX_nnn_grid1_xtal1_master.h5", "grid":1, "xtal":1},
            {"file":"XX_nnn_grid5_xtal99/XX_nnn_grid5_xtal99_master.h5", "grid":5, "xtal":99}
            ]