Dicom fix XA10A parsing

[jdtournier]

OK, first stab at sorting out DICOM parsing for borked Siemens XA10A data. For the full low-down on how much havoc Siemens have generated with this one, take a look through @neurolabusc's run-down of all the issues...

I'm currently running this PR through the full DICOM testing suite, see whether this introduces any regressions.

In the meantime, anyone with any Siemens enhanced data (not just XA10A) is encouraged to put this through its paces. If any issues, please let me know, and even better send me the data so I can reproduce and eventually add them to the testing suite.

[bjeurissen]

It works!

With this warning:

mrconvert: [WARNING] mismatched series number and UID - this may cause problems with series grouping

but that is probably the whole point :)

[neurolabusc]

In my experience, XA30 is much cleaner than XA10. Hopefully, all XA10 systems have long since been upgraded. Therefore, the worst issues will be with archival datasets. This is timely, as Siemens is actively upgrading people from VE11 to XA30. My own center has deferred this upgrade. It would be great if users in the community to actively test tools and where possible provide sample datasets. As I recall, @josephmje @brainnetworklab and @lucijj were all instrumental in improving dcm2niix support, so perhaps they can also help with mrconvert.

[bjeurissen]

It works!

With this warning:

mrconvert: [WARNING] mismatched series number and UID - this may cause problems with series grouping

but that is probably the whole point :)

This was using XA31

[jdtournier]

Just a quick update: all my current DICOM tests pass with these changes. Full log below...

in "GE Discovery MR750w - DWI/"...
  [ 0 ] DTI AP pepolar 0: OK
  [ 1 ] PU:DTI AP pepolar 0: OK
in "GE Signa Excite/"...
  [ 0 ] Cor 3DSPGR 140: OK
  [ 1 ] depi 1NEX 45/6 96x96: OK
in "GE Signa HDxt/"...
  [ 0 ] cor T2 CUBE: OK
  [ 1 ] AXIAL Fibre: OK
  [ 2 ] ANGLE Fibre tracking: OK
in "MEDISO/"...
  [ x ] DW EPI CUSTOM DIR - AcqTime: 0:05:04, Slices: 17, NEX: 1, TR: 8236, TE: 71.3.: OK
in "Philips Achieva - cardiac/"...
  [ 1 ] SURVEY_L_SK8: OK
  [ 2 ] DTI_high_iso_E: OK
  [ 3 ] facDTI_high_iso_E SENSE: OK
  [ 4 ] facDTI_high_iso_E SENSE: OK
  [ 5 ] DTI_high_iso_E_b350: OK
  [ 6 ] facDTI_high_iso_E_b350 SENSE: OK
  [ 7 ] DWI_MSh: OK
  [ 8 ] facDWI_MSh SENSE: OK
in "Philips Achieva dStream - DWI/"...
  [ 0 ] FLAIR_Short: OK
  [ 1 ] fMRI_resting_std SENSE: OK
  [ 2 ] 3D_GRASE_PRASLOSKI: OK
  [ 3 ] DKI_25d_40d_700_1000: OK
  [ 4 ] DKI_75d_2800: OK
  [ 5 ] DKI_6d_revphase: OK
  [ 6 ] 3DTFE_mprage: OK
in "Philips Ingenia CX - head DTI/"...
  [ x ] *dti opt32 1000 avghighb SPIR 1.25iso 4bfactavg P: OK
in "Philips Ingenia CX - Muscle DTI/"...
  [ x ] DTI OPT12_NSA3_PB: OK
in "Philips Ingenia Elition X - inconsistent scaling/"...
  [ x ] DTI 1_MB2_sense1: OK
in "Philips Intera - DWI/"...
  [ x ] DT_SSh_high_2mm_NEW: OK
in "Philips - multiframe DWI/"...
  [ 1 ] DTI 16b: OK
  [ 2 ] DTI 16a: OK
in "Philips - multiframe multi-echo magnitude and phase/"...
  [ 0 ] QSM: OK
  [ 1 ] QSM: OK
in "Siemens Avanto - various/"...
  [ 0 ] fl3D_1x1x1_tra: OK
  [ 1 ] DWI_60: OK
in "Siemens Prisma - zero-filled mosaic/"...
  [ x ] DTI 64 Richtungen 2mm: OK
in "Siemens Skyra - phantom-mosaic/"...
  [ x ] ep2d_diff: OK
in "Siemens Skyra - various/"...
  [ 1 1 ] ASL_3D_tra_iso: OK
  [ 1 2 ] Perfusion_Weighted: OK
  [ 1 3 ] ASL_2D_tra: OK
  [ 1 4 ] MoCoSeries: OK
  [ 1 5 ] Perfusion_Weighted: OK
  [ 1 6 ] relCBF: OK
  [ 2 10 ] BRI 25-30 MDDW directions ep2d_diff_3scan_trace_p2_ColFA: OK
  [ 2 1 ] BRI 64 directions ep2d_diff_3scan_trace_p2: OK
  [ 2 2 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ADC: OK
  [ 2 3 ] BRI 64 directions ep2d_diff_3scan_trace_p2_TRACEW: OK
  [ 2 4 ] BRI 64 directions ep2d_diff_3scan_trace_p2_FA: OK
  [ 2 5 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ColFA: OK
  [ 2 6 ] BRI 25-30 MDDW directions ep2d_diff_3scan_trace_p2: OK
  [ 2 7 ] BRI 25-30 MDDW directions ep2d_diff_3scan_trace_p2_ADC: OK
  [ 2 8 ] BRI 25-30 MDDW directions ep2d_diff_3scan_trace_p2_TRACEW: OK
  [ 2 9 ] BRI 25-30 MDDW directions ep2d_diff_3scan_trace_p2_FA: OK
  [ 3 0 ] localizer: OK
  [ 3 1 ] BRI 64 directions ep2d_diff_3scan_trace_p2: OK
  [ 3 2 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ADC: OK
  [ 3 3 ] BRI 64 directions ep2d_diff_3scan_trace_p2_TRACEW: OK
  [ 3 4 ] BRI 64 directions ep2d_diff_3scan_trace_p2_FA: OK
  [ 3 5 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ColFA: OK
  [ 4 10 ] ep2d_bold_moco_p2 20CH: OK
  [ 4 11 ] act_MoCoSeries: OK
  [ 4 12 ] Design: OK
  [ 4 13 ] EvaSeries_GLM: OK
  [ 4 14 ] Mean_&_t-Maps: OK
  [ 4 2 ] ep2d_fid_basic_bold_p2_32 CH: OK
  [ 4 3 ] ep2d_bold_moco_p2: OK
  [ 4 4 ] act_MoCoSeries: OK
  [ 4 5 ] Design: OK
  [ 4 6 ] EvaSeries_GLM: OK
  [ 4 7 ] Mean_&_t-Maps: OK
  [ 4 9 ] ep2d_fid_basic_bold_p2 20CH: OK
  [ 5 2 10 ] intermediate t-Map: OK
  [ 5 2 11 ] Design: OK
  [ 5 2 12 ] EvaSeries_GLM: OK
  [ 5 2 13 ] Mean_&_t-Maps: OK
  [ 5 2 14 ] gre_field_mapping: OK
  [ 5 2 15 ] gre_field_mapping: OK
  [ 5 2 16 ] t2_tse_cor_fs_512_2.0mm: OK
  [ 5 2 17 ] Mag_Images: OK
  [ 5 2 18 ] Pha_Images: OK
  [ 5 2 19 ] mIP_Images(SW): OK
  [ 5 2 1 ] localizer: OK
  [ 5 2 20 ] SWI_Images: OK
  [ 5 2 21 ] t1_tir_cor_p2_2mm: OK
  [ 5 2 22 ] BRI 64 directions ep2d_diff_3scan_trace_p2: OK
  [ 5 2 23 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ADC: OK
  [ 5 2 24 ] BRI 64 directions ep2d_diff_3scan_trace_p2_TRACEW: OK
  [ 5 2 25 ] BRI 64 directions ep2d_diff_3scan_trace_p2_FA: OK
  [ 5 2 26 ] BRI 64 directions ep2d_diff_3scan_trace_p2_ColFA: OK
  [ 5 2 27 ] t1_tse_tra_3mm: OK
  [ 5 2 28 ] t1_fl2d_tra_3mm: OK
  [ 5 2 2 ] t1_mprage_sag_p2_iso_0.9 2 AVG_32ch: OK
  [ 5 2 3 ] t2_tse_cor_fs_448_2.5mm: OK
  [ 5 2 4 ] t2_tirm_cor_dark-fluid_3mm: OK
  [ 5 2 5 ] t2_tse_tra_576_3mm_high res: OK
  [ 5 2 6 ] t2_tse_cor_fs_576_2.5mm_4ave_iPAT3: OK
  [ 5 2 7 ] StartFMRI: OK
  [ 5 2 8 ] ep2d_bold_moco_p2_32ch: OK
  [ 5 2 9 ] act_MoCoSeries: OK
in "Siemens Sonata - various from CD/"...
  [ 0 ] localizer: OK
  [ 12 ] t2_tse_cor_3mm: OK
  [ 13 ] t2_tse_tra_3mm_restore: OK
  [ 14 ] loc_t2_tse_tra: OK
  [ 15 ] loc_t2_tse_sag: OK
  [ 16 ] loc_t2_tse_cor: OK
  [ 1 ] t2_flair_tra_4mm: OK
  [ 25 ] t1_se_tra_KM_4mmlueckenlos: OK
  [ 2 ] t1_se_tra: OK
  [ 3 ] Phlebo_tof_cor: OK
  [ 4 ] t2_stern_fl2d_tra: OK
  [ 5 ] diff_0_500_1000_ADC: OK
  [ 6 ] diff_0_500_1000_ADC_ADC: OK
  [ 7 ] tof_fi3d_tra_3_BL_0.6mm: OK
in "Siemens Trio - mosaic/"...
  [ 0 ] localizer_sf: OK
  [ 10 ] diff 20DW 2NEX AT TE110_ADC: OK
  [ 11 ] diff 20DW 2NEX AT TE110_TRACEW: OK
  [ 12 ] diff 20DW 2NEX AT TE110_FA: OK
  [ 13 ] diff 20DW 2NEX AT TE80: OK
  [ 14 ] diff 20DW 2NEX AT TE80_ADC: OK
  [ 15 ] diff 20DW 2NEX AT TE80_TRACEW: OK
  [ 16 ] diff 20DW 2NEX AT TE80_FA: OK
  [ 17 ] localizer_sf: OK
  [ 18 ] diff 20DW 2NEX AT TE80: OK
  [ 19 ] diff 20DW 2NEX AT TE80_ADC: OK
  [ 1 ] diff 20DW 2NEX ALL TIGHT: OK
  [ 20 ] diff 20DW 2NEX AT TE80_TRACEW: OK
  [ 21 ] diff 20DW 2NEX AT TE80_FA: OK
  [ 22 ] diff 20DW 2NEX TE90 TR6400: OK
  [ 23 ] diff 20DW 2NEX TE90 TR6400_ADC: OK
  [ 24 ] diff 20DW 2NEX TE90 TR6400_TRACEW: OK
  [ 25 ] diff 20DW 2NEX TE90 TR6400_FA: OK
  [ 26 ] diff 20DW 2NEX TE100TR6900: OK
  [ 27 ] diff 20DW 2NEX TE100TR6900_ADC: OK
  [ 28 ] diff 20DW 2NEX TE100TR6900_TRACEW: OK
  [ 29 ] diff 20DW 2NEX TE100TR6900_FA: OK
  [ 2 ] diff 20DW 2NEX ALL TIGHT_ADC: OK
  [ 30 ] diff 20DW 2NEX TE80 TR6400: OK
  [ 31 ] diff 20DW 2NEX TE80 TR6400_ADC: OK
  [ 32 ] diff 20DW 2NEX TE80 TR6400_TRACEW: OK
  [ 33 ] diff 20DW 2NEX TE80 TR6400_FA: OK
  [ 34 ] diff 20DW 2NEX TE80 TR6900: OK
  [ 35 ] diff 20DW 2NEX TE80 TR6900_ADC: OK
  [ 36 ] diff 20DW 2NEX TE80 TR6900_TRACEW: OK
  [ 37 ] diff 20DW 2NEX TE80 TR6900_FA: OK
  [ 3 ] diff 20DW 2NEX ALL TIGHT_TRACEW: OK
  [ 4 ] diff 20DW 2NEX ALL TIGHT_FA: OK
  [ 5 ] ep2d_diff_MDDW_AT_WIP: OK
  [ 6 ] ep2d_diff_MDDW_AT_WIP_ADC: OK
  [ 7 ] ep2d_diff_MDDW_AT_WIP_TRACEW: OK
  [ 8 ] ep2d_diff_MDDW_AT_WIP_FA: OK
  [ 9 ] diff 20DW 2NEX AT TE110: OK
in "Siemens Trio - various/"...
  [ 1 0 ] localiser: OK
  [ 1 12 ] diff60_b3000_2.3_iPat2+ADC: OK
  [ 1 13 ] diff60_b3000_2.3_iPat2+ADC_ADC: OK
  [ 1 14 ] diff60_b3000_2.3_iPat2+ADC_TRACEW: OK
  [ 1 15 ] diff60_b3000_2.3_iPat2+ADC_FA: OK
  [ 1 1 ] t1_mpr_1mm iso qk: OK
  [ 1 2 ] AX MPR T1: OK
  [ 1 4 ] COR MPR T1: OK
  [ 1 6 ] SAG MPR T1: OK
  [ 2 0 ] localiser: OK
  [ 2 1 ] t1_mpr_1mm iso qk: OK
  [ 2 2 ] diff60_b3000_2.3_iPat2+ADC: OK
  [ 2 3 ] diff60_b3000_2.3_iPat2+ADC_ADC: OK
  [ 2 4 ] diff60_b3000_2.3_iPat2+ADC_TRACEW: OK
  [ 2 5 ] diff60_b3000_2.3_iPat2+ADC_FA: OK
in "Siemens Verio - anonymised/"...
  [ 1 ] DTI 30 directions: 2 averages : SAG: OK
  [ 2 ] DTI 30 directions: 2 averages : SAG: OK
in "Siemens Vida - DTI DSI/"...
  [ 0 ] t1_mprage_sag_p2_iso: OK
  [ 10 ] ep2d_DSI257_s3p2_FA: OK
  [ 11 ] ep2d_DSI257_s3p2_ColFA: OK
  [ 12 ] ep2d_DSI257_s3p2_TENSOR_B0: OK
  [ 1 ] ep2d_mddw256_s3p2: OK
  [ 2 ] ep2d_mddw256_s3p2_ADC: OK
  [ 3 ] ep2d_mddw256_s3p2_TRACEW: OK
  [ 4 ] ep2d_mddw256_s3p2_FA: OK
  [ 5 ] ep2d_mddw256_s3p2_ColFA: OK
  [ 6 ] ep2d_mddw256_s3p2_TENSOR_B0: OK
  [ 7 ] ep2d_DSI257_s3p2: OK
  [ 8 ] ep2d_DSI257_s3p2_ADC: OK
  [ 9 ] ep2d_DSI257_s3p2_TRACEW: OK

[bjeurissen]

@jdtournier, would you like me to export the same data in Enhanced, Interoperability, and Mosaic mode?

[jdtournier]

That might be instructive... But based on @neurolabusc's experience, I have a feeling the conclusion will be that it's essentially unusable. May be worth collating the data anyway so I can investigate when I have nothing better to do (probably when I retire, at this rate...). But if nothing else, it would be useful to double-check whether there have been any changes to their export that might actually make it possible to support the data (the example data I've been sent were collected a couple of years ago).

[neurolabusc]

@bjeurissen would it be possible to have share your data as a public validation dataset, similar to dcm_qa? Failing that, can you share the data directly with me? I worked closely with the Siemens engineers between the release of XA10 and XA20 to provide feedback on their images. However, I have never seen XA31 data.

I do think it would be instructive to see all the data export types, to see how this information has evolved. However, beyond validation datasets, I strongly encourage XA users to only export to enhanced format.

Historically, my experience is this:

1. Siemens refers to XA mosaics as secondary capture images intended for quality assurance only. They do not include Type 1 DICOM tags like patient orientation, and are therefore not suitable for any post processing. It is unfortunate that Siemens provides this export option, as throughout the lifespan of the VA-VE series Siemens users have grown to learn that mosaics are the preferred export format for EPI data. Last I checked, the Siemens application specialists who train new users are not aware that the mosaics export is compromised.
2. At least back in 2018 Interoperability mode can increment the series number for every volume in a 4D dataset (fMRI, DTI), which foils traditional methods to parse images. According to Siemens: Having the option to export data in interoperability format will provide room to continue developmental activities for our research partners and give the chance to more third party processing software vendors to catch on.

[jdtournier]

@neurolabusc, thanks for sharing your experience - much appreciated. I'm not sure I agree with Siemens' description of their 'Interoperability' format though, feels like pretty much the opposite to me... 🤷‍♂️

I do think it would be instructive to see all the data export types, to see how this information has evolved.

👍

Otherwise, I've added a few more Vida datasets to my test suite, though they only cover XA10 & XA31 - no XA20. Still, it should increase coverage a bit...

One thing I am concerned about is that the fix for these data is to essentially ignore the SeriesNumber if a SeriesInstanceUID (0020, 000E) is found within a RelatedSeriesSequence (0008, 1250) (in which case this is the more relevant link). In that case, we treat them as a single series. But there is always the possibility that some other perfectly acceptable data also makes reference to some other series, and that may interfere with the correct interpretation of the data. So far, all the testing checks out, but there is always the potential that this could impact on other data now or in the future...

But for now, I'll merge anyway, and hope this doesn't break anything... 🤞