MICCAI 22 accepted paper TranSQ: Transformer-based Semantic Query for Medical Report Generation presents a method for generating medical reports using semantic queries.
Run the following scripts in ./preprocess: data_preprocess.py, generate_CE.py, and generate_sentence_gallery.py to obtain preprocessed data, semantic query category initialization, and build a sentence retrieval library.
Modify the transq/config.py configuration file and run the run.py script (refer to ./model/README.md for obtaining pre-trained models). For example:
python run.py with train_mimic_vit
- Test the trained model on the training set to gather data for averaging the positional order of semantic vectors.
- Run the
evaluation/NLG_eval/test_from_json.pyscript to calculate the average positional order of the mentioned sentences in the training set results. Perform a re-ranking of the test set results and compute the final metrics.
Note: Since there are slight differences between IU X-ray and MIMIC-CXR (different image numbers), we implemented the two datasets with two separate projects for convenience, and the main branch is for MIMIC-CXR.
The code implementation is modified from the project: https://github.com/dandelin/ViLT
If you have any problem, please feel free to contact me at [email protected]
If you use any part of this code and pre-trained weights for your own purpose, please cite our paper.
@article{GAO2024102982,
title = {Simulating doctors’ thinking logic for chest X-ray report generation via Transformer-based Semantic Query learning},
journal = {Medical Image Analysis},
volume = {91},
pages = {102982},
year = {2024},
issn = {1361-8415},
doi = {https://doi.org/10.1016/j.media.2023.102982},
url = {https://www.sciencedirect.com/science/article/pii/S1361841523002426},
author = {Danyang Gao and Ming Kong and Yongrui Zhao and Jing Huang and Zhengxing Huang and Kun Kuang and Fei Wu and Qiang Zhu},
keywords = {Medical report generation, Semantic query, Transformer, Computer-aided diagnosis, Deep learning}
}
or
@InProceedings{10.1007/978-3-031-16452-1_58,
author="Kong, Ming
and Huang, Zhengxing
and Kuang, Kun
and Zhu, Qiang
and Wu, Fei",
editor="Wang, Linwei
and Dou, Qi
and Fletcher, P. Thomas
and Speidel, Stefanie
and Li, Shuo",
title="TranSQ: Transformer-Based Semantic Query for Medical Report Generation",
booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2022",
year="2022",
publisher="Springer Nature Switzerland",
address="Cham",
pages="610--620"
}