A Collection of Radiology Report Generation Studies

This GitHub repository summarizes papers and resources related to the radiology report generation (RRG) task.

You can refer to my survey paper titled "A Systematic Review of Deep Learning-based Research on Radiology Report Generation" for more details of this topic.

If you find my survey paper helpful to your research, please cite it with the following BibTeX:

@misc{liu2023systematic,
      title={A Systematic Review of Deep Learning-based Research on Radiology Report Generation}, 
      author={Chang Liu and Yuanhe Tian and Yan Song},
      year={2023},
      eprint={2311.14199},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

If you have any suggestions about this repository, please feel free to start a new issue or pull requests.

Contents

• References
• Survey Papers
• Papers
  • Year 2024
  • Year 2023
  • Year 2022
  • Year 2021
  • Year 2020
  • Year 2019
  • Year 2018
• Datasets
• To-Do Lists

To-Do Lists

• Recent Papers
  • Update ACL 2024 Papers
  • Update CVPR 2024 Papers
  • Update AAAI 2024 Papers
    • Update PDFs and References
  • Regular Maintenance of Preprint arXiv Papers and Missed Papers
• Previous Papers
  • Published Papers on Conferences
    • Update ACL papers
    • Update EMNLP papers
    • Update CVPR papers
    • Update ICCV papers
    • Update ECCV papers
    • Update MICCAI papers
    • Update AAAI papers
    • Update NeurIPS papers
  • Published Papers on Journals
    • Update TMM papers
    • Update TMI papers

<🎯Back to Top>

Survey Papers

• [NAACL 2019] A Survey on Biomedical Image Captioning [Paper]
• [Artif. Intell. Medicine 2020] Deep Learning in Generating Radiology Reports: A Survey [Paper]
• [Knowledge and Information Systems 2022] Diagnostic Captioning: A Survey [Paper]
• [arXiv 2023] A Systematic Review on Deep Learning-based Research on Radiology Report Generation [Paper]

<🎯Back to Top>

Papers

• Year 2024
  • ACL
    • Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation [Paper] [Code]
  • CVPR
    • Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation [Paper] [Code] [Dataset]
  • ECCV
    • HERGen: Elevating Radiology Report Generation with Longitudinal Data [Paper] [Code]
  • AAAI
    • Bootstrapping Large Language Models for Radiology Report Generation [Paper] [Code]
    • PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation [Paper] [Code]
  • TMM
    • From Observation to Concept: A Flexible Multi-view Paradigm for Medical Report Generation [paper]
  • TMI
    • Complex Organ Mask Guided Radiology Report Generation [Paper] [Code]
• Year 2023
  • ACL
    • ORGAN: Observation-Guided Radiology Report Generation via Tree Reasoning [Paper] [Code]
    • Replace and Report: NLP Assisted Radiology Report Generation [Paper]
  • EACL
    • KGVL-BART: Knowledge Graph Augmented Visual Language BART for Radiology Report Generation [Paper] [Code]
  • EMNLP
    • Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting [Paper]
    • PhenotypeCLIP: Phenotype-based Contrastive Learning for Medical Imaging Report Generation [Paper]
    • RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning [Paper] [Code]
  • CVPR
    • Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation [Paper] [Code]
    • RGRG: Interactive and Explainable Region-guided Radiology Report Generation [Paper] [Code]
    • KiUT: Knowledge-injected U-Transformer for Radiology Report Generation [Paper]
    • METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens [Paper]
  • ICCV
    • Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation [Paper]
  • MICCAI
    • SGT: Scene Graph-Guided Transformer for Surgical Report Generation** [Paper] [Code]
    • Utilizing Longitudinal Chest X-Rays and Reports to Pre-Fill Radiology Reports [Paper] [Code]
  • ICASSP
    • MvCo-DoT: Multi-View Contrastive Domain Transfer Network for Medical Report Generation [Paper]
  • ICIP
    • Self Adaptive Global-Local Feature Enhancement for Radiology Report Generation [Paper]
  • WWW
    • Auxiliary Signal‑guided Knowledge Encoder‑decoder for Medical Report Generation [paper] [code]
  • AMI
    • Improving Chest X-ray Report Generation by Leveraging Warm Starting [Paper] [Code]
  • TMI
    • Attributed Abnormality Graph Embedding for Clinically Accurate X-Ray Report Generation [Paper]
    • SGT++: Improved Scene Graph-guided Transformer for Surgical Report Generation [Paper]
    • Joint Embedding of Deep Visual and Semantic Features for Medical Image Report Generation [paper]
    • Semi-supervised Medical Report Generation via Graph-guided Hybrid Feature Consistency [paper]
  • arXiv
    • LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation [Paper] [Code]
    • MAIRA-1: A Specialised Large Multimodal Model for Radiology Report Generation [Paper] [Project]
    • MedXChat: Bridging CXR Modalities with a Unified Multimodal Large Model [Paper]
    • Pragmatic Radiology Report Generation [paper]
    • RadLLM: A Comprehensive Healthcare Benchmark of Large Language Models for Radiology [Paper] [Project]
    • RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance [Paper] [Code]
    • Towards Generalist Biomedical AI [Paper] [Reproduced Code]
    • Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data [Paper] [Project] [Code]
    • Cross-Modal Causal Intervention for Medical Report Generation [Paper] [Code]
    • Medical Report Generation based on Segment-Enhanced Contrastive Representation Learning [Paper]
    • Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [Paper]
• Year 2022
  • AACL
    • Multimodal Generation of Radiology Reports using Knowledge-Grounded Extraction of Entities and Relations [Paper]
  • ACL
    • Reinforced Cross-modal Alignment for Radiology Report Generation [Paper] [Code]
  • EMNLP
    • Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards [Paper] [Code]
    • Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation [Paper]
  • CVPR
    • Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [Paper]
  • ECCV
    • Cross-Modal Prototype Driven Network for Radiology Report Generation [Paper]
  • MICCAI
    • RepsNet: Combining Vision with Language for Automated Medical Reports [Paper]
    • A Self-guided Framework for Radiology Report Generation [Paper] [Code]
    • A Medical Semantic-Assisted Transformer for Radiographic Report Generation [Paper]
    • Lesion Guided Explainable Few Weak-Shot Medical Report Generation [Paper]
    • TranSQ: Transformer-Based Semantic Query for Medical Report Generation [Paper] [Code]
  • BMVC
    • On the Importance of Image Encoding in Automated Chest X-Ray Report Generation [Paper] [Code]
  • ICBB
    • Clinically Coherent Radiology Report Generation with Imbalanced Chest X-rays [Paper]
  • MIA
    • Knowledge Matters: Chest Radiology Report Generation with General and Specific Knowledge [Paper] [Code]
  • TMI
    • Automated Radiographic Report Generation Purely on Transformer: A Multicriteria Supervised Approach [Paper]
  • arXiv
    • CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation [Paper]
    • Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty [Paper]
    • DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis [Paper]
• Year 2021
  • ACL
    • Cross-modal Memory Networks for Radiology Report Generation [Paper] [Code]
  • NAACL
    • Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation [Paper] [Code]
  • COLING
    • JPG - Jointly Learn to Align: Automated Disease Prediction and Radiology Report Generation [Paper]
  • EMNLP
    • Automated Generation of Accurate & Fluent Medical X-ray Reports [Paper] [Code]
    • Progressive Transformer-Based Generation of Radiology Reports [Paper] [Code]
    • Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation [Paper] [Code]
  • CVPR
    • Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation [Paper]
    • A Self-boosting Framework for Automated Radiographic Report Generation [Paper]
  • ICCV
    • Visual-Textual Attentive Semantic Consistency for Medical Report Generation [Paper]
  • MICCAI
    • RATCHET: Medical Transformer for Chest X-ray Diagnosis and Reporting [Paper] [Code]
    • Trust It or Not: Confidence-Guided Automatic Radiology Report Generation [Paper]
  • NeurIPS
    • Auto-encoding Knowledge Graph for Unsupervised Medical Report Generation [Paper]
• Year 2020
  • EMNLP
    • Generating Radiology Reports via Memory-driven Transformer [Paper] [Code]
    • Reinforcement Learning with Imbalanced Dataset for Data-to-Text Medical Report Generation [Paper]
  • AAAI
    • When Radiology Report Generation Meets Knowledge Graph [Paper]
  • ACCV
    • Hierarchical X-Ray Report Generation via Pathology tags and Multi Head Attention [Paper]
• Year 2019
  • ACL
    • Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports [Paper]
  • MICCAI
    • Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment [Paper]
  • AAAI
    • Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation [Paper]
  • BMVC
    • Addressing Data Bias Problems for Chest X-ray Image Report Generation [Paper]
• Year 2018
  • ACL
    • On the Automatic Generation of Medical Imaging Reports [Paper] [Code]
  • MICCAI
    • Multimodal Recurrent Model with Attention for Automated Radiology Report Generation [Paper] [Code]
  • NeurIPS
    • Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation [Paper]

<🎯Back to Top>

Datasets

[J. Am. Medical Informatics Assoc.] IU X-Ray: Preparing A Collection of Radiology Examinations for Distribution and Retrieval [Paper] [Dataset]

[Scientific Data] MIMIC-CXR, A De-identified Publicly Available Database of Chest Radiographs with Free-text Reports [Paper] [Dataset]

[arXiv] MIMIC-CXR-JPG, A Large Publicly Available Database of Labeled Chest Radiographs [Paper] [Dataset]

[NeurIPS 2021 Datasets & Benchmark] FFA-IR: Towards an Explainable and Reliable Medical Report Generation Benchmark [Paper] [Dataset] [GitHub]

[NeurIPS 2021 Datasets & Benchmark] RadGraph: Extracting Clinical Entities and Relations from Radiology Reports [Paper] [Dataset]

<🎯Back to Top>

Q&A

• Q: The conference sequence of this paper list?
  • This paper list is organized according to the following sequence:
    • Conferences
      • ACL
      • EACL
      • NAACL
      • COLING
      • EMNLP
      • CVPR
      • ICCV
      • ECCV
      • MICCAI
      • AAAI
      • NeurIPS
      • ACCV
      • BMCV
      • ICASSP
      • ICIP
      • ICBB
      • WWW
    • Journals
      • AMI
      • MIA
      • TMM
      • TMI
    • arXiv

<🎯Back to Top>

References

The reference.bib file summarizes bibtex references of up-to-date image inpainting papers, widely used datasets, and toolkits. Based on the original references, I have made the following modifications to make their results look nice in the LaTeX manuscripts:

• Refereces are normally constructed in the form of author-etal-year-nickname. Particularly, references of datasets and toolkits are directly constructed as nickname, e.g., imagenet.
• In each reference, all names of conferences/journals are converted into abbreviations, e.g., Computer Vision and Pattern Recognition -> CVPR.
• The url, doi, publisher, organization, editor, series in all references are removed.
• The pages of all references are added if they are missing.
• All paper names are in title case. Besides, I have added an additional {} to make sure that the title case would also work well in some particular templates.

If you have other demands of reference formats, you may refer to the original references of papers by searching their names in DBLP or Google Scholar.

<🎯Back to Top>