[WIP] Implementing `gemmi`-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

marinegor · 2024-09-20T21:57:20Z

Fixes #2367 and also extends #4303

Changes made in this Pull Request:

uses gemmi library (link) to parse mmcif files
adds a class MMCIFReader(base.SingleFrameReaderBase) and class MMCIFParser(TopologyReaderBase) classes for that

As a bonus, this implementation would potentially allow to read any of the gemmi-supported formats (source):

mmCIF (PDBx/mmCIF),
PDB (with popular extensions),
mmJSON

Also, this (with slight modifications) also would allow reading mmcif with multiple models sharing the same topology, as well as more feature-rich parsing of PDBs (the same code without changes can be used for parsing altlocs, charges, etc, from all of these formats).

However, I'm slightly lost on what's to be done next for this PR to be merged, so I'm asking if someone could help me navigate here (tagging @richardjgowers here as author of original PDBx implementation 4303).

PR Checklist

Tests?
Docs?
CHANGELOG updated?
Issue raised/referenced?

Developers certificate of origin

I certify that this contribution is covered by the LGPLv2.1+ license as defined in our LICENSE and adheres to the Developer Certificate of Origin.

📚 Documentation preview 📚: https://mdanalysis--4712.org.readthedocs.build/en/4712/

pep8speaks · 2024-09-20T21:57:28Z

Hello @marinegor! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file package/MDAnalysis/coordinates/MMCIF.py:

Line 28:80: E501 line too long (84 > 79 characters)
Line 41:80: E501 line too long (85 > 79 characters)
Line 42:80: E501 line too long (93 > 79 characters)
Line 61:80: E501 line too long (104 > 79 characters)
Line 65:80: E501 line too long (87 > 79 characters)
Line 67:80: E501 line too long (107 > 79 characters)

In the file package/MDAnalysis/topology/MMCIFParser.py:

Line 2:24: W291 trailing whitespace
Line 60:80: E501 line too long (111 > 79 characters)
Line 64:80: E501 line too long (88 > 79 characters)
Line 68:80: E501 line too long (123 > 79 characters)
Line 78:80: E501 line too long (122 > 79 characters)
Line 102:80: E501 line too long (108 > 79 characters)
Line 109:80: E501 line too long (80 > 79 characters)
Line 124:80: E501 line too long (91 > 79 characters)
Line 168:80: E501 line too long (81 > 79 characters)
Line 169:80: E501 line too long (126 > 79 characters)
Line 179:80: E501 line too long (125 > 79 characters)
Line 218:80: E501 line too long (126 > 79 characters)
Line 236:80: E501 line too long (140 > 79 characters)
Line 275:80: E501 line too long (87 > 79 characters)
Line 286:80: E501 line too long (90 > 79 characters)

In the file package/MDAnalysis/topology/PDBParser.py:

Line 49:80: E501 line too long (80 > 79 characters)
Line 50:80: E501 line too long (84 > 79 characters)

In the file package/MDAnalysis/topology/__init__.py:

Line 336:26: W292 no newline at end of file

In the file testsuite/MDAnalysisTests/datafiles.py:

Line 48:80: E501 line too long (103 > 79 characters)
Line 81:80: E501 line too long (80 > 79 characters)
Line 97:80: E501 line too long (86 > 79 characters)
Line 271:80: E501 line too long (90 > 79 characters)
Line 339:80: E501 line too long (104 > 79 characters)
Line 386:80: E501 line too long (83 > 79 characters)
Line 435:80: E501 line too long (80 > 79 characters)
Line 462:80: E501 line too long (80 > 79 characters)
Line 480:80: E501 line too long (80 > 79 characters)
Line 492:80: E501 line too long (80 > 79 characters)
Line 493:80: E501 line too long (80 > 79 characters)
Line 496:80: E501 line too long (83 > 79 characters)
Line 497:80: E501 line too long (86 > 79 characters)
Line 545:80: E501 line too long (82 > 79 characters)
Line 546:80: E501 line too long (82 > 79 characters)
Line 548:80: E501 line too long (88 > 79 characters)
Line 550:80: E501 line too long (88 > 79 characters)
Line 551:80: E501 line too long (81 > 79 characters)
Line 776:80: E501 line too long (81 > 79 characters)
Line 777:80: E501 line too long (87 > 79 characters)
Line 778:80: E501 line too long (84 > 79 characters)
Line 779:80: E501 line too long (85 > 79 characters)
Line 780:80: E501 line too long (83 > 79 characters)

In the file testsuite/MDAnalysisTests/topology/test_mmcif.py:

Line 37:80: E501 line too long (88 > 79 characters)
Line 49:80: E501 line too long (84 > 79 characters)

Comment last updated at 2024-10-02 17:52:18 UTC

github-actions · 2024-09-20T21:59:32Z

Linter Bot Results:

Hi @marinegor! Thanks for making this PR. We linted your code and found the following:

Some issues were found with the formatting of your code.

Code Location	Outcome
main package	⚠️ Possible failure
testsuite	⚠️ Possible failure

Please have a look at the darker-main-code and darker-test-code steps here for more details: https://github.com/MDAnalysis/mdanalysis/actions/runs/11148966346/job/30986736623

Please note: The black linter is purely informational, you can safely ignore these outcomes if there are no flake8 failures!

richardjgowers

Looks good so far, will require a small test file to check reader/parser halves.

richardjgowers · 2024-09-21T15:10:13Z

package/MDAnalysis/coordinates/MMCIF.py

+        pass
+
+
+class MMCIFWriter(base.WriterBase):


I wouldn't include this at this stage, Writer is optional

richardjgowers · 2024-09-21T15:10:31Z

package/MDAnalysis/topology/MMCIFParser.py

+from .base import TopologyReaderBase
+
+
+def _into_idx(arr: list[int]) -> list[int]:


Document what this does, ideally with an example

richardjgowers · 2024-09-21T15:11:33Z

package/MDAnalysis/topology/MMCIFParser.py

+            record_types,  # res.het_flag
+            tempfactors,  # at.b_iso
+            residx,  # _into_idx(res.seqid.num)
+        ) = map(  # this is probably not pretty, but it's efficient -- one loop over the mmcif


Are all the fields here guaranteed in a valid pdbx? One benefit to working column by column is that you can do optional columns

Do you have an example of a PDBx in mind, or like a test set for them? I've never actually worked with the format, since in RCSB afaik we have only pdb or mmcif

PDBx is mmcif. The download links here will give you an example file: https://www.rcsb.org/structure/4ake we use 4ake elsewhere in the testsuite. In my experience, sometimes the PDB / mmcif versions of the same entry aren't completely identical, so I wouldn't worry about trying to align the PDB & PDBx tests.

richardjgowers · 2024-09-21T15:12:17Z

package/MDAnalysis/topology/MMCIFParser.py

+            np.array,
+            list(
+                zip(
+                    *[


I'm struggling to follow the logic here, a comment breaking down what this double nested loop iteration into a zip is doing would be nice

richardjgowers · 2024-09-21T15:12:54Z

package/pyproject.toml

@@ -78,6 +78,7 @@ extra_formats = [
    "pytng>=0.2.3",
    "gsd>3.0.0",
    "rdkit>=2020.03.1",
+    "gemmi", # for mmcif format


This will probably be optional, so other imports will have to respect that too

marinegor added 13 commits May 22, 2024 20:04

Start working on MMCIF parser

aa2a88f

Add first (not working) version of MMCIFReader and MMCIF topology parser

218cf43

Do some squashing

7f78e02

Remove inherited docs

6682d6e

Try improving the parsing

817f3a0

Try three independent loops over the model

3cc8c80

Merge remote-tracking branch 'upstream/develop' into feature/mmcif

f1bf325

Add gemmi dependency

d21c220

necessary params

2a1be15

finished sorting atom attrs

77645e6

add function for transformation into *idx

91e6942

oh damn seems to finally be working

9a0c086

remove TODOs

9c731df

Remove debug prints

8b40ec7

richardjgowers reviewed Sep 21, 2024

View reviewed changes

marinegor added 13 commits September 23, 2024 00:31

Merge branch 'develop' into feature/mmcif

bdcbd73

try to pack things into separate class in utils?

401a4d3

remove unnecessary functions

9c336bd

copy all loops into separate functions

def88e4

Move loops over structures into functions

cabfd37

Move coordinate fetching into function for the coordinate reader as well

4c9d930

Fix imports

184491a

Start adding documentation

3de8565

Reference MMCIFParser in PDBParser

ca6ebbb

Add documentation for trajectory and topology parsers

45077ad

Add mmcif tests

9a1a59a

Update format specifications

27c10d6

Write simple tests

950cfcf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Implementing `gemmi`-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

[WIP] Implementing `gemmi`-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

marinegor commented Sep 20, 2024 •

edited by github-actions bot

Loading

pep8speaks commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading

richardjgowers left a comment

richardjgowers Sep 21, 2024

richardjgowers Sep 21, 2024

richardjgowers Sep 21, 2024

marinegor Sep 21, 2024

richardjgowers Sep 22, 2024

richardjgowers Sep 21, 2024

richardjgowers Sep 21, 2024

		from .base import TopologyReaderBase


		def _into_idx(arr: list[int]) -> list[int]:

[WIP] Implementing gemmi-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

Are you sure you want to change the base?

[WIP] Implementing gemmi-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

Conversation

marinegor commented Sep 20, 2024 • edited by github-actions bot Loading

PR Checklist

Developers certificate of origin

pep8speaks commented Sep 20, 2024 • edited Loading

Comment last updated at 2024-10-02 17:52:18 UTC

github-actions bot commented Sep 20, 2024 • edited Loading

Linter Bot Results:

richardjgowers left a comment

Choose a reason for hiding this comment

richardjgowers Sep 21, 2024

Choose a reason for hiding this comment

richardjgowers Sep 21, 2024

Choose a reason for hiding this comment

richardjgowers Sep 21, 2024

Choose a reason for hiding this comment

marinegor Sep 21, 2024

Choose a reason for hiding this comment

richardjgowers Sep 22, 2024

Choose a reason for hiding this comment

richardjgowers Sep 21, 2024

Choose a reason for hiding this comment

richardjgowers Sep 21, 2024

Choose a reason for hiding this comment

[WIP] Implementing `gemmi`-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

[WIP] Implementing `gemmi`-based mmcif reader (with easy extension to PDB/PDBx and mmJSON) #4712

marinegor commented Sep 20, 2024 •

edited by github-actions bot

Loading

pep8speaks commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading