Add CTF file reader and interactive crystal map plot example #451

IMBalENce · 2023-05-16T08:34:29Z

Description of the change

Following up #412, added a file reader for Oxford AZtec HKL .ctf file.
Added an example jupyter notebook for interactive plotting - clicking on an IPF map to retrieve phase name and Euler angles.

Progress of the PR

Docstrings for all functions
Unit tests with pytest for all lines
Clean code style by running black via pre-commit

Minimal example of the bug fix or new feature

>>> from orix import io
>>> xmap = io.load("data.ctf") 
>>> xmap
Phase     Orientations         Name  Space group  Point group  Proper point group     Color
   -1      8358 (2.5%)  not_indexed         None         None                None         w
    1   322978 (97.5%)     Iron fcc        Fm-3m         m-3m                 432  tab:blue
Properties: bands, error, MAD, BC, BS
Scan unit: um

For reviewers

The PR title is short, concise, and will make sense 1 year later.
New functions are imported in corresponding __init__.py.
New features, API changes, and deprecations are mentioned in the unreleased
section in CHANGELOG.rst.
Contributor(s) are listed correctly in __credits__ in orix/__init__.py and in
.zenodo.json.

hakonanes

Thank you for this reader, @IMBalENce! It will make orix accessible to many more people.

With this PR I could read the only AZtec HKL CTF file I have on my computer (I got it from you last year). The can be simplified a lot, since many parts adopted from the .ang reader are unnecessary, I think. See comments.

The example is very nice, and a great addition to the docs! The next thing would be to navigate with the arrow keys... Another PR for another time.

examples/plotting/interactive_IPF.py

hakonanes · 2023-05-23T20:32:41Z

orix/io/__init__.py

@@ -81,13 +81,19 @@ def loadctf(file_string: str) -> Rotation:
    file_string
        Path to the ``.ctf`` file. This file is assumed to list the
        Euler angles in the Bunge convention in the columns 5, 6, and 7.
+        The starting row for the data that contains Euler angles is relevant


With the new ctf reader you're adding, I suggest to leave this function as is and instead deprecate it in favor of the new one. We can then remove it in v0.13. What do you think?

Sounds good to me. I was also planning to add a ctf file writer at some stage.

Great. Please tell me if you need some hints on deprecation.

Thank you. If you could point me to some of the relevant information, I'd be very happy to take some reading.

Our dev docs briefly shows how to deprecate a function. You can catch a deprecation warning in tests like this.

orix/io/plugins/ctf.py

examples/plotting/interactive_IPF.py

IMBalENce · 2023-05-29T01:58:52Z

Thank you so much for your time on reviewing my changes @hakonanes. A lot of very helpful suggestions!

I just realize some of the test did not pass. I'll work on the test code.

hakonanes

Good work! Here are a few more comments.

I've found five different versions of the CTF format among the files collaborators have kindly shared with me over the years. Oxford Instruments, Bruker, NanoMegas ASTAR, EMsoft and MTEX can all write to this format. Your reader could read all of them (great!), but there are some slight differences and pitfalls which we need to address to support all of them. Are you OK with me contributing to your branch with a few commits to account for these differences?

Btw, I find it easiest if the reviewer resolves a conversation.

hakonanes · 2023-05-29T09:14:11Z

orix/io/__init__.py

@@ -81,13 +81,19 @@ def loadctf(file_string: str) -> Rotation:
    file_string
        Path to the ``.ctf`` file. This file is assumed to list the
        Euler angles in the Bunge convention in the columns 5, 6, and 7.
+        The starting row for the data that contains Euler angles is relevant


Great. Please tell me if you need some hints on deprecation.

orix/io/plugins/ctf.py

hakonanes · 2023-05-29T11:23:38Z

orix/io/plugins/ctf.py

+            0: [
+                "phase_id",
+                "x",
+                "y",
+                "bands",
+                "error",
+                "euler1",
+                "euler2",
+                "euler3",
+                "MAD",  # Mean angular deviation
+                "BC",  # Band contrast
+                "BS",  # Band Slope
+            ],
+        },


The columns are the same in the Bruker CTF files I have access to.

examples/plotting/interactive_IPF.py

hakonanes · 2023-11-03T16:00:41Z

@IMBalENce, do you think we could finish this work? If you don't have time, no worries, I can look at finishing myself.

IMBalENce · 2023-11-05T22:38:29Z

@hakonanes Sorry for taking so long. It was a bit hectic lately. I'll do my best to work this out in two weeks.

hakonanes · 2023-11-06T08:10:19Z

No problem! I've been off development myself lately.

IMBalENce · 2023-12-05T05:57:59Z

@hakonanes I tried my best to address each of your comments. I think most should be ok now.

I also found some issue in running pytest and can get stuck in test_io.test_load_unsupported_format. To pass all test, I temporarily comment out this section. I was wondering what would be the best practice here?

Additionally, I'm still working on the test coverage on this ctf reader. I'm hoping to complete it in a day or two.

hakonanes · 2023-12-18T20:15:25Z

I also found some issue in running pytest and can get stuck in test_io.test_load_unsupported_format.

By stuck, do you mean that pytest hangs? Or that you get an error? It would be great if you could open an issue. We'd be much closer to a solution, then!

CSSFrancis · 2024-03-26T14:51:56Z

Any movement on this lately? I can take a shot at debugging the failing test if that is helpful. Is that that only thing that is blocking this from merging?

CSSFrancis · 2024-03-26T15:32:25Z

@IMBalENce The failing test is related to the fact that there used to be a warning for trying to read a .ctf file.
(And now you can!)

Signed-off-by: IMBalENce <[email protected]>

Co-authored-by: Håkon Wiik Ånes <[email protected]>

IMBalENce · 2024-04-17T02:02:42Z

@CSSFrancis Sorry for the delay in moving this forward again. Thank you for the tips. I'm testing it now. Will get it back soon.

hakonanes · 2024-04-17T05:20:16Z

I'm ready to help and review, btw.

hakonanes · 2024-04-25T20:08:51Z

@IMBalENce, I'm looking over the PR now. I have a few suggestions for changes. I'll make a PR to your branch if that's OK.

Signed-off-by: Håkon Wiik Ånes <[email protected]>

IMBalENce · 2024-05-02T02:40:45Z

@IMBalENce, I'm looking over the PR now. I have a few suggestions for changes. I'll make a PR to your branch if that's OK.

Thank you @hakonanes. Have just merged your changes.

Signed-off-by: Håkon Wiik Ånes <[email protected]>

hakonanes · 2024-05-11T13:55:55Z

This should now be done. Could you have a brief look at my latest changes, @IMBalENce? These are the changes since you merged my PR to your branch: 2b3b7f0...a68d37e.

Signed-off-by: Håkon Wiik Ånes <[email protected]>

hakonanes

The CTF reader should now be able to read files created by Oxford Instruments, Bruker, ASTAR, EMsoft, and MTEX.

Note, I've contributed code to this PR myself, so a second look my another maintainer would be appreciated.

IMBalENce · 2024-05-13T01:26:02Z

Thank you, @hakonanes. It looks great!

hakonanes · 2024-05-13T19:00:39Z

Fantastic, thanks for having a look, @IMBalENce!

IMBalENce marked this pull request as ready for review May 16, 2023 08:40

hakonanes added enhancement New feature or request documentation Relates to the documentation labels May 16, 2023

hakonanes added this to the v0.12.0 milestone May 16, 2023

hakonanes requested changes May 24, 2023

View reviewed changes

hakonanes requested changes May 29, 2023

View reviewed changes

ondrolexa mentioned this pull request Dec 4, 2023

Hkl channel5 reader #465

Open

7 tasks

pc494 mentioned this pull request Mar 26, 2024

New production version, likely 0.12.0 #471

Closed

5 tasks

IMBalENce and others added 14 commits March 26, 2024 10:33

Add ctf file reader and interactive IPF example

3be235e

Signed-off-by: IMBalENce <[email protected]>

Update recommended review changes

069bfbc

Update examples/plotting/interactive_IPF.py

d792fdf

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Update orix/io/plugins/ctf.py

1e34a01

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Update orix/io/plugins/ctf.py

d8f4005

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Update orix/io/plugins/ctf.py

209f74c

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Refine the ctf reader for more efficient header reading

d3651fa

Add back Plugin description section to avoid io.load error

0db22b9

Update orix/io/plugins/ctf.py

e489eb5

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Update examples/plotting/interactive_IPF.py

25f9753

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Add depreciation warning for loadctf()

a9cd442

Fix test_io timeout issue, and simplify ctf reader

9c283a5

Enhance the interactive IPF plot example

e476960

Update orix/io/plugins/ctf.py

6ba09f3

Co-authored-by: Håkon Wiik Ånes <[email protected]>

Merge branch 'develop' into ctf_reader

b06875a

hakonanes added this to the v0.13.0 milestone Apr 25, 2024

hakonanes changed the title ~~Add ctf file reader and interactive IPF example~~ Add CTF file reader and interactive crystal map plot example Apr 25, 2024

hakonanes and others added 7 commits April 25, 2024 22:34

Rename interactive xmap plot, simplify and use CrystalMap.plot()

f250898

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Deprecate loadang (not just loadctf)

cba8ad1

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Prefer Formula over MaterialName for phase names from .ang files

e62b068

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Move private crystal map functions to allow calls from other files

bc70df1

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Update CTF to allow reading from ASTAR, EMsoft and MTEX files

0a8740b

Signed-off-by: Håkon Wiik Ånes <[email protected]>

List CTF reader in IO plugins in docs

de71123

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Merge branch 'hakonanes-ctf_reader' into ctf_reader

2b3b7f0

hakonanes added 8 commits May 11, 2024 13:32

Test private function in crystal map module

0d4429b

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Silence some test warnings from setuptools

8ff2eed

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Add test fixtures for CTF files in various formats

48c6505

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Test Oxford Instruments and Bruker CTF files

0170f7b

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Fix manifest exclude patterns

70e252b

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Use type hint classes for tuple and dict instead of types, valid for 3.8

1a39e98

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Complete testing of CTF reader

0b8f46b

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Exclude orix/tests directory from test coverage

2e2c8d9

Signed-off-by: Håkon Wiik Ånes <[email protected]>

hakonanes added 2 commits May 11, 2024 18:30

Remove a few more type hints using types (not supported on 3.8)

0b34bc2

Signed-off-by: Håkon Wiik Ånes <[email protected]>

Remove type hints using types also in CTF reader file

a68d37e

Signed-off-by: Håkon Wiik Ånes <[email protected]>

hakonanes mentioned this pull request May 11, 2024

Make a minor release 0.13.0 #503

Closed

5 tasks

hakonanes approved these changes May 12, 2024

View reviewed changes

hakonanes merged commit 006e85d into pyxem:develop May 13, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CTF file reader and interactive crystal map plot example #451

Add CTF file reader and interactive crystal map plot example #451

IMBalENce commented May 16, 2023 •

edited by hakonanes

Loading

hakonanes left a comment •

edited

Loading

hakonanes May 23, 2023

IMBalENce May 29, 2023

hakonanes May 29, 2023

hakonanes Jun 1, 2023

IMBalENce commented May 29, 2023 •

edited

Loading

hakonanes left a comment

hakonanes May 29, 2023

hakonanes May 29, 2023

hakonanes commented Nov 3, 2023

IMBalENce commented Nov 5, 2023

hakonanes commented Nov 6, 2023

IMBalENce commented Dec 5, 2023

hakonanes commented Dec 18, 2023

CSSFrancis commented Mar 26, 2024 •

edited

Loading

CSSFrancis commented Mar 26, 2024

IMBalENce commented Apr 17, 2024

hakonanes commented Apr 17, 2024

hakonanes commented Apr 25, 2024

IMBalENce commented May 2, 2024

hakonanes commented May 11, 2024 •

edited

Loading

hakonanes left a comment

IMBalENce commented May 13, 2024

hakonanes commented May 13, 2024

Add CTF file reader and interactive crystal map plot example #451

Add CTF file reader and interactive crystal map plot example #451

Conversation

IMBalENce commented May 16, 2023 • edited by hakonanes Loading

Description of the change

Progress of the PR

Minimal example of the bug fix or new feature

For reviewers

hakonanes left a comment • edited Loading

Choose a reason for hiding this comment

hakonanes May 23, 2023

Choose a reason for hiding this comment

IMBalENce May 29, 2023

Choose a reason for hiding this comment

hakonanes May 29, 2023

Choose a reason for hiding this comment

hakonanes Jun 1, 2023

Choose a reason for hiding this comment

IMBalENce commented May 29, 2023 • edited Loading

hakonanes left a comment

Choose a reason for hiding this comment

hakonanes May 29, 2023

Choose a reason for hiding this comment

hakonanes May 29, 2023

Choose a reason for hiding this comment

hakonanes commented Nov 3, 2023

IMBalENce commented Nov 5, 2023

hakonanes commented Nov 6, 2023

IMBalENce commented Dec 5, 2023

hakonanes commented Dec 18, 2023

CSSFrancis commented Mar 26, 2024 • edited Loading

CSSFrancis commented Mar 26, 2024

IMBalENce commented Apr 17, 2024

hakonanes commented Apr 17, 2024

hakonanes commented Apr 25, 2024

IMBalENce commented May 2, 2024

hakonanes commented May 11, 2024 • edited Loading

hakonanes left a comment

Choose a reason for hiding this comment

IMBalENce commented May 13, 2024

hakonanes commented May 13, 2024

IMBalENce commented May 16, 2023 •

edited by hakonanes

Loading

hakonanes left a comment •

edited

Loading

IMBalENce commented May 29, 2023 •

edited

Loading

CSSFrancis commented Mar 26, 2024 •

edited

Loading

hakonanes commented May 11, 2024 •

edited

Loading