ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release #3099

effigies · 2019-11-18T15:25:16Z

Summary

Updated Travis on master to actually install pre-release dependencies. Starting on deprecation warnings.

I would appreciate a careful review. In general, np.[as[any]]array(niimg.dataobj) is the more conservative change, keeping get_data() functionality, except for caching. When floats obviously make more sense, I'm switching to get_fdata(). We may want to specify dtype=np.float32 in these cases.

I'm also in passing removing unnecessary loading of data blocks, such as get_data().shape or get_data().ndim.

List of changes proposed in this PR (pull-request)

Purge get_data()
...?

Acknowledgment

(Mandatory) I acknowledge that this contribution will be available under the Apache 2 license.

codecov · 2019-11-18T21:11:55Z

Codecov Report

Merging #3099 into master will decrease coverage by <.01%.
The diff coverage is 34.48%.

@@            Coverage Diff             @@
##           master    #3099      +/-   ##
==========================================
- Coverage   67.79%   67.78%   -0.01%     
==========================================
  Files         295      295              
  Lines       39310    39304       -6     
  Branches     5181     5178       -3     
==========================================
- Hits        26651    26644       -7     
  Misses      11951    11951              
- Partials      708      709       +1

Flag	Coverage Δ
#smoketests	`51.23% <5.3%> (+0.18%)`	⬆️
#unittests	`65% <32.41%> (-0.04%)`	⬇️

Impacted Files	Coverage Δ
nipype/interfaces/dipy/preprocess.py	`26.96% <0%> (ø)`	⬆️
nipype/interfaces/dipy/simulate.py	`22.28% <0%> (-0.45%)`	⬇️
nipype/interfaces/utility/base.py	`87.2% <0%> (ø)`	⬆️
nipype/interfaces/nipy/preprocess.py	`44.66% <0%> (-0.11%)`	⬇️
nipype/interfaces/nipy/model.py	`24.61% <0%> (-0.01%)`	⬇️
nipype/interfaces/cmtk/parcellation.py	`12.5% <0%> (ø)`	⬆️
nipype/interfaces/dipy/reconstruction.py	`31.68% <0%> (ø)`	⬆️
nipype/algorithms/mesh.py	`29.7% <0%> (ø)`	⬆️
nipype/interfaces/cmtk/cmtk.py	`17.96% <0%> (-0.17%)`	⬇️
nipype/interfaces/dipy/tracks.py	`31.28% <0%> (ø)`	⬆️
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b5cb2aa...fcbaad8. Read the comment docs.

effigies · 2019-11-25T17:12:04Z

nipype/algorithms/confounds.py

@@ -664,7 +662,7 @@ def _run_interface(self, runtime):
            )

        components, filter_basis, metadata = compute_noise_components(
-            imgseries.get_data(),
+            imgseries.get_fdata(dtype=np.float32),


Please specifically verify that the desired behavior here would be to compute noise components on float32 data... float64 would be very large, but this data is presumably floating point, so using the dataobj interface seems needlessly cumbersome.

effigies · 2019-11-25T17:13:27Z

nipype/algorithms/confounds.py

            )

-        mask_images = self._process_masks(mask_images, imgseries.get_data())
+        mask_images = self._process_masks(mask_images, imgseries.dataobj)


I moved the array coercion to _process_masks, since the data may not be used.

effigies · 2019-11-25T18:23:35Z

This is ready for some quite tedious review. Sorry.

effigies · 2019-12-12T02:36:30Z

This really should get a second pair of eyes. It's a lot of changes.

effigies

Took another look with fresh eyes. Some proposed changes. Went ahead and cleaned up NUMPY_MMAP (see #3112) while I was at it.

I will commit the changes that are obviously needed. If someone would like to review, feel free to accept or reject the others.

effigies · 2019-12-14T15:11:01Z

nipype/algorithms/icc.py

        maskdata = np.logical_not(np.logical_or(maskdata == 0, np.isnan(maskdata)))

        session_datas = [
            [
-                nb.load(fname, mmap=NUMPY_MMAP).get_data()[maskdata].reshape(-1, 1)
+                nb.load(fname, mmap=NUMPY_MMAP).get_fdata()[maskdata].reshape(-1, 1)


To control memory usage:

Suggested change

nb.load(fname, mmap=NUMPY_MMAP).get_fdata()[maskdata].reshape(-1, 1)

nb.load(fname).get_fdata(dtype=np.float32)[maskdata].reshape(-1, 1)

effigies · 2019-12-14T15:15:44Z

nipype/algorithms/metrics.py

-        origdata1 = np.logical_and(
-            nii1.get_data() != 0, np.logical_not(np.isnan(nii1.get_data()))
-        )
+        origdata1 = np.asanyarray(nii1.dataobj)


This looks like we're finding a mask of a volume of real values, so I would probably go with fdata. The slight potential memory advantage of dataobj doesn't justify ugliness IMO.

Suggested change

origdata1 = np.asanyarray(nii1.dataobj)

origdata1 = nii1.get_fdata(dtype=np.float32)

the next line is origdata1 != 0, which may not work for float.

effigies · 2019-12-14T15:16:35Z

nipype/algorithms/metrics.py

-            nii1.get_data() != 0, np.logical_not(np.isnan(nii1.get_data()))
-        )
+        origdata1 = np.asanyarray(nii1.dataobj)
+        origdata1 = (origdata1 != 0) & ~np.isnan(origdata1)
        cog_t = np.array(center_of_mass(origdata1.copy())).reshape(-1, 1)


Not used again, so hard to see why a copy is needed.

Suggested change

cog_t = np.array(center_of_mass(origdata1.copy())).reshape(-1, 1)

cog_t = np.array(center_of_mass(origdata1)).reshape(-1, 1)

effigies · 2019-12-14T15:17:25Z

nipype/algorithms/metrics.py

-        origdata2 = np.logical_and(
-            nii2.get_data() != 0, np.logical_not(np.isnan(nii2.get_data()))
-        )
+        origdata2 = np.asanyarray(nii2.dataobj)


Again:

Suggested change

origdata2 = np.asanyarray(nii2.dataobj)

origdata2 = nii2.get_fdata(dtype=np.float32)

the next line is origdata2 != 0, which may not work for float.

effigies · 2019-12-14T15:18:01Z

nipype/algorithms/metrics.py

        else:
            return np.mean(min_dist_matrix)

    def _eucl_max(self, nii1, nii2):
        from scipy.spatial.distance import cdist

-        origdata1 = nii1.get_data()
+        origdata1 = np.asanyarray(nii1.dataobj)


Suggested change

origdata1 = np.asanyarray(nii1.dataobj)

origdata1 = nii1.get_fdata(dtype=np.float32)

ok, now i'm confused why this function uses get_fdata, but the previous one (_eucl_mean) uses dataobj

My criterion here was whether something appeared to be targeting integer values.

the next line is origdata1 == 0, which may not work for float.

nipype/interfaces/dipy/simulate.py

nipype/interfaces/freesurfer/tests/test_model.py

nipype/interfaces/nipy/preprocess.py

nipype/interfaces/nipy/utils.py

nipype/algorithms/confounds.py

effigies · 2019-12-15T13:51:12Z

@satra For context, the semantics are:

# Scales, but changes type only if slope/inter != 1/0
img.get_data() = np.asanyarray(img.dataobj)
img.dataobj[...] = np.asanyarray(img.dataobj)[...]  # May not load whole array into RAM

# Scales, and always sets dtype (default: float64)
img.get_fdata(dtype=dtype) = np.asanyarray(img.dataobj).astype(dtype)

# Does not scale, does not change type, may not be available for all images
img.dataobj.get_unscaled()

effigies · 2019-12-16T03:15:42Z

@satra Should I interpret 👍 as "go ahead and commit" and no response as not to commit?

satra · 2019-12-16T16:13:10Z

@effigies - i think it looks good to me after your semantics explanation. i have not gotten a chance to fully go through it. if you would like me to i can do it tomorrow.

effigies · 2019-12-16T16:24:49Z

I would prefer if somebody fully went through it.

The suggestions I made are optional, but I generally tried to justify them. If you agree with the justifications, please merge them. If not, just resolve the conversations.

satra · 2019-12-16T16:34:14Z

@effigies - i cannot merge them - i tried yesterday - it's because i don't have write access to your nipype repo.

effigies · 2019-12-16T16:47:45Z

Oh weird, did they change the policy on that? Should I merge the thumbs-upped?

satra

overall looks great. i left a few comments with float point comparisons. especially for those situations it seems we should convert to something where the inequality or equality holds.

does nibabel have a helper to change a float to an int that takes range into account? or we could do np.round(img).astype(np.int32)

satra · 2019-12-19T14:51:38Z

nipype/algorithms/confounds.py

-                mask = img.get_data() > 0
-            np.logical_or(mask, img.get_data() > 0, mask)
+                mask = img.get_fdata() > 0
+            np.logical_or(mask, img.get_fdata() > 0, mask)


a note that this may not be a good comparison for floating point data.

and there are a few places in this function where it happens. it's likely not a big difference in the context of this function.

satra · 2019-12-19T14:55:43Z

nipype/algorithms/metrics.py

        origdata2 = np.logical_not(np.logical_or(origdata2 == 0, np.isnan(origdata2)))

        if isdefined(self.inputs.mask_volume):
-            maskdata = nb.load(self.inputs.mask_volume).get_data()
+            maskdata = np.asanyarray(nb.load(self.inputs.mask_volume).dataobj)


the next line is maskdata == 0, which may not work for float.

satra · 2019-12-19T14:56:17Z

nipype/algorithms/metrics.py

        origdata1 = np.logical_not(np.logical_or(origdata1 == 0, np.isnan(origdata1)))
-        origdata2 = nii2.get_data()
+        origdata2 = np.asanyarray(nii2.dataobj)


the next line is origdata2 == 0, which may not work for float.

satra · 2019-12-19T14:56:44Z

nipype/algorithms/metrics.py

        else:
            return np.mean(min_dist_matrix)

    def _eucl_max(self, nii1, nii2):
        from scipy.spatial.distance import cdist

-        origdata1 = nii1.get_data()
+        origdata1 = np.asanyarray(nii1.dataobj)


the next line is origdata1 == 0, which may not work for float.

satra · 2019-12-19T14:57:39Z

nipype/algorithms/metrics.py

-        origdata2 = np.logical_and(
-            nii2.get_data() != 0, np.logical_not(np.isnan(nii2.get_data()))
-        )
+        origdata2 = np.asanyarray(nii2.dataobj)


the next line is origdata2 != 0, which may not work for float.

satra · 2019-12-19T14:58:11Z

nipype/algorithms/metrics.py

-        origdata1 = np.logical_and(
-            nii1.get_data() != 0, np.logical_not(np.isnan(nii1.get_data()))
-        )
+        origdata1 = np.asanyarray(nii1.dataobj)


the next line is origdata1 != 0, which may not work for float.

Selecting dtype=float32 for loading full time series

effigies · 2019-12-19T19:27:46Z

does nibabel have a helper to change a float to an int that takes range into account? or we could do np.round(img).astype(np.int32)

Not that I know of. It's hard to think of a general solution that doesn't get so parameterized that explicitly describing what you want with numpy doesn't make more sense.

Anyway, post-3.0 the easy way will be something like: np.int32(img.dataobj), which will scale and cast and be the equivalent (but possibly memory-saving variant) of np.asanyarray(img.dataobj).astype(np.int32).

effigies · 2019-12-19T20:53:04Z

@satra Thanks for the review. This is ready for merge, IMO.

satra · 2019-12-19T21:58:52Z

go for it, once it clears tests.

effigies force-pushed the mnt/nibabel3 branch from d87adf4 to d00267f Compare November 18, 2019 21:11

effigies force-pushed the mnt/nibabel3 branch from cc86dc5 to 4ceba42 Compare November 25, 2019 16:48

effigies commented Nov 25, 2019

View reviewed changes

effigies force-pushed the mnt/nibabel3 branch from 6768734 to dc50f92 Compare November 25, 2019 17:19

effigies mentioned this pull request Nov 25, 2019

FIX: Mark strings containing regex escapes as raw #3106

Merged

1 task

effigies changed the title ~~[WIP] ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release~~ ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release Nov 25, 2019

effigies added this to the 1.4.0 milestone Dec 14, 2019

effigies commented Dec 15, 2019

View reviewed changes

satra reviewed Dec 15, 2019

View reviewed changes

nipype/algorithms/confounds.py Show resolved Hide resolved

effigies mentioned this pull request Dec 19, 2019

REL: 1.4.0 #3116

Merged

11 tasks

satra reviewed Dec 19, 2019

View reviewed changes

effigies added 9 commits December 19, 2019 14:14

ENH: Remove deprecated get_data()

c1a6936

RF: Purge get_data from nipy, dipy interfaces

36732e5

RF: Purge get_data() from algorithms.misc

77aa4d9

TEST: Purge get_data() from algorithms.tests.*

4b618ec

RF: Purge get_data from algorithms.confounds

f737c26

Selecting dtype=float32 for loading full time series

RF: Finish purging get_data() from algorithms

68382d2

RF: Purge get_data() from interfaces.cmtk

530143e

RF: Purge get_data() from interfaces.nilearn

7fd0b80

TEST: Purge final get_data() calls from some tests

de8785b

effigies added 2 commits December 19, 2019 14:14

RF: Drop NUMPY_MMAP from surrounding code

c613c7d

FIX: Value 1 replaced with 2 by accident

8e44223

effigies force-pushed the mnt/nibabel3 branch from fbf853c to 8e44223 Compare December 19, 2019 19:28

effigies added 4 commits December 19, 2019 16:07

STY: Black

97bdbd5

FIX: Improved mask estimation

a5c8e0f

RF: Simplify data retrieval, avoid needless upcast

c87c6bf

FIX: Missing and unused imports

fcbaad8

effigies force-pushed the mnt/nibabel3 branch from ccc8dc2 to fcbaad8 Compare December 19, 2019 21:07

effigies merged commit c503c32 into nipy:master Dec 19, 2019

effigies deleted the mnt/nibabel3 branch December 19, 2019 23:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release #3099

ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release #3099

effigies commented Nov 18, 2019

codecov bot commented Nov 18, 2019 •

edited

Loading

effigies Nov 25, 2019

effigies Nov 25, 2019

effigies commented Nov 25, 2019

effigies commented Dec 12, 2019

effigies left a comment

effigies Dec 14, 2019

effigies Dec 14, 2019

satra Dec 19, 2019

effigies Dec 14, 2019

effigies Dec 14, 2019

satra Dec 19, 2019

effigies Dec 14, 2019

satra Dec 15, 2019

effigies Dec 15, 2019

satra Dec 15, 2019

satra Dec 19, 2019

effigies commented Dec 15, 2019

effigies commented Dec 16, 2019

satra commented Dec 16, 2019

effigies commented Dec 16, 2019

satra commented Dec 16, 2019

effigies commented Dec 16, 2019

satra left a comment

satra Dec 19, 2019

satra Dec 19, 2019

satra Dec 19, 2019

satra Dec 19, 2019

satra Dec 19, 2019

satra Dec 19, 2019

satra Dec 19, 2019

effigies commented Dec 19, 2019

effigies commented Dec 19, 2019

satra commented Dec 19, 2019

	nb.load(fname, mmap=NUMPY_MMAP).get_fdata()[maskdata].reshape(-1, 1)
	nb.load(fname).get_fdata(dtype=np.float32)[maskdata].reshape(-1, 1)

	origdata1 = np.asanyarray(nii1.dataobj)
	origdata1 = nii1.get_fdata(dtype=np.float32)

	cog_t = np.array(center_of_mass(origdata1.copy())).reshape(-1, 1)
	cog_t = np.array(center_of_mass(origdata1)).reshape(-1, 1)

	origdata2 = np.asanyarray(nii2.dataobj)
	origdata2 = nii2.get_fdata(dtype=np.float32)

ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release #3099

ENH: Pacify DeprecationWarnings caused by nibabel 3 pre-release #3099

Conversation

effigies commented Nov 18, 2019

Summary

List of changes proposed in this PR (pull-request)

Acknowledgment

codecov bot commented Nov 18, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

effigies commented Nov 25, 2019

effigies commented Dec 12, 2019

effigies left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

effigies commented Dec 15, 2019

effigies commented Dec 16, 2019

satra commented Dec 16, 2019

effigies commented Dec 16, 2019

satra commented Dec 16, 2019

effigies commented Dec 16, 2019

satra left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

effigies commented Dec 19, 2019

effigies commented Dec 19, 2019

satra commented Dec 19, 2019

codecov bot commented Nov 18, 2019 •

edited

Loading