Support timedelta64 dtype as input #2884

weiji14 · 2023-12-16T02:54:35Z

Description of proposed changes

Map np.timedelta64 to GMT_LONG in clib/session.py's DTYPES dictionary. Added a unit test in test_clib_put_vectors.py to test passing a numpy array with timedelta64 dtypes of various time units (year to microsecond).

TODO:

Allow passing in np.timedelta64 via put_vector
Handle np.timedelta64 min/max values in the region parameter

Example usage:

import numpy as np
import pygmt
    

fig = pygmt.Figure()
fig.basemap(
    projection="X8c/5c",
    region=[0, 8, 0, 10],
    frame=["WSne", "xaf+lForecast Days", "yaf+lRMSE"],
)
fig.plot(
    x=np.arange(np.timedelta64(0, "D"), np.timedelta64(8, "D")),
    y=np.geomspace(start=0.1, stop=9, num=8),
    style="c0.2c",
    pen="1p",
)
fig.show()

produces

Addresses #2848, extends #464 and #562.

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
If adding new functionality, add an example to docstrings or tutorials.
Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

/format: automatically format and lint the code
/test-gmt-dev: run full tests on the latest GMT development version

Map np.timedelta64 to GMT_LONG in clib/session.py's DTYPES dictionary. Added a unit test in test_clib_put_vectors.py to test passing a numpy array with timedelta64 dtypes of various time units (year to microsecond).

Make a 2D plot with Forecast Days (timedelta64) on the x-axis, and RMSE on the y-axis.

weiji14 · 2023-12-16T03:43:19Z

pygmt/tests/test_plot.py

+    fig = Figure()
+    fig.basemap(
+        projection="X8c/5c",
+        region=[0, 8, 0, 10],


Ideally, we could pass in np.timedelta64 into the region argument as mentioned at #2848 (comment).

Suggested change

region=[0, 8, 0, 10],

region=[np.timedelta64(0, "D"), np.timedelta64(8, "D"), 0, 10],

Ah ok, looks like I modified kwargs_to_strings 3 years ago at #562 to handle datetime objects in region, so could probably do something similar here for timedelta64.

github-actions · 2023-12-16T04:06:38Z

Summary of changed images

This is an auto-generated report of images that have changed on the DVC remote

Status	Path
added	pygmt/tests/baseline/test_plot_timedelta64.png

Image diff(s)

Added images

test_plot_timedelta64.png

Modified images

Path	Old	New

Report last updated at commit 4426d95

Was using 9.56.1 before.

Cast np.timedelta64 inputs to int, so that they can be understood by GMT.

weiji14 · 2023-12-16T08:45:16Z

pygmt/helpers/decorators.py

+                            if getattr(
+                                getattr(item, "dtype", ""), "name", ""
+                            ).startswith("timedelta"):
+                                # A np.timedelta64 item is cast to integer
+                                value[index] = item.astype("int")


This double getattr to obtain item.dtype.name isn't very nice, wondering if there's a good alternative.

weiji14 · 2023-12-16T08:51:22Z

pygmt/helpers/decorators.py

+    ...         np.timedelta64(0, "h"),
+    ...         np.timedelta64(24, "h"),
    ...     ]
    ... )
-    {'R': '2010-01-01T16:00:00/2020-01-01T12:23:45.000000'}
+    {'R': '2010-01-01T16:00:00/2020-01-01T12:23:45.000000/0/24'}


Apparently Python has a built-in datetime.timedelta objects, but it's super hard to cast it to an integer type while preserving the orignal unit (e.g. day, hour, minute, etc). E.g.

import datetime td = datetime.timedelta(hours=24) print(np.timedelta64(td)) # numpy.timedelta64(86400000000,'us') print(np.timedelta64(td).astype(int)) # 86400000000

Should we still support datetime.timedelta? Or only np.timedelta64?

weiji14 · 2023-12-17T04:04:00Z

pygmt/helpers/decorators.py

                        if " " in str(item):
-                            # Check if there is a space " " when converting
-                            # a pandas.Timestamp/xr.DataArray to a string.
-                            # If so, use np.datetime_as_string instead.
-                            # Convert datetime-like item to ISO 8601
-                            # string format like YYYY-MM-DDThh:mm:ss.ffffff.
-                            value[index] = np.datetime_as_string(
-                                np.asarray(item, dtype=np.datetime64)
-                            )
+                            # Check if there is a space " " in the item, which
+                            # is typically present in objects such as
+                            # np.timedelta64, pd.Timestamp, or xr.DataArray.
+                            # If so, convert the item to a numerical or string
+                            # type that is understood by GMT as follows:
+                            if getattr(
+                                getattr(item, "dtype", ""), "name", ""
+                            ).startswith("timedelta"):
+                                # A np.timedelta64 item is cast to integer
+                                value[index] = item.astype("int")
+                            else:
+                                # A pandas.Timestamp/xr.DataArray containing
+                                # a datetime-like object is cast to ISO 8601
+                                # string format like YYYY-MM-DDThh:mm:ss.ffffff
+                                value[index] = np.datetime_as_string(
+                                    np.asarray(item, dtype=np.datetime64)
+                                )


The if-then logic here might get a lot more complicated if we also decide to support passing PyArrow time/date types (xref #2800) into region/-R. At such, it might be good to isolate this logic into a separate function instead, so I'm thinking of maybe reverting commit d5bedc2 for now, but cherry-pick that change into a separate PR.

Specifically, we might want to create a dedicated _cast_datetime_to_str function. There's actually something similar used in solar (see 6406b43), so we could potentially re-use the function in many places.

Sounds good.

Ok, commit reverted in 849feda. Should be ready to review now.

This reverts commit d5bedc2.

seisman

Looks good, but maybe wait for #2694?

weiji14 · 2024-01-08T01:51:58Z

Ok, re-generated the baseline image using GMT 6.5.0 and ghostscript 10.02.1 at 4426d95, and test_plot_timedelta64 passes at 4426d95, so merging.

Support timedelta64 dtype as input

4730961

Map np.timedelta64 to GMT_LONG in clib/session.py's DTYPES dictionary. Added a unit test in test_clib_put_vectors.py to test passing a numpy array with timedelta64 dtypes of various time units (year to microsecond).

weiji14 added the feature Brand new feature label Dec 16, 2023

weiji14 added this to the 0.11.0 milestone Dec 16, 2023

weiji14 self-assigned this Dec 16, 2023

weiji14 marked this pull request as draft December 16, 2023 03:07

weiji14 mentioned this pull request Dec 16, 2023

Full support of numpy dtypes #2848

Open

24 tasks

weiji14 added 2 commits December 16, 2023 16:39

Add unit test for plotting timedelta64 data

d53138c

Make a 2D plot with Forecast Days (timedelta64) on the x-axis, and RMSE on the y-axis.

Lint

51bc55a

weiji14 commented Dec 16, 2023

View reviewed changes

weiji14 mentioned this pull request Dec 16, 2023

Refactor the kwargs_to_strings decorator to avoid try-except in loops #2885

Merged

7 tasks

weiji14 added 3 commits December 16, 2023 20:38

Merge branch 'main' into timedelta64

0c20c72

Recreate baseline image using ghostscript 9.54.0

49dbe83

Was using 9.56.1 before.

Allow np.timedelta64 inputs to region parameter

d5bedc2

Cast np.timedelta64 inputs to int, so that they can be understood by GMT.

weiji14 commented Dec 16, 2023

View reviewed changes

Fix lint error

44e62d6

weiji14 mentioned this pull request Dec 17, 2023

Support PyArrow arrays and dataframes #2800

Open

weiji14 commented Dec 17, 2023

View reviewed changes

weiji14 mentioned this pull request Dec 17, 2023

Bump ghostscript to 10.02.1 #2694

Merged

weiji14 added 2 commits December 21, 2023 23:04

Merge branch 'main' into timedelta64

3aa5e3a

Revert "Allow np.timedelta64 inputs to region parameter"

849feda

This reverts commit d5bedc2.

weiji14 marked this pull request as ready for review December 21, 2023 10:07

seisman approved these changes Dec 21, 2023

View reviewed changes

seisman added the final review call This PR requires final review and approval from a second reviewer label Dec 26, 2023

weiji14 added 2 commits January 8, 2024 13:49

Merge branch 'main' into timedelta64

4c3b8e9

Update test_plot_timedelta64 baseline img with GMT 6.5.0 and gs 10.02.1

4426d95

weiji14 merged commit d5dd5ac into main Jan 8, 2024
10 of 18 checks passed

weiji14 deleted the timedelta64 branch January 8, 2024 01:53

weiji14 removed the final review call This PR requires final review and approval from a second reviewer label Jan 8, 2024

seisman mentioned this pull request Oct 31, 2024

Test timedelta64 dtype arrays with various date/time units #3567

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support timedelta64 dtype as input #2884

Support timedelta64 dtype as input #2884

weiji14 commented Dec 16, 2023 •

edited

Loading

weiji14 Dec 16, 2023

weiji14 Dec 16, 2023

github-actions bot commented Dec 16, 2023 •

edited

Loading

Added images

Modified images

weiji14 Dec 16, 2023

weiji14 Dec 16, 2023

weiji14 Dec 17, 2023 •

edited

Loading

seisman Dec 17, 2023

weiji14 Dec 21, 2023

seisman left a comment

weiji14 commented Jan 8, 2024

	region=[0, 8, 0, 10],
	region=[np.timedelta64(0, "D"), np.timedelta64(8, "D"), 0, 10],

Support timedelta64 dtype as input #2884

Support timedelta64 dtype as input #2884

Conversation

weiji14 commented Dec 16, 2023 • edited Loading

weiji14 Dec 16, 2023

Choose a reason for hiding this comment

weiji14 Dec 16, 2023

Choose a reason for hiding this comment

github-actions bot commented Dec 16, 2023 • edited Loading

Summary of changed images

Image diff(s)

Added images

Modified images

weiji14 Dec 16, 2023

Choose a reason for hiding this comment

weiji14 Dec 16, 2023

Choose a reason for hiding this comment

weiji14 Dec 17, 2023 • edited Loading

Choose a reason for hiding this comment

seisman Dec 17, 2023

Choose a reason for hiding this comment

weiji14 Dec 21, 2023

Choose a reason for hiding this comment

seisman left a comment

Choose a reason for hiding this comment

weiji14 commented Jan 8, 2024

weiji14 commented Dec 16, 2023 •

edited

Loading

github-actions bot commented Dec 16, 2023 •

edited

Loading

weiji14 Dec 17, 2023 •

edited

Loading