feat: add new UDF `to_local_time()` #11

appletreeisyellow · 2024-07-03T15:39:20Z

Which issue does this PR close?

Help with

Rationale for this change

This PR adds a ScalarUDF function to_local_time():

this function converts a timezone-aware timestamp to local time (with no offset or timezone information). In other words, this function strips off the timezone from the timestamp, while keep the display value of the timestamp the same. See examples below
only accept 1 input with type Timestamp(..., *)
returns with type Timestamp(..., None)

Example

This is how to use it in datafusion-cli:

> select to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels');
+---------------------------------------------+
| to_local_time(Utf8("2024-04-01T00:00:20Z")) |
+---------------------------------------------+
| 2024-04-01T00:00:20                         |
+---------------------------------------------+
1 row(s) fetched.
Elapsed 0.010 seconds.

> select to_local_time('2024-04-01T00:00:20'::timestamp AT TIME ZONE 'Europe/Brussels');
+--------------------------------------------+
| to_local_time(Utf8("2024-04-01T00:00:20")) |
+--------------------------------------------+
| 2024-04-01T00:00:20                        |
+--------------------------------------------+
1 row(s) fetched.
Elapsed 0.008 seconds.

> select
  time,
  arrow_typeof(time) as type,
  to_local_time(time) as to_local_time,
  arrow_typeof(to_local_time(time)) as to_local_time_type
from (
  select '2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels' as time
);
+---------------------------+------------------------------------------------+---------------------+-----------------------------+
| time                      | type                                           | to_local_time       | to_local_time_type          |
+---------------------------+------------------------------------------------+---------------------+-----------------------------+
| 2024-04-01T00:00:20+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) | 2024-04-01T00:00:20 | Timestamp(Nanosecond, None) |
+---------------------------+------------------------------------------------+---------------------+-----------------------------+
1 row(s) fetched.
Elapsed 0.017 seconds.

Example of using `to_local_time()` in `date_bin()`

Combine to_local_time() with date_bin() will look like:

> select date_bin(interval '1 day', to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels'));
+----------------------------------------------------------------------------------------------------+
| date_bin(IntervalMonthDayNano("18446744073709551616"),to_local_time(Utf8("2024-04-01T00:00:20Z"))) |
+----------------------------------------------------------------------------------------------------+
| 2024-04-01T00:00:00                                                                                |
+----------------------------------------------------------------------------------------------------+


> select date_bin(interval '1 day', to_local_time('2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels')) AT TIME ZONE 'Europe/Brussels';
+----------------------------------------------------------------------------------------------------+
| date_bin(IntervalMonthDayNano("18446744073709551616"),to_local_time(Utf8("2024-04-01T00:00:20Z"))) |
+----------------------------------------------------------------------------------------------------+
| 2024-04-01T00:00:00+02:00                                                                          |
+----------------------------------------------------------------------------------------------------+

Click to see more examples of applying to array values

Write sample data

create or replace table t AS
VALUES
  ('2024-01-01T00:00:01Z'),
  ('2024-02-01T00:00:01Z'),
  ('2024-03-01T00:00:01Z'),
  ('2024-04-01T00:00:01Z'),
  ('2024-05-01T00:00:01Z'),
  ('2024-06-01T00:00:01Z'),
  ('2024-07-01T00:00:01Z'),
  ('2024-08-01T00:00:01Z'),
  ('2024-09-01T00:00:01Z'),
  ('2024-10-01T00:00:01Z'),
  ('2024-11-01T00:00:01Z'),
  ('2024-12-01T00:00:01Z')
;

create or replace view t_utc as
select column1::timestamp AT TIME ZONE 'UTC' as "column1"
from t;

create or replace view t_timezone 
as 
select column1::timestamp AT TIME ZONE 'Europe/Brussels' as "column1" 
from t;

See how tables look like

> select column1, arrow_typeof(column1) from t;
+----------------------+-------------------------+
| column1              | arrow_typeof(t.column1) |
+----------------------+-------------------------+
| 2024-01-01T00:00:01Z | Utf8                    |
| 2024-02-01T00:00:01Z | Utf8                    |
| 2024-03-01T00:00:01Z | Utf8                    |
| 2024-04-01T00:00:01Z | Utf8                    |
| 2024-05-01T00:00:01Z | Utf8                    |
| 2024-06-01T00:00:01Z | Utf8                    |
| 2024-07-01T00:00:01Z | Utf8                    |
| 2024-08-01T00:00:01Z | Utf8                    |
| 2024-09-01T00:00:01Z | Utf8                    |
| 2024-10-01T00:00:01Z | Utf8                    |
| 2024-11-01T00:00:01Z | Utf8                    |
| 2024-12-01T00:00:01Z | Utf8                    |
+----------------------+-------------------------+
12 row(s) fetched.
Elapsed 0.009 seconds.

> select column1, arrow_typeof(column1) from t_utc;
+----------------------+------------------------------------+
| column1              | arrow_typeof(t_utc.column1)        |
+----------------------+------------------------------------+
| 2024-01-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-02-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-03-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-04-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-05-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-06-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-07-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-08-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-09-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-10-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-11-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
| 2024-12-01T00:00:01Z | Timestamp(Nanosecond, Some("UTC")) |
+----------------------+------------------------------------+
12 row(s) fetched.
Elapsed 0.011 seconds.

> select column1, arrow_typeof(column1) from t_timezone;
+---------------------------+------------------------------------------------+
| column1                   | arrow_typeof(t_timezone.column1)               |
+---------------------------+------------------------------------------------+
| 2024-01-01T00:00:01+01:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-02-01T00:00:01+01:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-03-01T00:00:01+01:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-04-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-05-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-06-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-07-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-08-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-09-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-10-01T00:00:01+02:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-11-01T00:00:01+01:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
| 2024-12-01T00:00:01+01:00 | Timestamp(Nanosecond, Some("Europe/Brussels")) |
+---------------------------+------------------------------------------------+
12 row(s) fetched.
Elapsed 0.012 seconds.

Query using to_local_time()

> select column1, to_local_time(column1), arrow_typeof(to_local_time(column1)) from t_utc;
+----------------------+------------------------------+--------------------------------------------+
| column1              | to_local_time(t_utc.column1) | arrow_typeof(to_local_time(t_utc.column1)) |
+----------------------+------------------------------+--------------------------------------------+
| 2024-01-01T00:00:01Z | 2024-01-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-02-01T00:00:01Z | 2024-02-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-03-01T00:00:01Z | 2024-03-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-04-01T00:00:01Z | 2024-04-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-05-01T00:00:01Z | 2024-05-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-06-01T00:00:01Z | 2024-06-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-07-01T00:00:01Z | 2024-07-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-08-01T00:00:01Z | 2024-08-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-09-01T00:00:01Z | 2024-09-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-10-01T00:00:01Z | 2024-10-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-11-01T00:00:01Z | 2024-11-01T00:00:01          | Timestamp(Nanosecond, None)                |
| 2024-12-01T00:00:01Z | 2024-12-01T00:00:01          | Timestamp(Nanosecond, None)                |
+----------------------+------------------------------+--------------------------------------------+
12 row(s) fetched.
Elapsed 0.015 seconds.

> select column1, to_local_time(column1), arrow_typeof(to_local_time(column1)) from t_timezone;
+---------------------------+-----------------------------------+-------------------------------------------------+
| column1                   | to_local_time(t_timezone.column1) | arrow_typeof(to_local_time(t_timezone.column1)) |
+---------------------------+-----------------------------------+-------------------------------------------------+
| 2024-01-01T00:00:01+01:00 | 2024-01-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-02-01T00:00:01+01:00 | 2024-02-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-03-01T00:00:01+01:00 | 2024-03-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-04-01T00:00:01+02:00 | 2024-04-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-05-01T00:00:01+02:00 | 2024-05-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-06-01T00:00:01+02:00 | 2024-06-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-07-01T00:00:01+02:00 | 2024-07-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-08-01T00:00:01+02:00 | 2024-08-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-09-01T00:00:01+02:00 | 2024-09-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-10-01T00:00:01+02:00 | 2024-10-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-11-01T00:00:01+01:00 | 2024-11-01T00:00:01               | Timestamp(Nanosecond, None)                     |
| 2024-12-01T00:00:01+01:00 | 2024-12-01T00:00:01               | Timestamp(Nanosecond, None)                     |
+---------------------------+-----------------------------------+-------------------------------------------------+
12 row(s) fetched.
Elapsed 0.016 seconds.

Combine with date_bin()

> select date_bin(interval '1 day', to_local_time(column1)) AT TIME ZONE 'Europe/Brussels' as date_bin from t_utc;
+---------------------------+
| date_bin                  |
+---------------------------+
| 2024-01-01T00:00:00+01:00 |
| 2024-02-01T00:00:00+01:00 |
| 2024-03-01T00:00:00+01:00 |
| 2024-04-01T00:00:00+02:00 |
| 2024-05-01T00:00:00+02:00 |
| 2024-06-01T00:00:00+02:00 |
| 2024-07-01T00:00:00+02:00 |
| 2024-08-01T00:00:00+02:00 |
| 2024-09-01T00:00:00+02:00 |
| 2024-10-01T00:00:00+02:00 |
| 2024-11-01T00:00:00+01:00 |
| 2024-12-01T00:00:00+01:00 |
+---------------------------+
12 row(s) fetched.
Elapsed 0.023 seconds.

> select date_bin(interval '1 day', to_local_time(column1)) AT TIME ZONE 'Europe/Brussels' as date_bin from t_timezone;
+---------------------------+
| date_bin                  |
+---------------------------+
| 2024-01-01T00:00:00+01:00 |
| 2024-02-01T00:00:00+01:00 |
| 2024-03-01T00:00:00+01:00 |
| 2024-04-01T00:00:00+02:00 |
| 2024-05-01T00:00:00+02:00 |
| 2024-06-01T00:00:00+02:00 |
| 2024-07-01T00:00:00+02:00 |
| 2024-08-01T00:00:00+02:00 |
| 2024-09-01T00:00:00+02:00 |
| 2024-10-01T00:00:00+02:00 |
| 2024-11-01T00:00:00+01:00 |
| 2024-12-01T00:00:00+01:00 |
+---------------------------+
12 row(s) fetched.
Elapsed 0.011 seconds.

What changes are included in this PR?

New ScalarUDF function to_local_time() with tests

Are these changes tested?

Yes

Are there any user-facing changes?

No API changes.

chore: doc chore: doc chore: doc chore: doc

… `interval` (apache#11466) * Unparser rule for datatime cast (#10) * use timestamp as the identifier for date64 * rename * implement CustomDialectBuilder * fix * dialect with interval style (#11) --------- Co-authored-by: Phillip LeBlanc <[email protected]> * fmt * clippy * doc * Update datafusion/sql/src/unparser/expr.rs Co-authored-by: Andrew Lamb <[email protected]> * update the doc for CustomDialectBuilder * fix doc test --------- Co-authored-by: Phillip LeBlanc <[email protected]> Co-authored-by: Andrew Lamb <[email protected]>

appletreeisyellow · 2024-08-06T18:48:22Z

Superseded by apache#11347

appletreeisyellow added 10 commits July 3, 2024 10:38

feat: add UDF to_local_time()

e3fc8ea

chore: support column value in array

a3d0be9

chore: lint

98b4e58

chore: fix conversion for us, ms, and s

f3e49ad

Merge branch 'main' into chunchun/udf-to-localtime

e59ff24

chore: add more tests for daylight savings time

62d4cef

chore: add function description

8536095

refactor: update tests and add examples in description

ebeef93

chore: add description and example

d7ce07f

chore: doc

a49eca9

chore: doc chore: doc chore: doc chore: doc

appletreeisyellow force-pushed the chunchun/udf-to-localtime branch from 7427955 to a49eca9 Compare July 9, 2024 01:57

appletreeisyellow closed this Aug 6, 2024

appletreeisyellow deleted the chunchun/udf-to-localtime branch August 6, 2024 18:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add new UDF `to_local_time()` #11

feat: add new UDF `to_local_time()` #11

appletreeisyellow commented Jul 3, 2024 •

edited

Loading

appletreeisyellow commented Aug 6, 2024

feat: add new UDF to_local_time() #11

feat: add new UDF to_local_time() #11

Conversation

appletreeisyellow commented Jul 3, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

Example

Example of using to_local_time() in date_bin()

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

appletreeisyellow commented Aug 6, 2024

feat: add new UDF `to_local_time()` #11

feat: add new UDF `to_local_time()` #11

appletreeisyellow commented Jul 3, 2024 •

edited

Loading

Example of using `to_local_time()` in `date_bin()`