Added support of timestamp/date/time using curly brackets #297

matthewryanwells · 2023-07-13T23:38:56Z

Description

Added support for using timestamp/date/time using curly brackets in the following ways to fix failing date filters in PowerBI:

Definition	Type
{ts 'YYYY-MM-DD hh:mm:ss[.SSS]'}	Timestamp
{timestamp 'YYYY-MM-DD hh:mm:ss[.SSS]'}	Timestamp
{d 'YYYY-MM-DD'}	Date
{date 'YYYY-MM-DD'}	Date
{t 'hh:mm:ss[.SSS]'}	Time
{time 'hh:mm:ss[.SSS]'}	Time

Issues Resolved

opensearch-project#364

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2023-07-13T23:43:11Z

Codecov Report

Merging #297 (be1d805) into integ-bracketed-timestamp (4102b58) will not change coverage.
The diff coverage is n/a.

❗ Current head be1d805 differs from pull request most recent head 1657498. Consider uploading reports for the commit 1657498 to get more accurate results

@@                     Coverage Diff                      @@
##             integ-bracketed-timestamp     #297   +/-   ##
============================================================
  Coverage                        97.39%   97.39%           
  Complexity                        4603     4603           
============================================================
  Files                              401      401           
  Lines                            11397    11397           
  Branches                           835      835           
============================================================
  Hits                             11100    11100           
  Misses                             290      290           
  Partials                             7        7

Flag	Coverage Δ
sql-engine	`97.39% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Yury-Fridlyand

So far so good. Please add

{dt '...'} and {datetime ...} too
UT for parser where you verify that AST from select date '...' and select {d '...'} are equal (and similar tests).
Doctests

core/src/test/java/org/opensearch/sql/expression/datetime/DateTimeFunctionTest.java

integ-test/src/test/java/org/opensearch/sql/legacy/QueryIT.java

sql/src/main/java/org/opensearch/sql/sql/parser/AstExpressionBuilder.java

integ-test/src/test/java/org/opensearch/sql/legacy/QueryIT.java

integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFunctionIT.java

GumpacG · 2023-07-14T16:26:17Z

Could you add tests for invalid string literals? For example, {ts "hello world"} should throw an error.

Yury-Fridlyand · 2023-07-20T18:19:29Z

Checkstyle fails
I think you need to update integ branch too

Signed-off-by: Matthew Wells <[email protected]>

…e from test Signed-off-by: Matthew Wells <[email protected]>

…d timestamp to normal query, added documentation Signed-off-by: Matthew Wells <[email protected]>

Signed-off-by: Matthew Wells <[email protected]>

matthewryanwells · 2023-07-21T15:59:32Z

So far so good. Please add

{dt '...'} and {datetime ...} too

UT for parser where you verify that AST from select date '...' and select {d '...'} are equal (and similar tests).

Doctests

We have decided to not add dt/datetime as PowerBI does not use that datatype
I have added unit tests and also added integration tests comparing the output of non bracketed to bracketed outputs
Add doctests

matthewryanwells · 2023-07-21T15:59:51Z

Could you add tests for invalid string literals? For example, {ts "hello world"} should throw an error.

Added tests for invalid inputs

matthewryanwells · 2023-07-21T16:00:48Z

Checkstyle fails I think you need to update integ branch too

Fixed checkstyle failures and integ branch has been updated

Signed-off-by: Matthew Wells <[email protected]>

integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFunctionIT.java

…)" This reverts commit a7f0182.

* added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]>

…-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]>

…-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> (cherry picked from commit 1a7134b)

… (opensearch-project#1897) Signed-off-by: Yury-Fridlyand <[email protected]> Added support of timestamp/date/time using curly brackets (opensearch-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> Bump bwc version (opensearch-project#1876) Signed-off-by: Vamsi Manohar <[email protected]> Prometheus Query Exemplars (opensearch-project#1782) Signed-off-by: Vamsi Manohar <[email protected]> Adding google code changes for core/src/main/java/org/opensearch/sql/analysis core/src/main/java/org/opensearch/sql/ast core/src/main/java/org/opensearch/sql/data core/src/main/java/org/opensearch/sql/datasource Signed-off-by: Mitchell Gale <[email protected]> Disable checkstyle fore core module and enable spotless check for the files being checked. Signed-off-by: Mitchell Gale <[email protected]> Added back core checkstyle test in sql-test-workflow.yml Signed-off-by: Mitchell Gale <[email protected]> Revert "Prometheus Query Exemplars (opensearch-project#1782)" This reverts commit 430d7a9. Revert "Bump bwc version (opensearch-project#1876)" This reverts commit 501392e. Revert "Added support of timestamp/date/time using curly brackets (opensearch-project#1894)" This reverts commit 1a7134b. Revert "Statically init `typeActionMap` in `OpenSearchExprValueFactory`. (#310) (opensearch-project#1897)" This reverts commit c8d42a7. Revert "Fix create_index/create_index_with_IOException issue caused by OpenSearch PR change (opensearch-project#1899)" This reverts commit 7b932a7.

…-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> (cherry picked from commit 1a7134b)

… (opensearch-project#1897) Signed-off-by: Yury-Fridlyand <[email protected]> Added support of timestamp/date/time using curly brackets (opensearch-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> Bump bwc version (opensearch-project#1876) Signed-off-by: Vamsi Manohar <[email protected]> Prometheus Query Exemplars (opensearch-project#1782) Signed-off-by: Vamsi Manohar <[email protected]> Adding google code changes for core/src/main/java/org/opensearch/sql/analysis core/src/main/java/org/opensearch/sql/ast core/src/main/java/org/opensearch/sql/data core/src/main/java/org/opensearch/sql/datasource Signed-off-by: Mitchell Gale <[email protected]> Disable checkstyle fore core module and enable spotless check for the files being checked. Signed-off-by: Mitchell Gale <[email protected]> Added back core checkstyle test in sql-test-workflow.yml Signed-off-by: Mitchell Gale <[email protected]> Revert "Prometheus Query Exemplars (opensearch-project#1782)" This reverts commit 430d7a9. Revert "Bump bwc version (opensearch-project#1876)" This reverts commit 501392e. Revert "Added support of timestamp/date/time using curly brackets (opensearch-project#1894)" This reverts commit 1a7134b. Revert "Statically init `typeActionMap` in `OpenSearchExprValueFactory`. (#310) (opensearch-project#1897)" This reverts commit c8d42a7. Revert "Fix create_index/create_index_with_IOException issue caused by OpenSearch PR change (opensearch-project#1899)" This reverts commit 7b932a7. Signed-off-by: Mitchell Gale <[email protected]>

…-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> Signed-off-by: Mitchell Gale <[email protected]>

… (opensearch-project#1897) Signed-off-by: Yury-Fridlyand <[email protected]> Added support of timestamp/date/time using curly brackets (opensearch-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> Bump bwc version (opensearch-project#1876) Signed-off-by: Vamsi Manohar <[email protected]> Prometheus Query Exemplars (opensearch-project#1782) Signed-off-by: Vamsi Manohar <[email protected]> Adding google code changes for core/src/main/java/org/opensearch/sql/analysis core/src/main/java/org/opensearch/sql/ast core/src/main/java/org/opensearch/sql/data core/src/main/java/org/opensearch/sql/datasource Signed-off-by: Mitchell Gale <[email protected]> Disable checkstyle fore core module and enable spotless check for the files being checked. Signed-off-by: Mitchell Gale <[email protected]> Added back core checkstyle test in sql-test-workflow.yml Signed-off-by: Mitchell Gale <[email protected]> Revert "Prometheus Query Exemplars (opensearch-project#1782)" This reverts commit 430d7a9. Revert "Bump bwc version (opensearch-project#1876)" This reverts commit 501392e. Revert "Added support of timestamp/date/time using curly brackets (opensearch-project#1894)" This reverts commit 1a7134b. Revert "Statically init `typeActionMap` in `OpenSearchExprValueFactory`. (#310) (opensearch-project#1897)" This reverts commit c8d42a7. Revert "Fix create_index/create_index_with_IOException issue caused by OpenSearch PR change (opensearch-project#1899)" This reverts commit 7b932a7. Signed-off-by: Mitchell Gale <[email protected]>

…ets (opensearch-project#1908) * Added support of timestamp/date/time using curly brackets (opensearch-project#1894) * Added support of timestamp/date/time using curly brackets (#297) * added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]> * improved failing tests Signed-off-by: Matthew Wells <[email protected]> * simplified tests for checking for failure Signed-off-by: Matthew Wells <[email protected]> * fixed redundant tests and improved tests that should fail Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]> (cherry picked from commit 1a7134b) * fixed bad cherrypick merge conflict Signed-off-by: Matthew Wells <[email protected]> --------- Signed-off-by: Matthew Wells <[email protected]>

matthewryanwells requested review from MaxKsyunz, Yury-Fridlyand, forestmvey, acarbonetto and GumpacG as code owners July 13, 2023 23:38

Yury-Fridlyand reviewed Jul 13, 2023

View reviewed changes

acarbonetto approved these changes Jul 14, 2023

View reviewed changes

matthewryanwells force-pushed the dev-bracketed-timestamp branch from 9352312 to cb51853 Compare July 14, 2023 17:50

forestmvey approved these changes Jul 18, 2023

View reviewed changes

MitchellGale approved these changes Jul 18, 2023

View reviewed changes

matthewryanwells force-pushed the dev-bracketed-timestamp branch from 60e610b to bbc5aa5 Compare July 19, 2023 16:18

matthewryanwells added 6 commits July 20, 2023 14:03

added bracketed time/date/timestamp input, and tests

359c3ac

Signed-off-by: Matthew Wells <[email protected]>

Removed unused import, removed accidentilly added test, removed ignor…

b22760c

…e from test Signed-off-by: Matthew Wells <[email protected]>

Added unit tests, added invalid tests, added tests comparing brackete…

39c50da

…d timestamp to normal query, added documentation Signed-off-by: Matthew Wells <[email protected]>

fixed fetched rows missing on doctest

074103d

Signed-off-by: Matthew Wells <[email protected]>

fixed checkstyle errors

05b7648

Signed-off-by: Matthew Wells <[email protected]>

moved invalid failure tests to a different folder and made them work

be1d805

Signed-off-by: Matthew Wells <[email protected]>

matthewryanwells force-pushed the dev-bracketed-timestamp branch from 9683a57 to be1d805 Compare July 20, 2023 21:37

Yury-Fridlyand approved these changes Jul 21, 2023

View reviewed changes

matthewryanwells added 2 commits July 21, 2023 13:09

Add copyright header to new file and file missing one

f69671c

Signed-off-by: Matthew Wells <[email protected]>

Added extra newline after copyright header

1657498

Signed-off-by: Matthew Wells <[email protected]>

GumpacG requested changes Jul 21, 2023

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFunctionIT.java Show resolved Hide resolved

integ-test/src/test/java/org/opensearch/sql/sql/DateTimeFunctionIT.java Show resolved Hide resolved

matthewryanwells merged commit a7f0182 into integ-bracketed-timestamp Jul 21, 2023

matthewryanwells deleted the dev-bracketed-timestamp branch July 21, 2023 21:23

matthewryanwells added a commit that referenced this pull request Jul 21, 2023

Revert "Added support of timestamp/date/time using curly brackets (#297…

fa8ea3d

…)" This reverts commit a7f0182.

matthewryanwells added a commit that referenced this pull request Jul 26, 2023

Added support of timestamp/date/time using curly brackets (#297)

3704dc4

* added bracketed time/date/timestamp input, tests, and documentation Signed-off-by: Matthew Wells <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support of timestamp/date/time using curly brackets #297

Added support of timestamp/date/time using curly brackets #297

matthewryanwells commented Jul 13, 2023 •

edited by acarbonetto

Loading

codecov bot commented Jul 13, 2023 •

edited

Loading

Yury-Fridlyand left a comment

GumpacG commented Jul 14, 2023

Yury-Fridlyand commented Jul 20, 2023

matthewryanwells commented Jul 21, 2023

matthewryanwells commented Jul 21, 2023

matthewryanwells commented Jul 21, 2023

Added support of timestamp/date/time using curly brackets #297

Added support of timestamp/date/time using curly brackets #297

Conversation

matthewryanwells commented Jul 13, 2023 • edited by acarbonetto Loading

Description

Issues Resolved

Check List

codecov bot commented Jul 13, 2023 • edited Loading

Codecov Report

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

GumpacG commented Jul 14, 2023

Yury-Fridlyand commented Jul 20, 2023

matthewryanwells commented Jul 21, 2023

matthewryanwells commented Jul 21, 2023

matthewryanwells commented Jul 21, 2023

matthewryanwells commented Jul 13, 2023 •

edited by acarbonetto

Loading

codecov bot commented Jul 13, 2023 •

edited

Loading