Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: offline map type #3746

Merged
merged 3 commits into from
Mar 22, 2024
Merged

Conversation

aceforeverd
Copy link
Collaborator

@aceforeverd aceforeverd commented Feb 4, 2024

feat

  • query with map data type from offline
  • native row builder for offline

limitation

  • not able to native encode empty map value, native row builder need some work to take the correct type for empty map, since a empty map is translated to map() expression that will inferred as map<void, void>, it does not consider type in input schema
  • each row will encoded with distinct native instance due to improve use of LLJIT  #3748, a jit can not add multiple modules
  • unsaferow encoding for map values

@github-actions github-actions bot added batch-engine openmldb batch(offline) engine execute-engine hybridse sql engine storage-engine openmldb storage engine. nameserver & tablet labels Feb 4, 2024
Copy link
Contributor

github-actions bot commented Feb 4, 2024

SDK Test Report

101 files  ±0  101 suites  ±0   2m 10s ⏱️ +5s
346 tests +1  332 ✅ +1  14 💤 ±0  0 ❌ ±0 
472 runs  +1  458 ✅ +1  14 💤 ±0  0 ❌ ±0 

Results for commit a95893f. ± Comparison against base commit 5e7f266.

This pull request removes 30 and adds 10 tests. Note that renamed tests count towards both.
  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
 ) limit 10;](2)
 ) limit 10;](3)
 FROM db1.t1
 FROM t1
 WINDOW w1 AS (
 last join db2.t2 order by db2.t2.col1
…
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[,  SELECT sum(db1.t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM db1.t1
 last join db2.t2 order by db2.t2.col1
 on db1.t1.col1 = db2.t2.col1 and db1.t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](2)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[db1,  SELECT sum(t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlLastJoinWithMultipleDB[null,  SELECT sum(db1.t1.col1) over w1 as sum_t1_col1, db2.t2.str1 as t2_str1
 FROM db1.t1
 last join db2.t2 order by db2.t2.col1
 on db1.t1.col1 = db2.t2.col1 and db1.t1.col2 = db2.t2.col0
 WINDOW w1 AS (
  PARTITION BY db1.t1.col2 ORDER BY db1.t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](3)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Fail to transform data provider op: table t1 not exists in database []](4)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT db1.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: db1.t2.str1](2)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: .t2.col1](3)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[db1, SELECT t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Column Not found: .t2.str1](1)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlMultipleDBErrorTest[null, SELECT db2.t2.str1 as t2_str1
 FROM t1
 last join db2.t2 order by db2.t2.col1
 on t1.col1 = db2.t2.col1 and t1.col2 = db2.t2.col0;
, SQL parse error: Fail to transform data provider op: table t1 not exists in database []](5)
com._4paradigm.hybridse.sdk.SqlEngineTest ‑ sqlWindowLastJoin[ SELECT sum(t1.col1) over w1 as sum_t1_col1, t2.str1 as t2_str1
 FROM t1
 last join t2 order by t2.col1
 on t1.col1 = t2.col1 and t1.col2 = t2.col0
 WINDOW w1 AS (
  PARTITION BY t1.col2 ORDER BY t1.col1
  ROWS_RANGE BETWEEN 3 PRECEDING AND CURRENT ROW
 ) limit 10;](1)
com._4paradigm.openmldb.batch.end2end.TestProject ‑ Test end2end row project with map values

♻️ This comment has been updated with latest results.

Copy link

codecov bot commented Feb 4, 2024

Codecov Report

Attention: Patch coverage is 44.69453% with 172 lines in your changes are missing coverage. Please review.

Project coverage is 74.87%. Comparing base (1106259) to head (a95893f).
Report is 1 commits behind head on main.

Files Patch % Lines
hybridse/src/codec/fe_row_codec.cc 0.00% 63 Missing ⚠️
.../com/_4paradigm/openmldb/batch/SparkRowCodec.scala 40.20% 58 Missing ⚠️
..._4paradigm/openmldb/batch/utils/DataTypeUtil.scala 60.00% 20 Missing ⚠️
hybridse/include/codec/fe_row_codec.h 0.00% 7 Missing ⚠️
hybridse/src/vm/jit_wrapper.cc 63.15% 7 Missing ⚠️
..._4paradigm/openmldb/batch/utils/HybridseUtil.scala 50.00% 5 Missing ⚠️
hybridse/src/codegen/map_ir_builder.cc 69.23% 4 Missing ⚠️
hybridse/src/vm/jit.cc 72.72% 3 Missing ⚠️
hybridse/src/codegen/ir_base_builder_test.h 60.00% 2 Missing ⚠️
...adigm/openmldb/batch/nodes/SimpleProjectPlan.scala 50.00% 2 Missing ⚠️
... and 1 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3746      +/-   ##
============================================
- Coverage     74.96%   74.87%   -0.09%     
  Complexity      658      658              
============================================
  Files           742      741       -1     
  Lines        133492   133698     +206     
  Branches       1380     1384       +4     
============================================
+ Hits         100075   100110      +35     
- Misses        33113    33284     +171     
  Partials        304      304              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

github-actions bot commented Feb 4, 2024

HybridSE Mac Test Report

20 116 tests   20 114 ✅  7m 30s ⏱️
   256 suites       2 💤
    68 files         0 ❌

Results for commit a95893f.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Feb 4, 2024

HybridSE Linux Test Report

20 116 tests   20 114 ✅  6m 5s ⏱️
   256 suites       2 💤
    68 files         0 ❌

Results for commit a95893f.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Feb 5, 2024

Linux Test Report

 53 files  ±0   60 suites  ±0   28m 4s ⏱️ +6s
671 tests ±0  651 ✅ ±0  7 💤 ±0  13 ❌ ±0 
672 runs  ±0  652 ✅ ±0  7 💤 ±0  13 ❌ ±0 

For more details on these failures, see this check.

Results for commit a95893f. ± Comparison against base commit 5e7f266.

♻️ This comment has been updated with latest results.

Happy path only.

TODO:
- support empty map
ensure each row encoded with distinct jit instance, work-around to 4paradigm#3748
Copy link
Collaborator

@tobegit3hub tobegit3hub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aceforeverd aceforeverd merged commit 524dba1 into 4paradigm:main Mar 22, 2024
29 of 33 checks passed
@aceforeverd aceforeverd deleted the feat-offline-map-type branch March 22, 2024 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
batch-engine openmldb batch(offline) engine execute-engine hybridse sql engine storage-engine openmldb storage engine. nameserver & tablet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants