chore: test Mergeinto runtime filter performance and wizard #14212

JackTan25 · 2024-01-03T03:16:13Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

we try to merge the merge-into-runtime-filter branch.

we trun on the broadcast join button in merge into optimizer.
we use query optimizer result when change join order.
we test performance and corrects for this pr and passed wizard, test scripts, and more important performance
test which doesn't increase.

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

into add_cluster_test

github-actions · 2024-01-03T20:05:05Z

Docker Image for PR

tag: pr-14212-6b8c41b

note: this image tag is only available for internal use,
please check the internal doc for more details.

JackTan25 · 2024-01-04T04:16:34Z

Click me

deploy@(merge_into_runtime_filter)/test> merge into small_target as t1 using (select * from target_table_cluster) as t2 on t1.l_partkey = t2.l_partkey and t1.l_orderkey = t2.l_orderkey and t1.l_suppkey = t2.l_suppkey and t1.l_linenumber = t2.l_linenumber and t1.l_shipdate = t2.l_shipdate when matched  then update *;

MERGE INTO small_target AS t1 USING (
  SELECT
    *
  FROM
    target_table_cluster
) AS t2 ON t1.l_partkey = t2.l_partkey
AND t1.l_orderkey = t2.l_orderkey
AND t1.l_suppkey = t2.l_suppkey
AND t1.l_linenumber = t2.l_linenumber
AND t1.l_shipdate = t2.l_shipdate
WHEN matched THEN
UPDATE
  *

┌────────────────────────┐
│ number of rows updated │
│          Int32         │
├────────────────────────┤
│                 500000 │
└────────────────────────┘
1 row read in 9.242 sec. Processed 1 million row, 164.99 MiB (108.2 thousand row/s, 17.85 MiB/s)

deploy@(merge_into_runtime_filter)/test> show tables;

SHOW TABLES

┌────────────────────────┐
│ Tables_in_test │
│         String         │
├────────────────────────┤
│ cluster_source         │
│ cluster_target         │
│ large_source           │
│ small_target           │
│ source_table           │
│ target_table           │
│ target_table_120       │
│ target_table_24        │
│ target_table_cluster   │
│ tmp                    │
└────────────────────────┘
10 rows read in 1.251 sec. Processed 10 rows, 2.07 KiB (7.99 rows/s, 1.65 KiB/s)

deploy@(merge_into_runtime_filter)/test> merge into target_table_cluster as t1 using (select * from source_table) as t2 on t1.l_partkey = t2.l_partkey and t1.l_orderkey = t2.l_orderkey and t1.l_suppkey = t2.l_suppkey and t1.l_linenumber = t2.l_linenumber and t1.l_shipdate = t2.l_shipdate when matched  then update *;

MERGE INTO target_table_cluster AS t1 USING (
  SELECT
    *
  FROM
    source_table
) AS t2 ON t1.l_partkey = t2.l_partkey
AND t1.l_orderkey = t2.l_orderkey
AND t1.l_suppkey = t2.l_suppkey
AND t1.l_linenumber = t2.l_linenumber
AND t1.l_shipdate = t2.l_shipdate
WHEN matched THEN
UPDATE
  *

┌────────────────────────┐
│ number of rows updated │
│          Int32         │
├────────────────────────┤
│                 500000 │
└────────────────────────┘
1 row read in 11.430 sec. Processed 1 million row, 164.99 MiB (87.49 thousand row/s, 14.43 MiB/s)

after fix runtime filter constraint, we can get good performance.

JackTan25 · 2024-01-04T04:37:10Z

distributed mode passed in wizard:

Click me

Preparing to run MERGE-INTO-C1...
Executing command: bendsql --query=-- MERGE-INTO-C1: Asset Types Distribution
SELECT asset_type, COUNT(*) AS count
FROM assets
GROUP BY asset_type
ORDER BY count DESC, asset_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

Executing command: snowsql --query -- MERGE-INTO-C1: Asset Types Distribution
SELECT asset_type, COUNT(*) AS count
FROM assets
GROUP BY asset_type
ORDER BY count DESC, asset_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

OK - MERGE-INTO-C1
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

Preparing to run MERGE-INTO-C2...
Executing command: bendsql --query=

-- MERGE-INTO-C2: Aggregated Quantity and Value Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       SUM(value) AS total_value,
       AVG(value) AS average_value
FROM assets -D mergeinto
Command executed successfully. Output:
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

Executing command: snowsql --query 

-- MERGE-INTO-C2: Aggregated Quantity and Value Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       SUM(value) AS total_value,
       AVG(value) AS average_value
FROM assets --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

OK - MERGE-INTO-C2
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

Preparing to run MERGE-INTO-C3...
Executing command: bendsql --query=

-- MERGE-INTO-C3: Assets Counts by User
SELECT user_id, COUNT(*) AS count
FROM assets
GROUP BY user_id
ORDER BY count DESC, user_id
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

Executing command: snowsql --query 

-- MERGE-INTO-C3: Assets Counts by User
SELECT user_id, COUNT(*) AS count
FROM assets
GROUP BY user_id
ORDER BY count DESC, user_id
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

OK - MERGE-INTO-C3
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

Preparing to run MERGE-INTO-C4...
Executing command: bendsql --query=

-- MERGE-INTO-C4: Date Range Analysis of Last Update
SELECT CASE
           WHEN last_updated < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM assets
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-12-31        410014
Before 2022     39986

Executing command: snowsql --query 

-- MERGE-INTO-C4: Date Range Analysis of Last Update
SELECT CASE
           WHEN last_updated < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM assets
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-12-31        410014
Before 2022     39986

OK - MERGE-INTO-C4
After 2021-12-31        410014
Before 2022     39986

Preparing to run MERGE-INTO-C5...
Executing command: bendsql --query=

-- MERGE-INTO-C5: General Status Distribution
SELECT status, COUNT(*) AS count
FROM orders
GROUP BY status
ORDER BY count DESC, status
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

Executing command: snowsql --query 

-- MERGE-INTO-C5: General Status Distribution
SELECT status, COUNT(*) AS count
FROM orders
GROUP BY status
ORDER BY count DESC, status
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

OK - MERGE-INTO-C5
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

Preparing to run MERGE-INTO-C6...
Executing command: bendsql --query=

-- MERGE-INTO-C6: General Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM orders -D mergeinto
Command executed successfully. Output:
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

Executing command: snowsql --query 

-- MERGE-INTO-C6: General Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM orders --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

OK - MERGE-INTO-C6
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

Preparing to run MERGE-INTO-C7...
Executing command: bendsql --query=

-- MERGE-INTO-C7: New Orders vs Existing Orders Count
SELECT CASE
           WHEN order_id > 500000 THEN 'New Order'
           ELSE 'Existing Order'
           END AS order_category,
       COUNT(*) AS count
FROM orders
GROUP BY order_category
ORDER BY count DESC
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
New Order       1649995
Existing Order  750030

Executing command: snowsql --query 

-- MERGE-INTO-C7: New Orders vs Existing Orders Count
SELECT CASE
           WHEN order_id > 500000 THEN 'New Order'
           ELSE 'Existing Order'
           END AS order_category,
       COUNT(*) AS count
FROM orders
GROUP BY order_category
ORDER BY count DESC
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
New Order       1649995
Existing Order  750030

OK - MERGE-INTO-C7
New Order       1649995
Existing Order  750030

Preparing to run MERGE-INTO-C8...
Executing command: bendsql --query=

-- MERGE-INTO-C8: Order Type Distribution
SELECT order_type, COUNT(*) AS count
FROM orders
GROUP BY order_type
ORDER BY count DESC, order_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
buy     1274986
sell    1125039

Executing command: snowsql --query 

-- MERGE-INTO-C8: Order Type Distribution
SELECT order_type, COUNT(*) AS count
FROM orders
GROUP BY order_type
ORDER BY count DESC, order_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
buy     1274986
sell    1125039

OK - MERGE-INTO-C8
buy     1274986
sell    1125039

Preparing to run MERGE-INTO-C9...
Executing command: bendsql --query=

-- MERGE-INTO-C9: Date Range Analysis
SELECT CASE
           WHEN created_at < '2022-01-01' THEN 'Before 2022'
           WHEN created_at BETWEEN '2021-01-01' AND '2021-06-30' THEN 'First Half 2021'
           ELSE 'After 2021-06-30'
           END AS date_range,
       COUNT(*) AS count
FROM orders
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-06-30        150000
Before 2022     2250025

Executing command: snowsql --query 

-- MERGE-INTO-C9: Date Range Analysis
SELECT CASE
           WHEN created_at < '2022-01-01' THEN 'Before 2022'
           WHEN created_at BETWEEN '2021-01-01' AND '2021-06-30' THEN 'First Half 2021'
           ELSE 'After 2021-06-30'
           END AS date_range,
       COUNT(*) AS count
FROM orders
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-06-30        150000
Before 2022     2250025

OK - MERGE-INTO-C9
After 2021-06-30        150000
Before 2022     2250025

Preparing to run MERGE-INTO-C10...
Executing command: bendsql --query=

-- MERGE-INTO-C10: Price Analysis After Adjustments
SELECT SUM(price) AS total_price,
       AVG(price) AS average_price,
       MIN(price) AS min_price,
       MAX(price) AS max_price
FROM orders -D mergeinto
Command executed successfully. Output:
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

Executing command: snowsql --query 

-- MERGE-INTO-C10: Price Analysis After Adjustments
SELECT SUM(price) AS total_price,
       AVG(price) AS average_price,
       MIN(price) AS min_price,
       MAX(price) AS max_price
FROM orders --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

OK - MERGE-INTO-C10
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

Preparing to run MERGE-INTO-C11...
Executing command: bendsql --query=

-- MERGE-INTO-C11: Transaction Types Distribution
SELECT transaction_type, COUNT(*) AS count
FROM transactions
GROUP BY transaction_type
ORDER BY count DESC, transaction_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
trade   505920
deposit 26492
withdrawal      26129

Executing command: snowsql --query 

-- MERGE-INTO-C11: Transaction Types Distribution
SELECT transaction_type, COUNT(*) AS count
FROM transactions
GROUP BY transaction_type
ORDER BY count DESC, transaction_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
trade   505920
deposit 26492
withdrawal      26129

OK - MERGE-INTO-C11
trade   505920
deposit 26492
withdrawal      26129

Preparing to run MERGE-INTO-C12...
Executing command: bendsql --query=

-- MERGE-INTO-C12: Aggregated Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM transactions -D mergeinto
Command executed successfully. Output:
73627261.57298446       131.820692792444        0.00106899      664.06985910

Executing command: snowsql --query 

-- MERGE-INTO-C12: Aggregated Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM transactions --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
73627261.57298446       131.820692792444        0.00106899      664.06985910

OK - MERGE-INTO-C12
73627261.57298446       131.820692792444        0.00106899      664.06985910

Preparing to run MERGE-INTO-C13...
Executing command: bendsql --query=

-- MERGE-INTO-C13: Transaction Counts by User and Asset Type
SELECT user_id, asset_type, COUNT(*) AS count
FROM transactions
GROUP BY user_id, asset_type
ORDER BY count DESC, user_id, asset_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

Executing command: snowsql --query 

-- MERGE-INTO-C13: Transaction Counts by User and Asset Type
SELECT user_id, asset_type, COUNT(*) AS count
FROM transactions
GROUP BY user_id, asset_type
ORDER BY count DESC, user_id, asset_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

OK - MERGE-INTO-C13
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

Preparing to run MERGE-INTO-C14...
Executing command: bendsql --query=

-- MERGE-INTO-C14: Date Range Analysis of Transactions
SELECT CASE
           WHEN transaction_time < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM transactions
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-12-31        483526
Before 2022     75015

Executing command: snowsql --query 

-- MERGE-INTO-C14: Date Range Analysis of Transactions
SELECT CASE
           WHEN transaction_time < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM transactions
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-12-31        483526
Before 2022     75015

OK - MERGE-INTO-C14
After 2021-12-31        483526
Before 2022     75015

Preparing to run MERGE-INTO-C15...
Executing command: bendsql --query=

-- MERGE-INTO-C15: asserts
SELECT asset_type, SUM(quantity) AS total_quantity, SUM(value) AS total_value
FROM assets
GROUP BY asset_type ORDER BY asset_type ASC -D mergeinto
Command executed successfully. Output:
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

Executing command: snowsql --query 

-- MERGE-INTO-C15: asserts
SELECT asset_type, SUM(quantity) AS total_quantity, SUM(value) AS total_value
FROM assets
GROUP BY asset_type ORDER BY asset_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

OK - MERGE-INTO-C15
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

Preparing to run MERGE-INTO-C16...
Executing command: bendsql --query=

-- MERGE-INTO-C16: orders
SELECT asset_type, SUM(quantity) AS total_quantity, AVG(price) AS average_price
FROM orders
GROUP BY asset_type ORDER BY asset_type ASC -D mergeinto
Command executed successfully. Output:
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

Executing command: snowsql --query 

-- MERGE-INTO-C16: orders
SELECT asset_type, SUM(quantity) AS total_quantity, AVG(price) AS average_price
FROM orders
GROUP BY asset_type ORDER BY asset_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

OK - MERGE-INTO-C16
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

Preparing to run MERGE-INTO-C17...
Executing command: bendsql --query=

-- MERGE-INTO-C17: transactions
SELECT transaction_type, SUM(quantity) AS total_quantity
FROM transactions
GROUP BY transaction_type ORDER BY transaction_type ASC -D mergeinto
Command executed successfully. Output:
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

Executing command: snowsql --query 

-- MERGE-INTO-C17: transactions
SELECT transaction_type, SUM(quantity) AS total_quantity
FROM transactions
GROUP BY transaction_type ORDER BY transaction_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

OK - MERGE-INTO-C17
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

JackTan25 · 2024-01-04T04:50:15Z

standalone mode passed in wizard:

Click me

Preparing to run MERGE-INTO-C1...
Executing command: bendsql --query=-- MERGE-INTO-C1: Asset Types Distribution
SELECT asset_type, COUNT(*) AS count
FROM assets
GROUP BY asset_type
ORDER BY count DESC, asset_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

Executing command: snowsql --query -- MERGE-INTO-C1: Asset Types Distribution
SELECT asset_type, COUNT(*) AS count
FROM assets
GROUP BY asset_type
ORDER BY count DESC, asset_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

OK - MERGE-INTO-C1
NEW_ASSET       150000
BTC     104817
ETH     104808
XRP     90375

Preparing to run MERGE-INTO-C2...
Executing command: bendsql --query=

-- MERGE-INTO-C2: Aggregated Quantity and Value Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       SUM(value) AS total_value,
       AVG(value) AS average_value
FROM assets -D mergeinto
Command executed successfully. Output:
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

Executing command: snowsql --query 

-- MERGE-INTO-C2: Aggregated Quantity and Value Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       SUM(value) AS total_value,
       AVG(value) AS average_value
FROM assets --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

OK - MERGE-INTO-C2
165342725.20941540      367.428278243145        1653427252.09414242     3674.282782431428

Preparing to run MERGE-INTO-C3...
Executing command: bendsql --query=

-- MERGE-INTO-C3: Assets Counts by User
SELECT user_id, COUNT(*) AS count
FROM assets
GROUP BY user_id
ORDER BY count DESC, user_id
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

Executing command: snowsql --query 

-- MERGE-INTO-C3: Assets Counts by User
SELECT user_id, COUNT(*) AS count
FROM assets
GROUP BY user_id
ORDER BY count DESC, user_id
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

OK - MERGE-INTO-C3
0       6
2       6
4       6
6       6
8       6
10      6
12      6
14      6
16      6
18      6
20      6
22      6
24      6

Preparing to run MERGE-INTO-C4...
Executing command: bendsql --query=

-- MERGE-INTO-C4: Date Range Analysis of Last Update
SELECT CASE
           WHEN last_updated < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM assets
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-12-31        410014
Before 2022     39986

Executing command: snowsql --query 

-- MERGE-INTO-C4: Date Range Analysis of Last Update
SELECT CASE
           WHEN last_updated < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM assets
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-12-31        410014
Before 2022     39986

OK - MERGE-INTO-C4
After 2021-12-31        410014
Before 2022     39986

Preparing to run MERGE-INTO-C5...
Executing command: bendsql --query=

-- MERGE-INTO-C5: General Status Distribution
SELECT status, COUNT(*) AS count
FROM orders
GROUP BY status
ORDER BY count DESC, status
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

Executing command: snowsql --query 

-- MERGE-INTO-C5: General Status Distribution
SELECT status, COUNT(*) AS count
FROM orders
GROUP BY status
ORDER BY count DESC, status
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

OK - MERGE-INTO-C5
avg     1049532
completed       350986
pending 348300
cancelled       300714
Pending 150000
above_avg       100418
below_avg       100075

Preparing to run MERGE-INTO-C6...
Executing command: bendsql --query=

-- MERGE-INTO-C6: General Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM orders -D mergeinto
Command executed successfully. Output:
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

Executing command: snowsql --query 

-- MERGE-INTO-C6: General Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM orders --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

OK - MERGE-INTO-C6
1526855542.46298600     636.183182451427        0.00005807      8910.78518320

Preparing to run MERGE-INTO-C7...
Executing command: bendsql --query=

-- MERGE-INTO-C7: New Orders vs Existing Orders Count
SELECT CASE
           WHEN order_id > 500000 THEN 'New Order'
           ELSE 'Existing Order'
           END AS order_category,
       COUNT(*) AS count
FROM orders
GROUP BY order_category
ORDER BY count DESC
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
New Order       1649995
Existing Order  750030

Executing command: snowsql --query 

-- MERGE-INTO-C7: New Orders vs Existing Orders Count
SELECT CASE
           WHEN order_id > 500000 THEN 'New Order'
           ELSE 'Existing Order'
           END AS order_category,
       COUNT(*) AS count
FROM orders
GROUP BY order_category
ORDER BY count DESC
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
New Order       1649995
Existing Order  750030

OK - MERGE-INTO-C7
New Order       1649995
Existing Order  750030

Preparing to run MERGE-INTO-C8...
Executing command: bendsql --query=

-- MERGE-INTO-C8: Order Type Distribution
SELECT order_type, COUNT(*) AS count
FROM orders
GROUP BY order_type
ORDER BY count DESC, order_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
buy     1274986
sell    1125039

Executing command: snowsql --query 

-- MERGE-INTO-C8: Order Type Distribution
SELECT order_type, COUNT(*) AS count
FROM orders
GROUP BY order_type
ORDER BY count DESC, order_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
buy     1274986
sell    1125039

OK - MERGE-INTO-C8
buy     1274986
sell    1125039

Preparing to run MERGE-INTO-C9...
Executing command: bendsql --query=

-- MERGE-INTO-C9: Date Range Analysis
SELECT CASE
           WHEN created_at < '2022-01-01' THEN 'Before 2022'
           WHEN created_at BETWEEN '2021-01-01' AND '2021-06-30' THEN 'First Half 2021'
           ELSE 'After 2021-06-30'
           END AS date_range,
       COUNT(*) AS count
FROM orders
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-06-30        150000
Before 2022     2250025

Executing command: snowsql --query 

-- MERGE-INTO-C9: Date Range Analysis
SELECT CASE
           WHEN created_at < '2022-01-01' THEN 'Before 2022'
           WHEN created_at BETWEEN '2021-01-01' AND '2021-06-30' THEN 'First Half 2021'
           ELSE 'After 2021-06-30'
           END AS date_range,
       COUNT(*) AS count
FROM orders
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-06-30        150000
Before 2022     2250025

OK - MERGE-INTO-C9
After 2021-06-30        150000
Before 2022     2250025

Preparing to run MERGE-INTO-C10...
Executing command: bendsql --query=

-- MERGE-INTO-C10: Price Analysis After Adjustments
SELECT SUM(price) AS total_price,
       AVG(price) AS average_price,
       MIN(price) AS min_price,
       MAX(price) AS max_price
FROM orders -D mergeinto
Command executed successfully. Output:
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

Executing command: snowsql --query 

-- MERGE-INTO-C10: Price Analysis After Adjustments
SELECT SUM(price) AS total_price,
       AVG(price) AS average_price,
       MIN(price) AS min_price,
       MAX(price) AS max_price
FROM orders --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

OK - MERGE-INTO-C10
2874615680.22030825     1197.744056924535       0.00058074      10990.07262787

Preparing to run MERGE-INTO-C11...
Executing command: bendsql --query=

-- MERGE-INTO-C11: Transaction Types Distribution
SELECT transaction_type, COUNT(*) AS count
FROM transactions
GROUP BY transaction_type
ORDER BY count DESC, transaction_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
trade   505920
deposit 26492
withdrawal      26129

Executing command: snowsql --query 

-- MERGE-INTO-C11: Transaction Types Distribution
SELECT transaction_type, COUNT(*) AS count
FROM transactions
GROUP BY transaction_type
ORDER BY count DESC, transaction_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
trade   505920
deposit 26492
withdrawal      26129

OK - MERGE-INTO-C11
trade   505920
deposit 26492
withdrawal      26129

Preparing to run MERGE-INTO-C12...
Executing command: bendsql --query=

-- MERGE-INTO-C12: Aggregated Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM transactions -D mergeinto
Command executed successfully. Output:
73627261.57298446       131.820692792444        0.00106899      664.06985910

Executing command: snowsql --query 

-- MERGE-INTO-C12: Aggregated Quantity Statistics
SELECT SUM(quantity) AS total_quantity,
       AVG(quantity) AS average_quantity,
       MIN(quantity) AS min_quantity,
       MAX(quantity) AS max_quantity
FROM transactions --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
73627261.57298446       131.820692792444        0.00106899      664.06985910

OK - MERGE-INTO-C12
73627261.57298446       131.820692792444        0.00106899      664.06985910

Preparing to run MERGE-INTO-C13...
Executing command: bendsql --query=

-- MERGE-INTO-C13: Transaction Counts by User and Asset Type
SELECT user_id, asset_type, COUNT(*) AS count
FROM transactions
GROUP BY user_id, asset_type
ORDER BY count DESC, user_id, asset_type
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

Executing command: snowsql --query 

-- MERGE-INTO-C13: Transaction Counts by User and Asset Type
SELECT user_id, asset_type, COUNT(*) AS count
FROM transactions
GROUP BY user_id, asset_type
ORDER BY count DESC, user_id, asset_type
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

OK - MERGE-INTO-C13
804     ETH     15
1216    ETH     15
216     BTC     14
425     ETH     14
844     BTC     14
1231    ETH     14
1539    ETH     14
1603    ETH     14
1926    ETH     14
2609    BTC     14
2704    ETH     14
2827    ETH     14
2841    BTC     14

Preparing to run MERGE-INTO-C14...
Executing command: bendsql --query=

-- MERGE-INTO-C14: Date Range Analysis of Transactions
SELECT CASE
           WHEN transaction_time < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM transactions
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 -D mergeinto
Command executed successfully. Output:
After 2021-12-31        483526
Before 2022     75015

Executing command: snowsql --query 

-- MERGE-INTO-C14: Date Range Analysis of Transactions
SELECT CASE
           WHEN transaction_time < '2022-01-01' THEN 'Before 2022'
           ELSE 'After 2021-12-31'
           END AS date_range,
       COUNT(*) AS count
FROM transactions
GROUP BY date_range
ORDER BY date_range
    LIMIT 13 --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
After 2021-12-31        483526
Before 2022     75015

OK - MERGE-INTO-C14
After 2021-12-31        483526
Before 2022     75015

Preparing to run MERGE-INTO-C15...
Executing command: bendsql --query=

-- MERGE-INTO-C15: asserts
SELECT asset_type, SUM(quantity) AS total_quantity, SUM(value) AS total_value
FROM assets
GROUP BY asset_type ORDER BY asset_type ASC -D mergeinto
Command executed successfully. Output:
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

Executing command: snowsql --query 

-- MERGE-INTO-C15: asserts
SELECT asset_type, SUM(quantity) AS total_quantity, SUM(value) AS total_value
FROM assets
GROUP BY asset_type ORDER BY asset_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

OK - MERGE-INTO-C15
BTC     54537998.34092175       545379983.40921303
ETH     50387362.73309448       503873627.33094612
NEW_ASSET       15000000.00000000       150000000.00000000
XRP     45417364.13539917       454173641.35398327

Preparing to run MERGE-INTO-C16...
Executing command: bendsql --query=

-- MERGE-INTO-C16: orders
SELECT asset_type, SUM(quantity) AS total_quantity, AVG(price) AS average_price
FROM orders
GROUP BY asset_type ORDER BY asset_type ASC -D mergeinto
Command executed successfully. Output:
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

Executing command: snowsql --query 

-- MERGE-INTO-C16: orders
SELECT asset_type, SUM(quantity) AS total_quantity, AVG(price) AS average_price
FROM orders
GROUP BY asset_type ORDER BY asset_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

OK - MERGE-INTO-C16
BTC     577143846.60937705      1321.552983291813
ETH     532400471.20223598      1228.552050777431
NEW_ORDER       7500000.00000000        500.000000000000
XRP     409811224.65137297      1172.281276438386

Preparing to run MERGE-INTO-C17...
Executing command: bendsql --query=

-- MERGE-INTO-C17: transactions
SELECT transaction_type, SUM(quantity) AS total_quantity
FROM transactions
GROUP BY transaction_type ORDER BY transaction_type ASC -D mergeinto
Command executed successfully. Output:
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

Executing command: snowsql --query 

-- MERGE-INTO-C17: transactions
SELECT transaction_type, SUM(quantity) AS total_quantity
FROM transactions
GROUP BY transaction_type ORDER BY transaction_type ASC --dbname mergeinto --schemaname PUBLIC -o output_format=tsv -o header=false -o timing=false -o friendly=false --warehouse COMPUTE_WH
Command executed successfully. Output:
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

OK - MERGE-INTO-C17
deposit 5712183.27889012
trade   62705473.53457811
withdrawal      5209604.75951623

…25/databend into mergeinto-runtime-filter

…14212)" This reverts commit 8ba2c12.

JackTan25 and others added 30 commits December 13, 2023 22:39

add optimize comment

10773ee

add join optimizer top rule

fe324c0

add distributed optmize rule

07d5bac

add change join order flag

782546a

fix not append plan

429b7b0

fix typo

78ae91f

add more cluster test

93d259d

fix test

e9e2d62

finish new distributed pipeline

481f544

Merge branch 'main' into import_upper_optimizer_for_merge_into

8f066eb

fix typo

0f1442c

fix lint

d66d5c0

fix bugs

91d044a

fix static filter bug

41b6547

fix

2d8c2ca

Merge branch 'main' into add_cluster_test

9a4aa18

fix test

afe09f2

Merge branch 'add_cluster_test' of https://github.com/JackTan25/databend

87ee165

into add_cluster_test

Merge branch 'main' into add_cluster_test

7da2cde

fix test

b3b2d9a

rollback

992daee

remove static filter

cf6a395

Merge branch 'main' into add_cluster_test

595e9c8

fix typo

e8231a6

Merge branch 'add_cluster_test' of https://github.com/JackTan25/databend

ffceddc

into add_cluster_test

Merge branch 'main' into add_cluster_test

cd88f25

Merge branch 'main' into add_cluster_test

1b1a510

resolve conflict

6c1e7de

Merge branch 'main' into import_upper_optimizer_for_merge_into

0152495

Merge branch 'main' into add_cluster_test

ae4da86

JackTan25 added 3 commits January 4, 2024 12:51

fix lint

c5492d2

Merge branch 'mergeinto-runtime-filter' of https://github.com/JackTan…

db17f40

…25/databend into mergeinto-runtime-filter

fix lint

8c5fac8

JackTan25 force-pushed the mergeinto-runtime-filter branch from 1df3c67 to 8c5fac8 Compare January 4, 2024 04:57

JackTan25 and others added 5 commits January 4, 2024 12:58

Merge branch 'main' into mergeinto-runtime-filter

74b8ec9

Merge branch 'mergeinto-runtime-filter' of https://github.com/JackTan…

590875a

…25/databend into mergeinto-runtime-filter

fix lint

8094a5d

Merge branch 'main' into mergeinto-runtime-filter

506ef66

Merge branch 'main' into mergeinto-runtime-filter

cf85c21

JackTan25 marked this pull request as ready for review January 4, 2024 05:12

JackTan25 requested review from dantengsky, SkyFan2002 and xudong963 January 4, 2024 05:16

SkyFan2002 approved these changes Jan 4, 2024

View reviewed changes

JackTan25 and others added 4 commits January 4, 2024 17:49

Merge branch 'main' into mergeinto-runtime-filter

a967281

remove static filter file

a25be69

Merge branch 'mergeinto-runtime-filter' of https://github.com/JackTan…

19c9b59

…25/databend into mergeinto-runtime-filter

Merge branch 'main' into mergeinto-runtime-filter

2e5b489

dantengsky approved these changes Jan 4, 2024

View reviewed changes

dantengsky added this pull request to the merge queue Jan 4, 2024

Merged via the queue into databendlabs:main with commit 8ba2c12 Jan 4, 2024
71 checks passed

JackTan25 added a commit that referenced this pull request Jan 6, 2024

Revert "chore: test Mergeinto runtime filter performance and wizard (#…

e4a7e87

…14212)" This reverts commit 8ba2c12.

JackTan25 mentioned this pull request Jan 6, 2024

chore: Revert "chore: test Mergeinto runtime filter performance and wizard" #14248

Closed

This was referenced Jul 14, 2024

Link Checker Report databendlabs/databend-docs#955

Closed

Link Checker Report databendlabs/databend-docs#960

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: test Mergeinto runtime filter performance and wizard #14212

chore: test Mergeinto runtime filter performance and wizard #14212

JackTan25 commented Jan 3, 2024 •

edited

Loading

github-actions bot commented Jan 3, 2024

JackTan25 commented Jan 4, 2024 •

edited

Loading

JackTan25 commented Jan 4, 2024

JackTan25 commented Jan 4, 2024 •

edited

Loading

chore: test Mergeinto runtime filter performance and wizard #14212

chore: test Mergeinto runtime filter performance and wizard #14212

Conversation

JackTan25 commented Jan 3, 2024 • edited Loading

Summary

Tests

Type of change

github-actions bot commented Jan 3, 2024

Docker Image for PR

JackTan25 commented Jan 4, 2024 • edited Loading

JackTan25 commented Jan 4, 2024

JackTan25 commented Jan 4, 2024 • edited Loading

JackTan25 commented Jan 3, 2024 •

edited

Loading

JackTan25 commented Jan 4, 2024 •

edited

Loading

JackTan25 commented Jan 4, 2024 •

edited

Loading