Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: adjust default value of setting max_storage_io_requests #16680

Merged
merged 1 commit into from
Oct 25, 2024

Conversation

SkyFan2002
Copy link
Member

@SkyFan2002 SkyFan2002 commented Oct 24, 2024

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Using a small-sized warehouse (16c) for testing, 3,000 segments were randomly generated. Pruning was performed using different values for max_storage_io_requests. The results show that when max_storage_io_requests=64, the performance is close to optimal.

deploy@(bench-vacuum2-1)/default> set max_storage_io_requests = 16;

SET max_storage_io_requests = 16

0 row read in 0.027 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

deploy@(bench-vacuum2-1)/default> explain select * from test_order where id > 1;

EXPLAIN SELECT * FROM test_order WHERE id > 1

-[ EXPLAIN ]-----------------------------------
Filter
├── output columns: [test_order.id (#0), test_order.id1 (#1), test_order.id2 (#2), test_order.id3 (#3), test_order.id4 (#4), test_order.id5 (#5), test_order.id6 (#6), test_order.id7 (#7), test_order.s1 (#8), test_order.s2 (#9), test_order.s3 (#10), test_order.s4 (#11), test_order.s5 (#12), test_order.s6 (#13), test_order.s7 (#14), test_order.s8 (#15), test_order.s9 (#16), test_order.s10 (#17), test_order.s11 (#18), test_order.s12 (#19), test_order.s13 (#20), test_order.d1 (#21), test_order.d2 (#22), test_order.d3 (#23), test_order.d4 (#24), test_order.d5 (#25), test_order.d6 (#26), test_order.d7 (#27), test_order.d8 (#28), test_order.d9 (#29), test_order.d10 (#30), test_order.insert_time (#31), test_order.insert_time1 (#32), test_order.insert_time2 (#33), test_order.insert_time3 (#34), test_order.i (#35)]
├── filters: [is_true(test_order.id (#0) > 1)]
├── estimated rows: 1500001.08
└── TableScan
    ├── table: default.default.test_order
    ├── output columns: [id (#0), id1 (#1), id2 (#2), id3 (#3), id4 (#4), id5 (#5), id6 (#6), id7 (#7), s1 (#8), s2 (#9), s3 (#10), s4 (#11), s5 (#12), s6 (#13), s7 (#14), s8 (#15), s9 (#16), s10 (#17), s11 (#18), s12 (#19), s13 (#20), d1 (#21), d2 (#22), d3 (#23), d4 (#24), d5 (#25), d6 (#26), d7 (#27), d8 (#28), d9 (#29), d10 (#30), insert_time (#31), insert_time1 (#32), insert_time2 (#33), insert_time3 (#34), i (#35)]
    ├── read rows: 1500180
    ├── read size: 1.84 GiB
    ├── partitions total: 3000000
    ├── partitions scanned: 1500180
    ├── pruning stats: [segments: <range pruning: 3000 to 3000>, blocks: <range pruning: 3000000 to 1500180>]
    ├── push downs: [filters: [is_true(test_order.id (#0) > 1)], limit: NONE]
    └── estimated rows: 3000000.00

14 rows explain in 45.113 sec. Processed 0 rows, 0 B (0 row/s, 0 B/s)

----------------------------------------------- restart warehouse--------------------------------------------------
deploy@(bench-vacuum2-1)/default> alter warehouse 'bench-vacuum2-1' suspend;
processed in (1.085 sec)

deploy@(bench-vacuum2-1)/default> set max_storage_io_requests = 32;

SET max_storage_io_requests = 32

0 row read in 0.027 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

deploy@(bench-vacuum2-1)/default> explain select * from test_order where id > 1;

EXPLAIN SELECT * FROM test_order WHERE id > 1

-[ EXPLAIN ]-----------------------------------
Filter
├── output columns: [test_order.id (#0), test_order.id1 (#1), test_order.id2 (#2), test_order.id3 (#3), test_order.id4 (#4), test_order.id5 (#5), test_order.id6 (#6), test_order.id7 (#7), test_order.s1 (#8), test_order.s2 (#9), test_order.s3 (#10), test_order.s4 (#11), test_order.s5 (#12), test_order.s6 (#13), test_order.s7 (#14), test_order.s8 (#15), test_order.s9 (#16), test_order.s10 (#17), test_order.s11 (#18), test_order.s12 (#19), test_order.s13 (#20), test_order.d1 (#21), test_order.d2 (#22), test_order.d3 (#23), test_order.d4 (#24), test_order.d5 (#25), test_order.d6 (#26), test_order.d7 (#27), test_order.d8 (#28), test_order.d9 (#29), test_order.d10 (#30), test_order.insert_time (#31), test_order.insert_time1 (#32), test_order.insert_time2 (#33), test_order.insert_time3 (#34), test_order.i (#35)]
├── filters: [is_true(test_order.id (#0) > 1)]
├── estimated rows: 1500001.08
└── TableScan
    ├── table: default.default.test_order
    ├── output columns: [id (#0), id1 (#1), id2 (#2), id3 (#3), id4 (#4), id5 (#5), id6 (#6), id7 (#7), s1 (#8), s2 (#9), s3 (#10), s4 (#11), s5 (#12), s6 (#13), s7 (#14), s8 (#15), s9 (#16), s10 (#17), s11 (#18), s12 (#19), s13 (#20), d1 (#21), d2 (#22), d3 (#23), d4 (#24), d5 (#25), d6 (#26), d7 (#27), d8 (#28), d9 (#29), d10 (#30), insert_time (#31), insert_time1 (#32), insert_time2 (#33), insert_time3 (#34), i (#35)]
    ├── read rows: 1500180
    ├── read size: 1.84 GiB
    ├── partitions total: 3000000
    ├── partitions scanned: 1500180
    ├── pruning stats: [segments: <range pruning: 3000 to 3000>, blocks: <range pruning: 3000000 to 1500180>]
    ├── push downs: [filters: [is_true(test_order.id (#0) > 1)], limit: NONE]
    └── estimated rows: 3000000.00

14 rows explain in 32.248 sec. Processed 0 rows, 0 B (0 row/s, 0 B/s)

----------------------------------------------- restart warehouse--------------------------------------------------
deploy@(bench-vacuum2-1)/default> alter warehouse 'bench-vacuum2-1' suspend;
processed in (1.054 sec)

deploy@(bench-vacuum2-1)/default> set max_storage_io_requests = 64;

SET max_storage_io_requests = 64

0 row read in 0.027 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

deploy@(bench-vacuum2-1)/default> explain select * from test_order where id > 1;

EXPLAIN SELECT * FROM test_order WHERE id > 1

-[ EXPLAIN ]-----------------------------------
Filter
├── output columns: [test_order.id (#0), test_order.id1 (#1), test_order.id2 (#2), test_order.id3 (#3), test_order.id4 (#4), test_order.id5 (#5), test_order.id6 (#6), test_order.id7 (#7), test_order.s1 (#8), test_order.s2 (#9), test_order.s3 (#10), test_order.s4 (#11), test_order.s5 (#12), test_order.s6 (#13), test_order.s7 (#14), test_order.s8 (#15), test_order.s9 (#16), test_order.s10 (#17), test_order.s11 (#18), test_order.s12 (#19), test_order.s13 (#20), test_order.d1 (#21), test_order.d2 (#22), test_order.d3 (#23), test_order.d4 (#24), test_order.d5 (#25), test_order.d6 (#26), test_order.d7 (#27), test_order.d8 (#28), test_order.d9 (#29), test_order.d10 (#30), test_order.insert_time (#31), test_order.insert_time1 (#32), test_order.insert_time2 (#33), test_order.insert_time3 (#34), test_order.i (#35)]
├── filters: [is_true(test_order.id (#0) > 1)]
├── estimated rows: 1500001.08
└── TableScan
    ├── table: default.default.test_order
    ├── output columns: [id (#0), id1 (#1), id2 (#2), id3 (#3), id4 (#4), id5 (#5), id6 (#6), id7 (#7), s1 (#8), s2 (#9), s3 (#10), s4 (#11), s5 (#12), s6 (#13), s7 (#14), s8 (#15), s9 (#16), s10 (#17), s11 (#18), s12 (#19), s13 (#20), d1 (#21), d2 (#22), d3 (#23), d4 (#24), d5 (#25), d6 (#26), d7 (#27), d8 (#28), d9 (#29), d10 (#30), insert_time (#31), insert_time1 (#32), insert_time2 (#33), insert_time3 (#34), i (#35)]
    ├── read rows: 1500180
    ├── read size: 1.84 GiB
    ├── partitions total: 3000000
    ├── partitions scanned: 1500180
    ├── pruning stats: [segments: <range pruning: 3000 to 3000>, blocks: <range pruning: 3000000 to 1500180>]
    ├── push downs: [filters: [is_true(test_order.id (#0) > 1)], limit: NONE]
    └── estimated rows: 3000000.00

14 rows explain in 29.687 sec. Processed 0 rows, 0 B (0 row/s, 0 B/s)

----------------------------------------------- restart warehouse--------------------------------------------------
deploy@(bench-vacuum2-1)/default> alter warehouse 'bench-vacuum2-1' suspend;
processed in (1.040 sec)

deploy@(bench-vacuum2-1)/default> set max_storage_io_requests = 128;

SET max_storage_io_requests = 128

0 row read in 0.027 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)

deploy@(bench-vacuum2-1)/default> explain select * from test_order where id > 1;

EXPLAIN SELECT * FROM test_order WHERE id > 1

-[ EXPLAIN ]-----------------------------------
Filter
├── output columns: [test_order.id (#0), test_order.id1 (#1), test_order.id2 (#2), test_order.id3 (#3), test_order.id4 (#4), test_order.id5 (#5), test_order.id6 (#6), test_order.id7 (#7), test_order.s1 (#8), test_order.s2 (#9), test_order.s3 (#10), test_order.s4 (#11), test_order.s5 (#12), test_order.s6 (#13), test_order.s7 (#14), test_order.s8 (#15), test_order.s9 (#16), test_order.s10 (#17), test_order.s11 (#18), test_order.s12 (#19), test_order.s13 (#20), test_order.d1 (#21), test_order.d2 (#22), test_order.d3 (#23), test_order.d4 (#24), test_order.d5 (#25), test_order.d6 (#26), test_order.d7 (#27), test_order.d8 (#28), test_order.d9 (#29), test_order.d10 (#30), test_order.insert_time (#31), test_order.insert_time1 (#32), test_order.insert_time2 (#33), test_order.insert_time3 (#34), test_order.i (#35)]
├── filters: [is_true(test_order.id (#0) > 1)]
├── estimated rows: 1500001.08
└── TableScan
    ├── table: default.default.test_order
    ├── output columns: [id (#0), id1 (#1), id2 (#2), id3 (#3), id4 (#4), id5 (#5), id6 (#6), id7 (#7), s1 (#8), s2 (#9), s3 (#10), s4 (#11), s5 (#12), s6 (#13), s7 (#14), s8 (#15), s9 (#16), s10 (#17), s11 (#18), s12 (#19), s13 (#20), d1 (#21), d2 (#22), d3 (#23), d4 (#24), d5 (#25), d6 (#26), d7 (#27), d8 (#28), d9 (#29), d10 (#30), insert_time (#31), insert_time1 (#32), insert_time2 (#33), insert_time3 (#34), i (#35)]
    ├── read rows: 1500180
    ├── read size: 1.84 GiB
    ├── partitions total: 3000000
    ├── partitions scanned: 1500180
    ├── pruning stats: [segments: <range pruning: 3000 to 3000>, blocks: <range pruning: 3000000 to 1500180>]
    ├── push downs: [filters: [is_true(test_order.id (#0) > 1)], limit: NONE]
    └── estimated rows: 3000000.00

14 rows explain in 27.179 sec. Processed 0 rows, 0 B (0 row/s, 0 B/s)

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Oct 24, 2024
@SkyFan2002 SkyFan2002 marked this pull request as ready for review October 24, 2024 05:51
@BohuTANG
Copy link
Member

Does the bench restart the warehouse for each setting?

@SkyFan2002
Copy link
Member Author

Does the bench restart the warehouse for each setting?

Yes. Run alter warehouse 'bench-vacuum2-1' suspend; before each setting.

@BohuTANG BohuTANG merged commit ab4bcd5 into databendlabs:main Oct 25, 2024
84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-chore this PR only has small changes that no need to record, like coding styles.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants