Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rebalance lazy compression levels #2934

Merged
merged 6 commits into from
Dec 20, 2021
Merged

rebalance lazy compression levels #2934

merged 6 commits into from
Dec 20, 2021

Conversation

Cyan4973
Copy link
Contributor

@Cyan4973 Cyan4973 commented Dec 16, 2021

This is a proposed update to rebalance compression levels.
It makes levels stronger than then used to be in v1.5.0, and closer to (though generally still faster than) v1.4.9, which achieves a more gradual scale towards level 13 (the first btlazy2 level).

It's still difficult to create a "good" level 12, because rowhash is just very fast, so there's only so much that can be slowed down by increasing parameters to the max.

note: having looked at 256K table, there is no obvious update, the current table already tries to be balanced, and there is still a large gap between strongest lazy2 and fastest btlazy2, simply because lazy2 is very fast, even when pushed to its maximum.

the only value which makes sense is `hashlog-1`
as it mimics the real memory usage.
@Cyan4973
Copy link
Contributor Author

arf, tons of updates to do in the regression test ... :(

@Cyan4973
Copy link
Contributor Author

Cyan4973 commented Dec 20, 2021

After extensive testings on x64, x86 and aarch64, this new map of compression levels produces a more "regular" speed / compression ratio curve on these systems (than the map currently defined in dev).

Copy link
Contributor

@felixhandte felixhandte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed my concerns about memory monotonicity offline. Looks good to me.

This is definitely more regular:

Screenshot from 2021-12-20 14-32-00

@Cyan4973 Cyan4973 merged commit 85bd1e6 into dev Dec 20, 2021
@Cyan4973 Cyan4973 deleted the lazy_rebalance branch January 13, 2023 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants