[Enhancement] Skip tablet schema in rowset meta during ingestion. (backport #50873) #51843
+546
−65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mergify[bot]](https://avatars.githubusercontent.com/in/10562?s=60&v=4){kind=small}
…0873) ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues: 1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB. 2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small. These two issues can eventually reduce the efficiency of real-time imports. ## What I'm doing: This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat. One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema. However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata. So I choose a more conservative solution, and the main changes are as follows: 1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.** 2. **Update the RowsetMeta that does not store schema when updating the tablet schema.** If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema. Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema. Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat. However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance. To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR. Below is a test based on this PR: a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times. | Branch | Table type |Total cost time | |----------|----------|----------| | main-8f128b | Duplicate | 1043.77(s) | | this pr | Duplicate | 178.49(s) | | main-8f128b | Primary | 188.46(s) | | this pr | Primary | 186.68(s) | Signed-off-by: sevev <[email protected]> Signed-off-by: zhangqiang <[email protected]> (cherry picked from commit 3005729) # Conflicts: # be/src/common/config.h # be/src/storage/compaction_task.h # be/src/storage/tablet.cpp # be/src/storage/tablet.h # be/src/storage/tablet_meta.cpp # be/src/storage/tablet_meta.h # be/src/storage/txn_manager.cpp
|
Cherry-pick of 3005729 has failed: To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally |
24 tasks
|
@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr |
auto-merge was automatically disabled
October 12, 2024 09:11
Pull request was closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues:
These two issues can eventually reduce the efficiency of real-time imports.
What I'm doing:
This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat.
One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema.
However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata.
So I choose a more conservative solution, and the main changes are as follows:
If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema.
Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema.
Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat.
However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance.
To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR.
Below is a test based on this PR:
a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times.
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check:
This is an automatic backport of pull request #50873 done by [Mergify](https://mergify.com). ## Why I'm doing: Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues:
These two issues can eventually reduce the efficiency of real-time imports.
What I'm doing:
This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat.
One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema.
However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata.
So I choose a more conservative solution, and the main changes are as follows:
If the BE exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema.
Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema.
Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat.
However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance.
To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR.
Below is a test based on this PR:
a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times.
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist: