Commits on Oct 12, 2024

1. [Enhancement] Skip tablet schema in rowset meta during ingestion. (#5… …

   …0873)
   
   ## Why I'm doing:
   Since version 3.2, Rowset has been designed to save its own schema, and the complete tablet schema is stored in the metadata. This can lead to the following issues:
   
   1. **When the import frequency is very high**, and a large number of Rowsets are generated, the metadata in RocksDB grows, particularly in non-primary key (non-PK) table scenarios. This is because, in non-PK tables, each update to the tablet metadata rewrites the historical Rowset metadata, leading to a large amount of obsolete data in RocksDB.
   2. **When the tablet has a very large number of columns (e.g., 10,000 columns)**, the time taken to persist the Rowset metadata increases, especially when the imported data volume is small.
   
   These two issues can eventually reduce the efficiency of real-time imports.
   
   ## What I'm doing:
   This PR attempts to solve the issue of reduced import efficiency caused by metadata bloat.
   
   One feasible solution is to store the tablet schema only once for all Rowsets that share the same schema. Instead of saving the complete schema in each Rowset's metadata, a reference or marker would be saved in each Rowset’s metadata to point to the corresponding tablet schema.
   
   However, the issue with this solution is compatibility. In previous versions, each Rowset generated its corresponding schema based on its own metadata. If the system is upgraded and then rollback to an older version, the older version would not be able to locate the schema using a reference or marker. This would lead to the generation of incorrect schemas, as the previous versions expect the full schema to be included in each Rowset's metadata.
   
   So I choose a more conservative solution, and the main changes are as follows:
   1. **Skip the schema in Rowset meta during import if the Rowset's schema is identical to the latest tablet schema.**
   2. **Update the RowsetMeta that does not store schema when updating the tablet schema.**
   
   If the BE  exits at any given time and restarts, those Rowsets that have not saved their own schema will be initialized using the tablet's current schema. Since the Rowset meta without schemas is updated each time the tablet schema is modified, it ensures that after the BE restarts, every Rowset can find its corresponding schema.
   
   Moreover, this logic is backward compatible with older versions, so even after an upgrade and subsequent downgrade, the BE will still be able to retrieve the correct schema.
   
   Compared to imports, DDL operations can be considered low-frequency tasks. As a result, in most cases, the Rowset meta generated during imports will not carry the schema, which helps alleviate metadata bloat.
   
   However, there can still be some bad cases. For example, in non-PK tables, during the period between an alter operation and the deletion of outdated Rowset meta, if the number of outdated Rowsets is particularly large, the system will still rewrite all outdated Rowsets each time the tablet meta is saved. This can still lead to a decline in import performance.
   
   To solve this issue, we need to resolve the problem of storing multiple copies of the same schema. I think we can first support downgrading and then resolve this issue, allowing for an iteration based on this PR.
   
   Below is a test based on this PR:
   a table with 200 columns, one bucket, writing one row of data at a time, with 10 concurrent threads, executed 1,000 times.
   
   | Branch | Table type |Total cost time |
   |----------|----------|----------|
   |  main-8f128b  | Duplicate  | 1043.77(s)  |
   |  this pr  |  Duplicate  | 178.49(s)  |
   |  main-8f128b  | Primary  | 188.46(s)  |
   |  this pr  |  Primary  | 186.68(s)  |
   
   Signed-off-by: sevev <[email protected]>
   Signed-off-by: zhangqiang <[email protected]>
   (cherry picked from commit 3005729)
   
   # Conflicts:
   #	be/src/common/config.h
   #	be/src/storage/compaction_task.h
   #	be/src/storage/tablet.cpp
   #	be/src/storage/tablet.h
   #	be/src/storage/tablet_meta.cpp
   #	be/src/storage/tablet_meta.h
   #	be/src/storage/txn_manager.cpp

   

   sevev authored and mergify[bot] committed Oct 12, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for efed819
   • Browse repository at this point

   Copy the full SHA

   efed819 View commit details

   Browse the repository at this point in the history