Add schema sync on read and simplify schema syncer interface and adjust mock stuff #108
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hanfei1991
reviewed
Jul 16, 2019
| const auto merge_tree = dynamic_cast<const StorageMergeTree *>(storage.get()); | ||
| if (!merge_tree || merge_tree->getData().merging_params.mode != MergeTreeData::MergingParams::Txn) | ||
| return; | ||
| if (schema_version == DEFAULT_SCHEMA_VERSION || !merge_tree || merge_tree->getData().merging_params.mode != MergeTreeData::MergingParams::Txn) |
There was a problem hiding this comment.
DEFAULT_SCHEMA_VERSION is confusing, how about INVALID_SCHEMA_VERSION
There was a problem hiding this comment.
You are right, I named it “default” mostly because it is just some setting’s default value. But I think “invalid” sounds more like you can’t satisfy someone’s schema request, rather than someone asks you to satisfy this schema request of this specific version. How about “unspecified”? I think it might be more appropriate saying that someone didn’t ask for a schema of a specific version.
hanfei1991
approved these changes
Jul 18, 2019
|
LGTM |
zanmato1984
added a commit
that referenced
this pull request
Aug 5, 2019
* Widen schema syncer with sync all semantic, add background schema sync service and do init sync on startup * Finish widen some data types to avoid data rewriting during apply ddl * Add update_timestamp into table info * Refine read and flush to be more robust for schema change * Fix nullable for overflow check when decode region cache * Add schema state enum * Support column type change in schema syncer * Fix comment * Add mock functions for TiDB DDL * Fix rvalue ref * Fix some bugs in schema syncer and mock tidb * Adjust test * Add alter test * Not do data conversion for alter txn table, as we already stored widen data * Fix typo * Fix mock tidb reverse type mapping * Enhance overflow checking when flushing * Enhance alter test * Refine some code * Remove table region mapping when drop table * Adjust drop order * Print file path in mutable test script when error * Fix insert widen data through sql * Fix some txn tests * Fix widen data write of import statement * DDL read and update by tikv's meta (#103) * DDL read and update by tikv's meta * add dbg back * refine test * address comments * Add schema version check before reading * Fix build error * Format * Reorg schema syncer code * Add lock control in schema syncer interface * Refinement * Add sync schema on read * fix bug during creating column: use a wrong db name (#107) * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * fix ddl sync for table drop and truncate (#109) * when we drop a table when flash server is closed, we should sync this drop when restart flash server. * Add schema sync on read and simplify schema syncer interface and adjust mock stuff (#108) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Fix build error * Fix build error * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Remove useless codes (#112) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Fix build fail * Fix alter tmt and pass schema tests (#114) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * sync default value from tidb to flash (#113) - still need to support current_timestamp - still need to support date/datetime/enum types. * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * [DDL] Pass all txn tests (#116) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Fix empty default value bug (#117) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read, and add drop on read test * Remove a lot useless code (#118) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read logic and add test (#120) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read, and add drop on read test * add schema_version for TableInfo (#122) * add schema_version for TableInfo In order to make read procedure more efficiently. * address comment * pass all tests * fix bug for drop table; (#123) * Support rename mock tidb table * Add rename tests * add logs for debug (#126) * Add mock truncate table and tests * Test rename table (#124) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read, and add drop on read test * Support rename mock tidb table * Add rename tests * Add mock truncate table and tests * tiny fix (#127) * Add rename/truncate on read/write test and alter on read test * Add rename/truncate on read/write test and alter on read test (#128) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read, and add drop on read test * Support rename mock tidb table * Add rename tests * Add mock truncate table and tests * Add rename/truncate on read/write test and alter on read test * support for alter column. (#129) * support for alter column. * add log * format * address comment * Fix mis-widen internal columns * Support rename table. (#130) * Fix fragile txn mock test * Fix mis-widen internal columns (#133) * Add sync schema on read * Simplify schema syncer interface and adjust mock stuff * Rename default schema version setting * Compensate last commit * Remove curl library * Remove curl from builder image * Remove useless codes, init schema syncer based on pd config * Minor fix to schema debug * Fix alter tmt and pass tests * Fix build fail * Add lock for mock schema syncer * Fix schema sync service init context * Adjust schema tests * Not sync if no schema change detected * Adjust txn mock tests * Fix default value bug * Rename some tests * Remove sync schema test * Remove a lot useless code * Refine schema sync on read, and add drop on read test * Support rename mock tidb table * Add rename tests * Add mock truncate table and tests * Add rename/truncate on read/write test and alter on read test * Fix mis-widen internal columns * Fix fragile txn mock test again * Remove useless code * Resolve a dead lock problem during bootstraping. (#134) * Resolve a dead lock problem during bootstraping. * Merger will handle the structure lock of a storage. If merging is earlier than syncing schema when bootstraping, merger is waiting for tmt context's init, and schema syncer is waiting for the structure lock, which cause dead lock :( * add err code * make log smarter * format * Fix dead lock in learner read (#135) * Adjust order of region check and lock by region scanner creation (#136) * Fix dead lock in learner read * Adjust order of region check and lock by region scanner creation * Remove useless invalid region version check * Address comments * address comment * [FLASH-358] Support Region DeleteRange (#139) * Address comments * Fix test config based on new ddl * fix drop db bug * refine log * Fix a ignore db config bug and add verbose pd addr * Enlarge pd/kv/db startup waiting time * Remove useless commands * format client-c * Address comments * address comments * hot fix * Fix failed test * Fix mutable tests by using no schema syncer * fix. optimize background thread pool. (#153)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.