More sort and finalize fixes #1799
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#### Reference Issues/PRs Fixes: #1753 #### What does this implement or fix? Both `finalize_staged_data` and `sort_and_finalize_staged_data` now return `VersionedItem`. `metadata` parameter was added to `sort_and_finalize_staged_data` #### Any other comments? #### Checklist <details> <summary> Checklist for code changes... </summary> - [ ] Have you updated the relevant docstrings, documentation and copyright notice? - [ ] Is this contribution tested against [all ArcticDB's features](../docs/mkdocs/docs/technical/contributing.md)? - [ ] Do all exceptions introduced raise appropriate [error messages](https://docs.arcticdb.io/error_messages/)? - [ ] Are API changes highlighted in the PR description? - [ ] Is the PR labelled as enhancement or bug so it appears in autogenerated release notes? </details> <!-- Thanks for contributing a Pull Request to ArcticDB! Please ensure you have taken a look at: - ArcticDB's Code of Conduct: https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md - ArcticDB's Contribution Licensing: https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing --> --------- Co-authored-by: Vasil Pashov <[email protected]>
…ize_staged_data instead of sort_and_finalize_staged_data
vasil-pashov
requested review from
alexowens90,
willdealtry and
poodlewars
as code owners
added 2 commits
September 3, 2024 00:53
… but different type * Update the tests to reflect how Arctic works with Pandas 1
added 11 commits
September 10, 2024 12:02
…ws_with_promotoable_types -> test_type_mismatch_in_staged_segments_throws_with_non_promotoable_types
vasil-pashov
force-pushed
the
more-sort-and-finalize-fixes
branch
from
3586eac
to
aae39b2
Compare
added 7 commits
September 11, 2024 20:56
alexowens90
requested changes
Sep 17, 2024
added 6 commits
September 17, 2024 12:41
…avoid duplication. Fix typo
willdealtry
reviewed
Sep 19, 2024
|
|
||
| ColumnInfo = namedtuple('ColumnInfo', ['name', 'dtype']) | ||
|
|
||
| COLUMN_DESCRIPTIONS = [ColumnInfo("a", "float"), ColumnInfo("b", "int64"), ColumnInfo("c", "str"), ColumnInfo("d", "datetime64[ns]")] | ||
| COLUMNS = [f"col_{i}" for i in range(0, 5)] |
There was a problem hiding this comment.
is it possible to have an unsigned type and a few more columns? Five is very narrow.
There was a problem hiding this comment.
Do you have any number on your mind? The only concern is that generating too many columns might slow the tests but we can play with it until we're happy.
alexowens90
approved these changes
Sep 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Fixes #1738
Fixes #1781
Fixes #1466
Fixes #1795
Fixes #1797
Fixes #1807
Fixes #1828
A notable change is that staged writes no longer validate the index is sorted. The validation is done at the moment compact_incompletes/finalize_staged_data/sort_and_finalize_staged_data is called. This is because sort_and_finalize_staged_data does not require the segments to be sorted, but the call for adding a staged segment is the same. We should add a separate call for that.
Note also that all incomplete keys for a symbol are deleted if any of the finalize calls fail. The other option is to leave the segments. In that case the user will have the responsibility of calling
delete_staged_data.What does this implement or fix?
Any other comments?
Checklist
Checklist for code changes...