Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: refactor the drop collection and drop partition #37348

Closed

Conversation

chyezh
Copy link
Contributor

@chyezh chyezh commented Nov 1, 2024

issue: #36858
pr: #37176

  • Remove DropVirtualChannel when DropCollection from datanode and deprecate the rpc.
  • Add DropCollection and DropPartition rpc for datacoord, and call it with stepTask when DropCollection and DropPartition happens.
  • Add CollectionTombstone at datacoord to store the tombstone of collection or partition.
  • Segment Meta modification will check the tombstone of collection or partition that segment belongs to before the modification to avoid resource leak when the collection or partition is dropped.

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: chyezh
To complete the pull request process, please assign wxyucs after the PR has been reviewed.
You can assign the PR to them by writing /assign @wxyucs in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@chyezh chyezh changed the title Fix streaming drop collection 2 5 enhance: refactor the drop collection and drop partition Nov 1, 2024
@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Nov 1, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Nov 1, 2024
Copy link

codecov bot commented Nov 1, 2024

Codecov Report

Attention: Patch coverage is 81.38223% with 132 lines in your changes missing coverage. Please review.

Project coverage is 80.85%. Comparing base (6b9b699) to head (c0c72be).
Report is 11 commits behind head on 2.5.

Files with missing lines Patch % Lines
internal/datacoord/services.go 50.79% 23 Missing and 8 partials ⚠️
internal/datacoord/meta.go 70.37% 17 Missing and 7 partials ⚠️
internal/datacoord/tombstone/collection.go 85.43% 16 Missing and 6 partials ⚠️
internal/rootcoord/garbage_collector.go 65.62% 9 Missing and 2 partials ⚠️
internal/datacoord/tombstone/singleton.go 52.94% 5 Missing and 3 partials ⚠️
internal/rootcoord/create_collection_task.go 75.86% 6 Missing and 1 partial ⚠️
internal/metastore/model/collection_tombstone.go 82.85% 4 Missing and 2 partials ⚠️
internal/datacoord/channel_manager.go 86.66% 3 Missing and 1 partial ⚠️
internal/datacoord/import_checker.go 76.92% 1 Missing and 2 partials ⚠️
internal/datacoord/server.go 70.00% 2 Missing and 1 partial ⚠️
... and 5 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              2.5   #37348      +/-   ##
==========================================
- Coverage   83.10%   80.85%   -2.26%     
==========================================
  Files        1031     1325     +294     
  Lines      157671   183467   +25796     
==========================================
+ Hits       131034   148341   +17307     
- Misses      21443    29897    +8454     
- Partials     5194     5229      +35     
Components Coverage Δ
Client ∅ <ø> (∅)
Core 66.93% <ø> (∅)
Go 83.11% <81.38%> (-0.02%) ⬇️
Files with missing lines Coverage Δ
internal/datacoord/channel_manager_factory.go 100.00% <100.00%> (ø)
internal/datacoord/compaction.go 72.52% <ø> (+0.39%) ⬆️
internal/datacoord/policy.go 85.71% <100.00%> (+0.16%) ⬆️
internal/datanode/data_node.go 81.46% <ø> (+3.68%) ⬆️
internal/distributed/datacoord/client/client.go 97.08% <100.00%> (+0.05%) ⬆️
internal/distributed/datacoord/service.go 91.24% <100.00%> (+0.87%) ⬆️
...rnal/flushcommon/pipeline/flow_graph_write_node.go 80.41% <ø> (-0.59%) ⬇️
internal/flushcommon/syncmgr/meta_writer.go 83.16% <ø> (+4.00%) ⬆️
internal/flushcommon/writebuffer/manager.go 96.15% <100.00%> (+2.07%) ⬆️
internal/flushcommon/writebuffer/write_buffer.go 74.00% <ø> (+3.62%) ⬆️
... and 24 more

... and 310 files with indirect coverage changes

@mergify mergify bot added the ci-passed label Nov 1, 2024
@chyezh
Copy link
Contributor Author

chyezh commented Nov 4, 2024

2.5 dropped

@chyezh chyezh closed this Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants