Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Milvus with streaming service make the datacoord cp metrics leak #36858

Closed
1 task done
chyezh opened this issue Oct 14, 2024 · 4 comments
Closed
1 task done

[Bug]: Milvus with streaming service make the datacoord cp metrics leak #36858

chyezh opened this issue Oct 14, 2024 · 4 comments
Assignees
@chyezh
Labels
feature/streaming node streaming node feature kind/bug Issues or changes related a bug priority/urgent Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@chyezh
Copy link
Contributor

chyezh commented Oct 14, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-d566b0ceff3ea586cea92a841a878b0479870653
- Deployment mode(standalone or cluster): 
- MQ type(rocksmq, pulsar or kafka): pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The metric milvus_datacoord_channel_checkpoint_unix_seconds should be cleared when collection is dropped, but not when enabling streaming service.

image

Expected Behavior

metric milvus_datacoord_channel_checkpoint_unix_seconds should be cleared when enabling streaming service.

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@chyezh chyezh added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 14, 2024
@chyezh
Copy link
Contributor Author

chyezh commented Oct 14, 2024

Streaming service deprecate the vchannel watch at datacoord.
So the DropVchannel is not called when DropCollection.

@yanliang567
Copy link
Contributor

/assign @chyezh
/unassign

@sre-ci-robot sre-ci-robot assigned chyezh and unassigned yanliang567 Oct 15, 2024
@yanliang567 yanliang567 added this to the 2.5.0 milestone Oct 15, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 15, 2024
@chyezh chyezh added the feature/streaming node streaming node feature label Oct 16, 2024
@chyezh chyezh mentioned this issue Oct 28, 2024
Open
@chyezh chyezh added the priority/urgent Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Oct 29, 2024
@chyezh chyezh mentioned this issue Nov 1, 2024
Closed
@chyezh
Copy link
Contributor Author

chyezh commented Nov 4, 2024

The DropCollection operation of streaming service works not properly.

This was referenced Nov 4, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 7, 2024
@chyezh
fix: drop collection failed if enable streaming service (#37444)
cae9e1c 
issue: #36858

- Start channel manager on datacoord, but with empty assign policy in
streaming service.
- Make collection at dropping state can be recovered by flusher to make
sure that
 milvus consume the dropCollection message.
- Add backoff for flusher lifetime.
- remove the proxy watcher from timetick at rootcoord in streaming
service.

Also see the better fixup: #37176

---------

Signed-off-by: chyezh <[email protected]>
@chyezh
Copy link
Contributor Author

chyezh commented Nov 7, 2024

Should be fixed.

@chyezh chyezh closed this as completed Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chyezh chyezh

Labels
feature/streaming node streaming node feature kind/bug Issues or changes related a bug priority/urgent Must be staffed and worked on either currently, or very soon, ideally in time for the next release. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
2 participants
@chyezh @yanliang567