Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Rootcoord tt delay metrics remains after node down #30832

Closed
1 task done
congqixia opened this issue Feb 26, 2024 · 0 comments
Closed
1 task done

[Bug]: Rootcoord tt delay metrics remains after node down #30832

congqixia opened this issue Feb 26, 2024 · 0 comments
Assignees
@congqixia
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@congqixia
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: e1e87d5 & 2.3.10
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When querynode/datanode go offline, the timetick delay metrics of rootcoord remains

Expected Behavior

tt delay metric shall be removed after querynode/ datanode go offline

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@congqixia congqixia added kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Feb 26, 2024
@congqixia congqixia self-assigned this Feb 26, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Feb 26, 2024
@congqixia
fix: Remove time tick delay metrics when nodes go offline
0060668 
See also milvus-io#30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Feb 26, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 28, 2024
@congqixia
fix: Remove time tick delay metrics when nodes go offline (#30833)
af31553 
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <[email protected]>
Signed-off-by: Congqi.Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Feb 28, 2024
@congqixia
fix: Remove time tick delay metrics when nodes go offline (milvus-io#…
4ce782c 
…30833)

See also milvus-io#30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <[email protected]>
Signed-off-by: Congqi.Xia <[email protected]>
@congqixia congqixia mentioned this issue Feb 28, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 28, 2024
@congqixia
fix: [Cherry-pick] Remove time tick delay metrics when nodes go offli…
df16bf6 
…ne (#30833) (#30879)

Cherry-pick from master
pr: #30833
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

---------

Signed-off-by: Congqi Xia <[email protected]>
Signed-off-by: Congqi.Xia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@congqixia congqixia

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@congqixia