enhance: Enable node assign policy on resource group #36968

weiliu1031 · 2024-10-17T11:13:13Z

issue: #36977
with node_label_filter on resource group, user can add label on querynode with env MILVUS_COMPONENT_LABEL, then resource group will prefer to accept node which match it's node_label_filter.

then querynode's can't be group by labels, and put querynodes with same label to same resource groups.

sre-ci-robot · 2024-10-17T11:13:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: weiliu1031
To complete the pull request process, please assign jiaoew1991 after the PR has been reviewed.
You can assign the PR to them by writing /assign @jiaoew1991 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-10-17T11:15:04Z

@weiliu1031 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

mergify · 2024-10-18T03:25:24Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

internal/util/sessionutil/session_util.go

internal/querycoordv2/meta/resource_group.go

mergify · 2024-10-21T11:40:40Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

weiliu1031 · 2024-10-21T11:59:07Z

/run-cpu-e2e

mergify · 2024-10-22T02:36:58Z

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

weiliu1031 · 2024-10-22T06:10:12Z

rerun go-sdk

internal/util/sessionutil/session_util.go

internal/querycoordv2/meta/resource_group.go

internal/querycoordv2/meta/resource_manager.go

mergify · 2024-10-23T12:24:53Z

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-23T12:54:19Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-10-24T02:37:27Z

@weiliu1031 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-10-24T03:47:49Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

weiliu1031 · 2024-10-24T03:54:23Z

/run-cpu-e2e

weiliu1031 · 2024-10-31T03:26:20Z

/run-cpu-e2e

weiliu1031 · 2024-10-31T03:26:40Z

rerun go-sdk

mergify · 2024-10-31T03:34:47Z

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-31T03:45:08Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-01T02:40:09Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-01T03:04:43Z

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T03:44:57Z

@weiliu1031 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T04:18:47Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

weiliu1031 · 2024-11-04T06:01:52Z

rerun cpp-unit-test

weiliu1031 · 2024-11-04T06:02:02Z

/run-cpu-e2e

mergify · 2024-11-04T06:35:52Z

@weiliu1031 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T07:27:47Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T00:52:26Z

@weiliu1031 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T00:55:38Z

@weiliu1031 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T01:03:02Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

weiliu1031 · 2024-11-05T01:48:55Z

/run-cpu-e2e

weiliu1031 · 2024-11-05T08:53:10Z

rerun go-sdk

weiliu1031 · 2024-11-05T08:53:20Z

rerun cpp-unit-test

chyezh · 2024-11-01T05:59:40Z

internal/util/sessionutil/session_util.go

+	ret := make(map[string]string)
+	switch role {
+	case "querynode":
+		supportedLabelPrefix := paramtable.Get().QueryNodeCfg.MilvusServerLabelPrefix.GetValue()


It may be better that hard code MILVUS_SERVER_LABEL_QUERYNODE_ directly?
We can conveniently add label like MILVUS_SERVER_LABEL_STREAMINGNODE_ for streamingnode, MILVUS_SERVER_LABEL_PROXY for proxy...
It is not easy to achieve compatibility and maintenance using environment variables that can be configured by another configuration.

chyezh · 2024-11-05T09:22:47Z

internal/querycoordv2/meta/resource_group.go

+
+	if len(dirtyNodes) > 0 {
+		return len(dirtyNodes)
+	}


Should be len(dirtyNodes) + min(rg.NodeNum() - int(rg.cfg.Limits.NodeNum), 0)?

chyezh · 2024-11-05T09:26:06Z

internal/querycoordv2/meta/resource_group.go

+		name:    rg.name,
+		nodes:   rg.nodes.Clone(),
+		cfg:     rg.GetConfigCloned(),
+		nodeMgr: rg.nodeMgr,
 	}
 }

 // MeetRequirement return whether resource group meet requirement.
 // Return error with reason if not meet requirement.
 func (rg *ResourceGroup) MeetRequirement() error {
 	// if len(node) is less than requests, new node need to be assigned.


Use RedundantNumOfNodes() > 0 || MissingNumOfNodes() > 0 to keep the consistency between api.

chyezh · 2024-11-05T09:37:39Z

internal/querycoordv2/meta/resource_manager.go

@@ -576,7 +577,7 @@ func (rm *ResourceManager) selectMissingRecoverSourceRG(rg *ResourceGroup) *Reso
 	// First, Transfer node from most redundant resource group first. `len(nodes) > limits`
 	if redundantRG := rm.findMaxRGWithGivenFilter(
 		func(sourceRG *ResourceGroup) bool {
-			return rg.GetName() != sourceRG.GetName() && sourceRG.RedundantNumOfNodes() > 0
+			return rg.GetName() != sourceRG.GetName() && sourceRG.RedundantNumOfNodes() > 0 && rg.AcceptNode(sourceRG.PreferRemovedNode())


Here could be a failure logic.
In current implementation, If the rg1's PerferRemoveNode() return a node that rg2 don't accept, the rg2 can never get node from the rg1.
So the PreferRemoveNode should be a logic that passing rg1 as a argument to determine the best and stable node that move from rg2 into rg1.

I prefer that the selectMissingRecoverSourceRG return the sourceRG and the target node id together to avoid unstable node selection

chyezh · 2024-11-05T09:38:15Z

internal/querycoordv2/meta/resource_manager.go

@@ -633,7 +634,7 @@ func (rm *ResourceManager) selectRedundantRecoverTargetRG(rg *ResourceGroup) *Re
 	// First, Transfer node to most missing resource group first.
 	if missingRG := rm.findMaxRGWithGivenFilter(
 		func(targetRG *ResourceGroup) bool {
-			return rg.GetName() != targetRG.GetName() && targetRG.MissingNumOfNodes() > 0
+			return rg.GetName() != targetRG.GetName() && targetRG.MissingNumOfNodes() > 0 && targetRG.AcceptNode(rg.PreferRemovedNode())


same with selectMissingRecoverSourceRG.

Signed-off-by: Wei Liu <[email protected]>

mergify · 2024-11-06T17:21:31Z

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

weiliu1031 · 2024-11-07T00:06:18Z

/run-cpu-e2e

sre-ci-robot added the area/dependency Pull requests that update a dependency file label Oct 17, 2024

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Oct 17, 2024

sre-ci-robot requested review from congqixia and jiaoew1991 October 17, 2024 11:13

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Oct 17, 2024

mergify bot added the do-not-merge/missing-related-issue label Oct 17, 2024

weiliu1031 changed the title ~~enhance: Enable node_label_filter on resource group~~ enhance: Enable node assign policy on resource group Oct 18, 2024

mergify bot removed the do-not-merge/missing-related-issue label Oct 18, 2024

chyezh reviewed Oct 21, 2024

View reviewed changes

internal/util/sessionutil/session_util.go Outdated Show resolved Hide resolved

internal/util/sessionutil/session_util.go Outdated Show resolved Hide resolved

internal/querycoordv2/meta/resource_group.go Outdated Show resolved Hide resolved

weiliu1031 force-pushed the enable_cross_az branch 2 times, most recently from 78cf4dc to cc5ad70 Compare October 21, 2024 11:17

weiliu1031 force-pushed the enable_cross_az branch from cc5ad70 to dd7df02 Compare October 22, 2024 02:05

mergify bot added the ci-passed label Oct 22, 2024

chyezh reviewed Oct 23, 2024

View reviewed changes

weiliu1031 force-pushed the enable_cross_az branch from dd7df02 to f9e1b06 Compare October 23, 2024 11:42

mergify bot removed the ci-passed label Oct 23, 2024

weiliu1031 force-pushed the enable_cross_az branch from f9e1b06 to bbf9b69 Compare October 24, 2024 02:17

weiliu1031 force-pushed the enable_cross_az branch from 155d22f to 8c6e534 Compare November 1, 2024 02:24

weiliu1031 force-pushed the enable_cross_az branch from 8c6e534 to a5321f3 Compare November 4, 2024 03:16

weiliu1031 force-pushed the enable_cross_az branch from a5321f3 to 1bb198f Compare November 5, 2024 00:43

chyezh reviewed Nov 5, 2024

View reviewed changes

mergify bot added the ci-passed label Nov 5, 2024

weiliu1031 force-pushed the enable_cross_az branch from 1bb198f to 7cf414c Compare November 6, 2024 16:09

mergify bot removed the ci-passed label Nov 6, 2024

enhance: Enable node assign policy on resource group

cf0e01b

Signed-off-by: Wei Liu <[email protected]>

weiliu1031 force-pushed the enable_cross_az branch from 7cf414c to cf0e01b Compare November 6, 2024 16:14

mergify bot added the ci-passed label Nov 7, 2024

enhance: Enable node assign policy on resource group #36968

Are you sure you want to change the base?

enhance: Enable node assign policy on resource group #36968

Conversation

weiliu1031 commented Oct 17, 2024 • edited Loading

sre-ci-robot commented Oct 17, 2024

mergify bot commented Oct 17, 2024

mergify bot commented Oct 18, 2024

mergify bot commented Oct 21, 2024

weiliu1031 commented Oct 21, 2024

mergify bot commented Oct 22, 2024

weiliu1031 commented Oct 22, 2024

mergify bot commented Oct 23, 2024

mergify bot commented Oct 23, 2024

mergify bot commented Oct 24, 2024

mergify bot commented Oct 24, 2024

weiliu1031 commented Oct 24, 2024

weiliu1031 commented Oct 31, 2024

weiliu1031 commented Oct 31, 2024

mergify bot commented Oct 31, 2024

mergify bot commented Oct 31, 2024

mergify bot commented Nov 1, 2024

mergify bot commented Nov 1, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

weiliu1031 commented Nov 4, 2024

weiliu1031 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

weiliu1031 commented Nov 5, 2024

weiliu1031 commented Nov 5, 2024

weiliu1031 commented Nov 5, 2024

chyezh Nov 1, 2024

Choose a reason for hiding this comment

chyezh Nov 5, 2024

Choose a reason for hiding this comment

chyezh Nov 5, 2024

Choose a reason for hiding this comment

chyezh Nov 5, 2024

Choose a reason for hiding this comment

chyezh Nov 5, 2024

Choose a reason for hiding this comment

chyezh Nov 5, 2024

Choose a reason for hiding this comment

mergify bot commented Nov 6, 2024

weiliu1031 commented Nov 7, 2024

weiliu1031 commented Oct 17, 2024 •

edited

Loading