[Bug]: [Nightly] Load collection failed reporting node not found

[NicoYuan1986]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: e6cb7d6
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):     kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load collection failed reporting node not found.

[pytest : test] [2024-10-21 20:12:42 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[pytest : test] [2024-10-21 20:13:15 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[pytest : test] /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[pytest : test] /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[pytest : test] /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
[pytest : test] /workspace/source/internal/distributed/querynode/client/client.go:173 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:178 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadPartitions.func1
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:175 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/job/utils.go:105 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/job/job_load.go:188 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute: node not found)>, <Time:{'RPC start': '2024-10-21 20:12:42.841134', 'RPC error': '2024-10-21 20:13:15.470586'}> (decorators.py:140)
[pytest : test] [2024-10-21 20:13:15 - ERROR - ci_test]: Traceback (most recent call last):
[pytest : test]   File "/milvus/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[pytest : test]     res = func(*args, **_kwargs)
[pytest : test]   File "/milvus/tests/python_client/utils/api_request.py", line 63, in api_request
[pytest : test]     return func(*arg, **kwargs)
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 429, in load
[pytest : test]     conn.load_collection(
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 141, in handler
[pytest : test]     raise e from e
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 137, in handler
[pytest : test]     return func(*args, **kwargs)
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 176, in handler
[pytest : test]     return func(self, *args, **kwargs)
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 116, in handler
[pytest : test]     raise e from e
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
[pytest : test]     return func(*args, **kwargs)
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1171, in load_collection
[pytest : test]     check_status(response)
[pytest : test]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 63, in check_status
[pytest : test]     raise MilvusException(status.code, status.reason, status.error_code)
[pytest : test] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[pytest : test] /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[pytest : test] /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[pytest : test] /workspace/source/internal/distributed/querynode/client/client.go:90 github.com/milvus-io/milvus/internal/distributed/querynode/client.wrapGrpcCall[...]
[pytest : test] /workspace/source/internal/distributed/querynode/client/client.go:173 github.com/milvus-io/milvus/internal/distributed/querynode/client.(*Client).LoadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:178 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadPartitions.func1
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:271 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).send
[pytest : test] /workspace/source/internal/querycoordv2/session/cluster.go:175 github.com/milvus-io/milvus/internal/querycoordv2/session.(*QueryCluster).LoadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/job/utils.go:105 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[pytest : test] /workspace/source/internal/querycoordv2/job/job_load.go:188 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute: node not found)>
[pytest : test]  (api_request.py:45)
[pytest : test] [2024-10-21 20:13:15 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=call query coordinator LoadCollection: stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[pytest : test] /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*...... (api_request.py:46)

Expected Behavior

load successfully

Steps To Reproduce

No response

Milvus Log

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI(new)/detail/master/155/pipeline/153
2. log: artifacts-milvus-distributed-kafka-mdk-master-155-py-n-155-e2e-logs.tar.gz
3. collection name: query_P9s5r4hl
4. failed time: [pytest : test] [gw2] [ 48%] FAILED testcases/test_query.py::TestQueryParams::test_query_expr_json_contains_combined_with_normal[False-json_contains]

Anything else?

No response

[yanliang567]

/assign @shaoting-huang
/unassign

[sre-ci-robot]

@yanliang567: GitHub didn't allow me to assign the following users: shaoting-huang.

Note that only milvus-io members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @shaoting-huang
/unassign

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[congqixia]

Querynode panicked during the nightly run.
The related code has been updated during last two weeks. Could please check recent nightly run plz?
/unassign
/assign @NicoYuan1986

[NicoYuan1986]

fixed. master-20241106-8275e40f