Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster] DQL RT bursts under long-term(164h) concurrent DQL(load/query/search) #36677

Open
1 task done
wangting0128 opened this issue Oct 8, 2024 · 2 comments
Open
1 task done

[Bug]: [benchmark][cluster] DQL RT bursts under long-term(164h) concurrent DQL(load/query/search) #36677

wangting0128 opened this issue Oct 8, 2024 · 2 comments
Assignees
@bigsheeper
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20240927-1f271e39-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-ks9wq
test case name: test_concurrent_locust_1b_ivf_sq8_ddl_dql_cluster

server:

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
holiday-1b-ddl-dql-etcd-0                                         1/1     Running            0                 9d      10.104.21.145   4am-node24   <none>           <none>
holiday-1b-ddl-dql-etcd-1                                         1/1     Running            0                 9d      10.104.18.145   4am-node25   <none>           <none>
holiday-1b-ddl-dql-etcd-2                                         1/1     Running            0                 9d      10.104.30.105   4am-node38   <none>           <none>
holiday-1b-ddl-dql-milvus-datanode-56457dd5b-blz5d                1/1     Running            0                 9d      10.104.23.176   4am-node27   <none>           <none>
holiday-1b-ddl-dql-milvus-indexnode-74d586475d-l2hzv              1/1     Running            1 (10d ago)       10d     10.104.15.108   4am-node20   <none>           <none>
holiday-1b-ddl-dql-milvus-mixcoord-76cdc795dc-c24nt               1/1     Running            1 (10d ago)       10d     10.104.21.26    4am-node24   <none>           <none>
holiday-1b-ddl-dql-milvus-proxy-7b57cff57f-9p6hx                  1/1     Running            1 (10d ago)       10d     10.104.15.107   4am-node20   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-48pfv              1/1     Running            1 (10d ago)       10d     10.104.16.84    4am-node21   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-5n8bm              1/1     Running            1 (10d ago)       10d     10.104.21.27    4am-node24   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-6rqdj              1/1     Running            2 (10d ago)       10d     10.104.25.90    4am-node30   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-96pml              1/1     Running            3 (10d ago)       10d     10.104.23.118   4am-node27   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-p96kw              1/1     Running            1 (10d ago)       10d     10.104.14.108   4am-node18   <none>           <none>
holiday-1b-ddl-dql-milvus-querynode-58554c58cd-rz799              1/1     Running            1 (10d ago)       10d     10.104.32.194   4am-node39   <none>           <none>
holiday-1b-ddl-dql-minio-0                                        1/1     Running            0                 10d     10.104.18.240   4am-node25   <none>           <none>
holiday-1b-ddl-dql-minio-1                                        1/1     Running            0                 10d     10.104.30.94    4am-node38   <none>           <none>
holiday-1b-ddl-dql-minio-2                                        1/1     Running            0                 10d     10.104.20.244   4am-node22   <none>           <none>
holiday-1b-ddl-dql-minio-3                                        1/1     Running            0                 10d     10.104.21.37    4am-node24   <none>           <none>
holiday-1b-ddl-dql-pulsar-bookie-0                                1/1     Running            0                 10d     10.104.21.35    4am-node24   <none>           <none>
holiday-1b-ddl-dql-pulsar-bookie-1                                1/1     Running            0                 10d     10.104.18.245   4am-node25   <none>           <none>
holiday-1b-ddl-dql-pulsar-bookie-2                                1/1     Running            0                 10d     10.104.20.247   4am-node22   <none>           <none>
holiday-1b-ddl-dql-pulsar-broker-0                                1/1     Running            0                 10d     10.104.1.95     4am-node10   <none>           <none>
holiday-1b-ddl-dql-pulsar-proxy-0                                 1/1     Running            0                 10d     10.104.9.60     4am-node14   <none>           <none>
holiday-1b-ddl-dql-pulsar-recovery-0                              1/1     Running            0                 10d     10.104.30.90    4am-node38   <none>           <none>
holiday-1b-ddl-dql-pulsar-zookeeper-0                             1/1     Running            0                 10d     10.104.18.239   4am-node25   <none>           <none>
holiday-1b-ddl-dql-pulsar-zookeeper-1                             1/1     Running            0                 10d     10.104.17.114   4am-node23   <none>           <none>
holiday-1b-ddl-dql-pulsar-zookeeper-2                             1/1     Running            0                 10d     10.104.34.50    4am-node37   <none>           <none>

holiday-1b-ddl-dql-milvus-proxy-7b57cff57f-9p6hx_warn.log

The average RT of search and query is less than 7 seconds, but it suddenly bursts to 60 seconds during the process, causing timeouts
截屏2024-10-08 12 09 52

截屏2024-10-08 12 12 26 截屏2024-10-08 12 12 57 截屏2024-10-08 12 13 12

client log:

[2024-10-04 12:38:26,689 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.670437', 'gRPC error': '2024-10-04 12:38:26.689490'}> (decorators.py:157)
[2024-10-04 12:38:26,693 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:26.686743923+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 698cb00c-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,693 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:26.686743923+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 12:38:26,695 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.676897', 'gRPC error': '2024-10-04 12:38:26.695973'}> (decorators.py:157)
[2024-10-04 12:38:26,696 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686788674+00:00"}"
>, [requestId: 698dad68-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,696 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686788674+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:26,696 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.672007', 'gRPC error': '2024-10-04 12:38:26.696720'}> (decorators.py:157)
[2024-10-04 12:38:26,697 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686752439+00:00"}"
>, [requestId: 698cef18-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,697 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686752439+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:26,697 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.673472', 'gRPC error': '2024-10-04 12:38:26.697383'}> (decorators.py:157)
[2024-10-04 12:38:26,697 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686764696+00:00"}"
>, [requestId: 698d2744-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,697 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686764696+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:26,698 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.675114', 'gRPC error': '2024-10-04 12:38:26.698374'}> (decorators.py:157)
[2024-10-04 12:38:26,698 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:26.68677603+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 698d69e8-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,698 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:26.68677603+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 12:38:26,699 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:26.666737', 'gRPC error': '2024-10-04 12:38:26.699104'}> (decorators.py:157)
[2024-10-04 12:38:26,699 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686744326+00:00"}"
>, [requestId: 698c20e2-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:26,699 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:26.686744326+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:27,181 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.180572', 'gRPC error': '2024-10-04 12:38:27.181500'}> (decorators.py:157)
[2024-10-04 12:38:27,182 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.181071817+00:00"}"
>, [requestId: 69da83a4-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:27,182 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.181071817+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:27,184 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.183112', 'gRPC error': '2024-10-04 12:38:27.184220'}> (decorators.py:157)
[2024-10-04 12:38:27,184 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.183981436+00:00"}"
>, [requestId: 69daed8a-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:27,184 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.183981436+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:27,380 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.379575', 'gRPC error': '2024-10-04 12:38:27.380485'}> (decorators.py:157)
[2024-10-04 12:38:27,381 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.380033907+00:00"}"
>, [requestId: 69f8e54c-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:27,381 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.380033907+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:27,383 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.381950', 'gRPC error': '2024-10-04 12:38:27.383287'}> (decorators.py:157)
[2024-10-04 12:38:27,383 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.383041037+00:00"}"
>, [requestId: 69f94424-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:27,383 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T12:38:27.383041037+00:00"}"
> (func_check.py:101)
[2024-10-04 12:38:28,379 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:28.378161', 'gRPC error': '2024-10-04 12:38:28.379487'}> (decorators.py:157)
[2024-10-04 12:38:28,380 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:28.379027432+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 6a9140da-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:28,380 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:28.379027432+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 12:38:28,383 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:28.380491', 'gRPC error': '2024-10-04 12:38:28.383598'}> (decorators.py:157)
[2024-10-04 12:38:28,384 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:28.380980352+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 6a91a16a-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:38:28,384 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:28.380980352+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 12:39:14,138 - ERROR - fouram]: grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.181200', 'gRPC error': '2024-10-04 12:39:14.138485'}> (decorators.py:157)
[2024-10-04 12:39:14,139 - ERROR - fouram]: (api_response) : [Collection.query] <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:27.182031796+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 69dabf90-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:39:14,139 - ERROR - fouram]: [CheckFunc] query request check failed, response:<_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:27.182031796+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 12:39:16,197 - ERROR - fouram]: grpc RpcError: [query], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 12:37:27.382764', 'gRPC error': '2024-10-04 12:39:16.197671'}> (decorators.py:157)
[2024-10-04 12:39:16,198 - ERROR - fouram]: (api_response) : [Collection.query] <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:27.383055825+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: 69f981fa-824d-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 12:39:16,198 - ERROR - fouram]: [CheckFunc] query request check failed, response:<_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T12:38:27.383055825+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:43,990 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:43.989329', 'gRPC error': '2024-10-04 17:05:43.990587'}> (decorators.py:157)
[2024-10-04 17:05:43,991 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.990099235+00:00"}"
>, [requestId: c08957fa-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:43,991 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.990099235+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:43,995 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:43.994518', 'gRPC error': '2024-10-04 17:05:43.995394'}> (decorators.py:157)
[2024-10-04 17:05:43,995 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.995035511+00:00"}"
>, [requestId: c08a21bc-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:43,995 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.995035511+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:43,998 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:43.997027', 'gRPC error': '2024-10-04 17:05:43.998616'}> (decorators.py:157)
[2024-10-04 17:05:43,998 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.99802254+00:00"}"
>, [requestId: c08a7270-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:43,999 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.99802254+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,000 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:43.999348', 'gRPC error': '2024-10-04 17:05:44.000843'}> (decorators.py:157)
[2024-10-04 17:05:44,001 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.999967955+00:00"}"
>, [requestId: c08ad756-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,001 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:43.999967955+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,003 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.000942', 'gRPC error': '2024-10-04 17:05:44.003113'}> (decorators.py:157)
[2024-10-04 17:05:44,004 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.001984549+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: c08b2148-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,004 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.001984549+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:44,006 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.002533', 'gRPC error': '2024-10-04 17:05:44.006161'}> (decorators.py:157)
[2024-10-04 17:05:44,006 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.002969592+00:00"}"
>, [requestId: c08b5c94-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,006 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.002969592+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,008 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.004945', 'gRPC error': '2024-10-04 17:05:44.008403'}> (decorators.py:157)
[2024-10-04 17:05:44,008 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.005969411+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: c08bbdec-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,008 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.005969411+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:44,176 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.174942', 'gRPC error': '2024-10-04 17:05:44.176450'}> (decorators.py:157)
[2024-10-04 17:05:44,177 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.176011579+00:00"}"
>, [requestId: c0a5aafe-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,177 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.176011579+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,387 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.385383', 'gRPC error': '2024-10-04 17:05:44.387542'}> (decorators.py:157)
[2024-10-04 17:05:44,388 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.386891472+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: c0c5aa98-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,388 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.386891472+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:44,395 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.388600', 'gRPC error': '2024-10-04 17:05:44.394950'}> (decorators.py:157)
[2024-10-04 17:05:44,395 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.390070764+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: c0c629aa-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,395 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.390070764+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:44,598 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.596490', 'gRPC error': '2024-10-04 17:05:44.598790'}> (decorators.py:157)
[2024-10-04 17:05:44,599 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.598102937+00:00"}"
>, [requestId: c0e5d94e-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,599 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.598102937+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,604 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.599658', 'gRPC error': '2024-10-04 17:05:44.604056'}> (decorators.py:157)
[2024-10-04 17:05:44,604 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.601080039+00:00"}"
>, [requestId: c0e65932-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,605 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:44.601080039+00:00"}"
> (func_check.py:101)
[2024-10-04 17:05:44,608 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:44.603162', 'gRPC error': '2024-10-04 17:05:44.608037'}> (decorators.py:157)
[2024-10-04 17:05:44,608 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.604054347+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
>, [requestId: c0e6e1f4-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:44,608 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-10-04T17:05:44.604054347+00:00", grpc_status:4, grpc_message:"Deadline Exceeded"}"
> (func_check.py:101)
[2024-10-04 17:05:45,279 - ERROR - fouram]: grpc RpcError: [search], <_InactiveRpcError: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-10-04 17:04:45.278497', 'gRPC error': '2024-10-04 17:05:45.279501'}> (decorators.py:157)
[2024-10-04 17:05:45,280 - ERROR - fouram]: (api_response) : [Collection.search] <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:45.279067207+00:00"}"
>, [requestId: c14e0cd0-8272-11ef-8233-b2da2819070d] (api_request.py:57)
[2024-10-04 17:05:45,280 - ERROR - fouram]: [CheckFunc] search request check failed, response:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"Deadline Exceeded", grpc_status:4, created_time:"2024-10-04T17:05:45.279067207+00:00"}"
> (func_check.py:101)

Expected Behavior

No response

Steps To Reproduce

1. create a collection with fields: id(INT64 primary key), float_vector(128dim, sift)
2. build index IVF_SQ8
3. insert 1 billion data
4. flush collection
5. build index with the same params
6. load collection
7. concurrent request: <- conurrent_number=20
   - load
   - query  <- request timeout
   - search <- request timeout

Milvus Log

No response

Anything else?

test result:

[2024-10-06 23:09:18,213 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-10-06 23:09:18,214 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]: grpc     load                                                                          219886     0(0.00%) |   6713      11  108357   6700 |    0.36        0.00 (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]: grpc     query                                                                         439879     2(0.00%) |   5244     807  108815   5200 |    0.73        0.00 (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]: grpc     search                                                                       4400570    26(0.00%) |   1886     218   60033   1900 |    7.28        0.00 (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]:          Aggregated                                                                   5060335    28(0.00%) |   2387      11  108815   1900 |    8.37        0.00 (stats.py:789)
[2024-10-06 23:09:18,214 -  INFO - fouram]:  (stats.py:790)
[2024-10-06 23:09:18,215 -  INFO - fouram]: [PerfTemplate] Report data:
{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'holiday-1b-ddl-dql-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_1b_ivf_sq8_ddl_dql_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2', 'dim': 128, 'dataset_name': 'sift', 'dataset_size': 1000000000, 'ni_per': 50000},
                                 'collection_params': {'other_fields': [], 'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_SQ8', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '168h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 20,
                                                       'params': {'nq': 10,
                                                                  'top_k': 10,
                                                                  'search_param': {'nprobe': 16},
                                                                  'timeout': 60,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response'}},
                                                      {'type': 'query',
                                                       'weight': 2,
                                                       'params': {'expr': 'id in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]',
                                                                  'timeout': 60,
                                                                  'check_task': 'check_response'}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'replica_number': 1, 'timeout': 30, 'check_task': 'check_response', 'check_items': None}}]},
            'run_id': 2024092996215544,
            'datetime': '2024-09-29 03:13:41.190357',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 28655.8958},
                            'insert': {'total_time': 39349.7288, 'VPS': 25413.1358, 'batch_time': 1.9675, 'batch': 50000},
                            'flush': {'RT': 2.5391},
                            'load': {'RT': 101.5247},
                            'Locust': {'Aggregated': {'Requests': 5060335,
                                                      'Fails': 28,
                                                      'RPS': 8.37,
                                                      'fail_s': 0.0,
                                                      'RT_max': 108815.15,
                                                      'RT_avg': 2387.9,
                                                      'TP50': 1900.0,
                                                      'TP99': 8000.0},
                                       'load': {'Requests': 219886,
                                                'Fails': 0,
                                                'RPS': 0.36,
                                                'fail_s': 0.0,
                                                'RT_max': 108357.43,
                                                'RT_avg': 6713.11,
                                                'TP50': 6700.0,
                                                'TP99': 11000.0},
                                       'query': {'Requests': 439879,
                                                 'Fails': 2,
                                                 'RPS': 0.73,
                                                 'fail_s': 0.0,
                                                 'RT_max': 108815.15,
                                                 'RT_avg': 5244.92,
                                                 'TP50': 5200.0,
                                                 'TP99': 8300.0},
                                       'search': {'Requests': 4400570,
                                                  'Fails': 26,
                                                  'RPS': 7.28,
                                                  'fail_s': 0.0,
                                                  'RT_max': 60033.35,
                                                  'RT_avg': 1886.19,
                                                  'TP50': 1900.0,
                                                  'TP99': 2900.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Oct 8, 2024
@wangting0128 wangting0128 added this to the 2.5.0 milestone Oct 8, 2024
@yanliang567
Copy link
Contributor

/assign @weiliu1031
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 8, 2024
@czs007 czs007 assigned bigsheeper and unassigned weiliu1031 Nov 5, 2024
@wangting0128
Copy link
Contributor Author

Rerunning, trying to reproduce the problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bigsheeper bigsheeper

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
4 participants
@wangting0128 @bigsheeper @yanliang567 @weiliu1031