[Bug]: query failed: Assert "index < this->counter_" => index out of range, index=0, counter_=0

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20241013-44564f04-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

milvus server

deploy a cluster with image 2.4-20241013-44564f04-amd64

• with config

      config:
        queryNode:
          levelZeroForwardPolicy: RemoteLoad
        rootCoord:
          enableActiveStandby: true
        queryCoord:
          enableActiveStandby: true
        indexCoord:
          enableActiveStandby: true
        dataCoord:
          enableActiveStandby: true
          segment:
            enableLevelZero: true
        log:
          level: debug
        trace:
          exporter: jaeger
          sampleFraction: 1
          jaeger:
            url: http://tempo-distributor.tempo:14268/api/traces

test steps

1. create a collection with a int64-pk field and a 128dim vector field
2. create hnsw index
3. insert 10m entities with random pk values
4. flush
5. index again -> load
6. concurrent search + query + upsert

'concurrent_params': {'concurrent_number': 30, 'during_time': '5h', 'interval': 60, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 15,
                                                       'params': {'nq': 100,
                                                                  'top_k': 100,
                                                                  'search_param': {'ef': 120},
                                                                  'expr': 'id >= 10',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': None,
                                                                  'output_fields': ['id'],
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 120,
                                                                  'random_data': True,
                                                                  'check_task': 'check_search_output',
                                                                  'check_items': {'nq': 100}}},
                                                      {'type': 'query',
                                                       'weight': 10,
                                                       'params': {'ids': None,
                                                                  'expr': '',
                                                                  'output_fields': None,
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': None,
                                                                  'timeout': 120,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 500,
                                                                  'random_range': [0, 20000000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'upsert',
                                                       'weight': 5,
                                                       'params': {'nb': 300,
                                                                  'timeout': 60,
                                                                  'random_id': True,
                                                                  'random_vector': True,
                                                                  'varchar_filled': False,
                                                                  'start_id': 0,
                                                                  'shuffle_id': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},

results

Only one search requests failed: with query failed: Assert "index < this->counter_" => index out of range, index=0, counter_=0 at /workspace/source/internal/core/src/mmap/ChunkVector.h:12

[2024-10-14 12:57:08,548 - ERROR - fouram]: RPC error: [search], <MilvusException: (code=65535, message=fail to search on QueryNode 2: worker(2) query failed: Assert "index < this->counter_"  => index out of range, index=0, counter_=0 at /workspace/source/internal/core/src/mmap/ChunkVector.h:126
)>, <Time:{'RPC start': '2024-10-14 12:57:07.696279', 'RPC error': '2024-10-14 12:57:08.547992'}> (decorators.py:147)
[2024-10-14 12:57:08,549 - ERROR - fouram]: (api_response) : [Collection.search] <MilvusException: (code=65535, message=fail to search on QueryNode 2: worker(2) query failed: Assert "index < this->counter_"  => index out of range, index=0, counter_=0 at /workspace/source/internal/core/src/mmap/ChunkVector.h:126
)>, [requestId: d19f9322-8a2b-11ef-9efb-caae83df8f3a] (api_request.py:57)
[2024-10-14 12:57:08,549 - ERROR - fouram]: [CheckFunc] search request check failed, response:<MilvusException: (code=65535, message=fail to search on QueryNode 2: worker(2) query failed: Assert "index < this->counter_"  => index out of range, index=0, counter_=0 at /workspace/source/internal/core/src/mmap/ChunkVector.h:126
)> (func_check.py:106)

Expected Behavior

No response

Steps To Reproduce

- [argo workflow](https://argo-workflows.zilliz.cc/archived-workflows/qa/be3623e6-1769-4fdb-8585-3411613ca407?nodeId=zong-rolling-upgrade-all-hjwcd-205957643)

Milvus Log

pods:

zong-rolling-upgrade-all-hjwcd-milvus-datanode-6864db75b5-6vkdc     Running     0            1m      10.104.34.9       4am-node37     
zong-rolling-upgrade-all-hjwcd-milvus-datanode-6864db75b5-wp9sp     Running     0            1m      10.104.25.168     4am-node30     
zong-rolling-upgrade-all-hjwcd-milvus-indexnode-75d6c7f47-p5kjx     Running     0            1m      10.104.21.250     4am-node24     
zong-rolling-upgrade-all-hjwcd-milvus-indexnode-75d6c7f47-ts6ch     Running     0            1m      10.104.6.34       4am-node13     
zong-rolling-upgrade-all-hjwcd-milvus-mixcoord-579dcdd8fc-lrnn5     Running     0            1m      10.104.25.167     4am-node30     
zong-rolling-upgrade-all-hjwcd-milvus-proxy-b799c74d5-gbkgt         Running     0            1m      10.104.5.49       4am-node12     
zong-rolling-upgrade-all-hjwcd-milvus-querynode-0-55f689648nkv9     Running     0            1m      10.104.34.10      4am-node37     
zong-rolling-upgrade-all-hjwcd-milvus-querynode-0-55f68964ptv29     Running     0            1m      10.104.30.29      4am-node38     
zong-rolling-upgrade-all-hjwcd-milvus-querynode-0-55f68964q7b9v     Running     0            1m      10.104.5.50       4am-node12

Anything else?

No response

[yanliang567]

/assign @weiliu1031
/unassign

[ThreadDao]

/assign @cqy123456
/unassign @weiliu1031

[xiaofan-luan]

/assign @sunby

[xiaofan-luan]

/assign @cqy123456

[xiaofan-luan]

this seems to be an growing mmap issue

[cqy123456]

Growing insert is not locked. During the insert process, the vector chunks will be cleared (chunk number counter = 0)after the growing index is built. And the growing segment uses indexing_record_.SyncDataWithIndex to get whether the growing index has been successfully built.

From the log, it can be seen that the chunks are cleared and the vector BF search at the same time, and there is a consistency problem in the access of indexing_record_.SyncDataWithIndex.
SyncDataWithIndex = fasle ->jump to the BF search logic->SyncDataWithIndex = true -> try_remove_chunks -> BF search.

[yanliang567]

@cqy123456 @ThreadDao any updates for this issue?