Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [null & default] The "nullable" property is lost and data is not "None" after backup and restore the collection with the nullable fields #36341

Closed
1 task done
binbinlv opened this issue Sep 18, 2024 · 5 comments
Closed
1 task done

[Bug]: [null & default] The "nullable" property is lost and data is not "None" after backup and restore the collection with the nullable fields #36341

binbinlv opened this issue Sep 18, 2024 · 5 comments
Assignees
@wayblink @smellthemoon @binbinlv
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@binbinlv
Copy link
Contributor

binbinlv commented Sep 18, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20240914-5ce99e25-amd64
- Deployment mode(standalone or cluster): both
- MQ type(rocksmq, pulsar or kafka):    all 
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc78
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The "nullable" property is lost and data is not "None" after backup the collection with the nullable fields

data from "None" to 0.0 for int field., and from "None" to '' for varchar field.

before backup:

Does collection hello_milvus2 exist in Milvus: True
{'auto_id': True, 'description': 'hello_milvus2', 'fields': [{'name': 'pk', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'random', 'description': '', 'type': <DataType.DOUBLE: 11>, 'nullable': True}, {'name': 'var', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}, 'nullable': True}, {'name': 'embeddings', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 8}}], 'enable_dynamic_field': False}
Number of entities in Milvus: hello_milvus2 : 6000

=== Start Creating index IVF_FLAT  ===


=== Start loading                  ===


=== Start searching based on vector similarity ===

hit: id: 452625224862197802, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862197802, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200803, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200803, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862197803, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862197803, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200804, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200804, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, var field: None
search latency = 0.3032s

=== Start querying with `random > 0.5` ===

query result:
-{'random': None, 'var': None, 'embeddings': [0.07026907, 0.43795532, 0.72754675, 0.82399035, 0.65299004, 0.2594087, 0.53980845, 0.8232887], 'pk': 452625224862197429}
search latency = 0.7342s

=== Start hybrid searching with `random > 0.5` ===

hit: id: 452625224862197802, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862197802, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200803, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200803, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862197803, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862197803, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862200804, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862200804, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, random field: None
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, var field: None
search latency = 0.2668s

=== Drop collection hello_milvus2  ===

After backup and restore:

Does collection hello_milvus2_recover exist in Milvus: True
{'auto_id': True, 'description': 'hello_milvus2', 'fields': [{'name': 'pk', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'random', 'description': '', 'type': <DataType.DOUBLE: 11>}, {'name': 'var', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}}, {'name': 'embeddings', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 8}}], 'enable_dynamic_field': False}
Number of entities in Milvus: hello_milvus2_recover : 6000

=== Start Creating index IVF_FLAT  ===


=== Start loading                  ===


=== Start searching based on vector similarity ===

hit: id: 452625224862197802, distance: 0.0, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862197802, distance: 0.0, entity: {'var': '', 'random': 0.0}, var field:
hit: id: 452625224862200803, distance: 0.0, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862200803, distance: 0.0, entity: {'var': '', 'random': 0.0}, var field:
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'var': '', 'random': 0.0}, var field:
hit: id: 452625224862197803, distance: 0.0, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862197803, distance: 0.0, entity: {'var': '', 'random': 0.0}, var field:
hit: id: 452625224862200804, distance: 0.0, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862200804, distance: 0.0, entity: {'var': '', 'random': 0.0}, var field:
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'var': '', 'random': 0.0}, random field: 0.0
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'var': '', 'random': 0.0}, var field:
search latency = 0.2511s

=== Start querying with `random > 0.5` ===

query result:
-{'pk': 452625224862197429, 'random': 0.0, 'var': '', 'embeddings': [0.07026907, 0.43795532, 0.72754675, 0.82399035, 0.65299004, 0.2594087, 0.53980845, 0.8232887]}
search latency = 1.0258s

=== Start hybrid searching with `random > 0.5` ===

hit: id: 452625224862197802, distance: 0.0, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862197802, distance: 0.0, entity: {'random': 0.0, 'var': ''}, var field:
hit: id: 452625224862200803, distance: 0.0, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862200803, distance: 0.0, entity: {'random': 0.0, 'var': ''}, var field:
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862200391, distance: 0.07805602252483368, entity: {'random': 0.0, 'var': ''}, var field:
hit: id: 452625224862197803, distance: 0.0, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862197803, distance: 0.0, entity: {'random': 0.0, 'var': ''}, var field:
hit: id: 452625224862200804, distance: 0.0, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862200804, distance: 0.0, entity: {'random': 0.0, 'var': ''}, var field:
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': 0.0, 'var': ''}, random field: 0.0
hit: id: 452625224862198387, distance: 0.11571306735277176, entity: {'random': 0.0, 'var': ''}, var field:
search latency = 0.1761s

=== Drop collection hello_milvus2_recover ===

Expected Behavior

Data with "None" is not lost

Steps To Reproduce

1. prepare data with "None" data inserted
2. backup
3. restore

https://milvus.io/docs/milvus_backup_cli.md

Milvus Log

No response

Anything else?

No response
@binbinlv binbinlv added kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Sep 18, 2024
@binbinlv binbinlv added this to the 2.5.0 milestone Sep 18, 2024
@smellthemoon smellthemoon mentioned this issue Sep 25, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Sep 26, 2024
@smellthemoon
enhance: support null in bulk insert of binlog to help backup null (#…
b60164b 
…36526)

#36341

Signed-off-by: lixinguo <[email protected]>
Co-authored-by: lixinguo <[email protected]>
@smellthemoon
Copy link
Contributor

/assign @wayblink

@smellthemoon
Copy link
Contributor

related with zilliztech/milvus-backup#440, could you help to verify it? @binbinlv

@binbinlv
Copy link
Contributor Author

binbinlv commented Oct 23, 2024 •
edited
Loading

/assign

@binbinlv
Copy link
Contributor Author

working on verify

@binbinlv binbinlv self-assigned this Oct 23, 2024
@binbinlv
Copy link
Contributor Author

binbinlv commented Oct 23, 2024 •
edited
Loading

Verified and fixed: (after the latest backup commit: Support nullable field and default value (#440))

before backup:

Does collection hello_milvus2 exist in Milvus: True
{'auto_id': True, 'description': 'hello_milvus2', 'fields': [{'name': 'pk', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'random', 'description': '', 'type': <DataType.DOUBLE: 11>, 'nullable': True}, {'name': 'var', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}, 'nullable': True}, {'name': 'embeddings', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 8}}], 'enable_dynamic_field': False}
Number of entities in Milvus: hello_milvus2 : 6000

=== Start Creating index IVF_FLAT  ===


=== Start loading                  ===


=== Start searching based on vector similarity ===

hit: id: 453378225374913371, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913371, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374916372, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374916372, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374915960, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374915960, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374913372, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913372, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374916373, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374916373, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374913956, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913956, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, var field: None
search latency = 7.5056s

=== Start querying with `random > 0.5` ===

query result:
-{'random': None, 'var': None, 'embeddings': [0.07026907, 0.43795532, 0.72754675, 0.82399035, 0.65299004, 0.2594087, 0.53980845, 0.8232887], 'pk': 453378225374912998}
search latency = 7.7685s

=== Start hybrid searching with `random > 0.5` ===

hit: id: 453378225374913371, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913371, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374916372, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374916372, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374915960, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374915960, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374913372, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913372, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374916373, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374916373, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374913956, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374913956, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, var field: None
search latency = 1.5785s

=== Drop collection hello_milvus2  ===

After backup and restore:

Does collection hello_milvus2_recover exist in Milvus: True
{'auto_id': True, 'description': 'hello_milvus2', 'fields': [{'name': 'pk', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': True}, {'name': 'random', 'description': '', 'type': <DataType.DOUBLE: 11>, 'nullable': True}, {'name': 'var', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 65535}, 'nullable': True}, {'name': 'embeddings', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 8}}], 'enable_dynamic_field': False}
Number of entities in Milvus: hello_milvus2_recover : 6000

=== Start Creating index IVF_FLAT  ===


=== Start loading                  ===


=== Start searching based on vector similarity ===

hit: id: 453378225374931377, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374931377, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374934378, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374934378, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374933966, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374933966, distance: 0.07805602252483368, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374931378, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374931378, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374934379, distance: 0.0, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374934379, distance: 0.0, entity: {'random': None, 'var': None}, var field: None
hit: id: 453378225374931962, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, random field: None
hit: id: 453378225374931962, distance: 0.11571306735277176, entity: {'random': None, 'var': None}, var field: None
search latency = 0.6720s

=== Start querying with `random > 0.5` ===

query result:
-{'random': None, 'var': None, 'embeddings': [0.07026907, 0.43795532, 0.72754675, 0.82399035, 0.65299004, 0.2594087, 0.53980845, 0.8232887], 'pk': 453378225374931004}
search latency = 0.7793s

=== Start hybrid searching with `random > 0.5` ===

hit: id: 453378225374931377, distance: 0.0, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374931377, distance: 0.0, entity: {'var': None, 'random': None}, var field: None
hit: id: 453378225374934378, distance: 0.0, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374934378, distance: 0.0, entity: {'var': None, 'random': None}, var field: None
hit: id: 453378225374933966, distance: 0.07805602252483368, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374933966, distance: 0.07805602252483368, entity: {'var': None, 'random': None}, var field: None
hit: id: 453378225374931378, distance: 0.0, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374931378, distance: 0.0, entity: {'var': None, 'random': None}, var field: None
hit: id: 453378225374934379, distance: 0.0, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374934379, distance: 0.0, entity: {'var': None, 'random': None}, var field: None
hit: id: 453378225374931962, distance: 0.11571306735277176, entity: {'var': None, 'random': None}, random field: None
hit: id: 453378225374931962, distance: 0.11571306735277176, entity: {'var': None, 'random': None}, var field: None
search latency = 0.5485s

=== Drop collection hello_milvus2_recover ===

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wayblink wayblink

@smellthemoon smellthemoon

@binbinlv binbinlv

Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
3 participants
@wayblink @smellthemoon @binbinlv