feat: Geospatial Data Type and GIS Function Support for milvus #37417

tasty-gumi · 2024-11-04T10:48:23Z

Main Goals

Create and describe collections with geospatial fields, enabling both client and server to recognize and process geo fields.
Insert geospatial data as payload values in the insert binlog, and print the values for verification.
Load segments containing geospatial data into memory.
Ensure query outputs can display geospatial data.
Support filtering on GIS functions for geospatial columns.

Solution

Add Type: Modify the Milvus core by adding a Geospatial type in both the C++ and Go code layers, defining the Geospatial data structure and the corresponding interfaces.
Dependency Libraries: Introduce necessary geospatial data processing libraries. In the C++ source code, use Conan package management to include the GDAL library. In the Go source code, add the go-geom library to the go.mod file.
Protocol Interface: Revise the Milvus protocol to provide mechanisms for Geospatial message serialization and deserialization.
Data Pipeline: Facilitate interaction between the client and proxy using the WKT format for geospatial data. The proxy will convert all data into WKB format for downstream processing, providing column data interfaces, segment encapsulation, segment loading, payload writing, and cache block management.
Query Operators: Implement simple display and support for filter queries. Initially, focus on filtering based on spatial relationships for a single column of geospatial literal values, providing parsing and execution for query expressions.
Index Construction: Consider building an H3 index, utilizing the C interface provided by the H3 system.
Client Modification: Enable the client to handle user input for geospatial data and facilitate end-to-end testing.Check the modification in pymilvus.

delete incomplete H3 Index development and useless generated files.
fix conanfiles in milvus conan repo so that local can fetch the packages to build libraries

sre-ci-robot · 2024-11-04T10:48:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tasty-gumi
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-11-04T10:56:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T10:58:06Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T11:35:49Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T11:36:31Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

czs007 · 2024-11-04T12:12:13Z

rerun go-sdk

czs007 · 2024-11-04T12:12:24Z

rerun cpp-unit-test

mergify · 2024-11-04T12:52:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T13:10:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-11-04T13:15:43Z

/run-cpu-e2e

mergify · 2024-11-04T13:19:53Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T13:21:14Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:09:21Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:43:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

MrPresent-Han · 2024-11-05T02:20:51Z

internal/core/unittest/test_utils/DataGen.h

+    return geometry;
+}
+
+inline GeneratedData


The order of function DataGen is changed, the parameter 'repeat_count' become the 6th parameter, this will lead to the failures of test_group_by, which pass a non-zero repeat_count but got a zero repeat count after this change. Please modify the corresponding ut case to avoid Signal 8 (SIGFPE)

mergify · 2024-11-05T08:36:01Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T08:57:51Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T09:11:29Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T12:02:43Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

tasty-gumi · 2024-11-05T12:04:12Z

rerun go-sdk

mergify · 2024-11-05T12:26:56Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T12:35:05Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T12:42:31Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

add geospatial interface in src common change type define and add segcore support add storage & chunkdata support feature: go package storage & proxy & typeutil support geospatial type in internal and typeutil in pkg Signed-off-by: tasty-gumi <[email protected]> add geospatial interface in src common change type define and add segcore support change: use wkb only in core Signed-off-by: tasty-gumi <[email protected]> fix:the geospatial only use std::string as FieldDataImpl template paramters && add geospatial data generation && pass chunk ,growing , sealed test fix : merge confilcts after rebase ,test nullable not pass due to upstream feat:basic GIS Function expr and visitor impl and GIS proto support && add:storage test of geo data Signed-off-by: tasty-gumi <[email protected]> feat:add proxy validate (pass httpserver test) && plan parser of geospatialfunction fix:sealedseg && go tidy fix:go mod feat:can produce wkt result for pymilvus client feat: add parser and query operator for geos filed && print geos binlog as wkt fix:fielddataimpl interface Signed-off-by: tasty-gumi <[email protected]> fix: some format of code && segmentfault debug for rebase Signed-off-by: tasty-gumi <[email protected]> add: import util test for parquet and mix compaction test Signed-off-by: tasty-gumi <[email protected]> fix: delete useless file and fix error for rebase Signed-off-by: tasty-gumi <[email protected]> fix: git rebase for custom function feat Signed-off-by: tasty-gumi <[email protected]> fix:rename geospatial field && update proto && rewrite Geometry class with smart pointer Signed-off-by: tasty-gumi <[email protected]> add:last commit miss add files Signed-off-by: tasty-gumi <[email protected]> fix: geospatial name replace in test files && fix geomertry and parser fix:remove some file change for dev Signed-off-by: tasty-gumi <[email protected]> fix:remove size in if && add destory in ~Geometry() Signed-off-by: tasty-gumi <[email protected]> add:conan file gdal rep Signed-off-by: tasty-gumi <[email protected]> remove:gdal fPIC Signed-off-by: tasty-gumi <[email protected]> fix: for rebase Signed-off-by: tasty-gumi <[email protected]> remove:log_warn Signed-off-by: tasty-gumi <[email protected]> remove:gdal shared Signed-off-by: tasty-gumi <[email protected]> remove:tbbproxy Signed-off-by: tasty-gumi <[email protected]> fix:add gdal option && update go mod Signed-off-by: tasty-gumi <[email protected]> dev:change some scripts Signed-off-by: tasty-gumi <[email protected]> remove: dev scripts Signed-off-by: tasty-gumi <[email protected]> add:conan files dependency of gdal Signed-off-by: tasty-gumi <[email protected]> fix:fmt cpp code Signed-off-by: tasty-gumi <[email protected]> add:delete geos-config in cmake_bulid/bin which may cause permission deny Signed-off-by: tasty-gumi <[email protected]> fix: add go client geometry interface && fix group by test Signed-off-by: tasty-gumi <[email protected]> fix: mod tidy for tests go client Signed-off-by: tasty-gumi <[email protected]> fix:memory leak in test and go fmt Signed-off-by: tasty-gumi <[email protected]> fix: datagen function remove pkoffset Signed-off-by: tasty-gumi <[email protected]>

mergify · 2024-11-06T07:20:55Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

tasty-gumi · 2024-11-06T07:22:18Z

/run-cpu-e2e

mergify · 2024-11-06T07:52:29Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2024-11-06T08:45:16Z

Codecov Report

Attention: Patch coverage is 47.17742% with 131 lines in your changes missing coverage. Please review.

Project coverage is 67.17%. Comparing base (b3de4b0) to head (c050d46).
Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
...core/src/exec/expression/GISFunctionFilterExpr.cpp	0.00%	33 Missing ⚠️
internal/core/src/common/Geometry.h	38.29%	29 Missing ⚠️
internal/core/src/query/PlanProto.cpp	6.25%	15 Missing ⚠️
internal/core/src/common/FieldData.cpp	0.00%	14 Missing ⚠️
internal/core/src/exec/expression/Expr.cpp	0.00%	7 Missing ⚠️
internal/core/src/expr/ITypeExpr.h	0.00%	7 Missing ⚠️
internal/core/src/common/Array.h	0.00%	6 Missing ⚠️
internal/core/src/common/Types.h	16.66%	5 Missing ⚠️
...l/core/src/exec/expression/GISFunctionFilterExpr.h	0.00%	5 Missing ⚠️
internal/core/src/common/FieldDataInterface.h	87.87%	4 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #37417       +/-   ##
===========================================
- Coverage   83.25%   67.17%   -16.09%     
===========================================
  Files        1015      293      -722     
  Lines      157480    25626   -131854     
===========================================
- Hits       131116    17214   -113902     
+ Misses      21172     8412    -12760     
+ Partials     5192        0     -5192

Components	Coverage Δ
Client	`∅ <ø> (∅)`
Core	`67.17% <47.17%> (∅)`
Go	`∅ <ø> (∅)`

Files with missing lines	Coverage Δ
internal/core/src/common/FieldData.h	`100.00% <100.00%> (ø)`
internal/core/src/query/PlanProto.h	`18.51% <ø> (ø)`
internal/core/src/segcore/ConcurrentVector.cpp	`97.26% <100.00%> (ø)`
internal/core/src/segcore/InsertRecord.h	`83.66% <100.00%> (ø)`
internal/core/src/segcore/SegmentGrowingImpl.cpp	`77.60% <100.00%> (ø)`
internal/core/src/segcore/SegmentSealedImpl.cpp	`86.18% <100.00%> (ø)`
internal/core/src/storage/Event.cpp	`83.39% <100.00%> (ø)`
internal/core/src/storage/Util.cpp	`76.93% <100.00%> (ø)`
internal/core/src/mmap/Utils.h	`81.60% <84.61%> (ø)`
internal/core/src/common/FieldDataInterface.h	`59.10% <87.87%> (ø)`
... and 10 more

... and 1288 files with indirect coverage changes

Signed-off-by: tasty-gumi <[email protected]>

mergify · 2024-11-06T09:17:10Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

tasty-gumi · 2024-11-06T09:28:09Z

rerun cpp-unit-test

mergify · 2024-11-06T09:53:27Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

sre-ci-robot requested review from bigsheeper, congqixia and czs007 November 4, 2024 10:48

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/dependency Pull requests that update a dependency file area/test sig/testing test/integration integration test labels Nov 4, 2024

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 4, 2024

MrPresent-Han reviewed Nov 5, 2024

View reviewed changes

tasty-gumi force-pushed the geo_pr0905 branch from 3b943e5 to ebf2daa Compare November 5, 2024 08:04

tasty-gumi force-pushed the geo_pr0905 branch from fe08100 to 85d2f9a Compare November 6, 2024 06:51

fix: go-client test add entity.geometry

c050d46

Signed-off-by: tasty-gumi <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Geospatial Data Type and GIS Function Support for milvus #37417

feat: Geospatial Data Type and GIS Function Support for milvus #37417

tasty-gumi commented Nov 4, 2024

sre-ci-robot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

MrPresent-Han Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

tasty-gumi commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 6, 2024

tasty-gumi commented Nov 6, 2024

mergify bot commented Nov 6, 2024

codecov bot commented Nov 6, 2024 •

edited

Loading

mergify bot commented Nov 6, 2024

tasty-gumi commented Nov 6, 2024

mergify bot commented Nov 6, 2024

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Are you sure you want to change the base?

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Conversation

tasty-gumi commented Nov 4, 2024

Main Goals

Solution

sre-ci-robot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

MrPresent-Han Nov 5, 2024

Choose a reason for hiding this comment

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

tasty-gumi commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 6, 2024

tasty-gumi commented Nov 6, 2024

mergify bot commented Nov 6, 2024

codecov bot commented Nov 6, 2024 • edited Loading

Codecov Report

mergify bot commented Nov 6, 2024

tasty-gumi commented Nov 6, 2024

mergify bot commented Nov 6, 2024

codecov bot commented Nov 6, 2024 •

edited

Loading