Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]: Optimize memory usage of deserialize delta log. #30404

Closed
1 task done
aoiasd opened this issue Jan 31, 2024 · 5 comments
Closed
1 task done

[Enhancement]: Optimize memory usage of deserialize delta log. #30404

aoiasd opened this issue Jan 31, 2024 · 5 comments
Assignees
@aoiasd
Labels
kind/enhancement Issues or changes related to enhancement stale indicates no udpates for 30 days

Comments

@aoiasd
Copy link
Contributor

aoiasd commented Jan 31, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

Delta log deserialize has a large overheads at memory peak.
Optimize memory usage of deserialize delta log.

Why is this needed?

relate: #30110

Anything else?

No response
@aoiasd aoiasd added the kind/enhancement Issues or changes related to enhancement label Jan 31, 2024
@congqixia
Copy link
Contributor

/assign @aoiasd

@aoiasd aoiasd changed the title [Enhancement]: Delta log deserialize has a large overheads at memory peak. [Enhancement]: Optimize memory usage of deserialize delta log. Jan 31, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Jan 31, 2024
@congqixia
enhance: Add PrimaryKeys interface to reduce memory usage
b8b02d8 
See also milvus-io#30404

`PrimaryKey` is used to hold pk values for both int64 & varchar data
type. Since it is an interface it may occupies more memory than pure
slices when holding a group of pks.

This PR add `PrimaryKeys` interface when some other module need to hold
lots of PrimaryKeys.
By using this interface, it could reduce the memory of pk slice to half
when using Int64 Pk data type and reduce interface cost for each row of
varchar as well.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Jan 31, 2024
Merged
@aoiasd aoiasd mentioned this issue Jan 31, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 1, 2024
@congqixia
enhance: Add PrimaryKeys interface to reduce memory usage (#30405)
e677af1 
See also #30404

`PrimaryKey` is used to hold pk values for both int64 & varchar data
type. Since it is an interface it may occupies more memory than pure
slices when holding a group of pks.

This PR add `PrimaryKeys` interface when some other module need to hold
lots of PrimaryKeys.
By using this interface, it could reduce the memory of pk slice to half
when using Int64 Pk data type and reduce interface cost for each row of
varchar as well.

---------

Signed-off-by: Congqi Xia <[email protected]>
sre-ci-robot pushed a commit that referenced this issue Feb 6, 2024
@aoiasd
enhance: delete codc deserialize data by stream batch (#30407)
a053715 
relate: #30404

---------

Signed-off-by: aoiasd <[email protected]>
@tedxu tedxu mentioned this issue Feb 20, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Feb 21, 2024
@tedxu
enhance: Adding a generic stream payload reader (#30682)
12acaf3 
See: #30404

---------

Signed-off-by: Ted Xu <[email protected]>
@stale Stale
Copy link

stale bot commented Mar 1, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Mar 1, 2024
@congqixia
Copy link
Contributor

still WIP

@stale stale bot closed this as completed Mar 14, 2024
@xiaofan-luan xiaofan-luan reopened this Mar 15, 2024
@stale stale bot removed the stale indicates no udpates for 30 days label Mar 15, 2024
@xiaofan-luan
Copy link
Collaborator

keep this issue

@stale Stale
Copy link

stale bot commented Apr 15, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Apr 15, 2024
@stale stale bot closed this as completed Apr 23, 2024
congqixia added a commit to congqixia/milvus that referenced this issue Oct 28, 2024
@congqixia
enhance: Return deltadata for DeleteCodec.Deserialize
15e7e21 
Related to milvus-io#35303 milvus-io#30404

This PR change return type of `DeleteCodec.Deserialize`
from `storage.DeleteData` to `DeltaData`, which
reduces the memory usage of interface header.

Also refine `storage.DeltaData` methods to make it
easier to usage.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Oct 28, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 29, 2024
@congqixia
enhance: Return deltadata for DeleteCodec.Deserialize (#37214)
3106384 
Related to #35303 #30404

This PR change return type of `DeleteCodec.Deserialize` from
`storage.DeleteData` to `DeltaData`, which
reduces the memory usage of interface header.

Also refine `storage.DeltaData` methods to make it easier to usage.

Signed-off-by: Congqi Xia <[email protected]>
congqixia added a commit to congqixia/milvus that referenced this issue Oct 29, 2024
@congqixia
enhance: Return deltadata for DeleteCodec.Deserialize (milvus-io#37214
60ec291 
)

Related to milvus-io#35303 milvus-io#30404

This PR change return type of `DeleteCodec.Deserialize` from
`storage.DeleteData` to `DeltaData`, which
reduces the memory usage of interface header.

Also refine `storage.DeltaData` methods to make it easier to usage.

Signed-off-by: Congqi Xia <[email protected]>
@congqixia congqixia mentioned this issue Oct 29, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 30, 2024
@congqixia
enhance: [2.5] Return deltadata for DeleteCodec.Deserialize (#37214) (
ee6ce8c 
#37264)

Cherry pick from master
pr: #37214
Related to #35303 #30404

This PR change return type of `DeleteCodec.Deserialize` from
`storage.DeleteData` to `DeltaData`, which
reduces the memory usage of interface header.

Also refine `storage.DeltaData` methods to make it easier to usage.

Signed-off-by: Congqi Xia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aoiasd aoiasd

Labels
kind/enhancement Issues or changes related to enhancement stale indicates no udpates for 30 days
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aoiasd @xiaofan-luan @congqixia