Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use more array programming style to improve firesquid speed #78

Open
DylanVerstraete opened this issue Jan 11, 2023 · 1 comment
Open
Labels
priority_minor type_feature New feature or request
Milestone

Comments

@DylanVerstraete
Copy link
Contributor

DylanVerstraete commented Jan 11, 2023

For some object we already use "array programming" meaning we collect all creates/updates inside a batch of blocks and save to the database later.

For farms, nodes, contracts, we still save one by one due to relations they have. In a future version, we should convert this to array programming style.

For example, you can read block by block and save entities to the db inbetween
or with batch processing you can: read X amount of blocks, save entities to be saved in memory and save to the DB whenever you processed the X amount of blocks => higher performance

@DylanVerstraete DylanVerstraete added this to the 3.0.0 milestone Jan 11, 2023
@rkhamis rkhamis removed this from 3.10.x Mar 14, 2023
@rkhamis rkhamis modified the milestones: 3.0.0, later Mar 14, 2023
@renauter
Copy link
Contributor

renauter commented Jul 27, 2023

More insights from @DylanVerstraete

Should be: we collect all creates/updates inside a batch of blocks and save to the database later
Array style programming could be phrased as batch style processing
For example, you can read block by block and save entities to the db in between or with batch processing you can: read X amount of blocks, save entities to be saved in memory and save to the DB whenever you processed the X amount of blocks => higher performance
Since every "save to the db" opens a connection to the postgres and has some overhead, opening the connection only once and saving an entire bunch of entities is therefor more performant

@sameh-farouk sameh-farouk moved this to Accepted in 3.15.x Jun 24, 2024
@sameh-farouk sameh-farouk modified the milestones: later, 2.13.0 Jun 24, 2024
@ashraffouda ashraffouda removed this from 3.15.x Sep 18, 2024
@ashraffouda ashraffouda modified the milestones: 2.13.0, 3.0.0 Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority_minor type_feature New feature or request
Projects
None yet
Development

No branches or pull requests

5 participants