Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to serialize a source #43

Open
kyleaoman opened this issue Mar 15, 2024 · 0 comments
Open

Option to serialize a source #43

kyleaoman opened this issue Mar 15, 2024 · 0 comments
Assignees
Milestone

Comments

@kyleaoman
Copy link
Owner

It's probably reasonable to expect a user to be able to hold (1) the particle data of the source and (2) the datacube in memory simultaneously as a minimum memory requirement. As currently implemented, however, martini is more memory-hungry than this because it pre-computes and stores the spectra of all particles simultaneously, requiring nchannels * nparticles floating point values, which for some setups could be a very large number. Pre-computing the spectra is done for speed, but it would be possible to cut down on memory by segmenting the source into a series of chunks of ~equal numbers of particles. The simplest implementation would be to run the source insertion loop once for each such chunk, which is a significant hit in terms of cpu time cost. Giving some flexibility to keep memory requirements under control would be good though. Could explore other options like optionally using a memmap to store the spectra on disk rather than in memory. Costs a bit of access speed but maybe better than running the main loop more than once.

@kyleaoman kyleaoman added this to the v2.2 milestone Mar 15, 2024
@kyleaoman kyleaoman self-assigned this Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant