Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long-term development targets #4

Open
sigmafelix opened this issue Aug 24, 2023 · 3 comments
Open

Long-term development targets #4

sigmafelix opened this issue Aug 24, 2023 · 3 comments
Assignees
Labels
longterm long-term preparation. discussion required

Comments

@sigmafelix
Copy link
Collaborator

  • Bridging python packages (dask-geopandas, geopolars, cuSpatial) to R workflow
  • Optimal splitting of computational regions regarding shape complexity and input data resolutions
  • Streamlined integration of sparklyr and sf-derived functions

--- more ideas to comments ---

@sigmafelix sigmafelix added the longterm long-term preparation. discussion required label Aug 28, 2023
@sigmafelix
Copy link
Collaborator Author

  • Establish a temporary PostgreSQL/PostGIS setting for performance (ad hoc)
  • Compare mirai and crew for parallelization backend with future

@sigmafelix sigmafelix self-assigned this Jan 31, 2024
@sigmafelix
Copy link
Collaborator Author

  • mirai keeps using the low rate of CPU and ends up taking longer time than future runs.
  • future.batchtools integration into targets pipeline in other projects are promising, but I want to note that it will not guarantee to reduce the total runtime compared to running separate pipeline nodes with future.
  • crew is relevant to targets pipeline.

@sigmafelix
Copy link
Collaborator Author

sigmafelix commented Mar 12, 2024

  • Integration of callr for background processing (semi-automatic and asynchronous, I would say a la parallel) -- future.callr can handle this.
  • Passing paths instead of sf/terra objects in vector processing use cases
  • Memory saving solutions: no global object export as default

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
longterm long-term preparation. discussion required
Projects
None yet
Development

No branches or pull requests

1 participant