Skip to content

Commit

Permalink
Test diff cov fix
Browse files Browse the repository at this point in the history
- Multirasters tests are not detected by codecov?
  • Loading branch information
Insang Song committed Dec 18, 2023
1 parent 0e3727a commit 44de531
Show file tree
Hide file tree
Showing 23 changed files with 408 additions and 62 deletions.
4 changes: 2 additions & 2 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Package: scomps
Title: Scalable R geospatial computation
Version: 0.1.1.12152023
Version: 0.1.1.12192023
Authors@R:
person("Insang", "Song", , "[email protected]", role = c("aut", "cre"),
comment = c(ORCID = "0000-0001-8732-3256"))
Expand Down Expand Up @@ -33,4 +33,4 @@ Suggests:
VignetteBuilder: knitr
Config/testthat/edition: 3
LitrVersionUsed: 0.9.0
LitrId: 81666995d76fe29d4d2a16477742c9eb
LitrId: 7aaf35ea9a4ab72b74d998b76280d9a7
2 changes: 2 additions & 0 deletions R/scale_process.R
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ distribute_process_grid <-
if ("id" %in% names(formals(fun_dist))) {
detected_id <- list(...)
detected_id <- detected_id$id
} else {
detected_id <- "id"
}
colnames(fallback)[1] <- detected_id
return(fallback)
Expand Down
Binary file added figure/generate-compregion-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added figure/load-srtm-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added figure/point-generation-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added figure/unnamed-chunk-3-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added figure/vis-results-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added figure/vis-results-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
26 changes: 19 additions & 7 deletions scomps_rmarkdown_litr.html
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

<meta name="author" content="Insang Song" />

<meta name="date" content="2023-12-15" />
<meta name="date" content="2023-12-18" />

<title>Creating the scomps R package</title>

Expand Down Expand Up @@ -353,7 +353,7 @@
<h1 class="title toc-ignore">Creating the <code>scomps</code> R
package</h1>
<h4 class="author">Insang Song</h4>
<h4 class="date">2023-12-15</h4>
<h4 class="date">2023-12-18</h4>

</div>

Expand All @@ -368,7 +368,7 @@ <h2>Package setup</h2>
path = &quot;.&quot;,
fields = list(
Package = params$package_name,
Version = &quot;0.1.1.12152023&quot;,
Version = &quot;0.1.1.12192023&quot;,
Title = &quot;Scalable R geospatial computation&quot;,
Description = &quot;A package for scalable geospatial computation for environmental health research&quot;,
`Authors@R` = person(
Expand Down Expand Up @@ -2297,6 +2297,8 @@ <h3>Create functions</h3>
if (&quot;id&quot; %in% names(formals(fun_dist))) {
detected_id &lt;- list(...)
detected_id &lt;- detected_id$id
} else {
detected_id &lt;- &quot;id&quot;
}
colnames(fallback)[1] &lt;- detected_id
return(fallback)
Expand Down Expand Up @@ -2803,6 +2805,8 @@ <h3>Create functions</h3>
func = &quot;mean&quot;
)
)
testthat::expect_s3_class(res, &quot;data.frame&quot;)
testthat::expect_true(!anyNA(res))

testfiles_corrupted &lt;- c(testfiles, &quot;/home/runner/fallin.tif&quot;)
testthat::expect_warning(
Expand Down Expand Up @@ -2902,6 +2906,7 @@ <h2>Documenting the package and building</h2>
## Writing &#39;distribute_process_multirasters.Rd&#39;
## Writing &#39;switch_packbound.Rd&#39;
## Writing &#39;validate_and_repair_vectors.Rd&#39;</code></pre>
<p>[1] TRUE</p>
<p>── R CMD build
───────────────────────────────────────────────────────────────── *
checking for file
Expand All @@ -2910,11 +2915,11 @@ <h2>Documenting the package and building</h2>
installing the package to build vignettes * creating vignettes … OK *
checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories * building
‘scomps_0.1.1.12152023.tar.gz’ Warning: invalid uid value replaced by
‘scomps_0.1.1.12192023.tar.gz’ Warning: invalid uid value replaced by
that for user ‘nobody’ Warning: invalid gid value replaced by that for
user ‘nobody’</p>
<p>[1]
“/Users/songi2/Documents/GitHub/Scalable_GIS/scomps_0.1.1.12152023.tar.gz”</p>
“/Users/songi2/Documents/GitHub/Scalable_GIS/scomps_0.1.1.12192023.tar.gz”</p>
<p>/Users/songi2/Documents/GitHub/Scalable_GIS/scomps</p>
<p>[1] “/Users/songi2/Documents/GitHub/Scalable_GIS/scomps”</p>
<pre class="r"><code># here we return to the current working directory &quot;you see&quot;
Expand All @@ -2932,6 +2937,7 @@ <h2>Documenting the package and building</h2>

devtools::install(build = FALSE, dependencies = FALSE, build_vignettes = TRUE, quick = TRUE)
</code></pre>
<pre><code>## [1] TRUE</code></pre>
<pre><code>## ── R CMD build ─────────────────────────────────────────────────────────────────
## * checking for file ‘/Users/songi2/Documents/GitHub/Scalable_GIS/scomps/DESCRIPTION’ ... OK
## * preparing ‘scomps’:
Expand All @@ -2940,11 +2946,17 @@ <h2>Documenting the package and building</h2>
## * creating vignettes ... OK
## * checking for LF line-endings in source and make files and shell scripts
## * checking for empty or unneeded directories
## * building ‘scomps_0.1.1.12152023.tar.gz’
## * building ‘scomps_0.1.1.12192023.tar.gz’
## Warning: invalid uid value replaced by that for user &#39;nobody&#39;
## Warning: invalid gid value replaced by that for user &#39;nobody&#39;</code></pre>
<pre><code>## [1] &quot;/Users/songi2/Documents/GitHub/Scalable_GIS/scomps_0.1.1.12152023.tar.gz&quot;</code></pre>
<pre><code>## [1] &quot;/Users/songi2/Documents/GitHub/Scalable_GIS/scomps_0.1.1.12192023.tar.gz&quot;</code></pre>
<pre><code>## sf (1.0-14 -&gt; 1.0-15) [CRAN]</code></pre>
<pre><code>## Installing 1 packages: sf</code></pre>
<pre><code>## Installing package into &#39;/Users/songi2/Library/R/arm64/4.3/library&#39;
## (as &#39;lib&#39; is unspecified)</code></pre>
<pre><code>##
## The downloaded binary packages are in
## /var/folders/58/7rn_bn5d6k3_cxwnzdhswpz4n0z2n9/T//RtmptTZeUl/downloaded_packages
## Running /Library/Frameworks/R.framework/Resources/bin/R CMD INSTALL \
## /Users/songi2/Documents/GitHub/Scalable_GIS/scomps --install-tests \
## --no-docs --no-multiarch --no-demo
Expand Down
8 changes: 6 additions & 2 deletions scomps_rmarkdown_litr.rmd
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Creating the ``r params$package_name`` R package"
author: "Insang Song"
date: "2023-12-15"
date: "2023-12-18"
knit: litr::render
output: litr::litr_html_document
params:
Expand All @@ -21,7 +21,7 @@ usethis::create_package(
path = ".",
fields = list(
Package = params$package_name,
Version = "0.1.1.12152023",
Version = "0.1.1.12192023",
Title = "Scalable R geospatial computation",
Description = "A package for scalable geospatial computation for environmental health research",
`Authors@R` = person(
Expand Down Expand Up @@ -2171,6 +2171,8 @@ distribute_process_grid <-
if ("id" %in% names(formals(fun_dist))) {
detected_id <- list(...)
detected_id <- detected_id$id
} else {
detected_id <- "id"
}
colnames(fallback)[1] <- detected_id
return(fallback)
Expand Down Expand Up @@ -2691,6 +2693,8 @@ testthat::test_that("Processes are properly spawned and compute over multiraster
func = "mean"
)
)
testthat::expect_s3_class(res, "data.frame")
testthat::expect_true(!anyNA(res))
testfiles_corrupted <- c(testfiles, "/home/runner/fallin.tif")
testthat::expect_warning(
Expand Down
2 changes: 2 additions & 0 deletions tests/testthat/tests.R
Original file line number Diff line number Diff line change
Expand Up @@ -854,6 +854,8 @@ testthat::test_that("Processes are properly spawned and compute over multiraster
func = "mean"
)
)
testthat::expect_s3_class(res, "data.frame")
testthat::expect_true(!anyNA(res))

testfiles_corrupted <- c(testfiles, "/home/runner/fallin.tif")
testthat::expect_warning(
Expand Down
Binary file added tools/tarballs/scomps_0.1.1.12192023.tar.gz
Binary file not shown.
Binary file added tools/testdata/ncroads_first.gpkg
Binary file not shown.
104 changes: 98 additions & 6 deletions tools/vignettes-sources/v02_distribute_process_original.Rmd.orig
Original file line number Diff line number Diff line change
Expand Up @@ -170,18 +170,20 @@ plot(ncpoints_m[, "mean"], main = "Multi-thread")

## Distribute computation through geographic hierarchy
- We consider "hierarchy," which is usually embedded in many geospatial datasets. Suppose we want to summarize elevation across census geographies in North Carolina. The census geographies are highly organized in order; think of state, counties, zip code areas, census tracts, block groups, and blocks. Some of these are exhaustive to the higher order geographies (e.g., census block groups are exhaustively delineated in a census tract), whereas some do not (not all zip code areas are exhaustively covered by a county).
- In this example, we consider
- In this example, we consider census tracts and counties to distribute the computation task for mean elevation values at census tracts over counties.


```{r get-hierarchy}
nc_county <- tigris::counties(state = "NC", cb = TRUE, year = 2020)
nc_tracts <- tigris::tracts(cb = TRUE, state = "NC", year = 2020)
nc_county <- file.path("../testdata/nc_hierarchy.gpkg")
nc_county <- sf::st_read(nc_county, layer = "county")

nc_tracts <- file.path("../testdata/nc_hierarchy.gpkg")
nc_tracts <- sf::st_read(nc_tracts, layer = "tracts")

nc_county <- sf::st_transform(nc_county, "EPSG:5070")
nc_tracts <- sf::st_transform(nc_tracts, "EPSG:5070")
nc_tracts$COUNTY <-
paste0(nc_tracts$STATEFP, nc_tracts$COUNTYFP)

substr(nc_tracts$GEOID, 1, 5)
```

## Single-thread processing
Expand Down Expand Up @@ -215,4 +217,94 @@ system.time(

```

It is clearly shown that several thousands of features may not get benefits from parallel processing. However, this approach will be helpful when each geographic area is large and the raster data is too large to handle.
It is clearly shown that several thousands of features may not get benefits from parallel processing. However, this approach will be helpful when each geographic area is large and the raster data is too large to handle.



## Multi-thread processing over large rasters

In many cases users have parallelized over multiple files, i.e., large raster files. Leveraging memory-saving `terra`'s C++ pointers, users will want to use `distribute_process_multirasters`, where they are assumed to have the raster file paths to distribute. In standard vector-raster overlays, `distribute_process_multirasters` will help especially when users have multiple sizable raster files to extract values with moderately sized vectors.


```{r multirasters}
ncpath <- "../testdata/nc_hierarchy.gpkg"
nccnty <- terra::vect(ncpath, layer = "county")
ncelev <- terra::unwrap(readRDS("../testdata/nc_srtm15_otm.rds"))
terra::crs(ncelev) <- "EPSG:5070"
names(ncelev) <- c("srtm15")
tdir <- tempdir()
terra::writeRaster(ncelev, file.path(tdir, "test1.tif"), overwrite = TRUE)
terra::writeRaster(ncelev, file.path(tdir, "test2.tif"), overwrite = TRUE)
terra::writeRaster(ncelev, file.path(tdir, "test3.tif"), overwrite = TRUE)
terra::writeRaster(ncelev, file.path(tdir, "test4.tif"), overwrite = TRUE)
terra::writeRaster(ncelev, file.path(tdir, "test5.tif"), overwrite = TRUE)

testfiles <- list.files(tempdir(), pattern = "*.tif$", full.names = TRUE)
testfiles
```


```{r multirasters-processing}
res <- distribute_process_multirasters(
filenames = testfiles,
fun_dist = extract_with_polygons,
polys = nccnty,
surf = ncelev,
id = "GEOID",
func = "mean"
)

knitr::kable(head(res))
```

## Distributing sf/terra functions

Users also can pass generic terra functions to multiple threads.

Some considerations apply:
- If a function to calculate the nearest distance, edge cases may present especially when the target dataset are too sparse to be located in the padded calculation extent. For example, when users want to distribute `terra::nearest()` to calculate the nearest distance from ranch locations to the primary roads, some ranch locations might have no destination primary roads if the calculation grids are too finely defined.
- We strongly assume that users have unique identifiers in input data. The passed argument in `distribute_process_*` functions is expected to include `id` argument to designate the origin locations' unique identifiers.


### Example
- The example below shows how to distribute a `terra` function `nearest` is distributed over regular computational grids.


```{r generic-distribution}
pnts <- readRDS("../testdata/nc_random_point.rds")
pnts <- terra::vect(pnts)
rd1 <- terra::vect(
file.path("../testdata/ncroads_first.gpkg"))

pnts <- terra::project(pnts, "EPSG:5070")
rd1 <- terra::project(rd1, "EPSG:5070")


nccompreg <-
get_computational_regions(
input = pnts,
mode = "grid",
nx = 6L,
ny = 4L,
padding = 3e4L)

future::plan(future::multicore, workers = 6L)

system.time(
res <-
distribute_process_grid(
grids = nccompreg,
fun_dist = terra::nearest,
x = pnts,
y = rd1)
)



system.time(
restr <- terra::nearest(x = pnts, y = rd1)
)
knitr::kable(head(res))
knitr::kable(head(restr))
```

9 changes: 9 additions & 0 deletions tools/vignettes-sources/v03_extract_with.Rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
title: "Using extract_with"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Using extract_with}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
35 changes: 35 additions & 0 deletions tools/vignettes-sources/v04_calc_sedc.Rmd
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: "Exponentially decaying contributions"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
%\VignetteIndexEntry{Exponentially decaying contributions}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---



## Exponentially decaying contributions (EDC)
### Formula




### Sum of EDC



## Get EDC
### Concept



### Under-the-hood processes






## References
Binary file modified vignettes/figure/figure/generate-compregion-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified vignettes/figure/figure/load-srtm-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified vignettes/figure/figure/point-generation-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified vignettes/figure/figure/unnamed-chunk-3-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified vignettes/figure/figure/vis-results-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified vignettes/figure/figure/vis-results-2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit 44de531

Please sign in to comment.