How to use tarchetypes::tar_files() with AWS S3?

[mzorko]

Hi everyone,

We are trying to use tarchetypes::tar_files() with format = "aws_file".

We set-up our S3 bucket using:

targets::tar_option_set(
  resources = targets::tar_resources(
    aws = targets::tar_resources_aws(
      bucket = "our-bucket"
    )
  )
)

Upload example works and creates files in our defined bucket:

write_mean <- function(data) {
  tmp <- tempfile()
  writeLines(as.character(mean(data)), tmp)
  tmp
}

list(
  targets::tar_target(data, rnorm(5), format = "aws_qs"),
  targets::tar_target(mean_file, write_mean(data), format = "aws_file")
)

When we try to run tar_files() (full file path is s3://our-bucket/our_data/test_file.csv)

list(tarchetypes::tar_files(raw_data, "our_data/test_file.csv", format = "aws_file"))

we get following output:

> targets::tar_make()
• start target raw_data_files
• built target raw_data_files
• start branch raw_data_76b7f586
x error branch raw_data_76b7f586
• end pipeline
Error : _paths_ missing files: our_data/test_file.csv

We also tried to modify prefix using targets::tar_resources_aws().

Can you help us understand this?

[wlandau]

Sorry about that, it turns out tar_files() and functions are not compatible with format = "aws_file" because the original files no longer exist locally after they get uploaded to AWS. I just changed the documentation to reflect this.

[mzorko]

Hi Will, is there a way to track files already uploaded to S3 (that never existed locally on our machine), e.g., files someone else uploaded to our S3 bucket?

[mzorko]

Hi Will, would that work with private (not public) URLs?

[wlandau]

I am not sure, you may have to set extra environment variables in your project's .Renviron or something. Another way to check is with aws.s3 or paws.

[mzorko]

Hi Will, we set up all credentials to access S3 and we can use both {paws} and {aws.s3} to communicate (read & write) with S3. Issue is our object is not public in our S3 bucket and we are not able to access it using url. Even in our browser, we are getting access denied - same error we get in {targets}. Currently, it looks like format = "url" can only be used with public buckets, accessible via url.

[wlandau]

Well shucks, I could have sworn AWS would make it easier than having to modify an API call. (So you really can't just set an env var and have the URLs work? I haven't scoured the docs myself.)

The workarounds I can think of would depend on how many URLs you have and how often you expect the data to change. If you know the data in the buckets is not going to change, then you can just create a target for each URL that downloads the data in the bucket. If you expect the data to change and don't have an unreasonable number of URLs, then you could define a global object in _targets.R to first get the ETag, then let the target that downloads the data depend on it. Sketch:

# _targets.R
library(targets)
source("functions.R")
your_etag <- get_etag_of_object()
list(
  tar_target(x, depend_on_etag_and_download_object(your_etag, other_args))
)

[mzorko]

Hi Will,

Thanks for this input. We will test this approach in the following week, but for now we found another way of using {targets} that fit us currently. We are mostly using it with Spark ({sparklyr}) and AWS. In any case, we are really amazed with the power of {targets} itself.

We will definitely keep in touch with our findings.

Thanks,
Mislav