issue accessing data decrypted at project level

[SpikeLewis]

Problem

I'm trying to use RuODK but having trouble accessing the data that has been encrypted at the project level. sorry if im missing something obvious.

Reproducible example

# insert reprex here

Session Info

# utils::sessionInfo()

[florianm]

What functions did you use?
Encryption is not supported in the OData and RESTful API endpoints, so https://docs.ropensci.org/ruODK/reference/odata_submission_get.html or https://docs.ropensci.org/ruODK/reference/submission_get.html won't work, but you can use https://docs.ropensci.org/ruODK/reference/submission_export.html (the CSV.ZIP export). Does the latter work for you?

FYI we're working on a vignette explaining encryption a bit better: #112

[SpikeLewis]

Hi Florian, Thank you for this message. I'm getting error 400 "Error in response %>% httr::stop_for_status(task = glue::glue("get desired response from server {url} as user \"{un}\".\n\n", : Bad Request (HTTP 400). Failed to get desired response from server https://myserver as user "my_email". Using this (I've checked all the credentials, and they are correct). Sorry this has stumped me! Spike

…

________________________________ From: Florian Mayer ***@***.***> Sent: Friday, April 8, 2022 2:56 AM To: ropensci/ruODK ***@***.***> Cc: SpikeLewis ***@***.***>; Author ***@***.***> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134) What functions did you use? Encryption is not supported in the OData API endpoints, but you can use https://docs.ropensci.org/ruODK/reference/submission_export.html (the CSV.ZIP export). Does the latter work for you? — Reply to this email directly, view it on GitHub<#134 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYTKCFHYQT2OEGJ6AJMKSX3VD6G5XANCNFSM5SZKNHMQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[florianm]

Could you provide a reprex please? At least answer my question above re which function?

Status 400 is an authentication error, so it points to a wrong credential or maybe a lack of permissions of that account in ODK Central.

[SpikeLewis]

Hi Florian, Sorry I'm not sure what function you're asking about, but hopefully the following is reproducible. if (!requireNamespace("remotes")) install.packages("remotes") # Full install remotes::install_github( ***@***.***", dependencies = TRUE, upgrade = "always", build_vignettes = TRUE) usethis::edit_r_environ(scope = "user") library(ruODK) ru_setup( url = Sys.getenv("ODKC_URL"), un = Sys.getenv("ODKC_UN"), pw = Sys.getenv("ODKC_PW"), pp = Sys.getenv("ODKC_PP"), tz = "Australia/Perth", pid = Sys.getenv("ODKC_PID"), odkc_version = "1.1", fid = Sys.getenv("ODKC_FID"), test_pw = Sys.getenv("ODKC_TEST_PW"), test_un = Sys.getenv("ODKC_TEST_UN"), test_pp = Sys.getenv("ODKC_TEST_PP"), test_url = Sys.getenv("ODKC_TEST_URL"), test_pid = "1", test_fid = Sys.getenv("ODKC_TEST_FID"), test_fid_zip = Sys.getenv("ODKC_TEST_FID_ZIP", ""), test_fid_gap = Sys.getenv("ODKC_TEST_FID_ATT", ""), test_fid_att = Sys.getenv("ODKC_TEST_FID_ATT", ""), test_odkc_version = "1.1", test_fid_wkt = Sys.getenv("ODKC_TEST_FID_WKT", ""), verbose = TRUE) submission_export( local_dir = here::here(), overwrite = TRUE, media = TRUE, repeats = TRUE, pid = get_default_pid(), fid = get_default_fid(), url = get_default_url(), un = get_default_un(), pw = get_default_pw(), pp = get_default_pp(), retries = get_retries(), odkc_version = get_default_odkc_version(), verbose = get_ru_verbose() )

…

________________________________ From: Florian Mayer ***@***.***> Sent: Wednesday, April 13, 2022 2:46 AM To: ropensci/ruODK ***@***.***> Cc: SpikeLewis ***@***.***>; Author ***@***.***> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134) Could you provide a reprex please? At least answer my question above re which function? Status 400 is an authentication error, so it points to a wrong credential or maybe a lack of permissions of that account in ODK Central. — Reply to this email directly, view it on GitHub<#134 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYTKCFGISKYYBN7O5OBRKOLVEYROXANCNFSM5SZKNHMQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[yanokwa]

I've confirmed that there is an issue with @SpikeLewis's passphrase. It works when exporting directly from Central, but not with ruODK. I will continue to investigate to see if we can get a repro.

[SpikeLewis]

Hi all, Just to update it seems to not be an issue with passphrase (it works on a dummy project). My only other thought is that it is the number of forms within the project (I'm on 3000+ forms). As I said, I can get in, and see that there are submissions, it's just throwing the error up on decryption. Going in via central itself, there is no issues downloading and decrypting from there. Best, Spike

…

________________________________ From: Yaw Anokwa ***@***.***> Sent: Thursday, April 14, 2022 6:58 PM To: ropensci/ruODK ***@***.***> Cc: SpikeLewis ***@***.***>; Mention ***@***.***> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134) FYI, I've confirmed that there is an issue with @SpikeLewis<https://github.com/SpikeLewis>'s passphrase. It works when exporting directly from Central, but not with ruODK. I will continue to investigate to see if we can get a repro. — Reply to this email directly, view it on GitHub<#134 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYTKCFH6JR74L7POHYMVB3LVFBMDDANCNFSM5SZKNHMQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[yanokwa]

I've now tried a project with 3000 forms with no success. I'm now thinking it might be a network issue with OP's connection.

[florianm]

Thanks both for the troubleshooting. Looking at the code for submission_export, there are some limitations in ruODK:

• Only the first encryption key is used
• If the server throws an internal server error (HTTP status 500), ruODK terminates the export as not to exhaust server memory: https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L183-L188 - seeing you've got a different error caught in the next command (stop_for_status) tells me we're not getting a HTTP status 500, so the encryption key seems to have worked.

@SpikeLewis some more diagnostic questions:

• Can you reproduce this error with the same form, same encryption method, and fewer submissions?
• Would you know how many different encryption keys your form uses?
• How long does submission_export run until you get the HTTP status 500?
• Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168

[florianm]

@SpikeLewis did you get to investigate this further?

[SpikeLewis]

Hi all, Apologies for the delay (appendix). The same exact code seems to be now working, with no errors (Yaw suggested it might be an internet stability thing). I'm now unable to replicate the errors (apologies for consuming any of your time on this). just to note: * Can you reproduce this error with the same form, same encryption method, and fewer submissions? * Yes on my dummy central project (with only 1 form and 2 submissions), the same form with fewer submissions worked (my real server has few submissions but 1000s of forms). * Would you know how many different encryption keys your form uses? * I have been using project level encryption via central. * How long does submission_exportrun until you get the HTTP status 500? * This took seconds (but now working) * Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168 * No I did not. Again thank you for your time, and fingers crossed for a continued stable network. Best, Spike *

…

________________________________ From: Florian Mayer ***@***.***> Sent: Wednesday, April 20, 2022 3:58 AM To: ropensci/ruODK ***@***.***> Cc: SpikeLewis ***@***.***>; Mention ***@***.***> Subject: Re: [ropensci/ruODK] issue accessing data decrypted at project level (Issue #134) Thanks both for the troubleshooting. Looking at the code for submission_export<https://github.com/ropensci/ruODK/blob/main/R/submission_export.R>, there are some limitations in ruODK: * Only the first encryption key is used * If the server throws an internal server error (HTTP status 500), ruODK terminates the export as not to exhaust server memory: https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L183-L188 - seeing you've got a different error caught in the next command (stop_for_status) tells me we're not getting a HTTP status 500, so the encryption key seems to have worked. @SpikeLewis<https://github.com/SpikeLewis> some more diagnostic questions: * Can you reproduce this error with the same form, same encryption method, and fewer submissions? * Would you know how many different encryption keys your form uses? * How long does submission_export run until you get the HTTP status 500? * Did you see this info message (with verbose=TRUE): https://github.com/ropensci/ruODK/blob/main/R/submission_export.R#L167-L168 — Reply to this email directly, view it on GitHub<#134 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AYTKCFHJAQ4AJBZEVOESCETVF5XE7ANCNFSM5SZKNHMQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[florianm]

Brilliant, thanks for the detailed answers!
Seeing the same ruODK code worked on different forms and occasions I doubt there's a bug in there. However, we are always at the mercy of the internet pigeons flying straight and true.
I'm closing this issue with the most likely cause being network timeouts. If the pigeons start missing home again feel free to re-open this issue or create a new one!