Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ncbi_searcher Error in taxonomy(zz = outsum) : Too Many Requests (RFC 6585) (HTTP 429). #120

Closed
mpnelsen opened this issue Aug 17, 2020 · 8 comments
Labels
Milestone

Comments

@mpnelsen
Copy link

mpnelsen commented Aug 17, 2020

Hi,

I'm trying to conduct searches for accessions derived from a single gene in a single genus, but am getting an error (below). I'm wondering if there might be a way I can slow down the taxonomy search inside the function to circumvent this issue? Thanks for any suggestions!

-Matt.

api_key<-"xxxxxxx"
set_entrez_key(api_key)
search.res<-ncbi_searcher(taxa="Aster",seqrange="350:5000",verbose=TRUE,fuzzy=TRUE,entrez_query="rbcL")
══ 1 queries ═══════════════

Retrieving data for taxon 'Aster'

✔ Found: Aster
══ Results ═════════════════

● Total: 1
● Found: 1
● Not Found: 0
Working on Aster...
...retrieving sequence IDs...
...retrieving available genes and their lengths...
Error in taxonomy(zz = outsum) : Too Many Requests (RFC 6585) (HTTP 429).

traceback()
7: stop(http_condition(x, "error", task = task, call = call))
6: stop_for_status(res)
5: taxonomy(zz = outsum)
4: parseres(query_res, hypothetical)
3: download_summary(seq_ids, hypothetical = hypothetical, ...)
2: ncbi_searcher_foo(id, getrelated = getrelated, verbose = verbose,
seqrange = seqrange, entrez_query = entrez_query, fuzzy = fuzzy,
limit = limit, hypothetical = hypothetical)
1: ncbi_searcher(taxa = "Aster", seqrange = "350:5000", verbose = TRUE,
fuzzy = TRUE, entrez_query = "rbcL")

Session Info
> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS

Matrix products: default
BLAS/LAPACK: /home/xxxxxxxxx

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] traits_0.4.2   devtools_2.2.2 usethis_1.5.1 

loaded via a namespace (and not attached):
 [1] zoo_1.8-7         remotes_2.1.1     taxize_0.9.97     lattice_0.20-40  
 [5] vctrs_0.3.1       testthat_2.3.2    rlang_0.4.7       pkgbuild_1.1.0   
 [9] pillar_1.4.4      httpcode_0.3.0    glue_1.4.1        withr_2.2.0      
[13] rappdirs_0.3.1    sessioninfo_1.1.1 uuid_0.1-4        foreach_1.4.8    
[17] lifecycle_0.2.0   plyr_1.8.6        stringr_1.4.0     rvest_0.3.6      
[21] codetools_0.2-16  memoise_1.1.0     callr_3.4.3       ps_1.3.3         
[25] curl_4.3          parallel_3.6.1    hoardr_0.5.2      fansi_0.4.1      
[29] Rcpp_1.0.5        readr_1.3.1       backports_1.1.8   desc_1.2.0       
[33] pkgload_1.1.0     jsonlite_1.6.1    fs_1.3.2          hms_0.5.3        
[37] digest_0.6.25     stringi_1.4.6     processx_3.4.3    rprojroot_1.3-2  
[41] grid_3.6.1        cli_2.0.2         tools_3.6.1       magrittr_1.5     
[45] tibble_3.0.2      bold_1.1.0        crul_1.0.0        crayon_1.3.4     
[49] ape_5.4           pkgconfig_2.0.3   ellipsis_0.3.1    data.table_1.12.8
[53] xml2_1.2.2        prettyunits_1.1.1 assertthat_0.2.1  reshape_0.8.8    
[57] httr_1.4.1        iterators_1.0.12  R6_2.4.1          conditionz_0.1.0 
[61] nlme_3.1-144      compiler_3.6.1   
@sckott
Copy link
Contributor

sckott commented Aug 17, 2020

thanks, having a look

@sckott sckott added the bug label Aug 17, 2020
@sckott sckott added this to the v0.5 milestone Aug 17, 2020
@sckott sckott closed this as completed in 8aab069 Aug 17, 2020
@sckott
Copy link
Contributor

sckott commented Aug 17, 2020

@mpnelsen reinstall remotes::install_github("ropensci/traits") and try again, we weren't sending the Entrez API key along with requests to NCBI.

@mpnelsen
Copy link
Author

Thanks a ton, @sckott

@mpnelsen
Copy link
Author

mpnelsen commented Aug 17, 2020

Actually...I'm still getting an error thrown after the update...

api_key<-"xxxxxxx"
set_entrez_key(api_key)
search.res<-ncbi_searcher(taxa="Aster",seqrange="350:5000",verbose=TRUE,fuzzy=TRUE,entrez_query="rbcL")
══ 1 queries ═══════════════

Retrieving data for taxon 'Aster'

✔ Found: Aster
══ Results ═════════════════

● Total: 1
● Found: 1
● Not Found: 0
Working on Aster...
...retrieving sequence IDs...
...retrieving available genes and their lengths...
Error: Too Many Requests (HTTP 429)

traceback()
7: stop(sprintf("%s (HTTP %s)", x$message, x$status_code), call. = FALSE)
6: res$raise_for_status()
5: taxonomy(zz = outsum)
4: parseres(res, hypothetical)
3: download_summary(seq_ids, hypothetical = hypothetical, ...)
2: ncbi_searcher_foo(id, getrelated = getrelated, verbose = verbose,
seqrange = seqrange, entrez_query = entrez_query, fuzzy = fuzzy,
limit = limit, hypothetical = hypothetical)
1: ncbi_searcher(taxa = "Aster", seqrange = "350:5000", verbose = TRUE,
fuzzy = TRUE, entrez_query = "rbcL")

@sckott
Copy link
Contributor

sckott commented Aug 17, 2020

did you restart R?

turn on verbose curl output to make sure the key is being sent. you should see it as a query parameter.

crul::set_verbose()
ncbi_searcher(...)

@mpnelsen
Copy link
Author

mpnelsen commented Aug 17, 2020

I did...and it looks like it's sending it along (changed to a string of x's here). Thanks much for your help.

library(traits)
Registered S3 method overwritten by 'hoardr':
method from
print.cache_info httr
library(rentrez)
api_key<-"xxxxxxxxx"
set_entrez_key(api_key)
crul::set_verbose()

search.res<-ncbi_searcher(taxa="Aster",seqrange="350:5000",verbose=TRUE,fuzzy=TRUE,entrez_query="rbcL")
══ 1 queries ═══════════════

Retrieving data for taxon 'Aster'

✔ Found: Aster
══ Results ═════════════════

● Total: 1
● Found: 1
● Not Found: 0
Working on Aster...
...retrieving sequence IDs...
=> GET /entrez/eutils/esearch.fcgi?db=nuccore&retmax=500&term=xXarbitraryXx%5Bporgn%3A__txid41479%5D%20AND%20350%3A5000%5BSLEN%5D%20AND%20rbcL&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 8
<= NCBI-PHID: 939B08B8DE1D889500002DFAEB0E37E3.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: CB11C7035279321A_9371SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=CB11C7035279321A_9371SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:46 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
...retrieving available genes and their lengths...
=> POST /entrez/eutils/esummary.fcgi HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=> Content-Length: 1236
=> Content-Type: multipart/form-data; boundary=------------------------ef48f488793a79e9
=>

--------------------------ef48f488793a79e9
Content-Disposition: form-data; name="db"

nucleotide
--------------------------ef48f488793a79e9
Content-Disposition: form-data; name="api_key"

xxxxxxxxx
--------------------------ef48f488793a79e9
Content-Disposition: form-data; name="id"

1677621541 1677621441 1677620947 1677620639 1677619655 1677619383 1677619133 1445137842 1445137840 1445137838 1445137836 1409006572 1389385086 1351233385 1328838772 1316025274 1316024818 1316023806 1316022968 1316021096 1284846308 399139827 1243024303 1063823660 1048218690 1048218688 1048218686 1046840259 313770804 313770802 290585993 290585959 667668116 685425357 685425355 685425353 685425351 685425349 685425347 685425345 685425343 685425341 685425339 685425337 685425335 685425333 685425331 685425329 685425327 685425325 685425323 685425321 685425319 685425317 685425315 685425313 685425311 685425309 685425307 685425305 685425303 685425301 685425299 685425297 685425295 685425293 685425291 685425289 685425287 685425285 685425283 685425281 685425279 685425277 685425275 685425273 685425271 685425269 335353484 557836626 459927280 306482374 306482372 125857596
--------------------------ef48f488793a79e9--

<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 7
<= NCBI-PHID: 939B08B8DE1D8895000049FAEC68837C.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 0250E390F08314AE_4B34SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=0250E390F08314AE_4B34SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:47 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=588669&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 8
<= NCBI-PHID: 939B08B8DE1D8895000042FAECF91351.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 711D1CA8017D53A6_CC76SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=711D1CA8017D53A6_CC76SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:47 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=714451&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 7
<= NCBI-PHID: 939B08B8DE1D889500004EFAED009AB9.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 018D7022D47CF466_A496SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=018D7022D47CF466_A496SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=2291102&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 6
<= NCBI-PHID: 939B08B8DE1D8895000026FAED107C21.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 543862F9D31251F9_C454SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=543862F9D31251F9_C454SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=616994&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 5
<= NCBI-PHID: 939B08B8DE1D889500003FFAED21FB6A.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 8433E5333566D958_AB8CSID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=8433E5333566D958_AB8CSID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=1196498&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 4
<= NCBI-PHID: 939B08B8DE1D8895000053FAED2D8647.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: F0A3884FCB3EFBF8_6316SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=F0A3884FCB3EFBF8_6316SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=1196492&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 3
<= NCBI-PHID: 939B08B8DE1D8895000021FAED37CB39.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: FE40B0B1872E7134_CE72SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=FE40B0B1872E7134_CE72SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=2294043&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:48 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 2
<= NCBI-PHID: 939B08B8DE1D889500002FFAED40B853.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 0DA1F85AF605F7FC_F41CSID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=0DA1F85AF605F7FC_F41CSID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=983245&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:48 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 1
<= NCBI-PHID: 939B08B8DE1D889500003CFAED494366.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: CD8CBB83F7B1430A_BA10SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=CD8CBB83F7B1430A_BA10SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=415155&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 200 OK
<= Date: Mon, 17 Aug 2020 22:06:48 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 0
<= NCBI-PHID: 939B08B8DE1D8895000045FAED513EC4.1.1.m_1
<= Cache-Control: private
<= NCBI-SID: 4DD3914E484C1F99_BB79SID
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= Access-Control-Allow-Origin: *
<= Content-Type: text/xml; charset=UTF-8
<= Set-Cookie: ncbi_sid=4DD3914E484C1F99_BB79SID; domain=.nih.gov; path=/; expires=Tue, 17 Aug 2021 22:06:48 GMT
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<= Transfer-Encoding: chunked
<=
=> GET /entrez/eutils/esummary.fcgi?db=taxonomy&id=2065853&api_key=xxxxxxxxx HTTP/1.1
=> Host: eutils.ncbi.nlm.nih.gov
=> User-Agent: libcurl/7.65.3 r-curl/4.3 crul/1.0.0
=> Accept-Encoding: gzip, deflate
=> Accept: application/json, text/xml, application/xml, /
=>
<= HTTP/1.1 429 Too Many Requests
<= Date: Mon, 17 Aug 2020 22:06:47 GMT
<= Server: Finatra
<= Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
<= Content-Security-Policy: upgrade-insecure-requests
<= X-RateLimit-Remaining: 0
<= content-encoding: gzip
<= X-RateLimit-Limit: 10
<= content-length: 128
<= Content-Type: application/json
<= Retry-After: 2
<= X-UA-Compatible: IE=Edge
<= X-XSS-Protection: 1; mode=block
<=
Error: Too Many Requests (HTTP 429)

traceback()
7: stop(sprintf("%s (HTTP %s)", x$message, x$status_code), call. = FALSE)
6: res$raise_for_status()
5: taxonomy(zz = outsum)
4: parseres(res, hypothetical)
3: download_summary(seq_ids, hypothetical = hypothetical, ...)
2: ncbi_searcher_foo(id, getrelated = getrelated, verbose = verbose,
seqrange = seqrange, entrez_query = entrez_query, fuzzy = fuzzy,
limit = limit, hypothetical = hypothetical)
1: ncbi_searcher(taxa = "Aster", seqrange = "350:5000", verbose = TRUE,
fuzzy = TRUE, entrez_query = "rbcL")

sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-conda_cos6-linux-gnu (64-bit)
Running under: Ubuntu 18.04.4 LTS

Matrix products: default
BLAS/LAPACK: /home/...

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] rentrez_1.2.2 traits_0.4.2.93

loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 pillar_1.4.4 compiler_3.6.1 plyr_1.8.6
[5] iterators_1.0.12 tools_3.6.1 digest_0.6.25 uuid_0.1-4
[9] jsonlite_1.6.1 lifecycle_0.2.0 tibble_3.0.2 nlme_3.1-144
[13] lattice_0.20-40 pkgconfig_2.0.3 rlang_0.4.7 foreach_1.4.8
[17] cli_2.0.2 crul_1.0.0 curl_4.3 parallel_3.6.1
[21] httr_1.4.1 stringr_1.4.0 xml2_1.2.2 rappdirs_0.3.1
[25] vctrs_0.3.1 hms_0.5.3 triebeard_0.3.0 grid_3.6.1
[29] reshape_0.8.8 glue_1.4.1 httpcode_0.3.0 data.table_1.12.8
[33] R6_2.4.1 fansi_0.4.1 conditionz_0.1.0 XML_3.99-0.3
[37] hoardr_0.5.2 readr_1.3.1 magrittr_1.5 urltools_1.7.3
[41] codetools_0.2-16 ellipsis_0.3.1 assertthat_0.2.1 bold_1.1.0
[45] rvest_0.3.6 taxize_0.9.97 ape_5.4 stringi_1.4.6
[49] crayon_1.3.4 zoo_1.8-7

@sckott
Copy link
Contributor

sckott commented Aug 17, 2020

change in 42148b5

@mpnelsen reinstall again. added a sleep parameter to this fxn, e.g, i tried it with your eg and with sleep=2L and it kept the rate remaining at 9 the whole time.

also i wonder if we have different rate limits b/c i wasn't hitting the 429 error you hit. anyway, this should do it i hope

@mpnelsen
Copy link
Author

That did it. Thx a lot, @sckott

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants