Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LORIS pipelines to crawl LORIS' projects and candidates endpoints #103

Closed
wants to merge 4 commits into from

Conversation

cmadjar
Copy link

@cmadjar cmadjar commented Apr 30, 2021

Description

This is a refactoring of the scripts that were present in #102 and #13. One of the pipeline present in #67 has also been added to the PR but not changed.

It contains two files:

  • loris-projects-api.py: to query the different LORIS endpoints specific to projects (a.k.a. under /projects/) - still needs work
  • loris-candidate-api.py: to query the LORIS endpoints specific to a candidate (a.k.a. under /candidates/) - for now this is an exact copy of what was in Add template for Loris Candidate API #67 that was submitted by @mathdugre a while back

@codecov
Copy link

codecov bot commented Apr 30, 2021

Codecov Report

Merging #103 (b313073) into master (7905869) will decrease coverage by 2.20%.
The diff coverage is 22.22%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #103      +/-   ##
==========================================
- Coverage   81.33%   79.12%   -2.21%     
==========================================
  Files          57       59       +2     
  Lines        4650     4829     +179     
==========================================
+ Hits         3782     3821      +39     
- Misses        868     1008     +140     
Impacted Files Coverage Δ
datalad_crawler/pipelines/loris-projects-api.py 15.10% <15.10%> (ø)
datalad_crawler/pipelines/loris-candidate-api.py 45.00% <45.00%> (ø)
datalad_crawler/version.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03710b8...b313073. Read the comment docs.

@cmadjar
Copy link
Author

cmadjar commented Apr 30, 2021

@yarikoptic I think you can close #13 since the loris-project-api.py script was based out of it and has been refactored multiple times since the PR was sent. The last refactoring was to try to render it more flexible so that we do not have to create a new pipeline every time we need to query a different projects LORIS endpoint, which is what we've been doing so far...

#67 can probably be closed as well if @mathdugre is happy to have his script ported in this PR. As you guys wish.

If you have initial feedbacks, don't hesitate. The loris-project-api.py pipeline is still a work in progress.

…roject crawlers.

Could not test unfortunately...
Facing an eternal loop when I try to login even though password, URL and username are all correct (can curl).
Looking at the datalad code, it seems like when we get into the __init__ function of the class
LORIS_Token, keyring is empty when I try to print it on the console. Could this be why???
Line of code is: https://github.com/datalad/datalad/blob/4f8fe6e4a8a4113501db418d3942930ac8977764/datalad/downloaders/credentials.py#L342
@yarikoptic
Copy link
Member

Per @mih insistence of seeking for ultimate purity in all of our lives, I am closing this PR due to being inactive for considerable amount of time. But "closed" is not "gone forever". If there is an interest to resurrect this, or alternative/complimentary #13 -- just let us know.

@yarikoptic yarikoptic closed this Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants