Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new script for benchmarking #157

Merged

Conversation

HealthyPear
Copy link
Member

@HealthyPear HealthyPear commented Aug 25, 2021

Requirements

Description

This PR adds a script that can be called as protopipe-BENCHMARK which has 2 commands,

  • list, which lists all available benchmark notebooks under docs/contribute/benchmarks,
  • launch, that allows to launch 1 notebook via papermill (which is here added as a development dependency) and produce not only its executed version but also an HTML page.

Using papermill allows to have default variables for input and optimizations, but the script can also read them from a configuration file bechmarks.yaml which should be copied with all the other ones by the interface (under testing in this PR).
Variables can be also overwritten on the go via the CLI flag --kwargs.

At the time of the first launch, a Jupyter Book template is filled, which pages are the single notebooks.
Such books are supposed to populate the new repository for the storage of performances.

TODO

@HealthyPear HealthyPear added documentation Documentation or services hosting it enhancement New feature or request benchmarks labels Aug 25, 2021
@codecov
Copy link

codecov bot commented Aug 30, 2021

Codecov Report

Merging #157 (2e300aa) into master (f5827fc) will decrease coverage by 1.90%.
The diff coverage is 1.09%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #157      +/-   ##
==========================================
- Coverage   64.95%   63.05%   -1.91%     
==========================================
  Files          27       28       +1     
  Lines        2802     2885      +83     
==========================================
- Hits         1820     1819       -1     
- Misses        982     1066      +84     
Impacted Files Coverage Δ
protopipe/scripts/data_training.py 91.83% <ø> (ø)
protopipe/scripts/launch_benchmark.py 0.00% <0.00%> (ø)
protopipe/scripts/write_dl2.py 80.76% <ø> (ø)
protopipe/pipeline/utils.py 48.06% <100.00%> (+1.25%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f5827fc...2e300aa. Read the comment docs.

@HealthyPear HealthyPear mentioned this pull request Oct 1, 2021
5 tasks
@HealthyPear HealthyPear added this to the v0.5.0 milestone Feb 1, 2022
@HealthyPear HealthyPear linked an issue Feb 16, 2022 that may be closed by this pull request
- upgraded script
- updated installation requirements
kosack
kosack previously approved these changes Mar 1, 2022
Copy link
Contributor

@kosack kosack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@HealthyPear HealthyPear marked this pull request as ready for review March 2, 2022 08:29
@HealthyPear HealthyPear requested a review from kosack March 2, 2022 08:30
@HealthyPear HealthyPear merged commit e80fe30 into cta-observatory:master Mar 2, 2022
@HealthyPear HealthyPear deleted the feature-add_protopipe_BENCHMARK branch March 2, 2022 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmarks documentation Documentation or services hosting it enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add analysis book template
2 participants