Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General benchmarking framework #86

Merged
merged 9 commits into from
Dec 29, 2023
Merged

General benchmarking framework #86

merged 9 commits into from
Dec 29, 2023

Conversation

b-fg
Copy link
Member

@b-fg b-fg commented Dec 15, 2023

Automated benchmarks generation to improve in reproducibility, and profiling of new code.
When completed, it should be able to automatically run and compare benchmarks across multiple:

  • Julia versions
  • Backends
  • Cases (including size, simulation time/max. steps, data type)

@b-fg b-fg self-assigned this Dec 15, 2023
Copy link

codecov bot commented Dec 15, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ff83ce9) 89.47% compared to head (ba88cc9) 91.07%.
Report is 11 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master      #86      +/-   ##
==========================================
+ Coverage   89.47%   91.07%   +1.60%     
==========================================
  Files           9        9              
  Lines         399      538     +139     
==========================================
+ Hits          357      490     +133     
- Misses         42       48       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@weymouth
Copy link
Collaborator

Looks cool. A couple things.

  • We probably need to add documentation on how to use this.
  • This adds dependencies which aren't needed for just running the code. Can we make these optional by including them as part of a new target? (I've been thinking we should try to reduce/organize our dependencies better.)

@b-fg
Copy link
Member Author

b-fg commented Dec 16, 2023

Yes, I was planning to look into better handling the extra dependencies too. And documentation will also come in due course :)

…e configuration, backends.

Reverted the dependencies added in WaterLily, and these are now included in the benchmarks environment.
To run the benchmarks, the benchmarks environment is activated, while WaterLily is marked as a development package.
@b-fg
Copy link
Member Author

b-fg commented Dec 19, 2023

Pretty much done :) Feel free to give it a try!

  • Documentation can be found within the benchmark.sh file.
  • Fixed project dependencies by moving them to a subproject environment.

Output example on the TGV case:

@b-fg b-fg changed the title General benchmarking framework [WIP] General benchmarking framework Dec 19, 2023
benchmark/benchmark.sh Outdated Show resolved Hide resolved
benchmark/benchmark.sh Outdated Show resolved Hide resolved
benchmark/benchmark.sh Outdated Show resolved Hide resolved
benchmark/benchmark.sh Outdated Show resolved Hide resolved
benchmark/benchmark.sh Outdated Show resolved Hide resolved
b-fg and others added 3 commits December 28, 2023 18:20
Co-authored-by: Mosè Giordano <[email protected]>
 - juliaup now uses the +{version} approach instead of modifying the default juliaup version
 - Pkg.develop now uses dirname(@__DIR__)

Also moved docs to the new README.md file to improve readability.
@b-fg
Copy link
Member Author

b-fg commented Dec 29, 2023

I think this is finished @weymouth. We can add more benchmark examples in the future, additionally to the tgv.jl.
After merging this PR, and figuring out optional dependencies we could release the minor update.

@weymouth weymouth merged commit 6092807 into master Dec 29, 2023
14 checks passed
@b-fg b-fg deleted the benchmarks branch December 31, 2023 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants