Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set joblib compress to 3 #947

Merged
merged 1 commit into from
Mar 21, 2022
Merged

set joblib compress to 3 #947

merged 1 commit into from
Mar 21, 2022

Conversation

vuillaut
Copy link
Member

We don't use any compression for the RF dump on disk.

I picked compress=3 from a quick study (see below) that seems to give a good compromise: 1.2s --> 12.1s to dump, 1.5-->3.8s to read, 788MB-->179MB on disk

Study:

In [13]: for i in range(9):
    ...:     print(i)
    ...:     %time joblib.dump(disp_sign,f'disp_sign_{i}.sav', compress=i)
    ...: 
0
CPU times: user 339 ms, sys: 890 ms, total: 1.23 s
Wall time: 1.41 s
1
CPU times: user 8.62 s, sys: 967 ms, total: 9.58 s
Wall time: 9.59 s
2
CPU times: user 9.11 s, sys: 599 ms, total: 9.71 s
Wall time: 9.72 s
3
CPU times: user 11.6 s, sys: 530 ms, total: 12.1 s
Wall time: 12.1 s
4
CPU times: user 12.7 s, sys: 504 ms, total: 13.2 s
Wall time: 13.2 s
5
CPU times: user 16.8 s, sys: 469 ms, total: 17.3 s
Wall time: 17.3 s
6
CPU times: user 28 s, sys: 517 ms, total: 28.5 s
Wall time: 28.5 s
7
CPU times: user 41.3 s, sys: 604 ms, total: 41.9 s
Wall time: 41.9 s
8
CPU times: user 2min 12s, sys: 839 ms, total: 2min 13s
Wall time: 2min 13s
In [15]: for i in range(9):
    ...:     print(i)
    ...:     %time joblib.load(f'disp_sign_{i}.sav')
    ...: 
0
CPU times: user 639 ms, sys: 894 ms, total: 1.53 s
Wall time: 1.53 s
1
CPU times: user 3.62 s, sys: 631 ms, total: 4.25 s
Wall time: 4.26 s
2
CPU times: user 3.39 s, sys: 475 ms, total: 3.87 s
Wall time: 3.87 s
3
CPU times: user 3.28 s, sys: 488 ms, total: 3.77 s
Wall time: 3.77 s
4
CPU times: user 3.35 s, sys: 496 ms, total: 3.85 s
Wall time: 3.85 s
5
CPU times: user 3.24 s, sys: 461 ms, total: 3.7 s
Wall time: 3.7 s
6
CPU times: user 3.12 s, sys: 507 ms, total: 3.62 s
Wall time: 3.62 s
7
CPU times: user 3.14 s, sys: 474 ms, total: 3.62 s
Wall time: 3.62 s
8
CPU times: user 3.09 s, sys: 452 ms, total: 3.55 s
Wall time: 3.55 s
In [17]: ls -sh disp_sign*.sav
788M disp_sign_0.sav  
198M disp_sign_1.sav
187M disp_sign_2.sav 
179M disp_sign_3.sav 
181M disp_sign_4.sav
172M disp_sign_5.sav
167M disp_sign_6.sav 
166M disp_sign_7.sav
162M disp_sign_8.sav

@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@codecov
Copy link

codecov bot commented Mar 21, 2022

Codecov Report

Merging #947 (b856209) into master (2271df3) will decrease coverage by 0.03%.
The diff coverage is 44.44%.

@@            Coverage Diff             @@
##           master     #947      +/-   ##
==========================================
- Coverage   85.53%   85.50%   -0.04%     
==========================================
  Files          78       78              
  Lines        6464     6464              
==========================================
- Hits         5529     5527       -2     
- Misses        935      937       +2     
Impacted Files Coverage Δ
lstchain/reco/dl1_to_dl2.py 74.46% <0.00%> (ø)
lstchain/tests/test_lstchain.py 96.87% <100.00%> (ø)
lstchain/reco/r0_to_dl1.py 93.08% <0.00%> (-0.63%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2271df3...b856209. Read the comment docs.

@vuillaut vuillaut merged commit 290f895 into master Mar 21, 2022
@vuillaut vuillaut deleted the joblib_compress branch March 21, 2022 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants