Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Julia] Stablize the results #400

Merged
merged 1 commit into from
Nov 2, 2023
Merged

[Julia] Stablize the results #400

merged 1 commit into from
Nov 2, 2023

Conversation

Moelf
Copy link
Contributor

@Moelf Moelf commented Nov 1, 2023 •
edited
Loading

tested on the same VM. I now suspect the vGPU has some policy if it sees you idle it will start shifting resource away from your VM or something

before:

Benchmark 1: julia --project=Related -e "using Related; main()"
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 23 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 25 milliseconds
Processing time (w/o IO): 30 milliseconds
Processing time (w/o IO): 31 milliseconds
Processing time (w/o IO): 25 milliseconds

after:

Benchmark 1: julia --project=Related -e "using Related; main()"
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 25 milliseconds
Processing time (w/o IO): 25 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 24 milliseconds
Processing time (w/o IO): 25 milliseconds
Processing time (w/o IO): 24 milliseconds

Moelf
Moelf commented Nov 1, 2023
View reviewed changes
julia/Related/src/Related.jl
@@ -98,4 +98,6 @@ function main()
end
end

main()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried on the VM, this is the only way to precompile without cause performance loss

Moelf
Moelf commented Nov 1, 2023
View reviewed changes
run.sh
@@ -260,9 +260,9 @@ run_julia() {
julia -e 'using Pkg; Pkg.activate("Related"); Pkg.instantiate()' &&
if [ $HYPER == 1 ]; then

capture "Julia" hyperfine -r $runs -w $warmup --show-output "julia --project=Related -e \"using Related; main()\""
capture "Julia" hyperfine -r $runs -w $warmup --show-output "julia --startup-file=no --project=Related -e \"using Related; main()\""
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not really needed but it helps with local development because often a Julia user would have startup-files

@Tortar
Copy link
Contributor

Tortar commented Nov 1, 2023 •
edited
Loading

Interesting...but this doesn't solve actually the problem on my computer...in the 20k posts enabled, this is what I see:

bob@bob-Victus-by-HP-Gaming-Laptop-15-fb0xxx:~$ julia # still needs precompilation

julia> using Related
[ Info: Precompiling Related [892d0954-52c7-4a59-ae8f-9a164aba3227]
Processing time (w/o IO): 4586 milliseconds
julia> main()
Processing time (w/o IO): 267 milliseconds # it went good, consistently these results
13262912
julia> main()
Processing time (w/o IO): 272 milliseconds
13262912
julia> main()
Processing time (w/o IO): 265 milliseconds
13262912
bob@bob-Victus-by-HP-Gaming-Laptop-15-fb0xxx:~$ julia # restart

julia> using Related
julia> main()
Processing time (w/o IO): 268 milliseconds
13262912
julia> main()
Processing time (w/o IO): 269 milliseconds
13262912
julia> main()
Processing time (w/o IO): 269 milliseconds
13262912

bob@bob-Victus-by-HP-Gaming-Laptop-15-fb0xxx:~$ julia # just added a space in the related.jl file to trigger precompilation

julia> using Related
[ Info: Precompiling Related [892d0954-52c7-4a59-ae8f-9a164aba3227]
Processing time (w/o IO): 4669 milliseconds
julia> main()
Processing time (w/o IO): 361 milliseconds # it went bad
13262912
julia> main()
Processing time (w/o IO): 364 milliseconds
13262912
julia> main()
Processing time (w/o IO): 360 milliseconds
13262912
julia> main()
Processing time (w/o IO): 360 milliseconds
13262912

bob@bob-Victus-by-HP-Gaming-Laptop-15-fb0xxx:~$ julia

julia> using Related
julia> main()
Processing time (w/o IO): 360 milliseconds # forever bad
13262912
julia> main()
Processing time (w/o IO): 361 milliseconds
13262912
julia> main()
Processing time (w/o IO): 358 milliseconds
13262912

bob@bob-Victus-by-HP-Gaming-Laptop-15-fb0xxx:~$ julia

julia> using Related
julia> main()
Processing time (w/o IO): 361 milliseconds
13262912
julia> main()
Processing time (w/o IO): 367 milliseconds
13262912

@Tortar
Copy link
Contributor

Tortar commented Nov 1, 2023

notice the comments in the REPL output

@Tortar
Copy link
Contributor

Tortar commented Nov 1, 2023

To me seems like some sort of bug in precompilation, but maybe I'm wrong

@Moelf
Copy link
Contributor Author

Moelf commented Nov 1, 2023

Or maybe hyperfine warmup round is weird

@Tortar
Copy link
Contributor

Tortar commented Nov 1, 2023

on my pc I'm not using that machinery actually, I'm just running the Related package from the REPL, so I think it should be unrelated

@jinyus
Copy link
Owner

jinyus commented Nov 2, 2023

Main:
Julia | 27.85 ms | 442.00 ms | 3.88 s

This PR:

Julia:

        Benchmark 1: julia --startup-file=no --project=Related -e "using Related; main()"
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
        Processing time (w/o IO): 24 milliseconds
          Time (mean ± σ):      1.005 s ±  0.019 s    [User: 0.889 s, System: 0.220 s]
          Range (min … max):    0.982 s …  1.028 s    10 runs
         
Julia:

        Benchmark 1: julia --startup-file=no --project=Related -e "using Related; main()"
        Processing time (w/o IO): 363 milliseconds
        Processing time (w/o IO): 363 milliseconds
        Processing time (w/o IO): 362 milliseconds
          Time (mean ± σ):      1.458 s ±  0.022 s    [User: 1.322 s, System: 0.240 s]
          Range (min … max):    1.443 s …  1.473 s    2 runs
         
Julia:

        Benchmark 1: julia --startup-file=no --project=Related -e "using Related; main()"
        Processing time (w/o IO): 3168 milliseconds
        Processing time (w/o IO): 3164 milliseconds
        Processing time (w/o IO): 3165 milliseconds
          Time (mean ± σ):      4.629 s ±  0.027 s    [User: 4.429 s, System: 0.294 s]
          Range (min … max):    4.610 s …  4.647 s    2 runs

@jinyus jinyus merged commit 8cd2a27 into jinyus:main Nov 2, 2023
@Moelf Moelf deleted the stablize_julia branch November 2, 2023 06:59
@Moelf Moelf mentioned this pull request Nov 2, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Moelf @Tortar @jinyus