Fix or suppress some noisy tests 🏌️‍♂️ #44444

IanButterworth · 2022-03-04T04:31:17Z

The noise from these in the CI logs seemed unnecessary

stdlib/REPL/test/repl.jl

DilumAluthge · 2022-03-04T06:08:48Z

Are you saying we should forgo printing the output from these tests?

timholy · 2022-03-04T08:53:47Z

Are you saying we should forgo printing the output from these tests?

The large majority of tests are silent, and that's good because when there is a problem it's easy to find in the log.

Thanks for taming the stragglers!

DilumAluthge · 2022-03-04T08:55:07Z

I worry that my comment about FORgoing the output was perhaps too subtle.

DilumAluthge · 2022-03-04T08:55:19Z

Or should I say, FOURgoing?

test/backtrace.jl

IanButterworth · 2022-03-04T15:26:08Z

@timholy is this one noise? I'm guessing not because the test that should trigger that catches the log, so this seems like it's an error?

loading                                  (2) |        started at 2022-03-04T07:36:51.142
      From worker 2:	��� Error: active project callback #19 failed
      From worker 2:	��� @ Base initdefs.jl:325

timholy · 2022-03-04T17:51:25Z

It is noise. That said, we could also remove the deliberately-errorring callback earlier in the test. See

julia/test/loading.jl

Lines 229 to 236 in b63ae3b

    
           push!(Base.active_project_callbacks, () -> watcher_counter[] += 1) 
        
           push!(Base.active_project_callbacks, () -> error("broken")) 
        
           push!(empty!(LOAD_PATH), joinpath(@__DIR__, "project")) 
        
           append!(empty!(DEPOT_PATH), [mktempdir(), joinpath(@__DIR__, "depot")]) 
        
           @test watcher_counter[] == 0 
        
           @test_logs (:error, r"active project callback .* failed") Base.set_active_project(nothing) 
        
           @test watcher_counter[] == 1

and

julia/test/loading.jl

Line 735 in b63ae3b

for _ = 1:2 pop!(Base.active_project_callbacks) end

You could pop! the second callback off after the first block, as long as you get rid of the loop in the second block.

Thanks again for doing this!

IanButterworth · 2022-03-05T12:55:37Z

Ok, I think this is ready to go.

There's room for improvement in the way the logs are suppressed in Pidfile, as I ended up just using a NullLogger because I couldn't figure out which lines were causing the logs, such that a @test_logs could be used. Perhaps you have a better idea of the locations @vtjnash (I added a TODO)

I marked this for 1.8 backport, but it will need manual intervention because Pidfile isn't on 1.8

IanButterworth · 2022-03-05T20:35:33Z

I just found another case in LibGit2 where Base.runtests wasn't able to intercept the printing of testset reports because they were generated in a subprocess. Changing them to begin blocks is a simple fix, otherwise each test file would need to know how to suppress them.

IanButterworth · 2022-03-05T21:26:18Z

This has grown a bit and is now in a place where I believe all unnecessary noise is fixed or suppressed, with the help of:

Move threaded testsets to main process to reduce Base test noise JuliaSparse/SuiteSparse.jl#63
Switch info for debug log JuliaCrypto/SHA.jl#84 (bumped in this PR)

Given its size I'd appreciate a review, so requested you @timholy if you don't mind

(cherry picked from commit 7287e64)

timholy

LGTM! Thanks for doing this, we'll all appreciate it.

stdlib/FileWatching/test/pidfile.jl

test/boundscheck_exec.jl

test/loading.jl

test/misc.jl

(cherry picked from commit d971465)

This warning was un-ironically introduced by "Fix or suppress some noisy tests". ``` ┌ Error: mktemp cleanup │ exception = │ IOError: unlink("C:\\Windows\\TEMP\\jl_A49B.tmp"): resource busy or locked (EBUSY) │ Stacktrace: │ [1] uv_error │ @ .\libuv.jl:100 [inlined] │ [2] unlink(p::String) │ @ Base.Filesystem .\file.jl:968 │ [3] rm(path::String; force::Bool, recursive::Bool) │ @ Base.Filesystem .\file.jl:283 │ [4] rm │ @ .\file.jl:274 [inlined] │ [5] mktemp(fn::Main.Test49Main_misc.var"#110#113", parent::String) │ @ Base.Filesystem .\file.jl:736 │ [6] mktemp(fn::Function) │ @ Base.Filesystem .\file.jl:730 │ [7] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1065 [inlined] │ [8] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [9] top-level scope │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1060 │ [10] include │ @ .\Base.jl:427 [inlined] │ [11] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:24 [inlined] │ [12] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [13] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:23 [inlined] │ [14] macro expansion │ @ .\timing.jl:440 [inlined] │ [15] runtests(name::String, path::String, isolate::Bool; seed::UInt128) │ @ Main C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:21 │ [16] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}}) │ @ Base .\essentials.jl:798 │ [17] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})() │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 │ [18] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool) │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:70 │ [19] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 [inlined] │ [20] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})() │ @ Distributed .\task.jl:490 └ @ Base.Filesystem file.jl:738 ```

This warning was un-ironically introduced by "Fix or suppress some noisy tests". ``` ┌ Error: mktemp cleanup │ exception = │ IOError: unlink("C:\\Windows\\TEMP\\jl_A49B.tmp"): resource busy or locked (EBUSY) │ Stacktrace: │ [1] uv_error │ @ .\libuv.jl:100 [inlined] │ [2] unlink(p::String) │ @ Base.Filesystem .\file.jl:968 │ [3] rm(path::String; force::Bool, recursive::Bool) │ @ Base.Filesystem .\file.jl:283 │ [4] rm │ @ .\file.jl:274 [inlined] │ [5] mktemp(fn::Main.Test49Main_misc.var"#110#113", parent::String) │ @ Base.Filesystem .\file.jl:736 │ [6] mktemp(fn::Function) │ @ Base.Filesystem .\file.jl:730 │ [7] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1065 [inlined] │ [8] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [9] top-level scope │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1060 │ [10] include │ @ .\Base.jl:427 [inlined] │ [11] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:24 [inlined] │ [12] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [13] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:23 [inlined] │ [14] macro expansion │ @ .\timing.jl:440 [inlined] │ [15] runtests(name::String, path::String, isolate::Bool; seed::UInt128) │ @ Main C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:21 │ [16] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}}) │ @ Base .\essentials.jl:798 │ [17] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})() │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 │ [18] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool) │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:70 │ [19] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 [inlined] │ [20] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})() │ @ Distributed .\task.jl:490 └ @ Base.Filesystem file.jl:738 ``` (cherry picked from commit d28a0dd)

This warning was un-ironically introduced by "Fix or suppress some noisy tests". ``` ┌ Error: mktemp cleanup │ exception = │ IOError: unlink("C:\\Windows\\TEMP\\jl_A49B.tmp"): resource busy or locked (EBUSY) │ Stacktrace: │ [1] uv_error │ @ .\libuv.jl:100 [inlined] │ [2] unlink(p::String) │ @ Base.Filesystem .\file.jl:968 │ [3] rm(path::String; force::Bool, recursive::Bool) │ @ Base.Filesystem .\file.jl:283 │ [4] rm │ @ .\file.jl:274 [inlined] │ [5] mktemp(fn::Main.Test49Main_misc.var"#110#113", parent::String) │ @ Base.Filesystem .\file.jl:736 │ [6] mktemp(fn::Function) │ @ Base.Filesystem .\file.jl:730 │ [7] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1065 [inlined] │ [8] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [9] top-level scope │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\misc.jl:1060 │ [10] include │ @ .\Base.jl:427 [inlined] │ [11] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:24 [inlined] │ [12] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Test\src\Test.jl:1360 [inlined] │ [13] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:23 [inlined] │ [14] macro expansion │ @ .\timing.jl:440 [inlined] │ [15] runtests(name::String, path::String, isolate::Bool; seed::UInt128) │ @ Main C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\testdefs.jl:21 │ [16] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, UInt128, Tuple{Symbol}, NamedTuple{(:seed,), Tuple{UInt128}}}) │ @ Base .\essentials.jl:798 │ [17] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})() │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 │ [18] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool) │ @ Distributed C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:70 │ [19] macro expansion │ @ C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\stdlib\v1.9\Distributed\src\process_messages.jl:285 [inlined] │ [20] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})() │ @ Distributed .\task.jl:490 └ @ Base.Filesystem file.jl:738 ``` (cherry picked from commit d28a0dd) Co-authored-by: Jameson Nash <[email protected]>

IanButterworth added 5 commits March 4, 2022 00:50

suppress time prints

1968011

avoid replacing module Foo warning

3236051

suppress empty info log

761fc2a

Artifacts: make test artifact downloads quiet

3f9d426

LazyArtifacts: make test install quiet

1040a17

IanButterworth force-pushed the ib/denoise_tests branch from fd77d40 to 564a78e Compare March 4, 2022 05:51

IanButterworth commented Mar 4, 2022

View reviewed changes

stdlib/REPL/test/repl.jl Show resolved Hide resolved

REPL: suppress foo echo tests

f5ad556

IanButterworth force-pushed the ib/denoise_tests branch from 564a78e to f5ad556 Compare March 4, 2022 06:23

Merge branch 'master' into ib/denoise_tests

375f5e8

giordano reviewed Mar 4, 2022

View reviewed changes

test/backtrace.jl Outdated Show resolved Hide resolved

IanButterworth commented Mar 4, 2022

View reviewed changes

test/backtrace.jl Outdated Show resolved Hide resolved

IanButterworth added 2 commits March 4, 2022 11:17

make GC.enable_logging test capture and test print

22784d5

capture tested deprecation warnings

3dc7217

IanButterworth added 2 commits March 4, 2022 18:50

switch with_logger to test_logs

a437d7f

tidy up Base.active_project_callbacks earlier to avoid error log

4d1090a

IanButterworth force-pushed the ib/denoise_tests branch from 8e262ed to 4d1090a Compare March 4, 2022 23:50

IanButterworth added 2 commits March 4, 2022 22:28

FileWatching: suppress Pidfile warning logs

75c98a2

instead of suppressing empty info log, use a debug

6f03d26

IanButterworth changed the title ~~Suppress some noisy tests 🏌️‍♂️~~ Fix or suppress some noisy tests 🏌️‍♂️ Mar 5, 2022

IanButterworth mentioned this pull request Mar 5, 2022

Switch info for debug log JuliaCrypto/SHA.jl#84

Merged

IanButterworth added 2 commits March 5, 2022 08:29

convert a noisy boundscheck testset to regular tests

2f21222

add missing SecretBuffer shred!s

d45b13f

IanButterworth added the backport 1.8 Change should be backported to release-1.8 label Mar 5, 2022

IanButterworth mentioned this pull request Mar 5, 2022

Move threaded testsets to main process to reduce Base test noise JuliaSparse/SuiteSparse.jl#63

Merged

IanButterworth removed the merge me PR is reviewed. Merge when all tests are passing label Mar 5, 2022

LibGit2: don't use testsets in subprocessed tests to prevent print leak

d090657

IanButterworth added 2 commits March 5, 2022 16:05

avoid method redefinition in syntax tests

b4590df

redirect stderr around intentional method redefinition to hide warning

02c5d1d

IanButterworth force-pushed the ib/denoise_tests branch from 8b52814 to 02c5d1d Compare March 5, 2022 21:22

IanButterworth requested a review from timholy March 5, 2022 21:23

IanButterworth and others added 2 commits March 5, 2022 17:13

add note

28901b7

🤖 Bump the SHA stdlib from 57c3a8c to 2d1f84e

960eae0

(cherry picked from commit 7287e64)

IanButterworth mentioned this pull request Mar 5, 2022

🤖 Bump the SHA stdlib from 57c3a8c to 2d1f84e #44477

Closed

timholy reviewed Mar 6, 2022

View reviewed changes

stdlib/FileWatching/test/pidfile.jl Outdated Show resolved Hide resolved

test/boundscheck_exec.jl Show resolved Hide resolved

test/loading.jl Show resolved Hide resolved

test/misc.jl Outdated Show resolved Hide resolved

review suggestions

41b38d2

IanButterworth added the merge me PR is reviewed. Merge when all tests are passing label Mar 6, 2022

IanButterworth merged commit d971465 into JuliaLang:master Mar 6, 2022

IanButterworth deleted the ib/denoise_tests branch March 6, 2022 12:59

DilumAluthge removed the merge me PR is reviewed. Merge when all tests are passing label Mar 6, 2022

KristofferC mentioned this pull request Mar 7, 2022

Backports for julia 1.8.0-beta2 #44324

Merged

47 tasks

IanButterworth added a commit that referenced this pull request Mar 7, 2022

Fix or suppress some noisy tests 🏌️‍♂️ (#44444)

406b293

(cherry picked from commit d971465)

KristofferC removed the backport 1.8 Change should be backported to release-1.8 label Mar 15, 2022

KristofferC mentioned this pull request May 17, 2022

remove testset from a test in boundscheck_exec.jl #42384

Closed

KristofferC added a commit that referenced this pull request Nov 7, 2022

revert part of #44444 which seems to make buildkite unhappy

f69b361

staticfloat mentioned this pull request Nov 7, 2022

Backports for Julia 1.8.3 #46984

Merged

37 tasks

vchuravy pushed a commit to JuliaPackaging/LazyArtifacts.jl that referenced this pull request Oct 2, 2023

Fix or suppress some noisy tests 🏌️‍♂️ (JuliaLang/julia#44444)

cca9759

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix or suppress some noisy tests 🏌️‍♂️ #44444

Fix or suppress some noisy tests 🏌️‍♂️ #44444

IanButterworth commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

timholy commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

IanButterworth commented Mar 4, 2022

timholy commented Mar 4, 2022 •

edited

Loading

IanButterworth commented Mar 5, 2022 •

edited

Loading

IanButterworth commented Mar 5, 2022

IanButterworth commented Mar 5, 2022 •

edited

Loading

timholy left a comment

Fix or suppress some noisy tests 🏌️‍♂️ #44444

Fix or suppress some noisy tests 🏌️‍♂️ #44444

Conversation

IanButterworth commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

timholy commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

DilumAluthge commented Mar 4, 2022

IanButterworth commented Mar 4, 2022

timholy commented Mar 4, 2022 • edited Loading

IanButterworth commented Mar 5, 2022 • edited Loading

IanButterworth commented Mar 5, 2022

IanButterworth commented Mar 5, 2022 • edited Loading

timholy left a comment

Choose a reason for hiding this comment

timholy commented Mar 4, 2022 •

edited

Loading

IanButterworth commented Mar 5, 2022 •

edited

Loading

IanButterworth commented Mar 5, 2022 •

edited

Loading