how to run tests from individual modules

[dpinol]

Hi,
as with SafeTestsets.jl, I was trying to run my tests with Jive from different modules to avoid collisions and ensure that the test are independent.

I tried wrapping the calls to Jive.runtests in independent modules, but it does not work because Jive loads the tests files Main.include instead of doing include.

Is there are way to ensure a better independence between tests? thanks

[wookay]

could you test it again with master branch?

[dpinol]

Hi,
thank you for the very quick patch!

Unfortunately I didn't get the desired effect.
Even when running Jive.runtests from different modules, the tests are executed from the same module and I still get namespace pollution (test2.jl is affected by "using Dates" in

Running "julia test.jl" on JiveInclude.zip

I get the following and I'd like that test2.jl would fail because it contain "using Dates".

Also, the warning that appears may be confusing to many developers

************************Main.M1
1/1 test1.jl
┌ Warning: Package Jive does not have JiveInclude in its dependencies:
│ - If you have Jive checked out for development and have
│   added JiveInclude as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Jive
└ Loading JiveInclude into Jive from project dependency, future warnings for Jive are suppressed.
Hello World!************************Jive
    Pass 1  (compile: 0.47, elapsed: 0.47 seconds)
✅  All 1 test has been completed.  (compile: 0.47, elapsed: 0.47 seconds)
************************Main.M2
1/1 test2.jl
Hello World!************************Jive
    Pass 1  (compile: 0.00, elapsed: 0.00 seconds)
✅  All 1 test has been completed.  (compile: 0.00, elapsed: 0.00 seconds)
     Testing JiveInclude tests passed 

What I've ended up doing (for Jive release) is wrapping `Main.include' so that each included file is imported into a different module.

worldAge = Base.get_world_counter()
function Main.include(path::AbstractString)
    @eval module $(Symbol(path))
    Base.invoke_in_world($worldAge, Base.include, @__MODULE__, $path)
    end
    return nothing
end

To make it less hackish, it would be ideal if Jive.runtest could have an optional callback argument which would delegate the including process to the tested project. This would allow:

• skipping files based on their extension or location (eg. test helpers)
• wrapping in different modules
• ...

[wookay]

thanks for the attachment.
I just updated using anonymous at master branch.

do you expected like this?

JiveInclude/test julia --project=.. runtests.jl
************************Main.M1
1/1 test1.jl
Hello World!************************Main.anonymous
    Pass: 1  (compile: 0.58, elapsed: 1.83 seconds)
✅  All 1 test has been completed.  (compile: 0.58, elapsed: 1.83 seconds)
************************Main.M2
1/1 test2.jl
Hello World!************************Main.anonymous
: Error During Test at JiveInclude/test/test2.jl:7
  Got exception outside of a @test
  LoadError: UndefVarError: Dates not defined
  Stacktrace:
   [1] top-level scope
     @ :0
   [2] include
     @ ./Base.jl:386 [inlined]
   [3] normal_run(dir::String, tests::Vector{String}, start_idx::Int64, stop_on_failure::Bool)
     @ Jive ~/.julia/dev/Jive/src/runtests.jl:479
   [4] runtests(dir::String; skip::Vector{String}, node1::Vector{Any}, targets::Vector{String}, enable_distributed::Bool, stop_on_failure::Bool)
     @ Jive ~/.julia/dev/Jive/src/runtests.jl:87
   [5] runtests(dir::String)
     @ Jive ~/.julia/dev/Jive/src/runtests.jl:71
   [6] top-level scope
     @ JiveInclude/test/runtests.jl:14
   [7] include(mod::Module, _path::String)
     @ Base ./Base.jl:386
   [8] exec_options(opts::Base.JLOptions)
     @ Base ./client.jl:285
   [9] _start()
     @ Base ./client.jl:485
  in expression starting at JiveInclude/test/test2.jl:7
    Error: 1    Pass: 1  (compile: 0.00, elapsed: 2.47 seconds)
❗️  Test run finished with 1 error.  (compile: 0.00, elapsed: 2.47 seconds)
ERROR: LoadError: Test run finished with errors.

[dpinol]

Hi,
I confirm that the new version gets rid of the warning , and runs each runtest invokation in a different module. Thanks!

On the other hand, this stills works, due to the cross-pollution of imports.

using Jive
runtests("test")

Looking at the last patch, what do you think about moving modul = Module() within the loop rigt before Base.include(modul, filepath) (line 479)? This might accomplish the same effect as SafeTestsets.jl without the need to alter any test file

best

[wookay]

thanks for the feedback.

I've added context, that is

runtests(dir::String; ..., context::Union{Nothing,Module}=nothing)

context: module that to be used in Base.include. nothing means to be safe that using anonymous module for every test file.

[dpinol]

it works like a charm
thanks!