More error resilient test runner #2036
Conversation
Ensures that test-runner spec expansion works properly on first run Fails test-runner, if big SUITE cannot be compiled
Generate specs after big tests are compiled
This comment has been minimized.
This comment has been minimized.
Codecov Report
@@ Coverage Diff @@
## master #2036 +/- ##
==========================================
+ Coverage 75.06% 76.95% +1.89%
==========================================
Files 315 351 +36
Lines 28675 37929 +9254
==========================================
+ Hits 21524 29189 +7665
- Misses 7151 8740 +1589
Continue to review full report at Codecov.
|
|
5810.1 / Erlang 19.3 / small_tests / 7172d25 5810.3 / Erlang 19.3 / mysql_redis / 7172d25 mod_global_distrib_SUITE:mod_global_distrib:test_pm_with_ungraceful_reconnection_to_different_server{error,
{{assertion_failed,assert,is_chat_message,
[<<"Hi again from Europe1!">>],
{xmlel,<<"message">>,
[{<<"from">>,<<"alicE0.501035@localhost/res1">>},
{<<"to">>,<<"eve0.469205@localhost">>},
{<<"xml:lang">>,<<"en">>},
{<<"type">>,<<"chat">>},
{<<"id">>,<<"-576459798885834869">>}],
[{xmlel,<<"body">>,[],[{xmlcdata,<<"Hi from Europe1!">>}]}]},
"<message from='alicE0.501035@localhost/res1' to='eve0.469205@localhost' xml:lang='en' type='chat' id='-576459798885834869'><body>Hi from Europe1!</body></message>"},
[{escalus_new_assert,assert_true,2,
[{file,
"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
{line,84}]},
{mod_global_distrib_SUITE,
'-test_pm_with_ungraceful_reconnection_to_different_server/1-fun-0-',
4,
[{file,"mod_global_distrib_SUITE.erl"},{line,613}]},
{escalus_story,story,4,
[{file,
"/home/travis/build/esl/MongooseIM/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1529}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1045}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,977}]}]}}5810.2 / Erlang 19.3 / internal_mnesia / 7172d25 5810.4 / Erlang 19.3 / odbc_mssql_mnesia / 7172d25 5810.5 / Erlang 19.3 / ldap_mnesia / 7172d25 5810.6 / Erlang 19.3 / elasticsearch_and_cassandra_mnesia / 7172d25 5810.8 / Erlang 20.0 / pgsql_mnesia / 7172d25 5810.9 / Erlang 21.0 / riak_mnesia / 7172d25 |
|
@arcusfelis could you explain this paragraph:
I see that both big tests with preset mysql_redis and small tests' ejabberd_sm_SUITE with redis backend passed even though redis on travis is not run in docker. |
|
@michalslaski More like: Basically, the problem with the current behavior is it kinda confusing not to see redis in if you provide --db option without redis in it, we would not try to start it from the test runner script. I.e. something like Alternative behavior WOULD be (need to be coded, if we want it) to skip starting some database, if the port is already used by something. |
This PR addresses some errors, because test runner assumes that someone run it before. And it expects beam files of SUITEs to be available for spec generation and autocompletion.
Be aware, that with this patch travis test would stop, if
redis-serveris running without a docker container. Use--dbin this case.Later we can add code, that would test database options sanity before running any tests (gen_tcp connect? actually probing connection with some library? idk).