Enable CircleCI Test Insights #3899
Conversation
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
df88afd
to
9b04c9e
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Codecov ReportBase: 83.16% // Head: 83.13% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #3899 +/- ##
==========================================
- Coverage 83.16% 83.13% -0.04%
==========================================
Files 535 535
Lines 34112 34112
==========================================
- Hits 28370 28359 -11
- Misses 5742 5753 +11
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
9b04c9e
to
de2a980
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
2 times, most recently
from
294b791
to
bcdc6fb
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
bcdc6fb
to
aa4e3e4
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
aa4e3e4
to
2b6925a
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
2b6925a
to
7be0dca
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
7be0dca
to
2a1c1aa
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
2 times, most recently
from
7ac75c6
to
5ec1fc5
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
5ec1fc5
to
9fbe8df
Compare
chrzaszcz
changed the title
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
9fbe8df
to
9108acb
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
9108acb
to
3418701
Compare
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
3418701
to
edd3580
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
07d220c
to
1850e46
Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
small_tests_24 / small_tests / 199f1b4 small_tests_25 / small_tests / 199f1b4 dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 199f1b4 ldap_mnesia_24 / ldap_mnesia / 199f1b4 dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 199f1b4 ldap_mnesia_25 / ldap_mnesia / 199f1b4 dynamic_domains_mysql_redis_25 / mysql_redis / 199f1b4 internal_mnesia_25 / internal_mnesia / 199f1b4 pgsql_mnesia_24 / pgsql_mnesia / 199f1b4 elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 199f1b4 dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 199f1b4 riak_mnesia_24 / riak_mnesia / 199f1b4 mysql_redis_25 / mysql_redis / 199f1b4 pgsql_mnesia_25 / pgsql_mnesia / 199f1b4 mssql_mnesia_25 / odbc_mssql_mnesia / 199f1b4 s2s_SUITE:node1_tls_optional_node2_tls_required_trusted_with_cachain:simple_message{error,
{timeout_when_waiting_for_stanza,
[{escalus_client,wait_for_stanza,
[{client,<<"alice_simple_message_3127@fed1/res1">>,escalus_tcp,
<0.5214.3>,
[{event_manager,<0.5089.3>},
{server,<<"fed1">>},
{username,<<"alice_simple_message_3127">>},
{resource,<<"res1">>}],
[{event_client,
[{event_manager,<0.5089.3>},
{server,<<"fed1">>},
{username,<<"alice_simple_message_3127">>},
{resource,<<"res1">>}]},
{resource,<<"res1">>},
{username,<<"alice_simple_message_3127">>},
{server,<<"fed1">>},
{host,<<"localhost">>},
{port,5242},
{auth,{escalus_auth,auth_plain}},
{wspath,undefined},
{username,<<"alice_simple_message_3127">>},
{server,<<"fed1">>},
{host,<<"localhost">>},
{port,5242},
{password,<<"makota2">>},
{stream_id,<<"ce7d29a601539f30">>}]},
10000],
[{file,
"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
{line,136}]},
{s2s_SUITE,'-simple_message/1-fun-0-',2,
[{file,"/home/circleci/project/big_tests/tests/s2s_SUITE.erl"},
{line,128}]},
{escalus_story,story,4,
[{file,
"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file... |
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
199f1b4
to
046fe28
Compare
|
small_tests_24 / small_tests / 046fe28 small_tests_25 / small_tests / 046fe28 dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 046fe28 ldap_mnesia_24 / ldap_mnesia / 046fe28 pgsql_mnesia_24 / pgsql_mnesia / 046fe28 dynamic_domains_mysql_redis_25 / mysql_redis / 046fe28 ldap_mnesia_25 / ldap_mnesia / 046fe28 dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 046fe28 elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 046fe28 riak_mnesia_24 / riak_mnesia / 046fe28 internal_mnesia_25 / internal_mnesia / 046fe28 dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 046fe28 pgsql_mnesia_25 / pgsql_mnesia / 046fe28 pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription{error,
{{badmatch,
[{xmlel,<<"message">>,
[{<<"from">>,
<<"alice_unsubscribe_after_presence_unsubscription_2694@localhost">>},
{<<"to">>,
<<"bob_unsubscribe_after_presence_unsubscription_2694@localhost/res1">>},
{<<"type">>,<<"headline">>}],
[{xmlel,<<"event">>,
[{<<"xmlns">>,
<<"http://jabber.org/protocol/pubsub#event">>}],
[{xmlel,<<"items">>,
[{<<"node">>,<<"DtHw9RnkY1WhFECpTyhZ5g==">>}],
[{xmlel,<<"item">>,
[{<<"id">>,<<"salmon">>}],
[{xmlel,<<"entry">>,
[{<<"xmlns">>,
<<"http://www.w3.org/2005/Atom">>}],
[]}]}]}]},
{xmlel,<<"headers">>,
[{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
[]}]}]},
[{pep_SUITE,'-unsubscribe_after_presence_unsubscription/1-fun-0-',2,
[{file,"/home/circleci/project/big_tests/tests/pep_SUITE.erl"},
{line,384}]},
{escalus_story,story,4,
[{file,
"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1291}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1223}]}]}}mysql_redis_25 / mysql_redis / 046fe28 mssql_mnesia_25 / odbc_mssql_mnesia / 046fe28 pep_SUITE:pep_tests:unsubscribe_after_presence_unsubscription{error,
{{badmatch,
[{xmlel,<<"message">>,
[{<<"from">>,
<<"alice_unsubscribe_after_presence_unsubscription_2656@localhost">>},
{<<"to">>,
<<"bob_unsubscribe_after_presence_unsubscription_2656@localhost/res1">>},
{<<"type">>,<<"headline">>}],
[{xmlel,<<"event">>,
[{<<"xmlns">>,
<<"http://jabber.org/protocol/pubsub#event">>}],
[{xmlel,<<"items">>,
[{<<"node">>,<<"rOkgTOpRJ9lXhQwUhgIKjA==">>}],
[{xmlel,<<"item">>,
[{<<"id">>,<<"salmon">>}],
[{xmlel,<<"entry">>,
[{<<"xmlns">>,
<<"http://www.w3.org/2005/Atom">>}],
[]}]}]}]},
{xmlel,<<"headers">>,
[{<<"xmlns">>,<<"http://jabber.org/protocol/shim">>}],
[]}]}]},
[{pep_SUITE,'-unsubscribe_after_presence_unsubscription/1-fun-0-',2,
[{file,"/home/circleci/project/big_tests/tests/pep_SUITE.erl"},
{line,384}]},
{escalus_story,story,4,
[{file,
"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1291}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1223}]}]}} |
The goal is to add the 'cth_surefire' hook, which will collect test results for CircleCI Insights.
The Insights need results in the JUnit XML format. This is achieved by using the 'cth_surefire' hook.
- Support parallel test failures - Close groups when end_per_suite is skipped or fails - Include group name in error messages
chrzaszcz
force-pushed
the
circleci-test-insights
branch
from
046fe28
to
651bdd5
Compare
|
small_tests_24 / small_tests / 651bdd5 small_tests_25 / small_tests / 651bdd5 ldap_mnesia_24 / ldap_mnesia / 651bdd5 dynamic_domains_pgsql_mnesia_24 / pgsql_mnesia / 651bdd5 dynamic_domains_pgsql_mnesia_25 / pgsql_mnesia / 651bdd5 dynamic_domains_mysql_redis_25 / mysql_redis / 651bdd5 muc_SUITE:hibernation:hibernated_room_can_be_queried_for_archive{error,{{assertion_failed,assert,is_groupchat_message,
[<<"Restorable message">>],
undefined,"undefined"},
[{escalus_new_assert,assert_true,2,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_new_assert.erl"},
{line,84}]},
{muc_SUITE,wait_for_mam_result,3,
[{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
{line,4394}]},
{muc_SUITE,'-hibernated_room_can_be_queried_for_archive/1-fun-0-',3,
[{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
{line,4130}]},
{escalus_story,story,4,
[{file,"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_story.erl"},
{line,72}]},
{muc_SUITE,hibernated_room_can_be_queried_for_archive,1,
[{file,"/home/circleci/project/big_tests/tests/muc_SUITE.erl"},
{line,4126}]},
{test_server,ts_tc,3,[{file,"test_server.erl"},{line,1782}]},
{test_server,run_test_case_eval1,6,
[{file,"test_server.erl"},{line,1291}]},
{test_server,run_test_case_eval,9,
[{file,"test_server.erl"},{line,1223}]}]}}ldap_mnesia_25 / ldap_mnesia / 651bdd5 pgsql_mnesia_24 / pgsql_mnesia / 651bdd5 dynamic_domains_mssql_mnesia_25 / odbc_mssql_mnesia / 651bdd5 inbox_extensions_SUITE:regular:one_to_one:pagination:max_queries_can_fetch_ahead{error,
{timeout_when_waiting_for_stanza,
[{escalus_client,wait_for_stanza,
[{client,
<<"[email protected]/res1">>,
escalus_tcp,<0.11021.1>,
[{event_manager,<0.10994.1>},
{server,<<"domain.example.com">>},
{username,<<"mike_max_queries_can_fetch_ahead_1538">>},
{resource,<<"res1">>}],
[{event_client,
[{event_manager,<0.10994.1>},
{server,<<"domain.example.com">>},
{username,<<"mike_max_queries_can_fetch_ahead_1538">>},
{resource,<<"res1">>}]},
{resource,<<"res1">>},
{username,<<"mike_max_queries_can_fetch_ahead_1538">>},
{server,<<"domain.example.com">>},
{host,<<"localhost">>},
{port,5222},
{auth,{escalus_auth,auth_plain}},
{wspath,undefined},
{username,<<"mike_max_queries_can_fetch_ahead_1538">>},
{server,<<"domain.example.com">>},
{host,<<"localhost">>},
{password,<<"nicniema">>},
{stream_id,<<"b7b0be8e60d58937">>}]},
5000],
[{file,
"/home/circleci/project/big_tests/_build/default/lib/escalus/src/escalus_client.erl"},
{line,136}]},
{inbox_helper,'-given_conversations_between/2-fun-1-',4,
[{file,"/home/circleci/project/big_tests/tests/inbox_helper.erl"},
{line,543}]},
{lists,foldl_1,3,[{file,"lists.erl"},{line,1355}]...elasticsearch_and_cassandra_25 / elasticsearch_and_cassandra_mnesia / 651bdd5 internal_mnesia_25 / internal_mnesia / 651bdd5 riak_mnesia_24 / riak_mnesia / 651bdd5 pgsql_mnesia_25 / pgsql_mnesia / 651bdd5 mysql_redis_25 / mysql_redis / 651bdd5 mssql_mnesia_25 / odbc_mssql_mnesia / 651bdd5 |
The goal is to enable CircleCI Test Insights for big tests, that can automatically detect flaky and most failing tests.
This can be achieved by using the
cth_surefirecommon test hook.However, the hook contained a few bugs, so:
cth_surefireis copied from Erlang/OTP to big tests.When reviewing, check the last commit to see the diff in
cth_surefire.Here you can see the Insights in action:
Screenshot of a flaky test from reports:
