Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data-driven testing -- print out the deepest DOCTEST_SUBCASE #215

Closed
jktjkt opened this issue Mar 27, 2019 · 2 comments
Closed

Data-driven testing -- print out the deepest DOCTEST_SUBCASE #215

jktjkt opened this issue Mar 27, 2019 · 2 comments

Comments

@jktjkt
Copy link
Contributor

jktjkt commented Mar 27, 2019

Description

I'm trying to use what the Qt landscape calls a "data driven testing". Here's my code:

TEST_CASE("converting a byte buffer to a float")
{
    float f;
    std::string buffer;

    DOCTEST_SUBCASE("0x41480000 (= 12.5)") { buffer = "\x00\x00\x48\x41"s; f = 12.5f; }
    DOCTEST_SUBCASE("0x3dcccccd (= 0.1)") { buffer = "____"s; f = 0.1f; }
    DOCTEST_SUBCASE("0x3a83126f (= 0.001)") { buffer = "\x6f\x12\x83\x3a"s; f = 0.001f; }
    DOCTEST_SUBCASE("0x43160000 (= 150.0)") { buffer = "\x00\x00\x16\x43"s; f = 150.0f; }

    auto it = buffer.cbegin();
    float result = cla::utils::parseLittleEndianBuffer<float>(it);
    REQUIRE(it == buffer.cend());
    REQUIRE(result == f);
}

As I'm migrating from Catch where it required a patch which tweaked the test nesting that is printed out upon a failure. And indeed, it seems that a similar thing is needed with doctest. With its current dev branch, the test failure report only includes "converting a byte buffer to a float", but it does not show that the offending data came from the 0x3dcccccd (= 0.1) subcase:

[doctest] doctest version is "2.3.1"
[doctest] run with "--help" for options
===============================================================================
tests/numbers.cpp:213:
TEST CASE:  converting a byte buffer to a float

tests/numbers.cpp:225: FATAL ERROR: REQUIRE( it == buffer.cend() ) is NOT correct!
  values: REQUIRE( {?} == {?} )

===============================================================================
[doctest] test cases:      4 |      3 passed |      1 failed |      0 skipped
[doctest] assertions:    109 |    108 passed |      1 failed |
[doctest] Status: FAILURE!

I read the docs about parametrized tests. I think that the "subcases" example is a bit contrived. I actually do not mind wrapping the test data in several DOCTEST_SUBCASE stanzas at all; I also think that adding a nice, human-readable name to the subcase block is a nice feature.

Can this be made to work without that extra call to CAPTURE()?
@onqtam
Copy link
Member

onqtam commented Mar 27, 2019 •
edited
Loading

this might be the same thing: #125 (comment) - there I link to the same Catch issue.

Perhaps I should really think about this - not having it on by default but at least some option... Also with the current XML output the last entered subcases could be inferred.

But I'm reeeally busy in the upcoming weeks/months :|

@onqtam onqtam mentioned this issue Sep 22, 2019
Closed
jktjkt added a commit to jktjkt/doctest that referenced this issue Apr 30, 2020
@jktjkt
Print subcase stack upon a subcase entry
f0d2eda 
This is largely a hack to give us at least *something* for data driven
testing (doctest#215). In that pattern, it's common to structure code in a way
that first "generates the data", and only then executes the actual test.
It can look like this:

  TEST_CASE("parsing stuff") {
    std::string input;
    Something expected;

    DOCTEST_SUBCASE("single-line") {
      DOCTEST_SUBCASE("empty") {}
      DOCTEST_SUBCASE("trivial") {
        input = "create foo";
	expected = {OP_CREATE, {"foo",}};
      }
      DOCTEST_SUBCASE("two args") {
        input = "create blah bar";
	expected = {OP_CREATE, {"blah", "bar",}};
      }
    }
    DOCTEST_SUBCASE("multi-line") {
      DOCTEST_SUBCASE("trailing whitespace") {
        input = "create foo\n\n";
	expected = {OP_CREATE, {"foo,}};
      }
      DOCTEST_SUBCASE("one word per line, two words") {
        input = "create\nfoo";
	expected = {OP_CREATE, {"foo",}};
      }
    }

    REQUIRE(parse(input) == expected);
  }

The important part is that once we're executing actual tests, the
subsections will have been exited already, so the usual ways of showing
the subcase stack can be no longer applied.

In Catch2, there was a patch which tried to track the deepest subcase
stack so far. Let's try to implement something very simple, just
printing the current state of the stack once it is entered. It does not
tie into any error handling, so the output will be always there, and it
also probably does not support any fancy filtering when skipping
subcases. However, it's something which at least shows what is going on
when an error gets reported.

Cc: doctest#215
Cc: doctest#125
jktjkt added a commit to jktjkt/doctest that referenced this issue Apr 30, 2020
@jktjkt
Print subcase stack upon a subcase entry
300ed25 
This is largely a hack to give us at least *something* for data driven
testing (doctest#215). In that pattern, it's common to structure code in a way
that first "generates the data", and only then executes the actual test.
It can look like this:

  TEST_CASE("parsing stuff") {
    std::string input;
    Something expected;

    DOCTEST_SUBCASE("single-line") {
      DOCTEST_SUBCASE("empty") {}
      DOCTEST_SUBCASE("trivial") {
        input = "create foo";
	expected = {OP_CREATE, {"foo",}};
      }
      DOCTEST_SUBCASE("two args") {
        input = "create blah bar";
	expected = {OP_CREATE, {"blah", "bar",}};
      }
    }
    DOCTEST_SUBCASE("multi-line") {
      DOCTEST_SUBCASE("trailing whitespace") {
        input = "create foo\n\n";
	expected = {OP_CREATE, {"foo,}};
      }
      DOCTEST_SUBCASE("one word per line, two words") {
        input = "create\nfoo";
	expected = {OP_CREATE, {"foo",}};
      }
    }

    REQUIRE(parse(input) == expected);
  }

The important part is that once we're executing actual tests, the
subsections will have been exited already, so the usual ways of showing
the subcase stack can be no longer applied.

In Catch2, there was a patch which tried to track the deepest subcase
stack so far. Let's try to implement something very simple, just
printing the current state of the stack once it is entered. It does not
tie into any error handling, so the output will be always there, and it
also probably does not support any fancy filtering when skipping
subcases. However, it's something which at least shows what is going on
when an error gets reported.

Cc: doctest#215
Cc: doctest#125
@jktjkt jktjkt mentioned this issue Apr 30, 2020
Closed
onqtam added a commit that referenced this issue May 17, 2020
@onqtam
fixed #215 fixed #125 relates #364 - printing the deepest subcase sta…
1a65a92 
…ck reached - even if at the point of failure of an assertion the stack has become smaller - that way users can determine the exact path of data initialization in previous subcases before the failing assert.
@onqtam
Copy link
Member

onqtam commented May 17, 2020

done - for a program like this:

TEST_CASE("data-driven") {
    int a;
    SUBCASE("subcase 1")
        a = 5;
    SUBCASE("subcase 2")
        a = 42;
    CHECK(a == 6);
}

the output will be something like this:

===============================================================================
E:\doctest\scripts\playground\test.cpp:61:
TEST CASE:  data-driven

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  subcase 1

E:\doctest\scripts\playground\test.cpp:67: ERROR: CHECK( a == 6 ) is NOT correct
!
  values: CHECK( 5 == 6 )

===============================================================================
E:\doctest\scripts\playground\test.cpp:61:
TEST CASE:  data-driven

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  subcase 2

E:\doctest\scripts\playground\test.cpp:67: ERROR: CHECK( a == 6 ) is NOT correct
!
  values: CHECK( 42 == 6 )

that message will be printed only if the deepest subcase stack is different from the current one. Also the current subcase stack will be printed below the TEST CASE: as before.

@onqtam onqtam closed this as completed May 17, 2020
onqtam added a commit that referenced this issue May 17, 2020
@onqtam
fixed #215 fixed #125 relates #364 - printing the deepest subcase sta…
72bf24e 
…ck reached - even if at the point of failure of an assertion the stack has become smaller - that way users can determine the exact path of data initialization in previous subcases before the failing assert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jktjkt @onqtam