Define Setup and TearDown callbacks as TEST greatest_<>_cb(void *udata);

[soul4soul]

I would like to make better use of Setup and TearDown for suites but in order to do I must be able to return a test result from them.

Take this example code which represents one of my test suites. In this test application my test suites are split up by functionality of a device. This is a paradigm I commonly use to organize tests.

TEST TestSetup()
{
    S16BIT s16Result;

    SKIP_TRUE(sInput.sFunc.sApi.IsValid); //Test if there is API support
    SKIP_TRUE(sInput.sFunc.sApi.u8DevNum == sInput.sDut.u8DevNum); // Test the DUT will be the device tested by this suite

    s16Result = My_Open_Dev(sInput.sFunc.sApi.u8DevNum);
    ASSERT_EQm("Failed to open API device", ERR_SUCCESS, s16Result);

    PASS();
}

/* This function will run after every test in this suite */
static void TearDown_CB(void* data)
{
    S16BIT s16Result;

    s16Result = My_Close_Dev(sInput.sFunc.sApi.u8DevNum);
    //ASSERT_EQm("Failed to close API device", s16Result == ERR_SUCCESS ? ERR_SUCCESS : ERR_DEV_NOT_INIT, s16Result);

    //PASS();
}

TEST ApiSelfTestResult()
{
    S16BIT s16Result;
    U32BIT u32TestResults;
    U8BIT u8DevNum = sInput.sFunc.sApi.u8DevNum;

    CHECK_CALL(TestSetup());
    s16Result = My_ApiSelfTest(u8DevNum, &u32TestResults);

    ASSERT_ENUM_EQ(ERR_SUCCESS, s16Result, Error2Str);
    ASSERT_EQ_FMTm("ApiSelfTest test results do not equal 0", 0, u32TestResults, "%X");

    PASS();
}

SUITE(apioperations_func_suite)
{
    SET_TEARDOWN(TearDown_CB, NULL);

    RUN_TEST(ApiStatusRead_ReadStatus);
}

In the setup I would like to check that the DUT has the necessary features that the suite will be testing. In the setup I would also like to call some functions to get the DUT into a common know state which will be used for every test in the suite. These function calls may fail as any other and I want to test them. The reason to use a setup function is because the steps that need to be performed are common to every test and need to be run at the start of each test in the suite. However by using a setup I am no longer able to skip a test or assert. There is a work around which is not too painful. A function can be define that does everything I would do in setup but its return type would be TEST. At the very beginning of each test CHECK_CALL can be used. This workaround can be see in the code above.

In the TearDown I would like to perform some clean up and check that things ended in a known condition. So once again I need to be able to assert. Additionally it is required that the TearDown is always run. I have not found a suitable workaround for TearDown. The technique I used for the Setup won't work for the teardown because a test is exited immediately on failure so a CHECK_CALL to a teardown function as the last line of a test is not valid. In my case it was more important to clean things so I chose to use the teardown callback and not make any additional asserts.

[silentbicycle]

A function can be define that does everything I would do in setup but its return type would be TEST. At the very beginning of each test CHECK_CALL can be used. This workaround can be see in the code above.

I wouldn't consider that a workaround -- it's one of the reasons why I added CHECK_CALL in the first place. I should be more explicit about that in the README: "if your setup hook should be able to fail/skip the test, then use CHECK_CALL(setup(data)); in the body of the test instead".

teardown is a more complicated matter, though. I prefer to check that things are in a known condition during setup, at least in the general case, and do little to nothing in teardown itself. Making teardown able to fail an otherwise passing test would be an interface change (so it would have to wait until the next major version); while adding a function that can only be called from inside a teardown hook, which causes the most recently run test to fail could work, it seems kind of hacky, and it wouldn't work with the existing ASSERT macros.

Based on my understanding of your situation (testing with an external, stateful device -- I've been there), I suggest using a function at the start of test cases (via CHECK_CALL) which closes a previous connection, if any, and then does your SKIP_TRUE and then open calls. In cases where the test has otherwise passed, but would fail in teardown because it can't close the connection, I'd suggest putting a checked close at the end of your tests.

Also, while it doesn't directly address your issue, you may find using SHUFFLE_TESTS helpful for tracking down issues caused by incomplete test setup/teardown cleanup or unanticipated coupling between tests.