Testing

Four different categories of tests are used: unit tests, functional tests, scenario tests, and live scenario tests.

Commonly, only unit tests are required to validate changes. For large changes, functional or integration tests are often needed. Live scenario tests can be used to test end to end scenarios for these larger changes.

Unit tests

To run these tests (and the functional tests), from the root of the repo run the following script:

./scripts/verify.sh local

In Windows, use:

./scripts/verify.bat local

This command also runs PyLint to check code style.

Unit tests should never depend on external connections or local resources such as file I/O. Be mindful of how long a unit test runs. Unit tests should never run for longer than a few seconds.

Functional tests

These tests currently check that the HTTP requests generated by calling an sfctl command is correct. It verifies the request by calling the command against a mock cluster that is created by the test. The test then uses the module VCR to record the transaction. The recorded file is then read by the test to ensure correctness. The file is then deleted.

These tests can be found in <root>/src/sfctl/tests/request_generation_test.py

Any command that generates an HTTP request to the Service Fabric cluster should include at least one of these tests.

Scenario tests

Scenario tests are designed to be repeatable functional tests. These tests take advantage of recording thanks to vcrpy in the knack CLI framework.

For examples of these tests, take a look at scenario_test.py in the tests module.

These tests can be run from recordings, or against live Azure Service Fabric clusters. By default these tests are disabled. To run these more intensive tests, environment variables must specify a Service Fabric cluster endpoint to connect to. The expected variable name is as follows:

EXPORT SF_TEST_ENDPOINT=http://test.azure.com:19080

In Windows, the command is

SET SF_TEST_ENDPOINT=http://test.azure.com:19080

Here, the endpoint should be the HTTP gateway URI of a Service Fabric cluster.

Live scenario tests

For more complicated scenarios that cannot run in automation, use a live scenario test. Run live scenario tests to validate complex workflows such as application life cycle or cluster upgrade.

The follow sections document each live scenario test.

Application life cycle (application_lifecycle_test)

Basic application life cycle test, excluding application upgrade. Note, this test may result in altered Service Fabric cluster state, be sure to clean up any state if the test fails before subsequent runs.

Enable

Enable the application life cycle test by specifying a path to a Service Fabric application package:

EXPORT SF_TEST_APP_PATH=/Users/test/Downloads/sample_apps/CalcApp/

Documentation tests

Documentation tests serve two purposes: they ensure that errors do not occur when running the help command, and they ensure that all groups, commands, and arguments have descriptive text explaining their purpose.

These tests can be found in <root>/src/sfctl/tests/help_text_test.py.

Add a test by adding or modifying an entry for the command validate_output. These commands are split into groups, and the name of each command in a group must be provided. Further details are documented in the test file.