diff --git a/README.md b/README.md index 72db7c0053..fd3222f87b 100644 --- a/README.md +++ b/README.md @@ -910,6 +910,7 @@ The Variable EXIV2\_PORT or EXIV2\_HTTP can be set to None to skip http tests. [TOC](#TOC)
+ ### 4.1 Running tests on Unix-like systems You can run tests directly from the build: @@ -945,6 +946,7 @@ $ [TOC](#TOC)
+ ### 4.2 Running tests on Visual Studio builds from cmd.exe **Caution:** _The python3 interpreter must be on the PATH, build for DOS, and called python3.exe._ I copied the python.exe program: @@ -995,6 +997,7 @@ set EXIV2_PORT= [TOC](#TOC)
+ ### 4.3 Unit Tests The code for the unit tests is in `/unitTests`. To include unit tests in the build, use the *cmake* option `-DEXIV2_BUILD_UNIT_TESTS=On`. @@ -1014,6 +1017,7 @@ $ popd [TOC](#TOC)
+ ### 4.4 Bugfix Tests You can run the bugfix tests from the build directory: diff --git a/tests/README-TESTS.md b/tests/README-TESTS.md index 41655834bd..20378185b5 100644 --- a/tests/README-TESTS.md +++ b/tests/README-TESTS.md @@ -3,6 +3,7 @@ - [README-TESTS](#README-TESTS) - [Running the test suite](#running-the-test-suite) - [Writing new tests](#writing-new-tests) + - [Test data](#test-data) - [Test suite](#test-suite) - [Configuration](#configuration) - [INI style](#ini-style) @@ -25,6 +26,7 @@
+ # README-TESTS This test suite is intended for system tests, i.e. for running a binary with @@ -34,6 +36,7 @@ of new features where unit testing is not feasible, e.g. to test new command line parameters.
+ ## Running the test suite The test suite is written for Python 3 and is not compatible with Python 2, thus @@ -65,6 +68,7 @@ standard output. [TOC](#TOC)
+ ## Writing new tests The test suite is intended to run a binary and compare its standard output, @@ -111,7 +115,22 @@ When creating new tests, follow roughly these steps: [TOC](#TOC) +
+ +## Test data +All test files (e.g. images, `.exv`, etc) are stored in the `$data_path` +directory, with any temporary copies in the `$tmp_path` directory (see +[Creating file copies](#creating-file-copies)). + +When writing tests, try to reuse existing data files rather than add new ones. +If new files are required, prefer adding only the metadata (`.exv`) instead of +the whole image. These steps help to reduce the size of the test data +directory. + +[TOC](#TOC) +
+ ## Test suite The test suite itself uses the builtin `unittest` module of Python to discover @@ -131,6 +150,7 @@ group test cases. ### Configuration
+ #### INI style The test suite is configured via `INI` style files using Python's builtin @@ -181,6 +201,7 @@ some_var:some value with whitespaces before and after [TOC](#TOC)
+ #### Parameters The test suite's configuration file should have the following form: @@ -276,6 +297,7 @@ variable names in Python. [TOC](#TOC)
+ ### Test cases The test cases are defined in Python source files utilizing the unittest module, @@ -363,6 +385,7 @@ writing 0 ten times. The same is of course possible for strings. [TOC](#TOC)
+ ### Multiline strings It is generally recommended to use Python's multiline strings (strings starting @@ -401,6 +424,7 @@ character. To achieve that put the `"""` on the following line. [TOC](#TOC)
+ ### Paths Some test cases require the specification of paths (e.g. to the location of test @@ -430,6 +454,7 @@ class AnInformativeName(metaclass=CaseMeta): [TOC](#TOC)
+ ## Advanced test cases This section describes more advanced features that are probably not necessary @@ -437,6 +462,7 @@ the "standard" usage of the test suite.
+ ### Providing standard input to commands The test suite supports providing a standard input to commands in a similar @@ -483,6 +509,7 @@ test suite will implicitly assume `None` for every command. [TOC](#TOC)
+ ### Using a different output encoding The test suite will try to interpret the program's output as utf-8 encoded @@ -526,6 +553,7 @@ encodings can be found [TOC](#TOC)
+ ### Working with binary output Some programs output binary data directly to stdout or stderr. Such programs can @@ -595,28 +623,30 @@ and your test case will get only the specified environment variables. [TOC](#TOC)
+ ### Creating file copies For tests that modify their input file it is useful to run these with a -disposable copy of the input file and not with the original. For this purpose +disposable copy of the input file and not with the original. For this purpose, the test suite features a decorator which creates a copy of the supplied files -and deletes the copies after the test ran. +and preserves the copies after the test finishes. All temporary copies are +stored in the `$tmp_path` directory. Example: ```python # -*- coding: utf-8 -*- -import system_tests +import system_tests, CopyTmpFiles -@system_tests.CopyFiles("$filename", "$some_path/another_file.txt") +@CopyTmpFiles("$data_path/invalid_input_file.txt") class AnInformativeName(metaclass=system_tests.CaseMeta): - filename = "invalid_input_file" + filename = path("$tmp_path/invalid_input_file.txt") commands = [ - "$binary -c $import_file -i $filename" - ] + "$binary --option $filename" + ] retval = ["$abort_exit_value"] stdout = ["Reading $filename"] stderr = [ @@ -626,16 +656,16 @@ error in $filename ] ``` -In this example, the test suite would automatically create a copy of the files -`invalid_input_file` and `$some_path/another_file.txt` (`some_path` would be of -course expanded too) named `invalid_input_file_copy` and -`$some_path/another_file_copy.txt`. After the test ran, the copies are -deleted. Please note that variable expansion in the filenames is possible. +In this example, the test suite would automatically create a copy of the file +`$data_path/invalid_input_file.txt` called `$tmp_path/invalid_input_file.txt`. +After the test runs, the temporary copy is preserved. Please note that variable +expansion in the filenames is possible. [TOC](#TOC)
+ ### Customizing the output check Some tests do not require a "brute-force" comparison of the whole output of a @@ -694,6 +724,7 @@ class AnInformativeName(metaclass=system_tests.CaseMeta): [TOC](#TOC)
+ ### Running all commands under valgrind The test suite can run all commands under a memory checker like @@ -736,6 +767,7 @@ into account: [TOC](#TOC)
+ ### Manually expanding variables in strings In case completely custom checks have to be run but one still wants to access @@ -791,6 +823,7 @@ will result in `another_string` being "foo" and not "bar". [TOC](#TOC)
+ ### Hooks The `Case` class provides two hooks that are run after each command and after @@ -848,6 +881,7 @@ class AnInformativeName(metaclass=system_tests.CaseMeta): [TOC](#TOC)
+ ## Bash test cases - Previously, Exiv2 had some bash test scripts, which were saved as the file `EXIV2_DIR/test/*.sh`. We're going to rewrite them as Python test scripts and save them to the directory `EXIV2_DIR/tests/bash_tests`.