Merge pull request #8 from openMSL/7-separate-model-behavior-tests-fr…

…om-integration-tests

Update CL3 documentation

doc/modules/test-architecture/pages/cl3.adoc

@@ -1,18 +1,29 @@
 # CL 3: Quantitative Verification
 
-For credibility assessment level 3, the model is again employed in a co-simulation with a trace file player or scenario engine.
-During the simulation, a trace file is generated containing the model output of every simulation time step.
-This is done by connecting a https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] to the model output in the SSP specification of the test.
+For credibility assessment level 3, the goal is to verify the behavior of a model.
+A specific test folder (_test/behavior_) contains the tests for CL3.
+The model is employed in a co-simulation with a trace file player.
+The input trace file can either be placed in the test folder (as shown https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/001_detected_object[here]) or be generated by a python script directly in the CI pipeline (as shown https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/002_object_outside_fov[here]).
+During the simulation, an output trace file is generated containing the model output of every simulation time step.
+This is done by connecting a https://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] to the model output in the SSP specification of the test.
 The trace file writer is pre-built in every test environment.
-The FMU is located at _/tmp/tracefile_writer_fmu/osi-sensordata-trace-file-writer.fmu_.
-After the co-simulation is finished, the GitHub action scans the test folder for python files.
-If it finds one, the python script is called with the path to the just generated trace file as an argument.
-The file name of the python file does not matter.
-However, if there is more than one python file in the directory, the analysis will not be performed since it is unclear, which file to use.
-If supplementary files are necessary for the analysis, they are to be placed in a subfolder.
-
-The purpose is to test the proper implementation of the model code in terms of its quantifiable requirements.
+The FMU is located at _/tmp/tracefile_writer_fmu/osi-trace-file-writer.fmu_.
+After the co-simulation is finished, the GitHub action scans the behavior test folder for file named _analyze.py_.
+If it is found, the python script is called with the path to the just generated output trace file as an argument.
+
+All tests in CL 3 are considered behavior tests, since they all verify the implemented behavior of the model.
+Therefore, every test is located in an individual folder in _test/behavior_.
+The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number.
+The folder SHALL contain a https://ssp-standard.org/publications/SSP10RC1/SystemStructureAndParameterization10RC1.pdf[system structure definition file (.ssd)].
+In this file, the utilized model input (trace file player or esmini) as well as the https://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] are specified.
+The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion.
+This README.md SHALL follow the same https://github.com/openMSL/.github/blob/main/doc/integration_test_readme_template.md[template] as the integration tests.
+Other simulation artefacts such as the trace file or scenario to be played as well as the python analysis script are also to be placed in that test folder.
+Example implementations in the _test/behavior_ folder can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/[sensor model template repository].
+
+The purpose is to test the proper implementation of the model behavior in terms of its quantifiable requirements.
 E.g. the sensor is rotated around all 3 axis, and it is evaluated, if the coordinate transformations in the model are correct.
+Or sensor model is detecting objects in its field of view correctly.
 This can be achieved in two ways:
 
 1. The simulation trace file is analyzed by a custom standalone python script.
@@ -28,12 +39,12 @@ image::cl3_test.svg[Test setup for credibility assessment level 3,600]
 
 The OSI trace file generated by the co-simulation is read in by a python script after the simulation has finished.
 The python script can perform a variety of tests on the trace file, depending on the specific scenario and use-case.
-For example, the scenario places an object at a certain distance inside the field of view of a sensor.
+For example, the ground truth contained in the input trace file places an object at a certain distance inside the field of view of a sensor.
 The python script can analyze the sensor model output and check, if the object was detected at the expected distance.
 Another example would be to place an object outside of the field of view and check, that it was not detected.
 Furthermore, distribution functions for detections or expected number of detections on an object can be checked.
 
-An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/004_tracefile_analysis[sensor model template repository].
+An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/001_detected_object[sensor model template repository].
 
 ## CL 3.2 Trace File Comparison