From 03471a9dee1285ccdf2a177de92ec71319ee55d1 Mon Sep 17 00:00:00 2001 From: ClemensLinnhoff Date: Wed, 20 Mar 2024 11:23:54 +0100 Subject: [PATCH 1/4] Update CL3 documentation Signed-off-by: ClemensLinnhoff --- doc/modules/test-architecture/pages/cl3.adoc | 23 +++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/doc/modules/test-architecture/pages/cl3.adoc b/doc/modules/test-architecture/pages/cl3.adoc index 6f77910..f383536 100644 --- a/doc/modules/test-architecture/pages/cl3.adoc +++ b/doc/modules/test-architecture/pages/cl3.adoc @@ -1,18 +1,31 @@ # CL 3: Quantitative Verification -For credibility assessment level 3, the model is again employed in a co-simulation with a trace file player or scenario engine. +For credibility assessment level 3, the goal is to verify the behavior of a model. +A specific test folder (_test/behavior_) contains the tests for CL3. +The model is again employed in a co-simulation with a trace file player or scenario engine. During the simulation, a trace file is generated containing the model output of every simulation time step. This is done by connecting a https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] to the model output in the SSP specification of the test. The trace file writer is pre-built in every test environment. The FMU is located at _/tmp/tracefile_writer_fmu/osi-sensordata-trace-file-writer.fmu_. -After the co-simulation is finished, the GitHub action scans the test folder for python files. +After the co-simulation is finished, the GitHub action scans the behavior test folder for python files. If it finds one, the python script is called with the path to the just generated trace file as an argument. The file name of the python file does not matter. However, if there is more than one python file in the directory, the analysis will not be performed since it is unclear, which file to use. If supplementary files are necessary for the analysis, they are to be placed in a subfolder. -The purpose is to test the proper implementation of the model code in terms of its quantifiable requirements. +All tests in CL 3 are considered behavior tests, since they all verify the implemented behavior of the model. +Therefore, every test is located in an individual folder in _test/behavior_. +The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number. +The folder SHALL contain a https://ssp-standard.org/publications/SSP10RC1/SystemStructureAndParameterization10RC1.pdf[system structure definition file (.ssd)]. +In this file, the utilized model input (trace file player or esmini) as well as the https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] are specified. +The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion. +This README.md SHALL follow the same https://github.com/openMSL/.github/blob/main/doc/integration_test_readme_template.md[template] as the integration tests. +Other simulation artefacts such as the trace file or scenario to be played as well as the python analysis script are also to be placed in that test folder. +Example implementations in the _test/behavior_ folder can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/[sensor model template repository]. + +The purpose is to test the proper implementation of the model behavior in terms of its quantifiable requirements. E.g. the sensor is rotated around all 3 axis, and it is evaluated, if the coordinate transformations in the model are correct. +Or sensor model is detecting objects in its field of view correctly. This can be achieved in two ways: 1. The simulation trace file is analyzed by a custom standalone python script. @@ -28,12 +41,12 @@ image::cl3_test.svg[Test setup for credibility assessment level 3,600] The OSI trace file generated by the co-simulation is read in by a python script after the simulation has finished. The python script can perform a variety of tests on the trace file, depending on the specific scenario and use-case. -For example, the scenario places an object at a certain distance inside the field of view of a sensor. +For example, the ground truth contained in the input trace file places an object at a certain distance inside the field of view of a sensor. The python script can analyze the sensor model output and check, if the object was detected at the expected distance. Another example would be to place an object outside of the field of view and check, that it was not detected. Furthermore, distribution functions for detections or expected number of detections on an object can be checked. -An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/integration/004_tracefile_analysis[sensor model template repository]. +An example implementation of this test can be found in the https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/001_detected_object[sensor model template repository]. ## CL 3.2 Trace File Comparison From 7be76c129ef3874fdfb07dde8bbf868bec4a5317 Mon Sep 17 00:00:00 2001 From: ClemensLinnhoff Date: Wed, 20 Mar 2024 18:12:31 +0100 Subject: [PATCH 2/4] Add section about generating a test trace file Signed-off-by: ClemensLinnhoff --- doc/modules/test-architecture/pages/cl3.adoc | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/doc/modules/test-architecture/pages/cl3.adoc b/doc/modules/test-architecture/pages/cl3.adoc index f383536..45b4bfb 100644 --- a/doc/modules/test-architecture/pages/cl3.adoc +++ b/doc/modules/test-architecture/pages/cl3.adoc @@ -2,16 +2,14 @@ For credibility assessment level 3, the goal is to verify the behavior of a model. A specific test folder (_test/behavior_) contains the tests for CL3. -The model is again employed in a co-simulation with a trace file player or scenario engine. -During the simulation, a trace file is generated containing the model output of every simulation time step. -This is done by connecting a https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] to the model output in the SSP specification of the test. +The model is employed in a co-simulation with a trace file player. +The input trace file can either be placed in the test folder (as shown https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/001_detected_object[here]) or be generated by a python script directly in the CI pipeline (as shown https://github.com/openMSL/sl-1-0-sensor-model-repository-template/tree/main/test/behavior/002_object_outside_fov[here]). +During the simulation, an output trace file is generated containing the model output of every simulation time step. +This is done by connecting a https://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] to the model output in the SSP specification of the test. The trace file writer is pre-built in every test environment. -The FMU is located at _/tmp/tracefile_writer_fmu/osi-sensordata-trace-file-writer.fmu_. -After the co-simulation is finished, the GitHub action scans the behavior test folder for python files. -If it finds one, the python script is called with the path to the just generated trace file as an argument. -The file name of the python file does not matter. -However, if there is more than one python file in the directory, the analysis will not be performed since it is unclear, which file to use. -If supplementary files are necessary for the analysis, they are to be placed in a subfolder. +The FMU is located at _/tmp/tracefile_writer_fmu/osi-trace-file-writer.fmu_. +After the co-simulation is finished, the GitHub action scans the behavior test folder for file named _analyze.py_. +If it is found, the python script is called with the path to the just generated output trace file as an argument. All tests in CL 3 are considered behavior tests, since they all verify the implemented behavior of the model. Therefore, every test is located in an individual folder in _test/behavior_. From 568ec1554996074fd1382837ba3dee45e98d9fe1 Mon Sep 17 00:00:00 2001 From: ClemensLinnhoff Date: Mon, 25 Mar 2024 11:20:25 +0100 Subject: [PATCH 3/4] Fix trace file writer link Signed-off-by: ClemensLinnhoff --- doc/modules/test-architecture/pages/cl3.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/modules/test-architecture/pages/cl3.adoc b/doc/modules/test-architecture/pages/cl3.adoc index 45b4bfb..9e2a270 100644 --- a/doc/modules/test-architecture/pages/cl3.adoc +++ b/doc/modules/test-architecture/pages/cl3.adoc @@ -15,7 +15,7 @@ All tests in CL 3 are considered behavior tests, since they all verify the imple Therefore, every test is located in an individual folder in _test/behavior_. The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number. The folder SHALL contain a https://ssp-standard.org/publications/SSP10RC1/SystemStructureAndParameterization10RC1.pdf[system structure definition file (.ssd)]. -In this file, the utilized model input (trace file player or esmini) as well as the https://github.com/Persival-GmbH/osi-sensordata-trace-file-writer[trace file writer] are specified. +In this file, the utilized model input (trace file player or esmini) as well as the hhttps://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] are specified. The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion. This README.md SHALL follow the same https://github.com/openMSL/.github/blob/main/doc/integration_test_readme_template.md[template] as the integration tests. Other simulation artefacts such as the trace file or scenario to be played as well as the python analysis script are also to be placed in that test folder. From 0b47ea378b566603841694aa08487cccd2941369 Mon Sep 17 00:00:00 2001 From: ClemensLinnhoff Date: Mon, 25 Mar 2024 11:20:56 +0100 Subject: [PATCH 4/4] Fix trace file writer link Signed-off-by: ClemensLinnhoff --- doc/modules/test-architecture/pages/cl3.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/modules/test-architecture/pages/cl3.adoc b/doc/modules/test-architecture/pages/cl3.adoc index 9e2a270..0b8b59b 100644 --- a/doc/modules/test-architecture/pages/cl3.adoc +++ b/doc/modules/test-architecture/pages/cl3.adoc @@ -15,7 +15,7 @@ All tests in CL 3 are considered behavior tests, since they all verify the imple Therefore, every test is located in an individual folder in _test/behavior_. The individual test folder SHALL follow the naming scheme "xxx_short_description", where xxx is a three digit consecutive number. The folder SHALL contain a https://ssp-standard.org/publications/SSP10RC1/SystemStructureAndParameterization10RC1.pdf[system structure definition file (.ssd)]. -In this file, the utilized model input (trace file player or esmini) as well as the hhttps://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] are specified. +In this file, the utilized model input (trace file player or esmini) as well as the https://github.com/openMSL/sl-5-6-osi-trace-file-writer[trace file writer] are specified. The test folder SHALL additionally contain a README.md file, which describes the test system, scenario and pass/fail criterion. This README.md SHALL follow the same https://github.com/openMSL/.github/blob/main/doc/integration_test_readme_template.md[template] as the integration tests. Other simulation artefacts such as the trace file or scenario to be played as well as the python analysis script are also to be placed in that test folder.