Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Launch-based Testing #161

Closed
hidmic opened this issue Jan 14, 2019 · 3 comments
Closed

Launch-based Testing #161

hidmic opened this issue Jan 14, 2019 · 3 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@hidmic
Copy link
Contributor

hidmic commented Jan 14, 2019

Feature request

This is a feature request to enable launch-based testing, using (and likely extending) the current launch API. Acceptance criteria is limited to running both gtest and stdout-matching launch tests through regular build tools (e.g. colcon).

Feature description

In ROS, the rostest integraton with roslaunch allows for running test nodes in otherwise regular launch files.
When run with the rostest CLI, these nodes (either gtest or nosetests based executables) make the whole launch test fail (or succeed) based on their return code. Fixture nodes can also cause a test failure by means of the required tag. All test nodes have user specified timeouts. Integration with catkin (i.e. CMake) is readily available too.

Similarly, though in greater generality, launch_testing should provide the means to:

  • Describe an executable under test, along with an executable or a group thereof to act as test fixture.
  • Assert outcome of the executable under test (return code, stdout, stderr, etc.), which may
    or may not be using a testing framework itself (e.g. gtest, pytest, etc.).
  • Assert a failure after user specified test timeout.
  • Optionally assert a failure upon normal or abnormal termination of test fixture executables.
  • Teardown the whole running launch test description upon assertion failure.
  • Aggregate test outcome in a standard file format for latter recollection (e.g. xunit files).
  • Integrate launch test descriptions as yet another package test instance, using specific CMake API.

Implementation considerations

launch_testing

To maximize code reuse, test descriptions must compose executable descriptions (see
#114 for a precedent on this pattern).

For test descriptions, assertions can be coupled or decoupled from their respective conditions. Compare:

GTest(Node(package='demos_rclcpp', node_executable='talker')),
MatchingTest(Node(package='demos_rclpy', node_executable='listener'), matching_pattern='.*ERROR.*')

where each new test kind is a subclass of a base Test action (an Action subclass itself), with

Assert(GTestPasses(Node())),
Expect(OutputMatches(Node()))

where two Action subclasses, Assert (terminating) and Expect (non-terminating), take a Condition instance that in turn deals with the executables.

In either case, the composing objects will setup timers to timeout and process event handlers to collect output, deal with testing framework details if need be (e.g. argument passing) and emit specific events (i.e. AssertionError) upon test failure. Then, a tests-aware launch.LaunchService may react accordingly, shutting down the whole launch description and aggregating tests data.

Note that in (1), defining launch description behavior upon executable termination becomes necessary (orthogonal to testing, but instrumental to mimic'ing ROS required tag usage for fixture executables). See the legacy exit handlers implementation for feature reference (though implementation in this case may rely on events).

ros2launch

CLI must be updated with test verb to easily invoke a test-aware launch.LaunchService.

@hidmic
Copy link
Contributor Author

hidmic commented Jan 14, 2019

On a first step, we will:

  1. Write some testing examples on how the API would look like. (https://github.com/ros2/examples/issues/223).
  2. Put together API basic functionality (no result aggregation, single test per launch, etc.),
    migrating legacy features if needed. (Add actions for running gtest and pytests #176)
  3. Add basic CMake integration. ([launch] Add basic CMake support to run launch-based tests #162)
  4. Migrate existing legacy launch API use cases. (Remove legacy API #159)

@wjwwood wjwwood added this to the dashing milestone Jan 14, 2019
@cottsay cottsay added the enhancement New feature or request label Jan 17, 2019
@mjcarroll mjcarroll added the ready Work is about to start (Kanban column) label Jan 24, 2019
@hidmic
Copy link
Contributor Author

hidmic commented Mar 7, 2019

Basic testing functionality is provided by the launch_testing package after #167. A feature complete equivalent of ROS 1 rostest is on the making at ApexAI/apex_rostest.

@hidmic hidmic added in progress Actively being worked on (Kanban column) and removed ready Work is about to start (Kanban column) labels Mar 7, 2019
@hidmic hidmic added in review Waiting for review (Kanban column) and removed in progress Actively being worked on (Kanban column) labels May 16, 2019
@hidmic
Copy link
Contributor Author

hidmic commented May 16, 2019

Functionality provided here has been superseded by that introduced on #208. Refer to that issue for further reference.

@hidmic hidmic closed this as completed May 16, 2019
@hidmic hidmic removed the in review Waiting for review (Kanban column) label May 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants