Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to async state service request #461

Merged
merged 7 commits into from
Dec 10, 2020
Merged

Switch to async state service request #461

merged 7 commits into from
Dec 10, 2020

Conversation

iche033
Copy link
Contributor

@iche033 iche033 commented Nov 17, 2020

The scene broadcsater system provides a synchronous state service which blocks until the state is filled in the next iteration. In normal ign gazebo usage this works fine but sometimes causes deadlock when running the distributed_levels demo due to a race condition. This PR adds a new non-blocking async state service in the scene broadcaster system, and switches the GUI to call this async state service instead.

More info:

The NetworkManager classes in distributed simulation rely heavily on ign-transport for synchronizing all secondaries with the primary. The deadlock situation happens when the state service call is invoked while the secondaries are being stepped at the same time by the primary:

  1. Primary sends msgs to step all Secondaries
  2. A state service request is received by the scene broadcaster system in Primary and blocks until the state service is filled in the next iteration
  3. step ack msgs sent back by the Secondaries are never received by the Primary because the ign-transport thread is being blocked.
  4. Primary cannot step the simulation without ack msg, which means the scene broadcaster can not fill the state required by the state service call
  5. Deadlock!

Another option is to update the distributed system stepping synchronization strategy - I started with this fix and have a workaround implemented but ultimately I came back to the async fix because I think that it would be good to avoid a blocking service call in general.

Signed-off-by: Ian Chen [email protected]

Signed-off-by: Ian Chen <[email protected]>
@iche033 iche033 changed the base branch from ign-gazebo4 to ign-gazebo3 November 17, 2020 06:48
@github-actions github-actions bot added the 🔮 dome Ignition Dome label Nov 17, 2020
@codecov
Copy link

codecov bot commented Nov 17, 2020

Codecov Report

Merging #461 (2127a27) into ign-gazebo3 (01fc2ff) will increase coverage by 11.67%.
The diff coverage is 82.24%.

Impacted file tree graph

@@               Coverage Diff                @@
##           ign-gazebo3     #461       +/-   ##
================================================
+ Coverage        65.71%   77.38%   +11.67%     
================================================
  Files              127      205       +78     
  Lines             6238    11049     +4811     
================================================
+ Hits              4099     8550     +4451     
- Misses            2139     2499      +360     
Impacted Files Coverage Δ
include/ignition/gazebo/EntityComponentManager.hh 100.00% <ø> (ø)
include/ignition/gazebo/Link.hh 100.00% <ø> (ø)
include/ignition/gazebo/Model.hh 100.00% <ø> (ø)
include/ignition/gazebo/SdfEntityCreator.hh 100.00% <ø> (ø)
include/ignition/gazebo/Server.hh 100.00% <ø> (ø)
include/ignition/gazebo/ServerConfig.hh 100.00% <ø> (ø)
include/ignition/gazebo/components/Component.hh 100.00% <ø> (ø)
include/ignition/gazebo/gui/GuiEvents.hh 0.00% <0.00%> (ø)
include/ignition/gazebo/gui/GuiRunner.hh 100.00% <ø> (+100.00%) ⬆️
src/SimulationRunner.hh 100.00% <ø> (ø)
... and 185 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92c2cf7...8f2663b. Read the comment docs.

Copy link
Contributor

@chapulina chapulina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The general idea looks good to me. I just have some questions.

src/systems/scene_broadcaster/SceneBroadcaster.cc Outdated Show resolved Hide resolved
src/systems/scene_broadcaster/SceneBroadcaster.cc Outdated Show resolved Hide resolved
src/gui/GuiRunner.cc Show resolved Hide resolved
@gazebosim gazebosim deleted a comment from chapulina Nov 17, 2020
@azeey azeey removed their request for review November 25, 2020 22:23
Copy link
Contributor

@caguero caguero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me. I just left two minor comments.

src/systems/scene_broadcaster/SceneBroadcaster.cc Outdated Show resolved Hide resolved
src/systems/scene_broadcaster/SceneBroadcaster.cc Outdated Show resolved Hide resolved
@chapulina chapulina merged commit 4341bb2 into ign-gazebo3 Dec 10, 2020
@chapulina chapulina deleted the async_state branch December 10, 2020 04:29
doisyg pushed a commit to wyca-robotics/ign-gazebo that referenced this pull request Dec 11, 2020
Signed-off-by: Ian Chen <[email protected]>
Signed-off-by: Guillaume Doisy <[email protected]>
doisyg pushed a commit to wyca-robotics/ign-gazebo that referenced this pull request Dec 11, 2020
Signed-off-by: Ian Chen <[email protected]>
Signed-off-by: Guillaume Doisy <[email protected]>
Lobotuerk added a commit to Lobotuerk/ign-gazebo that referenced this pull request Jan 11, 2021
* Initial commit

Signed-off-by: Martiño Crespo <[email protected]>

* Added plugin files

Signed-off-by: Martiño Crespo <[email protected]>

* Initial .sdf file

Signed-off-by: Martiño Crespo <[email protected]>

* Pipeline for getting sensor contacts

Signed-off-by: Martiño Crespo <[email protected]>

* Added visualization of postion and forces

Signed-off-by: Martiño Crespo <[email protected]>

* Added plugin for Transform Control

Signed-off-by: Martiño Crespo <[email protected]>

* Added initial interpolation

Signed-off-by: Martiño Crespo <[email protected]>

* Code check

Signed-off-by: Martiño Crespo <[email protected]>

* Update info after specified milliseconds

Signed-off-by: Martiño Crespo <[email protected]>

* Set marker lifetime for better performance

Signed-off-by: Martiño Crespo <[email protected]>

* Added Depth Camera

Signed-off-by: Martiño Crespo <[email protected]>

* Added callback and unpacking to Depth Camera messages

Signed-off-by: Martiño Crespo <[email protected]>

* Compute and visualize normal forces

Signed-off-by: Martiño Crespo <[email protected]>

* PR Feedback 1

Signed-off-by: Martiño Crespo <[email protected]>

* Update sdf example to a more realistic environment

Signed-off-by: Martiño Crespo <[email protected]>

* Allow moving Depth Camera from model origin

Signed-off-by: Martiño Crespo <[email protected]>

* Make markers dimensions available as parameters

Signed-off-by: Martiño Crespo <[email protected]>

* Visualize sensor as a marker instead of <visual> and filter out normal forces outside sensor

Signed-off-by: Martiño Crespo <[email protected]>

* PR Feedback 2

Signed-off-by: Martiño Crespo <[email protected]>

* PR Feedback 3

Signed-off-by: Martiño Crespo <[email protected]>

* Minor fixes for sdf, sensor marker and profiler

Signed-off-by: Martiño Crespo <[email protected]>

* PR Feedback 4

Signed-off-by: Martiño Crespo <[email protected]>

* fix mac warning

Signed-off-by: Mabel Zhang <[email protected]>

* Helper function to set component data (gazebosim#436)

Signed-off-by: Louise Poubel <[email protected]>

* Remove unneeded if statement (gazebosim#432)

Signed-off-by: John Shepherd <[email protected]>

* Fixes flaky RecordAndPlayback test in INTEGRATION_log_system (gazebosim#463)

The flakiness comes from two sources:
1. Poses recorded by the LogRecorder are published by the
   SceneBroadcaster system throttled at 60 Hz. The throttle mechanism
   uses real-time instead of sim-time which causes a variance in the
   number of recorded poses from run to run. However, the expected number of
   recorded poses was calculated with the assumption that the simulation
   would run with a 1.0 RTF. If the CPU load is high, there could be a
   mismatch between the expected and the actual number of recorded
   poses, which causes the test to fail.
   This can be checked by running the test with
`cpulimit -l 20 -f  bin/INTEGRATION_log_system  -- --gtest_filter="*RecordAndPlayback"`

2. An attempt is made to match up played back poses with the closest
   timestamp in the recorded file. These poses are again published by
   the SceneBroadcaster, so they are subject to the same kind of timing
   issues as the recorded poses.

The solution in this patch for the first issue is to determine the
expected number of recorded poses by counting them in a separate system
that mimics the throttling mechanism of ign-transport. For the second
issue, a testing system is added to the playback server bypassing the
SceSceneBroadcaster altogether.

Signed-off-by: Addisu Z. Taddese <[email protected]>

* Make PeerTracker test more robust (gazebosim#452)

Signed-off-by: Louise Poubel <[email protected]>

Co-authored-by: Michael Carroll <[email protected]>

* Clarify how sim time is interpreted in a System's step (gazebosim#467)

Signed-off-by: Ashton Larkin <[email protected]>

* 3 ➡️  4: fixes for gazebosim#463 (gazebosim#469)

Signed-off-by: Louise Poubel <[email protected]>

* Link link tutrial (gazebosim#472)

Signed-off-by: Louise Poubel <[email protected]>

* Switch to async state service request (gazebosim#461)

Signed-off-by: Ian Chen <[email protected]>

* Use a std::promise/std::future to avoid busy waiting the step ack messages in NetworkManagerPrimary (gazebosim#470)

Signed-off-by: Ivan Santiago Paunovic <[email protected]>
Signed-off-by: Louise Poubel <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>

* Fix tests that use Fuel paths with uppercase letters (gazebosim#480)

As of gazebosim/gz-fuel-tools#130, fuel paths use all lowercase letters. This fixes some tests that had uppercase letters.
Signed-off-by: Addisu Z. Taddese <[email protected]>

* 4.1.0 (gazebosim#485)

Signed-off-by: Ashton Larkin <[email protected]>

* Update key event handling (gazebosim#466)

Signed-off-by: John Shepherd <[email protected]>

* Fix slot in Plotting plugin (gazebosim#490)

Signed-off-by: Alejandro Hernández <[email protected]>

* Tape Measure Plugin (gazebosim#456)

Signed-off-by: John Shepherd <[email protected]>

* Satisfy make codecheck (gazebosim#491)

Signed-off-by: Louise Poubel <[email protected]>

* Move deselect and preview termination to render thread (gazebosim#493)

Signed-off-by: John Shepherd <[email protected]>

* Fix codecheck (gazebosim#499)

Signed-off-by: Nate Koenig <[email protected]>

Co-authored-by: Nate Koenig <[email protected]>

* Logical Audio Sensor Plugin (gazebosim#401)

Signed-off-by: Ashton Larkin <[email protected]>

* Codecheck - initialize primitive in LogicalAudio component (gazebosim#502)

Signed-off-by: Ashton Larkin <[email protected]>

* 3 to 4: fix codecheck for ign-gazebo4

Signed-off-by: Ashton Larkin <[email protected]>

* 3 to 4: resolve codecheck warnings

Signed-off-by: Ashton Larkin <[email protected]>

* add frame_id and child_frame_id attribute support for DiffDrive (gazebosim#361)

Add configuration of the odom frame_id and child_frame_id fields from sdf attributes <frame_id> and <child_frame_id>

Signed-off-by: Guillaume <[email protected]>
Signed-off-by: Guillaume Doisy <[email protected]>

* Require ign-gui 4.1.0 (gazebosim#505)

Signed-off-by: Louise Poubel <[email protected]>

* Fix shadow artifacts by disabling double sided rendering (gazebosim#446)

* read double sided sdf param

Signed-off-by: Ian Chen <[email protected]>

* update migration

Signed-off-by: Ian Chen <[email protected]>

* Make the tunnels example world more interesting (gazebosim#462)

Signed-off-by: Louise Poubel <[email protected]>

* add double sided msg to sdf conversion

Signed-off-by: Ian Chen <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>

* Add ability to record video based on sim time (gazebosim#414)

* add ability to record video from gui camera using sim time

Signed-off-by: Ian Chen <[email protected]>

* add msg

Signed-off-by: Ian Chen <[email protected]>

* use QueryBoolText

Signed-off-by: Ian Chen <[email protected]>

Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Louise Poubel <[email protected]>

* Add lockstep mode to video recording (gazebosim#419)

Signed-off-by: Ian Chen <[email protected]>
Signed-off-by: Louise Poubel <[email protected]>

Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Louise Poubel <[email protected]>

* Kinetic energy monitor plugin (gazebosim#492)

Signed-off-by: Gonzalo de Pedro <[email protected]>
Signed-off-by: Nate Koenig <[email protected]>

Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Louise Poubel <[email protected]>

* Disable right click menu when using measuring tool (gazebosim#458)

Signed-off-by: John Shepherd <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>

* Bump to 3.6.0 (gazebosim#524)

Signed-off-by: Louise Poubel <[email protected]>

* Don't make docs on macOS (gazebosim#528)

add comment about doxygen bug

Signed-off-by: Louise Poubel <[email protected]>
Signed-off-by: Steve Peters <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>

* Updates to ardupilot migration tutorial (gazebosim#525)

Signed-off-by: Louise Poubel <[email protected]>

* Update gtest to 1.10.0 for Windows compilation (ign-gazebo3) (gazebosim#506)

* Compile new gtest with c++11
* Use INSTANTIATE_TEST_SUITE_P instead of deprecated -INSTANTIATE_TEST_CASE_P

Signed-off-by: Jose Luis Rivero <[email protected]>

* Apply suggestions from code review

Signed-off-by: Michael Carroll <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>

* change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527)

See: https://bugreports.qt.io/browse/QTBUG-89114
Signed-off-by: acxz <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>
Signed-off-by: Louise Poubel <[email protected]>

* Generate valid topics everywhere (support names with spaces) (gazebosim#522)

Signed-off-by: Louise Poubel <[email protected]>

* One more tutorial version bump

Signed-off-by: Michael Carroll <[email protected]>

* Fix bad merge

Signed-off-by: Michael Carroll <[email protected]>

* change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527)

See: https://bugreports.qt.io/browse/QTBUG-89114
Signed-off-by: acxz <[email protected]>

Co-authored-by: Louise Poubel <[email protected]>
Signed-off-by: Louise Poubel <[email protected]>

* Generate valid topics everywhere (support names with spaces) (gazebosim#522)

Signed-off-by: Louise Poubel <[email protected]>

* Change deprecated test case->suite

Signed-off-by: Michael Carroll <[email protected]>

* 3 ➡️ 4 (gazebosim#533)

* Clarify how sim time is interpreted in a System's step (gazebosim#467)
* add frame_id and child_frame_id attribute support for DiffDrive (gazebosim#361)
* Add ability to record video based on sim time (gazebosim#414)
* add ability to record video from gui camera using sim time
* add msg
* use QueryBoolText
* Add lockstep mode to video recording (gazebosim#419)
* Disable right click menu when using measuring tool (gazebosim#458)
* Bump to 3.6.0 (gazebosim#524)
* Don't make docs on macOS (gazebosim#528)
* Updates to ardupilot migration tutorial (gazebosim#525)
* Update gtest to 1.10.0 for Windows compilation (ign-gazebo3) (gazebosim#506)
* Compile new gtest with c++11
* Use INSTANTIATE_TEST_SUITE_P instead of deprecated -INSTANTIATE_TEST_CASE_P
* Apply suggestions from code review
* One more tutorial version bump
* Fix bad merge
* change nullptr to a int ptr for qt 5.15.2 bug (gazebosim#527)
* Generate valid topics everywhere (support names with spaces) (gazebosim#522)
* Change deprecated test case->suite

Co-authored-by: Ashton Larkin <[email protected]>
Co-authored-by: G.Doisy <[email protected]>
Co-authored-by: Ian Chen <[email protected]>
Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Louise Poubel <[email protected]>
Co-authored-by: John Shepherd <[email protected]>
Co-authored-by: Steve Peters <[email protected]>
Co-authored-by: Jose Luis Rivero <[email protected]>
Co-authored-by: Akash Patel <[email protected]>

* Add support for topic statistics on breadcrumb deployments (gazebosim#532)

* Add support for topic statistics on breadcrumb deployments

Signed-off-by: Nate Koenig <[email protected]>

* Require version 9.1 of ignition transport

Signed-off-by: Nate Koenig <[email protected]>

* Move to after mutex

Signed-off-by: Nate Koenig <[email protected]>

Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Carlos Agüero <[email protected]>

Co-authored-by: Martiño Crespo <[email protected]>
Co-authored-by: Louise Poubel <[email protected]>
Co-authored-by: Mabel Zhang <[email protected]>
Co-authored-by: John Shepherd <[email protected]>
Co-authored-by: Addisu Z. Taddese <[email protected]>
Co-authored-by: Michael Carroll <[email protected]>
Co-authored-by: Ashton Larkin <[email protected]>
Co-authored-by: Ian Chen <[email protected]>
Co-authored-by: Ivan Santiago Paunovic <[email protected]>
Co-authored-by: Alejandro Hernández Cordero <[email protected]>
Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: Nate Koenig <[email protected]>
Co-authored-by: G.Doisy <[email protected]>
Co-authored-by: Gonzo <[email protected]>
Co-authored-by: Steve Peters <[email protected]>
Co-authored-by: Jose Luis Rivero <[email protected]>
Co-authored-by: Akash Patel <[email protected]>
Co-authored-by: Carlos Agüero <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔮 dome Ignition Dome
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants