Replace ign common plugin loader with ign-plugin #38

ahcorde · 2020-08-19T12:18:51Z

This PR is related with this issue #18

I have some issue. When I run the tests for ign-sensors4 everything look good. Then problem is inside ign-gazebo code, when the library is loaded for some plugins is not able to find the pluginNames.

For example:

Camera
- In the test: LoadLib() returns that the loaded plugin is N8ignition7sensors2v416SensorTypePluginINS1_12CameraSensorEEE -> ignition::sensors::v4::SensorTypePluginignition::sensors::v4::CameraSensor
- Inside ign-gazebo code: the LoadLib() returns a empty structure. (No loaded plugins)
Depth Camera or GPU lidar are working properly.

Maybe @mxgrey or @scpeters can add some light here.

Signed-off-by: ahcorde [email protected]

Signed-off-by: ahcorde <[email protected]>

mxgrey · 2020-08-20T06:57:31Z

Off the top of my head, I can't think of why no plugins are showing up for ign-gazebo, but I would guess maybe something funny is going on with the linker. Does the problem occur on all platforms (Linux, Mac, Windows)?

ahcorde · 2020-08-21T08:47:49Z

I try to explain what I think it's happening here:

ign-gazebo is linking against camera from ign-sensor in the Sensor.cc. This is the CMakeLists.txt. When the program starts then the symbols from libignition-gazebo4-sensors-system.so are loaded which also includes libignition-sensors4-camera.so because as I explained before the CMake is linking with the sensor library.
This same file (Sensor.cc) is loading the camera plugin (libignition-sensors4-camera.so) using dlsym, In this line of code if there is a <sensor name="camera" type="camera"> in the sdf file. Here is where the problem arises

The Macro IGN_SENSORS_REGISTER_SENSOR(CameraSensor) is creating a struct in an anonymous namespace. Inside this namespace a static variable is created which is in charge of loading the symbols. The first time that this struct is generated is when the class is called (linked by the CMake) but the second time that we need to read the symbols (in the "createSensor" method) then this static variable is no being initialized again.

I have create a simple demo to reproduce this here.

The right example to hace a look is ./main_dl_link_cmake. This is doing exactely the same thing.

The hello library contains a struct and a static variable inside an anonymous namespace. In the main this library is linked with cmake and then is being loaded using dlsym(). The text is only printed once and it should be printed twice

Signed-off-by: ahcorde <[email protected]>

ahcorde · 2020-08-21T11:21:10Z

I think I found a solution. If the symbols are already loaded, why we need to reload it again? There is no need to do it.

I added in the Manager class a method called AddSensor() 25be8d7 to add a sensor to the list of loaded sensors.

The I will use this class in ign-gazebo. Here it's the PR using this new method. gazebosim/gz-sim#304

Signed-off-by: ahcorde <[email protected]>

src/CameraSensor.cc

src/CMakeLists.txt

src/AltimeterSensor.cc

Signed-off-by: ahcorde <[email protected]>

ahcorde · 2020-09-14T11:39:10Z

waiting this PR #46 that fix the Github actions

chapulina

It looks like plugins are not being loaded correctly on CI

chapulina · 2020-09-16T00:01:24Z

src/AirPressureSensor.cc

@@ -19,6 +19,8 @@
 #include <ignition/common/Profiler.hh>
 #include <ignition/transport/Node.hh>

+#include <ignition/plugin/Register.hh>


Is this include needed at every sensor? Shouldn't SensorFactory.hh be enough?

According with the migration guide https://github.com/ignitionrobotics/ign-plugin/blob/94482af3e9b2cc1cbb7c2dbeb2c7846f8283f0f6/MIGRATION.md#registering-a-plugin

I added this header for registering plugins. All sensors are individual components that why I need to use ignition/plugin/Register.hh in the cc file. I tried to include this header (or #include <ignition/plugin/RegisterMore.hh>) in the SensorFactory.hh but I was not able to compile the code because an undefined reference:

``/usr/bin/ld: ../lib/libignition-sensors4-imu.so.4.0.0~pre1: undefined reference `IgnitionPluginHook'

According with the migration guide

I see. I mean, we'd still be including that header, just indirectly. The way ign-sensors is exposing the plugin functionality is through SensorFactory. So technically each sensor doesn't need to know about ign-plugins, they just know they need to use the IGN_SENSORS_REGISTER_SENSOR macro from the SensorFactory.hh header.

I tried to include this header (or #include <ignition/plugin/RegisterMore.hh>) in the SensorFactory.hh but I was not able to compile the code because an undefined reference:

SensorFactory.hh uses the IGNITION_ADD_PLUGIN macro from ignition/plugin/Register.hh, so I believe it should include that header. Otherwise, that header is not usable on its own.

And it probably needs to link against ignition-plugin${IGN_PLUGIN_VER}::register to fix the undefined reference. I believe that if you link to it publicly, then each individual sensor won't need to also link against register, only against ign-sensors's core library?

@chapulina, I was trying your suggestion but I'm not able to compile the code. I was having a look to other packages that make use of ignition-plugin and the approach it's similar to the one that I used here. For example:

Ign-gui plugins include the register header in the .cc files. Not in the core library. Link1 Link2

ign-gazebo. the same idea. Link3 Link4

ign-rendering: Link5 Link6 Link7

it's similar to the one that I used here

I think the key difference here is that the IGNITION_ADD_PLUGIN macro is used in SensorFactory, not in each plugin. All the examples you gave use that macro directly on the plugin files.

Does SensorFactory.hh work if you don't include Register.hh above it? If it doesn't, that sounds like an architecture flaw. A header should work when included by itself, without the need for extra includes above it, right?

All plugin are using the MACRO but it's hidden under this other one IGN_SENSORS_REGISTER_SENSOR

@chapulina I can remove IGN_SENSORS_REGISTER_SENSOR and make use of `IGNITION_ADD_PLUGIN.

I can remove IGN_SENSORS_REGISTER_SENSOR and make use of `IGNITION_ADD_PLUGIN.

That sounds like a big migration effort for users. But since our support for custom sensors is not really there yet (#9), maybe this isn't a big deal. I'd be ok with it. What do you think, @iche033 ?

src/SensorFactory.cc

ahcorde · 2020-09-16T08:29:20Z

I noticed that the CI is not able to run the tests. But I'm not able to reproduce it in my machine. I will try on a docker container.

chapulina

I noticed that the CI is not able to run the tests. But I'm not able to reproduce it in my machine.

I'm seeing the same failures locally, i.e.

$ ./build/ignition-sensors4/bin/INTEGRATION_imu_plugin
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from ImuSensorTest
[ RUN      ] ImuSensorTest.CreateImu
[Err] [SensorFactory.cc:81] Unable to load sensor plugin file for [/home/chapulina/dome_ws/install/lib/libignition-sensors4-imu.so]
[Err] [SensorFactory.cc:166] Unable to instantiate sensor plugin for [ignition-sensors4-imu]
[Err] [SensorFactory.hh:133] Failed to create sensor of type[imu]
/home/chapulina/dome_ws/src/ign-sensors/test/integration/imu_plugin.cc:85: Failure
Value of: sensor != nullptr
  Actual: false
Expected: true
Segmentation fault (core dumped)

It looks like the shared library is generated and installed, but it doesn't have the correct symbols inside it.

src/SensorFactory.cc

Signed-off-by: ahcorde <[email protected]>

chapulina

A couple more comments.

And great work cleaning up the Windows warnings! Those could even be pushed to a separate PR so they can get merged faster.

chapulina · 2020-09-21T16:35:05Z

include/ignition/sensors/LogicalCameraSensor.hh

@@ -23,12 +23,20 @@

 #include <sdf/sdf.hh>

-#include <ignition/common/PluginMacros.hh>
+#include <ignition/plugin/RegisterMore.hh>


No other plugins have RegisterMore on their headers, can it be removed here?

chapulina · 2020-09-21T16:42:05Z

src/ImuSensor_TEST.cc

@@ -191,19 +195,14 @@ TEST(ImuSensor_TEST, CreateImuSensor)
  sdf::ElementPtr imuSDF = ImuSensorToSDF(name, update_rate, topic,
    accelNoise, gyroNoise, always_on, visualize);

-  // Create an ImuSensor
-  auto sensor = mgr.CreateSensor<ignition::sensors::ImuSensor>(imuSDF);


We shouldn't remove the CreateSensor tests, that function is still supported, right?

Instead, we should add separate tests for the Load functionality that you're using below.

chapulina · 2020-09-21T17:27:02Z

Just talked to @ahcorde , we'll save this for Ign-E so we have more time to try and test the new approach. Removing it from the Dome beta.

chapulina · 2021-02-04T21:37:04Z

This PR is outdated, it would be nice to sync it with the current main branch

Signed-off-by: Louise Poubel <[email protected]>

chapulina · 2021-03-23T16:35:55Z

Removing beta label, we won't have time to wrap this up before code freeze. Let's retarget at Ignition-F.

chapulina · 2021-08-12T16:21:51Z

The plugin interface was completely removed on #90.

Replace ign common plugin loader with ign-plugin

a027783

Signed-off-by: ahcorde <[email protected]>

ahcorde added enhancement New feature or request 🔮 dome Ignition Dome labels Aug 19, 2020

ahcorde requested a review from iche033 August 19, 2020 12:18

ahcorde self-assigned this Aug 19, 2020

ahcorde added 2 commits August 21, 2020 10:53

Updated includes

50feecc

Signed-off-by: ahcorde <[email protected]>

Added AddSensor method to Manager

25be8d7

Signed-off-by: ahcorde <[email protected]>

ahcorde mentioned this pull request Aug 21, 2020

Use the symbols if they are already loaded gazebosim/gz-sim#304

Closed

ahcorde mentioned this pull request Aug 24, 2020

Use ign-plugin #18

Closed

Merge branch 'master' into ahcorde/replace/pluginload

21badd7

scpeters changed the title ~~[WIP] Replace ign common plugin loader with ign-plugin~~ Replace ign common plugin loader with ign-plugin Aug 24, 2020

ahcorde added 2 commits August 27, 2020 12:37

make linters happy

b78594d

Signed-off-by: ahcorde <[email protected]>

make linters happy

c5d0cdb

Signed-off-by: ahcorde <[email protected]>

iche033 reviewed Sep 2, 2020

View reviewed changes

src/CameraSensor.cc Outdated Show resolved Hide resolved

src/CMakeLists.txt Show resolved Hide resolved

src/AltimeterSensor.cc Outdated Show resolved Hide resolved

ahcorde added 2 commits September 3, 2020 14:34

Removed unnecessary code

f3dfca3

Signed-off-by: ahcorde <[email protected]>

Fixed headers

a0a0e65

Signed-off-by: ahcorde <[email protected]>

chapulina added the beta Targeting beta release of upcoming collection label Sep 8, 2020

Merge branch 'master' into ahcorde/replace/pluginload

dd97599

chapulina reviewed Sep 16, 2020

View reviewed changes

src/SensorFactory.cc Show resolved Hide resolved

chapulina mentioned this pull request Sep 17, 2020

Source the libraries before executing the tests gazebo-tooling/action-gz-ci#24

Closed

Fixed windows build error

5b4b7a9

Signed-off-by: ahcorde <[email protected]>

ahcorde added 2 commits September 21, 2020 10:46

Improved WaitForMessageTestHelper - test

69ba811

Signed-off-by: ahcorde <[email protected]>

make linters happy

c15ad7c

Signed-off-by: ahcorde <[email protected]>

chapulina reviewed Sep 21, 2020

View reviewed changes

chapulina removed beta Targeting beta release of upcoming collection 🔮 dome Ignition Dome labels Sep 21, 2020

chapulina mentioned this pull request Sep 22, 2020

ignition::sensors::SensorFactory.CreateSensor(...) returns null in several tests #49

Closed

chapulina changed the base branch from master to main October 3, 2020 01:27

ahcorde mentioned this pull request Nov 2, 2020

Load global symbols gazebosim/gz-plugin#30

Closed

chapulina mentioned this pull request Nov 3, 2020

👩‍🌾 Prevent segfaults on test failures, make tests verbose #56

Merged

chapulina added the 🏢 edifice Ignition Edifice label Nov 3, 2020

chapulina mentioned this pull request Nov 4, 2020

👩‍🌾 Suggestions on addressing Windows warnings #59

Merged

Merge branch 'main' into ahcorde/replace/pluginload

7448fbe

nkoenig changed the base branch from main to ign-sensors5 December 23, 2020 23:35

chapulina changed the base branch from ign-sensors5 to main February 1, 2021 17:14

ahcorde added the needs upstream release Blocked by a release of an upstream library label Feb 3, 2021

chapulina added a commit that referenced this pull request Feb 5, 2021

Use ign-plugin (alternative to #38)

ccedda3

Signed-off-by: Louise Poubel <[email protected]>

chapulina mentioned this pull request Feb 5, 2021

Remove plugin interface and support custom sensors #90

Merged

chapulina added beta Targeting beta release of upcoming collection and removed beta Targeting beta release of upcoming collection labels Mar 17, 2021

chapulina added 🏯 fortress Ignition Fortress and removed 🏢 edifice Ignition Edifice labels Apr 1, 2021

chapulina closed this Aug 12, 2021

chapulina deleted the ahcorde/replace/pluginload branch August 12, 2021 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace ign common plugin loader with ign-plugin #38

Replace ign common plugin loader with ign-plugin #38

ahcorde commented Aug 19, 2020

mxgrey commented Aug 20, 2020

ahcorde commented Aug 21, 2020

ahcorde commented Aug 21, 2020

ahcorde commented Sep 14, 2020

chapulina left a comment

chapulina Sep 16, 2020

ahcorde Sep 16, 2020

chapulina Sep 16, 2020 •

edited

Loading

ahcorde Sep 17, 2020

chapulina Sep 17, 2020 •

edited

Loading

ahcorde Sep 17, 2020

ahcorde Sep 17, 2020

chapulina Sep 17, 2020

ahcorde commented Sep 16, 2020

chapulina left a comment

chapulina left a comment

chapulina Sep 21, 2020

chapulina Sep 21, 2020

chapulina commented Sep 21, 2020

chapulina commented Feb 4, 2021

chapulina commented Mar 23, 2021

chapulina commented Aug 12, 2021

Replace ign common plugin loader with ign-plugin #38

Replace ign common plugin loader with ign-plugin #38

Conversation

ahcorde commented Aug 19, 2020

mxgrey commented Aug 20, 2020

ahcorde commented Aug 21, 2020

ahcorde commented Aug 21, 2020

ahcorde commented Sep 14, 2020

chapulina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chapulina Sep 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chapulina Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahcorde commented Sep 16, 2020

chapulina left a comment

Choose a reason for hiding this comment

chapulina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chapulina commented Sep 21, 2020

chapulina commented Feb 4, 2021

chapulina commented Mar 23, 2021

chapulina commented Aug 12, 2021

chapulina Sep 16, 2020 •

edited

Loading

chapulina Sep 17, 2020 •

edited

Loading