port python_orocos_kdl to ros2 #4

mjbogusz · 2017-12-08T18:22:00Z

This PR enables PyKDL usage in ROS2.

Fixes in SIP are by @rkojcev and @vmayoral, so is basic adaptation of package.xml and CMakeLists.txt to ament, I've additionally added triggering PYTHONPATH hook (as ament does not support SIP packages yet) and improved installation of compiled objects.

mikaelarguedas · 2017-12-08T18:47:56Z

python_orocos_kdl/PyKDL/std_string.sip

@@ -39,7 +39,7 @@
    // If argument is a Python string, assume it's UTF-8
     if (sipIsErr == NULL)
 #if PY_MAJOR_VERSION < 3
-        return (PyString_Check(sipPy) || PyUnicode_Check(sipPy));
+        return (PyBytes_Check(sipPy) || PyUnicode_Check(sipPy));


It looks like the PyBytes_X functions are Python3 only (https://docs.python.org/2.7/c-api/string.html) so the PyString_X calls within the #if PY_MAJOR_VERSION < 3 should stay unchanged AFAICT

Seems right. As stated in the PR, changes within the SIPs are by @rkojcev, but if needed I can rebase and remove those changes.

mjbogusz · 2017-12-08T18:56:04Z

python_orocos_kdl/PyKDL/std_string.sip

-     if (PyString_Check(sipPy)) {
-        *sipCppPtr = new std::string(PyString_AS_STRING(sipPy));
+     if (PyBytes_Check(sipPy)) {
+        *sipCppPtr = new std::string(PyBytes_AS_STRING(sipPy));


As above, this is within a #if PY_MAJOR_VERSION < 3 block so probably it can be left untouched.

mjbogusz · 2017-12-12T16:14:54Z

I've removed the unnecessary changes and added one additional python2/python3 check so that it should work with both.
@mikaelarguedas fyi

mikaelarguedas · 2017-12-14T12:42:22Z

thanks @mjbogusz for iterating.

Do you have any reference code you use for testing the good functionment of this package ? (as it doesn't have tests)

mjbogusz · 2017-12-14T14:58:24Z

Currently I do not, I'm yet to create my modules utilizing PyKDL (as of now I'm only creating some frames and multiply them together - this much seems to work).

However if you'd point me towards a good example of how to set up tests in ament_cmake I can try to run the ones included in python_orocos_kdl/tests. Is ros2/system_tests, specifically test_communication, a good point of reference besides the overcomplicated CMakeLists?

mjbogusz · 2017-12-15T03:01:34Z

I've fixed the included tests (Class.None, opening a file in binary mode and some whitespace problems) and they all pass.

I'm working on including them in CMakeLists.txt.

Edit: I've just noticed that those tests use unittest while ament only supports pytest... This might take a while.

mjbogusz · 2017-12-15T03:31:53Z

Tests are in 👍

mikaelarguedas · 2017-12-15T14:49:15Z

Thanks @mjbogusz !

I've additionally added triggering PYTHONPATH hook (as ament does not support SIP packages yet) and improved installation of compiled objects.

Yeah ament_python has some macros that go in that direction but nothing that would work out of the box for sip packages AFAICT.

I've removed the unnecessary changes and added one additional python2/python3 check so that it should work with both.

👍

I don't have write access to this branch so I pushed a few fixups to https://github.com/ros2/orocos_kinematics_dynamics/tree/ros2-pykdl (mostly to get the tests to run without having to source the workspace). Would you mind cherry-picking them on this branch ?

I think with the fixups it's good to be merged 👍

Looks like just installing sip was not enough to get it to build on our farm:

Linux
Linux-aarch64
macOS
Windows

I'll take a closer look at it early next week

mjbogusz · 2017-12-16T14:40:10Z

Would you mind cherry-picking them on this branch ?

You mean, merging them into my PR branch?

mikaelarguedas · 2017-12-16T21:53:41Z

You mean, merging them into my PR branch?

Yes please 👍 , any solution bringing the changes from 20b998a into this PR branch is fine

mjbogusz · 2017-12-17T00:17:39Z

Done
I've also moved find_package(ament_cmake_pytest) inside the if(BUILD_TESTING) block so that it's not required without --build-tests - this way it also builds on R2B3 (I'm yet to port my project onto 1.0 - I have a feeling this should've been beta4...).

Edit: I've also added a status message in FindSIP.cmake - I have a suspicion that FindSIP.py is being run with the default python (as in: python2).
Edit2: I'm quite sure of it - the reported error is ImportError while Python3 would report ModuleNotFoundError. No idea why it's not respecting the PYTHON_EXECUTABLE variable though.

@vmayoral

…n ROS 2, fix Python3 compilation with @vmayoral

…ronment hook

…eLists.txt

mikaelarguedas · 2017-12-17T01:15:08Z

Edit: I've also added a status message in FindSIP.cmake - I have a suspicion that FindSIP.py is being run with the default python (as in: python2).
Edit2: I'm quite sure of it - the reported error is ImportError while Python3 would report ModuleNotFoundError. No idea why it's not respecting the PYTHON_EXECUTABLE variable though.

As we are using Ubuntu Xenial (python3.5) the error would be ImportError (ModuleNotFoundError is a 3.6 feature AFAICT). If you look at the OSX and Windows jobs that failed above (they use 3.6.3) they do raise ModuleNotFoundError as expected.

I looked a bit into it and I it looks like its using the right python interpreter its because of a mismatch between the sip modules installed when installing python3-sip from apt and sip from pip3

The one from apt (python3-sip(-dev)) provides a sipconfig module as well as a sip executable. The one from pip doesn't provide any of those only a sip python module. I started modifying the script to support both configurations but it's not working/tested yet: 465eb14

mjbogusz · 2017-12-17T01:51:42Z

I've run this file on some configurations. Seems just about right 👌 maybe except the doubled output when sipconfig is not available ;)

However while I'm not sure if that's an issue here, SIP's pip page says:

All wheels include the sip extension module but do not include the code generator.

Anyway, here are outputs from different configs:

Arch with python 3.6.3 and SIP installed via pacman (python-sip - python 3 is the default here):

sip_version:041306
sip_version_str:4.19.6
sip_bin:/usr/bin/sip
default_sip_dir:/usr/share/sip
sip_inc_dir:/usr/include/python3.6m

WSL (aka Bash on Windows - here acting as Xenial) with python 3.5.2 and SIP from pip3:

could not import 'sipconfig' module, try to import 'sip' instead
sip_version:041306
sip_version_str:4.19.6
sip_bin:/usr/bin/sip
default_sip_dir:/usr/share/sip
sip_inc_dir:/usr/include/python3.5m
sip_version:041306
sip_version_str:4.19.6
sip_bin:/usr/bin/sip
default_sip_dir:/usr/share/sip
sip_inc_dir:/usr/include/python3.5m

Same WSL with SIP from apt (python3-sip):

sip_version:041100
sip_version_str:4.17
sip_bin:/usr/bin/sip
default_sip_dir:/usr/share/sip
sip_inc_dir:/usr/include/python3.5m

Also it's quite interesting what files are inside the packages available for Xenial:

apt (https://packages.ubuntu.com/xenial/amd64/python3-sip/filelist ):

/usr/lib/python3/dist-packages/sip.cpython-35m-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/sipconfig.py
/usr/lib/python3/dist-packages/sipconfig_nd5.py
/usr/share/doc/python3-sip/changelog.Debian.gz
/usr/share/doc/python3-sip/copyright
/usr/share/python3/dist/python3-sip

pip3 (printed when uninstalling):

  /usr/local/lib/python3.5/dist-packages/sip-4.19.6.dist-info/INSTALLER
  /usr/local/lib/python3.5/dist-packages/sip-4.19.6.dist-info/METADATA
  /usr/local/lib/python3.5/dist-packages/sip-4.19.6.dist-info/RECORD
  /usr/local/lib/python3.5/dist-packages/sip-4.19.6.dist-info/WHEEL
  /usr/local/lib/python3.5/dist-packages/sip.so

mjbogusz · 2017-12-17T02:03:46Z

Also while scrolling through jenkins logs I've found out that it installs SIP from both apt (while building dockerfile, at 14:34:05 of this specific run) and from pip3 (while installing python packages within the container in venv, 14:40:48, 14:40:54 and 14:40:55)

mikaelarguedas · 2017-12-17T02:30:00Z

maybe except the doubled output when sipconfig is not available ;)

Haha yeah its just me printing my debug output to stderr. That's not going to stay :)

Also while scrolling through jenkins logs I've found out that it installs SIP from both apt (while building dockerfile, at 14:34:05 of this specific run) and from pip3 (while installing python packages within the container in venv, 14:40:48, 14:40:54 and 14:40:55)

Yes you're right, sorry I should have pointed it out earlier, we currently do the same venv setup for all platforms (including linux regardless of what has been installed via apt in the Dockerfile), that's definitely something we could iterate on the later. The main thing I wanted to confirm is "is installing sip from pip is enough to build this code?".

As you saw in the release notes and as I confirmed locally: the answer is no (we need the sip headers, libraries and generator for this package to build). I confirmed that building sip from source and installing it provides all the necessary files so tweaking the FindSIP.py script will become irrelevant (as the from source install does provide the sipconfig python module).

So what we need is a way to get the full sip on all platforms:

on MacOS: sip is available from homebrew so it "should be easy": (brew install sip)
on Windows: I didnt see a chocolatey package for it: we need to either:
- package it ourselves: We did this for some dependencies: https://github.com/ros2/choco-packages/tree/latest/package
- create a vendor package (package that tries to find SIP on the system and if it doesn't find it, it downloads and builds it from source). We do this for a couple other dependencies (examples: tinyxml2_vendor or poco_vendor)

mjbogusz · 2017-12-17T03:25:58Z

I'd say that a chocolatey package is better - both vendor packages you've linked are working OK, but some are not as good, e.g. rviz_yaml_cpp_vendor (ros2/rviz#152 and related PRs).

Also I think it'll save some time when RQT tools will start getting ported to ROS2 - I may be mistaken but I think PyQT required SIP.

mikaelarguedas · 2017-12-17T03:38:22Z

I may be mistaken but I think PyQT required SIP.

You're totally right, I actually think that SIP was developed by the PyQt effort.

I'd say that a chocolatey package is better - both vendor packages you've linked are working OK, but some are not as good, e.g. rviz_yaml_cpp_vendor (ros2/rviz#152 and related PRs).

Yeah in the case of the ones I linked, they provide a failsafe as if users don't have the dep on their system it will download and install it for them. So they work "in addition to" the chocolatey packages.
So maybe I shouldn't have presented them as mutually exclusive. Either option would get us something that will allow to build python_orocos_kdl. Having both would provide a better user experience in general.

I'll test the MacOS test case on Monday.

If it works out of the box:

let's focus on the chocolatey package

If it doesnt work out of the box and seems painful:

let's focus on the vendor package (as it will solve both the MacOS and Windows use case).

How does that sound ?

mjbogusz · 2017-12-18T00:35:47Z

Sounds A-OK.

I've tried looking into creating a choco package, but I've spent way too much time setting up the whole required VC++ stack.

Also I have no idea how to install python modules using chocolatey - assuming e.g. "C:\Python36\Lib\site-packages" is not the best option. And last but not least we have to install sip.h in a way it's accessible for the code generated by SIP via #include <sip.h>...

mikaelarguedas · 2017-12-22T17:18:24Z

So it looks like python modules installed via brew are not findable in a python virtualenv, they recommend using pip instead. As the sip version doesnt provide us the files we need we'll have to do a vendor package for now.

Also I have no idea how to install python modules using chocolatey - assuming e.g. "C:\Python36\Lib\site-packages" is not the best option. And last but not least we have to install sip.h in a way it's accessible for the code generated by SIP via #include <sip.h>...

That;s a good question, we haven't packages any mixed python/C++ packages with choco yet. Maybe @nuclearsandwich will have some insights on the preferred install locations

mjbogusz · 2017-12-22T17:53:05Z

python modules installed via brew are not findable in a python virtualenv

It should work if one was to believe the docs... And as per https://stackoverflow.com/a/26322102/3217805 (the 'edit' part) it might be required to manually set PYTHONPATH:

editing PYTHONPATH in ~/.bash_profile forces virtual environments to inherit global packages

However I have no experience with brew (or development on osx) whatsoever so that's just a

About choco: I've found out that PYTHONPATH env variable should work under Windows more or less as expected and choco can set it for us (caveat: we might have to add python's main site-packages to that variable too).

Also, if I understand it well enough, VsDevCmd.bat (VS's native tools command prompt) sets the include directories for VC++ so we should be able to add our path to those too:

@if not "%WindowsSdkDir%" == "" @set PATH=%WindowsSdkDir%bin\x86;%PATH%
@if not "%WindowsSdkDir%" == "" @set INCLUDE=%WindowsSdkDir%include\%WindowsSDKVersion%shared;%WindowsSdkDir%include\%WindowsSDKVersion%um;%WindowsSdkDir%include\%WindowsSDKVersion%winrt;%INCLUDE%
@if not "%WindowsSdkDir%" == "" @set LIB=%WindowsSdkDir%lib\%WindowsSDKLibVersion%um\x86;%LIB%
@if not "%WindowsSdkDir%" == "" @set LIBPATH=%WindowsLibPath%;%ExtensionSDKDir%\Microsoft.VCLibs\14.0\References\CommonConfiguration\neutral;%LIBPATH%
:: and then
@if exist "%VCINSTALLDIR%ATLMFC\INCLUDE" set INCLUDE=%VCINSTALLDIR%ATLMFC\INCLUDE;%INCLUDE%
@if exist "%VCINSTALLDIR%INCLUDE" set INCLUDE=%VCINSTALLDIR%INCLUDE;%INCLUDE%

mjbogusz · 2018-01-22T21:20:40Z

A colleague of mine has tried using this package in Ubuntu 17.10 and 16.04 and he was consistently encountering a segmentation fault when requiring PyKDL (stacktrace pointed at missing strcmp_avx2 or something like that).
Seems like there's something wrong either with the functions called in ifdefs or with those ifdefs themselves.
I'll try to reproduce that behaviour and possibly fix it, but for now I'm considering this PR broken.

As a sidenote, any news on the vendor package?

allenh1 · 2019-12-06T21:09:20Z

@clalancette is anyone working on this at the moment?

clalancette · 2019-12-06T21:24:30Z

@clalancette is anyone working on this at the moment?

Not that I know of; @mjbogusz may still be interested. I think there is the open question about SIP, but it is worth looking at again.

ivanpauno · 2020-02-13T18:24:13Z

Closing due to long time inactivity._

Flova · 2021-12-09T18:39:47Z

Any updates on this?
It breaks the tf2_geometry_msgs package which is quite essential for many python nodes that use tf2.
Are there any obvious workarounds for this, that I have missed?

Flova · 2021-12-09T18:40:56Z

See ros2/geometry2#285

mikaelarguedas reviewed Dec 8, 2017

View reviewed changes

mjbogusz commented Dec 8, 2017

View reviewed changes

mikaelarguedas added the in review Waiting for review (Kanban column) label Dec 11, 2017

mjbogusz force-pushed the ros2-pykdl branch from 2175675 to 4090f83 Compare December 12, 2017 16:13

mjbogusz force-pushed the ros2-pykdl branch 2 times, most recently from f0e3f64 to 7ef822c Compare December 12, 2017 20:15

mikaelarguedas mentioned this pull request Dec 15, 2017

add python sip ros2/ci#119

Closed

mikaelarguedas self-assigned this Dec 15, 2017

mjbogusz force-pushed the ros2-pykdl branch from c49b41e to 98bcb3f Compare December 17, 2017 00:14

rkojcev and others added 5 commits December 17, 2017 01:27

python_orocos_kdl: modify package.xml and CMakeLists.txt to compile i…

5ab2924

…n ROS 2, fix Python3 compilation with @vmayoral

python_orocos_kdl: remove AMENT_IGNORE and add manual PYTHONPATH envi…

463c5e3

…ronment hook

python_orocos_kdl: fix tests for python3 and pytest, add them to CMak…

d0684bc

…eLists.txt

add PyKDL from build folder to PYTHONPATH to run the tests

a632dbd

remove debug print

7ce8c58

mjbogusz force-pushed the ros2-pykdl branch from 98bcb3f to 7ce8c58 Compare December 17, 2017 00:28

clalancette added in progress Actively being worked on (Kanban column) and removed in review Waiting for review (Kanban column) labels Jan 18, 2018

mikaelarguedas added ready Work is about to start (Kanban column) and removed in progress Actively being worked on (Kanban column) labels Mar 22, 2018

clalancette unassigned mikaelarguedas Oct 18, 2018

This was referenced Dec 17, 2019

installed python tf2_kdl and ported tf2_kdl tests ros2/geometry2#206

Merged

installed python tf2_geometry_msgs ros2/geometry2#207

Merged

ivanpauno added backlog and removed ready Work is about to start (Kanban column) labels Feb 13, 2020

ivanpauno closed this Feb 13, 2020

sloretz mentioned this pull request Aug 6, 2020

PyKDL ModuleNotFoundError: No module named 'PyKDL' by importing import tf2_geometry_msgs ros2/geometry2#285

Closed

Flova mentioned this pull request Mar 4, 2022

Add core parts of the ipm library ros-sports/ipm#1

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

port python_orocos_kdl to ros2 #4

port python_orocos_kdl to ros2 #4

mjbogusz commented Dec 8, 2017

mikaelarguedas Dec 8, 2017

mjbogusz Dec 8, 2017

mjbogusz Dec 8, 2017

mjbogusz commented Dec 12, 2017

mikaelarguedas commented Dec 14, 2017

mjbogusz commented Dec 14, 2017

mjbogusz commented Dec 15, 2017 •

edited

Loading

mjbogusz commented Dec 15, 2017

mikaelarguedas commented Dec 15, 2017

mjbogusz commented Dec 16, 2017 •

edited

Loading

mikaelarguedas commented Dec 16, 2017

mjbogusz commented Dec 17, 2017 •

edited

Loading

mikaelarguedas commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mikaelarguedas commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mikaelarguedas commented Dec 17, 2017 •

edited

Loading

mjbogusz commented Dec 18, 2017

mikaelarguedas commented Dec 22, 2017

mjbogusz commented Dec 22, 2017

mjbogusz commented Jan 22, 2018

allenh1 commented Dec 6, 2019

clalancette commented Dec 6, 2019

ivanpauno commented Feb 13, 2020

Flova commented Dec 9, 2021

Flova commented Dec 9, 2021

port python_orocos_kdl to ros2 #4

port python_orocos_kdl to ros2 #4

Conversation

mjbogusz commented Dec 8, 2017

mikaelarguedas Dec 8, 2017

Choose a reason for hiding this comment

mjbogusz Dec 8, 2017

Choose a reason for hiding this comment

mjbogusz Dec 8, 2017

Choose a reason for hiding this comment

mjbogusz commented Dec 12, 2017

mikaelarguedas commented Dec 14, 2017

mjbogusz commented Dec 14, 2017

mjbogusz commented Dec 15, 2017 • edited Loading

mjbogusz commented Dec 15, 2017

mikaelarguedas commented Dec 15, 2017

mjbogusz commented Dec 16, 2017 • edited Loading

mikaelarguedas commented Dec 16, 2017

mjbogusz commented Dec 17, 2017 • edited Loading

mikaelarguedas commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mikaelarguedas commented Dec 17, 2017

mjbogusz commented Dec 17, 2017

mikaelarguedas commented Dec 17, 2017 • edited Loading

mjbogusz commented Dec 18, 2017

mikaelarguedas commented Dec 22, 2017

mjbogusz commented Dec 22, 2017

mjbogusz commented Jan 22, 2018

allenh1 commented Dec 6, 2019

clalancette commented Dec 6, 2019

ivanpauno commented Feb 13, 2020

Flova commented Dec 9, 2021

Flova commented Dec 9, 2021

mjbogusz commented Dec 15, 2017 •

edited

Loading

mjbogusz commented Dec 16, 2017 •

edited

Loading

mjbogusz commented Dec 17, 2017 •

edited

Loading

mikaelarguedas commented Dec 17, 2017 •

edited

Loading