MAVSDK verson on macOS aarch64

[cryptik]

I am working on a Mac M1 (aarch64) and was able to install mavsdk from homebrew, which installs mavsdk_server version 1.4.12. However, this does not install the Python3 bindings. When trying to install mavsdk-python from PyPi via pip3 install mavsdk, the only versions available are 0.5.2, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.9.1, 0.10.0, 0.11.0, 0.12.0. None of these versions include the mavsdk_server executable and all are really old. This does not seem to be the case if installing via pip on x86 Mac, as the latest versions are available.

I am sure it is possible to compile the latest version (1.4.12) of the python bindings, but I am not sure how to do that or how to install it into my python virtual environment (miniconda) once it is built.

[julianoes]

Yes, that's currently broken. The main pain point is that GitHub actions doesn't have macOS M1/2 runners, and so it's not easy to release the binary.

Besides that, something is wrong with how the architecture gets downloaded or is compatible and so the recent releases don't work.

Also see:
https://github.com/mavlink/MAVSDK-Python/blob/29f2502d03c540a21270651610ebbbe72cfcc8f5/setup.py#L55-L73

mavlink/MAVSDK-Python#319

Pull requests to fix it would be most welcome.

[cryptik]

@julianoes quick question... so I was able to install the aarch64 version of mavsdk_server v1.4.12 via home-brew... but this only includes the C++ interface library, not the python version. Can I build/install the mavsdk_python package from the source? I am not sure how this is done as I have never built a python package that could then be installed via pip. Do you know where I can find a guide on how to build the pip package?

Also, looking at the two posts you provided above... here...

        elif sys.platform.startswith('darwin'):
            return 'macos'

This only checks if the platform is macOS.. and not the architecture. This probably defaults to x86. The code for Linux checks the architecture. I can probably fix that and submit a pull request if I can figure out how to build/install the python package. Any pointers would be most helpful.

[cryptik]

@julianoes scratch my last. I was able to clone the repo and followed the instructions at the bottom of the README.md. When I first saw them, I did not realize that they build the python sdk. I was expecting something talking about building a wheel and what not. I have never built a python package from source.

Anyway, the instructions worked flawlessly and I was able to build MAVSDK version 1.4.12 for aarch64 on M1. Seems to work just fine. I can see mavsdk installed in pip as well as the mavsdk_server in the local bin folder.

[JonasVautherin]

That's great! Are you detecting the M1 around here?

Then the problem we have is that we don't have a CI for the M1 that could build mavsdk_server and release it. I wish that the github CI had that, but it doesn't at the moment...

[cryptik]

So for my current immediate need, I used mavsdk_server installed from homebrew, and then used the MAVSDK repo instructions to build the python bindings (as you know, it won't create the executable for MacOS aarch64). Both versions numbers match at this point.

The code block that you highlighted is one of the problems. It appears to detect that the platform is MacOS (i.e., Darwin) but does not check the architecture. I am also still determining how a python library is built (i.e., setup.py) for distribution, so I need to understand how that works. I use them a lot.. just never had to build one :)

[JonasVautherin]

So for my current immediate need, I used mavsdk_server installed from homebrew

Oh, so homebrew actually builds MAVSDK for the M1 and ships the binary? Is there a way to download that binary directly (say with curl)? Because then we could leverage that in the CI: MAVSDK-Python could ship with the homebrew binary for M1...

Maybe @ZhongRuoyu or @chenrui333 would have an insight about that? It would solve our problem. Right now we fetch mavsdk_server from our Github releases (that's done here in setup.py).

[cryptik]

They do have a formula for x86 MacOS as well as Apple Silicon for Ventura, Monterey, and Big Sur, but looking at the formula, it appears to actually build the executable on the user's workstation (using gcc or clang) rather than installing a pre-compiled binary. Does the python package distribution system allow one to do something similar (i.e., build the binary locally rather than download it)?

The recipe code is here

Until GitHub has the CI runners for Apple aarch64, it will still be an issue of where the actual build happens. I am surprised they do not support it yet. I did see a blog post that their CI system supports "self-hosted" M1 runners. Still, I don't see how that is different from the developer building on their workstation and copying the executable up to the release folder. Self-hosted, in this case, does not appear to be that helpful.

[JonasVautherin]

Oh actually @julianoes wrote the original recipe, I had forgotten that 😆.

Does the python package distribution system allow one to do something similar?

Probably, but at this point, maybe we could just have MAVSDK-Python rely on a system-wide mavsdk_server for the M1. For instance here, it could check for M1, and if it is M1, it could check that mavsdk_server is in the PATH. If not, it could throw an exception saying that it can be installed through homebrew.

Then around here, instead of running the embedded bin/mavsdk_server, for M1 it would again test if it is in the PATH and run that instead.

What do you think?

[cryptik]

Perfect! The homebrew install path differs for M1 (/opt/homebrew/bin vs. /usr/local/bin/brew), but these paths are added to the user's PATH when they install brew. So, for the M1 install script, it would be easy to search for mavsdk_server in the /opt/homebrew/bin folder. This is a great idea!

I use the mavsdk-python bindings inside a Conda environment, so having the server executable outside of the site_packages folder structure is a better setup for me... however, users will need to be mindful that the pip installed package may have a different version number than the brew installed executable. I don't think this is a big deal... but it could cause strange behavior if the versions differ significantly.

[JonasVautherin]

however, users will need to be mindful that the pip installed package may have a different version number than the brew installed executable.

Oh that's a very good point... it may cause strange behaviors. I guess to warn users we could add a way to call mavsdk_server --version, and MAVSDK-Python could check the version and warn if it doesn't match.

But maybe let's start by having it take the mavsdk_server from the PATH for the M1, that should already unblock quite some people...

[ZhongRuoyu]

Oh, so homebrew actually builds MAVSDK for the M1 and ships the binary?

Yes, we do. The pre-built binaries are called bottles, and they are hosted here on GitHub.

Is there a way to download that binary directly (say with curl)? Because then we could leverage that in the CI: MAVSDK-Python could ship with the homebrew binary for M1...

You may use brew fetch mavsdk to download that. To force a specific bottle, use e.g. brew fetch --bottle-tag=arm64_monterey. The bottle tags are the ones shown in the formula.

But please note that bottles are not necessarily ready for use; they need to be poured (processed) by Homebrew before that. ¹ Here's what you'll likely see when download the fetched bottle, extract it, and run the executable. So, from what I learned from the context of this issue, I'm afraid that this may not work.

$ ./bin/mavsdk_server
dyld[73411]: Library not loaded: @@HOMEBREW_PREFIX@@/opt/[email protected]/lib/libssl.1.1.dylib
  Referenced from: <8EF3AA53-B6AC-3F6F-B75F-7F0C22B73FC6> /Users/ruoyu.zhong/Downloads/mavsdk/1.4.12_1/lib/libmavsdk_server.1.4.12.dylib
  Reason: tried: '/System/Volumes/Preboot/Cryptexes/OS@@HOMEBREW_PREFIX@@/opt/[email protected]/lib/libssl.1.1.dylib' (no such file), '/usr/local/lib/libssl.1.1.dylib' (no such file), '/usr/lib/libssl.1.1.dylib' (no such file, not in dyld cache)Library not loaded: @@HOMEBREW_PREFIX@@/opt/tinyxml2/lib/libtinyxml2.9.dylib
  Referenced from: <5F93883E-9924-325F-8523-A9DCA0DA4D6C> /Users/ruoyu.zhong/Downloads/mavsdk/1.4.12_1/lib/libmavsdk.1.4.12.dylib
  Reason: tried: '/System/Volumes/Preboot/Cryptexes/OS@@HOMEBREW_PREFIX@@/opt/tinyxml2/lib/libtinyxml2.9.dylib' (no such file), '/usr/local/lib/libtinyxml2.9.dylib' (no such file), '/usr/lib/libtinyxml2.9.dylib' (no such file, not in dyld cache)
[1]    73411 abort      ./bin/mavsdk_server

Footnotes

1. Although this is not the case for mavsdk, some bottles may not be relocatable. That means they will only work when installed to the same location as when they were built. (But mavsdk's macOS bottles are relocatable, as they are tagged :any). ↩

[JonasVautherin]

@ZhongRuoyu: thanks a lot for the insights! So I guess the best way is to tell the user to install it with homebrew when the M1 is detected (instead of automatically trying to fetch the executable).

[julianoes]

Done in #2277.