Package mdio-tool

[bleykauf]

As suggested by @doronbehar in #277 (comment).

[doronbehar]

I was also thinking regarding this topic: Although mdio-tool could be useful for other people, perhaps it'll be best to embed it's functionality in linien-server? This way you'll have 1 dependency less to handle.

[bleykauf]

I would like to integrate mdio-tool in linien-server. One of the ideas of the refractor in #277 is to separate responsibilities better, to allow for example to port it to other platforms, see #275. mdio-tool is only needed for the RedPitaya.

[doronbehar]

In #298 I suggest to copy the code for mdio-tool as is into this repo, and put it in linien-server's .deb package as is. This solution seems like it'll take the least effort, and a debian package for linien-server is highly required anyway because we want a systemd service file to be distributed along with it and pip is not recommended for that (see this).

[doronbehar]

Creating a Debian package for linien-server is harder then I thought, and it will require too much maintenance in the future so I'm dropping this idea.. The reason is that linien-server has too many python dependencies that are not packaged yet in debian's repository.

I'm investigating whether it'll be possible to cross compile linien with Nix and bundle it as a debian package. This will also enable putting mdio-tool with even less effort, adding the systemd service easily, and even using Python3.10! Not to mention all other features of Nix...

However, cross compiling is not always working.. I think I'm close to cross compile scipy (issues listed upstream) but I'm waiting in the meantime for mesonbuild/meson-python#122 .

If that Nix based cross compilation will not work, I will revert to classic pip install linien-server on ssh on the server.

[doronbehar]

I'll note here, that I'm currently waiting for NixOS/nixpkgs#214549 regarding the implementation of the nix bundle idea.

With some patches to scipy and to it's nix-build expression, I managed to cross compile it but I haven't tested it at all on a real armv7h machine.

The progress in linien-org/pyrp3#6 surely helps!

[doronbehar]

Hello again @bleykauf . I managed to make a notable progress using Nix' and Nixpkgs' 1st class cross compilation support 🎉.

In my nixFlake branch I wrote a flake.nix and more Nix files that Nix reads and can cross-compile linien-server and all of it's dependencies for redpitaya. With Nix you can also create the .deb package that will include linien-server and it's dependencies (cross compiled), it's systemd service, and mdio-tool (cross compiled as well). You can build and transfer to your redpitaya this .deb package by checking out my fork, and with nix installed, run the following:

scp $(nix build --no-link --print-out-paths -L .\#linien-server-deb-armv7l-hf-multiplatform) [email protected]:linien-server.deb
ssh [email protected] dpkg --install linien-server.deb

Run dpkg --contents on this package to view it's contents:

dpkg --contents $(nix build --print-out-paths --no-link -L .\#linien-server-deb-armv7l-hf-multiplatform) | less

You can also build and run linien-gui with Nix using:

$ nix build -L .\#linien-gui
./result/bin/linien

Note that all of the above commands build a fork of scipy that include Nix flakes support and other cross compilation fixes (see my comment there), so the build may take time.

Anyway, I installed linien-server on my redpitaya as described above, and it seems to be fully functional. I will have to spend more time to actually lock a spectral line with it 😅 ..

[bleykauf]

Happy that your nix project is working now :)

As for using this in Linien, I am not convinced that this will make things easier. RedPitaya OS 2.0 beta with Python 3.10 is now available (still some issues #322 but can be resolved for sure). The installation process is completely automatic from the user perspective and is pretty simple after all: install screen via apt, install linien-server via pip3 and compile and copy mdio-tool. Only mdio-tool is not installed in the proper way, so I am still interested in getting the dep file for it but I am not familiar with nix...

Using this for in the main project would not change anything for the user (installation is automatic, just running different commands) but adds overhead regarding maintenance.

This could however be useful in regards to the GUI on Linux, since the current Github Action workflow using pyinstaller is not working, see #263.

[doronbehar]

The installation process is completely automatic from the user perspective and is pretty simple after all: install screen via apt, install linien-server via pip3 and compile and copy mdio-tool. Only mdio-tool is not installed in the proper way

I think that the screen dependence and usage are also improper - this was my original motivation (along with mdio-tool dependence) for trying to deploy this application with Nix.

I also think that installing such an application via pip3 is a recipe for deployment and updating issues. Not only because a systemd service is not included but mostly because linien is actively developed and bugs are discovered and fixed frequently, hence scientists must have a good track over the linien versions they install on their red pitayas.

As for mdio-tool, I just managed (in this commit) to compile with Nix mdio-tool statically - meaning that it should work as is on the redpitaya. This may help deployment if you have other ideas still relying on the pypi package (build it with nix build -L .\#mdio-tool-static).

This could however be useful in regards to the GUI on Linux, since the current Github Action workflow using pyinstaller is not working, see #263.

I'm afraid Nix is not useful for that - it's not sophisticated enough yet to generate a static single executables for python packages (including all dependencies). Haven't played with pyinstaller yet, but it's not even available on Nixpkgs, and it seems to have issues for Nix users.

Using this for in the main project would not change anything for the user (installation is automatic, just running different commands) but adds overhead regarding maintenance.

It's OK, we don't have to agree on that now. I'm using Nix for a long time and I can testify it's benefits overcome the maintenance. It takes time for people to appreciate the benefits, the keywords of the benefits are clean and expected deployments.

[doronbehar]

This can be closed now since version 2.0