Add AppImage support to Linux builds #3688
Conversation
|
Where can the resulting AppImage built by Travis CI be downloaded? |
So far, we haven't integrated this into Travis-CI yet (and we only upload our stable releases currently). I'd probably defer that task to @lukas-w once we have the portable installation working properly. The primary purpose of this PR is to make the AppImage available to developers to create and test to get the kinks with the software worked out. |
|
Hi @tresf how would developers test this if it cannot be downloaded? In my AppImage generation scripts, I use http://transfer.sh to get "out" the build products from Travis CI by uploading them to a temporary location where they will be available for 14 days. |
Those interested (which tend to be very few) would compile the code, like any other feature. The temporary upload is a nice feature. I generally do something similar but manually when I need testers and which doesn't add additional logic to Travis-CI that would later have to be pulled out just prior to merge. We should consider uploading all PRs to a temporary location as you've suggested. It'll be a welcome addition as we ramp up for nightly builds. |
cmake/linux/package_linux.sh.in
Outdated
| chmod +x "$LINUXDEPLOYQT" | ||
| echo " [success]" | ||
|
|
||
| # Make skeleton AppDir |
There was a problem hiding this comment.
Why do you think this is needed? When you install your build products with make INSTALL_ROOT="$APPDIR" install then they will end up there.
There was a problem hiding this comment.
This is post-install step, so it's irrelevant.
There was a problem hiding this comment.
I don't understand this comment, why are you doing it when it is irrelevant?
There was a problem hiding this comment.
Your comment is irrelevant. You're telling a project how to specify the install target, which is too late when it's a post-install step. We can script the entire thing like we do for Windows, but then it takes away the granularity of developers specifying their own build environment.
We copy the install instead of messing it all up. Ideally, the /usr mandate would be dropped and then we can use whatever folder we want. That's how macdeployqt works for us.
It's really not your place to tell a project, or a user for that matter where to drop an install target. This may make sense for a small project. But (I feel) for large, complex projects, it's presumptuous.
There was a problem hiding this comment.
So where would you ideally like to install? The reasonis that (at least for some software), the path specified with DCMAKE_INSTALL_PREFIX ends up in the compiled binaries, which is what I generally want to avoid, hence recommending /usr as the generic prefix. This is not a hard requirement though and you are free to use another prefix.
There was a problem hiding this comment.
where would you ideally like to install
Wherever the user says to, would be the ideal location. For our tutorials, this is usually /home/user/lmms/target, but I often run a second copy from /home/user/Desktop/lmms/target and others may prefer their own place.
DCMAKE_INSTALL_PREFIX ends up in the compiled binaries, which is what I generally want to avoid
This is the part that confuses me... I would expect a relinking tool to be agnostic of the prefix, but to be fair, I've also never written one. :)
cmake/linux/package_linux.sh.in
Outdated
| mkdir -p "$APPDIR" | ||
| echo " [success]" | ||
|
|
||
| # Clone install to AppDir |
There was a problem hiding this comment.
Why do you think this is needed? When you install your build products with make INSTALL_ROOT="$APPDIR" install then they will end up there.
There was a problem hiding this comment.
This is a post-install step, so it's irrelevant.
There was a problem hiding this comment.
I don't understand this comment, why are you doing it when it is irrelevant?
cmake/linux/package_linux.sh.in
Outdated
| cp -R "$INSTALL/usr" "$APPDIR" | ||
| echo " [success]" | ||
|
|
||
| # Prepare the launcher |
There was a problem hiding this comment.
Why do you think this is needed? linuxdeployqt will copy the desktop file and icon when you invoke it with linuxdeployqt $APPDIR/usr/share/applications/*.desktop
There was a problem hiding this comment.
Per portablelinuxapps.com, I thought that was the way to do it.
Create an AppDir structure that looks (as a minimum) like this:
MyApp.AppDir/
MyApp.AppDir/AppRun
MyApp.AppDir/myapp.desktop
MyApp.AppDir/myapp.png
MyApp.AppDir/usr/bin/myapp
MyApp.AppDir/usr/lib/libfoo.so.0
There was a problem hiding this comment.
That documentation describes how to do things by hand, it is from 2010 (7 years old). Nowadays we have tools like linuxdeployqt and appimagetool which can do the work.
There was a problem hiding this comment.
Nowadays we have tools like
linuxdeployqtandappimagetoolwhich can do the work.
I started researching the layout when I had initial problems and there was no argument against it in #3558 so I thought I was on the right track. But we'll switch to using the recommended approach instead, thanks. 👍
cmake/linux/package_linux.sh.in
Outdated
| mv "$APPDIR/usr/share/pixmaps/@[email protected]" "$APPDIR" | ||
| echo " [success]" | ||
|
|
||
| echo -e "${GREEN}Cleaning up...${PLAIN}" |
There was a problem hiding this comment.
Don't do this. $APPDIR/usr/share/applications/ and $APPDIR/usr/share/icons/ shall still be populated because that is what the optional appimaged daemon will use to integrate your application into the desktop (menus, icons, etc.)
cmake/linux/package_linux.sh.in
Outdated
| # Workaround per https://github.com/probonopd/linuxdeployqt/issues/52 | ||
| # export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/pulseaudio/:$LD_LIBRARY_PATH | ||
| # ^--- Shouldn't be needed anymore via https://github.com/probonopd/linuxdeployqt/commit/9a93e030cff2cb10e9e672ebb3360d30b3099c6d | ||
| export LD_LIBRARY_PATH="$APPDIR/usr/lib/lmms/":$LD_LIBRARY_PATH |
There was a problem hiding this comment.
Indeed should not be needed anymore, did you try without this?
There was a problem hiding this comment.
Not yet. It's on the todo list.
There was a problem hiding this comment.
Tried it, still fails. Re-enabling.
ERROR: ldd outputLine: "libZynAddSubFxCore.so => not found"
ERROR: Please ensure that all libraries can be found by ldd. Aborting.
make[3]: *** [cmake/linux/CMakeFiles/appimage] Error 1
make[2]: *** [cmake/linux/CMakeFiles/appimage.dir/all] Error 2
make[1]: *** [cmake/linux/CMakeFiles/appimage.dir/rule] Error 2
make: *** [appimage] Error 2
cmake/linux/package_linux.sh.in
Outdated
|
|
||
| echo -e "${GREEN}Bundling dependencies...${PLAIN}" | ||
| # Bundle both qt and non-qt dependencies into appimage format | ||
| "$LINUXDEPLOYQT" "$APPDIR/usr/bin/@CMAKE_PROJECT_NAME@" -executable="$APPDIR/usr/lib/@CMAKE_PROJECT_NAME@/RemoteZynAddSubFx" -bundle-non-qt-libs |
There was a problem hiding this comment.
Please move it to $APPDIR/usr/bin/RemoteZynAddSubFx and retry. I think linuxdeployqt fails because it expects executables to be in usr/bin and not in a subdirectory of usr/lib.
There was a problem hiding this comment.
That's not where it lives, so it will be more work. We can try it.
There was a problem hiding this comment.
I'd like to clarify a bit on this... RemoteZynAddSubFx is not a binary usable by the system, so it's not something we'd ever install to $PATH. That's why it doesn't live there.
We have a similar problem with this executable on Mac and I haven't figured it out. Mac too makes mention that executables should be in a specific directory for macdeployqt to work properly, it's just not as straightforward as moving it. It may work for this PR, but it wouldn't make sense in general and when the remote process is launched from C++ it would need to know the new place to find it as well.
| @@ -0,0 +1,95 @@ | |||
| #!/usr/bin/env bash | |||
| # Creates Linux ".AppImage" for @PROJECT_NAME_UCASE@ | |||
There was a problem hiding this comment.
Why are you going through all the hassle of making this part of CMake when a few lines of .travis.yml or shell script would do? Just curious. Do you think it would be possible/advantageous to have a generic CMake AppImage module/plugin? If so, would you know where to start?
There was a problem hiding this comment.
Why are you going through all the hassle of making this part of CMake when a few lines of .travis.yml or shell script would do?
I hope I explained this (same?) question properly in #3688 (comment).
Do you think it would be possible/advantageous to have a generic CMake AppImage module/plugin? If so, would you know where to start?
Hesitant "yes" to "advantageous ... cmake module" question.
A "no" to knowing where to start. The first that comes to mind is CMake's NSIS integration, but it's really ugly to use.
CMake also offers CPackDMG for Mac, but it wasn't really useful for us. Our project requires macdeployqt combined with manual relinking. So from my perspective it's much easier to test, troubleshoot and patch in a common scripting language (which is bash for the time being)
cmake/linux/package_linux.sh.in
Outdated
| INSTALL="@CMAKE_BINARY_DIR@/../target" | ||
|
|
||
| # linuxdeployqt assumes installations contain /usr/ prefixes. | ||
| # 1. Make a directory containing /usr |
There was a problem hiding this comment.
When you install your build products with make INSTALL_ROOT="$APPDIR" install then they will end up there. The needed directories will be created for you.
There was a problem hiding this comment.
First, you make mention of INSTALL_ROOT but your wiki references DESTDIR. Is there a difference?
# quoted from wiki
CMake wants DESTDIR instead:
- cmake . -DCMAKE_INSTALL_PREFIX=/usr
- make -j$(nproc)
- make DESTDIR=appdir install ; find appdir/
Second, I did wonder if the parent directory would be created. We normally use cmake flags, as then we can use them in our scripts as well. If we can specify this through cmake instead, then we can configure it in and not have to guess. (Problem 2 above).
The package should be identical to this. I've simply mirrored the steps outlined in #3558 and chain them into a post-cmake-install step, which is (mostly) how we package for the other platforms. Windows has direct I'll see if I can add the build artifacts through Travis-CI. We normally don't override the |
|
Please describe exactly what you need with regard to |
Well, ideally we'd never want to pass install parameters directly to the Our current build process simply states to the developer that if they want to install to a non-standard location, they can provide it as This all may seem like overkill when something like Travis-CI can install the app to system space and then bundle it all up nicely, but Travis-CI is essentially a temporary sandbox, where our workstations aren't. As one of the primary package maintainers, I need the process to occur 100% in user space, even if Travis-CI won't necessarily benefit from it. Edit: Does that help? |
|
Since linuxdeployqt not only bundles libraries but also resources like icons, desktop files, mime types and so on, we need a FHS-like tree somewhere with directories like Can you provide such a tree in some (=whatever) writeable location to run linudeployqt on without much hassle? It does not have to be named |
By default, this is how we do it, but the tool didn't work when I called it against
I'm not sure what this means. |
|
Ok... backtracking a bit... I'm re-reading our Gitter conversation from May... I guess my question after re-reading all of this is, what is LHS (FHS?) and how do I tell it to stop assuming everything is in Conversation quoted below. @tresf wrote:
@probonopd wrote:
@tresf wrote:
@tresf wrote:
@tresf wrote:
@probonopd wrote:
|
|
|
|
Right, these are the points made in my very first conversation. I would not consider these "fine". :) There are still several unanswered questions. If they're not answered I'll assume the previous assumptions were and continue to be true. |
As discussed above, it is not a
Alternatively, copy your tree prior to executing
As per 2., you don't need to use it.
Did you try putting it into
Did you install the optional |
If this works, that would remove |
That's great news as well, thanks. |
|
@probonopd What about this question...
|
|
Updates...
|
|
You don't need to use an |
Not sure how to parse the payload executable from this line, and if we allow this, there might be countless other more complex ones, including If you urgently need the But if |
Where did you get your |
|
Added debug logs: https://gist.github.com/tresf/28bf36131c090a385e9fdf47e851e5c3
I compiled it like this: git clone https://github.com/probonopd/linuxdeployqt
cd linuxdeployqt
qmake
make
mv bin/linuxdeployqt ~/bin/
Is there an order of operations that avoids a hacked solution? In the If not, fine. Perhaps we're allowing the application to assume too much because I'm also not opposed to separating
This is an environmental variable picked up by Unity. It was added per #488. It's a desktop workaround and has no place inside the C++ codebase. |
Do you also have
Yes:
It's not just about
|
Thanks that got it working on i686. 👍
That makes perfect sense. Ok... I'll see what I can hack together. I'd prefer if we didn't have to hardcode this in two separate places so I'll have to think about it a bit more in regards to how to wrap it into the portable image. It's a minor inconvenience, so it's certainly not a show stopper. |
I've refreshed the link with some fixes to the jack implementation. We'll be providing these as official builds on our downloads page when 1.2.0 is official released. |
|
I don't know where is better to write, but can I somehow add Carla to the appimage build ? |
|
@vlad1777d probably a separate bug report would be the place. I don't know how portable Carla is and I also don't know what happens if we build with system (non-portable) support for it and it's missing. I recommend tagging For starters, you can try to use |
|
Thanks, @tresf , I created new bugreport and tagged you there. If I wrote something wrong or didn't tell something - correct my message. |
|
Hello @tresf . Here is it's ourput: |
|
To find out whether AppImage is at fault here, please extract the AppImage with |
|
@probonopd ,
Unpacked appimage ran:
LMMS 1.1.3 ran:
Appimage output: Unpacked appimage output: LMMS 1.1.3 output: Seems, they (packed and unpacked RC4) load the same time. Also after pressing button "Add effect" in "FX-Mixer", now left near 3.30 minutes, nothing loaded, seems, it's a freeze. I tried pressing this button several times. For comparison, LMMS 1.1.3 loaded in 3 seconds, after pressing "Add effect" to appearing window with list of effects was spend near 1 second. (I use standard jack installed with "aptitude install qjackctl") System: Linux Mint 18.2 x64 Cinnamon. |
|
So that AppImage is actually faster than the unpacked binaries? That should rule out the AppImage itself as the root cause. I cannot say what causes the contents of the AppImage to be that slow, but I doubt it's normal. |
|
@probonopd , updated answer. I think that it's inaccuracy. They load near the same time (packed and unpacked), but they load in times slower than LMMS 1.1.3. System: Linux Mint 18.2 x64 Cinnamon. LMMS 1.1.3 loads near 3-8 seconds, I think it's obvious that somewhere is something wrong =) |
|
@probonopd @vlad1777d running each of these cases once on some machine with unspecified specs which is also used for other stuff in the background doesn't produce meaningful results (even running 100 times won't in my opinion). |
|
Yes, my only point was that I don't think the slowdown is coming from AppImage. |
|
Calling strace or using a profiler might help. |
|
I can do this if you tell me certain steps, because I'm not familiar with those things. |
|
I figured out. It was so when I turned on jack manually before launching RC4. (in both cases in settings "Jack" is chosen as audio driver) But problem with long loading after pressing "Add effect" button in "FX-Mixer" is still present. |
|
No, that was not the matter. I figured out now a real matter. Was Jack runned manually or automatically by LMMS it doesn't matter. But anyway, LMMS 1.1.3 runs in near 3 seconds with present link too. I created that link not to select each time "Computer/media/username/Storm/instruments/sf2/drums/hpa_drumkit/file.sf2" when I'm trying to load instrument. And the problem in waiting near 4 minutes after pressing "Add effect" in "FX-Mixer" is still present. |
|
@vlad1777d call |
|
@TheAssassin , |
|
|
Strace: LMMS (appimage) normal launch: Strace: LMMS (appimage) launch with link to instrumens folder in /home/user/lmms/samples:
Added separate bugreport about pressing "Add effect" button: #3990 |
|
a thousand times: "THANK YOU, FINALLY I CAN JUST USE IT ON ANY DISTRO" |
Signed-off-by: Patrick José Pereira <[email protected]>
@probonopd just following up, more as an FYI... we found a way to set it in C++ so we'll be back to just Reference: |
Add AppImage support
Help us test this on your OS:
.AppImagehere:https://lmms.io/download/
This Pull Request will create an
.AppImageinstaller in the build directory. Closes #3558 .Developer Usage:
make install && make appimageTodo:
linuxdeployqt, crosslink to a wiki or something..AppImageDESTINATION; avoid wildcard (Make it possible to set destination filename probonopd/linuxdeployqt#146) Update: Not needed if usingappimagetool.RemoteZynAddSubFXcan't be linked to by lmms and when it is, it can't find Qt (d4fe98a, 8bea395)CMAKE_INSTALL_PREFIX(e.g./usr) from being used (Add AppImage support to Linux builds #3688 (comment)) (bf9d74bQT_AUTO_SCREEN_SCALE_FACTOR=1to fix screen scaling (Qt 5.6+) (8bc9e9d, not in rc3.89 build though)(Upstream, not critical) Install mimetype icon (partially via eb5e819,Invalid/upstream bug: https://github.com/AppImage/AppImageKit/issues/430)appimagedisn't working yet with itstk/rawwavesmissing from install, breaks Mallets instrument (0382b69)ZynAddSubFXinstrument shows blank bank. (possible duplicate of Zyn - Blank Instrument Bank #1001)Investigate adding Carla to the buildOut of scope.Done anyway in RC4.117. :)Travis-CI to use latest Qt5 version:Moved to Switch Travis-CI to Qt5.8 #3885.