QGIS MacOS Package Improvements

[PeterPetrik]

QGIS Enhancement: QGIS MacOS Package Improvements

Date 2020/05/18

Author Peter Petrik (@PeterPetrik)

Contact [email protected]

maintainer @PeterPetrik

Version QGIS 3.14

Summary

In 2019, we upgraded the QGIS macOS packaging method to address several issues at that time. This was made possible with the help of the QGIS project and donors who supported this work. The new method is more transparent and uses an automated packaging approach but also comes with its own limitations, mostly related to its strong dependency on Homebrew.

Last month we finished a prototype for building of all required QGIS dependencies without Homebrew. This now allows the continuous integration of MacOS builds in the QGIS CI infrastructure. The set of dependencies used could be downloaded and used to easily compile QGIS master for new MacOS developers as described in the INSTALL file and this blog post

Within this QEP, we propose finalizing the prototype and using it for regular QGIS bundles. This would solve many packaging problems and automatically upgrade GDAL, PROJ, GRASS and others to more up-to-date versions.

Existing MacOS Packager

Normally, Homebrew is used to fetch QGIS dependencies and to compile only QGIS for the LTR, PR and nightly releases. The main problems with this approach are:

Homebrew can only support one version of a given package. This stops QGIS from being built against multiple versions of, for example the GDAL or Proj libraries. On one hand, we want QGIS nightlies to use more bleeding edge versions of the dependencies (e.g. for plugin or core developers) . On the other hand, we want to use more stable and tested versions of dependencies for the PR or LTR builds (e.g. for users and organisations). OSGeo4W manages to succeed in offering these different versions of QGIS in Windows but unfortunately, Homebrew does not offer such an option.

For packaging the QGIS bundle, we relied on Python scripting to automatically determine the linked libraries. Also we used absolute paths for links between dynamic libraries, which resulted in the bundle that worked only with specific default install location. The automatic system was difficult to maintain, due to various architecture choices made at the time, sometimes connected with how Homebrew works.

Proposed Solution

We propose using the new system of dependencies for QGIS builds. This would guarantee full control of versions and patches used for MacOS builds, similar to how OSGeo4W handles things. We also propose simplifying the logic in the bundler scripts, to rely more on a hardcoded list of requirements (e.g. add libproj to exactly this location) instead of having an automatic script that needs to decide what to bundle and how to position it in the final path. We would also use rpath instead of hardcoded absolute paths for the linker. This would allow relocation of the final bundle on the user's disk.

To take a closer look at the proposal, you can find the existing proof of the solution in the qgis/QGIS-Mac-Packager repository. The packager has these steps:

1. Build all dependencies in qgis_deps. This follows the logic used by OSGeo4A, where each dependency has a simple bash build script (e.g. qgis_deps/recipes/gdal/recipe.sh for GDAL). This is run only when some library has changed.
2. Build QGIS in qgis_build
3. Bundle it all together qgis_bundle. Here again each dependency has its own bash script to copy the files to the correct final destination and fix rpaths and other bits.
4. Sign the bundle and create the installer
5. Upload the final package to the QGIS download area

Issue Tracking ID(s)

• Update to GDAL3, PROJ6, GRASS 7.8.2 and other major updates link
• Fix the GRASS and SAGA providers: link link link and others
• Fix the bundle relocation. User would be able to move the installed QGIS bundle to an alternative location (e.g. ~/Applications instead of root /Application folder) link
• Address the notarisation process, so the users on MacOS 10.15 would not need to go around the default security settings of the OS link
• Fix issues related to _iconv library symbols link
• Add QtDesigner executable into bundle, so users would be able to design QGIS forms link
• Inclusion of pip in the bundled python environment so users could easily install additional Python packages into the bundled Python environment link

[timlinux]

Thank you so much... We really need this to put maços on the same footing as windows and Linux for both users and developers.

[kyngchaos]

A reminder that the QGIS source has a bundling function. It takes care of library path links so they are all relative to the QGIS app (and removes rpaths so it won't crash if Qt is installed, ie when running on the same dev Mac). I had intended it to be able to make a fully all-in-one app, but my interests never got beyond leaving GDAL and python external. It can still be expanded to the full bundling, and I think it should be - bundling external stuff is an installation operation, no need to duplicate this functionality externally.

Sure, the bundling is all in cmake, but if there is something that needs more flexibility cmake can call external scripts. Right now it handles explicit external items, the core/basics. It also needs to handle arbitrary externals, ie everything else, that can vary a lot depending on how that everything else is built. This might need a more flexible script than cmake.

[PeterPetrik]

When doing all-in-one bundle, we are talking roughly about 40+ libraries and equal number of python modules that needs to be packaged. Also there are 100+ text files (e.g. bash or other scripts) which sometimes needs to be patched (e.g. default build system uses absolute paths etc.) The bundling process highly depends on how the dependencies are linked (e.g. usage of rpath, absolute paths, etc). I agree that CMake system can be well used for bundling QGIS, Qt and maybe few libraries. But for bundling of the the whole stack I am really not convinced it is well maintainable and that is why I proposed this alternative solution based on simple bash scripts.

[phborba]

It is very rewarding to see your effort @PeterPetrik towards QGIS and the MacOS community!

Just one question: Will you address the external python package issues that QGIS all in one installer has in this WIP? I ask this because the python interpreter bundled with QGIS does not have pip, so we cannot install packages.

Once more, thank you very much for all your effort! Very nice job!

[PeterPetrik]

@phborba thanks a lot. Yes, pip will be part of the new stuff.

[phborba]

Very nice to hear that! Keep up the nice job!

[PeterPetrik]

Progress report: https://qgis.org/downloads/macos/nightly2/qgis_nightly2_master_20200814_023140.dmg

Fixed

• QGIS proj 5. Should probably use proj 6 & gdal3 QGIS-Mac-Packager#12
• Renaming macOS app causes authentication error QGIS#32163
• Symbol not found: _iconv QGIS-Mac-Packager#28
• Possibility to add extra python packages QGIS-Mac-Packager#33
• geoPDF GDAL driver

Build, Bundled, but still to test

• GRASS, GDAL and SAGA processing providers
• notarsation QGIS cannot be opened after updating to Mac OS Catalina (QGIS is not notarized) QGIS-Mac-Packager#31
• Qt Creator Bundle custom widget lib for Qt Designer  QGIS-Mac-Packager#3

Feel free to test and report the issues to the QGIS-Mac-Packager repo

[thayeray]

Thanks for this Peter. I know that you are working on it, but I thought I would let you know what happens on my machine with this build in case it helps you. On my system I am not able to get GRASS to work using the nightly2 master link you posted. I tried setting the GRASS7 folder in the processing options, but I think I am just setting it to a python wrapper without the underlying C binaries: /Applications/QGIS3.15.app/Contents/Resources/python/plugins/processing/algs/grass7 I also tried creating a new user to get the default path, but that is empty. You may have addressed this question elsewhere , but can you please tell me if the current build contains the GRASS binaries or just the QGIS parts that use GRASS, and GRASS needs to be installed separately, and then the location of GRASS set in the processing options? By the way, I am also still not able to get the GeoPDF option to light up when exporting a PDF from a layout. It says that the "GDAL PDF driver was not built with PDF read support. A build with PDF read support is required for GeoPDF creation.". On the plus side the About QGIS dialog lists GDAL/OGR 3.1.2 for both compiled and running against.  -Thayer 2020-08-14T10:13:50     CRITICAL    Problem with GRASS installation: GRASS was not found or is not correctly installed On Friday, August 14, 2020, 04:46:58 AM EDT, Peter Petrik <[email protected]> wrote: Progress report: https://qgis.org/downloads/macos/nightly2/qgis_nightly2_master_20200814_023140.dmg Fixed - qgis/QGIS-Mac-Packager#12 - qgis/QGIS#32163 - qgis/QGIS-Mac-Packager#28 - qgis/QGIS-Mac-Packager#33 - geoPDF GDAL driver Build, Bundled, but still to test - GRASS, GDAL and SAGA processing providers - notarsation qgis/QGIS-Mac-Packager#31 - Qt Creator qgis/QGIS-Mac-Packager#3 Feel free to test and report the issues to the QGIS-Mac-Packager repo — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[PeterPetrik]

@thayeray thanks, best to create new tickets directly in QGIS-Mac-Packager. I have created this one for you qgis/QGIS-Mac-Packager#72

[PeterPetrik]

all but notarisation should be fixed in the tomorrow's nightly2 build

[MtnBiker]

https://gisunchained.wordpress.com/2019/05/29/using-qgis-from-conda/ Is this still the up to date method? I tried with miniconda installed with brew. Not totally stable. But maybe I should start with a new instance/file.

I see that QGIS doesn't accept earmarked funding. Is there another way to get a US 501-c(3) funded effort for a Silicon version?