Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3 packaging for macos #1575

Closed
totaam opened this issue Jul 10, 2017 · 22 comments
Closed

python3 packaging for macos #1575

totaam opened this issue Jul 10, 2017 · 22 comments

Comments

@totaam
Copy link
Collaborator

totaam commented Jul 10, 2017

python3 tracker ticket: #1568.

TODO:

  • update moduleset to latest python (ie: 3.6.x, currently building 3.4!)
  • add all dependencies to moduleset
  • add a new meta module for python3 xpra
  • py2app packaging
    etc..
@totaam
Copy link
Collaborator Author

totaam commented Jul 12, 2017

This may fix issues like #761 and #404, worth a try.

Does the GTK3 macos port support opengl though? (#1569)

@totaam
Copy link
Collaborator Author

totaam commented Nov 12, 2017

Needed for window transparency: #1570

@totaam
Copy link
Collaborator Author

totaam commented Jan 3, 2018

2018-01-03 03:20:08: smo commented


I tried to build the up to date moduleset but had a problem with pygobject. Here is my contact to the mailing list which didn't really go anywhere.

https://mail.gnome.org/archives/gtk-osx-devel-list/2017-September/msg00000.html

@totaam
Copy link
Collaborator Author

totaam commented Jan 3, 2018

2018-01-03 15:45:00: antoine commented


Using a clean user account and following the build instructions from [https://wiki.gnome.org/Projects/GTK+/OSX/Building]:

wget --no-check-certificate \
    "https://git.gnome.org/browse/gtk-osx/plain/gtk-osx-build-setup.sh"
sh ./gtk-osx-build-setup.sh
PATH=/Users/gtk3/.local/bin:$PATH
jhbuild bootstrap

Got a few certificate errors with: xz, pkg-config, flex, expat, etc (download without checking certs and continue).
Then it fails to build python3 with another cert error:

jhbuild buildone python3
jhbuild buildone: could not download http://git.gnome.org/browse/gtk-osx/plain/modulesets-stable/gtk-osx.modules: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>

This time fixed by downloading the cacert.pem from here:
[https://curl.haxx.se/docs/caextract.html] and then telling jhbuild to use it:

CA_CERT_FILE=$HOME/cacert.pem jhbuild build python3

Make python3 the default (as per docs - though the upstream instructions look wrong for the second symlink):

jhbuild shell
cd $JHBUILD_PREFIX/bin
ln -s python3 python
ln -s python3-config python-config
jhbuild build openssl

(upstream notes apply: interrupt the build and run it by hand - we should upstream our openssl 1.1 build commands)

jhbuid build meta-gtk-osx-bootstrap meta-gtk-osx-gtk3

Got a minor failure building glib:

xzcat -d "/Users/gtk3/gtk/source/pkgs/glib-2.52.2.tar.xz" | tar xf -
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-gint64-long-long.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-gint64-long-long.patch"
patching file configure.ac
Hunk #1 succeeded at 3002 (offset 48 lines).
Hunk #2 succeeded at 3114 (offset 48 lines).
Hunk #3 succeeded at 3266 (offset 49 lines).
*** Applying patch https://git.gnome.org/browse/gtk-osx/plain/patches/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch *** [14/39]
patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch"
patching file gio/gosxappinfo.c
patching file gio/gosxcontenttype.c
Hunk #1 succeeded at 52 (offset -6 lines).
patching file gio/tests/contenttype.c
Hunk #1 succeeded at 323 with fuzz 1 (offset -38 lines).
Hunk #2 FAILED at 359.
1 out of 2 hunks FAILED -- saving rejects to file gio/tests/contenttype.c.rej
*** Error during phase checkout of glib: ########## Error running patch -p1 < "/Users/gtk3/.cache/jhbuild/glib-2.52-Fallback-to-CFStringGetCSTring-if-CFStringGetC.patch" *** [14/39]

Applied the one line from the reject file by hand and continued.

Everything built correctly after that.

@smo: whatever you did (which was not recorded anywhere AFAICT) must have been different?


Continuing from there:

  • r17840 makes the moduleset more reusable
  • add our moduleset to jhbuild:
cat > $HOME/.jhbuildrc-custom << EOF
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
EOF
  • and build some things we will need:
jhbuild buildone sshpass -f
jhbuild build meta-subversion-xpra
jhbuild build meta-osx-xpra-pkgtools
jhbuild build meta-osx-xpra-codec-deps

Got one failure we should fix on lame:

gzip -dc "/Users/gtk3/gtk/source/pkgs/lame-3.99.5.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [8/23]
patch -p0 < "/Users/gtk3/.cache/jhbuild/lame-channels.patch"
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/libmp3lame/set_get.c
|+++ b/libmp3lame/set_get.c
--------------------------
File to patch: 
Skip this patch? [y] 
Skipping patch.
patch: **** malformed patch at line 10:

Next we have to build python and switch back to python3 afterwards so we can build the python deps against python3:

jhbuild buildone python
jhbuild shell
cd $JHBUILD_PREFIX/bin
rm python python-config
ln -s python3 python
ln -s python3-config python-config

Build the rest:

jhbuild build meta-osx-xpra-python-deps

lz4 had a problem (will fix).
Then python3 had a problem finding ssl, it just needed to be rebuilt now that we brought in a newer openssl lib version:

jhbuild buildone python3 -f

At that point, everything is built OK, AFAICT.

Still TODO:

  • merge stable moduleset updates (oh joy..)
  • add python3 / gtk3 macos handling to our setup.py

@totaam
Copy link
Collaborator Author

totaam commented Jan 4, 2018

@smo: why are we so far behind [https://git.gnome.org/browse/gtk-osx/tree/]?
Even gtk2 is outdated here. (only our gstreamer and network modulesets are slightly ahead)
Is there anything there that we should not merge and why?

I've merged most of the changes:

  • r17846: mostly things we don't even use - and maybe we shouldn't copy them then, just use an https module reference and be done with it - and we can still override some packages with newer ones, only when we need to keep an older version do we really need to copy the moduleset, right?
  • r17847: bootstrap, network, python
  • r17848: main gtk-osx moduleset
  • r17849: glib for gobject-introspection (will check the python2 re-build later)
  • r17850 + r17851: moduleset updates for packaging tools
  • r17855: gtkosx_application import for GTK3 (gi bindings)
  • r17856: python-cryptography new requirement?
  • r17857: tiny py3k string fix
  • r17858 + r17859: force py2app to not include the desktop server (raising ImportError does it)
  • r17860: a really annoying issue is that although the headers look the same, py3cairo and pycairo install in different places, but our source needs to reference just one header file.. so this a manual step is still needed:
ln -sf pycairo.h /Users/gtk3/gtk/inst/include/pycairo/py3cairo.h

Also added latest gdb by hand:

wget https://ftp.gnu.org/gnu/gdb/gdb-8.0.1.tar.gz
tar -zxvf gdb-8.0.1.tar.gz 
cd gdb-8.0.1
./configure --prefix=${JHBUILD_PREFIX}
make && make install
sudo chgrp procmod ${JHBUILD_PREFIX}/bin/gdb
sudo chmod g+s ${JHBUILD_PREFIX}/bin/gdb

But... still needs signing to be used: codesign gdb.

@totaam
Copy link
Collaborator Author

totaam commented Jan 4, 2018

More work on this:

  • r17861: gtk-mac-bundler requires python2 to run
  • r17864: yet another new python-lz4 dependency (sigh), r17868 + r17872: macos packaging for it
  • r17870: support new versions of python-lz4 (applies to all arches)
  • r17866: gtk-quartz-engine isn't in the moduleset any more? it is gtk2 and contains build errors... remove it
  • r17865: fixup r17855 gtkosx application import
  • r17867 + r17881: packaging tweaks: ship more gi bindings, skip (py)gtk2 stuff, etc
  • r17876: clipboard support for gtk3
  • r17877 + r17878: tray, clipboard and gui porting

Still TODO:

  • gdk bindings need porting (input devices stuff, see Wacom Tablet OSX Support #1157#comment:24) unless GTK3 gives us the devices and the correct values out of the box?
  • opengl: need to implement the gl backend glue, unfortunately there is no easy way to get the "nsview" from a gdk window in GTK3... so we may have to use a Cython class
  • fix the GtkMenuBar invalid cast warnings, somehow

Apart from that, it looks usable. Beta builds posted here with the "Python3" filename: [https://xpra.org/beta/osx/].

@totaam
Copy link
Collaborator Author

totaam commented Jan 5, 2018

Notes after rebuilding again from scratch to verify:

  • pygobject3 needed this hack to avoid errors on "stack_size":
find . -name "Makefile" -exec sed -i '' 's+-Wl,-stack_size,1000000 ++g' {} \;
+--- a/libxml2-2.9.4/python/types.c	2016-02-09 03:17:33.000000000 -0700
++++ b/libxml2-2.9.4/python/types.c	2016-12-21 12:34:06.755650986 -0700
+@@ -31,8 +31,6 @@
+     const char *mode;
+ 
+     fd = PyObject_AsFileDescriptor(f);
+-    if (!_PyVerify_fd(fd))
+-        return(NULL);
+     /*
+      * Get the flags on the fd to understand how it was opened
+      */

Saves having to temporarily switch back to python2 to build it.

The python2-only builds started hitting the issue from comment:7, and now the mixed python2-python3 builds do too. Sigh.

One more weird issue started happening: the "py2app" step does something that loads gdk and tries to access the display server and crashes when it fails, means that the builds now have to be executed from a GUI shell. PITA.


So at this point, the python3 jhbuild env is almost straightforward (bar python3 vs python issues) with this file:

cat > .jhbuildrc-custom  <<EOF
_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "openssl", "python3", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core", "meta-gtk-osx-gtk3",
"libpng",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-mac-integration-python", "meta-osx-python3-xpra-deps"]
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
EOF

@totaam
Copy link
Collaborator Author

totaam commented Jan 6, 2018

Lots more updates (in large part because testing jhbuild is hard, so I commit then fix the numerous mistakes one at a time).
I've also sent some questions and pull requests upstream to try to reduce our diff:

Changesets:

With all these changes:

  • the modulesets are cleaner, and closer to upstream
  • the differences between the python2 and python3 / gtk3 modulesets are easier to see and maintain
  • building everything from scratch is easier:
cat > .jhbuildrc-custom <<EOF
_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.10", sdk_version="10.10", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = ["meta-osx-xpra-deps"]
moduleset="https://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk3.modules"
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"
EOF

Then just: jhbuild build.
The only difference for the "GTK2" moduleset is the moduleset URL:

moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/xpra-gtk2.modules"

Then when building the packages, we have to do:

PYTHON=python3 ./make-all.sh

(or PYTHON=python2 for GTK2 builds)

@smo: does that work for you?

@totaam
Copy link
Collaborator Author

totaam commented Jan 8, 2018

Note: the gi bingings were missing from the python2 builds (meaning no sound), to re-instate them I had to apply this patch: pygobject: remove references to deprecated GI_INFO_TYPE_ERROR_DOMAIN. (basically, just remove all references to GI_INFO_TYPE_ERROR_DOMAIN then rebuild with introspection enabled)
There might be a better way of doing this.

@totaam
Copy link
Collaborator Author

totaam commented Jan 10, 2018

2018-01-10 16:47:44: smo commented


gtk-engines fails when building for python3/gtk3 is this needed?

checking for GTK... no
configure: error: GTK+-2.12 is required to compile gtk-engines
*** Error during phase configure of gtk-engines: ########## Error running ./configure --prefix /Users/xpragtk3/gtk/inst    *** [47/129]

@totaam
Copy link
Collaborator Author

totaam commented Jan 10, 2018

gtk-engines fails when building for python3/gtk3 is this needed?

It isn't referenced anywhere in our modulesets.
Can you get some info on it? jhbuild info gtk-engines

@totaam
Copy link
Collaborator Author

totaam commented Jan 16, 2018

2018-01-16 18:02:24: smo commented


Name: gtk-engines
Module Set: gtk-osx-themes
Type: autogen
Install version: not installed
Install date: not installed
URL: http://ftp.gnome.org/pub/GNOME/sources/gtk-engines/2.20/gtk-engines-2.20.2.tar.bz2
Version: 2.20.2
Tree-ID: 2.20.2-d41d8cd98f00b204e9800998ecf8427e
Sourcedir: /Users/xpragtk3/gtk/source/gtk-engines-2.20.2
Requires: automake, libtool, make
Required by: meta-gtk-osx-themes
After: meta-gtk-osx-core

@totaam
Copy link
Collaborator Author

totaam commented Jan 17, 2018

meta-gtk-osx-themes is found here: https://github.com/GNOME/gtk-osx/blob/master/modulesets-stable/gtk-osx-themes.modules, but isn't referenced anywhere else in the modulesets.
There is a meta-gtk-osx-gtk3-core-themes, so I've switched to that in r18020.
@smo: does that fix things? Do we need to add icon-naming-utils?

@totaam
Copy link
Collaborator Author

totaam commented May 17, 2018

2018-05-17 20:41:02: smo commented


Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

ld: -stack_size option can only be used when linking a main executable

We can see the configure line here

./configure --prefix /Users/xpragtk3/gtk/inst --libdir="$JHBUILD_LIBDIR" --with-python  --with-python-install-dir=/Users/xpragtk3/gtk/inst/lib/python2.7/site-packages

I'm not sure where the --with-python-install-dir comes from it doesn't appear in the moduleset?

If you run configure again and change it to the correct python install dir it seems to install minus having to stop the build to use the same line as comment:11

python -c "import libxml2; print(libxml2)"
<module 'libxml2' from '/Users/xpragtk3/gtk/inst/lib/python3.6/site-packages/libxml2.py'>

seems to load properly now

@totaam
Copy link
Collaborator Author

totaam commented May 17, 2018

2018-05-17 21:01:31: smo commented


itstool 2.0.2 doesn't work with python3
itstool 2.0.4 works with python3 but segfaults when used building gtk-doc
itstool from git seems to work just fine had to install this manually

@totaam
Copy link
Collaborator Author

totaam commented May 17, 2018

2018-05-17 23:55:51: smo commented


seems to be some problems with building lame still as well

** Checking out lame *** [105/135]
curl --continue-at - -L http://winswitch.org/src/lame-3.100.tar.gz -o /Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3983  100  3983    0     0  11362      0 --:--:-- --:--:-- --:--:-- 11380
*** Error during phase checkout of lame: file hash is incorrect (expected ddfe36cab873794038ae2c1210557ad34857a4b6bdc515785d1da9e175b1da1e, got 30641cf26a0d4dc26ca6b97e719c52b4c72d49014a09cd28309a9d989a0408a2) *** [105/135]

Hash is wrong but the url is actually invalid for the download. Should this be switched or was there a good reason for having it on winswitch.org?

*** Checking out lame *** [105/135]
gzip -dc "/Users/xpragtk3/gtk/source/pkgs/lame-3.100.tar.gz" | tar xf -
*** Applying patch http://xpra.org/svn/Xpra/trunk/osx/jhbuild/patches/lame-channels.patch *** [105/135]
patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch"
patching file libmp3lame/set_get.c
Hunk #1 FAILED at 91.
1 out of 1 hunk FAILED -- saving rejects to file libmp3lame/set_get.c.rej
*** Error during phase checkout of lame: ########## Error running patch -p1 < "/Users/xpragtk3/.cache/jhbuild/lame-channels.patch" *** [105/135]

After getting the correct tarball this patch still fails.

If I ignore the patch failing and try to build I get build errors like these

Undefined symbols for architecture x86_64:
  "_lame_init_old", referenced from:
     -exported_symbol[s_list] command line option
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [libmp3lame.la] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of lame: ########## Error running make -j 3  *** [105/135]

@totaam
Copy link
Collaborator Author

totaam commented May 17, 2018

2018-05-17 23:57:13: smo commented


gst-plugins-base has issue building the examples

libtool: link: unable to infer tagged configuration
libtool:   error: specify a tag with '--tag'

Don't build the examples with --disable-examples configure flag and everything builds fine.

@totaam
Copy link
Collaborator Author

totaam commented May 18, 2018

2018-05-18 00:28:46: smo commented


scons doesn't build under python3

*** Building scons *** [131/135]
python setup.py build
scons: *** SCons does not run under Python version 3.6.4.
Python 3 and above are not yet supported.
*** Error during phase build of scons: ########## Error running python setup.py build *** [131/135]

need this to build serf and subversion

@totaam
Copy link
Collaborator Author

totaam commented May 18, 2018

2018-05-18 00:29:20: smo commented


Bunch of warnings I noticed as well when you start the build


W: meta-osx-xpra-python3-deps has a dependency on unknown "gst-python" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: meta-osx-xpra-python3-deps has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-pyasn1-modules has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module
W: python3-ldap has a dependency on unknown "python3-pyasn1" module

@totaam
Copy link
Collaborator Author

totaam commented May 21, 2018

Started from scratch again today ran into some issues with libxml2. It would always just build the python2 variant and then would also fail with

Weird, I didn't hit this one.

itstool from git seems to work just fine had to install this manually

Again, I didn't hit this one. (I'm building on 10.10)
We'll just wait for the next release and update itstool in the moduleset then.

seems to be some problems with building lame still as well

Yep. It's an upstream bug:

gst-plugins-base has issue building the examples

Fixed in r19356.

I hit a more fundamental issue rebuilding gstreamer:

  GEN      Gst-1.0.gir
(..)
ImportError: dlopen(/Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so, 2): Symbol not found: _PyInt_FromLong
  Referenced from: /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so
  Expected in: flat namespace
 in /Users/gtk3/gtk/inst/lib/gobject-introspection/giscanner/_giscanner.so

No idea when the broke, or why.
Had to rebuild gobject-introspection which caused dozens of other modules to get rebuilt.

Then something else was not quite right - looks like some headers aren't installed by gst-plugings-base, gst-libav fails with:

/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:10: fatal error: 'gst/audio/gstaudioaggregator.h' file not found
#include <gst/audio/gstaudioaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gst-plugins-base-1.14.1/gst-libs/gst/audio/gstaudioaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/audio/

Then after that:

In file included from /Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/audio.h:36:
/Users/gtk3/gtk/inst/include/gstreamer-1.0/gst/audio/gstaudioaggregator.h:27:10: fatal error: 'gst/base/gstaggregator.h' file not found
#include <gst/base/gstaggregator.h>
         ^
1 error generated.

Fixed by:

cp ${JHBUILD_SOURCE}/gstreamer-1.14.1/libs/gst/base/gstaggregator.h ${JHBUILD_PREFIX}/include/gstreamer-1.0/gst/base/

Bunch of warnings I noticed as well when you start the build
pyasn1 ...

r19355 fixes that.

@totaam
Copy link
Collaborator Author

totaam commented May 24, 2018

2018-05-24 20:12:43: smo commented


They seem to have a fix for the lame issue over at homebrew site

[https://github.com/Homebrew/homebrew-core/blob/master/Formula/lame.rb]

    # Fix undefined symbol error _lame_init_old
    # https://sourceforge.net/p/lame/mailman/message/36081038/
    inreplace "include/libmp3lame.sym", "lame_init_old\n", ""

Just remove the offending line from include/libmp3lame.sym and all is well. I'll add a patch.

@totaam
Copy link
Collaborator Author

totaam commented May 24, 2018

2018-05-24 22:51:48: smo commented


r19418 adds the patch to the repo
r19419 adds it to the moduleset

lame now builds without error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant