CPython integration help needed

[tlaurion]

Micropython would not fit the bill of wyng dependencies in the long run.

CPython could work with additional space freed with modified IFD and Neutered ME, but I would need to understand how to properly cross-compile it.

Outcome would be chipsec inclusion, wyng-backups inclusion permitting to have factory disk restore/integrity validation inside of Heads.

Linked #233

[osresearch]

I'm using micropython in other projects with good success, but those embedded systems don't have deep dependency trees.

I'll poke at cpython to see if there is an easy way to apply the fix from 360651

[osresearch]

I first build a native version for the build machine and then build a cross compiled version using the native as the $PYTHON_FOR_BUILD that appears to work in a chroot environment:

CC=$HEADS/crossgcc/bin/x86_64-linux-musl-gcc \
PYTHON_FOR_BUILD=`pwd`/../build/python \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
have_openssl=no \
../configure \
  --build x86_64-linux-musl \
  --host x86_64-linux-gnu \
  --disable-ipv6 \
  --prefix=/tmp/root/usr/local \

make "passes", although many of the extensions fail. It also appears that quite a bit of the code ends up ignoring $(CC) and building with gcc instead (all of the cpython/Modules directory?).

Another problem is that python wants openssl, which is a massive dependency.

[osresearch]

Yeah, the wrong compiler is definitely going to make this a mess:

~ # python
Python 3.9.0a4+ (heads/master-dirty:efc28bb, Mar  5 2020, 19:38:39) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: Error loading shared library libffi.so.6: No such file or directory (needed by /usr/local/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so)
>>> 

[osresearch]

Oops, I swapped the --host and --build values. Rebuilding with this for the configuration:

CC=~/heads/install/bin/musl-gcc \
 CXX=~/heads/install/bin/musl-gcc \
READELF=~/heads/crossgcc/x86_64-linux-musl/x86_64-linux-musl/bin/readelf \
PYTHON_FOR_BUILD=`pwd`/../build/python \
ac_cv_file__dev_ptmx=no \
ac_cv_file__dev_ptc=no \
have_openssl=no \
../configure \
  --host x86_64-linux-musl \
  --build x86_64-linux-gnu \
  --disable-ipv6  \
  --prefix=/tmp/root/usr/local \

Takes forever due to warnings in Modules/unicodename_db.h, but does appear to use the correct gcc this time. However, import ctypes produces the same error, so something is still borked.

[tlaurion]

@osresearch : Anything practical to provide here to help fix things upstream?

[osresearch]

I was idly wondering about building a fakeroot environment so that the build environment is "native" and doesn't have the cross compilation issues. This would mean building a bootstrap compiler to build the musl native compiler, although since we have busybox already built for musl we have most of the rest of the environment that we would need.

[tlaurion]

@osresearch : ping?

[tlaurion]

Some progress upstream https://bugs.python.org/issue45881

[tlaurion]

Was able to compile. 13mb static binary unpacked and unstripped, built only when host provides the same python version, so PoC under 3.9.2

[tlaurion]

Not bad, 800k stripped cost, compressed under intrd with xz

[tlaurion]

Not bad, 800k stripped cost, compressed under intrd with xz

Well, this is just the interpreter alone. Still not bad, but still not ready, everything else is missing and a lot of errors are still happening for Modules/unicodename_db.h and some modules fail to be compiled.

Buildroot seems to be used in other cross-compiling environments https://github.com/minghuadev/minecraft-tests/blob/master/other-p2js/arm-cross-python-386/readme-buildroot-howto

[tlaurion]

Still some issues upstream with musl-cross-make and python's assumptions, building target against host's buildstack, not defined one per environment variables, autotool config being faulty and hacks needed, pending upstream.

python/cpython#24502
python/cpython#95855

As of today with 3.9.2, python tries to link against host's libc and does not respect libc path given at configure step, resulting in build failing. Details in WiP inclusion PR at #1216 (comment)

[tlaurion]

Seems like the following might fix the problem and OpenWrt is a good source for those kind of issues

• https://github.com/openwrt/packages/blob/992fcd1bd8770bb44a56bc4173ac3befe0fa16ef/lang/python/python3/patches/030-bpo-43112-detect-musl-as-a-separate-SOABI-GH-24502.patch
• https://github.com/openwrt/packages/blob/38f01ad2c9738c6c5a5dbda85019bb9cf5bc10f8/lang/python/python3/patches/024-musl-find_library.patch

[tlaurion]

Opened issue python/cpython#97842

[tlaurion]

python/cpython#107221 seem to fix triplet and libc fixation fixing python/cpython#97842

Might be time to retry!