2022.7.x docker container broken on 1st gen Raspberry Pi

[abochmann]

The problem

I'm trying to run home-assistant:stable with the docker compose method on an Raspberry Pi Model B Rev 2 (512MB RAM / armv6l). This is obviously not a good platform, but it's what I had available for a test. The OS is Raspbian/bullseye, with docker ce 20.10.17

Running docker compose up, the container starts, but the homeassistant inside seems to crash:

Attaching to homeassistant
homeassistant  | s6-rc: info: service s6rc-oneshot-runner: starting
homeassistant  | s6-rc: info: service s6rc-oneshot-runner successfully started
homeassistant  | s6-rc: info: service fix-attrs: starting
homeassistant  | s6-rc: info: service fix-attrs successfully started
homeassistant  | s6-rc: info: service legacy-cont-init: starting
homeassistant  | s6-rc: info: service legacy-cont-init successfully started
homeassistant  | s6-rc: info: service legacy-services: starting
homeassistant  | services-up: info: copying legacy longrun home-assistant (no readiness notification)
homeassistant  | s6-rc: info: service legacy-services successfully started
homeassistant  | [22:08:03] INFO: Home Assistant Core finish process exit code 256
homeassistant  | [22:08:03] INFO: Home Assistant Core finish process received signal 4
homeassistant  | [22:08:09] INFO: Home Assistant Core finish process exit code 256
homeassistant  | [22:08:09] INFO: Home Assistant Core finish process received signal 4

A plain Home Assistant Core venv starts successfully on the same hardware, as does the previous release of the docker container using 2022.6.6@sha256:c2e7c0bb84e6f84d1c62ff34d9f286fb17486ccf2760a5612f65128848c59316.

What version of Home Assistant Core has the issue?

2022.7.5

What was the last working version of Home Assistant Core?

2022.6.6

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

When looking at the resulting core file with gdb, it reports an illegal instruction:

Core was generated by `python3 -m homeassistant --config /config'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0xb58fe7fa in ?? ()

I assume the build uses some compiler flag that creates instructions not supported by the BCM2835?

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[abochmann]

Edited to add that the 2022.6.6 container starts up on the same system.

[abochmann]

Edited subject to clarify this is a new issue in 2022.7, and the problem persists in 2022.7.5

It's possible to start python3 itself inside the container, so I'd assume the problem is in some module that gets pulled in by homeassistant:

$ docker ps
CONTAINER ID   IMAGE                                          COMMAND   CREATED              STATUS          PORTS     NAMES
e92eae05a31e   ghcr.io/home-assistant/home-assistant:latest   "/init"   About a minute ago   Up 56 seconds             homeassistant
$ docker exec e92eae05a31e python3 -V
Python 3.10.5

[richardzone]

Can confirm Raspberry Pi 1b works with 2022.6.7 as well, but it does not work with 2022.7.*

[abochmann]

Yesterday, I replaced the Pi 1b with a Pi 3b+, so I'm not directly affected anymore. I'll keep the old setup around for a couple more days in case anyone picks this up and needs some testing.

[danielrheinbay]

Duplicate of #74707?
See #74707 (comment)

[richardzone]

Duplicate of #74707? See #74707 (comment)

@danielrheinbay I don't think this is duplicate as the ARM architecture(armv6l vs armv7), error message, Linux kernel version(recent vs old) are all different.

The linux version on my RPi 1b running Raspbian/Debian Bullseye:

Linux richardpi 5.10.92+ #1514 Mon Jan 17 17:35:21 GMT 2022 armv6l GNU/Linux

[danielrheinbay]

@richardzone Fair enough, the environments do differ. I am also affected by this issue (the most recent version of hass core that runs on my Raspi 1b using Docker is 2022.6.7), so I reviewed existing issues and found @frenck's comment in #74707 pointing to the update of the Alpine base image. I was wondering if this update of was also what is causing 2022.7.x version of hass core to crash on Raspi 1s.

[abochmann]

I have not found any indication of a general problem with Alpine 3.16 on the RPi 1.

Maybe there's a connection to numpy/numpy#20765 and piwheels/packages#276 though? Unfortunately neither issue seems to mention how they fixed the build problem.

Anyway, when I run the Alpine python3 inside my homeassistant container and do an import numpy, it immediately crashes. I tried importing a couple of other random modules and saw no problem.

[mathmaniac43]

I am experiencing identical behavior, same platform, with the new 2022.8.1

[larsxschneider]

According to the debugger it breaks here:
https://github.com/home-assistant/core/blob/2022.8.1/homeassistant/__main__.py#L69

bash-5.1# python3 -m pdb -m homeassistant --config /config
> /usr/src/homeassistant/homeassistant/__main__.py(1)<module>()
-> """Start Home Assistant."""
(Pdb) break /usr/src/homeassistant/homeassistant/__main__.py:69
Breakpoint 1 at /usr/src/homeassistant/homeassistant/__main__.py:69
(Pdb) c
> /usr/src/homeassistant/homeassistant/__main__.py(69)get_arguments()
-> from . import config as config_util
(Pdb) n
Segmentation fault

Stepping into this call just gives me "frozen" gibberish:

-> from . import config as config_util
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(404)parent()
(Pdb) s
> <frozen importlib._bootstrap>(407)parent()
(Pdb) s
> <frozen importlib._bootstrap>(408)parent()
(Pdb) s

But poking around the config utils I found that import jwt creates a segfault:
https://github.com/home-assistant/core/blob/2022.8.1/homeassistant/auth/__init__.py#L10

Does this make any sense to someone? 😄

[larsxschneider]

Importing crypto libs seems so be the problem:

Python 3.10.5 (main, Jun 22 2022, 16:42:00) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cryptography.hazmat.primitives.asymmetric import ec, padding
Segmentation fault

But this is wrapped in a try/except block... so it might be a red herring: https://github.com/jpadilla/pyjwt/blob/0bef0fbff5c245668578a43774d8620bdba4a6f7/jwt/algorithms.py#L18-L57

[danielrheinbay]

Good catch, @larsxschneider!
Afaik, in Python, a segfault effectively crashes the interpreter itself, so there is no way to catch it using try/except (or any other way, for that matter). Common causes for segfaults (double frees, accessing beyond boundaries of allocated memory, use after free etc.) are typical for compiled modules written in e.g. C.

Indeed, cryptography contains compiled modules: https://cryptography.io/en/latest/installation/

[larsxschneider]

@danielrheinbay thanks! I think your link explains it - there is no ARM 32bit version. That's what my QNAP ARM would need I think.

[danielrheinbay]

The same goes for our RasPi 1 and 2, I guess :/

[danielrheinbay]

So I have reviewed older versions of the documentation and noticed:

• It does not explicitly say armhf (or any other 32-bit ARM arch) is not supported. It just says the other archs are supported and tested.
• I couldn't find any recent changelog entries relating to 32 bit support on ARM, e.g. no "deprecated" notice.
• Major Linux distros (e.g. Debian) ship python3-cryptography in their current versions for 32 bit ARM (e.g. armhf).

So we may have to dig further to identify the actual root cause.

[larsxschneider]

Just for completeness, on 2022.6.7 both imports, numpy and jwt, work without issues:

bash-5.1# cat /usr/src/homeassistant/homeassistant/const.py | grep VERSION | head -3
MAJOR_VERSION: Final = 2022
MINOR_VERSION: Final = 6
PATCH_VERSION: Final = "7"
bash-5.1#
bash-5.1# python
Python 3.9.12 (main, May 13 2022, 12:43:07)
[GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import jwt
>>>

Apparently cryptography changed something in their build pipeline some time ago which changes the supported architectures: pyca/cryptography#5771

However, I don't think that is the problem here because numpy crashes too. What is strange is, that numpy has no dependencies. So, I am still clueless what the real problem is.

[larsxschneider]

Maybe there's a connection to numpy/numpy#20765 and piwheels/packages#276 though? Unfortunately neither issue seems to mention how they fixed the build problem.

Re-reading the thread, I think @abochmann really found the core problem in this link above.

Apparently a newer piwheels should fix the problem?! See piwheels/packages#276 (comment)

[danielrheinbay]

Apparently a newer piwheels should fix the problem?!

What do you mean when you say "a newer piwheels", @larsxschneider?

From piwheels.org:

piwheels is a Python package repository providing Arm platform wheels (pre-compiled binary Python packages) specifically for the Raspberry Pi, making pip installations much faster. Packages are natively compiled on Raspberry Pi 3 and 4 hardware using the Mythic Beasts Pi cloud.

From piwheels/packages#276 (comment):

Since a recent update to our provider's OS images, we've noticed that the version of gcc we have is compiled strictly for armv7 and not compatible with armv6 as before. This means that although wheels work on armv7 models (Pi 2 and up, inc Zero 2), they don't work on armv6 models (Pi 1 and Zero). This causes a fatal "Illegal instruction" error.

The issue applies to all (I think) armv6 wheels built since around mid-December.

Then, two days later in piwheels/packages#276 (comment):

All packages have been rebuilt, this should have fixed the problem now.

So the way I understand it is this:
piwheels is a repository offering all sorts of pre-compiled Python packages, aka wheels, specifically compiled for Raspis. piwheels.org's cloud provider, Mythic Beasts Pi, updated their OS images. This update contained a gcc version which, when compiling Python packages for piwheels.org repositories, would produce binaries that are incompatible with armv6 Raspis, e.g. Raspi 1 and Raspi 2. This issue has bin fixed more than half a year ago, and all affected wheels have since been rebuilt.

[larsxschneider]

@danielrheinbay I am no home assistant and no python expert. So read my comment with that in mind.

Is it possible that the faulty version of Piwheels was backed into the Homeassistant docker image? If that is true, then the problem could still be present although the issue was fixed half a year ago.

[Wetzel402]

I am still seeing a boot loop as of version 2022.9.

[kuznetsss]

So the way I understand it is this:
piwheels is a repository offering all sorts of pre-compiled Python packages, aka wheels, specifically compiled for Raspis. piwheels.org's cloud provider, Mythic Beasts Pi, updated their OS images. This update contained a gcc version which, when compiling Python packages for piwheels.org repositories, would produce binaries that are incompatible with armv6 Raspis, e.g. Raspi 1 and Raspi 2. This issue has bin fixed more than half a year ago, and all affected wheels have since been rebuilt.

I found that Home Assistant has it's own wheels repo https://github.com/home-assistant/wheels
So maybe if the problem is related to wheels it would be useful to create an issue in that repo. What do you think?

[kuznetsss]

Also can confirm that import numpy inside container still causes a crash for version 2022.9.1 on Odroid C2 (aarch64 CPU)

bash-5.1# python
Python 3.10.5 (main, Jun 22 2022, 16:10:59) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)

I found the issue numpy/numpy#18131 in numpy repo but it has been fixed.

[Wetzel402]

Interestingly this issue is also referenced in the wheels issue 276. Issue 276 and the numpy issue were both closed around January 17 but I'm not seeing a clear commit in the commit history.

As @kuznetsss pointed out, HA has it's own wheels it would be worth while to verify the fix implemented Jan 17 is also in the HA variant.

[HarvsG]

Bricked my native python install trying to upgrade and interestingly it was getting stuck on building cryptography (for some reason no wheel available) - likely related to the rust dependency: piwheels/packages#276 (comment)

Cryptography which broke a lot of builds and wheels last year: pyca/cryptography#5771

So now fresh install, trying docker and came across this issue

interestingly jwt imports cryptography https://github.com/GehirnInc/python-jwt/blob/fd684745b2de884a32e6d7a423e659e4d262fb27/setup.py#L27

Although I cant find that numpy does

[AloneLiberty]

I've done some investigation about this issue, the issue is deeper than cryptography and numpy. It is expanded on all packages with compiled (С/C++/Rust) code.
When I recompiled packages on my Pi 0W, it seems to work. But what is strange - I tried to build through Github Actions and packages also work from it.
I don't know why packages from Home Assistant wheels index doesn't work, maybe it was temporary compiler issue (like piwheels/packages#276).
Can someone else try to compile numpy or any other failing packages on Github Actions and confirm/refute my theory? If the theory is true, we should ask developers to rebuild all armhf packages and issue will be solved.

[Wetzel402]

I confirmed this problem still exists on 2022.11.5. I tried to build 2022.11.5 on my QNAP NAS but so far I haven't been able to. I'm a noob when it comes to building from source. I tried downloading the HA core repo and adding the build command to my docker compose but it gives me an error saying I need to check my yaml. I will build on my machine if I can figure it out.

[Wetzel402]

I'm no expert so I'm not sure if I'm on to something or not. I noticed that this file was created in June specifically for armhf. Could this be our issue?

Edit: I've attempted to build the HA core container image using the Alpine and Alpine Python base images as well as the Debian base image. I always get errors installing dependencies for python astral and I'm not sure how to resolve that.

This is my docker file...

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-base-debian:latest #tried Alpine and Debian base images

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
COPY requirements.txt homeassistant/
COPY homeassistant/package_constraints.txt homeassistant/homeassistant/
RUN \
    apt-get update && apt-get -y upgrade && apt-get -y install python3 python3-pip #needed for Debian image since python and pip are not included
RUN \
    pip3 install --no-binary=:all: --find-links "${WHEELS_LINKS}" \ #build wheels
    -r homeassistant/requirements.txt --use-deprecated=legacy-resolver
COPY requirements_all.txt home_assistant_frontend-* homeassistant/
RUN \
    if ls homeassistant/home_assistant_frontend*.whl 1> /dev/null 2>&1; then \
        pip3 install --no-cache-dir --no-index homeassistant/home_assistant_frontend-*.whl; \
    fi \
    && pip3 install --no-binary=:all: --find-links "${WHEELS_LINKS}" \ #build wheels
    -r homeassistant/requirements_all.txt --use-deprecated=legacy-resolver

## Setup Home Assistant Core
COPY . homeassistant/
RUN \
    pip3 install --no-binary=:all: --find-links "${WHEELS_LINKS}" \ #build wheels
    -e ./homeassistant --use-deprecated=legacy-resolver \
    && python3 -m compileall homeassistant/homeassistant

# Home Assistant S6-Overlay
COPY rootfs /

WORKDIR /config

It should be noted it builds fine with wheels but then loops which is what we expect. I tried to build with wheels and the 2022.06 Alpine base image just to confirm that Alpine wasn't the problem. It loops as well.

The next step is to build an image that includes building wheels but I'm not going to get there without help since I keep getting errors while installing build dependencies.

[Jojo-1000]

TL;DR

I think the Github Actions are configured wrong to use armv7 instead of armv6 for the wheels packages.
From the logs of armhf base core images (2022.12.9):
Collecting numpy==1.23.2
#13 85.81 Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl


Long story (for troubleshooting in case that is not it)

I ran into this same issue on raspberry pi 1B rev2. It is very easy to reproduce with the built docker containers:

pi@raspberrypi:~ $ docker exec -ti homeassistant bash
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)
bash-5.1#

Because it was suggested that the wheels might be broken, I tried to only pull the wheels directly (my poor pi 1b does not like to compile or download the full requirements). So I created a Dockerfile which only installs the numpy package from the same repository as the Github Actions use and built the container.

Dockerfile:

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
#COPY requirements.txt homeassistant/
#COPY homeassistant/package_constraints.txt homeassistant/homeassistant/

# test install numpy package
RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy

Build output

pi@raspberrypi:~ $ docker build .
Sending build context to Docker daemon   2.56kB
Step 1/5 : ARG BUILD_FROM
Step 2/5 : FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c
 ---> cc6d4d2f7195
Step 3/5 : ENV     S6_SERVICES_GRACETIME=220000
 ---> Using cache
 ---> 70e2347fa3b1
Step 4/5 : WORKDIR /usr/src
 ---> Using cache
 ---> 7ab57f3baea6
Step 5/5 : RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy
 ---> Running in 50736869aad6
Looking in links: https://wheels.home-assistant.io/musllinux/
Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.2/10.2 MB 556.4 kB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-1.23.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 50736869aad6
 ---> 0d13df698c3d
Successfully built 0d13df698c3d

The goal was just to get a simple system where I may have a chance of manually building the single package or swap it to a different source. So I tried to run the import-crash code again and to my surprise it was working:

docker run -ti 0d13df698c3d bash
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.max([1, 2, 3])
3
>>>

I tried to pull the exact same image that was built in the Actions for that version and compared the numpy versions. That is where I noticed that the Actions are pulling from armv7, while my working version uses armv6:

Not working:

Collecting numpy==1.23.2
#13 85.81   Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Working:

Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)

This is the same for all of the packages (although I did not test that every single one is working).

So I think this easily explains the illegal instruction errors, unless I am mistaken and the Actions are only used for CI and the uploaded docker images come from somewhere else. I am not very familiar with the Actions workflows, but this should be pretty easy to fix without any code changes and without rebuilding the wheels. I noticed that the architecture everywhere is set to arm in the logs, so maybe that is why it defaults to armv7?

[samitheberber]

I noticed that linuxserver.io is providing own wheels, so got latest version working by switching to their container. I was thinking trying to figure out how to get wheels built by myself, but decided to go with this easy route for now.

[Wetzel402]

Okay, I built an image on my QNAP NAS with the following dockerfile...

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
COPY requirements.txt homeassistant/
COPY homeassistant/package_constraints.txt homeassistant/homeassistant/
RUN \
    apk --update --upgrade add \
  build-base \
  cairo \
  cairo-dev \
  cargo \
  curl \
  freetype-dev \
  gcc \
  gdk-pixbuf-dev \
  gettext \
  jpeg-dev \
  lcms2-dev \
  libffi-dev \
  libxml2-dev \
  libxslt-dev \
  musl-dev \
  openjpeg-dev \
  openssl-dev \
  pango-dev \
  poppler-utils \
  postgresql-client \
  postgresql-dev \
  py-cffi \
  python3-dev \
  rust \
  tcl-dev \
  tiff-dev \
  tk-dev \
  zlib-dev
RUN \
    pip3 install "setuptools<58.0" && \
    pip3 install -v --no-binary=:all: --find-links "${WHEELS_LINKS}" \
    -r homeassistant/requirements.txt --use-deprecated=legacy-resolver
COPY requirements_all.txt home_assistant_frontend-* homeassistant/
RUN \
    if ls homeassistant/home_assistant_frontend*.whl 1> /dev/null 2>&1; then \
        pip3 install -v --no-cache-dir --no-index home-assistant-frontend-*.whl; \
    fi \
    && pip3 install -v --no-cache-dir --no-index --only-binary=:all: --find-links "${WHEELS_LINKS}" \
    -r homeassistant/requirements_all.txt --use-deprecated=legacy-resolver

## Setup Home Assistant Core
COPY . homeassistant/
RUN \
    pip3 install -v --no-binary=:all: --find-links "${WHEELS_LINKS}" \
    -e ./homeassistant --use-deprecated=legacy-resolver \
    && python3 -m compileall homeassistant/homeassistant

# Home Assistant S6-Overlay
COPY rootfs /

WORKDIR /config

This image starts setting up integrations before giving the same errors when it tries to set up domain file_upload.

@samitheberber, I had no idea linuxserver.io published a HA image. I grabbed the latest and can confirm it works. I just updated 6 months worth of HA 👍

[Jojo-1000]

The linuxserver.io image does not work for me, I got a warning that it has arm v7 and I need arm v6. What is your architecture that they are working on @Wetzel402 @samitheberber ? (uname -a)

[Wetzel402]

@Jojo-1000, my QNAP NAS is arm v7 so that makes sense then why its working for me. I wonder if the linuxserver.io image has "good" wheels but is using the wrong numpy version like you observed for the HA in house image.

[Gerigot]

TL;DR

I think the Github Actions are configured wrong to use armv7 instead of armv6 for the wheels packages. From the logs of armhf base core images (2022.12.9): Collecting numpy==1.23.2 #13 85.81 Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Long story (for troubleshooting in case that is not it)

I ran into this same issue on raspberry pi 1B rev2. It is very easy to reproduce with the built docker containers:

pi@raspberrypi:~ $ docker exec -ti homeassistant bash
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Illegal instruction (core dumped)
bash-5.1#

Because it was suggested that the wheels might be broken, I tried to only pull the wheels directly (my poor pi 1b does not like to compile or download the full requirements). So I created a Dockerfile which only installs the numpy package from the same repository as the Github Actions use and built the container.

Dockerfile:

ARG BUILD_FROM
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0@sha256:37b3385d3a10a30344ba73f4d5551b7ecb63aca1606b1586e67f34624f592d6c

# Synchronize with homeassistant/core.py:async_stop
ENV \
    S6_SERVICES_GRACETIME=220000

WORKDIR /usr/src

## Setup Home Assistant Core dependencies
#COPY requirements.txt homeassistant/
#COPY homeassistant/package_constraints.txt homeassistant/homeassistant/

# test install numpy package
RUN pip3 install --no-cache-dir --no-index --only-binary=:all: --find-links "https://wheels.home-assistant.io/musllinux/" --use-deprecated=legacy-resolver numpy

Build output
The goal was just to get a simple system where I may have a chance of manually building the single package or swap it to a different source. So I tried to run the import-crash code again and to my surprise it was working:

docker run -ti 0d13df698c3d bash
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
bash-5.1# python
Python 3.10.7 (main, Nov 24 2022, 12:49:35) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.max([1, 2, 3])
3
>>>

I tried to pull the exact same image that was built in the Actions for that version and compared the numpy versions. That is where I noticed that the Actions are pulling from armv7, while my working version uses armv6:

Not working:

Collecting numpy==1.23.2
#13 85.81   Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv7l.whl

Working:

Collecting numpy
  Downloading https://wheels.home-assistant.io/musllinux/numpy-1.23.2-cp310-cp310-musllinux_1_2_armv6l.whl (10.2 MB)

This is the same for all of the packages (although I did not test that every single one is working).

So I think this easily explains the illegal instruction errors, unless I am mistaken and the Actions are only used for CI and the uploaded docker images come from somewhere else. I am not very familiar with the Actions workflows, but this should be pretty easy to fix without any code changes and without rebuilding the wheels. I noticed that the architecture everywhere is set to arm in the logs, so maybe that is why it defaults to armv7?

I think you're right and probably the only thing that is missing in the configuration of the armhf config is an environment variable, QEMU_CPU=arm1176, that will force the build container to think is on armv6 so that the build will just work for armv6 and then in the build for v7 remove that variable.

I'm trying to build an image on my linux computer using docker buildx and send it to my raspberry pi zero w if it works I will update this message.

[HarvsG]

I think you're right and probably the only thing that is missing in the configuration of the armhf config is an environment variable, QEMU_CPU=arm1176, that will force the build container to think is on armv6 so that the build will just work for armv6 and then in the build for v7 remove that variable.

Presumably HA's actions are open source and we can submit a PR?

[Gerigot]

I think you're right and probably the only thing that is missing in the configuration of the armhf config is an environment variable, QEMU_CPU=arm1176, that will force the build container to think is on armv6 so that the build will just work for armv6 and then in the build for v7 remove that variable.

Presumably HA's actions are open source and we can submit a PR?

I am trying to understand what is all going on behind the scene because there are lots of repository involved in the build process (builder, base, ecc..).
I'm not an expert in GitHub actions and on what I could understand I think the base image for armhf and armv7 are the same so I'm not sure where I have to really check.

[Wetzel402]

I don't think there is an issue with the Alpine base image. I think the issue is specific to wheels when building HA core. The linuxserver.io image works for those of us with armv7 CPUs and uses different wheels than the OEM version. Separately @Jojo-1000, seems to have found another issue where the wrong numpy wheel is being collected for armv6.

[Gerigot]

So I made it work on my raspberry pi zero w with arch armv6l, I had to compile it from the source and from another pc because the raspberry was too loaded and it struggled to create the image.

steps that I did:
Checkout from github the core repository

//Modified the Dockerfile
FROM ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0

//added
ENV QEMU_CPU=arm1176 // this is used to made it download the armv6 wheels instead of armv7

//run on a faster machine
docker buildx create --platform linux/arm/v6 --use
docker buildx build --platform=linux/arm/v6 -o type=tar,dest=out.tar .

//send "out.tar" file over to the raspberry pi zero w and run
cat out.tar | docker import - custom/homeassistant

//custom/homeassistant is the image tag you can use whatever you want...

with this I could install the newer version

Unfortunately I couldn't find how and if it's possible to modify it in the main github actions to make it run on raspberry pi zero w directly

EDIT: it's not necessary to replace ${WHEELS_LINK} because is present in the armhf-homeassitant-base image where the build begins ;)

I'm digging more into the details of the base Image and what I noticed is that the layers of the armhf-homeassistant-base are full of
RUN |7 BUILD_ARCH=armhf QEMU_CPU=arm1176
I think that maybe there is a solution for my and many others problem with raspberry's v6 version

UPDATE:
I think I found a way to get the parameter I will make a pull request if it works.

[Gerigot]

I've finally created a pull request I've noted that the builder used, if the architecture is armhf, in the actions will provide with the QEMU_CPU=arm1176 as an ARG.
The only thing needed was to add an ARG QEMU_CPU in the dockerfile and that should trigger the right process.

I tested it by the same process

Added

ARG QEMU_CPU
at the beginning of the dockerfile after the ENV S6......

then build it with
docker buildx create --platform linux/arm/v6 --use //needed only one time
docker buildx build --platform=linux/arm/v6 --build-arg QEMU_CPU=arm1176 --build-arg BUILD_FROM=ghcr.io/home-assistant/armhf-homeassistant-base:2022.11.0 -t homeassistant-home .

hope it will be merged soon so that we will be able to use the offical image from the repository.

thank you @Jojo-1000 without your analysis I would never been able to identify what was going on

[Metaln00b]

It's gemerged, but what to do when you to compose on :dev? Is a flag necessary?

[dudusandi]

@Metaln00b

Already available:

No flags needed

https://github.com/home-assistant/core/pkgs/container/home-assistant/64544700?tag=2023.2.0.dev20230120

Thanks all guys for support !!

[Gerigot]

Yes it has been merged almost 20 hours ago but we had to wait until a build was triggered.

That happened 2 hours ago with a development version
2023.2.0.dev20230120 as said by @dudusandi

I will try it and I hope it will be soon present also in a stable version

@Metaln00b if you are using the :dev Image Tag you should stop your container delete the image and then pull the new one.
Otherwise you can use the 2023.2.0.dev20230120 image Tag so that you'll have the specific build that solves the problem.

[Gerigot]

I've tested it on my raspberry pi zero W and it works like a charm I'm not sure about ti other but I think this issue can be closed as resolved.

[jonnyparris]

Also running smoothly on my 1st gen Pi 👍🏾 Thanks for the fix!

…

On Sun, 22 Jan 2023, 11:56 Patrick, ***@***.***> wrote: I've tested it on my raspberry pi zero W and it works like a charm I'm not sure about ti other but I think this issue can be closed as resolved. — Reply to this email directly, view it on GitHub <#75142 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQ2QADWWCZ3UUUGQCRDM53WTUN7VANCNFSM53P7TIWA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[aSauerwein]

can confirm 2023.2.0.dev20230120 is running fine on 1st gen Pi

[HarvsG]

Ironically I have just yesterday migrated to a Pi Zero W 2 to overcome this issue 😞

[dudusandi]

@HarvsG Its much faster then rpi0w ?

[MichalW]

can confirm 2023.2.0.dev20230120 is running fine on 1st gen Pi

I can confirm that this version works on raspberry pi b+.
Thanks a lot for this fix

[HarvsG]

@HarvsG Its much faster then rpi0w ?

Yes and I managed to install full HAOS, which really simplified my config. So not wasted time.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.