Broken core20 armhf builds on launchpad

[Hook25]

Bug Description

When building on launchpad via remote-build a core20 snap for armhf, the traceback in this output is printed. This happens deterministically in our daily builds that are dispatched via snapcraft 7.

To Reproduce

I think this should happen building any core20 snap via remote-build (as there is nothing special in what we do and if fails to run as soon as it is picked up for any core20 armhf snap we build)

git clone https://github.com/canonical/checkbox
cd checkbox/checkbox-snap
./prepare.sh series_classic20
snap refresh snapcraft --channel=7
snapcraft remote-build --build-for=armhf

Environment

Workflow: https://github.com/canonical/checkbox/blob/main/.github/workflows/checkbox-snap-daily-builds.yml

snapcraft.yaml

name: checkbox
summary: Checkbox test runner
description: |
  Checkbox is a flexible test automation software.
  It’s the main tool used in Ubuntu Certification program.
adopt-info: version-calculator
grade: stable
confinement: classic

base: core20

apps:
  checkbox-cli:
    command-chain: [bin/wrapper_local]
    command: bin/checkbox-cli-wrapper
  test-runner:
    command-chain: [bin/wrapper_local]
    command: bin/test-runner
  sru:
    command-chain: [bin/wrapper_local]
    command: bin/sru
  client-cert-iot-server:
    command-chain: [bin/wrapper_local]
    command: bin/client-cert-iot-server
  odm-certification:
    command-chain: [bin/wrapper_local]
    command: bin/odm-certification-app
  configure:
    command: bin/configure
  agent:
    command-chain: [bin/wrapper_local]
    command: bin/checkbox-cli-wrapper run-agent
    daemon: simple
    restart-condition: always
    restart-delay: 1s
  shell:
    command-chain: [bin/wrapper_local]
    command: bin/shell-wrapper

passthrough:
  hooks:
    configure:
      command-chain: [bin/wrapper_local]

parts:
  version-calculator:
    plugin: dump
    source: .
    override-pull: |
      snapcraftctl pull
      # version.txt created by prepare.sh
      export version=`cat $SNAPCRAFT_PART_SRC/version.txt`
      [ $version ] || exit 1
      snapcraftctl set-version $version
    stage:
      - version.txt
  launchers:
    plugin: dump
    source: launchers/
    organize:
      '*': bin/
  config-variables:
    plugin: dump
    source: .
    stage: [config_vars]
  workaround-automated-review-issue:
    plugin: nil
    override-build: |
      # This is a workaround for an upstream bug in the automated snap review tool
      # remove this once this issue is resolved:
      # https://bugs.launchpad.net/review-tools/+bug/2049093
      # Note: this cats /dev/urandom because the squashfs will collapse any
      #       predictable output to less than the desired size making this
      #       ineffective
      echo "This file is a workaround for a bug in the automated snap review tool" > $SNAPCRAFT_PART_INSTALL/size_workaround
      echo "this only contains random bytes to pad the snap to 16kb" >> $SNAPCRAFT_PART_INSTALL/size_workaround
      echo "see: https://bugs.launchpad.net/review-tools/+bug/2049093" >>  $SNAPCRAFT_PART_INSTALL/size_workaround
      head -c 16384 /dev/urandom >> $SNAPCRAFT_PART_INSTALL/size_workaround

Relevant log output

Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/snapcraft_legacy/__init__.py", line 352, in <module>
    from snapcraft_legacy._store import (  # noqa
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/snapcraft_legacy/_store.py", line 30, in <module>
    import craft_store
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/craft_store/__init__.py", line 23, in <module>
    from .auth import Auth
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/craft_store/auth.py", line 33, in <module>
    from keyring.backends import SecretService
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/keyring/backends/SecretService.py", line 16, in <module>
    import secretstorage
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/secretstorage/__init__.py", line 13, in <module>
    from secretstorage.collection import Collection, create_collection, \
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/secretstorage/collection.py", line 24, in <module>
    from secretstorage.item import Item
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/secretstorage/item.py", line 17, in <module>
    from secretstorage.util import DBusAddressWrapper, \
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/secretstorage/util.py", line 23, in <module>
    from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in <module>
    from cryptography.hazmat.primitives.ciphers.base import (
  File "/snap/snapcraft/12614/lib/python3.10/site-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in <module>
    from cryptography.hazmat.bindings._rust import openssl as rust_openssl
RuntimeError: OpenSSL 3.0's legacy provider failed to load. This is a fatal error by default, but cryptography supports running without legacy algorithms by setting the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY. If you did not expect this error, you have likely made a mistake with your OpenSSL configuration.

Additional context

See it failing here: https://github.com/canonical/checkbox/actions/workflows/daily-builds.yml
Relevant logs:
checkbox_armhf.txt
checkbox_armhf.4.txt
checkbox_armhf.3.txt
checkbox_armhf.2.txt
checkbox_armhf.1.txt

[Saviq]

Can confirm for my base: core20 snaps, seems to have started over the last few days.

No remote-build involved, just plain LP builds:

https://launchpad.net/~mir-team/+snap/ubuntu-frame-osk-20-beta

[mr-cal]

This is also failing when snapcraft builds its own wheels, which means we have a way to debug this locally:

#5061

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3514.

This message was autogenerated

[mr-cal]

Plan:

• Pull in @dariuszd21's fix in fix: correctly set PIP_NO_BINARY #5061 to reproduce locally
• Look at crypto 43 changelog and issues
• Try building with OPENSSL_STATIC=1 (https://cryptography.io/en/latest/installation/#static-wheels)
• Try adding openssl as a stage-package, based on rust bindings (/snap/snapcraft/current/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so)

[mr-cal]

We have a fix and should be able to get snapcraft released this week or next week.

Thanks for the report!

[mr-cal]

Fixed via #5084

[mr-cal]

This is in 8.4.2, which is currently in the candidate channel. I'm planning to release to stable on 2024-Oct-08.

[mr-cal]

This has also been fixed in snapcraft 7.5.8 with #5105.