[Snap] Snap builds failing in automation

[benfrancis]

STR:

• Push a commit to master

Expected:

• The snapcraft remote build servers are automatically triggered to generate builds for all architectures
• Snaps for all architectures are successfully built and released on the edge channel in the Snap Store

Actual:

• Builds fail in automation with the following error:

Building webthings-gateway
:: + craftctl default
:: ++ npm --version
:: /build/webthings-gateway/parts/webthings-gateway/run/build.sh: line 5: npm: command not found
:: + NPM_VERSION=
:: error:
:: Traceback (most recent call last):
::   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 111, in main
::     ret = CraftCtl.run(cmd, args)
::   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 45, in run
::     _client(cmd, args)
::   File "/snap/snapcraft/current/lib/python3.10/site-packages/craft_parts/ctl.py", line 98, in _client
::     raise RuntimeError(message)
:: RuntimeError: Failed to run the build script for part 'webthings-gateway'.
'override-build' in part 'webthings-gateway' failed with code 1.

Note: The snap was successfully building locally for the AMD64 architecture using the snapcraft command.

[benfrancis]

Although previously working, I just reproduced the same error when running $ snapcraft locally on AMD64:

/root/parts/webthings-gateway/run/build.sh: line 5: npm: command not found

After running $ snapcraft clean && snapcraft I then got a different error:

2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.574 :: + cp -av build /root/parts/webthings-gateway/install/lib/node_modules/webthings-gateway/
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.575 :: cp: 'build' and '/root/parts/webthings-gateway/install/lib/node_modules/webthings-gateway/build' are the same file
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.743 'override-build' in part 'webthings-gateway' failed with code 1.
2024-09-09 17:01:28.937 :: Review the scriptlet and make sure it's correct.
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753 Traceback (most recent call last):
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753   File "/snap/snapcraft/12503/lib/python3.10/site-packages/snapcraft/application.py", line 351, in main
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753     return app.run()
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753   File "/snap/snapcraft/12503/lib/python3.10/site-packages/snapcraft/application.py", line 179, in run
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753     return_code = super().run()
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753   File "/snap/snapcraft/12503/lib/python3.10/site-packages/craft_application/application.py", line 492, in run
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753     dispatcher = self._get_dispatcher()
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753   File "/snap/snapcraft/12503/lib/python3.10/site-packages/snapcraft/application.py", line 298, in _get_dispatcher
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753     raise errors.ClassicFallback()
2024-09-09 17:01:28.937 :: 2024-09-09 17:01:27.753 snapcraft.errors.ClassicFallback

As far as I can tell nothing changed on my end since adding the snapcraft.yaml.

@ogra1 Does this look like the same issue that was reproducing in automation and has it now propagated to my local version of snapcraft, or do you think it's a separate issue?

[benfrancis]

@ogra1 I noticed that my snapcraft.yaml had local modifications to change the source of each part to "." in order to use my local checkout of the source code (with no other local changes). If I revert that change back to the remote GitHub repo as the source it builds successfully.

That may explain why my local builds started failing, although I'm not sure why it fails because I thought that configuration was previously working and I need it to work to test local changes.

Am I missing something that needs resetting in the local build environment in order to build my local source code successfully?

[benfrancis]

I noticed that my snapcraft.yaml had local modifications to change the source of each part to "."

I just tried a fresh checkout of the GitHub repo and changed the sources to "." and it built fine. I can't figure out what local state was previously causing it to fail, but I can at least generate builds of my local repo now.

[benfrancis]

Update: Since Canonical's hotfitx for the remote builds landed, earlier in the week I manually triggered a re-build on snapcraft.io to see whether it had fixed the remote builds yet.

As far as I can tell the npm: command not found error no longer occurs, but the remote builds are now failing with other errors. On AMD64 it appears to build successfully but then suddenly fail after copying across the contents of the build directory to the snap. On ARM64 there appears to be a segfault whilst cross-compiling an npm dependency.

[ogra1]

i created #3169 for this

[benfrancis]

@ogra1 Thanks very much for that patch.

• Good news: Now building successfully in automation for AMD64
• Bad news: Builds are still failing on all other architectures, including ARM64:

:: > [email protected] install /build/webthings-gateway/parts/webthings-gateway/build/node_modules/segfault-handler
:: > node-gyp rebuild
::
:: make: Entering directory '/build/webthings-gateway/parts/webthings-gateway/build/node_modules/segfault-handler/build'
::   CXX(target) Release/obj.target/segfault-handler/src/segfault-handler.o
:: In file included from ../../nan/nan.h:60,
::                  from ../src/segfault-handler.cpp:2:
:: /root/.cache/node-gyp/10.24.1/include/node/node.h:573:7: warning: cast between incompatible function types from ‘void (*)(Nan::ADDON_REGISTER_FUNCTION_ARGS_TYPE)’ {aka ‘void (*)(v8::Local<v8::Object>)’} to ‘node::addon_register_func’ {aka ‘void (*)(v8::Local<v8::Object>, v8::Local<v8::Value>, void*)’} [-Wcast-function-type]
::   573 |       (node::addon_register_func) (regfunc),                          \
::       |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:: /root/.cache/node-gyp/10.24.1/include/node/node.h:607:3: note: in expansion of macro ‘NODE_MODULE_X’
::   607 |   NODE_MODULE_X(modname, regfunc, NULL, 0)  // NOLINT (readability/null_usage)
::       |   ^~~~~~~~~~~~~
:: ../src/segfault-handler.cpp:346:3: note: in expansion of macro ‘NODE_MODULE’
::   346 |   NODE_MODULE(segfault_handler, init)
::       |   ^~~~~~~~~~~
:: In file included from /root/.cache/node-gyp/10.24.1/include/node/node.h:63,
::                  from ../../nan/nan.h:60,
::                  from ../src/segfault-handler.cpp:2:
:: /root/.cache/node-gyp/10.24.1/include/node/v8.h: In instantiation of ‘void v8::PersistentBase<T>::SetWeak(P*, typename v8::WeakCallbackInfo<P>::Callback, v8::WeakCallbackType) [with P = node::ObjectWrap; T = v8::Object; typename v8::WeakCallbackInfo<P>::Callback = void (*)(const v8::WeakCallbackInfo<node::ObjectWrap>&)]’:
:: /root/.cache/node-gyp/10.24.1/include/node/node_object_wrap.h:84:25:   required from here
:: /root/.cache/node-gyp/10.24.1/include/node/v8.h:9502:16: warning: cast between incompatible function types from ‘v8::WeakCallbackInfo<node::ObjectWrap>::Callback’ {aka ‘void (*)(const v8::WeakCallbackInfo<node::ObjectWrap>&)’} to ‘Callback’ {aka ‘void (*)(const v8::WeakCallbackInfo<void>&)’} [-Wcast-function-type]
::  9502 |                reinterpret_cast<Callback>(callback), type);
::       |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:: /root/.cache/node-gyp/10.24.1/include/node/v8.h: In instantiation of ‘void v8::PersistentBase<T>::SetWeak(P*, typename v8::WeakCallbackInfo<P>::Callback, v8::WeakCallbackType) [with P = Nan::ObjectWrap; T = v8::Object; typename v8::WeakCallbackInfo<P>::Callback = void (*)(const v8::WeakCallbackInfo<Nan::ObjectWrap>&)]’:
:: ../../nan/nan_object_wrap.h:64:57:   required from here
:: /root/.cache/node-gyp/10.24.1/include/node/v8.h:9502:16: warning: cast between incompatible function types from ‘v8::WeakCallbackInfo<Nan::ObjectWrap>::Callback’ {aka ‘void (*)(const v8::WeakCallbackInfo<Nan::ObjectWrap>&)’} to ‘Callback’ {aka ‘void (*)(const v8::WeakCallbackInfo<void>&)’} [-Wcast-function-type]
::   SOLINK_MODULE(target) Release/obj.target/segfault-handler.node
::   COPY Release/segfault-handler.node
:: make: Leaving directory '/build/webthings-gateway/parts/webthings-gateway/build/node_modules/segfault-handler/build'
::
:: > [email protected] postinstall /build/webthings-gateway/parts/webthings-gateway/build/node_modules/@babel/polyfill/node_modules/core-js
:: > node -e "try{require('./postinstall')}catch(e){}"
::
:: �[96mThank you for using core-js (�[94m https://github.com/zloirock/core-js �[96m) for polyfilling JavaScript standard library!�[0m
::
:: �[96mThe project needs your help! Please consider supporting of core-js on Open Collective or Patreon: �[0m
:: �[96m>�[94m https://opencollective.com/core-js �[0m
:: �[96m>�[94m https://www.patreon.com/zloirock �[0m
::
:: �[96mAlso, the author of core-js (�[94m https://github.com/zloirock �[96m) is looking for a good job -)�[0m
::
::
:: > [email protected] postinstall /build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle
:: > node lib/install.js
::
[30/Sep/2024:13:48:52 +0000] "CONNECT raw.githubusercontent.com:443 HTTP/1.1" 200 725197 "-" "-"
:: Command failed: /build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle/vendor/gifsicle --version
:: /build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle/vendor/gifsicle: 1: �ELF����: not found
:: /build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle/vendor/gifsicle: 2: @8: not found
:: /build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle/vendor/gifsicle: 3: Syntax error: ")" unexpected
::
::
:: gifsicle pre-build test failed
:: compiling from source
:: Error: Command failed: /bin/sh -c autoreconf -ivf
:: autoreconf: export WARNINGS=
:: autoreconf: Entering directory '.'
:: autoreconf: configure.ac: not using Gettext
:: autoreconf: running: aclocal --force
:: Can't exec "aclocal": No such file or directory at /usr/share/autoconf/Autom4te/FileUtils.pm line 274.
:: autoreconf: error: aclocal failed with exit status: 2
::
::
::     at Promise.all.then.arr (/build/webthings-gateway/parts/webthings-gateway/build/node_modules/bin-build/node_modules/execa/index.js:231:11)
::     at process._tickCallback (internal/process/next_tick.js:68:7)
:: npm WARN rollback Rolling back [email protected] failed (this is probably harmless): ENOTEMPTY: directory not empty, rmdir '/build/webthings-gateway/parts/webthings-gateway/build/node_modules/gifsicle'
:: npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
:: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"arm64"})
::
:: npm ERR! code ELIFECYCLE
:: npm ERR! errno 1
:: npm ERR! [email protected] postinstall: `node lib/install.js`
:: npm ERR! Exit status 1
:: npm ERR!
:: npm ERR! Failed at the [email protected] postinstall script.
:: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
::
:: npm ERR! A complete log of this run can be found in:
:: npm ERR!     /root/.npm/_logs/2024-09-30T13_48_57_113Z-debug.log
'override-build' in part 'webthings-gateway' failed with code 1.
Review the scriptlet and make sure it's correct.
Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240930-134533.688707.log'
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 266, in run
    self.build()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 250, in build
    self.run_build_command(["snapcraft"], cwd=output_path, env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/operation.py", line 62, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 718, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-jammy-arm64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/lpbuildd-git-proxy', '--env', 'SNAPPY_STORE_NO_CDN=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-92543604", "build-request-timestamp": "2024-09-30T13:41:17Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/d6fc6b405a4a45eb3b743b2b7e8b0f36/+build/2613474"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'SNAPCRAFT_BUILD_FOR=arm64', '--', '/bin/sh', '-c', 'cd /build/webthings-gateway && linux64 snapcraft']' returned non-zero exit status 1.
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=jammy --arch=arm64 SNAPBUILD-2613474
Scanning for processes to kill in build SNAPBUILD-2613474

[benfrancis]

Additionally, the AMD64 snap appears to fail to start, see #3171