Build fails with WASI SDK 21 #113516

proman2019 · 2023-12-27T15:54:10Z

Bug report

Bug description:

Hello!
when executing
python3 Tools/wasm/wasi.py make-host
on ubuntu 22.04 with wasi-ld version: LLD 17.0.6 (tried earlier versions too)
Errors are as follows:

/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/_testimportmultiple.o   -o Modules/_testimportmultiple.cpython-313d-wasm32-wasi.so
/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/_testmultiphase.o   -o Modules/_testmultiphase.cpython-313d-wasm32-wasi.so
/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/_testsinglephase.o   -o Modules/_testsinglephase.cpython-313d-wasm32-wasi.so
/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/xxlimited.o   -o Modules/xxlimited.cpython-313d-wasm32-wasi.so
/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/xxlimited_35.o   -o Modules/xxlimited_35.cpython-313d-wasm32-wasi.so
wasm-ld: error: unknown argument: -Wl,--stack-first
wasm-ld: error: unknown argument: -Wl,--stack-first
wasm-ld: error: unknown argument: -Wl,--initial-memory=10485760
wasm-ld: error: unknown argument: -Wl,--initial-memory=10485760
wasm-ld: error: unknown argument: -Wl,--stack-first
wasm-ld: error: unknown argument: -Wl,--initial-memory=10485760
wasm-ld: error: unknown argument: -Wl,--stack-first
wasm-ld: error: unknown argument: -Wl,--initial-memory=10485760
make: *** [Makefile:3134: Modules/_testimportmultiple.cpython-313d-wasm32-wasi.so] Error 1

if "-Wl" flag removed then clang compiler fails.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

The text was updated successfully, but these errors were encountered:

brettcannon · 2024-01-03T23:05:00Z

What version of the WASI-SDK did you use?

proman2019 · 2024-01-03T23:22:41Z

Hi - I believe I was failing with wasi-sdk-20. It builds fine for wasi-sdk 16. From: Brett Cannon ***@***.***> Sent: Wednesday, January 3, 2024 6:05 PM To: python/cpython ***@***.***> Cc: Romanczyk, Piotr ***@***.***>; Author ***@***.***> Subject: [External] Re: [python/cpython] wasi build appears to fail (Issue #113516) What version of the WASI-SDK did you use? - Reply to this email directly, view it on GitHub<#113516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMBG4R5W4VEQWQCU6ZJIGMDYMXP2NAVCNFSM6AAAAABBEQFICCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZWGA4DSMZQGA>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> **** Resideo WARNING: This email was sent from outside of Resideo and the sender cannot be verified. Use CAUTION before opening file attachments, links, or replying to the email. For additional information please contact the Resideo Service Desk ****

brettcannon · 2024-01-04T18:28:20Z

It builds fine for me under WASI-SDK 20.

Can you share all 4 commands you used w/ wasi.py? Do you get a different result if you use wasi.py build?

proman2019 · 2024-01-06T20:43:40Z

Sorry for the delay... I've used exact same commands as listed on ./Tools/wasm README.md for WASI builds: python Tools/wasm/wasi.py configure-build-python python Tools/wasm/wasi.py make-build-python python Tools/wasm/wasi.py configure-host python Tools/wasm/wasi.py make-host I am a bit confused about the wasi-sdk versions - when I get wasi-sdk-16 tagged source I get this: ./wasi-ld -version LLD 14.0.4 The above version builds just fine from cpython main I thought it was v 20 failing and I can't tell exactly which wasi-sdk version is the one failing in my setup but the one that fails has this version of lld installed: ./wasi-ld -version LLD 17.0.6 Hope that helps From: Brett Cannon ***@***.***> Sent: Thursday, January 4, 2024 1:29 PM To: python/cpython ***@***.***> Cc: Romanczyk, Piotr ***@***.***>; Author ***@***.***> Subject: [External] Re: [python/cpython] wasi build appears to fail (Issue #113516) It builds fine for me under WASI-SDK 20. Can you share all 4 commands you used w/ wasi.py? Do you get a different result if you use wasi.py build? - Reply to this email directly, view it on GitHub<#113516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMBG4R5S7YJEPGJKURS22FDYM3YE7AVCNFSM6AAAAABBEQFICCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGU3DSNZSGQ>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> **** Resideo WARNING: This email was sent from outside of Resideo and the sender cannot be verified. Use CAUTION before opening file attachments, links, or replying to the email. For additional information please contact the Resideo Service Desk ****

brettcannon · 2024-01-08T19:53:32Z

Unfortunately w/o knowing what WASI-SDK version is having issues I don't know how to help since I don't have these failures under WASI-SDK 20. If it's WASI-SDK 21 (which we have not tried to support yet), then that might be the cause, but then that means they may have broken something.

brettcannon · 2024-01-08T19:56:32Z

I opened WebAssembly/wasi-sdk#372 to see if there's an easy way to find this out.

proman2019 · 2024-01-08T19:59:24Z

Understood - no problem. I've got it to build and working with wasi-sdk-16.0 - I will try wasi-sdk-20.0 later this week. Thanks! From: Brett Cannon ***@***.***> Sent: Monday, January 8, 2024 2:54 PM To: python/cpython ***@***.***> Cc: Romanczyk, Piotr ***@***.***>; Author ***@***.***> Subject: [External] Re: [python/cpython] wasi build appears to fail (Issue #113516) Unfortunately w/o knowing what WASI-SDK version is having issues I don't know how to help since I don't have these failures under WASI-SDK 20. If it's WASI-SDK 21 (which we have not tried to support yet), then that might be the cause, but then that means they may have broken something. - Reply to this email directly, view it on GitHub<#113516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMBG4R53OOAI34QHAPIP6Y3YNRFEPAVCNFSM6AAAAABBEQFICCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRG4ZDQNJSGQ>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> **** Resideo WARNING: This email was sent from outside of Resideo and the sender cannot be verified. Use CAUTION before opening file attachments, links, or replying to the email. For additional information please contact the Resideo Service Desk ****

proman2019 · 2024-01-08T19:59:58Z

Sounds good! Thanks. From: Brett Cannon ***@***.***> Sent: Monday, January 8, 2024 2:57 PM To: python/cpython ***@***.***> Cc: Romanczyk, Piotr ***@***.***>; Author ***@***.***> Subject: [External] Re: [python/cpython] wasi build appears to fail (Issue #113516) I opened WebAssembly/wasi-sdk#372<WebAssembly/wasi-sdk#372> to see if there's an easy way to find this out. - Reply to this email directly, view it on GitHub<#113516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMBG4R6E32SP5HZO6PA75U3YNRFPXAVCNFSM6AAAAABBEQFICCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRG4ZTEMRYGY>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>> **** Resideo WARNING: This email was sent from outside of Resideo and the sender cannot be verified. Use CAUTION before opening file attachments, links, or replying to the email. For additional information please contact the Resideo Service Desk ****

encukou · 2024-02-02T15:01:46Z

I will close the issue. If you have more information, please comment here or file a new one.

kushaldas · 2024-02-12T08:14:03Z

Fails to build using wasi-sdk-21.

/opt/wasi-sdk/bin/clang -c -fno-strict-overflow -Wsign-compare -g -Og -Wall    -std=c11 -Wextra -Wno-unused-parameter -Wno-int-conversion -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include    -DPy_BUILD_CORE -o Python/frozen.o ../../Python/frozen.c
_PYTHON_HOSTRUNNER='/home/kdas/.wasmtime/bin/wasmtime run --wasm max-wasm-stack=8388608 --wasm threads=y --wasi threads=y --dir /home/kdas/code/cpython::/ --env PYTHONPATH=/cross-build/wasm32-wasi/build/lib.wasi-wasm32-3.13-pydebug' _PYTHON_PROJECT_BASE=/home/kdas/code/cpython/cross-build/wasm32-wasi _PYTHON_HOST_PLATFORM=wasi-wasm32 PYTHONPATH=/home/kdas/code/cpython/cross-build/wasm32-wasi/build/lib.wasi-wasm32-3.13-pydebug:../../Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_d_wasi_wasm32-wasi /home/kdas/code/cpython/cross-build/build/python -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
/opt/wasi-sdk/bin/wasm-ld   -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760   Modules/_testimportmultiple.o   -o Modules/_testimportmultiple.cpython-313d-wasm32-wasi.so
wasm-ld: error: unknown argument: -Wl,--stack-first
wasm-ld: error: unknown argument: -Wl,--initial-memory=10485760
make: *** [Makefile:3183: Modules/_testimportmultiple.cpython-313d-wasm32-wasi.so] Error 1
make: *** Waiting for unfinished jobs....

kushaldas · 2024-02-12T08:18:13Z

Filed WebAssembly/wasi-sdk#385 asking for help/input.

brettcannon · 2024-02-14T02:49:41Z

There may have been a change in clang 17 (which WASI SDK 21 uses) where LDFLAGS goes straight to the linker, which means -Wl, is unnecessary as that parts meant for the compiler to pass the arg to the linker. So maybe that line in configure.ac just needs the compiler-specific part stripped and the rest left in?

brettcannon · 2024-02-15T00:10:09Z

OK, I think I have figured it out. 😮‍💨

If you look at the failure it corresponds to trying to build _testimportmultiple. OK, why is that in any way special in how it gets built? Well, it turns out it requires dlopen() which is new in WASI SDK 21:

cpython/configure.ac

Line 7458 in 3e7b7df

    
           PY_STDLIB_MOD([_testimportmultiple], [test "$TEST_MODULES" = yes], [test "$ac_cv_func_dlopen" = yes])

So now we know why this problem is only showing up for WASI SDK 21, but why the call to wasm-ld w/ LDFLAGS_NODIST that's only meant for clang?

cpython/configure.ac

Lines 2238 to 2240 in 3e7b7df

    
               dnl increase initial memory and stack size, move stack first 
        
               dnl https://github.com/WebAssembly/wasi-libc/issues/233 
        
               AS_VAR_APPEND([LDFLAGS_NODIST], [" -z stack-size=524288 -Wl,--stack-first -Wl,--initial-memory=10485760"])

If you look at the generated Makefile, you will see a line for building _testimportmultiple:

Modules/_testimportmultiple$(EXT_SUFFIX):  Modules/_testimportmultiple.o; $(BLDSHARED)  Modules/_testimportmultiple.o $(MODULE__TESTIMPORTMULTIPLE_LDFLAGS)  -o Modules/_testimportmultiple$(EXT_SUFFIX)

So it seems BLDSHARED is where wasm-ld is suddenly appearing where clang should be. So why is that happening? If it isn't explicitly set it turns out to inherit from LDSHARED:

cpython/configure.ac

Line 3317 in 4684301

BLDSHARED=${BLDSHARED-$LDSHARED}

And why does this matter?

cpython/Tools/wasm/wasi.py

Line 168 in 4684301

"LDSHARED": "wasm-ld", "AR": "llvm-ar", "RANLIB": "ranlib"}

And there's the wasm-ld! This was written this way thanks to where I copied how to set these environment variables:

cpython/Tools/wasm/wasi-env

Line 58 in 4684301

LDSHARED="${WASI_SDK_PATH}/bin/wasm-ld"

Dropping that environment variable gets compiling to work! I will get a PR pulled together for this.

But then test_importlib fails. ☹️ It probably has to do w/ dlopen() now existing.

brettcannon · 2024-02-15T00:32:34Z

FYI the fix for wasi-env will need to be backported.

…ythonGH-115495)

) (GH-115496)

…ythonGH-115495) (pythonGH-115496) (cherry picked from commit 0e4f73b) Co-authored-by: Brett Cannon <[email protected]>

…H-115495) (GH-115496) (GH-115497) [3.12] GH-113516: don't set `LDSHARED` when building for WASI (GH-115495) (GH-115496) (cherry picked from commit 0e4f73b) Co-authored-by: Brett Cannon <[email protected]>

brettcannon · 2024-02-15T01:35:32Z

The fixes to get the builds to work are now in, so getting test_importlib to pass is next.

proman2019 · 2024-02-15T02:03:05Z

Awesome! Great investigation! I am looking forward to test that build.

…

________________________________ From: Brett Cannon ***@***.***> Sent: Wednesday, February 14, 2024 8:35:45 PM To: python/cpython ***@***.***> Cc: Romanczyk, Piotr ***@***.***>; Author ***@***.***> Subject: [External] Re: [python/cpython] Build fails with WASI SDK 21 (Issue #113516) The fixes to get the builds to work are now in, so getting test_importlib to pass is next. — Reply to this email directly, view it on GitHub<#113516 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AMBG4R7T4EED26VWVYWJ26LYTVQ7DAVCNFSM6AAAAABBEQFICCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBVGIZDKNJXGA>. You are receiving this because you authored the thread.Message ID: ***@***.***> **** Resideo WARNING: This email was sent from outside of Resideo and the sender cannot be verified. Use CAUTION before opening file attachments, links, or replying to the email.****

brettcannon · 2024-02-26T23:32:10Z

I opened #115979 to track the test_importlib failures. As such, I'm closing this issue as the builds do now succeed.

proman2019 added the type-bug An unexpected behavior, bug, or error label Dec 27, 2023

Eclips4 added the OS-wasi label Dec 27, 2023

brettcannon self-assigned this Jan 3, 2024

brettcannon mentioned this issue Jan 8, 2024

How to tell what version of the WASI-SDK is installed? WebAssembly/wasi-sdk#372

Closed

encukou closed this as completed Feb 2, 2024

kushaldas reopened this Feb 12, 2024

kushaldas changed the title ~~wasi build appears to fail~~ Build fails with wasi-sdk-21 Feb 12, 2024

brettcannon added a commit to brettcannon/cpython that referenced this issue Feb 15, 2024

pythonGH-113516: don't set LDSHARED when building for WASI

98ad9c1

bedevere-app bot mentioned this issue Feb 15, 2024

GH-113516: don't set LDSHARED when building for WASI #115495

Merged

brettcannon added 3.11 only security fixes 3.12 bugs and security fixes 3.13 bugs and security fixes labels Feb 15, 2024

brettcannon changed the title ~~Build fails with wasi-sdk-21~~ Build fails with WASI SDK 21 Feb 15, 2024

brettcannon added a commit that referenced this issue Feb 15, 2024

GH-113516: don't set LDSHARED when building for WASI (GH-115495)

18343c0

bedevere-app bot mentioned this issue Feb 15, 2024

[3.12] GH-113516: don't set LDSHARED when building for WASI (GH-115495) #115496

Merged

brettcannon added a commit to brettcannon/cpython that referenced this issue Feb 15, 2024

[3.12] pythonGH-113516: don't set LDSHARED when building for WASI (p…

51ae736

…ythonGH-115495)

brettcannon added a commit that referenced this issue Feb 15, 2024

[3.12] GH-113516: don't set LDSHARED when building for WASI (GH-115495

0e4f73b

) (GH-115496)

bedevere-app bot mentioned this issue Feb 15, 2024

[3.11] [3.12] GH-113516: don't set LDSHARED when building for WASI (GH-115495) (GH-115496) #115497

Merged

brettcannon closed this as completed Feb 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build fails with WASI SDK 21 #113516

Build fails with WASI SDK 21 #113516

proman2019 commented Dec 27, 2023 •

edited by bedevere-app bot

Loading

brettcannon commented Jan 3, 2024

proman2019 commented Jan 3, 2024 via email

brettcannon commented Jan 4, 2024

proman2019 commented Jan 6, 2024 via email

brettcannon commented Jan 8, 2024

brettcannon commented Jan 8, 2024

proman2019 commented Jan 8, 2024 via email

proman2019 commented Jan 8, 2024 via email

encukou commented Feb 2, 2024

kushaldas commented Feb 12, 2024

kushaldas commented Feb 12, 2024

brettcannon commented Feb 14, 2024

brettcannon commented Feb 15, 2024 •

edited

Loading

brettcannon commented Feb 15, 2024

brettcannon commented Feb 15, 2024

proman2019 commented Feb 15, 2024 via email

brettcannon commented Feb 26, 2024

Build fails with WASI SDK 21 #113516

Build fails with WASI SDK 21 #113516

Comments

proman2019 commented Dec 27, 2023 • edited by bedevere-app bot Loading

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

brettcannon commented Jan 3, 2024

proman2019 commented Jan 3, 2024 via email

brettcannon commented Jan 4, 2024

proman2019 commented Jan 6, 2024 via email

brettcannon commented Jan 8, 2024

brettcannon commented Jan 8, 2024

proman2019 commented Jan 8, 2024 via email

proman2019 commented Jan 8, 2024 via email

encukou commented Feb 2, 2024

kushaldas commented Feb 12, 2024

kushaldas commented Feb 12, 2024

brettcannon commented Feb 14, 2024

brettcannon commented Feb 15, 2024 • edited Loading

brettcannon commented Feb 15, 2024

brettcannon commented Feb 15, 2024

proman2019 commented Feb 15, 2024 via email

brettcannon commented Feb 26, 2024

proman2019 commented Dec 27, 2023 •

edited by bedevere-app bot

Loading

brettcannon commented Feb 15, 2024 •

edited

Loading