Support for FreeBSD-arm64 #71338
Comments
ghost
added
the
|
OSX/BSD builds use the target system's libunwind: which makes it add only |
|
Yup, I think we just need to update the list of compatible context registers for freebsd-arm64 to make it work. |
|
If that would only make sense to me I would maybe help more, still learing on the fly. I found #40149 and doing quick change: diff --git a/src/coreclr/pal/src/include/pal/context.h b/src/coreclr/pal/src/include/pal/context.h
index fec5139fd5e..2cc4941d653 100644
--- a/src/coreclr/pal/src/include/pal/context.h
+++ b/src/coreclr/pal/src/include/pal/context.h
@@ -407,7 +407,7 @@ inline void *FPREG_Xstate_Ymmh(const ucontext_t *uc)
#if defined(HOST_ARM64)
-#ifndef TARGET_OSX
+#if !defined(TARGET_OSX) && !defined(TARGET_FREEBSD)
#define MCREG_X0(mc) ((mc).regs[0])
#define MCREG_X1(mc) ((mc).regs[1])
@@ -482,7 +482,7 @@ const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
}
-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX
#define MCREG_X0(mc) ((mc)->__ss.__x[0])
#define MCREG_X1(mc) ((mc)->__ss.__x[1])native compile fail with: Is this the context registers that needs to be updated (looked at the code and there's no ARM64 for any of the targets)? |
|
@sec, what are the detected features (the HAVE_xxx) for the FreeBSD arm64 build? When you do a clean build, it lists them to the console. From those, I should be able to tell where to update the context registers extraction. |
|
@janvorli There you go: edit: also attached full log from build attempt. |
|
It seems to me that your last change was correct. But you also need to add #elif (defined(HOST_UNIX) && defined(HOST_ARM64)) before the #else with #error that fires. In that #elif, you'd add: ASSIGN_REG(Pc) \
ASSIGN_REG(Sp) \
ASSIGN_REG(Fp) \
ASSIGN_REG(Lr) \
ASSIGN_REG(X19) \
ASSIGN_REG(X20) \
ASSIGN_REG(X21) \
ASSIGN_REG(X22) \
ASSIGN_REG(X23) \
ASSIGN_REG(X24) \
ASSIGN_REG(X25) \
ASSIGN_REG(X26) \
ASSIGN_REG(X27) \
ASSIGN_REG(X28)I hope I got it right. |
|
@janvorli Tried that, it failed with bunch of errors for every register: Leaving this empty, will go on but then fail again with similar errors. If I'm looking at this correctly, I think it's missing some #def for HOST_ARM64 and TARGET_FREEBSD inside And also there's again I wodner, if there's libunwind used (which state FreeBSD is supported under x64 and arm64) - shouldn't this be used (if I assume this code here is about that). As I saw @sdmaclea did the OSX part of changes - maybe some hints? :) Just to make sure, this is native build under ARM64 FreeBSD host. |
|
Ah, I haven't noticed that your change around line 490 was like this: -#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSXThat way, you have eliminated both definitions of the MCREG_xxx for arm64. What does the ucontext_t struct look like on FreeBSD arm64? We'll either need to define the MCREG_xxx for the FreeBSD separately or reuse the one for Linux / OSX if it happens to be the same. |
|
Ah, found it in FreeBSD sources. So it is different and we will need to have another #define MCREG_X21(mc) (mc.mc_gpregs.gp_x[21])
#define MCREG_X22(mc) (mc.mc_gpregs.gp_x[22])
#define MCREG_X23(mc) (mc.mc_gpregs.gp_x[23])
#define MCREG_X24(mc) (mc.mc_gpregs.gp_x[24])
#define MCREG_X25(mc) (mc.mc_gpregs.gp_x[25])
#define MCREG_X26(mc) (mc.mc_gpregs.gp_x[26])
#define MCREG_X27(mc) (mc.mc_gpregs.gp_x[27])
#define MCREG_X28(mc) (mc.mc_gpregs.gp_x[28])
#define MCREG_Fp(mc) (mc.mc_gpregs.gp_fp)
#define MCREG_Lr(mc) (mc.mc_gpregs.gp_lr)
#define MCREG_Sp(mc) (mc.mc_gpregs.gp_sp)
#define MCREG_Pc(mc) (mc.mc_gpregs.gp_pc)As for the floating point access we will also need an extra code here. Take the following with a grain of salt, it might not be exactly right or might need extra change at another place too: inline
struct fpregs* GetNativeSigSimdContext(native_context_t *mc)
{
return &(mc->mc_fpregs);
}
inline
const struct fpregs* GetConstNativeSigSimdContext(const native_context_t *mc)
{
return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
} |
|
@janvorli Thanks, I was looking at the same files and was about to post :) Managed to get past some errors, but now a lot of: I haven't saw any existing I'm starting to think does this road make sense. Shouldn't this reuse existing ARM64 code and/or use machine specific libunwind for the rest? |
|
We cannot reuse the code, because the structure of the |
|
Can you please share the log from the clean build after you've applied my previous fix? It is not clear to me how come we are compiling code using the BSDREG_xxx stuff when the |
|
build_log.txt @janvorli Here's build output and edit: accoring to CMakeOutput.txt, BSD_REG check passed: Run Build Command(s):/usr/local/bin/gmake -f Makefile cmTC_c2fef/fast && /usr/local/bin/gmake -f CMakeFiles/cmTC_c2fef.dir/build.make CMakeFiles/cmTC_c2fef.dir/build edit 2: added CMakeFiles output, maybe it will help also |
|
Ah, I've forgotten that the And then in if (CLR_CMAKE_HOST_ARCH_AMD64)
set(BSD_REGS_STYLE "((reg).r_##rr)")
elseif(CLR_CMAKE_HOST_ARCH_ARM64)
set(BSD_REGS_STYLE "((reg).rr)")
else()
message(FATAL_ERROR "Unknown FreeBSD architecture")
endif() |
|
Thanks, will try that - any idea what about missing MCREG_Pc and MCREG_Fp - can't see those inside ucontext struct (or the name is not close to that) ? |
|
Ah, sorry, the Fp is X29, so it should be #define MCREG_Fp(mc) (mc.mc_gpregs.gp_x[29])And the MCREG_Pc - hmm, that's strange. I can see that there is gp_lr and gp_elr. I think that maybe the gp_elr is in fact the Pc. Also related to the x29 above, for the most recent change I've suggested, this is wrong: #define BSDREG_X29(reg) BSD_REGS_STYLE(reg,X[29],x[29])
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,Fp,fp)It should be just #define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,X[29],x[29]) |
|
With everything so far: (Note: using Ubuntu 18.04 with FreeBSD AARCH64 crossroot) |
|
Ok, sounds like again elr vs pc. #define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Elr,elr)Last thing is the floating point context. In context.cpp at line 743, we'll need to add before the #elif TARGET_FREEBSD
const struct fpregs* fp = GetConstNativeSigSimdContext(native);
if (fp)
{
lpContext->Fpsr = fp->fp_sr;
lpContext->Fpcr = fp->fp_cr;
for (int i = 0; i < 32; i++)
{
lpContext->V[i] = *(NEON128*) &fp->q[i];
}
}Similarly at line 599, we need to add before the #elif TARGET_FREEBSD
struct fpregs* fp = GetNativeSigSimdContext(native);
if (fp)
{
fp->fp_sr = lpContext->Fpsr;
fp->fp_cr = lpContext->Fpcr;
for (int i = 0; i < 32; i++)
{
*(NEON128*) &fp->q[i] = lpContext->V[i];
}
} |
|
Including above code: Not sure why its throwing that. |
|
@janvorli Big thanks for help/pointers. I was able to compile coreclr part (there were some typos to fix): Also I've pushed those changes here main...sec:runtime:freebsd.arm64 @Thefrank I also took few changes from your patch regarding RID's - thanks :) Would you be able to try to cross compile under Linux? I've tried, but on my setup (Linux ARM64) it fails with: Also if you'll take my branch, those can also be needed - don't know if those are needed or it's just my setup: diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index d5dfc13504b..478029410b7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -133,7 +133,11 @@ elseif(FREEBSD)
set(CMAKE_CXX_COMPILER_TARGET ${triple})
set(CMAKE_ASM_COMPILER_TARGET ${triple})
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
- set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ if(CLR_CMAKE_HOST_OS STREQUAL FreeBSD)
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+ else()
+ set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -stdlib=libc++ -Qunused-arguments")
+ endif()
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld")
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -fuse-ld=lld")
elseif(ILLUMOS)
diff --git a/eng/native/configuretools.cmake b/eng/native/configuretools.cmake
index 6697524c659..b44bcc3417c 100644
--- a/eng/native/configuretools.cmake
+++ b/eng/native/configuretools.cmake
@@ -54,7 +54,11 @@ if(NOT WIN32 AND NOT CLR_CMAKE_TARGET_BROWSER)
set(TOOLSET_PREFIX ${ANDROID_TOOLCHAIN_PREFIX})
elseif(CMAKE_CROSSCOMPILING AND NOT DEFINED CLR_CROSS_COMPONENTS_BUILD AND
CMAKE_SYSTEM_PROCESSOR MATCHES "^(armv8l|armv7l|armv6l|aarch64|arm|s390x|ppc64le)$")
- set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+ if(FREEBSD)
+ set(TOOLSET_PREFIX "llvm-")
+ else()
+ set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+ endif()
else()
set(TOOLSET_PREFIX "")
endif()What should be next steps to get full SDK - as under FreeBSD ARM64 I don't have bootstrapped one, I think I will move to AMD64 FreeBSD and try there to cross compile? edit: @Thefrank for the error, it's edit 2: cross build under FreeBSD is going well... does this mean I will have working SDK on FreeBSD-arm64? 🤞 |
|
@sec I was happy to help. As for testing whether these changes are correct, running coreclr tests is the best way. These code paths are exercised by exception handling. src/tests/build.sh checked -priority1(Please note that you have to have the runtime clr+libs part built there before building the tests.) And then run them using src/tests/run.sh checked |
|
Hm, not that easy @janvorli :) While doing crosscompile under FreeBSD-amd64, using command What's going on in here? @Thefrank how's your Linux crossbuild? |
|
The cross components that are failing to build are compiled for x64, so they should be built on the host without the rootfs usage. It seems like there is some problem causing that the rootfs is still being used. |
|
So for my Ubuntu 18.04 AMD64 -> FreeBSD AARCH64 an old bug has presented itself: Previously under net6, this could be worked around in diff --git a/eng/native/gen-buildsys.sh b/eng/native/gen-buildsys.sh
index bf04c26f2b1..7d8793dd92c 100755
--- a/eng/native/gen-buildsys.sh
+++ b/eng/native/gen-buildsys.sh
@@ -97,7 +97,7 @@ if [[ "$build_arch" == "wasm" ]]; then
cmake_command="emcmake $cmake_command"
fi
-cmake_args_to_cache="$scan_build\n$SCAN_BUILD_COMMAND\n$generator\n$__UnprocessedCMakeArgs"
+cmake_args_to_cache=$(printf "$scan_build${SCAN_BUILD_COMMAND:+ }${SCAN_BUILD_COMMAND} $generator $__UnprocessedCMakeArgs")
cmake_args_cache_file="$2/cmake_cmd_line.txt"
if [[ -z "$__ConfigureOnly" ]]; then
if [[ -e "$cmake_args_cache_file" ]]; thenThis whole codeblock was removed in d22acef so this hacky-workaround won't work. |
|
I would also go for it. Arm64 need one small patch to compile, so having
rids will be nice.
|
|
I did some updates to port and did complete build on clean 14 arm64 VM in Azure (had to comment out this compat line and copy those libs by hand). Changes here https://github.com/sec/freebsd-ports/tree/lang-dotnet-arm64 atm I don't have 13.2 with arm64 available (Azure have only 14 image available) for the build - will try to hack something and test build on that, but it should be the same result. During the build, I hit one weird error during Maybe we should patch the build scripts to terminate hanging processes during build stage of each repo? |
|
JFYI, I have updated |
Left over processes during build can happen. The stack overflow is an odd one. I thought most of the crossgen issues were resolved in libunwind 1.8.0. The log generated by the sourcebuild process for that leg of the build should have more info. check
I think most build systems clean up left over processes on their own. It might be rather challenging determining which of the numerous Changing gears a bit: can the ARM64 version use NativeAOT? This mostly pertains to patching in RIDs to installer. |
I don't know what is the default size of stack for secondary threads on FreeBSD. It might be too small (e.g. for Linux MUSL, we had to explicitly change the default to about 1.5MB since the default was around 100kB IIRC). You could possibly enable the same code path for FreeBSD if it has a small secondary stack size. You can also override it for testing by setting the |
|
@Thefrank you mean can I do this? @janvorli looking at I did build on 13.2 under QEMU and hit that stack overflow on crossgen2 during runtime build. Have coredump for that, looked at And clean build on 14 just finished without any errors - @arrowd would it be OK to merge the changes I made into the port to enable arm64 builds also? Also I tried to build |
|
@sec I have started building your port on my RockPro64. Hope it won't hit a resource cap. |
|
I wasn't able to build .NET on my hardware because of out-of-memory issues. But the bootstrap provided by @sec seems to work, so I pushed these changes to the tree. Thanks Szczepan, we should now have .NET on FreeBSD aarch64 too! |
|
Ugh, it failed on FreeBSD 15 in the Not sure what to do with it now apart from marking it BROKEN for now. |
|
From the log: "/wrkdirs/usr/ports/lang/dotnet/work/dotnet-8.0.0/src/installer/artifacts/source-build/self/src/artifacts/sourcebuild.binlog" should give a good idea what exactly happened. edit: it also lists /wrkdirs/usr/ports/lang/dotnet/work/dotnet-8.0.0/artifacts/logs/installer.log which should also work but might not be as detailed |
|
Looking at this The failed stage was in Maybe some verbose flags could be added to build to have more info? As for BROKEN - I'm afraid if you do that, it will stay that way for a long time, as I don't have idea how to even start to debug this issue, as it's not 100% reproducable - some help from .net team on this topic would be valueable, if possible. edit: maybe it will give more info to add |
Can you give me a patch that plugs this flag in? I have no idea where to put it. |
The problem is somewhere deeper, for now leave it as it is - I saw that port build went fine under 13 arm64, so maybe it will catch up later.
Make sure to use latest packages and |
|
I don't have a hardware to reproduce this locally and I wasn't able to arrange access to the FreeBSD builder. I ended up marking dotnet as broken on 15-CURRENT AArch64. |
|
Back to Error happen at almost latest stage of the build, while building I have both binlog's and coredump. Looking at binlog, it fail at one of the Fun fact - when I try to run this by hand, it works and emit proper files, but during build it fails with After build fail, there's one zombie dotnet process It's using No idea how to debug this further :/ Machine have enough disk space and almost 20GB+ RAM free (and 4 cores). Any idea? Suggestions? Any magic switch? Binlog attached. |
|
AFAIK Runtime, ASPNetCore, and Installer all use ASPNetCore handles it all in one project: https://github.com/dotnet/aspnetcore/blob/268a2dfc29b33e3fdb73cbac6eb198c05314d77e/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj Installer handles crossgen targets here: https://github.com/dotnet/installer/blob/33549194e255cbaff0eda0d3cf688677b07e42ba/src/redist/targets/Crossgen.targets The functioning and command line parameters of it here: https://github.com/dotnet/installer/blob/33549194e255cbaff0eda0d3cf688677b07e42ba/src/core-sdk-tasks/Crossgen.cs Binlog is already |
|
I think that binlog won't help much in figuring out the culprit. A crashdump would be helpful though. From the call stack, we should at least get some idea on where it is crashing. |
|
Here's the edit: runtime went ok after few tries, here's another one from aspnet stage - https://gist.github.com/sec/76d5d4d6565fe07940890be862bee31f |
|
Hmm, the crashes are interesting, the crashing thread has zero instruction pointer, so it looks some context restoring went astray or some indirect jump went there or something like that. I think you may still be able to dig out details from the stack of that thread. I'd try to dump piece of the stack at current sp as data and search for addresses that can be looked up as being in libcoreclr.so (return addresses on the stack). That could allow us to reconstruct where it came from to that place. |
|
@janvorli Thanks (again) for looking into this - anything more I could provide/extract (and how) to maybe help? |
|
You can use |
|
Here you go @janvorli Isn't this connected with dotnet/source-build#4007 - just saw this... edit: using crossgen2 binary from 8.0.100 SDK makes the build complete without errors... so the issue is still in 8.0.3 (as I assume this crossgen2 was built during the vmr build) ? edit 2: also hitting the issue with cross-compiled 9.0.100-preview.2.24103.2 used to compile 9-preview.2 from vmr :( |
|
Has this been resolved in net9p3? |
|
The bug is in |
|
You can possibly temporarily switch to using the internal libunwind copy. It is essentially the same code as upstream with a couple of patches that we remove as soon as a fix is made upstream. |
|
AFAIK there should only be two places that need changes: runtime/src/coreclr/pal/src/CMakeLists.txt Lines 1 to 5 in 7dc3669 runtime/src/native/corehost/apphost/static/CMakeLists.txt Lines 225 to 227 in 7dc3669 That should change the behavior back to using the internal version |
|
Right, that should be all that's needed. |
|
Small update for arm64 interested. Doing crossbuild for 9-preview.7 is possible (there are 2 patches needed for ILC, more here #104497). Using that as bootstrap for building natively runtime and aspnet mostly work - produced ILC segfaults becuase of missing symbols export. sdk/installer build and produce working SDK and the rest. Should be possible to perform VMR build, when ILC thing is resolved. edit: ilc bug was resolved, it was my fault not to replace old nuget, so building all repos is a go. |
and others
Hi,
Not to spam #14537 (as it's already quite big) :)
As full native build of either v6 and v7 is possible under FreeBSD-amd64, I went ahead and tried to make the same process work for ARM64.
After getting configure to success, I've tried:
As it was used before, to cross compile from Linux-amd64 to Freebsd-x64, this fail at very early stage with:
Command I've used
ROOTFS_DIR=/home/ubuntu/arm64 ./src/coreclr/build-runtime.sh -arm64 -debug clang14 -cross -os FreeBSD.This does look like something missing in configure step, as those headers are present on rootfs_dir and host system:
so I guess something during configure stage should be added to look in those directories?
Trying with both
-crossand without, it fails with the same error:Command used was the same as with ad. 1. This looks more complicated, as it have something to do with libunwind being used? No idea how to get over that :)
Including patches I've come up with.
If anyone expierenced with cross build could comment on that, to even confirm it's possible in the current state to make that work or would it require far more work?
If someone is interested in helping/testing/making this work I can provide access to either FreeBSD or Linux ARM64 box that I use.
runtime_freebsd_arm64.txt
The text was updated successfully, but these errors were encountered: