Build experience and issues on Fedora 33 and SELinux

[Xrayez]

#72 is merged and I've decided to give a try at compiling the engine this way. Also note that #12 has some advanced information as well (for those not familiar with podman or container technology in general). This is a summary of discussion at Godot Contributors chat.

I've managed to build images and managed to build something, but stumbled upon various errors in the process on a clear installation of Fedora 33 (which also made mono crash on Windows).

• First thing first, do not attempt to use sudo, as podman is designed to work in rootless mode. Otherwise this may lead to permission errors once you build with user (for instance, failing to write to logs with tee may prevent other builds from being run at all).
• Had to dnf install patch manually.
• A lot of build steps resulted in permission errors with SELinux enforced. At first I added the :z tag to each -v path argument in build.sh, but it didn't resolve all permission errors, and even crashed in some cases (around building Windows).
  • Using sudo setenforce 0 allows to circumvent this before building.
• Also installed dnf install wine on the host to be sure, may be required.
• Note sure if mono runtime should be installed on the host as well, as resulted in mono: command not found in some cases.
  • Build experience and issues on Fedora 33 and SELinux #74 (comment)
• Errors like err:seh:install_bpf Unexpected status 0xffffffff, errno 1.
  • seems to be innocent.
• Compilation error for macOS: Build experience and issues on Fedora 33 and SELinux #74 (comment).
• Build issues with UWP: Build experience and issues on Fedora 33 and SELinux #74 (comment).

Worth documenting that export REGISTRY="localhost" must be written in build script if building locally.

Feel free to close this at some point, I'd just like to document the limitations so I can refer to them in case I forget, and may be useful for other newbies like me.

[Xrayez]

So, after second try with sudo setenforce 0, I've rebuilt containers, but I'm stumbling upon the same problem with Windows mono builds (crash due to missing prebuilt API assemblies):

PS J:\tools-mono> .\godot.windows.opt.tools.64.mono.exe
Godot Engine v3.2.3.stable.mono.official - https://godotengine.org
OpenGL ES 3.0 Renderer: GeForce GTX 750 Ti/PCIe/SSE2

Mono: Log file is: 'C:\Users\Xrayez\AppData\Roaming/Godot/mono/mono_logs/2021_02_27 04.08.27 (11496).txt'
ERROR: Failed to load one of the prebuilt API assemblies.
   At: modules/mono/mono_gd/gd_mono.cpp:930

=================================================================
        Native Crash Reporting
=================================================================
Got a UNKNOWN while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

=================================================================
        Managed Stacktrace:
=================================================================
=================================================================

When comparing the contents of GodotSharp folder with official builds and mine, I get this output:

PS C:\Users\Xrayez> $a = Get-ChildItem -Recurse -path "J:\tools-mono\GodotSharp"
PS C:\Users\Xrayez> $b = Get-ChildItem -Recurse -path "G:\Downloads\Godot_v3.2.3-stable_mono_win64\Godot_v3.2.3-stable_mono_win64\GodotSharp"
PS C:\Users\Xrayez> Compare-Object -ReferenceObject $a -DifferenceObject $b

InputObject                                                 SideIndicator
-----------                                                 -------------
Microsoft.Build.Engine.dll                                  =>
Microsoft.Build.Tasks.v4.0.dll                              =>
Microsoft.Build.Utilities.v4.0.dll                          =>
Microsoft.CodeAnalysis.CSharp.dll                           =>
Microsoft.CodeAnalysis.CSharp.Scripting.dll                 =>
Microsoft.CodeAnalysis.dll                                  =>
Microsoft.CodeAnalysis.Scripting.dll                        =>
Microsoft.CodeAnalysis.VisualBasic.dll                      =>
Microsoft.VisualC.dll                                       =>
Mono.Data.Tds.dll                                           =>
Mono.Messaging.RabbitMQ.dll                                 =>
Mono.Profiler.Log.dll                                       =>
Mono.XBuild.Tasks.dll                                       =>
RabbitMQ.Client.dll                                         =>
System.ComponentModel.Composition.dll                       =>
System.Data.DataSetExtensions.dll                           =>
System.Data.Entity.dll                                      =>
System.Data.Services.Client.dll                             =>
System.Data.Services.dll                                    =>
System.Deployment.dll                                       =>
System.DirectoryServices.Protocols.dll                      =>
System.Dynamic.dll                                          =>
System.Json.dll                                             =>
System.Json.Microsoft.dll                                   =>
System.Reactive.Core.dll                                    =>
System.Reactive.Debugger.dll                                =>
System.Reactive.Experimental.dll                            =>
System.Reactive.Interfaces.dll                              =>
System.Reactive.Linq.dll                                    =>
System.Reactive.Observable.Aliases.dll                      =>
System.Reactive.PlatformServices.dll                        =>
System.Reactive.Providers.dll                               =>
System.Reactive.Runtime.Remoting.dll                        =>
System.Reactive.Windows.Forms.dll                           =>
System.Reactive.Windows.Threading.dll                       =>
System.Reflection.Context.dll                               =>
System.Reflection.Metadata.dll                              =>
System.Threading.Tasks.Extensions.dll                       =>
System.Web.Mobile.dll                                       =>
System.Web.RegularExpressions.dll                           =>
System.Windows.dll                                          =>
System.Windows.Forms.DataVisualization.dll                  =>
System.Workflow.Activities.dll                              =>
System.Workflow.ComponentModel.dll                          =>
System.Workflow.Runtime.dll                                 =>
System.Xml.Serialization.dll                                =>
WebMatrix.Data.dll                                          =>
Microsoft.Win32.Primitives.dll                              =>
Microsoft.Win32.Registry.AccessControl.dll                  =>
Microsoft.Win32.Registry.dll                                =>
netstandard.dll                                             =>
System.AppContext.dll                                       =>
System.Collections.Concurrent.dll                           =>
System.Collections.dll                                      =>
System.Collections.NonGeneric.dll                           =>
System.Collections.Specialized.dll                          =>
System.ComponentModel.Annotations.dll                       =>
System.ComponentModel.dll                                   =>
System.ComponentModel.EventBasedAsync.dll                   =>
System.ComponentModel.Primitives.dll                        =>
System.ComponentModel.TypeConverter.dll                     =>
System.Console.dll                                          =>
System.Data.Common.dll                                      =>
System.Data.SqlClient.dll                                   =>
System.Diagnostics.Contracts.dll                            =>
System.Diagnostics.Debug.dll                                =>
System.Diagnostics.FileVersionInfo.dll                      =>
System.Diagnostics.Process.dll                              =>
System.Diagnostics.StackTrace.dll                           =>
System.Diagnostics.TextWriterTraceListener.dll              =>
System.Diagnostics.Tools.dll                                =>
System.Diagnostics.TraceEvent.dll                           =>
System.Diagnostics.TraceSource.dll                          =>
System.Diagnostics.Tracing.dll                              =>
System.Drawing.Primitives.dll                               =>
System.Dynamic.Runtime.dll                                  =>
System.Globalization.Calendars.dll                          =>
System.Globalization.dll                                    =>
System.Globalization.Extensions.dll                         =>
System.IO.Compression.ZipFile.dll                           =>
System.IO.dll                                               =>
System.IO.FileSystem.AccessControl.dll                      =>
System.IO.FileSystem.dll                                    =>
System.IO.FileSystem.DriveInfo.dll                          =>
System.IO.FileSystem.Primitives.dll                         =>
System.IO.FileSystem.Watcher.dll                            =>
System.IO.IsolatedStorage.dll                               =>
System.IO.MemoryMappedFiles.dll                             =>
System.IO.Pipes.dll                                         =>
System.IO.UnmanagedMemoryStream.dll                         =>
System.Linq.dll                                             =>
System.Linq.Expressions.dll                                 =>
System.Linq.Parallel.dll                                    =>
System.Linq.Queryable.dll                                   =>
System.Net.AuthenticationManager.dll                        =>
System.Net.Cache.dll                                        =>
System.Net.Http.Rtc.dll                                     =>
System.Net.HttpListener.dll                                 =>
System.Net.Mail.dll                                         =>
System.Net.NameResolution.dll                               =>
System.Net.NetworkInformation.dll                           =>
System.Net.Ping.dll                                         =>
System.Net.Primitives.dll                                   =>
System.Net.Requests.dll                                     =>
System.Net.Security.dll                                     =>
System.Net.ServicePoint.dll                                 =>
System.Net.Sockets.dll                                      =>
System.Net.Utilities.dll                                    =>
System.Net.WebHeaderCollection.dll                          =>
System.Net.WebSockets.Client.dll                            =>
System.Net.WebSockets.dll                                   =>
System.ObjectModel.dll                                      =>
System.Reflection.dll                                       =>
System.Reflection.Emit.dll                                  =>
System.Reflection.Emit.ILGeneration.dll                     =>
System.Reflection.Emit.Lightweight.dll                      =>
System.Reflection.Extensions.dll                            =>
System.Reflection.Primitives.dll                            =>
System.Reflection.TypeExtensions.dll                        =>
System.Resources.Reader.dll                                 =>
System.Resources.ReaderWriter.dll                           =>
System.Resources.ResourceManager.dll                        =>
System.Resources.Writer.dll                                 =>
System.Runtime.CompilerServices.VisualC.dll                 =>
System.Runtime.dll                                          =>
System.Runtime.Extensions.dll                               =>
System.Runtime.Handles.dll                                  =>
System.Runtime.InteropServices.dll                          =>
System.Runtime.InteropServices.RuntimeInformation.dll       =>
System.Runtime.InteropServices.WindowsRuntime.dll           =>
System.Runtime.Numerics.dll                                 =>
System.Runtime.Serialization.Formatters.dll                 =>
System.Runtime.Serialization.Json.dll                       =>
System.Runtime.Serialization.Primitives.dll                 =>
System.Runtime.Serialization.Xml.dll                        =>
System.Security.AccessControl.dll                           =>
System.Security.Claims.dll                                  =>
System.Security.Cryptography.Algorithms.dll                 =>
System.Security.Cryptography.Csp.dll                        =>
System.Security.Cryptography.DeriveBytes.dll                =>
System.Security.Cryptography.Encoding.dll                   =>
System.Security.Cryptography.Encryption.Aes.dll             =>
System.Security.Cryptography.Encryption.dll                 =>
System.Security.Cryptography.Encryption.ECDiffieHellman.dll =>
System.Security.Cryptography.Encryption.ECDsa.dll           =>
System.Security.Cryptography.Hashing.Algorithms.dll         =>
System.Security.Cryptography.Hashing.dll                    =>
System.Security.Cryptography.Primitives.dll                 =>
System.Security.Cryptography.ProtectedData.dll              =>
System.Security.Cryptography.RandomNumberGenerator.dll      =>
System.Security.Cryptography.RSA.dll                        =>
System.Security.Cryptography.X509Certificates.dll           =>
System.Security.Principal.dll                               =>
System.Security.Principal.Windows.dll                       =>
System.Security.SecureString.dll                            =>
System.ServiceModel.Duplex.dll                              =>
System.ServiceModel.Http.dll                                =>
System.ServiceModel.NetTcp.dll                              =>
System.ServiceModel.Primitives.dll                          =>
System.ServiceModel.Security.dll                            =>
System.ServiceProcess.ServiceController.dll                 =>
System.Text.Encoding.CodePages.dll                          =>
System.Text.Encoding.dll                                    =>
System.Text.Encoding.Extensions.dll                         =>
System.Text.RegularExpressions.dll                          =>
System.Threading.AccessControl.dll                          =>
System.Threading.dll                                        =>
System.Threading.Overlapped.dll                             =>
System.Threading.Tasks.dll                                  =>
System.Threading.Tasks.Parallel.dll                         =>
System.Threading.Thread.dll                                 =>
System.Threading.ThreadPool.dll                             =>
System.Threading.Timer.dll                                  =>
System.ValueTuple.dll                                       =>
System.Xml.ReaderWriter.dll                                 =>
System.Xml.XDocument.dll                                    =>
System.Xml.XmlDocument.dll                                  =>
System.Xml.XmlSerializer.dll                                =>
System.Xml.XPath.dll                                        =>
System.Xml.XPath.XDocument.dll                              =>
System.Xml.Xsl.Primitives.dll                               =>
aot-compilers                                               =>
Microsoft.Build.dll                                         =>
Microsoft.Build.Framework.dll                               =>
System.Collections.Immutable.dll                            =>
System.Memory.dll                                           =>
System.Runtime.CompilerServices.Unsafe.dll                  =>
System.Threading.Tasks.Dataflow.dll                         =>
iphone-arm64                                                =>
aarch64-apple-darwin-mono-sgen                              =>

So yeah, that's a lot of stuff missing (around 30 MB in total), and I have no clue why those libs wouldn't compile on my machine.

Classical builds work fine.

Again, I suspect this has something to do with how wine may be (mis)configured. I have mounted binfmt as suggested, installed wine on host with dnf install wine, don't know what else could be done on the user level.

Looking at git history here, stumbled upon #32, with changes that happen to occur exactly on roslyn-binaries:

/bin/sh ./../../mkinstalldirs ../../build/deps
mkdir -p -- ../../build/deps
touch ../../build/deps/.stamp
make install-local
make[6]: Entering directory '/root/mono-6.12.0.114/mcs/tools/gensources'
/bin/sh ./../../mkinstalldirs ../../class/lib/build-win32/tmp/
mkdir -p -- ../../class/lib/build-win32/tmp/
touch ../../class/lib/build-win32/tmp//.stamp
CSC     [build-win32] gensources.exe
'/root/mono-6.12.0.114/mcs/tools/gensources/Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.6.0;Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.5.0;Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.4.0;Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.3.1;Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.3.0;Z' in MONO_PATH doesn't exist or has wrong permissions.
'/root/mono-6.12.0.114/mcs/tools/gensources/\root\mono-6.12.0.114\external\roslyn-binaries\Microsoft.Net.Compilers\3.1.0;' in MONO_PATH doesn't exist or has wrong permissions.
/bin/sh ./../../mkinstalldirs ../../class/lib/build-win32/
touch ../../class/lib/build-win32//.stamp
CYGWIN=winsymlinks:nativestrict ln -s /root/mono-6.12.0.114/mcs/class/lib/build-win32 /root/mono-6.12.0.114/mcs/class/lib/build
make[6]: Leaving directory '/root/mono-6.12.0.114/mcs/tools/gensources'
make[5]: Leaving directory '/root/mono-6.12.0.114/mcs/tools/gensources'

---

Attaching build-containers build logs in case someone is interested in helping out, any pointers are greatly appreciated (look for windows.log specifically):
godot.build-containers.logs.zip

But I think I'll resolve this eventually...

Thanks!

[Xrayez]

Reporting that I've managed to build godot-windows with mono, happy-happy! 😂

Turns out I was running out of RAM, and the default installation of Fedora 33 didn't create a swap file, but /dev/zram with 4GB. My laptop has only 8GB so compilation failed with fatal error: Killed signal terminated program cc1 in some cases, which lead to the situation that the rest build steps from make got skipped... Swap usage peaked at somewhere 6GB...

The default installation uses the new Btrfs filesystem, so refer to https://wiki.archlinux.org/index.php/Swap#Swap_file with Btrfs-specific instructions on how to create such file.

Comparing with official mono builds:

PS C:\Users\Xrayez> $a = Get-ChildItem -Recurse -path "I:\tools-mono\GodotSharp"
PS C:\Users\Xrayez> $b = Get-ChildItem -Recurse -path "D:\bin\godot\3.2\GodotSharp"
PS C:\Users\Xrayez> Compare-Object -ReferenceObject $a -DifferenceObject $b

InputObject                    SideIndicator
-----------                    -------------
aot-compilers                  =>
iphone-arm64                   =>
aarch64-apple-darwin-mono-sgen =>

I haven't build AOT compilers since I haven't yet tried to compile for iOS/macOS.

---

I suspect that the MONO_PATH doesn't exist or has wrong permissions errors are a byproduct of differences between PATH separators on Linux/Windows. That said, I think it properly converts the path for Windows with winepath but somehow it interferes with Linux host.

Regarding err:seh:install_bpf Unexpected status 0xffffffff, errno 1 coming from wine (lots of them), this also seems innocent (somehow it uses wine-staging maybe that's why).

[Xrayez]

Succeeded with macOS/iOS builds now (including mono). But had to add this to modules/webm/libvpx/SCSub:

env_libvpx_neon.Append(CPPDEFINES=["__ARM_NEON__"])

Not sure whether it's just a workaround or a solution, but this allowed me to at least prevent the compilation error:

#error "NEON support not enabled"

This may be due to NEON detection bugs while using/building OSXCross, no idea. In either case, we have a bug somewhere.

Some more links:

• [macOS, 3.2] Add support for the Apple Silicon (ARM64) build target. godot#39943
• godotengine/godot-build-scripts@c44015d

[Xrayez]

Tried to setup UWP builds, seems like WIP because the build script still references private repository instead of localhost:

https://github.com/godotengine/godot-build-scripts/blob/f3a5097159933becc1df63f0b6c567c2ff4830fa/build.sh#L200-L201

Probably official builds use older version of msvc2017.tar, because my own build failed due to referencing non-existing paths in msvc-fixup.sh script. So, I think the buildsystem has regressed a little bit.

I've also had to increase ulimit just like for godot-windows, otherwise I got errors coming from Python.

That said, currently it will likely be faster to compile UWP export templates on native Windows machine than trying to fixup build scripts, so I suggest avoiding this step if you're a user as of now.

[Xrayez]

Succeeded with macOS/iOS builds now (including mono). But had to add this to modules/webm/libvpx/SCSub:

env_libvpx_neon.Append(CPPDEFINES=["__ARM_NEON__"])

Turns out it was an issue with 3.2.3, compiles fine on 3.2.4-rc3 without that patch. Probably the 3.2 branch of this repository should be used if you want to compile 3.2.3, but since 3.2.4 is close to release, it doesn't matter anyways.

The same applies to Javascript/web builds (threads + gdnative).

[Xrayez]

For UWP build, the link to download vs_buildtools.exe is too new, found this:

https://stackoverflow.com/a/57808976/2564620

cd "C:\Program Files (x86)\"
7z a "D:/msvc2017.tar" "./Microsoft Visual Studio" -sse -xr@"msvc2017_exclude.txt"

msvc2017_exclude.txt:

"Microsoft Visual Studio/2017/Community"
"Microsoft Visual Studio/2019"

But it still complains about missing Microsoft SDKs directory, don't know how to resolve yet. wine complains about some dotnet45 packages. Perhaps Microsoft SDKs directory should be packed as well?

[Xrayez]

Most issues resolved by now, except for UWP which is outdated, see #78.

[Xrayez]

Stumbled upon useful info regarding podman/SELinux: https://github.com/containers/container-selinux.