Build ShaderCompilerWorker when building Editor target in ue4 build

[TBBle]

On Windows, I was trying to work out why my Editor was hanging at 39% when launching a project using a clean git checkout and a successful ue4 build Development Editor.

I noticed the following in the logs right before it hung:

'XGEControlWorker.exe' is not recognized as an internal or external command, operable program or batch file.

This was UE4 trying to use Incredibuild to compiler Shaders during startup.

It turns out XGEControlWorker.exe is a post-build copy of ShaderCompileWorker.exe. Poking around, I discovered that on OS X and Linux, the setup instructions/scripts build ShaderCompileWorker explicitly before building UE4Editor.

On Linux, it also suggests building CrashReportClient, UnrealLightmass, and UnrealPak.

On OS X, Build.sh and XcodeBuild.sh take a parameter -buildscw in ($4 or $5) to build ShaderCompilerWorker before passing the command line over to UnrealBuildTool.exe.

On Windows, the VC Project Files generate Build.bat calls and add ShaderCompilerWorker as an extra target when building the Editor, based on a default-true config option bEditorDependsOnShaderCompileWorker.

You can see where it adds ShaderCompilerWorker it also adds LiveCodingConsole if enabled (defaults to true in Win64 non-Shipping non-Programs).

To fix the editor, I had to manually compile ShaderCompilerWorker.exe:

D:\Unreal\UnrealEngine\Engine\Build\BatchFiles\Build.bat ShaderCompileWorker Win64 Development

and everything worked fine after that.

In the end, what I'm hoping for is some piece of magic in ue4 build when it's given or defaults to an Editor target which will also build the appropriate extra targets for the platform.

[adamrehn]

The ue4 build command is concerned only with building the project or plugin in the current directory, so building Engine tools is decidedly outside the scope of its responsibilities. I could potentially add a separate subcommand to ue4cli for building Engine components when working with source builds, but I'm not sure if it's entirely necessary in this particular instance. Under Linux, the makefile generator creates a handy target called StandardSet that builds the Editor and all related tools in one hit, and under Windows the UE4 target in the Visual Studio solution invokes UBT to compile both the UE4Editor and ShaderCompileWorker targets.

[TBBle]

Source builds are where this matters; installed builds already build all these tools.

ShaderCompileWorker is the same as UE4Editor or UnrealBuildTool in this sense, and ue4 build happily builds those when needed.

I don't think a separate sub-command makes sense, I expect ue4 build Editor to leave me with the same working Editor setup as building the "Development Editor" target in my project in Visual Studio (on Windows).

Maybe it's a difference in expectations? It sounds like you created ue4 build as being more of a thin wrapper around Build.bat and Build.sh, in which case a separate sub-command (ue4 make?) seems more justified.

That might even depend on the appropriate toolchain generator, although it'd be annoying when I'm working in Visual Studio Code to have my builds require generating Visual Studio solutions as a first step, so that the generated NMake command can be run, to call Build.bat with the right extra parameters.

---

Just to clarify a possible disconnect... At no point in the workflow I was using here did I open a code editor or generate solutions. It was very-much a clone-compile-runEditor flow, like I might give to a non-programmer working with assets or blueprints against project HEAD. In my case, I was validating a backend change, so just needed to be sure that what normally works, still works.

---

It might also be useful to have a ue4 buildprogram sub-command that passes the right parameters to Build.bat to compile non-project-specific Programs, which would simplify my workaround from

D:\Unreal\UnrealEngine\Engine\Build\BatchFiles\Build.bat ShaderCompileWorker Win64 Development

to

ue4 buildprogram ShaderCompilerWorker Development

but that's still a workaround I have to carry into every UE4 project.

This might even just be a --program parameter to ue4 build, since all it needs to do (I think) is not prepend the project name to the target.

[adamrehn]

I'd always assumed that the standard expectation for users working with source builds of the Engine is that they follow the GitHub README instructions by first generating project files for the Engine itself and then building the Editor and related tooling before ever attempting to open or build an Unreal project. Regarding this particular use case, Epic created the UnrealGameSync tool specifically to distribute prebuilt binaries for in-house Engine versions to artists to avoid them needing to perform compilation at all. Is this practice of having users clone the source and then build the entire Engine the first time they work with a project widespread?

[adamrehn]

(Just FYI, I've added the requested functionality to ue4 build in commit 259d03c, but I'm still genuinely curious to know how often this situation arises, since it invalidates my existing assumptions around how developers typically use source builds of the Unreal Engine.)

[TBBle]

In our studio, it seems that UnrealGameSync has been slow to gain traction. I believe some teams are working with getting it regularly used, but it's certainly not the default. I think the game teams who aren't on UGS are committing their engine binaries, as they're using the project-forked-UE4 layout (Engine/ next to Project/, so that the build-system detects it by default).

I expect most people on my team (we're backend-tech focused, and so only own some UE4 Plugins) are using VS or VS Code workflows, so it might be just me who uses (and recommends) ue4cli as a fast way to script up a whole bunch of stuff.

Honestly, I had not looked at the GitHub README in a long time. My use of custom source is generally only when there's some issue with the Epic Launcher's Installed build, and I either need a debug-compiled engine, or to create or test an engine patch. So in that case, I just change root using ue4cli, and then clean and rebuild, and continue exactly as I was. Sometimes I'm doing that with a project's custom source as well, and it's quite handy to be able to easily chop-and-change ue4 targets when doing plugin development. ^_^

[adamrehn]

Fair enough, thanks for the insight into the use cases where this is handy. The change is now live in ue4cli version 0.0.42.