Clean up Makefile #21031

khyperia · 2017-07-21T16:49:25Z

Every time I opened up the Makefile, I got really sad. This cleans it up to be a tad more reasonable.

List of changes:

Use immediate sets instead of recursive sets in many places
Don't assume $(shell pwd) is the same thing as the location of the Makefile (use MAKEFILE_LIST instead)
Make the WTF of HOME_DIR = $(shell cd ~ && pwd) more clear with docs, and be more sane about setting it (and use Make's env-setting command, instead of copypasting HOME=$(HOME_DIR) everywhere)
Clean up adding dotnet to PATH (again, use Make's export command, instead of copy-pasting PATH=...)
Shuffle around whatever the heck MSBUILD_ADDITIONALARGS was (just use MSBUILD_ARGS)
Tabs to spaces in areas that aren't rules
Error if OS_NAME is unknown, instead of failing obscurely later
Don't do bizarre things with pushd and single huge commands passed to the shell
Root paths in commands with vars calculated earlier (e.g. why were we even calculating SRC_PATH and then never using it...)
Pipe dotnet-install.sh into bash, instead of doing wacky writes to disk (where we have to worry about chmod +x and the current directory)
Use the specified version of cli for dotnet-install, instead of hardcoding 1.0.0
Remove HOME bizzare-ness from restore.sh, as we're setting that in the makefile
Remove strange comment from tests.sh

I'm still really suspicious of strange things happening in the bootstrap rule, but that seems like a very invasive and potentially behavior-changing change, so I've left that for now.

Using pr-for-personal-review-only until I take a look at ci results to make sure nothing is terribly broken.

jaredpar · 2017-07-21T17:34:08Z

Makefile

-DOTNET_VERSION = 1.0.1
-DOTNET = $(BINARIES_PATH)/dotnet-cli/dotnet
-TARGET_FX = netcoreapp1.1
+SHELL := /usr/bin/env bash


Can you explain the distinction here between = and :=?

= is lazy, := is strict.

foo = hi $(bar) bar = one $(info $(foo)) bar = two $(info $(foo))

prints "hi one" "hi two"

foo := hi $(bar) bar := one $(info $(foo)) bar := two $(info $(foo))

prints "hi " "hi " (empty var).

The first behavior seems incredibly insane, so having the default for most variable assignments be := is good style.

Note it gets weird with recursive variables (e.g. our use of MSBUILD_ARGS):

MSBUILD_ARGS = $(MSBUILD_ARGS) /some:switch

makes things get upset ("Recursive variable 'MSBUILD_ARGS' references itself (eventually). Stop.").

(so the recursive case must use :=)

Also, btw, in my list of changes, I listed this as "Use immediate sets instead of recursive sets in many places" (which is the documented name of each type of assignment - recursive is also called lazy, deferred, etc)

Yeah ... def seems like we should be using := then.

jaredpar · 2017-07-21T17:36:40Z

Makefile

+DOTNET_VERSION := 1.0.1
+DOTNET_PATH := $(BINARIES_PATH)/dotnet-cli
+DOTNET := $(DOTNET_PATH)/dotnet
+export PATH := $(DOTNET_PATH):$(PATH)


This will change the $PATH of all executed processes correct?

Correct. However, we were setting PATH in every rule anyway (export PATH="$(BINARIES_PATH)/dotnet-cli:$(PATH)" at the start of every rule), so this has no effect. (Except, technically, the $(DOTNET): rule, but that creates DOTNET_PATH, so there's no harm on having it already there. Also clean I guess, but that's just an rm.)

I presume the export turns this into a real environment variable instead of just a make variable?

@agocke Correct - Make docs "To pass down, or export, a variable, make adds the variable and its value to the environment for running each line of the recipe." (those docs are for sub-makes, there might be actual docs for export, but it gets the point across)

agocke

LGTM

agocke · 2017-07-21T18:23:53Z

Makefile

+DOTNET_VERSION := 1.0.1
+DOTNET_PATH := $(BINARIES_PATH)/dotnet-cli
+DOTNET := $(DOTNET_PATH)/dotnet
+export PATH := $(DOTNET_PATH):$(PATH)


I presume the export turns this into a real environment variable instead of just a make variable?

jaredpar · 2017-07-21T18:21:09Z

Makefile

-DOTNET_VERSION = 1.0.1
-DOTNET = $(BINARIES_PATH)/dotnet-cli/dotnet
-TARGET_FX = netcoreapp1.1
+SHELL := /usr/bin/env bash


Yeah ... def seems like we should be using := then.

jaredpar · 2017-07-21T18:23:28Z

Makefile

-	dotnet publish -r $(RUNTIME_ID) src/Test/DeployCoreClrTestRuntime -o $(BINARIES_PATH)/$(BUILD_CONFIGURATION)/CoreClrTest -p:RoslynRuntimeIdentifier=$(RUNTIME_ID) && \
-	build/scripts/tests.sh $(BUILD_CONFIGURATION)
+	dotnet publish -r $(RUNTIME_ID) $(SRC_PATH)/Test/DeployCoreClrTestRuntime -o $(BINARIES_PATH)/$(BUILD_CONFIGURATION)/CoreClrTest -p:RoslynRuntimeIdentifier=$(RUNTIME_ID)
+	./build/scripts/tests.sh $(BUILD_CONFIGURATION)


What does the distinction of ./ do here vs. the old one? At a glance it seems to have same behavior: use build/scripts/tests.sh from the current directory (which I'm always afraid may not be the directory of the make file). Does this add some other subtle distinction?

No difference. Changed it for consistency: nearby we invoke ./build/scripts/restore.sh. I'll update both to use $(THIS_MAKEFILE_PATH) (which apparently ends with a slash, didn't read the Makefile docs for dir well enough, I'll update that too), and any other relative-to-pwd things I find.

khyperia · 2017-07-24T15:50:05Z

Ping @jaredpar @agocke - pushed another change. List of things:

Modify initial MSBUILD_ARGS to be identical to build.ps1
Use BootstrapBuildPath instead of CscToolExe/etc., like build.ps1 does
Only use /fl if file logger path is specified (again, like build.ps1 does)
Use var += ... instead of var := $(var) ...
Remove slashes after use of $(THIS_MAKEFILE_PATH), document the fact it has a slash in it, and root build/scripts/{restore,tests}.sh
Update .PHONY to have the actually-present rule names
Add -v Minimal --disable-parallel to CrossPlatform.sln restore, like we're already doing with BaseToolset.csproj (otherwise the build can fail with "too many files open", there's an open issue somewhere for this I can dig up if necessary)

khyperia · 2017-07-24T15:58:06Z

Update: apparently /nodeReuse:false doesn't work on linux? ("unrecognized flag"). Removed it from the script. Also it seems like the specifying of CscCore.csproj is failing? I'll add in an explicit ".csproj" and see if that fixes it...

jaredpar · 2017-07-24T16:00:09Z

Update: apparently /nodeReuse:false doesn't work on linux?

Correct. That flag controls how they re-use the set of spawned MSBuild processes. That's not implemented at all on Linux yet.

Also root a couple paths, and modify restore.sh to be consistent. (without restore.sh change, build can fail with "too many files open")

There is an existing issue that warn-as-error surfaced - analyzer DLLs are getting loaded twice.

jaredpar · 2017-07-24T21:29:55Z

Makefile

@@ -29,7 +29,7 @@ else
    $(error "Unknown OS_NAME: $(OS_NAME)")
 endif

-MSBUILD_ARGS := /p:TreatWarningsAsErrors=true /warnaserror /nologo '/consoleloggerparameters:Verbosity=minimal;summary' /p:Configuration=$(BUILD_CONFIGURATION)


Can you file a bug to track us turning this back on?

agocke · 2017-07-24T22:40:12Z

Makefile

-	dotnet publish -c $(BUILD_CONFIGURATION) -r $(RUNTIME_ID) src/Compilers/VisualBasic/VbcCore -o $(BOOTSTRAP_PATH)/vbc
-	rm -rf Binaries/$(BUILD_CONFIGURATION)
+	@echo Building Bootstrap
+	$(BOOTSTRAP_CMD) $(SRC_PATH)/Compilers/CSharp/CscCore -o $(BOOTSTRAP_PATH)/csc


Isn't -o a dotnet build parameter, while the rest are msbuild parameters? Maybe we should try to keep the dotnet build parameters first, since I don't know how dotnet build decides to pass parameters to msbuild.

Yep - exactly what I just fixed (I struggled for a while trying to figure out how the heck it decides to pass things to msbuild - sorry for confusion for reviewers, I've been force-pushing a lot to re-run Jenkins tests. I just made an Ubuntu VM, though, so I can actually test locally now.).

The exact docs/syntax for dotnet build says Usage: dotnet build [arguments] [options] [args] which is... pretty much the most helpful wording ever.

"arguments" means "the project file"

"options" means "options understood by dotnet, e.g. -o"

"args" means "arguments passed through to msbuild"

The latest commits fix this ordering.

Also restructure Makefile to only be invoked once when bootstrapping/building/testing.

Also revert to CscToolPath/etc and remove MSBuildTask (fixing this would be a much more invasive change then Makefile refactoring)

agocke · 2017-07-25T17:31:07Z

Makefile

 endif

+MSBUILD_MAIN_ARGS := $(MSBUILD_ARGS)
+MSBUILD_BOOTSTRAP_ARGS := $(MSBUILD_ARGS)


It's kind of confusing to put dotnet build args in a list called MSBUILD_BOOTSTRAP_ARGS

I went to split this into MSBUILD_ARGS and DOTNET_ARGS, but then I found that the -r flag is just a simple passthrough to /p:RuntimeIdentifier=.... I changed it to use that, because we have similar things with /p:RoslynRuntimeIdentifier, and it's less confusing to have them be unified syntax.

The dotnet implementation is a simple pass-through to the parameter. Also, remove TARGET_FX, as it's unused.

khyperia · 2017-07-27T15:52:19Z

Replaced by #21146, this PR is no longer relevant.

Clean up Makefile

a6ab877

khyperia added Area-Infrastructure PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it. labels Jul 21, 2017

dnfclas added the cla-already-signed label Jul 21, 2017

jaredpar reviewed Jul 21, 2017

View reviewed changes

khyperia removed the PR For Personal Review Only The PR doesn’t require anyone other than the developer to review it. label Jul 21, 2017

jaredpar requested a review from agocke July 21, 2017 18:20

agocke approved these changes Jul 21, 2017

View reviewed changes

jaredpar approved these changes Jul 21, 2017

View reviewed changes

khyperia force-pushed the clean_up_makefile branch from debd205 to 2d7ae9c Compare July 24, 2017 15:54

khyperia force-pushed the clean_up_makefile branch from 2d7ae9c to 03c762f Compare July 24, 2017 15:59

Makefile cleanup: Unify some things with build.ps1

a1f140c

Also root a couple paths, and modify restore.sh to be consistent. (without restore.sh change, build can fail with "too many files open")

khyperia force-pushed the clean_up_makefile branch from 03c762f to a1f140c Compare July 24, 2017 16:45

Disable warn-as-error for now

898462a

There is an existing issue that warn-as-error surfaced - analyzer DLLs are getting loaded twice.

jaredpar reviewed Jul 24, 2017

View reviewed changes

jaredpar approved these changes Jul 24, 2017

View reviewed changes

khyperia mentioned this pull request Jul 24, 2017

Analyzer DLL warning in Linux build breaks turning on warn-as-error #21080

Open

khyperia force-pushed the clean_up_makefile branch 2 times, most recently from 90bbdc0 to 008f783 Compare July 24, 2017 22:37

agocke reviewed Jul 24, 2017

View reviewed changes

khyperia force-pushed the clean_up_makefile branch from 008f783 to a7b6bf6 Compare July 24, 2017 22:42

Include MSBuildTask in bootstrap build.

649d4e3

Also restructure Makefile to only be invoked once when bootstrapping/building/testing.

khyperia force-pushed the clean_up_makefile branch from a7b6bf6 to 649d4e3 Compare July 25, 2017 15:09

Inline restore.sh for maintainability

bf96ea4

Also revert to CscToolPath/etc and remove MSBuildTask (fixing this would be a much more invasive change then Makefile refactoring)

agocke reviewed Jul 25, 2017

View reviewed changes

Move runtime ID to msbuild arg instead of dotnet

3907851

The dotnet implementation is a simple pass-through to the parameter. Also, remove TARGET_FX, as it's unused.

khyperia mentioned this pull request Jul 26, 2017

Move logic from the Makefile to either cibuild.sh or build/scripts #21146

Merged

khyperia closed this Jul 27, 2017

khyperia deleted the clean_up_makefile branch July 27, 2017 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up Makefile #21031

Clean up Makefile #21031

khyperia commented Jul 21, 2017

jaredpar Jul 21, 2017

khyperia Jul 21, 2017

khyperia Jul 21, 2017

jaredpar Jul 21, 2017

jaredpar Jul 21, 2017

khyperia Jul 21, 2017

agocke Jul 21, 2017

khyperia Jul 21, 2017

agocke left a comment

agocke Jul 21, 2017

jaredpar Jul 21, 2017

jaredpar Jul 21, 2017

khyperia Jul 21, 2017

khyperia commented Jul 24, 2017

khyperia commented Jul 24, 2017

jaredpar commented Jul 24, 2017

jaredpar Jul 24, 2017

khyperia Jul 24, 2017

agocke Jul 24, 2017

khyperia Jul 25, 2017

agocke Jul 25, 2017

khyperia Jul 25, 2017

khyperia commented Jul 27, 2017

Clean up Makefile #21031

Clean up Makefile #21031

Conversation

khyperia commented Jul 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agocke left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khyperia commented Jul 24, 2017

khyperia commented Jul 24, 2017

jaredpar commented Jul 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khyperia commented Jul 27, 2017