Problem with TUP_VARIANTDIR based paths in Lua file

[edwinzrodriguez]

I have an existing system I'm trying to update to use the current head in master f26bc1e and have run into an issue using $(TUP_VARIANTDIR) in CFLAGS with rules generated by a lua script.

In the attached, certain headers are copied to build-dbg/root/nimble/include, then I add include directives like -I$(TUP_VARIANTDIR)/root/nimble/include.
For rules defined in a Tupfile, the resulting command for a file compiled in src/utils is -I../../build-dbg/root/nimble/include
But if I use a lua file src/build_tools/generate_bin_rules.lua to generate rules for files in src/utils/test results in a command with -I../build-dbg/root/nimble/include rather than the correct path -I../../../build-dbg/root/nimble/include
It appears that TUP_VARIANTDIR is expanded relative to the Tupfile at the root, rather than the directory calling the lua script.

` 4) [0.071s] [build-dbg] src/ver: echo ROOT=../.. TUP_VARIANTDIR=../../build-dbg/src/ver TUP_VARIANT_OUTPUTDIR=../../build-dbg/src/ver VARIANT_DIR=../../build-dbg VARIANT_OUTPUT_DIR=../../build-dbg/src/ver; ${DISTCC_ENABLE:+distcc} ${CXX:-/usr/bin/g++} -std=gnu++11 -Wa,--compress-debug-sections -frecord-gcc-switches -fdiagnostics-color=${GCC_COLOR:-always} -D__STDC_LIMIT_MACROS -DWITH_OPENSSL -DBOOST_SPIRIT_THREADSAFE -DBOOST_NO_CXX11_SCOPED_ENUMS -D_GLIBCXX_USE_CXX11_ABI=0 -Wall -Werror -fPIC -fno-omit-frame-pointer -I. -I../../build-dbg/root/nimble/include -I../../build-dbg/root/nimble/include/sm -I../../build-dbg -g -c version.cc -o ../../build-dbg/src/ver/version.o -DBUILDID=""$(cat ../../build-dbg/version.txt | sed -n "s/^BUILDNUM=(.*)$/\1/p")"" -DCHANGESET=""${COMMIT}"" -DBRANCHNAME=""${BRANCH}"" -DBUILDVARIANT=""dbg"" -D"PRECHECK_$(cat ../../build-dbg/build_tools/precheck_result)"
ROOT=../.. TUP_VARIANTDIR=../../build-dbg/src/ver TUP_VARIANT_OUTPUTDIR=../../build-dbg/src/ver VARIANT_DIR=../../build-dbg VARIANT_OUTPUT_DIR=../../build-dbg/src/ver
cat: ../../build-dbg/build_tools/precheck_result: No such file or directory

• 3. [build-dbg] src/utils/test: ${DISTCC_ENABLE:+distcc} ${CXX:-/usr/bin/g++} -std=gnu++11 -Wa,--compress-debug-sections -frecord-gcc-switches -fdiagnostics-color=${GCC_COLOR:-always} -D__STDC_LIMIT_MACROS -DWITH_OPENSSL -DBOOST_SPIRIT_THREADSAFE -DBOOST_NO_CXX11_SCOPED_ENUMS -D_GLIBCXX_USE_CXX11_ABI=0 -Wall -Werror -fPIC -fno-omit-frame-pointer -I. -I../build-dbg/root/nimble/include -I../build-dbg/root/nimble/include/sm -I../build-dbg -g -I../../../build_tools/include -c test_uuid.cc -o ../../../build-dbg/src/utils/test/test_uuid.o
     test_uuid.cc:7:10: fatal error: u_uuid.h: No such file or directory
     7 | #include "u_uuid.h"
     | ^~~~~~~~~~
     compilation terminated.
     *** tup messages ***
     *** Command ID=75 failed with return value 1

2. [2.151s] [build-dbg] src/utils: echo ROOT=../.. TUP_VARIANTDIR=../../build-dbg/src/utils TUP_VARIANT_OUTPUTDIR=../../build-dbg/src/utils VARIANT_DIR=../../build-dbg VARIANT_OUTPUT_DIR=../../build-dbg/src/utils; ${DISTCC_ENABLE:+distcc} ${CXX:-/usr/bin/g++} -std=gnu++11 -Wa,--compress-debug-sections -frecord-gcc-switches -fdiagnostics-color=${GCC_COLOR:-always} -D__STDC_LIMIT_MACROS -DWITH_OPENSSL -DBOOST_SPIRIT_THREADSAFE -DBOOST_NO_CXX11_SCOPED_ENUMS -D_GLIBCXX_USE_CXX11_ABI=0 -Wall -Werror -fPIC -fno-omit-frame-pointer -I. -I../../build-dbg/root/nimble/include -I../../build-dbg/root/nimble/include/sm -I../../build-dbg -g -c u_uuid.cc -o ../../build-dbg/src/utils/u_uuid.o
   ROOT=../.. TUP_VARIANTDIR=../../build-dbg/src/utils TUP_VARIANT_OUTPUTDIR=../../build-dbg/src/utils VARIANT_DIR=../../build-dbg VARIANT_OUTPUT_DIR=../../build-dbg/src/utils
   [ ETA~=<1s ] 81%
   *** tup: 1 job failed.`

[gittup]

I think this has to do with when strings are eval'd in a regular Tupfile vs the lua Tupfile. In a regular Tupfile, if you do something like:

FOO = 2
BAR = $(FOO)
FOO = 3 
: |> echo $(BAR) |>

The output will be 2, since $(FOO) is evaluated immediately when setting BAR. In the Lua parser, strings aren't evaluated until they are used in a tup.rule(), so the equivalent in a Tupfile.lua would output 3. Presumably the correct thing to do here would be to eval() the right-hand side when setting a variable in lua.

Do things work as you expect if you use the native lua functions instead? Eg:

CFLAGS += '-I' .. tup.getvariantdir() .. '/root/nimble/include'

This will evaluate the variantdir immediately when calling the function, rather than deferring it until CFLAGS is used in a rule.

I think a longer-term fix would be to eval strings immediately to avoid unwanted surprises like this, though.

[edwinzrodriguez]

Looks like 'getvariantdir' returns the path to the output dir - the lua script is in 'src/build_tools'
"-I../../../build-dbg/build_tools/root/nimble/include". But I can work around it with this:
'-I' .. tup.getvariantdir() .. '/../root/nimble/include '

Btw, why isn't there an equivalent lua function for TUP_VARIANT_OUTPUTDIR?

[gittup]

Btw, why isn't there an equivalent lua function for TUP_VARIANT_OUTPUTDIR?

Good question :) - I added this in commit 7da0d88.