Allow override_find_program to use an executable. #4055

espindola · 2018-08-19T15:30:22Z

With this it is now possible to do

foobar = executable('foobar', ...)
meson.override_find_program('foobar', foobar)

Which is convenient for a project like protobuf which produces both a
dependency and a tool. If protobuf is updated to use
override_find_program, it can be used as

protobuf_dep = dependency('protobuf', version : '>=3.3.1',
fallback : ['protobuf', 'protobuf_dep'])
protoc_prog = find_program('protoc')

With this it is now possible to do foobar = executable('foobar', ...) meson.override_find_program('foobar', foobar) Which is convenient for a project like protobuf which produces both a dependency and a tool. If protobuf is updated to use override_find_program, it can be used as protobuf_dep = dependency('protobuf', version : '>=3.3.1', fallback : ['protobuf', 'protobuf_dep']) protoc_prog = find_program('protoc')

textshell · 2018-08-19T15:37:35Z

Thanks for working on this. I have a project that will at some point need this.

But it needs checking against uses of that program directly in the meson file (see #3218 (comment) in the original PR for what i got from some related irc discussions).

Also does this at the moment take care of adding dependencies to the backends?

nirbheek · 2018-08-19T15:53:09Z

mesonbuild/interpreter.py

-                             '(overridden: %s)' % ' '.join(exe.command))
-                return ExternalProgramHolder(exe)
+                             '(overridden: %s)' % exe)
+                return self.holderify(exe)


I don't think we should start returning ExecutableHolder in find_program(). That's unexpected and breaks the abstraction. For instance, users will expect to be able to call all the methods that exist on ExternalProgramHolder, and then we need to either implement all those methods on ExecutableHolder and find a way to not allow those methods to be called on objects that are returned from executable(), which is impossible.

A better way would be to allow ExternalProgramHolder to "hold" Executable objects.

nirbheek · 2018-08-19T15:53:23Z

mesonbuild/interpreter.py

@@ -1753,7 +1756,7 @@ def override_find_program_method(self, args, kwargs):
            if not os.path.exists(abspath):
                raise InterpreterException('Tried to override %s with a file that does not exist.' % name)
            exe = dependencies.ExternalProgram(abspath)
-        if not isinstance(exe, dependencies.ExternalProgram):
+        if not isinstance(exe, (dependencies.ExternalProgram, build.Executable)):
            # FIXME, make this work if the exe is an Executable target.
            raise InterpreterException('Second argument must be an external program.')


Need to remove FIXME and update the exception message.

nirbheek · 2018-08-19T15:58:49Z

mesonbuild/dependencies/base.py

@@ -953,6 +953,9 @@ def __repr__(self):
        r = '<{} {!r} -> {!r}>'
        return r.format(self.__class__.__name__, self.name, self.command)

+    def __str__(self):
+        return ' '.join(self.command)


Why was this implemented? We do not allow implicit str() conversions for any objects.

nirbheek · 2018-08-19T15:58:58Z

mesonbuild/build.py

@@ -1304,6 +1304,9 @@ def __init__(self, name, subdir, subproject, is_cross, sources, objects, environ
        # Only linkwithable if using export_dynamic
        self.is_linkwithable = self.export_dynamic

+    def __str__(self):
+        return self.name


Why was this implemented? We do not allow implicit str() conversions for any objects.

nirbheek · 2018-08-19T16:00:06Z

mesonbuild/interpreter.py

@@ -803,6 +803,9 @@ class ExecutableHolder(BuildTargetHolder):
    def __init__(self, target, interp):
        super().__init__(target, interp)

+    def found(self):
+        return True


Don't implement this method on ExecutableHolder, see below.

nirbheek · 2018-08-19T16:08:53Z

Besides the changes in the review comments, as @textshell said:

We need to ensure that the backends know that executables must be added to dependencies for the custom target or generator that uses that ExternalProgramHolder, else we'll get build errors due to parallelization.
We need solve the edge-case of external programs that are called during configure with run_command() or configure_file().

An easy way to solve (2) is to error out when this happens. However, is that what people expect when the program was overriden but is also available on the system? Probably not. Subproject setups would be prone to breakage too.

A better way to fix this is to make this feature an opt-in. i.e., find_program() will only use Executable overrides if you pass it a kwarg, like so:

find_program('foo', allow_built_executable_overrides: true)

The name is a placeholder, bikeshedding welcome.

Without that kwarg set, find_program() would only look for overrides that are available at configure time (system binaries and configure_file()).

textshell · 2018-08-19T16:25:28Z

I don't think having overrides that only work with a special parameter is the way to go. I think having a override that is ignored will very likely just break in strange and hard to diagnose ways, let's try not to go there.

So i prefer always returning these and erroring out when they are used synchronously. Also keep in mind, that one important use case is for these overridden programs to be used by meson modules. For example a protobuf module in the future or the qt modules (when it's adopted to look there).

I think the most important synchronous use of this kind of program is checking version requirements. That is something we need to handle in the long run. For overrides i think the only way i can imagine working is to attach metadata with the override. Yes, this means projects that want to support this use case need special code, but that's still better than what i can imagine as alternatives.

e.g.

meson.override_find_program('moc', moc_exe, version: '5.11.0')

and later

moc = find_program('moc')
if moc.overridden()
  if moc.version().version_compare('<5.3')
     error('something')
  endif
else
  # traditional check
endif

Of course the metadata bit can be implemented in a different PR. But i believe it's part of the process of how we want to design this.

nirbheek · 2018-08-19T16:54:21Z

I don't think having overrides that only work with a special parameter is the way to go. I think having a override that is ignored will very likely just break in strange and hard to diagnose ways, let's try not to go there.

Why do you believe that? We already do caching and detection based on kwargs for dependency(), and this is far less complicated than that.

So i prefer always returning these and erroring out when they are used synchronously.

Let's take the following situation.

subproject/foo does:

meson.override_find_program('foobar', ...)`

subproject/bar does:

foobar = find_program('foobar')
ct = custom_target(..., command : [foobar, ...])

subproject/baz does:

foobar = find_program('foobar')
f = configure_file(..., command : [foobar, ...])

foobar is also available in PATH. You update foo and although bar keeps working, baz breaks and you have to stop using find_program to fix baz.

This is a pretty shitty situation.

If we don't want to make it opt-in, we should at least make it opt-out.

I think the most important synchronous use of this kind of program is checking version requirements.

That's #1609, and yes we should implement it.

nirbheek · 2018-08-19T16:56:35Z

mesonbuild/interpreter.py

@@ -1753,7 +1756,7 @@ def override_find_program_method(self, args, kwargs):
            if not os.path.exists(abspath):
                raise InterpreterException('Tried to override %s with a file that does not exist.' % name)
            exe = dependencies.ExternalProgram(abspath)
-        if not isinstance(exe, dependencies.ExternalProgram):
+        if not isinstance(exe, (dependencies.ExternalProgram, build.Executable)):


Need to handle the case where Executable is actually cross-built. In that case we'll want to run it with the exe wrapper, or error out if an exe wrapper is not found.

textshell · 2018-08-19T17:37:46Z

I don't think having overrides that only work with a special parameter is the way to go. I think having a override that is ignored will very likely just break in strange and hard to diagnose ways, let's try not to go there.

Why do you believe that? We already do caching and detection based on kwargs for dependency(), and this is far less complicated than that.

It's not about the complexity of implementing it. It's about keeping things understandable. I've often heard the mantra that goes along the lines if "if you have two different things with the same name they will get confused and leave a big mess". I believe mixing a "system install" and a subproject provided version will lead down to such a mess.

So i prefer always returning these and erroring out when they are used synchronously.
Let's take the following situation.
[...]

foobar is also available in PATH. You update foo and although bar keeps working, baz breaks and you have to stop using find_program to fix baz.

This is a pretty shitty situation.

Maybe. But it's clear what happens. What if the program from the system path produces a file that is incompatible to the generated files in the other part of the build or the library also produced by the subproject. That is very hard to diagnose and is at least as bad. Or if you add a subproject with new version of a library to test if that library works and "random" parts of your build don't use that override, causing you to release a broken version? I prefer breaking the build in obvious ways to breaking it in subtle ways.

If we don't want to make it opt-in, we should at least make it opt-out.

I don't think a subproject can ever have the needed situation awareness to make a sensible decision here. That might be more reasonable in a global override infrastructure.

I think the most important synchronous use of this kind of program is checking version requirements.

That's #1609, and yes we should implement it.

I don't think we should tie this into a 80% percent solution without escape hatch. But maybe a .overridden() style query would be enough to reuse #1609. Or maybe an improved version that also allows for version introspection (meson modules at least routinely work around quirks in older versions)

* Don't use __str__. * Use ExternalProgramHolder for holding an executable. * Add tests of the dependencies. * Mark executable in test as native. * Error if an executable is used with run_command. I went with an error on the last item because I think that there is no non error behavior that is always correct. If a program is not available on the system, we would need an error one way or the other. The interesting case then is when the program is available on the system but is overridden anyway. I would assume that this would normally happens because the one available on the system is too old. Should we use one version during configuration and another one during the build? That sounds dangerous as there is no guarantee that the two versions are compatible. It also seems inconsistent with the fact that we error if a program is overridden after a find_program (see test cases/failing/73 override used). BTW, is there any way to test which error caused a test in the failing subdirectory to fail?

espindola · 2018-08-20T21:02:43Z

I think commit 4303e4e addresses all the requested changes.

textshell · 2018-08-20T21:29:49Z

mesonbuild/interpreter.py

@@ -2152,6 +2151,8 @@ def run_command_impl(self, node, args, kwargs, in_builddir=False):
            'or configure_file(), or a compiler object; not {!r}'
        if isinstance(cmd, ExternalProgramHolder):
            cmd = cmd.held_object
+            if isinstance(cmd, build.Executable):
+                raise InterpreterException('Cannot use an executable during configuration')


We need to help our users more here. The error message at least needs to say that the executable was overriden and that as it is built it is not available for execution in the build file.

textshell · 2018-08-20T21:35:03Z

mesonbuild/build.py

@@ -1304,6 +1304,9 @@ def __init__(self, name, subdir, subproject, is_cross, sources, objects, environ
        # Only linkwithable if using export_dynamic
        self.is_linkwithable = self.export_dynamic

+    def desc(self):


I think this name is a bit on the cryptic side. We should try to come up with a more descriptive name here.

* Improve function name and add docstring. * Improve error message to say which program was overridden by which executable.

espindola · 2018-08-22T04:27:48Z

bafd632 should address the last round of requests.

espindola · 2018-08-27T15:17:01Z

ping?

textshell · 2018-08-27T18:50:17Z

As this is a behavior change this needs to be acked by @jpakkane

I don't know these parts of meson well enough to say if this is technically all good now. Assuming this gets acked, what is missing is an changelog snippet and documentation.

jpakkane · 2018-08-27T19:25:48Z

The common approach we do is that when something is unclear we detect it and make it a hard error and only add something later based on real world use cases. This needs the documentation fixes mentioned above but once they are done, this is ok to merge.

jpakkane · 2018-08-29T18:03:05Z

docs/markdown/Release-notes-for-0.48.0.md

+
+This is particularly useful for fallback dependencies like protobuf
+that also provide a tool like protoc.
+


This should not be in this file. It should be in the snippets directory as described in the chapter directly above this one.

jpakkane · 2018-09-03T18:36:20Z

Merged manually.

espindola · 2018-09-03T23:58:32Z

Thanks! Sorry for the slow response.

nirbheek requested changes Aug 19, 2018

View reviewed changes

nirbheek reviewed Aug 19, 2018

View reviewed changes

textshell reviewed Aug 20, 2018

View reviewed changes

Address review comments.

bafd632

* Improve function name and add docstring. * Improve error message to say which program was overridden by which executable.

Add documentation.

cee2960

jpakkane reviewed Aug 29, 2018

View reviewed changes

jpakkane closed this Sep 3, 2018

lealgo mentioned this pull request Jun 1, 2019

Android build with Cross Compilation for BLAS and OpenCL back-ends LeelaChessZero/lc0#848

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow override_find_program to use an executable. #4055

Allow override_find_program to use an executable. #4055

espindola commented Aug 19, 2018

textshell commented Aug 19, 2018

nirbheek Aug 19, 2018

nirbheek Aug 19, 2018

nirbheek Aug 19, 2018

nirbheek Aug 19, 2018

nirbheek Aug 19, 2018

nirbheek commented Aug 19, 2018

textshell commented Aug 19, 2018

nirbheek commented Aug 19, 2018

nirbheek Aug 19, 2018

textshell commented Aug 19, 2018

espindola commented Aug 20, 2018

textshell Aug 20, 2018

textshell Aug 20, 2018

espindola commented Aug 22, 2018

espindola commented Aug 27, 2018

textshell commented Aug 27, 2018

jpakkane commented Aug 27, 2018

jpakkane Aug 29, 2018

jpakkane commented Sep 3, 2018

espindola commented Sep 3, 2018


		This is particularly useful for fallback dependencies like protobuf
		that also provide a tool like protoc.

Allow override_find_program to use an executable. #4055

Allow override_find_program to use an executable. #4055

Conversation

espindola commented Aug 19, 2018

textshell commented Aug 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirbheek commented Aug 19, 2018

textshell commented Aug 19, 2018

nirbheek commented Aug 19, 2018

Choose a reason for hiding this comment

textshell commented Aug 19, 2018

espindola commented Aug 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

espindola commented Aug 22, 2018

espindola commented Aug 27, 2018

textshell commented Aug 27, 2018

jpakkane commented Aug 27, 2018

Choose a reason for hiding this comment

jpakkane commented Sep 3, 2018

espindola commented Sep 3, 2018