Current working directory-based config file loading?

[bitprophet]

This has come up a few times recently, though I can't find the ticket or discussion that I'm thinking of. But it affects me occasionally, and also some tests occasionally.

Background

Specifically, the "project" level config file: it's technically defined as "the config file living in the same directory as one's task namespace". In many cases, both files and the user are all hanging out in one's project / checkout / etc root directory, such as ~/code/myproject, and thus it's also often interpreted as "the config file that is closest to my CWD / the output of pwd / whatever."

However, right now, the latter isn't implemented anywhere; so in some edge cases folks get confused as to why things don't work right:

• CWD is in some "project" directory containing an expected-to-be-loaded conf file, but one is using inv -c othercollection, and othercollection lives in some other location on your PYTHONPATH. Conf file isn't loaded, sadness ensues.
• CWD contains both a conf file and the desired tasks collection...but the tasks collection is named nottasks.py (maybe there's a non-Invoke tasks on PYTHONPATH already) and one is expecting that the conf file, which states {'tasks': {'collection_name': 'nottasks'}}, will cause Invoke to load up nottasks.py. But it doesn't, since the file is a "project-level" conf file, whose loading is predicated on finding and loading the nearby collection; again, sadness ensues.
  • This is actually what I ran into in a test suite; I'm in the habit of using "project-level" conf files in support subdirectories to test out config overrides, but when I applied it to testing tasks.collection_name itself...well...that didn't go well.

Solutions

• Modify the project-level config file loading such that it considers CWD first and only then falls back to "local to collection".
  • Breaks corner case where one has a valid conf file in CWD and local to collection, as the latter would then be ignored.
  • Arguably, truly local-to-collection files are equally served by using collection-level config within the collection/tasks file, so maybe not a horrible change?
    • But that feels specious, I can see users legit preferring a "real" config file even in that case, and especially since it currently works and would then stop working, seems bad.
• Add a new config file level for this concept, say local or working_directory.
  • Arguably more work, but probably not by much
  • New feature, easy to understand, no backwards compat breakage
  • Adding new config system levels shouldn't be a huge problem anyway, that's the entire point

[jenisys]

I ran into similar issues with invoke-0.21.0 while trying out tasks.collection_name=mytasks (and tasks.search_root) from a "invoke.yaml" configuration file. Using invoke --debug -l and exploring the output, the configuration file "invoke.yaml" seems not to be considered or it does not become effective (especially when a "tasks.py" exists in a parent directory).

[pradyunsg]

Just a thought: It might be useful to walk up the CWD path to find the configuration. That way a user is not constrained to stay in the root of the project to call invoke.

Doing this does bring up the question of managing the CWD for the tasks... "Where the user is" and "where the configuration is" both seem reasonable choices. I'd say the second is better since it ensures your tasks work the same regardless of which directory inside your project you are.

eg: if the user is at /home/username/my-projects/project/subdir, invoke could check:

/home/username/my-projects/project/subdir/pyproject.toml
/home/username/my-projects/project/pyproject.toml
/home/username/my-projects/pyproject.toml
/home/username/pyproject.toml

Further, from #537, it's nice that pyproject.toml is expected to be in the root of the Python package. :P

[bitprophet]

That's exactly how the tasks loader works now, for what it's worth, so yea we'd likely reuse the same code and approach: http://docs.pyinvoke.org/en/1.0/concepts/loading.html#task-module-discovery

[dperetti]

I have an invoke.yaml like this (notice the doubt 😆):

tasks:
  search-root: ./test
  search_root: ./test

And invoking inv -l I expected Invoke to discover the tasks in ./test.
Can't get it to work.
Ss that what your ticket is about ? What's the workaround, then ? How can search_root be used ?