(How) To handle or not to handle environments

[daavoo]

Background

#5758 /#5758 (comment)

Hola!

Handling environments for DVC projects has been a necessity in my team since we started to adopt DVC; maybe this is only because of our infrastructure setup which is something like the Shared Development Server and the nature of our work which requires collaborating for training several different models for simultaneous projects.

I have always been curious about the lack of documentation/integrations for this matter and we have been iterating around different tools to complement DVC. I assume that this might be related with what @dberenbaum said:

This seems to challenge a core philosophy underneath DVC: that code and data are sufficient for reproducibility.

In our experience, the environment is the third key component needed for proper reproducibility but this debate might be out of scope.

---

Questions

To formalize the discussion I would like to know other opinions about two main questions:

• What is the proper way to specify in which environment to run DVC? / Should new features need to be implemented inside DVC for this?

We have multiple docker and conda environments in our shared development server, we are currently installing copies of DVC on each one of them because, as far as we know, there is no way to indicate DVC where to look for the python or other executables used in the stages. Keeping all DVC versions sync and updated is kind of a source of problems.

• What is the proper way to track an environment used in a DVC project? / Should new features need to be implemented inside DVC for this?

As stated before, we treat the environment like code and data in terms of reproducibility, we want to re-run a pipeline if a dependency in the environment has been changed. We are currently relying on complementary/alternative tools for this. We have been also exploring different hacky ways like treating the environment definition file as a dependency for each stage inside dvc.yaml

[dberenbaum]

What is the proper way to specify in which environment to run DVC? / Should new features need to be implemented inside DVC for this?

I'm reading back through the thread you linked and realized I misinterpreted @rogermparent in #5758 (comment). I like his suggestion that you can have a global dvc install and have project environments, which could be invoked in the dvc stage commands. Would that make sense for you?

How are you using dvc and different environments? If the different environments are part of the same project, then having a global dvc install makes sense. If they are for different projects, then tying them all to a single dvc version might create unwanted dependencies between them.

What is the proper way to track an environment used in a DVC project? / Should new features need to be implemented inside DVC for this?

See https://github.com/iterative/dvc.org/issues/2502 for a similar discussion.

[daavoo]

I'm reading back through the thread you linked and realized I misinterpreted @rogermparent in #5758 (comment). I like his suggestion that you can have a global dvc install and have project environments, which could be invoked in the dvc stage commands. Would that make sense for you?

I think @rogermparent suggestion is good and it would make sense for some use cases. However it won't quite work for ours, unless we re-define our "project environment" strategy. As far as I know, it's not currently possible to try that "global dvc install that invokes environments" approach, right?

How are you using dvc and different environments? If the different environments are part of the same project, then having a global dvc install makes sense. If they are for different projects, then tying them all to a single dvc version might create unwanted dependencies between them.

We have different projects with different environments and so we consider dvc as just another dependency for each "project environment". A global dvc install would require quite a few changes as some "legacy" projects use dvc 1.X, for example.