Can't find documentation for calling dvc from python module.

[JoeyCarson]

One of my utilities pulls in data from various sources and builds a large hierarchy where versioning may or may not be required. Ideally, the hierarchy needs to allow for users to simply changes files in certain directories from time to time and version the appropriate subdirectory of the data.

At first I wanted to have a simple command to track changes, e.g. dvc add rootdir. But that seems to require reindexing of the entire hierarchy, which is not suitable for my use case. Ideally, data import process would dvc add the subdirectory itself when creating it. So I'd like to do that from python.

Otherwise, it would be useful if I can somehow avoid reindexing the whole hierarchy by running dvc add at the root.

As a feature request, I would ask for either some better documentation to describe how to achieve this approach, or documentation for calling dvc API from python (without running shell commands) in order to better work with this style of data organization.

[shcheklein]

@JoeyCarson the API is not public yet, though we already have users and it should stay more or less stable.

Check the dvc.Repo class. It exposes all the commands, including dvc add.

It might look something along those lines:

from dvc.repo import Repo

repo = Repo()
repo.add(
               "test",
                recursive=False, <-- optional
                no_commit=False, <-- optional
                fname="test", <-- optional
            )

Closing this since it's a question, not an issue. Please feel free to leave any additional comments if something is not working for you.

[efiop]

For the record: our first line of public API will be documented per this ticket iterative/dvc.org#463

[JoeyCarson]

Thanks for the prompt response folks. This seems simple enough. Perhaps you could recommend a good strategy in terms of versioning the individual directories in a hierarchy.

For instance, consider that I run os.walk from the bottom up of a directory hierarchy, meaning that I will dvc add a directory full of items, and then add that directory's parent directory. Would dvc allow this and would it attempt to reindex the child directory again? Or would it instead somehow point to the child directory's .dvc file?

[efiop]

@JoeyCarson No, it won't allow it, as outputs of your stages will overlap that way. Do you need to ignore some dirs? If so, have you considered .dvcignore functionality?