Skip to content
This repository has been archived by the owner on Aug 10, 2024. It is now read-only.

GTO docs #199

Merged
merged 42 commits into from
Nov 23, 2022
Merged
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
45d5377
first drafts
aguschin Oct 24, 2022
5ba326e
yarn format
aguschin Oct 24, 2022
6a33a05
remove mlem copy-paste parts
aguschin Oct 24, 2022
faa1b53
write use-cases index page. add MR page in use-cases
aguschin Oct 24, 2022
f2f1527
updating limitations
aguschin Oct 24, 2022
217c675
add use-cases (forgot to commit), add todos to user-guide
aguschin Oct 24, 2022
7895da1
fix sidebar
aguschin Nov 1, 2022
803e7cf
Apply suggestions from code review
aguschin Nov 1, 2022
826f278
process feedback
aguschin Nov 1, 2022
e3b90bd
add Git tags format
aguschin Nov 1, 2022
eff9755
refine Git tags format
aguschin Nov 1, 2022
65f66ee
write page: artifacts.yaml metafile
aguschin Nov 1, 2022
129b17f
user-guide/{artifacts,config,downstream}
aguschin Nov 2, 2022
c596dc9
fix lint
aguschin Nov 2, 2022
9f42ace
fixes on feedback
aguschin Nov 2, 2022
931dac1
Apply suggestions from code review
aguschin Nov 10, 2022
76dd044
command-reference index page
aguschin Nov 10, 2022
c4485ae
merge main
aguschin Nov 10, 2022
f2e7af0
first version of command reference
aguschin Nov 10, 2022
0f5b833
lightweight fixes
aguschin Nov 10, 2022
9eab27f
update annotate, assign, check-ref
aguschin Nov 11, 2022
65c9f51
write all other commands
aguschin Nov 11, 2022
e5a75b3
Apply suggestions from code review
aguschin Nov 17, 2022
570b77a
return register to GS
aguschin Nov 17, 2022
d747e56
simplify the docs structure, hide 'why gto?'
aguschin Nov 17, 2022
21930be
shorten command-reference/index
aguschin Nov 17, 2022
0b25aec
hide next steps in gs
aguschin Nov 17, 2022
68f2b98
reveal why-gto section
aguschin Nov 17, 2022
cee3374
Merge branch 'main' into gto-docs
aguschin Nov 17, 2022
0c0931e
Update theme to 0.1.24 for args linker fix
rogermparent Nov 17, 2022
7da4618
Merge pull request #227 from iterative/gto-docs-theme-0.1.24
yathomasi Nov 18, 2022
f374516
fix links
aguschin Nov 21, 2022
6ddfb70
fix lint
aguschin Nov 21, 2022
0f70d82
remove some sections from sidebar
aguschin Nov 21, 2022
98a06c4
fix dvc link
aguschin Nov 21, 2022
44d9e2f
Merge branch 'main' into gto-docs
aguschin Nov 22, 2022
2fb378e
reset package.json and yarn.lock
aguschin Nov 22, 2022
d8fec09
fix some feedback about CI/CD in GS
aguschin Nov 22, 2022
892c1f7
fix some feedback about CI/CD in GS
aguschin Nov 22, 2022
2ab8a9b
Update content/docs/gto/get-started.md
jorgeorpinel Nov 22, 2022
8f9b6a3
Apply suggestions from code review
aguschin Nov 23, 2022
c7ae5ff
yarn format
aguschin Nov 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 85 additions & 0 deletions content/docs/gto/command-reference/annotate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# annotate

Create or update metainformation about an artifact.
aguschin marked this conversation as resolved.
Show resolved Hide resolved

## Synopsis

```usage
usage: gto annotate [-r <text>] [--type <text>] [--path <text>]
[-e] [--label <text>] [-d <text>] [--commit]
[--push] [-h]
name

arguments:
name Artifact name
```

## Description

The command adds the artifact annotation to `artifacts.yaml` file. Feel free to
modify the file directly!

For example,

```cli
$ gto annotate awesome-model \
--type model \
--path models/neural_network.h5 \
--label ml
--label cool
--description "This model is very cool"
```

will create

```yaml
awesome-model:
description: This model is very cool
labels:
- ml
- cool
path: models/neural_network.h5
type: model
```

This information can be later retrieved by running `gto describe` command.

<admon type="tip">

Don't forget to commit `artifacts.yaml` with Git to associate it with the latest
artifact version and stage in any copy of the repo.

</admon>

By default GTO saves artifact as `virtual`. Use the `--must_exist` flag to tell
GTO the artifact file is committed to Git.
jorgeorpinel marked this conversation as resolved.
Show resolved Hide resolved

<details>

### Virtual vs. Physical artifacts

- Physical files/directories are committed to the repo. When you create a new
version or assign a stage to it, Git guarantees that it's immutable -- you can
Comment on lines +57 to +62
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Minor or for a later follow-up: Ideally this deeper explanation would be in another section of the docs (in a guide, for example) and just briefly mentioned/explained here (linked to the guide).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move it to https://mlem.ai/doc/gto/user-guide#annotations-in-artifactsyaml ? Feel free to address any pending items here in #235.

return a year later and get the same artifact by providing a version.

- Virtual artifacts could be an external path (e.g. `s3://mybucket/myfile`) or a
local path to a metafile representing an externally stored artifact file (as
[with DVC](https://dvc.org/doc/start/data-management)). In this case, GTO
can't pin versions to a physical state of the artifact and guarantee its
immutability later, e.g. if `s3://mybucket/myfile` changes the registry won't
know it, nor have a way to recover the original file.

</details>

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--type <text>` - Artifact type
- `--path <text>` - Artifact path
- `-e`, `--must-exist` - Verify artifact is committed to Git
- `--label <text>` - Labels to add to artifact
- `-d <text>`, `--description <text>` - Artifact description
- `--commit` - Automatically commit changes due to this command (experimental)
- `--push` - Push created commit automatically (experimental) - will set
commit=True
- `-h`, `--help` - Show this message and exit.
61 changes: 61 additions & 0 deletions content/docs/gto/command-reference/assign.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# assign

Assign stage to specific artifact version.

## Synopsis

```usage
usage: gto assign [-r <text>] [--version <text>]
[--stage <text>] [-m <text>]
[--simple <text>] [--force] [--push] [--sr]
[-h]
name [ref]

arguments:
name Artifact name
[ref] Git reference to use
```

## Description

To assign an actionable stage for a specific artifact version use the same
`gto assign` command. Stages can mark the artifact readiness for a specific
consumer. You can plug in a real downsteam system via CI/CD or web hooks, e.g.
to redeploy an ML model.

```cli
$ gto assign awesome-model --version v0.0.1 --stage prod
Created git tag 'awesome-model#prod#1' that assigns stage to 'v0.0.1'
```

GTO creates a special Git tag in
[the standard format](/doc/gto/user-guide#git-tags-format).

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--version <text>` - If you provide REF, this will be used to name new version
- `--stage <text>` - Stage to assign
- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
[supported values: auto, true, false] [default: auto]
- `--force` - Create the Git tag even if it already exists and is in effect
- `--push` - Push created tag automatically (experimental)
- `--sr`, `--skip-registration` - Don't register a version at specified commit
- `-h`, `--help` - Show this message and exit.

## Examples

Assign artifact "nn" to "prod" at specific Git ref instead of supplying artifact
version (note that this will also register a version if it doesn't exist):

```cli
$ gto assign nn abcd123 --stage prod
```

Assign stage at specific Git ref and name the version explicitly (this assumes
that version was not registered yet):

```cli
$ gto assign nn abcd123 --version v1.0.0 --stage prod
```
36 changes: 36 additions & 0 deletions content/docs/gto/command-reference/check-ref.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# check-ref

Find out the artifact version registered/assigned with ref.

## Synopsis

```usage
usage: gto check-ref [-r <text>] [--json] [--name] [--version]
[--event] [--stage] [-h]
ref

arguments:
ref Git reference to analyze
```

## Description

You can use `gto check-ref` to interpret a Git tag:

```cli
$ gto check-ref -r build/example-gto churn#prod#3
✅ Stage "prod" was assigned to version "v3.0.0" of artifact "churn"
```

For machine-consumable format, use `--json` flag or output specific pieces of
information with `--name`, `--version`, `--stage` or `--event`.

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--json` - Print output in json format
- `--name` - Show artifact name
- `--version` - Output artifact version
- `--event` - Show event
- `--stage` - Show artifact stage
- `-h`, `--help` - Show this message and exit.
141 changes: 141 additions & 0 deletions content/docs/gto/command-reference/deprecate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# deprecate

Deprecate artifact, deregister a version, or unassign a stage.

## Synopsis

```usage
usage: gto deprecate [-r <text>] [-m <text>] [--simple <text>]
[--force] [-d] [--push] [-h]
name [version] [stage]

arguments:
name Artifact name
[version] Artifact version
[stage] Stage to unassign
```

## Description

The command supports three use cases:

```cli
# deprecate an artifact:
$ gto deprecate nn

# deprecate a version:
$ gto deprecate nn v0.0.1

# unassign a stage:
$ gto deprecate nn v0.0.1 prod
```

### Unassigning a stage

Sometimes you need to mark an artifact version no longer ready for a specific
consumer, and maybe signal a downstream system about this. You can use
`gto deprecate` for that:

```cli
$ gto deprecate awesome-model v0.0.1 prod
Created git tag 'awesome-model#prod!#2' that unassigns a stage from 'v0.0.1'
```

<details>

### Unassigning a stage: some details and options

GTO creates a special Git tag in
[the standard format](/doc/gto/user-guide#git-tags-format).

Note, that later you can create this stage again, if you need to, by calling
`$ gto assign` again.

You also may want to delete the git tag instead of creating a new one. This is
useful if you don't want to keep extra tags in you Git repo, don't need history
and don't want to trigger a CI/CD or another downstream system. For that, you
can use:

```cli
$ gto deprecate awesome-model v0.0.1 prod --delete
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'
To push the changes upstream, run:
git push origin awesome-model#prod#1 --delete
```

</details>

### Deregister a version

Sometimes you need mark a specific artifact version as a no longer ready for
usage. You could just delete a git tag, but if you want to preserve a history of
the actions, you may again use `gto deprecate`.

```cli
$ gto deprecate awesome-model v0.0.1
Created git tag '[email protected]!' that deregistered a version.
```

<details>

### Deregister a version: some details and options

If you want to deregister the version by deleting the Git tags itself, you could
use

```cli
$ gto deprecate awesome-model v0.0.1 --delete
Deleted git tag '[email protected]' that registered a version.
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
To push the changes upstream, run:
git push origin [email protected] awesome-model#prod#1 awesome-model#prod!#2 --delete
```

This includes all Git tags related to the version: a tag that registered it and
all tags that assigned stages to it.

</details>

### Deprecating an artifact

Sometimes you need to need to mark the artifact as "deprecated", usually meaning
it's outdated and will no longer be developed. To do this, you could run:

```cli
$ gto deprecate awesome-model
Created Git tag 'awesome-model@deprecated' that deprecates an artifact.
```

<details>

### Deprecating an artifact: some details and options

With `awesome-model@deprecated` Git tag the artifact will be considered
deprecated until you register a new version or assign a new stage to it after
the deprecation.

If you want to deprecate an artifact by deleting git tags, you'll need to delete
all of them for the artifact. You could do that with

```cli
$ gto deprecate awesome-model --delete
Deleted git tag '[email protected]' that registered a version.
Deleted git tag 'awesome-model#prod#1' that assigned a stage to 'v0.0.1'.
Deleted git tag 'awesome-model#prod!#2' that unassigned a stage to 'v0.0.1'.
To push the changes upstream, run:
git push origin [email protected] awesome-model#prod#1 awesome-model#prod!#2 --delete
```

</details>

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `-m <text>`, `--message <text>` - Message to annotate the Git tag with
- `--simple <text>` - Use simple notation, e.g. `rf#prod` instead of `rf#prod-5`
[supported values: auto, true, false] [default: auto]
- `--force` - Create the Git tag even if it already exists and is in effect
- `-d`, `--delete` - Delete the git tag(s) instead of creating the new one
- `--push` - Push created tag automatically (experimental)
- `-h`, `--help` - Show this message and exit.
38 changes: 38 additions & 0 deletions content/docs/gto/command-reference/describe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# describe

Display enrichments for an artifact.

## Synopsis

```usage
usage: gto describe [-r <text>] [--rev <text>] [--type] [--path]
[--description] [-h]
name

arguments:
name Artifact name
```

## Description

To get details about an artifact (from `artifacts.yaml`) use `gto describe`:

```cli
$ gto describe churn -r https://github.com/iterative/example-gto
{
"type": "model",
"path": "models/churn.pkl",
"virtual": false
}
```

The output is in JSON format for ease of parsing programatically.

## Options

- `-r <text>`, `--repo <text>` - Local or remote repository [default: .]
- `--rev <text>` - Repo revision to use
- `--type` - Show type
- `--path` - Show path
- `--description` - Show description
- `-h`, `--help` - Show this message and exit.
Loading