Skip to content

Commit

Permalink
Improve ML how-to guides
Browse files Browse the repository at this point in the history
  • Loading branch information
@npentrel
npentrel committed Oct 20, 2024
1 parent f2b87dd commit ed40114
Show file tree
Hide file tree
Showing 5 changed files with 109 additions and 122 deletions.
83 changes: 34 additions & 49 deletions docs/how-tos/create-custom-training-scripts.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@ viamresources: ["mlmodel", "data_manager"]
platformarea: ["ml"]
level: "Advanced"
date: "2024-08-29"
# updated: "" # When the tutorial was last entirely checked
updated: "2024-10-20" # When the tutorial was last entirely checked
cost: "0"
---

You can create your own custom Python training script that trains ML models to your specifications using the Machine Learning framework of your choice (PyTorch, Tensorflow, TFLite, ONNX, or any other framework).
Once added to the [Viam Registry](https://app.viam.com/registry?type=Training+Script), you can use the training script to build ML models based on your datasets.
You can create your own custom Python training script that trains ML models to your specifications using PyTorch, Tensorflow, TFLite, ONNX, or any other Machine Learning framework.
Once you upload your training script to the [Viam Registry](https://app.viam.com/registry?type=Training+Script), you can use it to build ML models in the Viam Cloud based on your datasets.

{{< alert title="In this page" color="tip" >}}

Expand All @@ -35,7 +35,9 @@ Once added to the [Viam Registry](https://app.viam.com/registry?type=Training+Sc

{{% expand "A dataset with data you can train an ML model on. Click to see instructions." %}}

Follow the instructions to [Create a dataset and label data](/how-tos/train-deploy-ml/#create-a-dataset-and-label-data) to create a dataset.
For image data, you can follow the instructions to [Create a dataset and label data](/how-tos/train-deploy-ml/#create-a-dataset-and-label-data) to create a dataset.

For other data you can use the [Data Client API](/appendix/apis/data-client/) to get data stored in the Viam Cloud.

{{% /expand%}}

Expand Down Expand Up @@ -67,7 +69,7 @@ my-training/
{{% tablestep %}}
**2. Add `setup.py` code**

Add the following code to `setup.py`:
Add the following code to `setup.py` and dd additional required packages on line 11:

```python {class="line-numbers linkable-line-numbers" data-line="11"}
from setuptools import find_packages, setup
Expand All @@ -85,8 +87,6 @@ setup(
)
```

Ensure you add additional required packages on line 11.

{{% /tablestep %}}
{{% tablestep %}}
**3. Create `__init__.py`**
Expand Down Expand Up @@ -296,11 +296,12 @@ if __name__ == "__main__":

{{% /tablestep %}}
{{< tablestep >}}
<b>5. Template script parsing functionality explained</b>

<p>You do not need to edit the scripts parsing functionality but if you want to understand the script fully, click on the following expanders:</p>
<p><strong>5. Understand template script parsing functionality</strong></p>
<p>When a training script is run, the Viam platform passes the dataset file for the training and the designated model output directory to the script.</p>
<p>The template contains functionality to parse the command line inputs and parse annotations from the dataset file.</p>

{{% expand "Parsing command line inputs" %}}
{{% expand "Click for more information on parsing command line inputs." %}}

The script you are creating must take the following command line inputs:

Expand All @@ -310,7 +311,8 @@ The script you are creating must take the following command line inputs:
The `parse_args()` function in the template parses your arguments.

{{% /expand %}}
{{%expand "Parsing annotations from dataset file" %}}

{{% expand "Click for more information on parsing annotations from dataset file" %}}

When you submit a training job to the Viam Cloud, Viam will pass a `dataset_file` to the training script when you train an ML model with it.
The file contains metadata from the dataset used for the training, including the file path for each data point and any annotations associated with the data.
Expand Down Expand Up @@ -387,6 +389,8 @@ Depending on if you are training a classification or detection model, the templa

{{% /expand%}}

<p>If the script you are creating does not use an image dataset, you only need the model output directory.</p>

{{% /tablestep %}}
{{% tablestep %}}
**6. Add logic to produce the model artifact**
Expand All @@ -412,7 +416,7 @@ As an example, you can refer to the logic from <file>model/training.py</file> fr
{{% tablestep %}}
**8. Update the main method**

Update the main to which calls the functions you have just created.
Update the main to call the functions you have just created.

{{% /tablestep %}}
{{% tablestep %}}
Expand Down Expand Up @@ -459,7 +463,7 @@ Use the `parse_filenames_and_labels_from_json` and `parse_filenames_and_bboxes_f
{{% tablestep %}}
**2. Run your training script locally**

Install any required dependencies and run your training script specifying the path to the <dataset.jsonl> file from your exported dataset:
Install any required dependencies and run your training script specifying the path to the <FILE>dataset.jsonl</FILE> file from your exported dataset:

```sh {class="command-line" data-prompt="$" data-output="1-10"}
python3 -m model.training --dataset_file=/path/to/dataset.jsonl --model_output_directory=.
Expand All @@ -476,8 +480,7 @@ To be able to use your training script in the Viam platform, you must upload it
{{% tablestep %}}
**1. Package the training script as a <file>tar.gz</file> source distribution**

To run your training script on datasets in Viam, compress your project folder into a tar.gz file.
You can run this command to create a .tar.gz archive from your project folder:
To run your training script on datasets in Viam, compress your project folder into a tar.gz file:

```sh {class="command-line" data-prompt="$" data-output="1-10"}
tar -czvf my-training.tar.gz my-training/
Expand All @@ -489,11 +492,9 @@ You can refer to the directory structure of this [example classification trainin

{{% /tablestep %}}
{{% tablestep %}}
**2. Upload a new training script (or a new version)**

To upload a custom training script to the registry, use the [`viam training-script upload`](/cli/#training-script) command.
**2. Upload a training script**

For example:
To upload a custom training script to the registry, use the `viam training-script upload` command.

{{< tabs >}}
{{% tab name="Usage" %}}
Expand Down Expand Up @@ -528,8 +529,8 @@ To find your organization's ID, run the following command:
viam organization list
```

After a successful upload, you'll receive a confirmation message with a link to view your changes online in the CLI.
Once uploaded, you can view the script by navigating to the [registry's **Training Scripts** page](https://app.viam.com/registry?type=Training+Script).
After a successful upload, the CLI displays a confirmation message with a link to view your changes online.
You can view uploaded training scripts by navigating to the [registry's **Training Scripts** page](https://app.viam.com/registry?type=Training+Script).

{{% /tablestep %}}
{{< /table >}}
Expand All @@ -547,20 +548,17 @@ After uploading the training script, you can run it by submitting a training job

{{<imgproc src="/services/ml/train.svg" class="fill alignleft" style="width: 200px" declaredimensions=true alt="Train models">}}

In the Viam app, navigate to your list of [**DATASETS**](https://app.viam.com/data/datasets) and select the one you want to train on.
In the Viam app, navigate to your list of [**DATASETS**](https://app.viam.com/data/datasets) and select the one you want to train a model on.

Click **Train model** and select **Train on a custom training script**, then follow the prompts.

{{% /tab %}}
{{% tab name="CLI" %}}

You can use [`viam train submit custom from-registry`](/cli/#positional-arguments-submit) to submit a training job from a training script already uploaded to the registry or `viam train submit custom from-upload` to upload a training script and submit a training job at the same time.
You can use [`viam train submit custom from-registry`](/cli/#positional-arguments-submit) to submit a training job.

For example:

{{< tabs >}}
{{% tab name="from-registry" %}}

```sh {class="command-line" data-prompt="$"}
viam train submit custom from-registry --dataset-id=<INSERT DATASET ID> \
--org-id=<INSERT ORG ID> --model-name=MyRegistryModel \
Expand All @@ -570,27 +568,7 @@ viam train submit custom from-registry --dataset-id=<INSERT DATASET ID> \

This command submits a training job to the previously uploaded `MyCustomTrainingScript` with another input dataset, which trains `MyRegistryModel` and publishes that to the registry.

{{% /tab %}}
{{% tab name="with-upload" %}}

```sh {class="command-line" data-prompt="$"}
viam train submit custom with-upload --dataset-id=<INSERT DATASET ID> \
--model-org-id=<INSERT ORG ID> --model-name=MyRegistryModel \
--model-type=single_label_classification --model-version=2 \
--version=1 --path=<path-to-tar.gz> \
--script-name=mycompany:MyCustomTrainingScript
```

This command uploads a script called `MyCustomTrainingScript` to the registry under the specified organization and also submits a training job to that script with the input dataset, which generates a new version of the single-classification ML model `MyRegistryModel` and publishes that to the registry.

To find the dataset ID of a given dataset, go to the [**DATASETS** subtab](https://app.viam.com/data/datasets) of the **DATA** tab on the Viam app and select a dataset.
Click **...** in the left-hand menu and click **Copy dataset ID**.

To find your organization's ID, navigate to your organization's **Settings** page in [the Viam app](https://app.viam.com/).
Find **Organization ID** and click the copy icon.

{{% /tab %}}
{{< /tabs >}}
You can get the dataset id from the dataset page or using the [`viam dataset list`](/cli/#dataset) command.

{{% /tab %}}
{{< /tabs >}}
Expand All @@ -599,7 +577,7 @@ Find **Organization ID** and click the copy icon.
{{% tablestep %}}
**2. Check on training job process**

Once submitted, you can view your training job on the **DATA** page's [**TRAINING** tab](https://app.viam.com/training).
You can view your training job on the **DATA** page's [**TRAINING** tab](https://app.viam.com/training).

Once the model has finished training, it becomes visible on the **DATA** page's [**MODELS** tab](https://app.viam.com/data/models).

Expand Down Expand Up @@ -630,6 +608,13 @@ You can also view your training jobs' logs with the [`viam train logs`](/cli/#tr

## Next steps

To use your new model with machines, you must deploy it with the [ML model service](/services/ml/deploy/).
Then you can use another service, such as the vision service, to apply the deployed model to any live data, such as camera feeds.

To see models in use with machines, see one of the following resources:

{{< cards >}}
{{% card link="/how-tos/train-deploy-ml/" %}}
{{% card link="/how-tos/detect-people/" %}}
{{% card link="/tutorials/projects/helmet/" %}}
{{% card link="/tutorials/projects/integrating-viam-with-openai/" %}}
{{< /cards >}}
80 changes: 38 additions & 42 deletions docs/how-tos/detect-color.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "Detect Color with a Webcam"
title: "Detect color with a webcam"
linkTitle: "Detect Color"
type: "docs"
description: "Detect colors using a webcam and the Viam vision service. Without writing code, you will be able to view your camera stream, with detection bounding boxes."
description: "Detect colors and their location in an image with any webcam and a vision service."
imageAlt: "camera stream displaying a color detection"
images: ["/tutorials/try-viam-color-detection/detected-example.png"]
tags: ["vision", "detector", "camera", "services"]
Expand All @@ -18,31 +18,39 @@ viamresources: ["vision", "camera"]
platformarea: ["ml"]
level: "Beginner"
date: "2022-12-16"
updated: "2024-09-03"
updated: "2024-10-20"
cost: "0"
# SMEs: Hazal
---

In this guide you will detect the color red using your computer's webcam and the Viam vision service.
Without writing any code, you will be able to view your camera stream, with detection bounding boxes, from the Viam app control interface.
WebRTC is a powerful technology that allows developers to build apps with video streams.
Adding computer vision allows machines to analyze images and gain meaningful information from video streams.
You can then program the machines to act based on this data, for example by alerting you.

Imagine a factory's storage unit.
To get an idea of stock levels there are cameras, so someone can view the camera feeds to get stock levels rather than having to check in person.

Computer vision let's us do even better.
The factory adds red paint to the walls of your factory at level where they need to restock.
Now, a computer can monitor the live stream of the stock levels and as soon as the red color becomes visible, it can alert a supervisor.

This guide will show you how to use any webcam alongside a computer to detect the color red with the vision service.

{{< alert title="You will learn" color="tip" >}}

- How to create a machine and install `viam-server`
- How to configure a webcam
- How to use the color detection vision service

{{< /alert >}}

## Requirements

You don't need to buy or own any hardware to complete this tutorial.
You don't need to buy or own any hardware to follow along.
If you have the following components, you can follow along on your own hardware:

- A Linux, macOS or WSL computer that can run `viam-server`.
- A Linux or macOS computer that can run `viam-server`.
- A webcam: this could be the webcam on your laptop or any other webcam you can connect to your computer.

Make sure to connect the webcam to your computer (if it's not built-in) before starting this guide.

{{% expand "No computer or webcam?" %}}
No problem.
Use [Try Viam](https://app.viam.com/try) to borrow a rover free of cost online.
Expand All @@ -54,6 +62,7 @@ Now you know what the rover can perceive.

If your rover is facing a wall, find the base configuration panel and click on its **Test** panel.
Use the controls to drive your rover to a different location.
You can use picture in picture mode on one of the cameras so you can see where you're driving.

Now that you have seen that the cameras on your Try Viam rover work, **continue with Step 4**.

Expand All @@ -63,16 +72,10 @@ Be aware that if you are running out of time during your rental, you can [extend

{{% /expand%}}

{{% expand "Have your own rover?" %}}

If you are running this tutorial on [your own Viam Rover](/appendix/try-viam/rover-resources/), make sure you have physically set up and [configured your rover](/appendix/try-viam/rover-resources/rover-tutorial-fragments/).
Go to the **CONFIGURE** tab of the machine, find the camera and click on the **Test** panel at the bottom of the camera's configuration panel to test the camera stream.

{{% /expand%}}

## Instructions

Follow these instructions to configure your machine and test detecting colors:
To use Viam with your device, you must install Viam and create a configuration that describes the connected camera.
Then you can add the vision service to detect colors from your camera's live feed.

{{% expand "Step 1: Create a new machine" %}}

Expand Down Expand Up @@ -100,16 +103,16 @@ Enter a name or use the suggested name for your camera and click **Create**.
Click the **Save** button in the top right corner of the page to save your configuration.
Then click on the **Test** panel at the bottom of the camera's configuration panel to test the camera stream.

If you don't see an image stream, you need to [configure the `video_path` attribute](/components/camera/webcam/#using-video_path).
If you don't see an image stream, use the video path dropdown to select your camera path.

For more detailed information, including optional attribute configuration, see the [`webcam` docs](/components/camera/webcam/).
For more detailed configuration information and troubleshooting, see the [`webcam` docs](/components/camera/webcam/).

{{% /expand %}}
{{% expand "Step 4: Configure a color detection vision service" %}}

The [vision service](/services/vision/) enables a robot to use its cameras to see and interpret the world around it.
The service also allows you to create different types of detectors with which the robot can recognize objects, scan QR codes, perform optical quality inspections, and more.
The `color_detector` model of the vision service is a heuristic-based detector that draws boxes around objects according to their hue.
The vision service enables a machine to use its cameras to see and interpret the world around it.
The service also allows you to create different types of detectors with which the machine can recognize objects, scan QR codes, perform optical quality inspections, and more.
The `color_detector` model of the vision service is a heuristic-based detector that detects colors and their location within an image.

In this guide, we use the color `#7a4f5c` or `rgb(122, 79, 92)` (a reddish color).

Expand All @@ -120,7 +123,7 @@ In this guide, we use the color `#7a4f5c` or `rgb(122, 79, 92)` (a reddish color
{{< tabs >}}
{{% tab name="Builder" %}}

On your machine's **CONFIGURE** tab, add a `vision` **Service** and select the `color detector` model.
On your machine's **CONFIGURE** tab, add a `vision` service and select the `color detector` model.

In the resulting vision service panel, click the color picker box to set the color to be detected.
For this tutorial, set the color to `rgb(122, 79, 92)` or use hex code `#7a4f5c`.
Expand Down Expand Up @@ -156,11 +159,7 @@ Add the vision service object to the services array in your rover’s JSON confi
{{% /tab %}}
{{< /tabs >}}

Click the **Save** button in the top right corner of the page to save your vision service configuration.

{{< alert title="Tip" color="tip" >}}
If you want to detect other colors, change the color parameter.
Object colors can vary dramatically based on the light source.
{{< alert title="Colors can vary dramatically based on the light source" color="tip" >}}
We recommend you verify the desired color detection value under the lighting conditions your machine will be in when in use.

To determine the color value from the camera stream, you can use a pixel color tool, like [Color Picker for Chrome](https://chrome.google.com/webstore/detail/color-picker-for-chrome/clldacgmdnnanihiibdgemajcfkmfhia).
Expand All @@ -171,13 +170,16 @@ Note that the detector does not detect black, perfect greys (greys where the red
{{% /expand %}}
{{% expand "Step 5: Test color detection" %}}

Now, test your color detection in the **Test** section of the computer vision service's configuration panel or on the **CONTROL** tab.
Click the **Save** button in the top right corner of the page to save your vision service configuration.

Now, test your color detection in the **Test** section of the computer vision service's configuration panel.

You will see your camera stream and see detections as labeled boxes on the images along with labels and confidence data:

![Detection of the color rose](/services/vision/rose-detection.png)

If the color is not reliably detected, change the color detectors configuration and increase the **Hue Tolerance** or decrease the segment size.
If the color is not reliably detected, change the color detectors configuration.
Increase the **Hue Tolerance** or decrease the segment size.

{{% /expand %}}
{{% expand "(Optional) Step 6: Limit the number of detections" %}}
Expand All @@ -191,18 +193,12 @@ You must close and reopen the panel for the new configuration to take effect.

## Next steps

You can now detect colors on a camera stream.
Of course these detections are not just accessible from the Viam app, but you can also use the [vision service API](/appendix/apis/services/vision/#api).

You can also use the vision service with Machine Learning models:
You can now detect colors on a camera stream using any device and any webcam.
You can also use the vision service with more sophisticated Machine Learning models.
To learn more about how to access the data from a vision service programmatically or use machine learning models with a vision service, see:

{{< cards >}}
{{% card link="/appendix/apis/services/vision/" customTitle="Vision Service API" %}}
{{% card link="/how-tos/detect-people/" %}}
{{% card link="/tutorials/projects/send-security-photo/" %}}
{{< /cards >}}

To learn about coding with Viam's SDKs, try [making a rover move in a square](/how-tos/drive-rover/).

{{< cards >}}
{{% card link="/how-tos/drive-rover/" %}}
{{% card link="/tutorials/projects/helmet/" %}}
{{< /cards >}}
Loading

0 comments on commit ed40114

Please sign in to comment.