Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Update model store documentation #1346

Merged
merged 1 commit into from
Dec 12, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 81 additions & 52 deletions docs/ModelStore.md
Original file line number Diff line number Diff line change
@@ -1,72 +1,101 @@
# Model Store


## Introduction
Model Store lists models in user-specified servers.
Users can imports models from Model Store into DIGITS.


## Setting up environment variable
The configuration of Model Store requires one environment variable DIGITS_MODEL_STORE_URL to be set.
NVIDIA plans to publish one public Model Store at http://developer.download.nvidia.com/compute/machine-learning/modelstore/4.5.0.
You can set up the environment variable with that url before launching DIGITS.
For example, run the following command in your Bash shell.
``` shell
export DIGITS_MODEL_STORE_URL='http://developer.download.nvidia.com/compute/machine-learning/modelstore/4.5.0'
```
If multiple model stores are available, specify their url's, separated by the comma (,).
``` shell
export DIGITS_MODEL_STORE_URL='http://localhost/mymodelstore,http://dlserver/teammodelstore'
```

DIGITS 5.0 introduces the concept of a "model store," which is a collection of trained models that can be used as pre-trained weights to accelerate training convergence.
A DIGITS server can be configured to connect to one or more model stores to download these trained models from the store to the server.

## Browse and import model
First launch DIGITS.
Click Pretrained Models tab, and select 'Retrieve from Model Store' under 'Load Model.'
The new page shows models available in model stores.
By default, DIGITS is configured to connect to the official NVIDIA model store.
See instructions below for creating and connecting to your own store[s].

![List models](images/model-store-list.png)
## Usage

Hover the Note field to show complete text.
Enter keyword in 'Filter list by' to limit results.
Click 'Update model list' button will retrieve the latest model list (see limitation).
Click 'Import' will import that model into DIGITS (may takes a few seconds, depends on network speed).
From the DIGITS home page, click on "Pretrained Models", then "Load Model" > "Images" > "Retreive from Model Store".

After successfully importing the model, DIGITS redirects the browser to Home page.
![Home page](images/model-store/home.jpg)

![Imported models](images/model-store-import.png)
On the model store page, you will see a list of all the models available for download.
Click on the small download icon (first column) to download a model from the store to the DIGITS server.

The Pretrained Models table will show the newly imported model.
At this moment, you can use that imported model like other pretrained models.
![Official store](images/model-store/official.jpg)

After downloading a model, you can return to the homepage to see the list of pretrained models.
To make use of the downloaded model for training, select it as a "Pretrained Network" in the "New Model" form.

## Create your own Model Store server
You can host your Model Store via Python's built-in SimpleHTTPServer.
First, in shell, cd to your own Model Store.
Then run the following command to start server at port 8000.
```
python -m SimpleHTTPServer 8000
```

At the top directory, create a master.json file (if your server does not support Apache directory listing).
The following is a sample master.json file.
You are welcome to create your own model store and use it to share models with others.
Here we will explain how to use Python's built-in SimpleHTTPServer to do it, but you can easily do the same thing with Apache or Nginx or whatever else you like.

Create a new directory on the server you'll be using for your store.
Collect some snapshot tarballs from DIGITS training jobs that you'd like to use for your store.
Unzip each tarball into its own directory (each should have a file called `info.json`).
Finally, create a file in the top directory called `master.json`.
```sh
$ cat master.json
{
"msg": "Luke's Model Store",
"children": [
"lenet",
"alexnet",
"googlenet",
"detectnet"
]
}
```
{"msg":"This is my own model store server.", "children":["Model01","Model02"]}
Your directory structure should look something like this:
```sh
$ tree -F
.
├── alexnet/
│   ├── deploy.prototxt
│   ├── info.json
│   ├── labels.txt
│   ├── mean.binaryproto
│   ├── snapshot_iter_11310.caffemodel
│   ├── solver.prototxt
│   └── train_val.prototxt
├── detectnet/
│   ├── deploy.prototxt
│   ├── info.json
│   ├── mean.binaryproto
│   ├── original.prototxt
│   ├── snapshot_iter_19140.caffemodel
│   ├── solver.prototxt
│   └── train_val.prototxt
├── googlenet/
│   ├── deploy.prototxt
│   ├── info.json
│   ├── labels.txt
│   ├── mean.binaryproto
│   ├── snapshot_iter_33450.caffemodel
│   ├── solver.prototxt
│   └── train_val.prototxt
├── lenet/
│   ├── deploy.prototxt
│   ├── info.json
│   ├── labels.txt
│   ├── mean.binaryproto
│   ├── snapshot_iter_28140.caffemodel
│   ├── solver.prototxt
│   └── train_val.prototxt
└── master.json
```
Model01 and Model02 are subdirectories containing the actual models.
Each model must at least include one info.json file, one weight file and one model file.
The info.json file is in the format of same file inside DIGITS downloaded model.
The subdirectory can optionally contain aux.json, which points to files not directly from DIGITS.
The following is a sample aux.json file.
Finally, use Python to start a server to serve these files
```sh
$ python -m SimpleHTTPServer
```
{"license": "3-clause BSD license", "logo": "logo.png", "dataset":"MNIST"}
Now, restart your DIGITS server and configure it to connect to your new server:
```sh
$ DIGITS_MODEL_STORE_URL=http://localhost:8000 ./digits-devserver
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add one more example to show how to configure multiple model store servers? Something like DIGITS_MODEL_STORE_URL="http://localhost:8000, http://newstore:8080"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

```
If `"license"` is defined, it will be shown on license field of that model.
The `license.txt` inside that subdirectory will be displayed when users clicking the license name.
`"logo"` is optional and once defined, DIGITS will display `logo.png` from the same subdirectory.
To use multiple model stores, just separate the URLs with a comma:
```sh
$ DIGITS_MODEL_STORE_URL="http://developer.download.nvidia.com/compute/machine-learning/modelstore/5.0,http://localhost:8000" ./digits-devserver
```

> NOTE: If you have installed DIGITS with a deb package, see [UbuntuInstall.md](UbuntuInstall.md) for instructions about how to reconfigure and restart your server.
When you have configured everything properly, you should see something like this:

## Limitation
Some web server may limit frequent requests from the same machine to stop malicious activities.
Therefore, DIGITS implemented a cache mechanism to reduce server-to-server communication.
The button, 'Update model list', invalidates cache and retrieve meta data from all models.
![Custom store](images/model-store/custom.jpg)
Binary file removed docs/images/model-store-import.png
Binary file not shown.
Binary file removed docs/images/model-store-list.png
Binary file not shown.
Binary file added docs/images/model-store/custom.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/model-store/home.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.