From 2eff6f77730d5c00b1ee5dc02324ada809828460 Mon Sep 17 00:00:00 2001 From: Di Wu Date: Tue, 13 Jul 2021 03:04:55 +0000 Subject: [PATCH] doc update --- docs/en_US/hpo_benchmark.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/en_US/hpo_benchmark.rst b/docs/en_US/hpo_benchmark.rst index 60370f51db..67eebacb8e 100644 --- a/docs/en_US/hpo_benchmark.rst +++ b/docs/en_US/hpo_benchmark.rst @@ -4,18 +4,19 @@ HPO Benchmarks .. toctree:: :hidden: - HPO Benchmark Example Statistics + HPO Benchmark Example Statistics -We provide a benchmarking tool to compare the performances of tuners provided by NNI (and users' custom tuners) on different tasks. The implementation of this tool is based on the automlbenchmark repository (https://github.com/openml/automlbenchmark), which provides services of running different *frameworks* against different *benchmarks* consisting of multiple *tasks*. The tool is located in ``examples/trials/benchmarking/automlbenchmark``. This document provides a brief introduction to the tool and its usage. +We provide a benchmarking tool to compare the performances of tuners provided by NNI (and users' custom tuners) on different +types of tasks. This tool uses the `automlbenchmark repository `_ to run different *benchmarks* on the NNI *tuners*. +The tool is located in ``examples/trials/benchmarking/automlbenchmark``. This document provides a brief introduction to the tool and its usage. Terminology ^^^^^^^^^^^ - * **task**\ : a task can be thought of as (dataset, evaluator). It gives out a dataset containing (train, valid, test), and based on the received predictions, the evaluator evaluates a given metric (e.g., mse for regression, f1 for classification). * **benchmark**\ : a benchmark is a set of tasks, along with other external constraints such as time and resource. * **framework**\ : given a task, a framework conceives answers to the proposed regression or classification problem and produces predictions. Note that the automlbenchmark framework does not pose any restrictions on the hypothesis space of a framework. In our implementation in this folder, each framework is a tuple (tuner, architecture), where architecture provides the hypothesis space (and search space for tuner), and tuner determines the strategy of hyperparameter optimization. -* **tuner**\ : a tuner or advisor defined in the hpo folder, or a custom tuner provided by the user. +* **tuner**\ : a `tuner or advisor `_ provided by NNI, or a custom tuner provided by the user. * **architecture**\ : an architecture is a specific method for solving the tasks, along with a set of hyperparameters to optimize (i.e., the search space). In our implementation, the architecture calls tuner multiple times to obtain possible hyperparameter configurations, and produces the final prediction for a task. See ``./nni/extensions/NNI/architectures`` for examples. Note: currently, the only architecture supported is random forest. The architecture implementation and search space definition can be found in ``./nni/extensions/NNI/architectures/run_random_forest.py``. The tasks in benchmarks "nnivalid" and "nnismall" are suitable to solve with random forests.