From 43e586d3f786ad3f79cf5b192c67929b37f2eb6d Mon Sep 17 00:00:00 2001 From: Srinivasan Parthasarathy Date: Wed, 23 Jun 2021 07:52:47 -0400 Subject: [PATCH] Fix broken links (#804) * some broken links fixed Signed-off-by: Srinivasan Parthasarathy * fixing broken links Signed-off-by: Srinivasan Parthasarathy * seldon tutorial section Signed-off-by: Srinivasan Parthasarathy --- mkdocs/docs/contributing/overview.md | 10 ++-- mkdocs/docs/contributing/tutorials.md | 2 +- .../quick-start/kfserving/tutorial.md | 8 +-- .../quick-start/knative/tutorial.md | 2 +- .../quick-start/seldon/tutorial.md | 4 +- mkdocs/docs/metrics/builtin.md | 4 +- mkdocs/docs/metrics/custom.md | 10 ++-- mkdocs/docs/metrics/using-metrics.md | 6 +-- mkdocs/docs/reference/experiment.md | 6 +-- mkdocs/docs/reference/metrics.md | 2 +- .../istio/rollout-strategies/fixed-split.md | 4 +- .../istio/testing-strategies/conformance.md | 2 +- .../istio/testing-strategies/slovalidation.md | 6 +-- .../rollout-strategies/fixed-split.md | 2 +- .../rollout-strategies/session-affinity.md | 2 +- .../kfserving/testing-strategies/ab.md | 2 +- .../kfserving/testing-strategies/hybrid.md | 10 ++-- .../knative/rollout-strategies/fixed-split.md | 6 +-- .../knative/rollout-strategies/progressive.md | 6 +-- .../rollout-strategies/user-segmentation.md | 2 +- .../knative/testing-strategies/conformance.md | 2 +- .../slovalidationprogressive.md | 8 +-- .../seldon/rollout-strategies/progressive.md | 54 +++++++++++++++++++ .../seldon/testing-strategies/hybrid.md | 7 +++ mkdocs/mkdocs.yml | 5 ++ mkdocs/overrides/home.html | 2 +- 26 files changed, 120 insertions(+), 54 deletions(-) create mode 100644 mkdocs/docs/tutorials/seldon/rollout-strategies/progressive.md create mode 100644 mkdocs/docs/tutorials/seldon/testing-strategies/hybrid.md diff --git a/mkdocs/docs/contributing/overview.md b/mkdocs/docs/contributing/overview.md index f83ac5aaf..a2ad9faf0 100644 --- a/mkdocs/docs/contributing/overview.md +++ b/mkdocs/docs/contributing/overview.md @@ -22,11 +22,11 @@ bug report and let us know! We welcome many different types of contributions including: -* [Iter8 documentation/tutorials](../tutorials) +* [Iter8 documentation/tutorials](tutorials.md) * New features - * [New K8s stack/service mesh/ingress](../newk8sstack) + * [New K8s stack/service mesh/ingress](newk8sstack.md) * Iter8 code samples for OpenShift - * [Analytics](../analytics) + * [Analytics](analytics.md) * Tasks * Builds, CI * Bug fixes @@ -35,7 +35,7 @@ We welcome many different types of contributions including: * Reviewing pull requests Not everything happens through a GitHub pull request. Please come to our -[community meetings](#come-to-meetings) or [contact us](../../getting-started/help) and let's discuss how we can work together. +[community meetings](#come-to-weekly-community-meetings) or [contact us](../getting-started/help.md) and let's discuss how we can work together. *** @@ -77,7 +77,7 @@ The best ways to reach us with a question when contributing is to ask on: * The original GitHub issue * `#development` channel in the [Iter8 Slack workspace](https://join.slack.com/t/iter8-tools/shared_invite/zt-awl2se8i-L0pZCpuHntpPejxzLicbmw) -* Bring your questions to our [community meetings](#come-to-meetings) +* Bring your questions to our [community meetings](#come-to-weekly-community-meetings) ## Pull Request Lifecycle diff --git a/mkdocs/docs/contributing/tutorials.md b/mkdocs/docs/contributing/tutorials.md index e046b8432..a7c778e2c 100644 --- a/mkdocs/docs/contributing/tutorials.md +++ b/mkdocs/docs/contributing/tutorials.md @@ -8,7 +8,7 @@ template: main.html Iter8 documentation uses [Mkdocs](https://www.mkdocs.org/user-guide/writing-your-docs/). The section on [linking to pages and images](https://www.mkdocs.org/user-guide/writing-your-docs/#writing-with-markdown) is especially useful for Iter8 tutorial authors. ## Test your tutorial -All Iter8 tutorials include e2e tests, either as [part of GitHub Actions workflows](https://github.com/iter8-tools/iter8/blob/master/.github/workflows/e2e-tests.yaml) or as a standalone test script [like this one](https://github.com/iter8-tools/iter8/blob/master/samples/knative/mirroring/e2etest.sh) if they require more resources than what is available in GitHub Actions workflows. When contributing a tutorial, please include relevant e2e tests. +All Iter8 tutorials include e2e tests, either as [part of GitHub Actions workflows](https://github.com/iter8-tools/iter8/blob/master/.github/workflows/e2e-tests.yaml) or as a standalone test script [like this one](https://github.com/iter8-tools/iter8/blob/master/samples/seldon/quickstart/e2etest.sh) if they require more resources than what is available in GitHub Actions workflows. When contributing a tutorial, please include relevant e2e tests. ## Locally serve Iter8 docs **Pre-requisite:** Python 3+. diff --git a/mkdocs/docs/getting-started/quick-start/kfserving/tutorial.md b/mkdocs/docs/getting-started/quick-start/kfserving/tutorial.md index 924880dc5..7b353e84d 100644 --- a/mkdocs/docs/getting-started/quick-start/kfserving/tutorial.md +++ b/mkdocs/docs/getting-started/quick-start/kfserving/tutorial.md @@ -5,11 +5,11 @@ template: main.html # A/B Testing !!! tip "Scenario: A/B testing and progressive rollout of KFServing models" - [A/B testing](../../../../../concepts/buildingblocks/#ab-testing) enables you to compare two versions of an ML model, and select a winner based on a (business) reward metric. In this tutorial, you will: + [A/B testing](../../../concepts/buildingblocks.md#ab-testing) enables you to compare two versions of an ML model, and select a winner based on a (business) reward metric. In this tutorial, you will: 1. Perform A/B testing. 2. Specify *user-engagement* as the reward metric. This metric will be mocked by Iter8 in this tutorial. - 3. Combine A/B testing with [progressive traffic shifting](../../../../../concepts/buildingblocks/#progressive-traffic-shift). Iter8 will progressively shift traffic towards the winner and promote it at the end as depicted below. + 3. Combine A/B testing with [progressive traffic shifting](../../../concepts/buildingblocks.md#progressive-traffic-shift). Iter8 will progressively shift traffic towards the winner and promote it at the end as depicted below. ![Quickstart KFServing](../../../images/quickstart-ab.png) @@ -19,7 +19,7 @@ template: main.html 3. [Go 1.13+](https://golang.org/doc/install). ## 1. Setup -* Setup your K8s cluster with KFServing and Iter8 as described [here](../platform-setup/). +* Setup your K8s cluster with KFServing and Iter8 as described [here](platform-setup.md). * Ensure that the `ITER8` environment variable is set to the root of your local Iter8 repo. ## 2. Create ML model versions @@ -152,7 +152,7 @@ kubectl apply -f $ITER8/samples/kfserving/quickstart/metrics.yaml ??? Note "Metrics in your environment" You can define and use custom metrics from any database in Iter8 experiments. - For your application, replace the mocked metric used in this tutorial with any custom metric you wish to optimize in the A/B test. Documentation on defining custom metrics is [here](../../../../metrics/custom/). + For your application, replace the mocked metric used in this tutorial with any custom metric you wish to optimize in the A/B test. Documentation on defining custom metrics is [here](../../../metrics/custom.md). ## 5. Launch experiment Iter8 defines a custom K8s resource called *Experiment* that automates a variety of release engineering and experimentation strategies for K8s applications and ML models. Launch the A/B testing & progressive traffic shift experiment as follows. diff --git a/mkdocs/docs/getting-started/quick-start/knative/tutorial.md b/mkdocs/docs/getting-started/quick-start/knative/tutorial.md index 13d80bd47..f3d592551 100644 --- a/mkdocs/docs/getting-started/quick-start/knative/tutorial.md +++ b/mkdocs/docs/getting-started/quick-start/knative/tutorial.md @@ -245,7 +245,7 @@ kubectl apply -f $ITER8/samples/knative/quickstart/metrics.yaml ??? Note "Metrics in your environment" You can define and use custom metrics from any database in Iter8 experiments. - For your application, replace the mocked user-engagement metric used in this tutorial with any custom metric you wish to optimize in the hybrid (A/B + SLOs) test. Documentation on defining custom metrics is [here](../../../../metrics/custom/). + For your application, replace the mocked user-engagement metric used in this tutorial with any custom metric you wish to optimize in the hybrid (A/B + SLOs) test. Documentation on defining custom metrics is [here](../../../metrics/custom.md). ## 5. Launch experiment Iter8 defines a custom K8s resource called *Experiment* that automates a variety of release engineering and experimentation strategies for K8s applications and ML models. Launch the hybrid (A/B + SLOs) testing & progressive traffic shift experiment as follows. diff --git a/mkdocs/docs/getting-started/quick-start/seldon/tutorial.md b/mkdocs/docs/getting-started/quick-start/seldon/tutorial.md index 9ff0c4a3a..ee7cc3bff 100644 --- a/mkdocs/docs/getting-started/quick-start/seldon/tutorial.md +++ b/mkdocs/docs/getting-started/quick-start/seldon/tutorial.md @@ -21,7 +21,7 @@ template: main.html 4. [Helm 3+](https://helm.sh/docs/intro/install/) ## 1. Setup -* Setup your K8s cluster with Seldon and Iter8 as described [here](../platform-setup/). +* Setup your K8s cluster with Seldon and Iter8 as described [here](platform-setup.md). * Ensure that the `ITER8` environment variable is set to the root of your local Iter8 repo. ## 2. Create ML model versions @@ -328,7 +328,7 @@ kubectl apply -f $ITER8/samples/seldon/quickstart/metrics.yaml ??? Note "Metrics in your environment" You can define and use custom metrics from any database in Iter8 experiments. - For your application, replace the mocked user-engagement metric used in this tutorial with any custom metric you wish to optimize in the hybrid (A/B + SLOs) test. Documentation on defining custom metrics is [here](../../../../metrics/custom/). + For your application, replace the mocked user-engagement metric used in this tutorial with any custom metric you wish to optimize in the hybrid (A/B + SLOs) test. Documentation on defining custom metrics is [here](../../../metrics/custom.md). ## 5. Launch experiment Iter8 defines a custom K8s resource called *Experiment* that automates a variety of release engineering and experimentation strategies for K8s applications and ML models. Launch the hybrid (A/B + SLOs) testing & progressive traffic shift experiment as follows. diff --git a/mkdocs/docs/metrics/builtin.md b/mkdocs/docs/metrics/builtin.md index 5e758f737..19050b965 100644 --- a/mkdocs/docs/metrics/builtin.md +++ b/mkdocs/docs/metrics/builtin.md @@ -25,7 +25,7 @@ The following are the set of builtin Iter8 metrics. | iter8-system | latency-99th-percentile | Gauge | 99th percentile response latency | ## Collecting builtin metrics -Use the [`metrics/collect` task](../../reference/tasks/metrics/) in an experiment to collect builtin metrics for your app/ML model versions. +Use the [`metrics/collect` task](../reference/tasks/metrics.md) in an experiment to collect builtin metrics for your app/ML model versions. ## Example -For an example of an experiment that uses builtin metrics, look inside the Knative experiment in [this tutorial](../../tutorials/testing-strategies/conformance/#7-launch-experiment). \ No newline at end of file +For an example of an experiment that uses builtin metrics, look inside the Knative experiment in [this tutorial](../../tutorials/testing-strategies/conformance/#5-launch-experiment). \ No newline at end of file diff --git a/mkdocs/docs/metrics/custom.md b/mkdocs/docs/metrics/custom.md index b8282397e..b82b065cd 100644 --- a/mkdocs/docs/metrics/custom.md +++ b/mkdocs/docs/metrics/custom.md @@ -340,7 +340,7 @@ The examples in this document focus on Prometheus, NewRelic, Sysdig, and Elastic Iter8 will substitute placeholders in the metric query based on the time elapsed since the start of the experiment, and information associated with each version in the experiment. -Suppose the [metrics defined above](#defining-metrics) are referenced within an experiment as follows. Further, suppose this experiment has started, Iter8 is about to do an iteration of this experiment, and the time elapsed since the start of the experiment is 600 seconds. +Suppose the [metrics defined above](#metrics-withwithout-auth) are referenced within an experiment as follows. Further, suppose this experiment has started, Iter8 is about to do an iteration of this experiment, and the time elapsed since the start of the experiment is 600 seconds. ??? abstract "Look inside sample experiment" ```yaml linenums="1" @@ -543,7 +543,7 @@ The metrics provider is expected to respond to Iter8's HTTP request with a JSON Iter8 uses [jq](https://stedolan.github.io/jq/) to extract the metric value from the JSON response of the provider. The `jqExpression` used by Iter8 is supplied as part of the metric definition. When the `jqExpression` is applied to the JSON response, it is expected to yield a number. === "Prometheus" - Consider the `jqExpression` defined in the [sample Prometheus metric](#defining-metrics). Let us apply it to the [sample JSON response from Prometheus](#json-response). + Consider the `jqExpression` defined in the [sample Prometheus metric](#metrics-withwithout-auth). Let us apply it to the [sample JSON response from Prometheus](#json-response). ```shell echo '{ "status": "success", @@ -560,7 +560,7 @@ Iter8 uses [jq](https://stedolan.github.io/jq/) to extract the metric value from Executing the above command results yields `21.7639`, a number, as required by Iter8. === "New Relic" - Consider the `jqExpression` defined in the [sample New Relic metric](#defining-metrics). Let us apply it to the [sample JSON response from New Relic](#json-response). + Consider the `jqExpression` defined in the [sample New Relic metric](#metrics-withwithout-auth). Let us apply it to the [sample JSON response from New Relic](#json-response). ```shell echo '{ "results": [ @@ -602,7 +602,7 @@ Iter8 uses [jq](https://stedolan.github.io/jq/) to extract the metric value from Executing the above command results yields `80275388`, a number, as required by Iter8. === "Sysdig" - Consider the `jqExpression` defined in the [sample Sysdig metric](#defining-metrics). Let us apply it to the [sample JSON response from Sysdig](#json-response). + Consider the `jqExpression` defined in the [sample Sysdig metric](#metrics-withwithout-auth). Let us apply it to the [sample JSON response from Sysdig](#json-response). ```shell echo '{ "data": [ @@ -620,7 +620,7 @@ Iter8 uses [jq](https://stedolan.github.io/jq/) to extract the metric value from Executing the above command results yields `6.481`, a number, as required by Iter8. === "Elastic" - Consider the `jqExpression` defined in the [sample Elastic metric](#defining-metrics). Let us apply it to the [sample JSON response from Elastic](#json-response). + Consider the `jqExpression` defined in the [sample Elastic metric](#metrics-withwithout-auth). Let us apply it to the [sample JSON response from Elastic](#json-response). ```shell echo '{ "aggregations": { diff --git a/mkdocs/docs/metrics/using-metrics.md b/mkdocs/docs/metrics/using-metrics.md index 9d6c74f58..bf37d5c55 100644 --- a/mkdocs/docs/metrics/using-metrics.md +++ b/mkdocs/docs/metrics/using-metrics.md @@ -7,7 +7,7 @@ template: main.html !!! tip "Iter8 metric resources" Iter8 defines a custom Kubernetes resource (CRD) called **Metric** that makes it easy to define and use metrics in experiments. - Iter8 installation includes a set of pre-defined [builtin metrics](../builtin) that pertain to app/ML model latency/errors. You can also [define custom metrics](../custom) that enable you to utilize data from Prometheus, New Relic, Sysdig, Elastic or any other database of your choice. + Iter8 installation includes a set of pre-defined [builtin metrics](builtin.md) that pertain to app/ML model latency/errors. You can also [define custom metrics](custom.md) that enable you to utilize data from Prometheus, New Relic, Sysdig, Elastic or any other database of your choice. ## List metrics Find the set Iter8 metrics available in your cluster using `kubectl get`. @@ -32,7 +32,7 @@ iter8-system request-count Counter Number of requests (Iter8 ## Referencing metrics within experiments -Use metrics in experiments by referencing them in the criteria section of the experiment manifest. Reference metrics using the `namespace/name` or `name` [format](../../../reference/apispec/#criteria). +Use metrics in experiments by referencing them in the criteria section of the experiment manifest. Reference metrics using the `namespace/name` or `name` [format](../reference/experiment.md#criteria). ??? example "Sample experiment illustrating the use of metrics" ```yaml @@ -55,4 +55,4 @@ Use metrics in experiments by referencing them in the criteria section of the ex ``` ## Observing metric values -During an experiment, Iter8 reports the metric values observed for each version. Use `iter8ctl` to observe these metric values in realtime. See [here](../../../getting-started/quick-start/#a-observe-metrics) for an example. \ No newline at end of file +During an experiment, Iter8 reports the metric values observed for each version. Use `iter8ctl` to observe these metric values in realtime. See [here](../getting-started/quick-start/kfserving/tutorial.md#a-observe-results) for an example. \ No newline at end of file diff --git a/mkdocs/docs/reference/experiment.md b/mkdocs/docs/reference/experiment.md index 1526401f8..1d6e3d5a7 100644 --- a/mkdocs/docs/reference/experiment.md +++ b/mkdocs/docs/reference/experiment.md @@ -5,7 +5,7 @@ template: main.html # Experiment Resource !!! abstract "Experiment resource" - Iter8's **Experiment** resource type enables application developers and service operators to automate A/B, A/B/n, Canary and Conformance experiments for Kubernetes apps/ML models. The controls provided by the experiment resource type encompass [testing, deployment, traffic engineering, and version promotion functions](../../../concepts/buildingblocks/). + Iter8's **Experiment** resource type enables application developers and service operators to automate A/B, A/B/n, Canary and Conformance experiments for Kubernetes apps/ML models. The controls provided by the experiment resource type encompass [testing, deployment, traffic engineering, and version promotion functions](../concepts/buildingblocks.md). ??? info "Sample experiment" ```yaml linenums="1" @@ -190,7 +190,7 @@ Standard Kubernetes [meta.v1/ObjectMeta](https://kubernetes.io/docs/reference/ge | Field name | Field type | Description | Required | | ----- | ---- | ----------- | -------- | | name | string | Identifies an Iter8 metric using the [`namespace/name` or `name` format](#criteria). | Yes | -| metric | [][Metric](#metric) | Iter8 metric object referenced by name. | No | +| metric | [][Metric](metrics.md) | Iter8 metric object referenced by name. | No | ### ExperimentCondition @@ -209,7 +209,7 @@ Standard Kubernetes [meta.v1/ObjectMeta](https://kubernetes.io/docs/reference/ge | Field name | Field type | Description | Required | | ----- | ------------ | ----------- | -------- | -| aggregatedBuiltinHists | [AggregatedBuiltinHists](#aggregatedbuiltinhists) | This field is used to store intermediate results from the [`metrics/collect` task](../tasks/metrics/#metrics-tasks) that enables [builtin metrics](../../metrics/builtin/). Reserved for Iter8 internal use. | No | +| aggregatedBuiltinHists | [AggregatedBuiltinHists](#aggregatedbuiltinhists) | This field is used to store intermediate results from the [`metrics/collect` task](tasks/metrics.md#metrics-tasks) that enables [builtin metrics](../metrics/builtin.md). Reserved for Iter8 internal use. | No | | aggregatedMetrics | [AggregatedMetricsAnalysis](#aggregatedmetricsanalysis) | Most recently observed metric values for all metrics referenced in the experiment criteria. | No | | winnerAssessment | [WinnerAssessmentAnalysis](#winnerassessmentanalysis) | Information about the `winner` of the experiment. | No | | versionAssessments | [VersionAssessmentAnalysis](#versionassessmentanalysis) | For each version, a summary analysis identifying whether or not the version is satisfying the experiment criteria. | No | diff --git a/mkdocs/docs/reference/metrics.md b/mkdocs/docs/reference/metrics.md index 4c1c5f1b9..e1664949f 100644 --- a/mkdocs/docs/reference/metrics.md +++ b/mkdocs/docs/reference/metrics.md @@ -11,7 +11,7 @@ template: main.html !!! note "Version" This document describes version `v2alpha2` of Iter8's metric API. -Metrics usage is documented [here](../../metrics/using-metrics/). Iter8 provides a set of builtin metrics that are documented [here](../../metrics/builtin/). Creation of custom metrics is documented [here](../../metrics/custom/). It is also possible to mock metric values; mock metrics are documented [here](../../metrics/mock). +Metrics usage is documented [here](../metrics/using-metrics.md). Iter8 provides a set of builtin metrics that are documented [here](../metrics/builtin.md). Creation of custom metrics is documented [here](../metrics/custom.md). It is also possible to mock metric values; mock metrics are documented [here](../metrics/mock.md). ??? example "Sample metric" ```yaml linenums="1" diff --git a/mkdocs/docs/tutorials/istio/rollout-strategies/fixed-split.md b/mkdocs/docs/tutorials/istio/rollout-strategies/fixed-split.md index 1365530f7..3b389ff22 100644 --- a/mkdocs/docs/tutorials/istio/rollout-strategies/fixed-split.md +++ b/mkdocs/docs/tutorials/istio/rollout-strategies/fixed-split.md @@ -122,8 +122,8 @@ kubectl apply -f $ITER8/samples/istio/fixed-split/experiment.yaml value: https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/istio/quickstart/vs-for-v2.yaml ``` -## 6. Observe experiment -Follow [Step 6 of the quick start tutorial](../../../getting-started/quick-start/istio/tutorial.md#6-observe-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`fixedsplit-exp`) in your `iter8ctl` and `kubectl` commands. +## 6. Understand the experiment +Follow [Step 6 of the quick start tutorial](../../../getting-started/quick-start/istio/tutorial.md#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`fixedsplit-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup ```shell diff --git a/mkdocs/docs/tutorials/istio/testing-strategies/conformance.md b/mkdocs/docs/tutorials/istio/testing-strategies/conformance.md index 32bf10314..6e0b11f3c 100644 --- a/mkdocs/docs/tutorials/istio/testing-strategies/conformance.md +++ b/mkdocs/docs/tutorials/istio/testing-strategies/conformance.md @@ -154,7 +154,7 @@ kubectl apply -f $ITER8/samples/istio/conformance/experiment.yaml value: bookinfo-iter8 ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for Istio](../../../../getting-started/quick-start/istio/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`conformance-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/istio/testing-strategies/slovalidation.md b/mkdocs/docs/tutorials/istio/testing-strategies/slovalidation.md index 03a3f7509..c5183529c 100644 --- a/mkdocs/docs/tutorials/istio/testing-strategies/slovalidation.md +++ b/mkdocs/docs/tutorials/istio/testing-strategies/slovalidation.md @@ -5,11 +5,11 @@ template: main.html # SLO Validation !!! tip "Scenario: SLO validation with progressive traffic shift" - This tutorial illustrates an [SLO validation experiment with two versions](../../../../concepts/buildingblocks/#slo-validation); the candidate version will be promoted after Iter8 validates that it satisfies service-level objectives (SLOs). You will: + This tutorial illustrates an [SLO validation experiment with two versions](../../../concepts/buildingblocks.md#slo-validation); the candidate version will be promoted after Iter8 validates that it satisfies service-level objectives (SLOs). You will: 1. Specify *latency* and *error-rate* based service-level objectives (SLOs). If the candidate version satisfies SLOs, Iter8 will declare it as the winner. 2. Use Prometheus as the provider for latency and error-rate metrics. - 3. Combine SLO validation with [progressive traffic shifting](../../../../concepts/buildingblocks/#progressive-traffic-shift). + 3. Combine SLO validation with [progressive traffic shifting](../../../concepts/buildingblocks.md#progressive-traffic-shift). ![SLO validation with progressive traffic shift](../../../images/slovalidationprogressive.png) @@ -84,7 +84,7 @@ kubectl apply -f $ITER8/samples/istio/slovalidation/experiment.yaml fieldPath: .spec.http[0].route[1].weight ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for Istio](../../../../getting-started/quick-start/istio/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`slovalidation-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/kfserving/rollout-strategies/fixed-split.md b/mkdocs/docs/tutorials/kfserving/rollout-strategies/fixed-split.md index f72bf4362..a99a7e3ea 100644 --- a/mkdocs/docs/tutorials/kfserving/rollout-strategies/fixed-split.md +++ b/mkdocs/docs/tutorials/kfserving/rollout-strategies/fixed-split.md @@ -110,7 +110,7 @@ kubectl apply -f $ITER8/samples/kfserving/fixed-split/experiment.yaml value: https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/kfserving/quickstart/promote-v2.yaml ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for KFServing](../../../../getting-started/quick-start/kfserving/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`fixedsplit-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/kfserving/rollout-strategies/session-affinity.md b/mkdocs/docs/tutorials/kfserving/rollout-strategies/session-affinity.md index 2e690e3fc..21d19f923 100644 --- a/mkdocs/docs/tutorials/kfserving/rollout-strategies/session-affinity.md +++ b/mkdocs/docs/tutorials/kfserving/rollout-strategies/session-affinity.md @@ -150,7 +150,7 @@ kubectl apply -f $ITER8/samples/kfserving/session-affinity/experiment.yaml value: https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/kfserving/quickstart/promote-v2.yaml ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for KFServing](../../../../getting-started/quick-start/kfserving/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`session-affinity-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/kfserving/testing-strategies/ab.md b/mkdocs/docs/tutorials/kfserving/testing-strategies/ab.md index 7ce606c00..d98d1af67 100644 --- a/mkdocs/docs/tutorials/kfserving/testing-strategies/ab.md +++ b/mkdocs/docs/tutorials/kfserving/testing-strategies/ab.md @@ -4,4 +4,4 @@ template: main.html # A/B Testing -The [quick start tutorial for KFServing](../../../getting-started/quick-start/kfserving/tutorial) demonstrates A/B testing. \ No newline at end of file +The [quick start tutorial for KFServing](../../../getting-started/quick-start/kfserving/tutorial.md) demonstrates A/B testing. \ No newline at end of file diff --git a/mkdocs/docs/tutorials/kfserving/testing-strategies/hybrid.md b/mkdocs/docs/tutorials/kfserving/testing-strategies/hybrid.md index f82ac34ed..cf9ff2699 100644 --- a/mkdocs/docs/tutorials/kfserving/testing-strategies/hybrid.md +++ b/mkdocs/docs/tutorials/kfserving/testing-strategies/hybrid.md @@ -5,12 +5,12 @@ template: main.html # Hybrid (A/B + SLOs) testing !!! tip "Scenario: Hybrid (A/B + SLOs) testing and progressive traffic shift of KFServing models" - [Hybrid (A/B + SLOs) testing](../../concepts/buildingblocks/#hybrid-testing) enables you to combine A/B or A/B/n testing with a reward metric on the one hand with SLO validation using objectives on the other. Among the versions that satisfy objectives, the version which performs best in terms of the reward metric is the winner. In this tutorial, you will: + [Hybrid (A/B + SLOs) testing](../../../concepts/buildingblocks.md#hybrid-testing) enables you to combine A/B or A/B/n testing with a reward metric on the one hand with SLO validation using objectives on the other. Among the versions that satisfy objectives, the version which performs best in terms of the reward metric is the winner. In this tutorial, you will: 1. Perform hybrid (A/B + SLOs) testing. 2. Specify *user-engagement* as the reward metric. 3. Specify *latency* and *error-rate* based objectives, for which data will be provided by Prometheus. - 4. Combine hybrid (A/B + SLOs) testing with [progressive traffic shift](../../../../../concepts/buildingblocks/#progressive-traffic-shift). Iter8 will progressively shift traffic towards the winner and promote it at the end as depicted below. + 4. Combine hybrid (A/B + SLOs) testing with [progressive traffic shift](../../../concepts/buildingblocks.md#progressive-traffic-shift). Iter8 will progressively shift traffic towards the winner and promote it at the end as depicted below. ![Hybrid testing](../../../images/quickstart-hybrid.png) @@ -20,7 +20,7 @@ template: main.html 3. [Go 1.13+](https://golang.org/doc/install). ## 1. Steps 1, 2, and 3 -* Follow [Steps 1, 2, and 3 of the KFServing quick start tutorial](getting-started/quick-start/kfserving/tutorial/). +* Follow [Steps 1, 2, and 3 of the KFServing quick start tutorial](../../../getting-started/quick-start/kfserving/tutorial.md). ## 4. Define metrics ```shell @@ -194,8 +194,8 @@ kubectl apply -f $ITER8/samples/kfserving/hybrid/experiment.yaml value: https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/kfserving/quickstart/promote-v2.yaml ``` -## 6. Observe experiment -Follow [Step 6 of the quick start tutorial for KFServing](../../../../getting-started/quick-start/kfserving/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`hybrid-exp`) in your `iter8ctl` and `kubectl` commands. +## 6. Understand the experiment +Follow [Step 6 of the quick start tutorial for KFServing](../../../getting-started/quick-start/kfserving/tutorial.md#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`hybrid-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup ```shell diff --git a/mkdocs/docs/tutorials/knative/rollout-strategies/fixed-split.md b/mkdocs/docs/tutorials/knative/rollout-strategies/fixed-split.md index 35289e31e..155fd9cb3 100644 --- a/mkdocs/docs/tutorials/knative/rollout-strategies/fixed-split.md +++ b/mkdocs/docs/tutorials/knative/rollout-strategies/fixed-split.md @@ -48,10 +48,10 @@ kubectl wait --for=condition=Ready ksvc/sample-app ``` ## 3. Generate requests -Please follow [Step 3 of the quick start tutorial](../../../../getting-started/quick-start/knative/tutorial/#3-generate-requests). +Please follow [Step 3 of the quick start tutorial](../../../getting-started/quick-start/knative/tutorial.md#3-generate-requests). ## 4. Define metrics -Please follow [Step 4 of the quick start tutorial](../../../../getting-started/quick-start/knative/tutorial/#4-define-metrics). +Please follow [Step 4 of the quick start tutorial](../../../getting-started/quick-start/knative/tutorial.md#4-define-metrics). ## 5. Launch experiment ```shell @@ -108,7 +108,7 @@ kubectl apply -f $ITER8/samples/knative/fixed-split/experiment.yaml value: candidate ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for Knative](../../../../getting-started/quick-start/knative/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`fixedsplit-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/knative/rollout-strategies/progressive.md b/mkdocs/docs/tutorials/knative/rollout-strategies/progressive.md index eeae8a10a..9c47972f9 100644 --- a/mkdocs/docs/tutorials/knative/rollout-strategies/progressive.md +++ b/mkdocs/docs/tutorials/knative/rollout-strategies/progressive.md @@ -5,19 +5,19 @@ template: main.html # Progressive Traffic Shift !!! tip "Scenario: Progressive traffic shift" - [Progressive traffic shift](../../../../concepts/buildingblocks/#progressive-traffic-shift) is a type of canary rollout strategy. It enables you to incrementally shift traffic towards the winning version over multiple iterations of an experiment as shown below. + [Progressive traffic shift](../../../concepts/buildingblocks.md#progressive-traffic-shift) is a type of canary rollout strategy. It enables you to incrementally shift traffic towards the winning version over multiple iterations of an experiment as shown below. ![Progressive traffic shift](../../../images/progressive.png) ## Tutorials with progressive traffic shift -The [hybrid testing (quick start)](../../../../getting-started/quick-start/knative/tutorial) and [the SLO validation](../../testing-strategies/slovalidationprogressive/) tutorials demonstrate progressive traffic shift. +The [hybrid testing (quick start)](../../../getting-started/quick-start/knative/tutorial.md) and [the SLO validation](../testing-strategies/slovalidationprogressive.md) tutorials demonstrate progressive traffic shift. ## Specifying `weightObjRef` Iter8 uses the `weightObjRef` field in the experiment resource to get the current traffic split between versions and/or modify the traffic split. Ensure that this field is specified correctly for each version. The following example demonstrates how to specify `weightObjRef` in experiments. ??? example "Example" - The [A/B testing quick start tutorial for Knative](../../../getting-started/quick-start/#7-launch-experiment) uses a Knative service for traffic shifting. Hence, the experiment manifest specifies the `weightObjRef` field for each version by referencing this Knative service and the traffic fields within the Knative service corresponding to the versions. + The [A/B testing quick start tutorial for Knative](../../../getting-started/quick-start/knative/tutorial.md#5-launch-experiment) uses a Knative service for traffic shifting. Hence, the experiment manifest specifies the `weightObjRef` field for each version by referencing this Knative service and the traffic fields within the Knative service corresponding to the versions. ```yaml versionInfo: diff --git a/mkdocs/docs/tutorials/knative/rollout-strategies/user-segmentation.md b/mkdocs/docs/tutorials/knative/rollout-strategies/user-segmentation.md index 9da6955d0..295abfd3e 100644 --- a/mkdocs/docs/tutorials/knative/rollout-strategies/user-segmentation.md +++ b/mkdocs/docs/tutorials/knative/rollout-strategies/user-segmentation.md @@ -226,7 +226,7 @@ kubectl apply -f $ITER8/samples/knative/user-segmentation/experiment.yaml fieldPath: .spec.http[0].route[1].weight ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for Knative](../../../../getting-started/quick-start/knative/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`user-segmentation-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/knative/testing-strategies/conformance.md b/mkdocs/docs/tutorials/knative/testing-strategies/conformance.md index 557f93e96..e64846d22 100644 --- a/mkdocs/docs/tutorials/knative/testing-strategies/conformance.md +++ b/mkdocs/docs/tutorials/knative/testing-strategies/conformance.md @@ -89,7 +89,7 @@ kubectl apply -f $ITER8/samples/knative/conformance/experiment.yaml name: sample-app-v1 ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for Knative](../../../../getting-started/quick-start/knative/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`conformance-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/knative/testing-strategies/slovalidationprogressive.md b/mkdocs/docs/tutorials/knative/testing-strategies/slovalidationprogressive.md index 242ee9b77..7335937ba 100644 --- a/mkdocs/docs/tutorials/knative/testing-strategies/slovalidationprogressive.md +++ b/mkdocs/docs/tutorials/knative/testing-strategies/slovalidationprogressive.md @@ -5,17 +5,17 @@ template: main.html # SLO Validation !!! tip "Scenario: SLO validation with progressive traffic shift" - This tutorial illustrates an [SLO validation experiment with two versions](../../../../concepts/buildingblocks/#slo-validation); the candidate version will be promoted after Iter8 validates that it satisfies service-level objectives (SLOs). You will: + This tutorial illustrates an [SLO validation experiment with two versions](../../../concepts/buildingblocks.md#slo-validation); the candidate version will be promoted after Iter8 validates that it satisfies service-level objectives (SLOs). You will: 1. Specify *latency* and *error-rate* based service-level objectives (SLOs). If the candidate version satisfies SLOs, Iter8 will declare it as the winner. 2. Use Prometheus as the provider for latency and error-rate metrics. - 3. Combine SLO validation with [progressive traffic shifting](../../../../concepts/buildingblocks/#progressive-traffic-shift). + 3. Combine SLO validation with [progressive traffic shifting](../../../concepts/buildingblocks.md#progressive-traffic-shift). ![SLO validation with progressive traffic shift](../../../images/slovalidationprogressive.png) ## Steps 1 to 4 -Please follow steps 1 through 4 of the [Knative quick start tutorial](../../../../getting-started/quick-start/knative/quick-start/#1-setup). +Please follow steps 1 through 4 of the [Knative quick start tutorial](../../../getting-started/quick-start/knative/tutorial.md#1-setup). ## 5. Launch experiment @@ -82,7 +82,7 @@ kubectl apply -f $ITER8/samples/knative/slovalidation/experiment.yaml value: candidate ``` -## 6. Observe experiment +## 6. Understand the experiment Follow [Step 6 of the quick start tutorial for KFServing](../../../../getting-started/quick-start/kfserving/tutorial/#6-understand-the-experiment) to observe metrics, traffic and progress of the experiment. Ensure that you use the correct experiment name (`slovalidation-exp`) in your `iter8ctl` and `kubectl` commands. ## 7. Cleanup diff --git a/mkdocs/docs/tutorials/seldon/rollout-strategies/progressive.md b/mkdocs/docs/tutorials/seldon/rollout-strategies/progressive.md new file mode 100644 index 000000000..580d96415 --- /dev/null +++ b/mkdocs/docs/tutorials/seldon/rollout-strategies/progressive.md @@ -0,0 +1,54 @@ +--- +template: main.html +--- + +# Progressive Traffic Shift + +!!! tip "Scenario: Progressive traffic shift" + [Progressive traffic shift](../../../concepts/buildingblocks.md#progressive-traffic-shift) is a type of canary rollout strategy. It enables you to incrementally shift traffic towards the winning version over multiple iterations of an experiment as shown below. + + ![Progressive traffic shift](../../../images/progressive.png) + +## Tutorials with progressive traffic shift + +The [hybrid (A/B + SLOs) testing](../../../getting-started/quick-start/seldon/tutorial.md) tutorial demonstrates progressive traffic shift. + +## Specifying `weightObjRef` + +Iter8 uses the `weightObjRef` field in the experiment resource to get the current traffic split between versions and/or modify the traffic split. Ensure that this field is specified correctly for each version. The following example demonstrates how to specify `weightObjRef` in experiments. + +??? example "Example" + The [hybrid (A/B + SLOs) testing](../../../getting-started/quick-start/seldon/tutorial.md) tutorial uses an Istio virtual service for traffic shifting. Hence, the experiment manifest specifies the `weightObjRef` field for each version by referencing this Istio virtual service and the traffic fields within the Istio virtual service corresponding to the versions. + + ```yaml + versionInfo: + baseline: + name: iris-v1 + weightObjRef: + apiVersion: networking.istio.io/v1alpha3 + kind: VirtualService + name: routing-rule + namespace: default + fieldPath: .spec.http[0].route[0].weight + candidates: + - name: iris-v2 + weightObjRef: + apiVersion: networking.istio.io/v1alpha3 + kind: VirtualService + name: routing-rule + namespace: default + fieldPath: .spec.http[0].route[1].weight + ``` + +## Traffic controls + +You can specify the maximum traffic percentage that is allowed for a candidate version during the experiment. You can also specify the maximum increase in traffic percentage that is allowed for a candidate version during a single iteration of the experiment. You can specify these two controls in the `strategy` section of an experiment as follows. + +```yaml +strategy: + weights: # additional traffic controls to be used during an experiment + # candidate weight will not exceed 75 in any iteration + maxCandidateWeight: 75 + # candidate weight will not increase by more than 20 in a single iteration + maxCandidateWeightIncrement: 20 +``` \ No newline at end of file diff --git a/mkdocs/docs/tutorials/seldon/testing-strategies/hybrid.md b/mkdocs/docs/tutorials/seldon/testing-strategies/hybrid.md new file mode 100644 index 000000000..0d294bebe --- /dev/null +++ b/mkdocs/docs/tutorials/seldon/testing-strategies/hybrid.md @@ -0,0 +1,7 @@ +--- +template: main.html +--- + +# Hybrid (A/B + SLOs) testing + +The [quick start tutorial for Seldon](../../../getting-started/quick-start/seldon/tutorial.md) demonstrates hybrid (A/B + SLOs) testing. diff --git a/mkdocs/mkdocs.yml b/mkdocs/mkdocs.yml index 4f7898b03..96b0cf88b 100644 --- a/mkdocs/mkdocs.yml +++ b/mkdocs/mkdocs.yml @@ -136,6 +136,11 @@ nav: - Progressive traffic shift: tutorials/kfserving/rollout-strategies/progressive.md - Fixed-%-split: tutorials/kfserving/rollout-strategies/fixed-split.md - Session affinity: tutorials/kfserving/rollout-strategies/session-affinity.md + - Seldon: + - Testing strategies: + - Hybrid (A/B + SLOs) testing: tutorials/seldon/testing-strategies/hybrid.md + - Rollout strategies: + - Progressive traffic shift: tutorials/seldon/rollout-strategies/progressive.md - Knative: - Testing strategies: - SLO validation: tutorials/knative/testing-strategies/slovalidationprogressive.md diff --git a/mkdocs/overrides/home.html b/mkdocs/overrides/home.html index 8654c1fd2..59492f9a2 100644 --- a/mkdocs/overrides/home.html +++ b/mkdocs/overrides/home.html @@ -168,7 +168,7 @@

Talk to us on slack

Come to meetings

- Meet