General improvements for iter8.tools text and Iter8 repo issue/PR tem…

…plates (#732)

* built in metrics

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* built in metrics installation

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* dummy jq expression

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* updated images

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* updated installer for v0.5

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* temporary image using personal repo

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* iter8 conformance test with builtin metrics

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* statistical rigor

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* expanded documentation for builtins

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* updated builtin docs

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* updated handler image

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* fixing builtins

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* mkdocs version pinning

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* contributor documentation

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* removed commented blocks on integrations

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* reverting experiment yaml https url

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* improved illustrations

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* PR and issue templates

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

* fixing issue and PR templates

Signed-off-by: Srinivasan Parthasarathy <[email protected]>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,8 +1,8 @@
 ---
 name: Bug report
 about: Create a report to help us improve
-title: "[BUG]"
-labels: bug
+title: ""
+labels: kind/bug
 assignees: ""
 
 ---

.github/ISSUE_TEMPLATE/documentation.md

@@ -1,8 +1,8 @@
 ---
 name: "Documentation issue"
 about: The documentation is unclear, missing informations, or could be improved
-title: "[Documentation Request]"
-labels: documentation
+title: ""
+labels: kind/docs
 assignees: ''
 ---
 

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,36 +1,32 @@
 ---
 name: Feature request
 about: Suggest an idea for Iter8
-title: '[Feature Request]'
-labels: 'enhancement'
+title: ""
+labels: kind/enhancement
 assignees: ''
 
 ---
 
-**Is your feature request related to a problem? Please describe.**
-A clear and concise description of the problem that will be solved by this feature.
+**Is your feature request related to a **problem**? Please describe the **problem**.**
+A clear and concise description of the **problem**.
 
-**Why is the feature useful for Iter8 users?**
-How will this feature add value to Iter8 users and improve Iter8 experiments.
-
-**Describe the solution you'd like**
+**Describe the feature/solution you'd like**
 A clear and concise description of what you want to happen.
 
-**Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+**Will this feature/solution bring new benefits for Iter8 users? What are they?**
+How will this feature/solution add value to Iter8 users.
+
+**Will this feature/solution bring new benefits for Iter8 developers? What are they?**
+How will this feature/solution add value to Iter8 developers.
 
 **Does this issue require a design doc/discussion? If there is a link to the design document/discussions, please provide it below.**
-You can use this issue thread, or a discussion thread in this repo, for design discussions. If you do so, please provide a link to the specific issue/discussion comment(s) that describes the design.
+You can use this issue thread, or a discussion thread in this repo, for design discussions. Please provide the appropriate links. You can use the issue/discussion to also discuss and describe alternatives.
 
 **How will this feature be tested?**
-[ ] A checklist of behaviors that will be tested as part of this feature implementation. 
-
-Mark this as TBD if testing requirements are unclear at this time.
+- [ ] A checklist of behaviors that will be tested as part of this feature implementation. 
 
 **How will this feature be documented?**
-[ ] A checklist which maybe a combination of user and developer documentation. This includes new (or modifications to existing) code-samples, references, or other forms of user documentation that will be added to https://iter8.tools as part of this feature implementation. Also includes developer documentation, that must always accompany code in the `iter8-analytics`, `etc3`, `handler`, and `iter8ctl` repos.
-
-Mark this as TBD if documentation requirements are unclear at this time.
+- [ ] Most new features require end-user documentation at https://iter8.tools. Describe the specific documentation that will be added.
 
 **Additional context**
 Add any other context or screenshots about the feature request here.

.github/PULL_REQUEST_TEMPLATE/pull_request_template.md

@@ -1,5 +1,5 @@
-### Description
-Please describe your pull request, specifically, the issue which this PR is intended to resolve.
+### Issue resolved/partially addressed by this PR
+Please describe your pull request, specifically, the issue which this PR is intended to resolve or partially address.
 
 ### Testing
 - [ ] Does this PR fix a bug? How is the bug covered by new test(s)? 

mkdocs/docs/concepts/buildingblocks.md

@@ -4,7 +4,7 @@ template: main.html
 
 # Building Blocks
 
-> Iter8 defines a Kubernetes resource called **Experiment** that automates A/B, A/B/n, Canary, and Conformance experiments. During an experiment, Iter8 can compare multiple versions, find,  and safely promote the **winning version (winner)** based on business metrics and SLOs.
+> Iter8 defines a Kubernetes resource called **Experiment** that automates SLO validation, A/B, and A/B/n testing experiments. During an experiment, Iter8 can compare multiple versions, find, and safely promote the **winning version (winner)** based on business metrics and performance metrics like latency and error-rate.
 
 We now introduce the building blocks of an Iter8 experiment.
 
@@ -75,10 +75,15 @@ A version of your app/ML model is considered **validated**, if it satisfies the
 
 ## Traffic engineering
 
-**Traffic engineering** refers to features such as **traffic mirroring/shadowing** and **user segmentation** that provide fine-grained controls over how traffic is routed to and from app versions.
+**Traffic engineering** refers to features such as **dark launch, traffic mirroring/shadowing, user segmentation** and **session affinity** that provide fine-grained controls over how traffic is routed to and from app versions.
 
 Iter8 enables you to take total advantage of all the traffic engineering features available in the service mesh, ingress technology, or networking layer present in your Kubernetes cluster.
 
+=== "Dark launch"
+    **Dark launch** enables you to deploy and experiment with a new version of your application/ML model in such a way that it is hidden from all (or most) of your end-users.
+
+    ![Canary](../images/mirroring.png)
+
 === "Traffic mirroring/shadowing"
     **Traffic mirroring** or **shadowing** enables experimenting with a *dark* launched version with zero-impact on end-users. Mirrored traffic is a replica of the real user requests[^1] that is routed to the dark version. Metrics are collected and evaluated for the dark version, but responses from the dark version are ignored.
 
@@ -104,22 +109,8 @@ Iter8 enables you to take total advantage of all the traffic engineering feature
 
 When two or more versions participate in an experiment, Iter8 **recommends a version for promotion**; if the experiment yielded a winner, then the version recommended for promotion is the winner; otherwise, the version recommended for promotion is the **baseline** version of your app/ML model.
 
-Iter8 can optionally **promote the recommended version** at the end of an experiment.
+Iter8 can **promote the recommended version** at the end of an experiment.
 
 ![Canary](../images/yamljson.png)
 
-## Resource config tools
-
-Iter8 can be easily integrated with Helm and Kustomize. This integration is especially useful if you use these tools for configuring Kubernetes resources needed for releases of your app/ML model.
-
-=== "Helm charts"
-    An experiment that uses `helm` charts for version promotion is illustrated below.
-
-    ![Canary](../images/helm.png)
-
-=== "Kustomize resources"
-    An experiment that uses `kustomize` resources for version promotion is illustrated below.
-
-    ![Canary](../images/kustomize.png)
-
 [^1]: It is possible to mirror only a certain percentage of the requests instead of all requests.

mkdocs/docs/concepts/whatisiter8.md

@@ -4,12 +4,14 @@ template: main.html
 
 # What is Iter8?
 
-**Iter8** is an AI-powered platform for cloud native release automation, validation, and experimentation. Iter8 makes it easy to unlock business value and guarantee SLOs by identifying the best performing app/ML model version and promoting it safely.
+**Iter8** is an AI-powered platform for cloud native release automation and experimentation platform that enables SLO validation, A/B testing and progressive delivery. 
 
-Iter8 is designed for **developers, SREs, service operators, data scientists, and ML engineers** who wish to maximize release velocity and business value with their apps/ML models while protecting end-user experience.
+Iter8 makes it easy to **unlock business value and guarantee SLOs** by identifying the best performing version of your app/ML model and promoting it safely.
+
+Iter8 is designed for **DevOps and MLOps teams** interested in maximizing release velocity and business value with their apps/ML models while protecting end-user experience.
 
 ## What is an Iter8 experiment?
-Iter8 defines a Kubernetes resource called **Experiment** that automates A/B, A/B/n, Canary, and Conformance experiments. During an experiment, Iter8 can compare multiple versions, find,  and safely promote the winning version based on business metrics and SLOs.
+Iter8 defines a Kubernetes resource called **Experiment** that automates SLO validation, A/B, and A/B/n testing experiments. During an experiment, Iter8 can compare multiple versions, find, and safely promote the **winning version (winner)** based on business metrics and performance metrics like latency and error-rate.
 
 ![Process automated by an Iter8 experiment](../images/whatisiter8.png)
 

mkdocs/docs/reference/tasks/common.md

@@ -8,49 +8,73 @@ template: main.html
 
 ### Overview
 
-The `common/exec` task executes a shell commands with any specified arguments. Arguments are specified using placeholders, or templated variables, that are dynamically instantiated at runtime using values defined when the task is. The `common/exec` task is used in most tutorials as part of a finish action to promote the winning version at the end of an experiment.
+The `common/exec` task executes a shell command with arguments. Arguments are specified using placeholders that are dynamically substituted at runtime. The `common/exec` task can be used as part of a finish action to promote the winning version at the end of an experiment.
 
-### Examples
+### Example
 
-The following example executes `kubectl apply` using the file defined by the placeholder  `.promte` when the experiment has completed.
+The following (partially-specified) experiment executes `kubectl apply` using a YAML manifest at the end of the experiment. The URL for the manifest contains a placeholder `.promote`, which is dynamically substituted at the end of the experiment.
 
 ```yaml
-- finish:
-  task: common/exec
-    with:
-    - cmd: /bin/bash
-    - args:
-      - -c
-      - |
-        kubectl apply -f {{ .promote }}
+kind: Experiment
+...
+spec:
+  ...
+  strategy:
+    ...
+    actions:
+      finish: # run the following sequence of tasks at the end of the experiment
+      - task: common/exec # promote the winning version      
+        with:
+          cmd: /bin/sh
+          args:
+          - "-c"
+          - |
+            kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/{{ .promote }}.yaml
+    ...
+  versionInfo:
+    # information about app versions used in this experiment
+    baseline:
+      name: sample-app-v1
+      variables:
+      - name: promote
+        value: baseline
+    candidates:
+    - name: sample-app-v2
+      variables:
+      - name: promote
+        value: candidate
 ```
 
-### Result
-
-The command will be executed; in this case, the yaml file will be applied to the cluster. If a task exits with a non-zero error code, the experiment to which it belongs will also fail.
-
-### Arguments
+### Inputs
 
 | Field name | Field type | Description | Required |
 | ----- | ---- | ----------- | -------- |
 | cmd | string | The command that should be executed | Yes |
 | args | []string | A list of command line arguments that should be passed to `cmd`. | No |
 | disableInterpolation | bool | Flag indicating whether or not to disable placeholder subsitution. For details, see [below](#disabling-placeholder-substitution). Default is `false`. | No |
 
-### Details
+### Result
 
-#### Dynamic Placeholder Substitution
+The command with the supplied arguments will be executed. 
 
-Inputs to tasks can contain placeholders, or template variables, which will be dynamically substituted when the task is executed by Iter8. For example, in the sample experiment above, one input is:
+In the [example above](#example), a YAML file corresponding to the baseline or candidate version will be applied to the cluster.
 
-    kubectl apply -f {{ .promote }}
+If this task exits with a non-zero error code, the experiment to which it belongs will fail.
+
+### Dynamic placeholder substitution
+
+Inputs to tasks can contain placeholders, or template variables, which will be dynamically substituted when the task is executed by Iter8. In the [example above](#example), one input is:
+```shell
+kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/knative/quickstart/{{ .promote }}.yaml
+```
+In this case, the placeholder is `{{ .promote }}`. 
 
-In this case, the placeholder is `{{ .promote }}`. Placeholder substitution in task inputs works as follows.
+Placeholder substitution in task inputs works as follows. 
 
-Iter8 will find the version recommended for promotion. This information is set by Iter8 as the experiment exectuted and in stored in the `status.versionRecommendedForPromotion` field of the experiment. The version recommended for promotion is the winner, if a winner has been found in the experiment. Otherwise, it is the baseline version supplied in the `spec.versionInfo` field of the experiment.
+Iter8 will find the version recommended for promotion which is determined by Iter8 during the  course of the experiment, and stored in the `status.versionRecommendedForPromotion` field of the experiment resource. The version recommended for promotion is the winner, if a winner has been found in the experiment. Otherwise, it is the baseline version supplied in the `spec.versionInfo` field of the experiment.
 
-If the placeholder is `{{ .name }}`, Iter8 will substitute it with the name of the version recommended for promotion. If it is any other variable, Iter8 will substitute it with the value of the corresponding variable for the version recommended for promotion. Variable values are specified in the variables field of the version detail. Note that variable values could have been supplied by the creator of the experiment, or by other tasks that may already have been executed by Iter8 as part of the experiment.
+If the placeholder is `{{ .name }}`, Iter8 will substitute it with the name of the version recommended for promotion. If it is any other variable, Iter8 will substitute it with the value of the corresponding variable for the version recommended for promotion. Variable values are specified in the variables field of the version detail when the experiment is created.
 
-#### Disabling Placeholder Substitution
+### Disabling placeholder substitution
 
 By default, the `common/exec` task will attempt to find the version recommended for promotion, and use the values defined for it (in the spec.versionInfo portion of the experiment). However, this behavior will lead to task failure when the version recommended for promotion is not available. This is usually the case when a start task is executed. To use the `common/exec` task as part of an experiment start action, set `disableInterpolation` to true.

mkdocs/docs/reference/tasks/notification.md

@@ -10,33 +10,33 @@ template: main.html
 
 The `notification/slack` task posts a slack message about current state of the experiment.
 
-### Examples
+### Example
 
-The following task creates a notification that will be posted to the slack channel with id `channel` during the execution of an action.
+The following task notifies a slack channel with id `C0138103183` and using the token contained in the secret `slack-token` in the `ns` namespace.
 
 ```yaml
 task: notification/slack
-  with:
-  - channel: channel
-  - secret: ns/slack-token
+with:
+  channel: C0138103183
+  secret: ns/slack-token
 ```
 
-### Result
-
-A slack message describing the experiment will be posted to the specified channel. It will look something like this:
-
-![Sample slack notificiation](../../images/slack-notification.png)
-
-### Arguments
+### Inputs
 
 | Field name | Field type | Description | Required |
 | ----- | ---- | ----------- | -------- |
 | channel | string | Name of the slack channel to which messages should be posted. | Yes |
 | secret | string | Identifies a secret containing a `token` to be used for authentication.  Expressed as `namespace/name`. If `namespace` is not specified, the namespace of the experiment is used. | Yes |
 
+### Result
+
+A slack message describing the experiment will be posted to the specified channel. Below is a sample slack notification from this task.
+
+![Sample slack notificiation](../../images/slack-notification.png)
+
 ### Requirements
 
-#### Slack API Token
+#### Slack API token
 
 An API token allowing posting messages to the desired slack channel is needed. To obtain a suitable token, see [Sending messages using Incoming Webhooks](https://api.slack.com/messaging/webhooks). Once you have the token, store it in a Kubernetes secret. For example, to create the secret _slack-secret_ in the default namespace:
 
@@ -54,35 +54,34 @@ kubectl apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samp
 
 ??? info "Inspect role and rolebinding"
     ```yaml linenums="1"
-        # This role enables reading of secrets
-        apiVersion: rbac.authorization.k8s.io/v1
-        kind: ClusterRole
-        metadata:
-        name: iter8-secret-reader
-        rules:
-        - apiGroups:
-        - ""
-        resources:
-        - secrets
-        verbs: ["get", "list"]
-        ---
-        # This role binding enables Iter8 handler to read secrets in the default namespace.
-        # To change the namespace apply to the target namespace
-        apiVersion: rbac.authorization.k8s.io/v1
-        kind: RoleBinding
-        metadata:
-        name: iter8-secret-reader-handler
-        roleRef:
-        apiGroup: rbac.authorization.k8s.io
-        kind: ClusterRole
-        name: iter8-secret-reader
-        subjects:
-        - kind: ServiceAccount
-        name: iter8-handlers
-        namespace: iter8-system
-
+    # This role enables reading of secrets
+    apiVersion: rbac.authorization.k8s.io/v1
+    kind: ClusterRole
+    metadata:
+      name: iter8-secret-reader
+    rules:
+    - apiGroups:
+      - ""
+      resources:
+      - secrets
+      verbs: ["get", "list"]
+    ---
+    # This role binding enables Iter8 handler to read secrets in the default namespace.
+    # To change the namespace apply to the target namespace
+    apiVersion: rbac.authorization.k8s.io/v1
+    kind: RoleBinding
+    metadata:
+      name: iter8-secret-reader-handler
+    roleRef:
+      apiGroup: rbac.authorization.k8s.io
+      kind: ClusterRole
+      name: iter8-secret-reader
+    subjects:
+    - kind: ServiceAccount
+      name: iter8-handlers
+      namespace: iter8-system
     ```
 
-#### Target Slack Channel ID
+#### Slack channel ID
 
 A slack channel is identified by an id. To find the id, open the slack channel in a web browser. The channel id is the portion of the URL of the form: `CXXXXXXXX`