expose /metrics in python wrapper

.gitignore

@@ -208,4 +208,5 @@ examples/ambassador/custom/ambassador_custom.py
 examples/ambassador/headers/ambassador_headers.py
 examples/ambassador/shadow/ambassador_shadow.py
 examples/models/metrics/metrics.py
+examples/models/custom_metrics/customMetrics.py
 examples/models/tracing/tracing.py

README.md

@@ -295,8 +295,8 @@ Below are some of the core components together with link to the logs that provid
 * [Java Language Wrapper [Incubating] ](https://docs.seldon.io/projects/seldon-core/en/latest/java/README.html)
 * [R Language Wrapper [ALPHA] ](https://docs.seldon.io/projects/seldon-core/en/latest/R/README.html)
 * [NodeJS Language Wrapper [ALPHA] ](https://docs.seldon.io/projects/seldon-core/en/latest/nodejs/README.html)
-* [Custom Language Wrapper [ALPHA] ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/custom_metrics.html)
-   
+* [Go Language Wrapper [ALPHA] ](https://docs.seldon.io/projects/seldon-core/en/latest/go/README.html)
+
 ### Ingress
 
 * [Ambassador Ingress ](https://docs.seldon.io/projects/seldon-core/en/latest/ingress/ambassador.html)
@@ -307,6 +307,7 @@ Below are some of the core components together with link to the logs that provid
 * [Supported API Protocols ](https://docs.seldon.io/projects/seldon-core/en/latest/graph/protocols.html)
 * [CI/CD MLOps at Scale ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/cicd-mlops.html)
 * [Metrics with Prometheus ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/analytics.html)
+* [Custom Metrics ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/custom_metrics.html)
 * [Payload Logging with ELK ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/logging.html)
 * [Distributed Tracing with Jaeger ](https://docs.seldon.io/projects/seldon-core/en/latest/graph/distributed-tracing.html)
 * [Autoscaling in Kubernetes ](https://docs.seldon.io/projects/seldon-core/en/latest/graph/autoscaling.html)

doc/source/analytics/analytics.md

@@ -28,27 +28,28 @@ Seldon Core provides an example Helm analytics chart that displays the above Pro
 ```bash
 helm install seldon-core-analytics seldon-core-analytics \
    --repo https://storage.googleapis.com/seldon-charts \
-   --set grafana_prom_admin_password=password \
-   --set persistence.enabled=false \
    --namespace seldon-system
 ```
 
-The available parameters are:
-
- * ```grafana_prom_admin_password``` : The admin user Grafana password to use.
- * ```persistence.enabled``` : Whether Prometheus persistence is enabled.
-
 Once running you can expose the Grafana dashboard with:
 
 ```bash
-kubectl port-forward $(kubectl get pods -n seldon-system -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') 3000:3000 -n seldon-system
+kubectl port-forward svc/seldon-core-analytics-grafana 3000:80 -n seldon-system
 ```
 
-You can then view the dashboard at http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1
+You can then view the dashboard at http://localhost:3000/dashboard/db/prediction-analytics
 
 ![dashboard](./dashboard.png)
 
+It is also possible expose Prometheus itself with:
+```bash
+kubectl port-forward svc/seldon-core-analytics-prometheus-seldon 3001:80 -n seldon-system
+```
+
+and then access it at http://localhost:3001/
+
+
+
 ## Example
 
 There is [an example notebook you can use to test the metrics](../examples/metrics.html).
-

doc/source/analytics/custom_metrics.md

@@ -1,6 +1,8 @@
 # Custom Metrics
 
-Seldon Core exposes basic metrics via Prometheus endpoints on its service orchestrator that include request count, request time percentiles and rolling accuracy for each running model. However, you may wish to expose custom metrics from your components which are automatically added to Prometheus. For this purpose you can supply extra fields in the returned meta data of the response object in the API calls to your components as illustrated below:
+Seldon Core exposes basic metrics via Prometheus endpoints on its service orchestrator that include request count, request time percentiles and rolling accuracy for each running model as described in [metrics](./analytics.md) documentation.
+However, you may wish to expose custom metrics from your components which are automatically added to Prometheus.
+For this purpose you can supply extra fields in the returned meta data of the response object in the API calls to your components as illustrated below:
 
 ```json
 {
@@ -10,7 +12,7 @@ Seldon Core exposes basic metrics via Prometheus endpoints on its service orches
 				"type": "COUNTER",
 				"key": "mycounter",
 				"value": 1.0,
-				"tags": {"mytag":"mytagvalue"}
+				"tags": {"mytag": "mytagvalue"}
 			},
 			{
 				"type": "GAUGE",
@@ -39,7 +41,7 @@ We provide three types of metric that can be returned in the meta.metrics list:
 
  * COUNTER : a monotonically increasing value. It will be added to any existing value from the metric key.
  * GAUGE : an absolute value showing a level, it will overwrite any existing value.
- * TIMER : a time value (in msecs).
+ * TIMER : a time value (in msecs), it will be aggregated into Prometheus' HISTOGRAM.
 
 Each metric, apart from the type, takes a key and a value. The proto buffer definition is shown below:
 
@@ -53,12 +55,53 @@ message Metric {
  string key = 1;
  MetricType type = 2;
  float value = 3;
- map<string,string> tags = 4; 
+ map<string,string> tags = 4;
 }
 ```
 
+## Metrics endpoints
 
-As we expose the metrics via Prometheus, if ```tags``` are added they must appear in every metric response and always have the same set of keys since Prometheus does not allow metrics to have varying numbers of tags. This condition is enforced by the [micrometer](https://micrometer.io/) library we use to expose the metrics. Exceptions will happen if this condition is violated.
+Custom metrics are exposed directly by the Python wrapper.
+In order for `Prometheus` to scrape multiple endpoints from a single `Pod` we use `metrics` name for ports that expose `Prometheus` metrics:
+```yaml
+ports:
+- containerPort: 6000
+  name: metrics
+  protocol: TCP
+```
+
+This require us to use a following entry
+```
+  - source_labels: [__meta_kubernetes_pod_container_port_name]
+    action: keep
+    regex: metrics(-.*)?
+```
+in the Prometheus [config](https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-analytics/files/prometheus/prometheus-config.yaml) together with following two annotations:
+```
+prometheus.io/scrape: "true"
+prometheus.io/path: "/prometheus"
+```
+
+Note: we do not use `prometheus.io/port` annotation in this configuration.
+
+
+Before Seldon Core 1.1 custom metrics have been returned to the orchestrator which exposed them all together to `Prometheus` via a single endpoint.
+We used to have at this time all three following annotations:
+```yaml
+prometheus.io/scrape: "true"
+prometheus.io/path: "/prometheus"
+prometheus.io/port: "8000"
+```
+
+
+## Labels
+
+As we expose the metrics via `Prometheus`, if ```tags``` are added they must appear in every metric response otherwise `Prometheus` will consider such metrics as a new time series, see official [documentation].
+
+Before Seldon Core 1.1 orchestrator enforced presence of same set of labels using the [micrometer](https://micrometer.io/) library to expose metrics. Exceptions would happen if this condition have been violated.
+
+
+## Supported wrappers
 
 At present the following Seldon Core wrappers provide integrations with custom metrics:
 
@@ -67,4 +110,4 @@ At present the following Seldon Core wrappers provide integrations with custom m
 
 ## Example
 
-There is an [example notebook illustrating a model with custom metrics in python](../examples/tmpl_model_with_metrics.html).
+There is an [example notebook illustrating a model with custom metrics in python](../examples/custom_metrics.html).

doc/source/examples/custom_metrics.nblink

@@ -0,0 +1,3 @@
+{
+  "path": "../../../examples/models/custom_metrics/customMetrics.ipynb"
+}

doc/source/go/go_wrapper_link.rst

@@ -0,0 +1 @@
+.. mdinclude:: ../../../incubating/wrappers/s2i/go/README.md

doc/source/index.rst

@@ -74,16 +74,16 @@ Documentation Index
    Java Language Wrapper [Incubating] <java/README.md>
    R Language Wrapper [ALPHA] <R/README.md>
    NodeJS Language Wrapper [ALPHA] <nodejs/README.md>
-   Custom Language Wrapper [ALPHA] <analytics/custom_metrics.md>
-   
-   
+   Go Language Wrapper [ALPHA] <go/go_wrapper_link.rst>
+
+
 .. toctree::
    :maxdepth: 1
    :caption: Ingress
 
    Ambassador Ingress <ingress/ambassador.md>
    Istio Ingress <ingress/istio.md>
-   
+
 .. toctree::
    :maxdepth: 1
    :caption: Production

doc/source/python/python_wrapping_docker.md

@@ -147,38 +147,37 @@ These arguments can be set when deploying in a Seldon Deployment. An example can
 
 ```
  "graph": {
-		    "name": "tfserving-proxy",
-		    "endpoint": { "type" : "REST" },
-		    "type": "MODEL",
-		    "children": [],
-		    "parameters":
-		    [
-			{
-			    "name":"grpc_endpoint",
-			    "type":"STRING",
-			    "value":"localhost:8000"
-			},
-			{
-			    "name":"model_name",
-			    "type":"STRING",
-			    "value":"mnist-model"
-			},
-			{
-			    "name":"model_output",
-			    "type":"STRING",
-			    "value":"scores"
-			},
-			{
-			    "name":"model_input",
-			    "type":"STRING",
-			    "value":"images"
-			},
-			{
-			    "name":"signature_name",
-			    "type":"STRING",
-			    "value":"predict_images"
-			}
-		    ]
+    "name": "tfserving-proxy",
+    "endpoint": {"type" : "REST"},
+    "type": "MODEL",
+    "children": [],
+    "parameters": [
+    	{
+    	    "name":"grpc_endpoint",
+    	    "type":"STRING",
+    	    "value":"localhost:8000"
+    	},
+    	{
+    	    "name":"model_name",
+    	    "type":"STRING",
+    	    "value":"mnist-model"
+    	},
+    	{
+    	    "name":"model_output",
+    	    "type":"STRING",
+    	    "value":"scores"
+    	},
+    	{
+    	    "name":"model_input",
+    	    "type":"STRING",
+    	    "value":"images"
+    	},
+    	{
+    	    "name":"signature_name",
+    	    "type":"STRING",
+    	    "value":"predict_images"
+    	}
+    ]
 },
 ```
 
@@ -191,14 +190,14 @@ The allowable ```type``` values for the parameters are defined in the [proto buf
 
 To add custom metrics to your response you can define an optional method ```metrics``` in your class that returns a list of metric dicts. An example is shown below:
 
-```
+```python
 class MyModel(object):
 
-    def predict(self,X,features_names):
+    def predict(self, X, features_names):
         return X
 
     def metrics(self):
-    	return [{"type":"COUNTER","key":"mycounter","value":1}]
+    	return [{"type": "COUNTER", "key": "mycounter", "value": 1}]
 ```
 
 For more details on custom metrics and the format of the metric dict see [here](../custom_metrics.md).
@@ -210,17 +209,12 @@ There is an [example notebook illustrating a model with custom metrics in python
 
 To add custom meta data you can add an optional method ```tags``` which can return a dict of custom meta tags as shown in the example below:
 
-```
+```python
 class MyModel(object):
 
-    def predict(self,X,features_names):
+    def predict(self, X, features_names):
         return X
 
     def tags(self):
-        return {"mytag":1}
+        return {"mytag": 1}
 ```
-
-
-
-
-