Unable to view feedback reward on Grafana dashboard

[Nithinbs18]

Dear team,

Greetings!
I am currently working with Seldon-analytics and unable to visualize the feedback data on the grafana dashboard.

Please find the below details;
Seldon deployment manifest:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  name: test2
spec:
  name: test2
  predictors:
    - graph:
        children: []
        implementation: MLFLOW_SERVER
        modelUri: s3://seldon/b
        envSecretRefName: seldon-init-container-secret
        name: classifier
      name: model-test2
      labels:
        test-nithin: nithin2
      replicas: 1
      componentSpecs:
        - spec:
            containers:
              - name: classifier
                env:
                  - name: FOO
                    value: bar
                livenessProbe:
                  failureThreshold: 3
                  initialDelaySeconds: 300
                  periodSeconds: 5
                  successThreshold: 1
                  tcpSocket:
                    port: http
                  timeoutSeconds: 1

Helm charts used to install the pre-requisites :

helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set usageMetrics.enabled=true --set ambassador.enabled=true --set executor.defaultEnvSecretRefName=seldon-core-init-container-secret

helm install seldon-core-analytics seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts --set grafana_prom_admin_password=password --set persistence.enabled=false

helm repo add stable https://kubernetes-charts.storage.googleapis.com

helm install ambassador stable/ambassador --set crds.keep=false

Client program ;

import pandas as pd
import numpy as np
from seldon_core.seldon_client import SeldonClient
import json
import time
import requests
sc = SeldonClient(
    gateway_endpoint="localhost:8003",
    gateway="ambassador",
    namespace="default",
    deployment_name='test2')

df = pd.read_csv("./wine-quality.csv")

def _get_reward(y, y_pred):
    if y == y_pred:
        return 500
    return 1 / np.square(y - y_pred)


def _test_row(row):
    input_features = row[:-1]
    feature_names = input_features.index.to_list()
    X = input_features.values.reshape(1, -1)
    y = row[-1]
    r = sc.predict(
        data=X,
        names=feature_names)
    print(r.request)
    print(r.response)
    y_pred = (r.response['data']['tensor']['values'])
    reward = _get_reward(y, y_pred)
    print((str(y))+"-------"+(str(y_pred))+"-------"+(str(reward)))
    sc.feedback(
        prediction_request=r.request,
        prediction_response=r.response,
        gateway='ambassador',
        gateway_endpoint="localhost:8003",
        namespace="default",
        deployment_name='test2',
        reward=float(reward))
    return reward[0]
df.apply(_test_row, axis=1)

I also tried manually posing the data using;
with Request:
meta {
}
data {
names: "fixed acidity"
names: "volatile acidity"
names: "citric acid"
names: "residual sugar"
names: "chlorides"
names: "free sulfur dioxide"
names: "total sulfur dioxide"
names: "density"
names: "pH"
names: "sulphates"
names: "alcohol"
tensor {
shape: 1
shape: 11
values: 8.3
values: 0.42
values: 0.62
values: 19.25
values: 0.04
values: 41.0
values: 172.0
values: 1.0002
values: 2.98
values: 0.67
values: 9.7
}
}

Response:
{'data': {'names': [], 'tensor': {'shape': [1], 'values': [5.277768952367847]}}, 'meta': {}}

Code:

    payload = json.dumps(r.response)
    response_raw = requests.post(
        "http://localhost:8003/seldon/default/test2/api/v1.0/feedback",
        json=payload
     )
    print(response_raw.status_code)
    print(response_raw.text)

This returns 400 with below message:

{"status":{"code":-1,"info":"Invalid JSON: Message type \"seldon.protos.Feedback\" has no field named \"{\".\n Available Fields(except extensions): ['request', 
'response', 'reward', 'truth']","reason":"MICROSERVICE_BAD_DATA","status":1}}

The prediction works absolutely fine but does not return anything in meta eg:

{'data': {'names': [], 'tensor': {'shape': [1], 'values': [5.277768952367847]}}, 'meta': {}}

Please find the screenshot:

The return value of feedback function:

Success:True message:
Request:
request {
  meta {
  }
  data {
    names: "fixed acidity"
    names: "volatile acidity"
    names: "citric acid"
    names: "residual sugar"
    names: "chlorides"
    names: "free sulfur dioxide"
    names: "total sulfur dioxide"
    names: "density"
    names: "pH"
    names: "sulphates"
    names: "alcohol"
    tensor {
      shape: 1
      shape: 11
      values: 7.0
      values: 0.27
      values: 0.36
      values: 20.7
      values: 0.045
      values: 45.0
      values: 170.0
      values: 1.001
      values: 3.0
      values: 0.45
      values: 8.8
    }
  }
}
response {
  meta {
  }
  data {
    tensor {
      shape: 1
      values: 5.283192947262262
    }
  }
}
reward: 1.9462358

Response:
{'data': {'tensor': {'shape': [0]}}, 'meta': {}}

Any help on this will be highly appreciated.
Thanks in advance.

[Nithinbs18]

Hi @adriangonz
Do you have any idea about this?
Thanks in advance.

[axsaucedo]

@Nithinbs18 I can confirm this is a bug, we're doing some work right now on metrics, and it seems that the feedback component wasn't updated with the executor. For the meantime you can still get it to work if you run it with backwards compatibility as outlined here https://docs.seldon.io/projects/seldon-core/en/latest/reference/upgrading.html#new-service-orchestrator, but otherwise we'll be looking to fix this in 1.3

[Nithinbs18]

Hello @axsaucedo

Thanks for your reply.
As suggested I am using the lower version of seldon-core-operator and analytics i.e. 1.1.0

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION
ambassador              default         1               2020-07-13 10:07:27.1890633 +0200 CEST  deployed        ambassador-5.3.2                0.86.1
seldon-core             default         1               2020-07-14 22:45:55.0822594 +0200 CEST  deployed        seldon-core-operator-1.1.0
seldon-core-analytics   default         1               2020-07-15 06:49:30.6030806 +0200 CEST  deployed        seldon-core-analytics-1.1.0

Now I have an issue with the feedback, I cannot feedback any rewards as the endpoint /seldon/default/test2/api/v1.0/feedback returns 404 when I try to use it in the seldon python client (v 1.1.0).
Could you please suggest what exact version can I use or if there is a workaround for this.
Thank you very much in advance.

[axsaucedo]

Oh @Nithinbs18 my suggestion wasn't to downgrade Seldon Core, it was to basically switch to the "engine" which is the old service orchestrator (the new one is the "executor"). You can still do this in the latest version 1.2.1. I would recommend you to still stick to the latest version, as I can confirm feedback is working correctly, and if you switch to the "engine" you should start seeing reward in the metrics. We are aiming to add the fix for /feedback, so you'll be able to get it working without the need of the backwards compatibility flag from ~1.3.

[Nithinbs18]

Hi @axsaucedo
My bad and thanks a lot for the tip it worked now I can see the reward values in the dashboard.
One query does the use of old "engine" somehow stops logging the other metric data like Req/Sec and Latency? because when I use the "engine" they are not visible.

[axsaucedo]

@Nithinbs18 that shouldn't be the case, the req/sec and latency should still be displayed. It may just be that given it's running on the older backwards compatibility, it may also need to run on an older version of seldon-core-analytics. If this is the case we should make sure to add that in the documentation - @RafalSkolasinski would actually be able to provide more insights specifically on the point about compatibility with the version of seldon-core-analytics.

[Nithinbs18]

Hi @axsaucedo & @RafalSkolasinski

I tried the below combination;
Seldon core 1.1.0 / Seldon Analytics 1.1.0 / with latest executor ---> No reward captured
Seldon core 1.2.1 / Seldon Analytics 1.1.0 / with latest executor ---> No reward captured
Seldon core 1.1.0 / Seldon Analytics 1.1.0 / with latest executor ---> No reward captured
Seldon core 1.2.1 / Seldon Analytics 1.2.1 / with legacy engine ---> Reward captured / No ReqperSec and Latency captured
Seldon core 1.2.1 / Seldon Analytics 1.1.0 / with legacy engine ---> Reward captured / No ReqperSec and Latency captured
Seldon core 1.1.0 / Seldon Analytics 1.2.1 / with legacy engine ---> no data captured
Seldon core 1.1.0 / Seldon Analytics 1.1.0 / with legacy engine ---> no data captured

Thanks and looking forward to 1.3 :)

[axsaucedo]

Thank you very much for doing this arduous testing @Nithinbs18, it may be useful for us to add into on our docuemntation in the upgrading section. We're currently having a look at this now and will be providing updates so you get notified once this is addressed.

[Nithinbs18]

Hello @axsaucedo & @RafalSkolasinski ,

I see that this issue is closed now thanks for that. Could you please let me know how to get this working now?
Thanks in advance.

[RafalSkolasinski]

Hi @Nithinbs18 ,

Fix is now in master. If you use 1.2.2 version of wrapper then rewards will be collecected automatically using custom metrics mechanism and exposed to prometheus. From there Granana should be able to pick it up without issues.

[Nithinbs18]

Hi @RafalSkolasinski

Thanks for the response, I tried using the charts available in the master (with 1.2.2-dev tag) for core-operator and analytics I still face the same issue when I use the latest executor the reward is not visible and when I use the legacy engine other metrics are not visible, can you please point what am I doing wrong?
Thanks

[RafalSkolasinski]

You need to build your model usibg latest s2i python wrapper - this is the fixed component

[RafalSkolasinski]

@Nithinbs18 I now see that you are using the implementation: MLFLOW_SERVER - version 1.2.2 of mlflow server have seldon-core (python package) installed with the fixed version.

[RafalSkolasinski]

@Nithinbs18 Could you use helm-charts from release-1.2.2 branch? It should sensure usage of the new MLFLOW server image.

Please make sure you are using 1.2.2 image, the -dev has not been updated in 6 days so it does not contain the fix.

[Nithinbs18]

Hi @RafalSkolasinski
It works for standard deployment and also for canary deployments with the chart in release-1.2.2 now, thank you very much for your time.