Grafana integration leaves no data dashboards

[natalytvinova]

Bug Description

I deployed COS and integrated it with envoy:

$   juju status --relations | grep envoy
envoy                                  res:oci-image@cc06b3e    active       1  envoy                     2.0/stable           101  10.152.183.122  no       
grafana-agent-envoy                    0.35.2                   active       1  grafana-agent-k8s         latest/stable         58  10.152.183.22   no       
envoy/5*                                  active       idle   192.168.13.250   9090,9901/TCP  
grafana-agent-envoy/0*                    active       idle   192.168.212.194                 
cos-loki:logging                                                   grafana-agent-envoy:logging-consumer                     loki_push_api             regular  
cos-prometheus:receive-remote-write                                grafana-agent-envoy:send-remote-write                    prometheus_remote_write   regular  
envoy:grafana-dashboards                                           cos-grafana:grafana-dashboard                            grafana_dashboard         regular  joining  
envoy:grafana-dashboards                                           grafana-agent-envoy:grafana-dashboards-consumer          grafana_dashboard         regular  
envoy:metrics-endpoint                                             grafana-agent-envoy:metrics-endpoint                     prometheus_scrape         regular  
grafana-agent-envoy:grafana-dashboards-provider                    cos-grafana:grafana-dashboard                            grafana_dashboard         regular  
grafana-agent-envoy:peers                                          grafana-agent-envoy:peers                                grafana_agent_replica     peer     
istio-pilot:ingress                                                envoy:ingress                                            ingress                   regular  
mlmd:grpc                                                          envoy:grpc                                               grpc                      regular  

As you can see a lot of dashboards have no data:

To Reproduce

1. Juju deploy COS
2. Juju deploy kubeflow 1.8
3. Integrate it

Environment

Relevant Log Output

-

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5424.

This message was autogenerated

[orfeas-k]

My guess is that all metrics work, but there are no data on some (I 've seen total requests work) because the metrics exposed are irrelevant to the kind of usage we do for envoy. But we 'll need to confirm and even if I am right, we need to adjust the dashboard to present data users need. So I 'll link this to the Grafana tracker issue.

[orfeas-k]

An update on this, I 've sent a PR that fixes some of the panels. That being said, and this is true for all dashboards, it's normal for some of the panels to have no data since a workload may not provide every metric at every point in time. Thus, if it hasn't provided any scrape data about a specific panel yet, it is expected for this panel to show "No data".