[Flow Visibility]Add Grafana dashboards

[heanlan]

Add flow-records, pod-to-pod, pod-to-external, pod-to-service, node-throughput, networkpolicy-allow dashboards.

Signed-off-by: heanlan [email protected]

[codecov-commenter]

Codecov Report

Merging #3191 (f92de9a) into main (2ee6ad1) will decrease coverage by 2.99%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3191      +/-   ##
==========================================
- Coverage   60.59%   57.59%   -3.00%     
==========================================
  Files         297      334      +37     
  Lines       25434    28632    +3198     
==========================================
+ Hits        15411    16492    +1081     
- Misses       8345    10339    +1994     
- Partials     1678     1801     +123     

Flag	Coverage Δ
kind-e2e-tests	45.13% <ø> (-1.85%)	⬇️
unit-tests	41.29% <ø> (+0.94%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...g/agent/apiserver/handlers/featuregates/handler.go	0.00% <0.00%> (-82.36%)	⬇️
...kg/apiserver/registry/system/supportbundle/rest.go	20.45% <0.00%> (-54.26%)	⬇️
pkg/support/dump_others.go	0.00% <0.00%> (-51.73%)	⬇️
pkg/support/dump.go	8.19% <0.00%> (-48.47%)	⬇️
...egator/apiserver/handlers/recordmetrics/handler.go	0.00% <0.00%> (-44.45%)	⬇️
...g/agent/apiserver/handlers/addressgroup/handler.go	0.00% <0.00%> (-40.00%)	⬇️
...agent/apiserver/handlers/appliedtogroup/handler.go	0.00% <0.00%> (-40.00%)	⬇️
...gregator/apiserver/handlers/flowrecords/handler.go	0.00% <0.00%> (-40.00%)	⬇️
pkg/apiserver/handlers/loglevel/handler.go	0.00% <0.00%> (-38.47%)	⬇️
pkg/agent/util/net.go	17.68% <0.00%> (-21.09%)	⬇️
... and 90 more

[heanlan]

[Updated]
flow-records

pod-to-pod

pod-to-external

pod-to-service

node-throughput

networkpolicy-allow

[tnqn]

Hi @heanlan I see the directory is new and no other file is referencing to the json files. Could you explain how they will be consumed? Do we plan to document it?

[heanlan]

Hi @heanlan I see the directory is new and no other file is referencing to the json files. Could you explain how they will be consumed? Do we plan to document it?

Hi Quan, this PR along with the other 4 PRs all together composite of the phase-1 dev plan of the new flow-visibility long-term architecture. We all share the same directory.

• Add Grafana and Clickhouse deployment file [Flow Visibility] Add Grafana and Clickhouse deployment file #3063 
• Add policy recommendation Spark job [Flow Visibility] Add policy recommendation Spark job #3064
• Add ClickHouse Client Add ClickHouse Client #3196
• Add data retention methods for in-memory Clickhouse deployment [Flow Visibility] Add data retention methods for in-memory Clickhouse deployment #3244

The json files will be uploaded manually by users in the Grafana-UI, similar to what we used to have in Kibana dashboard.
We plan to document the changes in antrea/docs/network-flow-visibility.md. As there are several separate changes happen at the same time, we are still discussing how we should update the doc efficiently.

Currently, this PR is not ready to be merged. Something is still pending. I also spoked to Ziyou about it. It will be hold until PR#3063 get merged.

[heanlan]

On pending:
Queries in the PR include a workaround. We are waiting for the 3rd-party grafana-clickhouse-plugin to release their latest fix of timeFilter.

[heanlan]

close this PR as move the dashboards into the deployment PR #3063