[Flow Visibility] Add Grafana and Clickhouse deployment file

[zyiou]

This PR is for adding deployment file and custom plugin for Grafana configuration and setup for Clickhouse, which are the UI solution and database solution for flow visibility long term architecture.

Deployment file:
It adds deployment for Clickhouse and Grafana with provisioning datasource and dashboards configuration.
Grafana sankey plugin:
A panel plugin is initialized from Grafana plugin template with Sankey diagram using react-google-chart.

[codecov-commenter]

Codecov Report

Merging #3063 (333076b) into main (3f30d84) will decrease coverage by 6.27%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #3063      +/-   ##
==========================================
- Coverage   60.85%   54.57%   -6.28%     
==========================================
  Files         266      373     +107     
  Lines       26520    51224   +24704     
==========================================
+ Hits        16139    27956   +11817     
- Misses       8591    20862   +12271     
- Partials     1790     2406     +616     

Flag	Coverage Δ
e2e-tests	53.54% <ø> (?)
integration-tests	36.31% <ø> (?)
kind-e2e-tests	47.67% <ø> (-0.29%)	⬇️
unit-tests	42.35% <ø> (+0.76%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/agent/cniserver/pod_configuration_linux.go	26.31% <0.00%> (-40.36%)	⬇️
pkg/controller/ipam/antrea_ipam_controller.go	48.71% <0.00%> (-31.57%)	⬇️
pkg/controller/networkpolicy/endpoint_querier.go	61.46% <0.00%> (-29.97%)	⬇️
pkg/controller/egress/controller.go	62.19% <0.00%> (-26.26%)	⬇️
.../registry/networkpolicy/clustergroupmember/rest.go	64.28% <0.00%> (-23.95%)	⬇️
pkg/agent/cniserver/ipam/antrea_ipam_controller.go	55.93% <0.00%> (-23.83%)	⬇️
pkg/agent/cniserver/ipam/antrea_ipam.go	55.55% <0.00%> (-23.62%)	⬇️
pkg/agent/util/ethtool/ethtool_linux.go	46.66% <0.00%> (-23.34%)	⬇️
pkg/controller/ipam/validate.go	57.95% <0.00%> (-22.05%)	⬇️
pkg/agent/util/iptables/lock.go	60.00% <0.00%> (-21.82%)	⬇️
... and 340 more

[zyiou]

Hi @antoninbas and @tnqn , this PR is still in progress, but I have a quick question about how/where to serve Grafana custom plugin (currently in build/yamls/flow-visibility/grafana-custom-plugin).
Grafana custom plugin is for creating some custom panels that are not supported by Grafana yet (in our case, sankey diagram for visualizing throughput over a time range). It is generated by grafana toolkit https://grafana.com/tutorials/build-a-panel-plugin/#create-a-new-plugin, so there are several components like license, default settings and configuration for React etc. To serve and use it, we need to sign the plugin, package it and upload with release (sample workflow: https://github.com/zyiou/grafana-custom-plugin/blob/main/.github/workflows/build_plugin.yml)
Since the programming language and layout are quite different from Antrea, I’m considering whether we can serve this in a separate repository. What do you suggest?

[antoninbas]

@zyiou could you put the plugin code under plugins/ maybe?

[heanlan]

A quick question @zyiou . Do we also need to include the README file in order to deploy the clickhouse operator?

[zyiou]

A quick question @zyiou . Do we also need to include the README file in order to deploy the clickhouse operator?

I'm planning to add all documentation once we have e2e work.

[heanlan]

I have another question: once we have the all the flow-visibility refactoring PRs merged, do we still want to keep the ELK related contents in the repo? shall we open another PR to remove all of them for v1.6 release? or postpone it to v1.7?

[heanlan]

By using "Clickhouse-Grafana" we might indicate it as an alternative to "ELK", so in that way it makes sense. On the other hand, it's also very long.

I have 2 observations:

• For consistency we might want to keep using "Flow Collector" terminology, at least for this release.
• Instead of referring to specific products making up the stack I'd just call it "Enhanced Flow Collector"

Thanks, Salvatore. cc @jianjuns @antoninbas for opinions. Thanks!

[dreamtalen]

I have another question: once we have the all the flow-visibility refactoring PRs merged, do we still want to keep the ELK related contents in the repo? shall we open another PR to remove all of them for v1.6 release? or postpone it to v1.7?

I think we could keep ELK related stuff in v1.6 release. In order to fully move to our new flow visibility arch, we also need some changes on Flow Aggregator side (deleting unused fields in new arch, debugging cqe test, etc), which may take a while before finished.

[jianjuns]

I feel it depends on how complete the new implementation is (which you guys know better). If it is not that complete, probably let us still keep ELK support for a while.

For naming, another way is to name it something you guys like, not necessary description of the functionalities.

[heanlan]

Another alternative I can think of is "Grafana Flow Collector".

• it's shorter
• satisfies the "Flow Collector" terminology consistency
• from the users perspective they will only directly interact with Grafana

I'll go-ahead and do the refactoring if that sounds good to all of us.

[heanlan]

Hi @antoninbas @jianjuns , would you mind having a look at this PR again?

[jianjuns]

I do not have further comments on the PR, but we need a reviewer with enough knowledge to approve.

[antoninbas]

Overall LGTM. Would be good if someone else beyond the person working on this PR could validate the installation instructions by trying them out.

[antoninbas]

Should we encourage users to edit the clickhouse-secret Secret and use non-default credentials? I think that may be a good idea.

[heanlan]

Thanks, I added a credential configuration section in the latest commit for review, where I also added the Grafana login credentials configuration. There's one issue: flow-aggregator.yml also has a Secret with the same contents as clickhouse-secret. Their namespaces are different, so they cannot share the same Secret. Users need to make the changes at two different locations.

I feel it can be addressed if we later decide to merge flow-aggregator and flow-visibility namespaces into one. But at the current stage, I'm not sure whether we want to add this customization support. Do you have any opinions on that?

[antoninbas]

that sounds fine for now. I want us to support Helm as an installation method later one, and I believe it will make it easier to keep everything consistent.

[yanjunz97]

We may need to mention Grafana flow collector in this paragraph

[heanlan]

We may need to mention Grafana flow collector in this paragraph

Thanks @yanjunz97 , could you please also help to try out the instructions of deployment?

[yanjunz97]

Overall LGTM. Would be good if someone else beyond the person working on this PR could validate the installation instructions by trying them out.

I have worked on the validation. I found an error in Grafana dashboard when trying to set a customized username and password for Clickhouse. After some discussions and troubleshooting with @heanlan , we found the root cause. Anlan has updated the files to fix it.

After this fix, I tried out all the installation instructions again and validated that they all work as expected.

[antoninbas]

LGTM

[heanlan]

/test-all

[antoninbas]

@heanlan can I go ahead and merge this PR?

[heanlan]

Yes, please @antoninbas