If you can't find help here, don't hesitate to open an issue or a Q&A. There are several repositories under netobserv github org, but it is fine to centralize these in network-observability-operator.
- Q&A
- How-to
- Troubleshooting
No! While some features are developed primarily for OpenShift, we want to keep it on track with other / "vanilla" Kubes. For instance, there has been some work to make the console plugin run as a standalone, or the operator to manage upstream (non-OpenShift) ovn-kubernetes.
And if something is not working as hoped with your setup, you are welcome to contribute to the project ;-)
It depends on which agent you want to use: ebpf or ipfix, and whether you want to get the OpenShift Console plugin.
What matters is the version of the Linux kernel: 4.18 or more is supported. Earlier versions are not tested.
Other than that, there are no known restrictions on the Kubernetes version.
OpenShift 4.10 or above, or upstream OVN-Kubernetes, are recommended, as the operator will configure OVS for you.
For other CNIs, you need to find out if they can export IPFIX, and configure them accordingly.
OpenShift 4.10 or above is required.
Make sure all pods are up and running:
# Assuming configured namespace is netobserv (default)
kubectl get pods -n netobservShould provide results similar to this:
NAME READY STATUS RESTARTS AGE
flowlogs-pipeline-5rrg2 1/1 Running 0 43m
flowlogs-pipeline-cp2lb 1/1 Running 0 43m
flowlogs-pipeline-hmwxd 1/1 Running 0 43m
flowlogs-pipeline-wmx4z 1/1 Running 0 43m
grafana-6dbddc9869-sxn62 1/1 Running 0 31m
loki 1/1 Running 0 43m
netobserv-controller-manager-7487d87dc-2ltq2 2/2 Running 0 43m
netobserv-plugin-7fb8c5477b-drg2z 1/1 Running 0 43m
Results may slightly differ depending on the installation method and the FlowCollector configuration. At least you should see flowlogs-pipeline pods in a Running state.
If you use the eBPF agent, check also for pods in privileged namespace:
# Assuming configured namespace is netobserv (default)
kubectl get pods -n netobserv-privilegedNAME READY STATUS RESTARTS AGE
netobserv-ebpf-agent-7rwtk 1/1 Running 0 7s
netobserv-ebpf-agent-c7nkv 1/1 Running 0 7s
netobserv-ebpf-agent-hbjz8 1/1 Running 0 7s
netobserv-ebpf-agent-ldj66 1/1 Running 0 7s
Finally, make sure Loki is correctly deployed, and reachable from pods via the URL defined in spec.loki.url. You can for instance check using this command:
kubectl exec $(kubectl get pod -l "app=flowlogs-pipeline" -o name) -- curl -G -s "`kubectl get flowcollector cluster -o=jsonpath={.spec.loki.url}`loki/api/v1/query" --data-urlencode 'query={app="netobserv-flowcollector"}' --data-urlencode 'limit=1'It should return some json in this form:
{"status":"success","data":{"resultType":"streams","result":[...],"stats":{...}}}
If using IPFIX (ie. spec.agent.type is IPFIX in FlowCollector), wait 10 minutes and check again. There is sometimes a delay, up to 10 minutes, before the flows appear. This is due to the IPFIX protocol requiring exporter and collector to exchange record template definitions as a preliminary step. The eBPF agent doesn't have such a delay.
Else, check for any suspicious error in logs, especially in the flowlogs-pipeline pods and the eBPF agent pods. You may also take a look at prometheus metrics prefixed with netobserv_: they can give you clues if flows are processed, if errors are reported, etc.
Finally, don't hesitate to open an issue.
Make sure your cluster version is at least OpenShift 4.10: prior versions have no (or incompatible) console plugin SDK.
Make sure that spec.consolePlugin.register is set to true (default).
If not, or if for any reason the registration seems to have failed, you can still do it manually by editing the Console Operator config:
kubectl edit console.operator.openshift.io clusterIf it's not already there, add the plugin reference:
spec:
plugins:
- netobserv-pluginEnsure console pods are all in Running state using the following command:
oc get pods -n openshift-console -l app=consoleIf you had previously used the console with the plugin installed, you may need to restart console pods to clear cache:
oc delete pods -n openshift-console -l app=consoleIf the new views still don't show up, try clearing your browser cache and refreshing. Check also the netobserv-console-plugin-... pod status and logs.
kubectl get pods -n netobserv -l app=netobserv-plugin
kubectl logs -n netobserv -l app=netobserv-pluginIt is also possible that the OpenShift Console is failing to restart. There is for instance a known issue with very small clusters, causing this problem.
kubectl get pods -n openshift-consoleNAME READY STATUS RESTARTS AGE
console-65486f987c-bk6zn 1/1 Running 0 44m
console-67fd7b6f49-l5nmq 0/1 Pending 0 10m
console-67fd7b6f49-wmhp4 0/1 Pending 0 10m
downloads-5fc6cc467f-2fttx 1/1 Running 0 45m
downloads-5fc6cc467f-2gfpw 1/1 Running 0 45m
To force a restart, kill the running console pod using the following command:
oc delete pods -n openshift-console -l app=console
Note that this will make the console unavailable for some time
I first deployed flowcollector, and then kafka. Flowlogs-pipeline is not consuming any flow from Kafka
This is a known bug in one of flowlogs-pipeline dependencies.
Please recreate the flowlogs-pipeline pods by either killing them maunally or deleting and recreating the flow collector object.
br-ex and br-int are virtual bridge devices,
so they operate at OSI Layer 2 (e.g. Ethernet level). The eBPF agent works at Layers 3 and 4
(IP and TCP level), so it is expected that traffic passing through br-int and br-ex is captured
by the agent when it is processed by other interfaces (e.g. physical host or virtual pod interfaces).
This means that, if you restrict the agent interfaces (using the interfaces or excludeInterfaces
properties) to attach only to br-int and/or br-ex, you won't be able to see any flow.