ClusterFlow exclude logs feature is not working properly

[sjanorkar]

Bugs should be filed for issues encountered whilst operating logging-operator.
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors. #logging-operator
Please provide as much detail as possible.

Describe the bug:
ClusterFlow exclude logs feature is not working properly. Even after excluding the logs with the help of labels, container names I can see the logs pushed to the output.

Expected behaviour:
All the labels, namespaces and container_names mentioned in match.exclude[] should not be present in the log destination

Steps to reproduce the bug:
Use default logging Operator config.

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: aicore-logs-flow
  namespace: logging
spec:
  filters:
    - tag_normaliser: {}
    - record_transformer:
        enable_ruby: true
        records:
          - namespace: ${record["kubernetes"]["namespace_name"]}
          - app: ${record["kubernetes"]["labels"]["app"]}
          - cluster_name: {{ .Values.environmentName }}
          - container: ${record["kubernetes"]["container_name"]}
          - pod: ${record["kubernetes"]["pod_name"]}
          - node_name: ${record["kubernetes"]["host"]}
          - ai_sap_com_tenantId: aicore
        remove_keys: kubernetes
  match:
    - exclude:
        labels:
          app: kubernetes # apiserver-proxy, kube-proxy-aalpha-worker, kube-proxy-loki, kube-proxy-node, kube-proxy-prometheus
    - exclude:
        container_names:
          - nginx
          - kube-proxy
    - exclude:
        labels:
          app: <label>
    - exclude:
        labels:
          app: <label>
    - exclude:
        labels: 
          k8s-app: <label>
    - exclude:
        namespaces:
          - prometheus
          - default 
          - tests
  globalOutputRefs:
    -logs-output

Additional context:
Add any other context about the problem here.

Environment details:

• Kubernetes version (e.g. v1.15.2):
• Cloud-provider/provisioner (e.g. AKS, GKE, EKS, PKE etc):
• logging-operator version (e.g. 2.1.1):
• Install method (e.g. helm or static manifests):
• Logs from the misbehaving component (and any other relevant logs):
• Resource definition (possibly in YAML format) that caused the issue, without sensitive data:

/kind bug

[pepov]

Do you have any other clusterflows in the system? Based on the above ClusterFlow you should not get any logs at all, since there should be at least a 'select: {}' rule at the end.

[sjanorkar]

No, this is the only active clusterflow. Actually I do see all the logs in the output. Seems like exclude feature isn't working properly.

[pepov]

Which logging operator version do you use?

[pepov]

Also what happens if you add a select statement, that filters for a non existing label at the end?

[pepov]

@sjanorkar any update? Which version are you running?