[aws-for-fluent-bit] unable to set [SERVICE] values

[Justin-DynamicD]

The current configmap tempalte is very limited, and doesn't allow any sort of configuration with the [SERVICES] block. For example, it doesn't seem possible to set the below at all:

[SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Listen               0.0.0.0
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M

in the above example, only the parsers file is able to be set. It would be nice that if, in addition to service.extraParsers, it would be nice if we could define additional values.

[endersonmaia]

just for reference

• https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/templates/configmap.yaml
• https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit/values.yaml#L178-L232

[endersonmaia]

The README has references to volumes and volumeMounts as a way to define fluent-bit configuration, I just don't know yet how to mount a volume and get it included as part of the fluet-bit daemon

https://github.com/aws/eks-charts/blob/master/stable/aws-for-fluent-bit/README.md?plain=1#L99-L100

[jseiser]

Just wanted to bump this as well, we are wanting to use this chart to configure Container Insights. AWS Supported pointed us this way like this chart would just be ready but Im not actually seeing how to make this happen.

[endersonmaia]

Just wanted to bump this as well, we are wanting to use this chart to configure Container Insights. AWS Supported pointed us this way like this chart would just be ready but Im not actually seeing how to make this happen.

It works, @jseiser.

It's just not flexible enough, if you need to make changes to [SERVICE], this issue's topic.

What problem are you facing ?

[endersonmaia]

The README has references to volumes and volumeMounts as a way to define fluent-bit configuration, I just don't know yet how to mount a volume and get it included as part of the fluet-bit daemon

https://github.com/aws/eks-charts/blob/master/stable/aws-for-fluent-bit/README.md?plain=1#L99-L100

you could just override a configmap named fluentbit-config with the configuration you need

it should work, but I don't think it's the better approach for a helm chart; ideally, we could leverage the templates and reuse a base config and extend it as needed

the link I posted in a previous comment shows a better approach, IMHO, having the fluent-bit config section being provided by different template sections, with sane defaults, and allowing you to easily change the sections you need

[jseiser]

@endersonmaia

It works, @jseiser.

It's just not flexible enough, if you need to make changes to [SERVICE], this issue's topic.

What problem are you facing ?

I do not see how to make this chart, reflect what is detailed here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

[endersonmaia]

@endersonmaia

It works, @jseiser.
It's just not flexible enough, if you need to make changes to [SERVICE], this issue's topic.
What problem are you facing ?

I do not see how to make this chart, reflect what is detailed here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html

the link you shared is not using the helm chart, that won't work

[jseiser]

@endersonmaia

That link, is detailing how AWS says to configure Fluent-bit for Container Insights. You would think the AWS Chart for Fluent bit, would easily be able to be deployed and configured to match the AWS documentation for deploying fluent bit.

[endersonmaia]

@endersonmaia

That link, is detailing how AWS says to configure Fluent-bit for Container Insights. You would think the AWS Chart for Fluent bit, would easily be able to be deployed and configured to match the AWS documentation for deploying fluent bit.

yep, that should be the case, but there's not only one way to configure a k8s deployment, right ?

and we're talking about AWS, https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/

jokes aside, if you want to use helm charts, don't follow that link

actually, I'm in the process of using the suggested configMap at this spec and maybe even use this chart instead

[jseiser]

@endersonmaia

actually, I'm in the process of using the suggested configMap at this spec and maybe even use this chart instead

I actually did that last night and it worked fine so far.

serviceAccount:
    create: "true"
    name: "fluentbit-sa"
    annotations:
      eks.amazonaws.com/role-arn: ${role_arn}

env:
  - name: HOST_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

config:
  service: |
    [SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        Health_Check              On
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M

  inputs: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_container.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      Off
    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/fluent-bit*
        Parser              docker
        DB                  /var/fluent-bit/state/flb_log.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      Off
    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/cloudwatch-agent*
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  cwagent_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_cwagent.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      Off
  filters: |
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
    [FILTER]
        Name                modify
        Match               dataplane.systemd.*
        Rename              _HOSTNAME                   hostname
        Rename              _SYSTEMD_UNIT               systemd_unit
        Rename              MESSAGE                     message
        Remove_regex        ^((?!hostname|systemd_unit|message).)*$

    [FILTER]
        Name                aws
        Match               dataplane.*
        imds_version        v1
  outputs: |
    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              us-gov-west-1
        log_group_name      /aws/containerinsights/${cluster_name}/application
        log_stream_prefix   $${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
        log_retention_days  7
        metric_namespace    amazon-cloudwatch


  customParsers: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

    [PARSER]
        Name                syslog
        Format              regex
        Regex               ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key            time
        Time_Format         %b %d %H:%M:%S

    [PARSER]
        Name                container_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

    [PARSER]
        Name                cwagent_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

extraFile:
  dataplane-log.conf: |
    [INPUT]
        Name                systemd
        Tag                 dataplane.systemd.*
        Systemd_Filter      _SYSTEMD_UNIT=docker.service
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        DB                  /var/fluent-bit/state/systemd.db
        Path                /var/log/journal
        Read_From_Tail      On

    [INPUT]
        Name                tail
        Tag                 dataplane.tail.*
        Path                /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_dataplane_tail.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      Off

    [FILTER]
        Name                modify
        Match               dataplane.systemd.*
        Rename              _HOSTNAME                   hostname
        Rename              _SYSTEMD_UNIT               systemd_unit
        Rename              MESSAGE                     message
        Remove_regex        ^((?!hostname|systemd_unit|message).)*$

    [FILTER]
        Name                aws
        Match               dataplane.*
        imds_version        v1

    [OUTPUT]
        Name                cloudwatch_logs
        Match               dataplane.*
        region              us-gov-west-1
        log_group_name      /aws/containerinsights/${cluster_name}/dataplane
        log_stream_prefix   $${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights

  host-log.conf: |
    [INPUT]
        Name                tail
        Tag                 host.dmesg
        Path                /var/log/dmesg
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_dmesg.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      Off

    [INPUT]
        Name                tail
        Tag                 host.messages
        Path                /var/log/messages
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_messages.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      Off

    [INPUT]
        Name                tail
        Tag                 host.secure
        Path                /var/log/secure
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_secure.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      Off

    [FILTER]
        Name                aws
        Match               host.*
        imds_version        v1

    [OUTPUT]
        Name                cloudwatch_logs
        Match               host.*
        region              us-gov-west-1
        log_group_name      /aws/containerinsights/${cluster_name}/host
        log_stream_prefix   $${HOST_NAME}.
        auto_create_group   true
        extra_user_agent    container-insights


daemonSetVolumes:
  - name: fluentbitstate
    hostPath:
      path: /var/fluent-bit/state
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: runlogjournal
    hostPath:
      path: /run/log/journal
  - name: dmesg
    hostPath:
      path: /var/log/dmesg

daemonSetVolumeMounts:
  - name: fluentbitstate
    mountPath: /var/fluent-bit/state
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: runlogjournal
    mountPath: /run/log/journal
    readOnly: true
  - name: dmesg
    mountPath: /var/log/dmesg
    readOnly: true

[svrviny1324]

Hi , here I am trying to implement that the logs which are related to same pod i want to send it as single json instead of multiple json into cloudwatch using fluent bit
how i can write input,filter,parser?
Note: Here iam using configmap same as above.
below is my logs present sending it to cloudwatch in this formate
{
"log": "2023-04-17T12:30:06.484012995Z stdout F ",
"kubernetes": {
"pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb",
"namespace_name": "",
"pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b",
"host": "",
"container_name": "resetter",
"docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c",
"container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298",
"container_image": ""
}
}
{
"log": "2023-04-17T12:30:06.484049686Z stdout F ",
"kubernetes": {
"pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb",
"namespace_name": "",
"pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b",
"host": "",
"container_name": "resetter",
"docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c",
"container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298",
"container_image": ""
}
}
iam expecting now is
{
"kubernetes": {
"pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb",
"namespace_name": "",
"pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b",
"host": "",
"container_name": "resetter",
"docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c",
"container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298",
"container_image": ""
},
{
"log": "2023-04-17T12:30:06.484049686Z stdout F "
},
{
"log": "2023-04-17T12:30:06.484049686Z stdout F "
}