[Elastic-Agent] [Docker] Discuss: accessing logs from different container

[mtojek]

I'd like to ask for your recommendation for users that prefer to run Elastic Agent in the container (let's say due to security reasons).

Let's discuss the scenario:

The integrated product is nginx running in a container. It produces logs stored locally in the image and which are rotated. As the agent is running in a different container, it can't simply access produced logs.

What is your recommendation in this particular case? Should the user expose somehow log files? Mirror them?

Background -
I had an interesting talk with @ycombinator about possibilites and testing scenarios and it looks that we will both have to nail this problem (force agent to watch logs produced in a different container).

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[mtojek]

/cc @ruflin @michalpristas @ph

[blakerouse]

I don't think we are quite there yet with the Docker/Kubernetes story with Agent. This really comes down to dynamic configurations and the ability for Agent to interact directly with Docker and Kubernetes.

[ycombinator]

This really comes down to dynamic configurations and the ability for Agent to interact directly with Docker and Kubernetes.

This is the effort being tracked in elastic/beats#19225, yes?

[mtojek]

I understand. Thanks for responding! The reason why am I asking is because we'll need to somehow fetch logs from containers and wouldn't like to enforce a nasty workaround if there is a sophisticated method available.

EDIT:

There were few options on the table (rather quick wins, workarounds):

1. Install an agent on top of the other Docker image (with product like nginx, apache, etc.) or merge two images.
2. Expose via FUSE logs between Docker containers.

[blakerouse]

@ycombinator elastic/beats#19255 is the one.

At the moment in 7.9 there is no sophisticated method.

[ruflin]

elastic/beats#19225 is part of the solution. @mtojek I assume we just beats, you solve this with autodiscovery? If you want all logs, I would assume the "old" trick around mounting volumes should still work? https://www.elastic.co/guide/en/beats/filebeat/master/running-on-docker.html#_volume_mounted_configuration It assumes the nginx logs are not stored inside the container but written to file by docker.

[mtojek]

Not sure if I follow your idea.

The Elastic Agent runs as binary in a Docker container, together with filebeat and metricbeat. My question is: how can I expose directories with logs to these processes?

According to what @blakerouse confirmed, there is no specific method, hence I'm asking for some official recommendation :)

[ruflin]

The way it is done today (see link I provided) is that the Container in which Filebeat is running (in your case the Agent), it mounts volumes from the Docker Host where these logs are stored. Now Filebeat (Agent) must be pointed to these directories with the logs and tail them. The above follows the assumption that you can the default json-file logging driver and nginx container writes to it (not the log file inside your container).

I think I miss something on why what is possible with Filebeat should not be possible with Agent?

[mtojek]

I think I miss something on why what is possible with Filebeat should not be possible with Agent?

Most likely that's possible, but I didn't measure performance here. I'm not sure if there were similar tests executed. I'm looking for the method which causes less problems, e.g. docker container restarts, suddenly unmounted volumes, missing permissions.

There are couple of follow up questions:
Which container exposes the volume (host vs guest)? What if Elastic Agent or Nginx got restarted?

[ruflin]

As this is a way we recommended for quite some time I would expect it to work fairly well. @exekias Perhaps you can chime in here?

For your follow up questions:

• Nginx does not need any volume as the logging driver writes it to the right place
• Agent container needs to mount the volume from the host machine. It is not that the 2 containers share volumes with each other
• This should also solve the problem around restarting. All container logs are shipped, so if it is a new container, it has a new id and on disk a new directory, but agent will just ship it too.

[lunarok]

Hi,

Just move to elastic-agent with Fleet management.
And yes, that's something missin the "autodiscovery" feature to retrive all logs from docker and at same time having an option in different integrations to map the discovered logs to them (thinking MySQL, Traefik, Apache, Nginx ...)
I think an integration "Docker Logs" with the autodiscovery options configurable in the fleet management will be a starting point and ability for any integration that request a log file to use this instead as index.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[icc-garciaju]

Hi all.
Sorry for resurrecting such an old thread, but I think it's not resolved yet.
I think it could be a good idea to be able to call modules as if they where processors.
Something like this:

- id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
  type: filestream
  use_output: default
  meta:
    package:
      name: kubernetes
      version: 1.29.2
  data_stream:
    namespace: default
  streams:
    - id: container-log-${kubernetes.pod.name}-${kubernetes.container.id}
      data_stream:
        dataset: kubernetes.container_logs
        type: logs
      prospector.scanner.symlinks: true
      paths:
        - /var/log/containers/*${kubernetes.container.id}.log
      processors:
        - module:
            name: nginx
            when: ${kubernetes.labels.app} == "nginx"