panic: runtime error: invalid memory address or nil pointer dereference

[sbancal]

Component(s)

receiver/filelog

What happened?

Description

Trying to set filelog receiver to delete_after_read, I get the panic runtime error described here.

Steps to Reproduce

Just run the app container (loki in this case), and docker.io/otel/opentelemetry-collector-contrib next to it, with an access to the volume where 1st container writes it's logs (/var/log/loki in this case).

Expected Result

otel collector to read content of every /var/log/loki/loki.log and delete that file when processed.

Actual Result

otel collector encounters an error (output copied below) and deletes the log file (which is probably the root cause for the repeated warn messages).

Note

The setup works before adding filelog.allowFileDeletion feature gate and:

    start_at: beginning
    delete_after_read: true

Collector version

0.95.0

Environment information

Environment

OS: Rocky 9

OpenTelemetry Collector configuration

---
exporters:
  loki:
    auth:
      authenticator: basicauth/otelcol
    endpoint: https://my-loki.example.com/loki/api/v1/push
  otlphttp/mimir:
    auth:
      authenticator: basicauth/otelcol
    endpoint: https://my-mimir.example.com/otlp
extensions:
  basicauth/otelcol:
    client_auth:
      password: p4s$w0rd
      username: otel_username
processors:
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: host, job, log.file.path
    - action: upsert
      key: job
      value: loki
    - action: upsert
      key: host
      value: my-loki.example.com
  batch: null
receivers:
  filelog:
    include:
    - /var/log/loki/*.log
    include_file_path: true
    start_at: beginning
    delete_after_read: true
    operators:
    - regex: ts=(?P<ts>\S+)
      timestamp:
        layout: '%Y-%m-%dT%H:%M:%S.%fZ'
        parse_from: attributes.ts
      type: regex_parser
    - field: attributes.ts
      type: remove
    - regex: level=(?P<level>\S+)
      type: regex_parser
  prometheus:
    config:
      scrape_configs:
      - job_name: loki
        scheme: http
        scrape_interval: 15s
        static_configs:
        - labels:
            instance: my-loki.example.com
          targets:
          - loki-loki:3100
service:
  extensions:
  - basicauth/otelcol
  pipelines:
    logs:
      exporters:
      - loki
      processors:
      - attributes
      - batch
      receivers:
      - filelog
    metrics:
      exporters:
      - otlphttp/mimir
      processors:
      - batch
      receivers:
      - prometheus
  telemetry:
    logs:
      level: warn

Log output

2024-02-23T13:53:47.886Z        warn    localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.       {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-02-23T13:53:48.286Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x41d3ddc]

goroutine 115 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd(0xc002a7ff10, {0x9191978, 0xc0029d3860})
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/internal/reader/reader.go:49 +0x3c
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).preConsume.func1(0xc0029d3720?)
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/file_other.go:47 +0x51
created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).preConsume in goroutine 96
        github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/file_other.go:45 +0x193
2024-02-23T13:53:48.785Z        warn    fileconsumer/file.go:46 finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:48.786Z        warn    localhostgate/featuregate.go:63 The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.       {"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-02-23T13:53:48.986Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:49.186Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:49.386Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:49.586Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:49.787Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:49.987Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-02-23T13:53:50.186Z        warn    fileconsumer/file.go:132        finding files: no files match the configured criteria   {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/filelog: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @sbancal, it looks like this was hit by another user as well, and should be resolved in the 0.96.0 release.

Could you try this again and see if it's resolved for you as well?

[sbancal]

Hi @crobert-1
I confirm that the 0.96.0 release fixes this issue.
Thank you : )