Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Observed timestamp in logs is not exported to clickhouse #34150

Open
pimeys opened this issue Jul 18, 2024 · 4 comments
Open

Observed timestamp in logs is not exported to clickhouse #34150

pimeys opened this issue Jul 18, 2024 · 4 comments
Labels
bug Something isn't working exporter/clickhouse

Comments

@pimeys
Copy link
Contributor

pimeys commented Jul 18, 2024

Component(s)

exporter/clickhouse

What happened?

Description

Exporting logs from opentelemetry-rust stores zero timestamps to clickhouse.

Steps to Reproduce

  1. Send logs to opentelemetry-collector-contrib server from a Rust application using opentelemetry-rust
  2. Export to clickhouse
  3. Timestamps are zero

Expected Result

The exporter should look first the timestamp and if null take the value from the observed timestamp field.

Actual Result

The exporter only reads the timestamp field, blindly converts it to timestamp and stores a zero value to clickhouse.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/exporter_logs.go#L93

Collector version

v0.105.0

Environment information

Environment

OS: The official collector docker image ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.105.0

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 100000
    timeout: 1s

exporters:
  clickhouse:
    endpoint: tcp://clickhouse:9000
    database: otel
    ttl: 48h
    timeout: 10s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhouse]

Log output

No response

Additional context

Should be a quick fix, I'm not a Go developer but planning to take a look :)
@pimeys pimeys added bug Something isn't working needs triage New item requiring triage labels Jul 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@pimeys pimeys mentioned this issue Jul 18, 2024
Merged
@github-actions github-actions bot mentioned this issue Jul 23, 2024
Closed
@crobert-1
Copy link
Member

crobert-1 commented Jul 29, 2024 •
edited
Loading

Code owners have approved the PR resolving this, removing needs triage.

@crobert-1 crobert-1 removed the needs triage New item requiring triage label Jul 29, 2024
codeboten added a commit that referenced this issue Aug 7, 2024
@pimeys @crobert-1
@mx-psi @codeboten
[exporter/clickhouse] use observed timestamp in logs (#34151)
6d8e6f6 
**Description:** Some senders, such as the rust-opentelemetry do _not_
send timestamp in their logs. In these cases, if the timestamp is not
set, we must use the observed timestamp instead.

**Link to tracking Issue:**
#34150

**Testing:** Added a test which does not define timestamp

---------

Co-authored-by: Curtis Robert <[email protected]>
Co-authored-by: Pablo Baeyens <[email protected]>
Co-authored-by: Alex Boten <[email protected]>
f7o pushed a commit to f7o/opentelemetry-collector-contrib that referenced this issue Sep 12, 2024
@pimeys @crobert-1
@mx-psi @codeboten @f7o
[exporter/clickhouse] use observed timestamp in logs (open-telemetry#…
a977c77 
…34151)

**Description:** Some senders, such as the rust-opentelemetry do _not_
send timestamp in their logs. In these cases, if the timestamp is not
set, we must use the observed timestamp instead.

**Link to tracking Issue:**
open-telemetry#34150

**Testing:** Added a test which does not define timestamp

---------

Co-authored-by: Curtis Robert <[email protected]>
Co-authored-by: Pablo Baeyens <[email protected]>
Co-authored-by: Alex Boten <[email protected]>
@github-actions GitHub Actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Sep 30, 2024
@SpencerTorres
Copy link
Member

This was fixed/merged in #34151 and should've been closed automatically

cc @crobert-1

@github-actions github-actions bot removed the Stale label Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working exporter/clickhouse
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pimeys @SpencerTorres @crobert-1