Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

alloy does not read logs with the same timestamp #1977

Open
nordby opened this issue Oct 25, 2024 · 0 comments
Open

alloy does not read logs with the same timestamp #1977

nordby opened this issue Oct 25, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@nordby
Copy link

nordby commented Oct 25, 2024

What's wrong?

Pod logs with the same timestamp are ignored by alloy, only the first one gets into Loki
Image
Below is a screenshot of the logs from the node, which shows the same timestamp and container logsImage
Image

Steps to reproduce

run any application that writes a log with the same timestamp

System information

kubernetes 1.29.5

Software version

Grafana Alloy 1.4.3

Configuration

logging {
level = "error"
format = "json"
}

discovery.kubernetes "kubernetes_pods" {
role = "pod"
}

discovery.relabel "kubernetes_pods" {
targets = discovery.kubernetes.kubernetes_pods.targets

rule {
source_labels = ["__meta_kubernetes_pod_controller_name"]
regex = "([0-9a-z-.]+?)(-[0-9a-f]{8,10})?"
target_label = "__tmp_controller_name"
}

rule {
source_labels = ["__meta_kubernetes_pod_label_app_io_name", "__meta_kubernetes_pod_label_app", "__tmp_controller_name", "__meta_kubernetes_pod_name"]
regex = "^;([^;]+)(;.)?$"
target_label = "app"
}

rule {
source_labels = ["__meta_kubernetes_pod_label_app_io_component", "__meta_kubernetes_pod_label_component"]
regex = "^;([^;]+)(;.)?$"
target_label = "component"
}

rule {
source_labels = ["__meta_kubernetes_pod_node_name"]
target_label = "node_name"
}

rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}

rule {
source_labels = ["namespace", "app"]
separator = "/"
target_label = "job"
}

rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}

rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}

rule {
source_labels = ["__meta_kubernetes_pod_label_service"]
target_label = "service"
}
}

loki.source.kubernetes "kubernetes_pods" {
targets = discovery.relabel.kubernetes_pods.output
forward_to = [loki.process.pod_logs.receiver]
}

loki.process "pod_logs" {
stage.cri { }
forward_to = [loki.write.default.receiver]
}

loki.write "default" {
endpoint {
url = "http://xxx.xxx.xxx.xxx:3100/loki/api/v1/push"
}

external_labels = { kubernetes = "XXX"}
}

Logs

No response

@nordby nordby added the bug Something isn't working label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant