Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stopping Alloy in a docker compose project only shuts down after receiving the SIGKILL signal #1980

Open
vpmedia opened this issue Oct 28, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@vpmedia
Copy link

vpmedia commented Oct 28, 2024

What's wrong?

If I try to stop my docker compose project's containers (docker compose down), Alloy only shuts down after receiving SIGKILL.
I would expect that it would shut down gracefully receiving SIGTERM signal.
https://docs.docker.com/compose/support-and-feedback/faq/#why-do-my-services-take-10-seconds-to-recreate-or-stop

✔ Container dev-alloy-1 Removed 10.1s

Steps to reproduce

Try to shutdown a docker compose project including Alloy.
For the full list of services being used see the log file below.

System information

MacOS 15 and Ubuntu Linux

Software version

grafana/alloy:v1.4.3

Configuration

livedebugging {
    enabled = true
}

logging {
    level  = "info"
    format = "json"
}

discovery.docker "flog_scrape" {
    host             = "unix:///var/run/docker.sock"
    refresh_interval = "10s"
}

discovery.relabel "flog_scrape" {
    targets = []

    rule {
        source_labels = ["__meta_docker_container_name"]
        regex         = "/(.*)"
        target_label  = "container"
    }

    rule {
        source_labels = ["__meta_docker_container_log_stream"]
        target_label  = "logstream"
    }

    rule {
        source_labels = ["__meta_docker_container_label_logging_jobname"]
        target_label  = "job"
    }

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
        target_label  = "service_name"
    }
}

loki.source.docker "flog_scrape" {
    host             = "unix:///var/run/docker.sock"
    targets          = discovery.docker.flog_scrape.targets
    forward_to       = [loki.write.default.receiver]
    relabel_rules    = discovery.relabel.flog_scrape.rules
    refresh_interval = "10s"
}

loki.write "default" {
    endpoint {
        url = "http://loki:3100/loki/api/v1/push"
    }
    external_labels = {
        deployment_environment = env("ALLOY_LOKI_LABEL_DEPLOYMENT_ENVIRONMENT"),
        host_name = env("ALLOY_LOKI_LABEL_HOST_NAME"),
    }
}

otelcol.receiver.otlp "default" {
    grpc {
        endpoint = "0.0.0.0:4317"
    }

    http {
        endpoint = "0.0.0.0:4318"
        cors {
            allowed_origins = [
                "http://127.0.0.1",
            ]
        }
    }

    output {
        metrics = [otelcol.processor.batch.default.input]
        logs    = [otelcol.processor.batch.default.input]
        traces  = [otelcol.processor.batch.default.input]
    }
}

otelcol.processor.batch "default" {
    output {
        metrics = [otelcol.exporter.prometheus.default.input]
        logs    = [otelcol.exporter.loki.default.input]
        traces  = [otelcol.exporter.otlp.default.input]
    }
}

otelcol.exporter.debug "default" {}

otelcol.exporter.otlp "default" {
    client {
        endpoint = "tempo:4317"

        tls {
            insecure = true
            insecure_skip_verify = true
        }
    }
}

otelcol.exporter.prometheus "default" {
        forward_to = [prometheus.remote_write.default.receiver]
    }

prometheus.remote_write "default" {
    endpoint {
        url = "http://prometheus:9090/api/v1/write"
        send_native_histograms = false
    }
}

otelcol.exporter.loki "default" {
    forward_to = [loki.write.default.receiver]
}

Logs

✔ Container dev-prometheus_mysql_exporter-1 Removed 0.3s
✔ Container dev-prometheus_redis_exporter-1 Removed 0.2s
✔ Container dev-prometheus_alertmanager-1 Removed 0.2s
✔ Container dev-grafana-1 Removed 0.3s
✔ Container dev-alloy-1 Removed 10.1s <- exited after default 10s shutdown timeout
✔ Container dev-prometheus_node_exporter-1 Removed 0.4s
✔ Container dev-prometheus_blackbox_exporter-1 Removed 0.2s
✔ Container dev-redis-1 Removed 0.2s
✔ Container dev-mysql-1 Removed 1.0s
✔ Container dev-prometheus-1 Removed 0.1s
✔ Container dev-loki-1 Removed 2.0s
✔ Container dev-tempo-1 Removed 1.9s
✔ Network dev_default Removed 0.1s

...

tempo-1 exited with code 0
loki-1 exited with code 0
alloy-1 exited with code 137 <- received SIGKILL

@vpmedia vpmedia added the bug Something isn't working label Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant