Client-side Throttling API calls for namespace when k8sobjectsreceiver no longer monitoring that namespace #31644

kenfinnigan · 2024-03-07T19:53:37Z

Component(s)

receiver/k8sobjects

What happened?

Description

Lumigo has an operator which uses the OTeL Collector builder to construct an instance of the collector for in cluster collection and processing before transmitting traces/logs/etc to a backend.

The operator tracks which namespaces need to be monitored, and causes the OTeL Collector instance to be rebuilt when namespaces are added or removed. For each namespace being monitored, it defines the k8sobjectsreceiver with k8s objects and events for a specific set of object kinds (Pod/Deployment/etc).

When a namespace is removed from monitoring, and the Collector is restarted with any receiver config for that namespace removed, there are large numbers of errors in the Collector logs about API requests being throttled.

Steps to Reproduce

Deploy the Lumigo operator to a cluster with Helm, as per this.
Follow the instructions to monitor a namespace
Verify the namespace is monitored by viewing the telemetry-proxy container (the internal Collector instance) logs
Remove the namespace from monitoring by running kubectl delete -n {ns} lumigo lumigo
Verify the telemetry-proxy container was restarted from the logs

Expected Result

Collector is no longer retrieving k8s events and objects for the namespace.

Actual Result

Errors in the log for kube api calls to the no longer monitored namespace being throttled:

I0307 16:43:34.491552       1 request.go:697] Waited for 1.393232136s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/statefulsets
I0307 16:43:44.491739       1 request.go:697] Waited for 1.391941742s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/replicasets
I0307 16:43:54.491976       1 request.go:697] Waited for 1.394410791s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/apis/apps/v1/namespaces/node2/daemonsets
I0307 16:44:04.492993       1 request.go:697] Waited for 1.385453868s due to client-side throttling, not priority and fairness, request: GET:https://{ip_address}:443/api/v1/namespaces/node2/events?fieldSelector=involvedObject.apiVersion%3Dapps%2Fv1%2CinvolvedObject.kind%3DStatefulSet%2CinvolvedObject.uid%21%3D%27%27%2CinvolvedObject.uid%21%3D

Collector version

v0.89.0

Environment information

Environment

OS: Alpine

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      http:
        auth:
          authenticator: lumigoauth/server
        include_metadata: true # Needed by `headers_setter/lumigo`
  k8sobjects/objects_ns_node1:
    auth_type: serviceAccount
    objects:
    - name: pods
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: daemonsets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: deployments
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: replicasets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: statefulsets
      group: apps
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: cronjobs
      group: batch
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: jobs
      group: batch
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
    - name: pods
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: daemonsets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: deployments
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: replicasets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: statefulsets
      group: apps
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: cronjobs
      group: batch
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
    - name: jobs
      group: batch
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
  k8sobjects/events_ns_node1:
    auth_type: serviceAccount
    objects:
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=v1,involvedObject.kind=Pod,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=DaemonSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=Deployment,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=ReplicaSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=StatefulSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=CronJob,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: watch
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=Job,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=v1,involvedObject.kind=Pod,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=DaemonSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=Deployment,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=ReplicaSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=apps/v1,involvedObject.kind=StatefulSet,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=CronJob,involvedObject.uid!='',involvedObject.uid!=
    - name: events
      mode: pull
      interval: 10m
      namespaces: [ node1 ]
      # Filter out events without the involved object's UID, e.g., the ones generated by our webhook
      field_selector: involvedObject.apiVersion=batch/v1,involvedObject.kind=Job,involvedObject.uid!='',involvedObject.uid!=

extensions:
  health_check:
  headers_setter/lumigo:
    headers:
    - key: authorization
      from_context: Authorization
      action: upsert

exporters:
  otlphttp/lumigo:
    endpoint: https://ga-otlp.lumigo-tracer-edge.golumigo.com
    auth:
      authenticator: headers_setter/lumigo
  logging:
    verbosity: detailed
    sampling_initial: 1
    sampling_thereafter: 1
  otlphttp/lumigo_ns_node1:
    endpoint: $LUMIGO_ENDPOINT
    auth:
      authenticator: lumigoauth/ns_node1

processors:
  transform/set_k8s_objects_scope:
    log_statements:
    - context: scope
      statements:
      - set(name, "lumigo-operator.k8s-objects")
      - set(version, "28")
  transform/set_k8s_events_scope:
    log_statements:
    - context: scope
      statements:
      - set(name, "lumigo-operator.k8s-events")
      - set(version, "28")
  batch/k8s_objects_ns_node1:
    send_batch_size: 100
    timeout: 1s
  batch/k8s_events_ns_node1:
    send_batch_size: 100
    timeout: 1s

service:
  telemetry:
    logs:
      level: debug
  extensions:
  - headers_setter/lumigo
  - health_check
  - lumigoauth/server
  - lumigoauth/ns_node1
  pipelines:
    traces:
      receivers:
      - otlp
      exporters:
      - otlphttp/lumigo
    logs/k8s_objects_ns_node1:
      receivers:
      - k8sobjects/objects_ns_node1
      processors:
      - transform/set_k8s_objects_scope
      - batch/k8s_objects_ns_node1
      exporters:
      - otlphttp/lumigo_ns_node1
    logs/k8s_events_ns_node1:
      receivers:
      - k8sobjects/events_ns_node1
      processors:
      - transform/set_k8s_events_scope
      - batch/k8s_events_ns_node1
      exporters:
      - logging
      - otlphttp/lumigo_ns_node1

Log output

2024-03-07T16:37:00.682Z	info	[email protected]/service.go:143	Starting lumigo-collector...	{"Version": "0.0.1", "NumCPU": 2}
2024-03-07T16:37:00.682Z	info	extensions/extensions.go:34	Starting extensions...
2024-03-07T16:37:00.682Z	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:37:00.682Z	info	extensions/extensions.go:45	Extension started.	{"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:37:00.682Z	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "health_check"}
2024-03-07T16:37:00.682Z	info	[email protected]/healthcheckextension.go:35	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-03-07T16:37:00.682Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:37:00.682Z	info	extensions/extensions.go:45	Extension started.	{"kind": "extension", "name": "health_check"}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs"}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "watch", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.683Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs", "gvr": "/v1, Resource=events", "mode": "pull", "namespaces": ["node2"]}
2024-03-07T16:37:00.705Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/objects_ns_node2", "data_type": "logs"}
2024-03-07T16:37:00.725Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "receiver", "name": "otlp", "data_type": "traces", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:37:00.725Z	info	[email protected]/otlp.go:101	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2024-03-07T16:37:00.725Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:37:00.725Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:37:00.726Z	info	healthcheck/handler.go:132	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}
2024-03-07T16:37:00.726Z	info	[email protected]/service.go:169	Everything is ready. Begin running and processing data.

Reloading configurations
2024-03-07T16:41:34.890Z	info	[email protected]/collector.go:258	Received signal from OS	{"signal": "hangup"}
2024-03-07T16:41:34.890Z	warn	[email protected]/collector.go:199	Config updated, restart service
2024-03-07T16:41:34.890Z	info	[email protected]/service.go:178	Starting shutdown...
2024-03-07T16:41:34.890Z	info	healthcheck/handler.go:132	Health Check state change	{"kind": "extension", "name": "health_check", "status": "unavailable"}
2024-03-07T16:41:34.891Z	info	[email protected]/receiver.go:82	Object Receiver stopped	{"kind": "receiver", "name": "k8sobjects/objects_ns_node2", "data_type": "logs"}
2024-03-07T16:41:34.891Z	info	[email protected]/receiver.go:82	Object Receiver stopped	{"kind": "receiver", "name": "k8sobjects/events_ns_node2", "data_type": "logs"}
2024-03-07T16:41:34.892Z	info	[email protected]/receiver.go:82	Object Receiver stopped	{"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:34.892Z	info	[email protected]/receiver.go:82	Object Receiver stopped	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:34.894Z	info	extensions/extensions.go:52	Stopping extensions...
2024-03-07T16:41:34.895Z	info	[email protected]/service.go:192	Shutdown complete.
2024-03-07T16:41:35.016Z	info	[email protected]/telemetry.go:85	Setting up own telemetry...
2024-03-07T16:41:35.016Z	info	[email protected]/telemetry.go:202	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2024-03-07T16:41:35.016Z	debug	[email protected]/extension.go:162	Beta component. May change in the future.	{"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.016Z	debug	[email protected]/extension.go:162	Stable component.	{"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.016Z	debug	[email protected]/extension.go:162	Stable component.	{"kind": "extension", "name": "lumigoauth/ns_node1"}
2024-03-07T16:41:35.016Z	debug	[email protected]/exporter.go:273	Beta component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "otlphttp/lumigo_ns_node1"}
2024-03-07T16:41:35.018Z	debug	[email protected]/processor.go:287	Stable component.	{"kind": "processor", "name": "batch/k8s_objects_ns_node1", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.018Z	debug	[email protected]/processor.go:287	Alpha component. May change in the future.	{"kind": "processor", "name": "transform/inject_operator_details_into_resource", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.018Z	debug	[email protected]/processor.go:287	Stable component.	{"kind": "processor", "name": "batch/k8s_events_ns_node1", "pipeline": "logs/k8s_events_ns_node1"}
2024-03-07T16:41:35.018Z	debug	[email protected]/processor.go:287	Alpha component. May change in the future.	{"kind": "processor", "name": "transform/set_k8s_objects_scope", "pipeline": "logs/k8s_objects_ns_node1"}
2024-03-07T16:41:35.019Z	debug	[email protected]/exporter.go:273	Stable component.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp/lumigo"}
2024-03-07T16:41:35.020Z	debug	[email protected]/receiver.go:294	Stable component.	{"kind": "receiver", "name": "otlp", "data_type": "traces"}
2024-03-07T16:41:35.020Z	debug	[email protected]/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.020Z	debug	[email protected]/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.020Z	info	[email protected]/service.go:143	Starting lumigo-collector...	{"Version": "0.0.1", "NumCPU": 2}
2024-03-07T16:41:35.020Z	info	extensions/extensions.go:34	Starting extensions...
2024-03-07T16:41:35.021Z	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.021Z	info	extensions/extensions.go:45	Extension started.	{"kind": "extension", "name": "lumigoauth/server"}
2024-03-07T16:41:35.021Z	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.021Z	info	[email protected]/healthcheckextension.go:35	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2024-03-07T16:41:35.021Z	warn	[email protected]/warning.go:40	Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks	{"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
2024-03-07T16:41:35.021Z	info	extensions/extensions.go:45	Extension started.	{"kind": "extension", "name": "health_check"}
2024-03-07T16:41:35.034Z	info	[email protected]/otlp.go:101	Starting HTTP server	{"kind": "receiver", "name": "otlp", "data_type": "traces", "endpoint": "0.0.0.0:4318"}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "/v1, Resource=pods", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=daemonsets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=deployments", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=replicasets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=statefulsets", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "batch/v1, Resource=cronjobs", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "batch/v1, Resource=jobs", "mode": "watch", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "/v1, Resource=pods", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=daemonsets", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:93	Started collecting	{"kind": "receiver", "name": "k8sobjects/objects_ns_node1", "data_type": "logs", "gvr": "apps/v1, Resource=deployments", "mode": "pull", "namespaces": ["node1"]}
2024-03-07T16:41:35.034Z	info	[email protected]/receiver.go:73	Object Receiver started	{"kind": "receiver", "name": "k8sobjects/events_ns_node1", "data_type": "logs"}
2024-03-07T16:41:35.035Z	info	healthcheck/handler.go:132	Health Check state change	{"kind": "extension", "name": "health_check", "status": "ready"}
2024-03-07T16:41:35.035Z	info	[email protected]/service.go:169	Everything is ready. Begin running and processing data.

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2024-03-07T19:53:54Z

Pinging code owners:

receiver/k8sobjects: @dmitryax @hvaghani221 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

TylerHelmuth · 2024-03-12T18:33:50Z

I suspect the setup/teardown of the watches isnt quite right.

kenfinnigan · 2024-03-12T20:29:17Z

That's what I was thinking as well, but not sure how to debug it further to narrow down the issue.

Any suggestions @TylerHelmuth? (Bearing in mind I'm reasonably new to Golang)

TylerHelmuth · 2024-03-12T20:51:47Z

Hopefully it is somewhere in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/k8sobjectsreceiver/receiver.go

github-actions · 2024-05-13T03:29:56Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

receiver/k8sobjects: @dmitryax @hvaghani221 @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-07-12T05:19:46Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

kenfinnigan added bug Something isn't working needs triage New item requiring triage labels Mar 7, 2024

github-actions bot added the receiver/k8sobjects label Mar 7, 2024

github-actions bot mentioned this issue Mar 12, 2024

Weekly Report: 2024-03-05 - 2024-03-12 #31693

Closed

TylerHelmuth added help wanted Extra attention is needed priority:p2 Medium and removed needs triage New item requiring triage labels Mar 12, 2024

TylerHelmuth mentioned this issue Mar 23, 2024

[receiver/k8sobjects] Fix memory leak caused by pull mode's interval ticker #31919

Merged

github-actions bot added the Stale label May 13, 2024

github-actions bot added the closed as inactive label Jul 12, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Client-side Throttling API calls for namespace when k8sobjectsreceiver no longer monitoring that namespace #31644

Client-side Throttling API calls for namespace when k8sobjectsreceiver no longer monitoring that namespace #31644

kenfinnigan commented Mar 7, 2024

github-actions bot commented Mar 7, 2024

TylerHelmuth commented Mar 12, 2024

kenfinnigan commented Mar 12, 2024

TylerHelmuth commented Mar 12, 2024

github-actions bot commented May 13, 2024

github-actions bot commented Jul 12, 2024