Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove ballast extension entirely from components. #5429

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

atoulme
Copy link
Contributor

@atoulme atoulme commented Sep 30, 2024

Description:
Remove the deprecated memory ballast extension from bundled components, which was replaced by GOMEMLIMIT

@atoulme atoulme requested review from a team as code owners September 30, 2024 06:22
CHANGELOG.md Outdated Show resolved Hide resolved
Copy link
Contributor

@crobert-1 crobert-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more reference that needs removed here:

@dmitryax
Copy link
Contributor

dmitryax commented Sep 30, 2024

We also have some env var magic around the ballast extension that we need to clean up

@hughesjj
Copy link
Contributor

hughesjj commented Oct 2, 2024

Looks good to me. While there's still some references to the ballast, it's mostly in testdata or readmes, or for backwards compatibility

cmd/migratecheckpoint/main_test.go:             Fingerprint: &Fingerprint{FirstBytes: []byte("2021-10-01T02:34:00.905393894Z stderr F 2021/10/01 02:34:00 main.go:189: Set config to /conf/relay.yaml\n2021-10-01T02:34:00.905426919Z stderr F 2021/10/01 02:34:00 main.go:272: Set ballast to 168 MiB\n2021-10-01T02:34:00.905433052Z stderr F 2021/10/01 02:34:00 main.go:286: Set memory limit to 460 MiB\n2021-10-01T02:34:00.931383015Z stderr F 2021-10-01T02:34:00.931Z\tinfo\tservice/collector.go:303\tStarting otelcol...\t{\"Version\": \"v0.33.1\", \"NumCPU\": 2}\n2021-10-01T02:34:00.931397472Z stderr F 2021-10-01T02:34:00.931Z\tinfo\tservice/collector.go:242\tLoading configuration...\n2021-10-01T02:34:01.092707836Z stderr F 2021-10-01T02:34:01.092Z\tinfo\tservice/collector.go:258\tApplying configuration...\n2021-10-01T02:34:01.10836105Z stderr F 2021-10-01T02:34:01.108Z\tinfo\tbuilder/exporters_builder.go:264\tExporter was built.\t{\"kind\": \"exporter\", \"name\": \"splunk_hec/platformMetrics\"}\n2021-10-01T02:34:01.108516901Z stderr F 2021-10-01T02:34:01.108Z\tinfo\tbuilder/exporters_builder.go:264\tExporter was built.\t{\"k")},
cmd/migratecheckpoint/main_test.go:             Fingerprint: &Fingerprint{FirstBytes: []byte("2021-10-01T02:34:00.905393894Z stderr F 2021/10/01 02:34:00 main.go:189: Set config to /conf/relay.yaml\r\n2021-10-01T02:34:00.905426919Z stderr F 2021/10/01 02:34:00 main.go:272: Set ballast to 168 MiB\r\n2021-10-01T02:34:00.905433052Z stderr F 2021/10/01 02:34:00 main.go:286: Set memory limit to 460 MiB\r\n2021-10-01T02:34:00.931383015Z stderr F 2021-10-01T02:34:00.931Z\tinfo\tservice/collector.go:303\tStarting otelcol...\t{\"Version\": \"v0.33.1\", \"NumCPU\": 2}\r\n2021-10-01T02:34:00.931397472Z stderr F 2021-10-01T02:34:00.931Z\tinfo\tservice/collector.go:242\tLoading configuration...\r\n2021-10-01T02:34:01.092707836Z stderr F 2021-10-01T02:34:01.092Z\tinfo\tservice/collector.go:258\tApplying configuration...\r\n2021-10-01T02:34:01.10836105Z stderr F 2021-10-01T02:34:01.108Z\tinfo\tbuilder/exporters_builder.go:264\tExporter was built.\t{\"kind\": \"exporter\", \"name\": \"splunk_hec/platformMetrics\"}\r\n2021-10-01T02:34:01.108516901Z stderr F 2021-10-01T02:34:01.108Z\tinfo\tbuilder/exporters_builder.go:264\tExporter was bui")},
cmd/migratecheckpoint/testdata/otelcollector.log:2021-10-01T02:34:00.905426919Z stderr F 2021/10/01 02:34:00 main.go:272: Set ballast to 168 MiB
CHANGELOG.md:- (Splunk) Remove deprecated memory ballast extension ([#5429](https://github.com/open-telemetry/opentelemetry-collector/pull/5429))
CHANGELOG.md:  - Deprecation and removal of `ballast` extension
CHANGELOG.md:- (Core) `processor/memorylimiter`: The memory limiter processor will no longer account for ballast size. ([#10696](https://github.com/open-telemetry/opentelemetry-collector/pull/10696))
CHANGELOG.md:  If you are already using GOMEMLIMIT instead of the ballast extension this does not affect you.
CHANGELOG.md:- (Core) `extension/memorylimiter`: The memory limiter extension will no longer account for ballast size. ([#10696](https://github.com/open-telemetry/opentelemetry-collector/pull/10696))
CHANGELOG.md:  If you are already using GOMEMLIMIT instead of the ballast extension this does not affect you.
CHANGELOG.md:- (Core) `service`: The service will no longer be able to get a ballast size from the deprecated ballast extension. ([#10696](https://github.com/open-telemetry/opentelemetry-collector/pull/10696))
CHANGELOG.md:  If you are already using GOMEMLIMIT instead of the ballast extension this does not affect you.
CHANGELOG.md:  After removal of memory_ballast extension in v0.97.0, the Go garbage collection is running more aggressively, which
CHANGELOG.md:- (Splunk) `memory_ballast` has been removed. If GOMEMLIMIT env var is not set, then 90% of the total available memory limit is set by default. ([#4404](https://github.com/signalfx/splunk-otel-collector/pull/4404))
CHANGELOG.md:- (Core) `extension/ballast`: Deprecate `memory_ballast` extension. ([#8343](https://github.com/open-telemetry/opentelemetry-collector/issues/8343))
CHANGELOG.md:- [Fixed a warning for Salt deployments to set the ballast memory size under an extension instead of memory_limiter processor](https://github.com/signalfx/splunk-otel-collector/pull/2379)
CHANGELOG.md:- [Remove handling of unsupported --mem-ballast-size-mib command line argument](https://github.com/signalfx/splunk-otel-collector/pull/2339)
CHANGELOG.md:- Fix Linux installer to set `SPLUNK_MEMORY_TOTAL_MIB` in the environment file if `--ballast` option is specified (#807)
CHANGELOG.md:- Handle backwards compatibility of internal ballast removal (#759)
CHANGELOG.md:- Enable `memory_ballast` extension in Fargate config (#675)
CHANGELOG.md:- Update to use the `memory_ballast` extension instead of the `--mem-ballast-size-mib` flag (#567)
internal/buildscripts/packaging/installer/install.ps1:    (OPTIONAL) Total memory in MIB to allocate to the collector; automatically calculates the ballast size (default: "512").
internal/confmapprovider/discovery/README.md:2023/02/24 19:54:23 settings.go:384: Set ballast to 168 MiB
internal/configconverter/testdata/with_memory_ballast_only.yaml:  memory_ballast:
internal/configconverter/testdata/with_memory_ballast_only.yaml:  extensions: [memory_ballast]
internal/configconverter/testdata/ballast_mem_limiter_custom.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/with_memory_ballast.yaml:  memory_ballast:
internal/configconverter/testdata/with_memory_ballast.yaml:  extensions: [health_check, http_forwarder, zpages, memory_ballast, smartagent]
internal/configconverter/testdata/disable_excessive_internal_metrics/all_metric_relabel_configs_are_present.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/metric_relabel_configs_with_other_actions.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/old_metric_relabel_configs_present_input.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/different_job.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/no_metric_relabel_configs_set.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/no_prom_receiver.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/no_scrape_configs.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_excessive_internal_metrics/old_metric_relabel_configs_present_output.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/partially_excluded_in_signalfx_exporter_input.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/no_kubeletstats_receiver.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/partially_included_in_signalfx_exporter_output.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/do_not_change_enabled_metrics_output.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/do_not_change_enabled_metrics_input.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/disable_all_metrics_output.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/utilization_metrics_disabled.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/all_metrics_included_in_signalfx_exporter.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/partially_included_in_signalfx_exporter_input.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/partially_excluded_in_signalfx_exporter_output.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/no_signalfx_exporter.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/disable_kubelet_utilization_metrics/disable_all_metrics_input.yaml:    ballast_size_mib: 64
internal/configconverter/testdata/ballast_mem_limiter.yaml:    ballast_size_mib: 64
deployments/ansible/CHANGELOG.md:- `splunk_ballast_size_mib` is deprecated and removed. For Splunk Otel Collector version `0.97.0` or greater, `GOMEMLIMIT` env var is introduced. The default is set to 90% of the `SPLUNK_TOTAL_MEM_MIB`. For more information regarding the usage, please follow the instructions ([here](https://github.com/signalfx/splunk-otel-collector?tab=readme-ov-file#from-0961-to-0970)).
deployments/salt/README.md:- `gomemlimit`: `splunk_ballast_size_mib` is deprecated and removed. For Splunk Otel Collector version `0.97.0` or greater, `GOMEMLIMIT` env var is introduced. The default is set to 90% of the `SPLUNK_TOTAL_MEM_MIB`. For more information regarding the usage, please follow the instructions ([here](https://github.com/signalfx/splunk-otel-collector?tab=readme-ov-file#from-0961-to-0970)).  (**default:** 90% of `splunk_memory_total_mib`, otherwise)
deployments/chef/CHANGELOG.md:- `splunk_ballast_size_mib` is deprecated and removed. For Splunk Otel Collector version `0.97.0` or greater, `GOMEMLIMIT` env var is introduced. The default is set to 90% of the `SPLUNK_TOTAL_MEM_MIB`. For more information regarding the usage, please follow the instructions ([here](https://github.com/signalfx/splunk-otel-collector?tab=readme-ov-file#from-0961-to-0970)).
deployments/puppet/CHANGELOG.md:- The `splunk_ballast_size_mib` option is deprecated and no longer effective. It is only applicable for Splunk OpenTelemetry Collector version < `0.97.0`.
deployments/puppet/README.md:|**DEPRECATED** `splunk_ballast_size_mib`            | Set the ballast size for the collector explicitly instead of the value calculated from the `$splunk_memory_total_mib` parameter.  This should be set to 1/3 to 1/2 of configured memory.  The `SPLUNK_BALLAST_SIZE_MIB` environment variable will be set with this value for the collector service. Only applicable if `collector_version` is < 0.97.0.                                                                                                                                                                                                                                                                                     | None                                                                                                                          |
README.md:- `memory_ballast` is no longer effective. The garbage collection is now controlled by the soft memory limit set to 90%
README.md:  If you haven't customized the `memory_ballast`, just remove it from the configuration.
README.md:  If you have customized it via `SPLUNK_BALLAST_SIZE_MIB` (or `extensions::memory_ballast::size_mib` config), you should
README.md:  remove the `memory_ballast` extension and use the [`GOMEMLIMIT`](https://pkg.go.dev/runtime) environment variable to set a custom soft memory limit:
README.md:- `ballast_size_mib` parameter moved from `memory_limiter` processor to `memory_ballast` extension
README.md:  Remove `ballast_size_mib` parameter from `memory_limiter` and make sure that it's added to
README.md:  `memory_ballast` extension as `size_mib` parameter instead:
README.md:    memory_ballast:
README.md:      size_mib: ${SPLUNK_BALLAST_SIZE_MIB}

@hughesjj
Copy link
Contributor

hughesjj commented Oct 2, 2024

We probably need this test to be passing as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants