Expose wider collector configuration #31
Conversation
| ingress.grpc.port: | ||
| description: "Port the collector is listening on to receive OTLP over gRPC" | ||
| default: 9100 | ||
| config: |
There was a problem hiding this comment.
Hi @acrmp,
Generally I'm all in about this change, but I wonder if we should do it this way, that we practically embed the Otel Collector config in the spec or we provide provide a parameter which can define the path to the Otel collector config yaml file.
If we use some manager to manager the fleet of Otel collectors which we have based on OpAMP, they can push configuration to the clients (Otel collectors), it might be more clear, readable, practical for further processing (merge of local and remote config) if the configuration is in a separate file.
There was a problem hiding this comment.
@chombium Can you expand on how you would see this working? Were you thinking of doing config merging outside of this release?
We're still exploring how we might want to distribute configuration. One option is that we could do something similar to what we do for the prom scraper and load configs that match a certain file name glob. A downside is that otel config changes would currently require BOSH to roll all the VMs.
We've looked into OpAMP a little. There still seem to be a few rough edges at the moment. For example see this recent issue we opened:
open-telemetry/opentelemetry-collector-contrib#33799
There was a problem hiding this comment.
Hi @acrmp, @mkocher sorry for the late reply.... :-/
I was thinking of having the Otel collector config in a separate file for better separation, but it doesn't hurt if it's inside the job spec. At the end if we use an OpAMP server for managing the configuration, the sever would ping the Otel Collector that there is new config available and its up to the collector to decide when it will download and apply the new config. I'm not against your solution at all ;)
|
We think it's not too onerous to maintain backwards compatibility here and continue to support the older properties for a period. We'll likely push some further changes for this tomorrow. |
- Replaces the existing `metric_exporters` and `trace_exporters` properties with the new `config` property. - Operators can now provide pipeline and processor configuration. - The default otlp receiver will be set on all pipelines. - The config can be validated with the following command: `/var/vcap/packages/otel-collector/otel-collector validate --config some-config.yml` Signed-off-by: Ausaf Ahmed <[email protected]>
- Continue to support the earlier properties to provide a migration path. - Operators should prefer the new `config` property going forwards. Signed-off-by: Matthew Kocher <[email protected]>
mkocher
force-pushed
the
complete-collector-config
branch
from
a3833fe
to
e68a410
Compare
will need toshould update the properties to match the format expected and define pipelines.metric_exportersandtrace_exportersproperties with the newconfigproperty.The example config provided in the PR refers to the batch processor that is not currently included in the CF distribution of the collector. We'll make a separate PR to include that processor.