Set env vars for service instead of machine

[pjanotti]

Description:
On Windows move the environment variables to the service scope. This servers two purposes:

1. Avoid issues with Splunk instrumentations that use some of the same environment variables as the collector and could end up sending data straight to the backend instead of the sending locally to the collector.
2. Reduce the scope of visibility of the access token, which should only be visible to the collector

Thanks @pellared for noticing this issue.

Review notes

• "Monkey see, monkey does": in retrospect, after learning a bit about some of the tech used here, I wanted to change some other things, e.g.: use an array instead of maps that a later converted to arrays in Chef, but, this is already taking too long and such improvements can be addressed separately.

• Sorting variables when writing to the registry: this facilitates human inspection and tests.

• Choco: if there was a previous version with environment variables on the machine scope, the environment variables are removed from the machine level. Not doing the same for other deployments because they don't seem to support upgrading deployments, even the install script requires removing the previously installed collector.

• Ansible:

  • Removed some redundant restarts of the service on deployments/ansible/roles/collector/tasks/otel_win_install.yml, the restart is done only after the environment variables for the service are updated.
  • Tests are failing on CI but passing on mac dev box.

• Puppet:

  • Unlike the other mass deployments Puppet doesn't support deploying SFx.NET to Windows+IIS.
  • deployments/puppet/manifests/collector_win_registry.pp: respective tests are at internal\buildscripts\packaging\tests\deployments\puppet\puppet_test.py (internal/buildscripts/packaging/tests/helpers/win_utils.py was modified to support new environment variables location).

Link to Splunk idea:
N/A

Testing:
Local tests and CI.

Documentation:
Updating CHANGELOG.md

cc @RassK @theletterf

[pjanotti]

Returning to draft until I can look at the test failure.

[jeffreyc-splunk]

The msi itself also needs to be updated: https://github.com/signalfx/splunk-otel-collector/blob/main/internal/buildscripts/packaging/msi/splunk-otel-collector.wxs#L42

[jeffreyc-splunk]

And we'll have to update our ansible, chef, and puppet modules since they currently only configure HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment.

[rmfitzpatrick]

Is this removal related to the env var changes? Wonder if there should be a new var to dictate if expected.

[pjanotti]

This was failing consistently for me because the status had changed to "started" - I confess that I don't understand how it was working before.

[rmfitzpatrick]

are these removals related and necessary?

[pjanotti]

Yes, the restart should only happen after the environment variables are set for the service.

[atoulme]

Approved, feel free to merge if enough CI coverage.