Experiment flag for welcomePage.experimental.extensionContributions

[digitarald]

To do a staged rollout of extension walkthroughs, we need an experiment treatment flag for workbench.welcomePage.experimental.extensionContributions.

Experiments should focus on one walkthrough for one extension, so we can track the results for each separately. This means we need have separate flags that allow walkthroughs for extensions.

Maybe we can use a similar pattern that flags use for replacing getting started strings, asking for each extension that has a walkthrough: walkthroughExtension.ms-vscode.cpptools: true.

Partner extensions could either check the same treatment flags on their side to prevent their existing welcome page logic. This makes #122592 obsolete.

[JacksonKearl]

The idea is to change the default value of that setting based on an experiment?

[digitarald]

@JacksonKearl just clarified the initial proposal with an allow list. The initial idea was changing the default; but I think we actually need to control this on a per-extension level. Wdyt?

[JacksonKearl]

Maybe have it keyed by extension ID and walkthrough ID in order to support @kieferrm's request for a way to run walkthroughs as experiments... so regardless of workbench.welcomePage.experimental.extensionContributions, if the treatment variable walkthroughExtension/ms-vscode.cpptools/walkthroughContentV1 is set, that walkthrough gets shown?

[JacksonKearl]

Should be covered by the when clause overriding.

[JacksonKearl]

With 302c638, we can override gettingStarted.overrideCategory.${extension id}.${walkthrough id}.when, so extensions can set this to false initially, then override it for specific cohorts. To test locally, the usual:

"experiments.override.gettingStarted.overrideCategory.jakearl.md-to-html.exampleProject.when": "true",

with an optional:

"experiments.overrideDelay": 5000,

to verify how things work in cases where the exp service takes a while to load. (note usually the result will be cached cross sessions, however the local overriding machinery does not account for this.

Edit: previously the separator between extension id and walkthrough id was #, it is now ..

[Colengms]

extensions can set this to false initially, then override it for specific cohorts.

Hi @JacksonKearl . Does this refer to something an extension would need to do in code (i.e. using the experimentation platform API, so after activation), or would this be controlled through experimentation settings managed by VS Code?

[JacksonKearl]

As an extension you would contribute the walkthrough with "when": "false", then as VS Code core we'd run an experiment to override the when clause to be whatever you want. You can test out how this would work locally using the override settings here #122966 (comment)

[connor4312]

I think I verified this correctly. Can't test end-to-end with EXP integration, though