Support multi-option contexts

[pq]

The Dart Analyzer treats the presence of an analysis_options.yaml file as a signal to create an analysis context, a discrete, hermetic environment for analysis. In many cases this is unneeded and the memory overhead of elements that could be shared but are duplicated is significant, contributing to poor IDE performance.

Plan: Decouple options from contexts, allowing one context to contain many analysis options files without any element model and file state duplication. To the user, there will be no change in semantics with reduced memory pressure and improved performance.

---

Follow-ups:

• remove ContextLocatorImpl.singleOptionContexts flag #56352

Blocking:

• test_fileSystem_deleteFile_packageConfigJsonFile fails #55580
• [multi-option contexts] spurious MISSING_DEPENDENCY reported on a flutter customer app #55413
• evaluate performance for multi-option contexts #55300
• Fix sort order of folders in options map to ensure most specific lookups #55252
• ensure exclude: path filtering works in multi-option contexts #54858
• verify that command line flags are updating multi-options #54849
• DAS diagnostics page is displaying *all* options files for each context #54816
• update context_locator to not create fresh contexts for nested options #54714
• add domain analysis tests that verify options are correct after deletions and aditions #54814
• add options map tests where an optionsFile is provided #54791
• update internal g3 linter rule tests to new options APIs #54770
• migrate away from (singleton) contextRoot.optionsFile accesses #54312
• analyzer: move strictCasts bool field out of TypeSystemImpl #53873
  • deprecate uses (and update callers)
  • remove API
• Add strictCasts API access to lints #54376
• Add an sdkVersionConstraint getter to WorkspacePackage #54043
• Redesign the DAS diagnostics page to display multiple options per context #54074
• Cache analysis options objects in FileStates #54124
• evaluate where we call workspace.findPackageFor(file) and seek improvements #54310

Related:

• Can AnalysisContext.getAnalysisOptionsForFile be removed in favor of access from ResolvedUnitResult? #54667
• Reconsider how we configure lint rules in analysis options objects #54045
• Add tests for target-specific analysis options #54000

[pq]

Status 02.02.2024

Doing some initial testing on a local build, I've confirmed that flipping the flag "works" and DAS appears to be working as intended.

For my local analyzer SDK development workspace, I'm seeing the expected reduction in contexts.

Multi-Option Contexts (future)

Single-Option Contexts (today)

It's interesting to note the less than dramatic difference in memory usage.

Likely this can be at least partly explained by the nature of the saved contexts (in all 3 cases they have very few libraries) but the fact that this is essentially a cold-start capture makes these numbers imprecise anyway. Much more interesting will be to measure memory footprint over time (and with more substantial nested contexts).

Testing

For the curious, to enable mutli-option contexts in an SDK, set singleOptionContexts to false

and do a build.

[rrousselGit]

I think the .analysisOption may have been deprecated a bit early.

If .getAnalysisOptionsForFile(file) is still experimental, we don't have a stable replacement for .analysisOptions

[pq]

Sorry for the confusion @rrousselGit. This is taking a bit longer than hoped (lots of moving parts).

Out of curiosity, how are you using analysis options objects? In plugins?

[rrousselGit]

In analyzer_plugins, yes. In custom_lint specifically, a wrapper around analyzer_plugin to make it more dev friendly and improve support for monorepo in terms of memory usage.

[pq]

Thanks! If it isn't too much trouble, could you tell me how you're using analysis options in custom_lint?

(fyi @srawlins.)

[srawlins]

Any updates on this effort, @pq , as a P1 feature?

[pq]

Thanks for the ping. This is currently blocked behind #54858. Will be revisiting that in the next week.

[pq]

Investigating current blocked (#55252) now.

[pq]

Fix for #55252 cleared up the pkg bots 🎉. Waiting on flutter try results. 🤞

[knopp]

I switched the singleOptionContexts flag to false, rebuilt the SDK, but I get same amount of context for our mono-repo project. Is this supposed to make any difference in monorepo projects?

[pq]

Is this supposed to make any difference in monorepo projects?

@keertip

[pq]

The singleOptionContexts flag will only make contexts due to nested analysis options files unnecessary so it's sort of orthogonal to monorepos.

For example, with singleOptionContexts set to true the following will produce two contexts

my_package/
  analysis_options.yaml
  lib/
  test/
    analysis_options.yaml

With it set to false, you should see only one.

If you don't have any nested options files, you won't see a difference.

[bwilkerson]

On the other hand, it's a necessary part of providing good monorepo support. Without this, every package in a monorepo would likely still have it's own analysis context, which would largely defeat the gains we expect to see from the monorepo support.

[knopp]

Thank you for the confirmation. I'll patiently wait for pub workspace support.

[bwilkerson]

Any updates?

[pq]

We're waiting for https://dart-review.googlesource.com/c/sdk/+/362442 to fully make it through a flutter roll before we can try to re-enable this. (See: #55413)

[pq]

22da6ed flips the bit but we've reverted related changes twice so I think we want to let it bake for a few more days at least before declaring victory.

[pq]

Looking at the latest engine roll (flutter/flutter#147373) which brings Dart in @b5f51d8, it looks like this change has safely made it's way into Flutter. Here's hoping this landing sticks. 🤞

[pq]

With #55580 fixed, all known issues have been addressed.

[srawlins]

I think we can close this? Can we mark AnalysisContext.getAnalysisOptionsForFile non-experimental?

[pq]

I'm in favor but have a few reservations. I've added it for discussion later today.