Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid rooting documents for languages that support syntax trees #67960

Merged
merged 4 commits into from
Apr 26, 2023

Conversation

sharwell
Copy link
Member

Addresses 17GB memory overhead observed in a local heap dump where 50+ documents were open.

@sharwell sharwell requested a review from a team as a code owner April 25, 2023 18:44
@dotnet-issue-labeler dotnet-issue-labeler bot added Area-IDE untriaged Issues and PRs which have not yet been triaged by a lead labels Apr 25, 2023
@CyrusNajmabadi
Copy link
Member

I need to look at this in front of a computer. However, my recollection is that this type exists only for c#/vb. But I def could be wrong on that. If this is only for those two, we can likely simplify this.

@sharwell
Copy link
Member Author

sharwell commented Apr 25, 2023

... If this is only for those two, we can likely simplify this.

I definitely agree, but it seems like that could be a follow-up item.

@CyrusNajmabadi
Copy link
Member

i'm ok with the general approach here (storing the root instead of the doc). But i feel you did the PR a disservice both with the nullability change and the static change. it just makes it a slog of passing data around :)

@@ -272,15 +282,18 @@ private async ValueTask ProcessChangesAsync(ImmutableSegmentedList<ITextSnapshot
Contract.ThrowIfTrue(snapshots.IsDefault || snapshots.IsEmpty);
var currentSnapshot = GetLatest(snapshots);

var classificationService = TryGetClassificationService(currentSnapshot);
if (classificationService == null)
if (TryGetClassificationService(currentSnapshot) is not (var solutionServices, var classificationService))
return;

var currentDocument = currentSnapshot.GetOpenDocumentInCurrentContextWithChanges();
if (currentDocument == null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aside, this whole pattern is wonky. we get the document or not... so we dont need the helper for getting the classificationservice/solution-service. We could always just get that from currentDocument. Would simplify a lot it seems.

@@ -69,7 +71,7 @@ internal partial class TagComputer
// get called.

private readonly object _gate = new();
private (ITextSnapshot? lastSnapshot, Document? lastDocument, SyntaxNode? lastRoot) _lastProcessedData;
private (ITextSnapshot lastSnapshot, SumType<SyntaxNode, Document> lastDocumentOrRoot)? _lastProcessedData;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would intentionally doc that we want to drop teh Doc in the case where we have the root to ensure we're not holding onto old solutions. Then maybe a note that this could potentially be bad for other languages, but we can revisit as needed.

{
// We don't have a syntax tree yet. Just do a lexical classification of the document.
AddLexicalClassifications(classificationService, span, classifiedSpans);
return;
}

// If we have a document, we must have a snapshot as well.
Contract.ThrowIfNull(lastProcessedSnapshot);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. i see why you made the invariant change. i'm ok with it :)

else
classificationService.AddSyntacticClassifications(document.Project.Solution.Services, root, span.Span.ToTextSpan(), tempList, cancellationToken);
classificationService.AddSyntacticClassifications(solutionServices, root, span.Span.ToTextSpan(), tempList, cancellationToken);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, either consider flipping the if/else, or just do TryGetSecond in the above. but i don't feel strongly, something just minorly confused me here.

Copy link
Member

@CyrusNajmabadi CyrusNajmabadi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. very clear. thanks!

@CyrusNajmabadi
Copy link
Member

all the feedback is nits, except the part about documenting things. i would ask that that be done :)

@sharwell
Copy link
Member Author

@dotnet/roslyn-compiler I'm getting hit hard by this problem locally, but I'm not able to get the fix merged because bootstrap build is failing with this assertion. Can I get some help resolving it?

ID=VBCSCompiler TID=16: Debug.Assert failed with message: Fail: 
Stack Trace
   at Microsoft.CodeAnalysis.CommandLine.ExitingTraceListener.Exit(String originalMessage)
   at Microsoft.CodeAnalysis.CommandLine.ExitingTraceListener.WriteLine(String message)
   at System.Diagnostics.TraceInternal.Fail(String message)
   at System.Diagnostics.Debug.Assert(Boolean condition)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitConditionalOperatorCore(BoundExpression node, Boolean isRef, BoundExpression condition, BoundExpression originalConsequence, BoundExpression originalAlternative)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitUnconvertedConditionalOperator(BoundUnconvertedConditionalOperator node)
   at Microsoft.CodeAnalysis.CSharp.BoundUnconvertedConditionalOperator.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.VisitExpressionWithStackGuard(Int32& recursionDepth, BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitRvalue(BoundExpression node, Boolean isKnownToBeAnLvalue)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitRvalueWithState(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.<>c.<VisitTupleExpression>b__261_0(BoundExpression a, NullableWalker w)
   at System.Collections.Immutable.ImmutableArray.CreateRange[TSource,TArg,TResult](ImmutableArray`1 items, Func`3 selector, TArg arg)
   at Microsoft.CodeAnalysis.ImmutableArrayExtensions.SelectAsArray[TItem,TArg,TResult](ImmutableArray`1 items, Func`3 map, TArg arg)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitTupleExpression(BoundTupleExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitTupleLiteral(BoundTupleLiteral node)
   at Microsoft.CodeAnalysis.CSharp.BoundTupleLiteral.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.VisitExpressionWithStackGuard(Int32& recursionDepth, BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitWithoutDiagnostics(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitConvertedTupleLiteral(BoundConvertedTupleLiteral node)
   at Microsoft.CodeAnalysis.CSharp.BoundConvertedTupleLiteral.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.VisitExpressionWithStackGuard(Int32& recursionDepth, BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitRvalue(BoundExpression node, Boolean isKnownToBeAnLvalue)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitRvalueWithState(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitOptionalImplicitConversion(BoundExpression expr, TypeWithAnnotations targetTypeOpt, Boolean useLegacyWarnings, Boolean trackMembers, AssignmentKind assignmentKind, Boolean delayCompletionForTargetType)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitOptionalImplicitConversion(BoundExpression expr, TypeWithAnnotations targetTypeOpt, Boolean useLegacyWarnings, Boolean trackMembers, AssignmentKind assignmentKind)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitAssignmentOperator(BoundAssignmentOperator node)
   at Microsoft.CodeAnalysis.CSharp.BoundAssignmentOperator.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitExpressionWithoutStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.VisitExpressionWithStackGuard(BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.VisitExpressionWithStackGuard(Int32& recursionDepth, BoundExpression node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitRvalue(BoundExpression node, Boolean isKnownToBeAnLvalue)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitExpressionStatement(BoundExpressionStatement node)
   at Microsoft.CodeAnalysis.CSharp.BoundExpressionStatement.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitStatementsWithLocalFunctions(BoundBlock block)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitBlock(BoundBlock node)
   at Microsoft.CodeAnalysis.CSharp.BoundBlock.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitLockStatement(BoundLockStatement node)
   at Microsoft.CodeAnalysis.CSharp.BoundLockStatement.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitStatement(BoundStatement statement)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitStatementsWithLocalFunctions(BoundBlock block)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.VisitBlock(BoundBlock node)
   at Microsoft.CodeAnalysis.CSharp.BoundBlock.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitMethodBodies(BoundBlock blockBody, BoundBlock expressionBody)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitNonConstructorMethodBody(BoundNonConstructorMethodBody node)
   at Microsoft.CodeAnalysis.CSharp.BoundNonConstructorMethodBody.Accept(BoundTreeVisitor visitor)
   at Microsoft.CodeAnalysis.CSharp.BoundTreeVisitor.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitWithStackGuard(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.VisitAlways(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node, Boolean expressionIsRead)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Visit(BoundNode node)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Scan(Boolean& badRegion)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Scan(Boolean& badRegion)
   at Microsoft.CodeAnalysis.CSharp.AbstractFlowPass`2.Analyze(Boolean& badRegion, Optional`1 initialState)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Analyze(NullableWalker walker, Symbol symbol, DiagnosticBag diagnostics, Optional`1 initialState, Builder snapshotBuilderOpt, Boolean requiresAnalysis)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.Analyze(CSharpCompilation compilation, Symbol symbol, BoundNode node, Binder binder, Conversions conversions, DiagnosticBag diagnostics, Boolean useConstructorExitWarnings, Boolean useDelegateInvokeParameterTypes, Boolean useDelegateInvokeReturnType, MethodSymbol delegateInvokeMethodOpt, VariableState initialState, MethodSymbol baseOrThisInitializer, Builder analyzedNullabilityMapOpt, Builder snapshotBuilderOpt, ArrayBuilder`1 returnTypesOpt, Boolean getFinalNullableState, VariableState& finalNullableState, Boolean requiresAnalysis)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.AnalyzeWithSemanticInfo(CSharpCompilation compilation, Symbol symbol, BoundNode node, Binder binder, VariableState initialState, DiagnosticBag diagnostics, Boolean createSnapshots, Boolean requiresAnalysis)
   at Microsoft.CodeAnalysis.CSharp.NullableWalker.AnalyzeAndRewrite(CSharpCompilation compilation, Symbol symbol, BoundNode node, Binder binder, VariableState initialState, DiagnosticBag diagnostics, Boolean createSnapshots, SnapshotManager& snapshotManager, ImmutableDictionary`2& remappedSymbols)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.BindMethodBody(MethodSymbol method, TypeCompilationState compilationState, BindingDiagnosticBag diagnostics, Boolean includeInitializersInBody, BoundNode initializersBody, Boolean reportNullableDiagnostics, ImportChain& importChain, Boolean& originalBodyNested, Boolean& prependedDefaultValueTypeConstructorInitializer, InitialState& forSemanticModel)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileMethod(MethodSymbol methodSymbol, Int32 methodOrdinal, ProcessedFieldInitializers& processedInitializers, SynthesizedSubmissionFields previousSubmissionFields, TypeCompilationState compilationState)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.CompileNamedType(NamedTypeSymbol containingType)
   at Microsoft.CodeAnalysis.CSharp.MethodCompiler.<>c__DisplayClass25_0.<CompileNamedTypeAsync>b__0()
   at Roslyn.Utilities.UICultureUtilities.<>c__DisplayClass5_0.<WithCurrentUICulture>b__0()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecution)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

@@ -291,7 +301,7 @@ private async ValueTask ProcessChangesAsync(ImmutableSegmentedList<ITextSnapshot

lock (_gate)
{
_lastProcessedData = (currentSnapshot, currentDocument, currentRoot);
_lastProcessedData = (currentSnapshot, currentRoot is not null ? currentRoot : currentDocument);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting hit hard by this problem locally, but I'm not able to get the fix merged because bootstrap build is failing with this assertion. Can I get some help resolving it?

Looking. The assertion is hit when nullable walker visits this conditional operator. If you want a workaround, I guess you could try changing the code, e.g.:

Suggested change
_lastProcessedData = (currentSnapshot, currentRoot is not null ? currentRoot : currentDocument);
_lastProcessedData = (currentSnapshot, currentRoot ?? currentDocument);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this expression is special. I'll see if adding explicit casts helps.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed #67975 and added the explicit casts, but then I realized I can use a modified version with ?? and switched to that.

@sharwell sharwell merged commit ccd822d into dotnet:main Apr 26, 2023
@ghost ghost added this to the Next milestone Apr 26, 2023
@sharwell sharwell deleted the syntactic branch April 26, 2023 16:12
@Cosifne Cosifne removed this from the Next milestone May 31, 2023
@Cosifne Cosifne added this to the 17.7 P2 milestone May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-IDE untriaged Issues and PRs which have not yet been triaged by a lead
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants