[WIP][Macro] Emit error if AccessorMacro is applied to node that is not a variable

[RayZhao1998]

This PR resolves #2207 .

[ahoppen]

Thanks for taking this on @RayZhao1998 and welcome to swift-syntax’s development! 🙏🏽

Instead of repeating the logic of which attributes we need to remove, I think we should collect the AttributeSyntax nodes of the attributes that we did indeed expand in MacroApplication and then remove those nodes for the AttributeRemover.

[RayZhao1998]

I've made some changes. But as mentioned in the issue, it fails testTypeWrapperTransform now. Please help me check whether this test is correct.

[ahoppen]

That’s why I mentioned

Instead of repeating the logic of which attributes we need to remove, I think we should collect the AttributeSyntax nodes of the attributes that we did indeed expand in MacroApplication and then remove those nodes for the AttributeRemover.

The problem here is that we are essentially duplicating the logic of whether a macro can be applied to a given declaration here and that logic doesn’t quite match what the rest of MacroApplication does.

[RayZhao1998]

Sorry, I misunderstood before.

I ran into a problem. Is there a way to check equal between two AttributeSyntax nodes? It seems the AttributeSyntax.id will change after expanding. Now I'm using the attributeName to compare but I don't think it's a reasonable way.

[ahoppen]

Ah, I forgot about that. A proper way of doing this might be to productize #2118 but I don’t think that’s happening in the near future.

I think macro expansions always add code after the location of the macro attribute. So, we should be able to use the attribute’s position to identify it.

[ahoppen]

Do we even need to check isInvalidAttachedMacro here? I would think that we should diagnose all macro attributes that didn’t actually expand. Or am I missing something?

[RayZhao1998]

I found out how it diagnose when AccessorMacro is applied to struct in the compiler and imitated it. Also the diagnostics I'v implemented below is just for attached macros(actually only AccessorMacro).

If we should diagnose all macro attributes that didn't expand, what error messages should we emit, a common error message or the same as what compiler emit? Also I don't know all the cases that attribute won't expand.

[RayZhao1998]

I made some changes. I've abstracted a method called diagnosticForUnexpandedMacro. Currently, it only implements diagnostics for AccessorMacro. The rest can be left for others to implement.

[ahoppen]

I don’t think we should make this method public. The entire logic with allowBlockNames is very specific to SwiftParserDiagnsotic. Using SyntaxKind.nameForDiagnostics would probably be the better choice.

[RayZhao1998]

Using SyntaxKind.nameForDiagnostics would probably be the better choice.

How to make nameForDiagnostics public? I found the file is generated.

By the way, if the files are generated, should we add generated files to .gitignore?

[ahoppen]

The files are being generated as described here:

https://github.com/apple/swift-syntax/blob/main/CONTRIBUTING.md#generating-source-code

We need to commit the generated files to resolve a circular dependency: The sources are being generated using swift-syntax itself, which means that you need a version of swift-syntax to generate the sources of swift-syntax. If the generated sources weren’t committed anywhere, you couldn’t build swift-syntax.

[ahoppen]

Is removing necessary for correctness or do we just do it to keep the number of elements in self.expandedAttributes low (which is a good reason on its own). Would be worth a comment though, I think.

[RayZhao1998]

It will be incorrect if it's possible that two attributes with same position are actually different during the process. For example, the list still contains an attribute expanded in the last visit, and in the same position there is attribute not expanded in this visit, I think it'll go wrong.

I think we should remove it from the list if it has been checked to make sure it's correct

[ahoppen]

Ah, I thought we were fine because macros only ever introduce code after the the macro attribute itself, so I thought that the position would still uniquely identify a macro within the source file but thinking about it again, I realize that isn’t true because the macro itself gets removed, which shifts positions forward and thus allows another macro to have the same position.

For example, you could have the following

@NoOpMacroAB
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
sturct Foo {
func foo() {}
}

which expands to

sturct Foo {
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
func foo() {}
}

And the two @MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro have the same location despite being two different macro expansions.

---

And yes, I think with the removal of attributes from self.expandedAttributes, this should be fine because we always clear that list whenever removing attributes.

I hope that I’m making sense here. I think it would be good to have a test case for what I described above and make sure that we do actually diagnose that MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro can’t be attached to func foo(). Also, feel free to pick a shorter name for MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro. I only picked it to convey the meaning of the attribute without writing an implementation.

[ahoppen]

Thanks for your continued work on this @RayZhao1998. I think together we are iterating towards a good solution here. Reading through it again, I have some more comments. I think the main one is the one I left on the declaration of expandedAttributes. Could you check if the idea I posted there works or if I missed something?

[ahoppen]

Could you add labels for these associated values so it’s easier to tell what the two strings do?

[ahoppen]

Are these imports still necessary if you use nameKindForDiagnostics? Same for the dependency declaration in the package manifest.

[RayZhao1998]

Yes, it doesn't need any more.

[ahoppen]

Suggested change

	if (!attachedTo.is(VariableDeclSyntax.self) && !attachedTo.is(SubscriptDeclSyntax.self)) {
	if !attachedTo.is(VariableDeclSyntax.self) && !attachedTo.is(SubscriptDeclSyntax.self) {

[ahoppen]

I think your error message '\(macroType)' macro cannot be attached to \(declType) is sufficiently generic that we could apply it for all cases, not just the accessor macro attached to a non-variable and non-subscript case.

[RayZhao1998]

If that's the case, I believe there's no need to diagnose based on the type of macroType or the type of DeclSyntax. A unified diagnosis with message '\(macroType)' macro cannot be attached to \(declType) will suffice.

[ahoppen]

Ah, I thought we were fine because macros only ever introduce code after the the macro attribute itself, so I thought that the position would still uniquely identify a macro within the source file but thinking about it again, I realize that isn’t true because the macro itself gets removed, which shifts positions forward and thus allows another macro to have the same position.

For example, you could have the following

@NoOpMacroAB
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
sturct Foo {
func foo() {}
}

which expands to

sturct Foo {
@MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro
func foo() {}
}

And the two @MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro have the same location despite being two different macro expansions.

---

And yes, I think with the removal of attributes from self.expandedAttributes, this should be fine because we always clear that list whenever removing attributes.

I hope that I’m making sense here. I think it would be good to have a test case for what I described above and make sure that we do actually diagnose that MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro can’t be attached to func foo(). Also, feel free to pick a shorter name for MacroThatExpandsToMemberAttributesMacroAndWhichIsAlsoAnAccessorMacro. I only picked it to convey the meaning of the attribute without writing an implementation.

[ahoppen]

I think this can be private.

---

Also, I just tried to suggest a doc comment for it and couldn’t really come up with a great explanation, which is usually a sign that the design isn’t quite right (maybe you can find a good explanation). Thinking about it made me wonder if it would it be easier to

• Change this variable to

/// Attributes that look like macro attributes and should be expanded. 
///
/// Attributes are added to this list from `visitAny`. Any attributes that weren’t actually expanded (eg. because they are accessor macros but not attached to a variable or subscript), will be diagnosed from `visitAny`.
var attributesToExpand: [AttributeSyntax] = []

• In visitAny set this variable to self.macroAttributes(attachedTo: declSyntax)
• Whenever we expand a macro, remove it from the list (I think we might even do this based on the ID now and don’t need to resort to the position).
• If there are macros left over in attributesToExpand, diagnose them.
• After visitAny is done, restore attributesToExpand to the value it had before we assigned it to self.macroAttributes(attachedTo: declSyntax).

Do you think that would work?

[RayZhao1998]

I think we might even do this based on the ID now and don’t need to resort to the position

I found that the id is still different. The macroAttributes method depends on the attachedTo node. In visitAny the declSyntax's root is SourceFileSyntax, while in visit(for example visit(_ node: VariableDeclSyntax)) the node's root is VariableDeclSyntax(itself).

It seems that in visitChildren, the node will have a new layout and generate a new syntax without origin root info. Is this by design?

[RayZhao1998]

I also delved into the implementation of AttributeSyntax.position. It's the position of the start of this node's leading trivia(not the root). For Attribute, it always represents its index in the AttributeList. So I don't think it's a proper way to compare two attributes by position

[ahoppen]

I think I’ll need to play around with the code myself a little to figure out how we want to continue, but am too busy with other things at the moment. I’ll definitely come back to you but it might take a few weeks.

[RayZhao1998]

Hi @ahoppen , are there any updates on this?

[ahoppen]

I was super busy implementing background indexing in SourceKit-LSP over the last few weeks. I’m hoping that this will calm down now and I’ll allow me to get back to this PR this week.

[ahoppen]

Sorry for the delay, I found some time now. Looks like we were really close, the only issue was that in visit(_: VariableDeclSyntax) we needed to call macroAttributes(attachedTo:ofType:) on the original node, not the one that we already processed using super.visit.

Diff

diff --git a/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift b/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
index 23127070..f350c097 100644
--- a/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
+++ b/Sources/SwiftSyntaxMacroExpansion/MacroSystem.swift
@@ -669,7 +669,7 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
   /// Store expanded extension while visiting member decls. This should be
   /// added to top-level 'CodeBlockItemList'.
   var extensions: [CodeBlockItemSyntax] = []
-  var expandedAttributes: [AttributeSyntax] = []
+  /// Attributes that we are expecting to expand from `visitAny`. As macros are expanded, they should be removed from this list. Any attributes that are still left in this array after a `visitAny` call will generate diagnostics.
+  var attributesToExpand: [(attributeNode: AttributeSyntax, spec: MacroSpec)] = []
 
   init(
     macroSystem: MacroSystem,
@@ -707,6 +707,11 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
       let attributedNode = node.asProtocol(WithAttributesSyntax.self),
       !attributedNode.attributes.isEmpty
     {
+      let previousAttributesToExpand = attributesToExpand
+      defer {
+        attributesToExpand = previousAttributesToExpand
+      }
+      attributesToExpand = self.macroAttributes(attachedTo: declSyntax)
       // Apply body and preamble macros.
       if let nodeWithBody = node.asProtocol(WithOptionalCodeBlockSyntax.self),
         let declNodeWithBody = nodeWithBody as? any DeclSyntaxProtocol & WithOptionalCodeBlockSyntax
@@ -720,14 +725,8 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
       skipVisitAnyHandling.remove(Syntax(declSyntax))
 
       let attributesToRemove = self.macroAttributes(attachedTo: visitedNode)
-      for (attribute, spec) in attributesToRemove {
-        if let index = self.expandedAttributes.firstIndex(where: { expandedAttribute in
-          expandedAttribute.position == attribute.position
-        }) {
-          self.expandedAttributes.remove(at: index)
-        } else {
-          self.diagnosticForUnexpandedMacro(attribute: attribute, macroType: spec.type, attachedTo: declSyntax)
-        }
+      for attribute in attributesToExpand {
+        self.diagnosticForUnexpandedMacro(attribute: attribute.attributeNode, macroType: attribute.spec.type, attachedTo: declSyntax)
       }
       return AttributeRemover(removingWhere: { attributesToRemove.map(\.attributeNode).contains($0) }).rewrite(
         visitedNode
@@ -961,10 +960,10 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
     )
   }
 
-  override func visit(_ node: VariableDeclSyntax) -> DeclSyntax {
-    var node = super.visit(node).cast(VariableDeclSyntax.self)
+  override func visit(_ originalNode: VariableDeclSyntax) -> DeclSyntax {
+    var node = super.visit(originalNode).cast(VariableDeclSyntax.self)
 
-    guard !macroAttributes(attachedTo: DeclSyntax(node), ofType: AccessorMacro.Type.self).isEmpty else {
+    guard !macroAttributes(attachedTo: DeclSyntax(originalNode), ofType: AccessorMacro.Type.self).isEmpty else {
       return DeclSyntax(node)
     }
 
@@ -1061,6 +1060,7 @@ extension MacroApplication {
     var result: [ExpandedNode] = []
 
     for macroAttribute in macroAttributes(attachedTo: decl, ofType: ofType) {
+      attributesToExpand = attributesToExpand.filter { $0.attributeNode != macroAttribute.attributeNode }
       do {
         if let expanded = try expandMacro(
           macroAttribute.attributeNode,
@@ -1068,7 +1068,6 @@ extension MacroApplication {
           macroAttribute.conformanceList
         ) {
           result += expanded
-          self.expandedAttributes.append(macroAttribute.attributeNode)
         }
       } catch {
         contextGenerator(Syntax(decl)).addDiagnostics(from: error, node: macroAttribute.attributeNode)
@@ -1216,7 +1215,7 @@ extension MacroApplication {
               from: newAccessors,
               indentationWidth: self.indentationWidth
             )
-            self.expandedAttributes.append(macro.attributeNode)
+            attributesToExpand = attributesToExpand.filter { $0.attributeNode != macro.attributeNode }
           }
         } else if let newAccessors = try expandAccessorMacroWithoutExistingAccessors(
           definition: macro.definition,
@@ -1239,7 +1238,7 @@ extension MacroApplication {
           } else {
             newAccessorsBlock = newAccessors
           }
-          self.expandedAttributes.append(macro.attributeNode)
+          attributesToExpand = attributesToExpand.filter { $0.attributeNode != macro.attributeNode }
         }
       } catch {
         contextGenerator(Syntax(storage)).addDiagnostics(from: error, node: macro.attributeNode)

[RayZhao1998]

Thanks for your help.

@@ -961,10 +960,10 @@ private class MacroApplication<Context: MacroExpansionContext>: SyntaxRewriter {
     )
   }
 
-  override func visit(_ node: VariableDeclSyntax) -> DeclSyntax {
-    var node = super.visit(node).cast(VariableDeclSyntax.self)
+  override func visit(_ originalNode: VariableDeclSyntax) -> DeclSyntax {
+    var node = super.visit(originalNode).cast(VariableDeclSyntax.self)
 
-    guard !macroAttributes(attachedTo: DeclSyntax(node), ofType: AccessorMacro.Type.self).isEmpty else {
+    guard !macroAttributes(attachedTo: DeclSyntax(originalNode), ofType: AccessorMacro.Type.self).isEmpty else {
       return DeclSyntax(node)
     }

The changes of visit here are weird. When should we use originalNode and when to use the processed node?

Should we use originalNode when expandAccessor? If not, the comparison of attributedNode in expandAccessor will not be equal too.

[ahoppen]

Looking at the code again, I don’t understand what I did myself 🤦🏽 Do you have a test at hand that shows the issue? And I’m getting the feeling that instead of working around things ad-hoc here, the right solution is to productize #2118, which will allow us to keep track of the node’s in the original tree.

[RayZhao1998]

I've pushed my code. As we discussed above, unified diagnostic for unexpanded macros is already implemented. Because of the unexpected result of comparing two attributedNode, a lot of tests will fail.

Thinking back to the problem I encountered at the beginning, it was to compare whether two attributeNodes were the same in the origin syntax tree. So I agree with that #2118 should be the right solution.

[ahoppen]

OK, let me try and push for #2118 to get merged again, everything else seems like a fairly hacky solution. I’m sorry that this PR has already been dragged along for this long but getting #2118 ready might take another couple of weeks as we are currently busy finishing up the 6.0 release. Once that has calmed down, I will try to get #2118 merged and when that’s done, this PR should be trivial to implement and become the first user of the node tracking logic.