review: feature: introduce the concept of "Pattern"

[pvojtechovsky]

This PR introduces a new concept Pattern, which is an improvement of TemplateMatcher, it allows to match EVERY attribute of Spoon model - not only a subset like current TemplateMatcher.

The concept of Pattern is like this:

• Pattern consists of List of RootNodes. Main implementation of RootNode is ConstantNode, which handles constant values of attributes and ElementNode, which knows the node type and a Map<CtRole, RootNode> of nodes for handling of specific attributes. So ElementNodes and ConstantNodes together makes up the tree of the pattern - it mirrors to be generated/matched Spoon AST.
• The variables in Pattern are represented by ParameterNode which links ParameterInfo, which defines which value from Map of parameters (ParameterValueProvider) has to substituted.
• Structure of Pattern is usually created by copying/mirroring of an example template Spoon AST, but after it is initialized the Pattern processing is based on implementations of RootNode interface only. It doesn't need origin Spoon AST - so the matching and generating algorithms are much simpler and more flexible then before. The pattern can be created dynamically by code. No need to declare it in source as Template class. But there is of course a way how to create Pattern from legacy Template class too because in some cases it is possible. More complex patters can be first created from Template (like before) and then adapted by extra PatternBuilder calls to adjust things which were not possible with legacy Templates.
• The Patterns are created by PatternBuilder, which accept any spoon AST. Even the legacy Template can be used to initialize Pattern. It is subject of future work to make a PatternBuilder intuitive and usable for clients.

Both algorithms generating of code from template and matching code by template are primary behavior of new patterns and are 100% compatible. Everything what can be generated can be matched and opposite.

[monperrus]

Cool! Could you give an example snippet of how to use this?

[pvojtechovsky]

Here are some examples
A)
https://github.com/INRIA/spoon/pull/1686/files#diff-dfa8a4b46f7408d58548e5797a9239e1R30
B)
https://github.com/INRIA/spoon/pull/1686/files#diff-b5d51a0eb70403715f1134a17f6fce51R31
C)
https://github.com/INRIA/spoon/pull/1686/files#diff-25f39a55233b37bb5f826e3ad922186cR23
https://github.com/INRIA/spoon/pull/1686/files#diff-70de42aee1116615f166a1b492efa1b0R19

[monperrus]

could you explain as comments what it's meant to do?

[pvojtechovsky]

Here is example of code which is matched by pattern above

try (ListPrinter lp = elementPrinterHelper.createListPrinter(false, "(", false, false, ",", true, false, ")")) {
	for (Entry<String, CtExpression> e : annotation.getValues().entrySet()) {
		lp.printSeparatorIfAppropriate();
		if ((annotation.getValues().size() == 1 && "value".equals(e.getKey())) == false) {
			//it is not a default value attribute. We must print a attribute name too.
			printer.writeIdentifier(e.getKey()).writeSpace().writeOperator("=").writeSpace();
		}
		elementPrinterHelper.writeAnnotationElement(annotation.getFactory(), e.getValue());
	}
}

The Pattern match provides a actual parameter values for each defined parameter. Such parameters + another pattern can be used to generate new code based on the actual values and then replace old code by new code -> to do a refactoring based on the pattern matching/generating. See CodeReplaceTest#testTemplateReplace for running example. This example transforms the matched code above by newly generated code like this:

elementPrinterHelper.printList(annotation.getValues().entrySet(), false, "(", false, false, ",", true, false, ")"), e -> {
	if ((annotation.getValues().size() == 1 && "value".equals(e.getKey())) == false) {
		//it is not a default value attribute. We must print a attribute name too.
		printer.writeIdentifier(e.getKey()).writeSpace().writeOperator("=").writeSpace();
	}
	elementPrinterHelper.writeAnnotationElement(annotation.getFactory(), e.getValue());
});

The example consists of these files:

• OldPattern.java - defines Pattern of old code
• NewPattern.java - defines Patter of new code
• NewPattern#replaceOldByNew(CtElement) - the method which does the replacement

	/**
	 * Looks for all matches of Pattern defined by `OldPattern_ParamsInNestedType.class`
	 * and replaces each match with code generated by Pattern defined by `NewPattern.class`
	 * @param rootElement the root element of AST whose children has to be transformed
	 */
	@SuppressWarnings("unchecked")
	public static void replaceOldByNew(CtElement rootElement) {
		CtType<?> targetType = (rootElement instanceof CtType) ? (CtType) rootElement : rootElement.getParent(CtType.class);
		Factory f = rootElement.getFactory();
		Pattern newPattern = NewPattern.createPattern(f);
		Pattern oldPattern = OldPattern.createPattern(f);
		oldPattern.forEachMatch(rootElement, (match) -> {
			RoleHandler rh = RoleHandlerHelper.getRoleHandlerWrtParent(match.getMatchingElement(CtElement.class, false));
			match.replaceMatchesBy(newPattern.applyToType(targetType, (Class) rh.getValueClass(), match.getParametersMap()));
		});
	}

It really works now. The tests are passing well.

[pvojtechovsky]

I was wrong. Adding of the removed part of inlined if test did not helped to cover more lines. :-)
So I made new test, which

• generates code using if, where one can select what is generated using a parameter
• this pattern is made in more complicated way (configurePatternParameters after configureInlineStatements) to cover even more lines.

What next?

[monperrus]

it seems that the next step is to merge #2016?

[pvojtechovsky]

Yes, I will work on it next days ... or if you would have time, I welcome any refactoring you do there. Just put note there so we are not doing it parallel. I will add comment into #2016 before I start too.

[pvojtechovsky]

I just merged master. Now I will check whether new metamodel is still fitting to Pattern needs and then will refactor Pattern to use new Metamodel.

[pvojtechovsky]

It is finished from my point of view.

Note: I moved hardcoded metamodel to test, so we can still check correctness of spoon metamodel.
See MetamodelTest.testRuntimeMetamodel - solution for #2047

[pvojtechovsky]

I moved MetamodelTest.testRuntimeMetamodel and related classes to #2050

[spoon-bot]

API changes: 4 (Detected by Revapi)

Old API: fr.inria.gforge.spoon:spoon-core:jar:6.3.0-20180609.225321-133 / New API: fr.inria.gforge.spoon:spoon-core:jar:6.3.0-SNAPSHOT

	Class was removed.
Old	class DefaultParameterMatcher
New	none
Breaking	binary: breaking,

	Class was removed.
Old	interface ParameterMatcher
New	none
Breaking	binary: breaking,

	Class was removed.
Old	class SubstitutionVisitor
New	none
Breaking	binary: breaking,

	The annotation no longer has an attribute.
Old	method Parameter#match()
New	none
Breaking	binary: breaking,

[monperrus]

Pavel, thanks a lot for this great contribution!

Having new and big features in Spoon is really awesome.

Next one on our roadmap: the sniper mode!

[pvojtechovsky]

Hallelujah :-)))

[monperrus]

There is a regression on the template side, in https://github.com/SpoonLabs/spoon-examples, see https://ci.inria.fr/sos/job/spoon-examples/507/console

Could you have a look at it?

[pvojtechovsky]

I will have a look in the evening ... if I make internet at home running after tonight storm ...