Java to EMF EPackage converters

[ideas-into-software]

@juergen-albert

This PR is a continuation of refactor of DTOToEPackageConverter code, related to "EMF model driven development of Jakarta RESTful OSGi applications" project you assigned to me.

That initial phase of refactor was shared via PR #60, which uses a separate branch ( JakartaRS-OSGi-Apps-EMF-MDE ) - please refer to that PR for more details regarding this inintial phase of refactor.

Since DTOToEPackageConverter started off as a static utility class, adding another converter, JavaReferenceTypeToEPackageConverter, which shares and builds upon most of that logic, necessitated extracting common parts as to avoid code duplication. That is one the points I mentioned during requirements elicitation phase for "EMF model driven development of Jakarta RESTful OSGi applications" project.

Unfortunately, since static methods cannot be overriden, I was unable to continue using this approach - i.e. having both converters be static utility classes and avoid duplicating most of the code.

Therefore, I created this PR ( #61 ) on a separate branch ( Converter-as-Singleton-Component), which continues with that initial refactor ( #60 ) using a different approach, which allows to easily share common logic, i.e.: extracted interface, moved logic shared between existing DTOToEPackageConverter and new JavaReferenceTypeToEPackageConverter to abstract class AbstractJavaToEPackageConverter, and changed both DTOToEPackageConverter and JavaReferenceTypeToEPackageConverter from static utility classes to OSGi singleton components.

In addition:

• support for sequential as well as extracted enum values and literals - this was especially needed for jakarta.ws.rs.core.Response.Status;
• support for converting methods to EOperations;
• new converter method, which allows to pass path to JAR ( and its dependencies ), whose classes should be converted to EPackage, in addition to existing method which accepts array of such classes;
• new integration test suite for JavaReferenceTypeToEPackageConverter;
• version information, which corresponds to "Jakarta RESTful WS API" version, added to generated dynamic EMF EPackage;

Please see new integration test suite for JavaReferenceTypeToEPackageConverter, where you will find two new test cases, through which you can test functionality of this new converter ( JavaReferenceTypeToEPackageConverter ). Attached is also output of one of those test cases, with generated dynamic EMF of entire "Jakarta RESTful WS API" serialized as jakarta.ws.rs-api-3.1.0.ecore.

Questions

1. Built-in Java classes (java.*), which are referred to from "Jakarta RESTful WS API" classes ( fields as well as methods' return types and input params, etc. ) are currently added to generated EMF model as EDataTypes, as I wanted to avoid having to built entire EClasses for such - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

2. All exception-related classes are currently added to generated EMF model as EDataTypes, for the same exact reason as mentioned in item no. 1 above - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

3. Should dedicated EDataTypes be created for array-types like Object[], Class[], Locale[], Variant[], Link[], NewCookie[] - similar to how such custom array-types those were defined in org.gecko.emf.converter.model ?

4. How should rationale behind extracting such common custom EDataTypes to org.gecko.emf.converter.model be applied to these cases mentioned in items no. 1-3 above ? I.e. during the course of generating such dynamic EMF different custom EDataTypes are created and added to generated EMF EPackage and such is serialized with no problems - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

5. Naming of inner classes is constructed via camel-casing coalesced ( dot removed ) full name of such class, less package name - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

6. Regarding this new converter method, which allows to pass path to JAR ( and its dependencies ), i.e. org.gecko.emf.converter.JavaToEPackageConverter.convert(String, String, String, Path, Path...), implemented in org.gecko.emf.converter.AbstractJavaToEPackageConverter.convert(String, String, String, Path, Path...) - perhaps there is a way to obtain such via classloader for a given bundle, after having added those to its bundle descriptor (.bnd) file, instead of passing absolute paths to JARs ? See org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertJakartaRESTfulWSAPI(ServiceAware<JavaToEPackageConverter>) for example of how this is used currently;

jakarta.ws.rs-api-3.1.0.ecore.zip

[juergen-albert]

1. Built-in Java classes (java.*), which are referred to from "Jakarta RESTful WS API" classes ( fields as well as methods' return types and input params, etc. ) are currently added to generated EMF model as EDataTypes, as I wanted to avoid having to built entire EClasses for such - please confirm / clarify whether this approach is acceptable to you or suggest different approach;
2. All exception-related classes are currently added to generated EMF model as EDataTypes, for the same exact reason as mentioned in item no. 1 above - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

This is totally okay.

[juergen-albert]

3. Should dedicated EDataTypes be created for array-types like Object[], Class[], Locale[], Variant[], Link[], NewCookie[] - similar to how such custom array-types those were defined in org.gecko.emf.converter.model ?

I suggest managed this via the upper Bound of the EAttribute. We can change this later, if we think this is necessay.

[juergen-albert]

How should rationale behind extracting such common custom EDataTypes to org.gecko.emf.converter.model be applied to these cases mentioned in items no. 1-3 above ? I.e. during the course of generating such dynamic EMF different custom EDataTypes are created and added to generated EMF EPackage and such is serialized with no problems - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

I suggest the following:
Get a ResourceSet injected and the scan the attached EPackagesRegistry and the EPackages provided for existing EDataTypes and reuse the ones provided there. EDataTypes you create because they don't exist already should then be part of the current EPackage.

[juergen-albert]

Naming of inner classes is constructed via camel-casing coalesced ( dot removed ) full name of such class, less package name - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

Makes sense.

[juergen-albert]

Regarding this new converter method, which allows to pass path to JAR ( and its dependencies ), i.e. org.gecko.emf.converter.JavaToEPackageConverter.convert(String, String, String, Path, Path...), implemented in org.gecko.emf.converter.AbstractJavaToEPackageConverter.convert(String, String, String, Path, Path...) - perhaps there is a way to obtain such via classloader for a given bundle, after having added those to its bundle descriptor (.bnd) file, instead of passing absolute paths to JARs ? See org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertJakartaRESTfulWSAPI(ServiceAware) for example of how this is used currently;

I pointed to the jar in the initial task as an example, and to provide a set of packages and classes to limit the scope. I thought more along the line of giving a package name or a list of classes to convert. You can find the classes in a Package e.g. with this method: https://www.baeldung.com/java-find-all-classes-in-package
From there you can start traversing the dependency tree as far as necessary.

[juergen-albert]

I looked at the ecore you provided and it looks largely as if we are on the right track. 2 Things jumped at me right away though:

1. Annotations should be marked as Interfaces.
2. The Converter should always produce a List of EPackages to represent the original Java Packages. The uri can be derived from the original full qualified package name. As an example:

jakarta.ws.rs 3.1.0 -
EPackage name: rs
nsUri: https://geckoprojects.org/jakarta/ws/rs/3.1.0/

If you add the GenModel -> basePackage - jakarta.ws as Annotation, we also don't loose the full name

[ideas-into-software]

3. Should dedicated EDataTypes be created for array-types like Object[], Class[], Locale[], Variant[], Link[], NewCookie[] - similar to how such custom array-types those were defined in org.gecko.emf.converter.model ?

I suggest managed this via the upper Bound of the EAttribute. We can change this later, if we think this is necessay.

Just for the record, those are mostly eOperations with eParameters, not eStructuralFeatures where this problem appears.

To summarize my understanding of what you suggest:

• for all such array-types ( i.e. Object[], Class[], Locale[], Variant[], Link[], NewCookie[], Annotation[], MediaType[]), do not create custom EDataTypes, instead
• use "single instance" equivalents as type in such cases and set upperBound to org.eclipse.emf.ecore.ETypedElement.UNBOUNDED_MULTIPLICITY;

So, for example:

<eClassifiers xsi:type="ecore:EClass" name="WebTarget">
(...)
  <eOperations name="request" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//InvocationBuilder">
    <eParameters name="arg0" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//MediaType[]"/>
  </eOperations>
(...)
</eClassifiers>

Would become:

<eClassifiers xsi:type="ecore:EClass" name="WebTarget">
  (...)
  <eOperations name="request" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//InvocationBuilder">
    <eParameters name="arg0" upperBound="-1" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//MediaType"/>
  </eOperations>
  (...)
</eClassifiers>

[ideas-into-software]

Regarding this new converter method, which allows to pass path to JAR ( and its dependencies ), i.e. org.gecko.emf.converter.JavaToEPackageConverter.convert(String, String, String, Path, Path...), implemented in org.gecko.emf.converter.AbstractJavaToEPackageConverter.convert(String, String, String, Path, Path...) - perhaps there is a way to obtain such via classloader for a given bundle, after having added those to its bundle descriptor (.bnd) file, instead of passing absolute paths to JARs ? See org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertJakartaRESTfulWSAPI(ServiceAware) for example of how this is used currently;

I pointed to the jar in the initial task as an example, and to provide a set of packages and classes to limit the scope. I thought more along the line of giving a package name or a list of classes to convert. You can find the classes in a Package e.g. with this method: https://www.baeldung.com/java-find-all-classes-in-package From there you can start traversing the dependency tree as far as necessary.

This is a misunderstanding, perhaps because you did not take a look at the code I shared via this PR, or at least relevant parts.

Class has to be loaded first before it can be passed for conversion, and there are interdependencies between classes, in addition to transitive dependencies, which prevent class to be loaded unless those conditions are met. My question was whether there is a better way that you know of, by accessing parent classloader, etc. ( see original question for details ) - what you mentioned does not address that question.

All this is already taken care of - as mentioned in PR details I shared ( #61 (comment) ):

(...)

• new converter method, which allows to pass path to JAR ( and its dependencies ), whose classes should be converted to EPackage, in addition to existing method which accepts array of such classes;

• new integration test suite for JavaReferenceTypeToEPackageConverter;

(...)

Please see new integration test suite for JavaReferenceTypeToEPackageConverter, where you will find two new test cases, through which you can test functionality of this new converter ( JavaReferenceTypeToEPackageConverter ). Attached is also output of one of those test cases, with generated dynamic EMF of entire "Jakarta RESTful WS API" serialized as jakarta.ws.rs-api-3.1.0.ecore.

(...)

Specifically, see:

• implementation of org.gecko.emf.converter.AbstractJavaToEPackageConverter.convert(String, String, String, Path, Path...) method;

• test case org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertJakartaRESTfulWSAPI(ServiceAware<JavaToEPackageConverter>), which calls that method, passing org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.JAKARTA_WS_RS_API_JAR_PATH and org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.JAKARTA_XML_BIND_API_JAR_PATH (one of such transitive dependencies);

[ideas-into-software]

How should rationale behind extracting such common custom EDataTypes to org.gecko.emf.converter.model be applied to these cases mentioned in items no. 1-3 above ? I.e. during the course of generating such dynamic EMF different custom EDataTypes are created and added to generated EMF EPackage and such is serialized with no problems - please confirm / clarify whether this approach is acceptable to you or suggest different approach;

I suggest the following: Get a ResourceSet injected and the scan the attached EPackagesRegistry and the EPackages provided for existing EDataTypes and reuse the ones provided there. EDataTypes you create because they don't exist already should then be part of the current EPackage.

Some follow-up questions on this topic:

a) Do you mean org.eclipse.emf.ecore.resource.ResourceSet.getPackageRegistry() ?

b) What about naming clashes ? I.e. EClassifier of same name existing in two ( or more ) EPackages ?

[ideas-into-software]

ad. "Annotations should be marked as Interfaces."

In addition to classes and annotations, there are also interfaces in "Jakarta RESTful WS API" - should those interfaces be also marked as interfaces ?

ad. "The Converter should always produce a List of EPackages to represent the original Java Packages"

"Jakarta RESTful WS API" defines the following packages:

• package jakarta.ws.rs
• package jakarta.ws.rs.client
• package jakarta.ws.rs.container
• package jakarta.ws.rs.core
• package jakarta.ws.rs.ext
• package jakarta.ws.rs.sse

As per your new requirement, the generated dynamic EMF of entire “Jakarta RESTful WS API”, when serialized, would contain the following enhancements:

  <eAnnotations source="http://www.eclipse.org/emf/2002/GenModel">
    <details key="basePackage" value="jakarta.ws"/>
  </eAnnotations>

(...)

  <eSubpackages name="rs" nsURI="https://geckoprojects.org/jakarta/ws/rs/3.1.0/">
    <eClassifiers xsi:type="ecore:EClass" name="SomeRsPackageClass1"/>
  </eSubpackages>
  <eSubpackages name="client">
    <eClassifiers xsi:type="ecore:EClass" name="SomeClientPackageClass1"/>
  </eSubpackages>
  <eSubpackages name="container">
    <eClassifiers xsi:type="ecore:EClass" name="SomeContainerPackageClass1"/>
  </eSubpackages>
  <eSubpackages name="core">
    <eClassifiers xsi:type="ecore:EClass" name="SomeCorePackageClass1"/>
  </eSubpackages>
  <eSubpackages name="ext">
    <eClassifiers xsi:type="ecore:EClass" name="SomeExtPackageClass1"/>
  </eSubpackages>
  <eSubpackages name="sse">
   <eClassifiers xsi:type="ecore:EClass" name="SomeSsePackageClass1"/>
  </eSubpackages>

It is not clear to me:

a) what should be defined as nsURI for all the other packages ?

b) should nsPrefix be defined at all ?

[juergen-albert]

3. Should dedicated EDataTypes be created for array-types like Object[], Class[], Locale[], Variant[], Link[], NewCookie[] - similar to how such custom array-types those were defined in org.gecko.emf.converter.model ?

I suggest managed this via the upper Bound of the EAttribute. We can change this later, if we think this is necessay.

Just for the record, those are mostly eOperations with eParameters, not eStructuralFeatures where this problem appears.

To summarize my understanding of what you suggest:

• for all such array-types ( i.e. Object[], Class[], Locale[], Variant[], Link[], NewCookie[], Annotation[], MediaType[]), do not create custom EDataTypes, instead
• use "single instance" equivalents as type in such cases and set upperBound to org.eclipse.emf.ecore.ETypedElement.UNBOUNDED_MULTIPLICITY;

So, for example:

<eClassifiers xsi:type="ecore:EClass" name="WebTarget">
(...)
  <eOperations name="request" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//InvocationBuilder">
    <eParameters name="arg0" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//MediaType[]"/>
  </eOperations>
(...)
</eClassifiers>

Would become:

<eClassifiers xsi:type="ecore:EClass" name="WebTarget">
  (...)
  <eOperations name="request" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//InvocationBuilder">
    <eParameters name="arg0" upperBound="-1" eType="ecore:EClass jakarta.ws.rs-api-3.1.0.ecore#//MediaType"/>
  </eOperations>
  (...)
</eClassifiers>

+1

[juergen-albert]

Some follow-up questions on this topic:

a) Do you mean org.eclipse.emf.ecore.resource.ResourceSet.getPackageRegistry() ?

yes.

b) What about naming clashes ? I.e. EClassifier of same name existing in two ( or more )

EClassifiers are unique inside their EPackage, similar to Java. If you find two EClassifiers, that would claim the same InstanceClass, I would simply use the first one found. If you need a unique Identifier for an EClassifier you can use EcoreUtil.getURI, which will produce a unique URI.

[juergen-albert]

a) what should be defined as nsURI for all the other packages ?

<eSubpackages name="client" nsURI="https://geckoprojects.org/jakarta/ws/rs/client/3.1.0/>
    <eClassifiers xsi:type="ecore:EClass" name="SomeClientPackageClass1"/>
  </eSubpackages>

b) should nsPrefix be defined at all ?

Yes, we need one, as it is mandatory for an EPackage. I suggest rs, rsClient, rsContainer and so on.

[juergen-albert]

Regarding your question with the dependent jar files. I understood what you asked of me. I simply stated, that I never intended to get a converter, we can hand a complete jar to convert.

Now that we have it though, we can later use it in the tooling. In the end, the tooling will need to provide all the dependent jars, as no one else will be able to guarantee that all dependent classes will be around. You can create the URLClassLoader you use with the current Classloader the converter has. This will give it at least access to the classes known to the current bundle. All other options like Dynamic Imports will be problematic at best.

[ideas-into-software]

@juergen-albert

Latest commits ( https://github.com/geckoprojects-org/org.gecko.emf.utils/pull/61/commits ) include all that you requested for this phase, including:

• 'nsURI' and 'nsPrefix' for each sub-package

• Instead of creating custom EDataTypes for array types, re-using "single instance" equivalents as type in such cases and set upperBound to 'ETypedElement.UNBOUNDED_MULTIPLICITY'

• Re-using EDataTypes from EPackagesRegistry attached to ResourceSet

• Using current Classloader when constructing URLClassLoader

.. in addition to refactoring related to above mentioned, as well as minor fixes and improvements.

Some additional clarifying questions, pertaining to implementation shared via recent commit ( 1117885 ) - see (a) and (b) below.

ad. "The Converter should always produce a List of EPackages to represent the original Java Packages" (c.d.)

(a)

In this requirement, as originally stated ( #61 (comment) ), you provided example of how names of EPackages should be constructed, which is logically incoherent with the other part of your statement "to represent the original Java Packages".

Since I could only choose one or the other, I based this implementation provided via recent commit ( 1117885 ) on original package names, only sanitized to make them compatible with ecore requirements (i.e. replacing dots with underscores).

I.e., if original package names of "Jakarta RESTful WS API" are the following:

• package jakarta.ws.rs
• package jakarta.ws.rs.client
• package jakarta.ws.rs.container
• package jakarta.ws.rs.core
• package jakarta.ws.rs.ext
• package jakarta.ws.rs.sse

.. then, EPackages in the generated dynamic EMF are named exactly the same, except for dot-to-underscores replacement, and not rs, rs.client, rs.container, etc.

Please see attached ecore file, generated via org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertJakartaRESTfulWSAPIAndSerializeToStaticEMF(ServiceAware<JavaToEPackageConverter>, ServiceAware<ResourceSet>) and clarify as needed / confirm this is sufficient; also attaching screenshot.

jakarta.ws.rs-api-3.1.0.ecore.zip

(b)

Continuing from (a), this one pertains to how basePackage is determined - see org.gecko.emf.converter.AbstractJavaToEPackageConverter.addBasePackageEAnnotation(EPackage, Class<?>...) for details of implementation I shared, but in short: since this is a public project and people will be "throwing" different JARs on this, please clarify what about JARs which contain packages from different namespaces - e.g. org.apache.felix:org.apache.felix.http.servlet-api:2.1.0, which contains packages from both jakarta.servlet and javax.servlet namespaces - and there are many JARs like such!

Please see attached ecore file, generated via org.gecko.emf.converter.tests.JavaReferenceTypeToEPackageConverterTest.testConvertOrgApacheFelixHttpServletApiAndSerializeToStaticEMF(ServiceAware<JavaToEPackageConverter>, ServiceAware<ResourceSet>) and clarify as needed / confirm this is sufficient; also attaching screenshot.

org.apache.felix.http.servlet-api-2.1.0.ecore.zip

[juergen-albert]

In EMF packages can't have a dot in their name, as the dot is the delimiter between subpackages.

I would do it as follows for e.g. the jakarta rs jar:

jakarta.ws.rs
jakarta.ws.rs.client
jakarta.ws.rs.container
jakarta.ws.rs.core
jakarta.ws.rs.ext
jakarta.ws.rs.sse

would result in the following EPackages (the |- marks a subpackage)
rs
|- client
|- container
|- core
|- ext
|- sse

So rs would be the smallest common denominator of all the packages you have in that example.

For your felix servlet api, there would be no common denominator. Thus the result would be two root packages. one starts with java and one with jakarta with subpackages as in my example above.

[juergen-albert]

for the last case: if you need a common root EPackage, simply call it root and give it some fixed URI. in such a case somebody needs to take the result and restructure it anyway.

[ideas-into-software]

@juergen-albert

Regarding dots - this was already fixed in commits ( https://github.com/geckoprojects-org/org.gecko.emf.utils/pull/61/commits ) several days ago. Please see updated comment #61 (comment) with examples and refer to that.

[juergen-albert]

Okay. but is your question answered by this comment?

[ideas-into-software]

@juergen-albert

No, it is not answered - you did not understand my question it seems. Please see screenshots and ecore files shared in #61 (comment) - what you propose does not address the fact that there are multiple other packages and cutting package names like you propose would result in (1) mixing classes from different name spaces (if using only last part of the package name ) and (2) makes it very difficult to correlate resulting generated ecore with what's included in the JAR itself.

Best would be familiarize yourself with examples share, and perhaps re-read those questions in that context ( partially re-phrased now in previous paragraph ) and provide answers this way.

Also, I need itemized list of what other changes are required to close this PR.

[juergen-albert]

I modified the ecore for the felix servlet api jar, to represent what I think the result should be.

org.apache.felix.http.servlet-api-2.1.0.zip

The same should work for the jakarta rs model as well.

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[ideas-into-software]

@juergen-albert

PR was marked as ready for review, as what you requested via #61 (comment) is now part of this PR ( 11c478e ).

Attached please see examples of ecore files generated for both EMF models we discussed so far - now, instead of flat list, you will find trees of EPackages, etc.

Regarding Version eAnnotation - I kept it at root level at it refers to version of JAR being converted. It would be confusing to add it on package level, as this usually means something different - i.e. when package-info.java is present and contains per-package API version, it is usually different from version of JAR itself.

---

jakarta.ws.rs-api-3.1.0.ecore.zip

org.apache.felix.http.servlet-api-2.1.0.ecore.zip