Resolve competing web module war definitions

Date	July 7 2020	Contacts	Jody Garnett
Status	proposed	Release	3.12
Resources		Ticket #	pending
Source code	pending
Funding	GeoCat, GeoNetwork Enterprise and professional services

Overview

The web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability.

This proposal refactors web module structure to resolve this issue.

This proposal should simplify the build experience for everyone, and preserve the developer workflow you are used to, and document the result so it is easier to understand where to make any changes.

Technical Details:

Initially proposed as a simple change to take advantage public OSGeo repository; this change has shown several limitations of the core-geonetwork build system for schema-plugins:

Use of schema-plugin version 3.7 across different branches of core-geonetwork, combined with transitive dependencies, ensure conflicts across active branches of core-geonetwork.
web app build copies contents folder to folder, rather than make use of schema plugin artifact zip and jar.
- The web app war and jetty configurations actually directly watch some folders across module boundaries (sidestepping the distinction between copying folder content or unpacking zip contents). This duplication gives some uncertainty when a change is noticed, and how to reliability pick up a change.
web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability
See "baseline" heading further down this page for a detailed review of the current build system

The above limitations result in an unstable build environment.

References:

Proposal: Refactor into distinct modules and setup seperate war and jetty staging

This approach:

Any required content is staged into a distinct target folder by process-resources
Respect that target/geonetwork is reserved for use by the war plugin (and jetty:run-war testing) and never stage content here directly
Make jetty:run a new folder for data_directory not in src (which is static) or target (which is subject to mvn clean).
Focuses on locking down the war definition using unpack to extract distinct dependencies zip artifacts managed by maven (with no copying between folders).
- Use of distinct target folders
- Use of a custom war assembly, if required, to produce exactly the war we want.
- There is an opportunity to use jetty:run-war to test the war build chain. While this would respond to mvn process-resources, it would not be intended for interactive development.
Allow the jetty configuration to work across folders
- In this workflow web-ui resources, and schema plugins folders would be used directly, no process-resources required.
- Care taken to only use the maven dependencies to take advantage of jetty ability to watch the maven reactor and notice mvn install being used.
- If this approach works using jetty:run would be quicker and just focused on rapid development, there would not be any need to use process-resources (just update files directly for html, javascript and xslt, or use mvn install for java files).

Discussion:

This proposal has the advantage of clearly supporting the two distinct workflows enjoyed by geonetwork developers.

This represent an improvement for developers as process-resources is no longer be required to pick up changes. However there is a duplication of effort, and there may be some frustration testing the war bundle as a result.

Example:
1. If you were using jetty:run and process-resources to try out a schematron rule change
2. And then packaged a war for a customer
3. You would of missed out including your rule change in the war
To fix:
1. Use mvn install to package the schema plugin into jar and zip files in the local maven repository
2. And then package war for customer (it will pick up the rule change which is provided by schema plugin zip artifact)
Consider use of web context to replace use of filters, have multiple jetty-context.xml files to test different dev, prod, inspire.

The use of jetty:run is primarily used with dev filter. The war:war makes use of prod and inspire filters are used to produce distinct war bundles.

This is a larger discussion, offering greater control to administrators, and will need to be the subject of its own proposal.
Q: The gn database is created in the current directory, can this move to target folder?

A: The only reason to keep in sources can be to avoid loosing the info with mvn clean, but maybe then can be configured a profile to externalize the data directory?
Q: Is it desirable to keep data geonetwork and gn database?

A: Yes, see above.
How to handle schema plugins sch files being compiled into xslt files when GeoNetwork first starts up?

If jetty:run is using schema plugin directly to ideally generate xlsx into a different location to avoid both the risk of accidental commit and accidentally including these files is generated war.

migrate

migrate module: a new module to package database migration into jar and zip artifacts.

src/main/java
src/main/resources

web-ui

Care needed to ensure zip artifact produced:

src/main/resources

web

The package-resources and compile stages use:

src/main/filters
src/main/java
src/main/resources
src/main/webapp
src/main/data
src/main/webResources
target/classes
target/doc - copied docs
target/schemas - schema plugins updated by mvn process-resources
target/webGenerated - generated webapp contents from filters and webResources

The war configuration uses:

src/assembly - custom war assembly to include license and avoid accidentally packaging contents
src/main/resources
src/main/webapp
src/main/data
target/classes
target/doc
target/schemas - schema plugins updated by mvn process-resources
target/webGenerated
target/migrate - unpacked content from migrate module
target/geonetwork/WEB-INF/lib/ - gn-web jar and dependencies staged here

The jetty configuration uses:

src/main/resources
src/main/webapp
test/java/org/fao/geonet/Start.java
test/resources/jetty-context.xml - these dev settings (assume inspire or prod not needed here)
test/webGenerated - generated webapp contents from filters/dev-env.properties and webResources
target/classes
data_directory/data/ - created from src/main/data for use by jetty
../web-ui/src/main/resources - direct use of web-ui files
../migrate/src/main/resources - direct use of sql files
../schemas/../**/plugin/ - direct use of schema plugins folders

Baseline GeoNetwork 3.10 web/pom.xml build process

The web module has quite a complex build chain supporting:

war creation of a geonetwork.war that includes schema plugins, a default data directory, and some environment specific customizations
jetty:run supporting something like:
- immediate recognition of changes to html files and javascript files
- delayed recognition of changes to XSLT files (using mvn process-resources)
- creation of gn h2 database in the web folder
- direct use of data directory in src/main/webapp

Background on Build Life Cycle

For this discussion keep in mind the maven build life cycle (common steps in italic, default war stages described):

validate
initialize
generate-sources
process-sources
generate-resources
process-resources - resources:resource goal
compile - compiler:compile goal
process-classes
generate-test-sources
process-test-sources
generate-test-resources
process-test-resources - resources:testResources goal
test-compile - compiler:testCompile goal
process-test-classes
test - surefire:test goal
prepare-package
package - war:war goal
pre-integration-test
integration-test
post-integration-test
verify
install - install:install goal
deploy - deploy:deploy goal

Each pom packaging type (pom, jar, war, ...) registers maven plugins to specific stages of the maven life cycle, along with appropriate plugin configuration. As an example war plugin configuration defines a default assembly definition that includes src/webapp.

References:

src folders

The war packaging for the module defines several src folders:

src/main/java

Define Geonetwork and other classes (compiled into target/classes)
src/main/resources

Define resources (compiled into target/classes)
src/webapp

Static outline of war contents.
- conversion
- doc
- htmlCache
- images
- loc - translations
- resources
- WEB-INF
  - web.xml
- xml
- xsl
- xslt
- geonetwork.css
- modalbox.css

The build generate

src/main/filters

Used to process webResources into target/webapp
src/webResources
- WEB-INF
Content processed into target/webapp

The generate-sources registers two additional src folders:

src/main/java - add-src folder containing Geonetwork class
src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate - add-src folder for sql migration

process-resources

The process-resources stages content into:

src/main/webapp/WEB-INF/data/config/schema_plugins

Collected from schema plugin content
src/webapp/doc - from documentation manuals
target/webapp

Copy filter src/main/webResource based on env configuration

compile

The compile stage builds:

target/classes - from java and resources

package

The maven-war-plugin generates jar:

target/geonetwork/WEB-INF/lib/web-app-3.11.0-SNAPSHOT.jar

Containing:
- src/main/java compiled classes
- src/main/resources contents
- src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate compiled classes
  
  This is unusual having java source code in src/webapp!

The maven-war-plugin generates war structure into target/geonetwork:

catalog
conversion
doc
htmlCache
images
loc
META-INF
resources
WEB-INF
xml
xsl
xslt
modalbox.css
geonetwork.css

Where content is collected from:

target/geonetwork: src/main/webapp files
target/geonetwork: target/webapp files that have been processed
target/geonetwork: ../web-ui/src/main/resources files
target/geonetwork/WEB-INF/data/config/schema_plugins: ../schemas files from each plugin folder
target/geonetwork/WEB-INF/lib/: maven dependencies
with many excludes ...
- xml/schemas/
- WEB-INF/data/*.db
- WEB-INF/data/index/**
- ...
These appear to be a safety measure to avoid including files produced when src/main/webapp is used as a live directory by jetty below.

Observations:

⚠️ The handling of sql is very unusual
The processing (using filters) of webResources appears to duplicate functionality that should be controlled by web container.
- Some work put into making the application configurable should remove the need to make custom wars for different environments.
- If needed for local testing a jetty context for each of these configurations would be a clean approach
⚠️ Working with the data directory in src/main/webapp results in lots of strange workarounds:
- Excludes in the war definition to avoid packaging "running" content
- Not quite documented process to "reset" your environment if you want a clean start
⚠️ : war definition including content form web-ui and schema plugins directly, while process-resource includes the same content

      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
            <manifestEntries>
              <Implementation-Build>${git.commit.id}</Implementation-Build>
            </manifestEntries>
          </archive>
          <archiveClasses>true</archiveClasses>
          <!-- Filter resources according to current profile properties
              (see src/main/filters) and copy them to the webapp -->
          <webResources>
            <resource>
              <directory>${project.basedir}/../schemas</directory>
              <includes>
                <include>**/src/main/plugin/*</include>
              </includes>
              <targetPath>WEB-INF/data/config/schema_plugins</targetPath>
            </resource>
            <resource>
              <directory>${project.basedir}/../web-ui/src/main/resources</directory>
            </resource>
            <resource>
              <directory>${build.webapp.resources}</directory>
            </resource>
          </webResources>
          <!-- <packagingExcludes>WEB-INF/data/**</packagingExcludes> -->
          <packagingExcludes>
            xml/schemas/**,
            WEB-INF/data/*.db,
            WEB-INF/data/index/**,
            WEB-INF/data/data/backup/**,
            WEB-INF/data/data/resources/htmlcache/**,
            catalog/lib/style/bootstrap/docs/**,
            catalog/lib/style/bootstrap/fonts/**,
            catalog/lib/style/bootstrap/grunt/**,
            catalog/lib/style/bootstrap/test-infra/**,
            catalog/lib/style/font-awesome/css/**,
            catalog/lib/style/font-awesome/src/**,
            catalog/lib/style/font-awesome/scss/**,
            catalog/lib/style/font-awesome/*.json,
            catalog/lib/style/font-awesome/Gemfile*,
            catalog/lib/style/font-awesome/.*,
            catalog/lib/style/font-awesome/*.txt,
            catalog/lib/style/font-awesome/*.md,
            catalog/lib/style/bootstrap/*.json,
            catalog/lib/style/bootstrap/Gemfile*,
            catalog/lib/style/bootstrap/.*,
            catalog/lib/style/bootstrap/*.txt,
            catalog/lib/style/bootstrap/*.md,
            catalog/lib/bootstrap-table/*.json,
            catalog/lib/bootstrap-table/Gemfile*,
            catalog/lib/bootstrap-table/.*,
            catalog/lib/bootstrap-table/*.txt,
            catalog/lib/bootstrap-table/*.md,
            catalog/lib/bootstrap-table/docs/**
          </packagingExcludes>
          <!--          <warSourceDirectory>src/main/geonetwork</warSourceDirectory> -->
          <webXml>${build.webapp.resources}/WEB-INF/web.xml</webXml>
          <attachClasses>true</attachClasses>
          <warName>${application.name}</warName>
          <webappDirectory>${project.build.directory}/geonetwork</webappDirectory>
        </configuration>
      </plugin>

jetty

The jetty-maven-plugin defines web application using:

maven dependencies
target/geonetwork
src/main/webapp
../web-ui/src/main/resources/
target/webapp
jett-context.xml: used to prevent jetty scanning jars during startup

Observations:

This configuration has been setup to use the "live" src/main/webapp location and pick up any changes each time mvn process-resources is called above.
⚠️ schemas plugin folders are included twice:
- in target/geonetwork via war definition
- in src/main/webapp/ via process-resource copy
⚠️ jars are included twice:
- target/geonetwork libs folder
- as maven dependency
- example: An error starting up (where domain jar has a conflict between jar included in libs and included by maven)

      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <configuration>
          <contextXml>${basedir}/jetty-context.xml</contextXml>
          <webAppSourceDirectory>${project.build.directory}/geonetwork</webAppSourceDirectory>
          <webApp>
            <contextPath>/${application.name}</contextPath>
            <descriptor>${project.build.directory}/WEB-INF/web.xml</descriptor>
            <baseResource implementation="org.eclipse.jetty.util.resource.ResourceCollection">
              <resourcesAsCSV>
                ${project.basedir}/src/main/webapp,
                ${rootProjectDir}/web-ui/src/main/resources/,
                ${build.webapp.resources}
              </resourcesAsCSV>
            </baseResource>
          </webApp>
          <systemProperties>
            <systemProperty>
              <name>org.eclipse.jetty.server.Request.maxFormContentSize</name>
              <value>5000000</value>
            </systemProperty>
          </systemProperties>
          <httpConnector>
            <port>${jetty.port}</port>
          </httpConnector>
          <stopKey>JETTY_TOP</stopKey>
          <stopPort>${jetty.stop.port}</stopPort>
        </configuration>
      </plugin>