Formatter / Cache & landing pages #4299

fxprunayre · 2019-12-10T15:44:30Z

Landing pages apply to public records only and provides a quick access to pre-built HTML pages based on a formatter. It is disabled by default. Landing page's main goal is to provide persistent pages that can be used as DOI registered URL for a record.

Formatters

GeoNetwork provides an advanced cache of record view modes (aka formatters). While viewing records, the default view is based on the search service response and is built by the JS client application. No cache in action here. The advanced view is using a formatter and other custom views can also be created using formatter (eg. a view based on an editor configuration, JSON-LD). While using a formatter a first level cache is an in-memory one. Requests are added to a queue which then populate a file system cache.

The file system cache is persisted in the data directory under resources/formatter-cache. A database store the current content of the cache and a public and private folder store formatter outputs (usually HTML) based on record's privileges.

All formatters can be cached but it depends on URL parameters (PDF outputs are not cached). If the call for the formatter contains a parameter not in formatterAllowedParameters it will not be cached. Mind the parameters available in a formatter to properly cache a version which depends on parameters (2 URLs with slightly different parameters pointing to the same formatter may result in caching in the same file). The cache key depends on record id and formatter id - URL parameters are not part of the key for now.

Landing pages

A specific cache is added to store landing pages for public records. A landing page may be used as the main entry point for a record when registering a DOI for example. The landing page by default is a simple HTML page containing all record information as HTML plus a JSON-LD representation for better indexing by search engine. The landing page is using the xsl-view formatter by default but this can be customized.

The landing page cache can be populated using the API and is updated depending on record changes (cf. FormatterCachePublishListener which takes care of updating formatters from private to public folder depending on publication status - event is MetadataIndexCompleted). Those HTML pages may be served directly by a web server to allow direct/quick access to those pages (in this case the landing page folder has to be mounted as a web folder).

Only public records are stored in this landing page cache.

File structure is:

DATA_DIR/resources/htmlcache/landing-page/uuid.html

The configuration is made in config.properties:

landingPage.formatter=xsl-view
landingPage.language=${language.default}

When exposing the landing page folder directly to the web, it may be relevant to setup a redirect to the catalog in case the requesting page does not exist. This will trigger the landing page creation. For this, create a .htaccess file in the landing page folder:

# Landing page redirect. 
# If the landing page requested does not exist, 
# redirect to the catalogue which will check 
# if it really does not exist or needs to be generated.
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*).html$ /geonetwork/srv/api/records/$1 [L,R=301]

Other changes

Restore the caching mechanism for the xsl-view formatter.
Move ALLOWED_PARAMETERS to bean config as it depends on catalogue available formatters.
Add API to get cache info (eg. cache current size).
Restore the option to skip popularity when using a formatter
Add option to force a cache refresh when using a formatter
Avoid NPE when a formatter is called before the app init while trying to retrive lang code
Portal link. Values can be:
- custom url using {{uuid}} to be replaced by the record UUID. eg. http://another.portal.org/${uuid}
- default (will link to the main application)
- group (will link to the record groupowner website URL if any, default if not)

Documentation

This PR description will be added to https://geonetwork-opensource.org/manuals/3.8.x/en/customizing-application/creating-custom-view.html

GeoNetwork provides an advanced cache of record view modes (aka formatters). While viewing records, the default view is based on the search service response and is built by the JS client application. No cache in action here. The advanced view is using a formatter and other custom views can also be created using formatter (eg. a view based on an editor configuration, JSON-LD). While using a formatter a first level cache is an in-memory one. Requests are added to a queue which then populate a file system cache. The file system cache is persisted in the data directory under `resources/formatter-cache`. A database store the current content of the cache and a `public` and `private` folder store formatter outputs (usually HTML) based on record's privileges. All formatters can be cached but it depends on URL parameters (PDF outputs are not cached). If the call for the formatter contains a parameter not in `formatterAllowedParameters` it will not be cached. Mind the parameters available in a formatter to properly cache a version depending on parameters. The cache key depends on record id and formatter id - URL parameters are not part of the key for now. A new cache is added to store landing pages for public records. A landing page may be used as the main entry point for a record when registering a DOI for example. The landing page by default is a simple HTML page containing all record information as HTML plus a JSON-LD representation for better indexing by search engine. The landing page is using the `xsl-view` formatter by default but this can be customized. The landing page cache can be populated using the API and is updated depending on record changes (TODO: check on which events). Those HTML pages may be served directly by a webserver to allow direct/quick access to those pages (in this case the landing page folder has to be mounted as a web folder). Only public records are stored in this landing page cache. File structure is ``` DATA_DIR/resources/htmlcache/landing-page/uuid.html ``` * Restore the caching mechanism for the xsl-view formatter. * Move ALLOWED_PARAMETERS to bean config as it depends on catalogue available formatters. * Add API to get cache info (eg. cache current size). * A formatter may depend on request language? Define one language in the landing page config? * Web server direct access / If a landing page does not exist? Trigger a redirect to check if the record exist or not ?

…g to make the landing page creation asynch (to not extend indexing time).

…build lp for templates.

…r with custom parameters.

pvgenuchten · 2020-01-09T15:45:58Z

francois, does the landingpage (and htaccess rewrite rule) respect the request headers language and accept (encoding)? For uri's such as doi, it's a good practice to support multiple output encodings (xml,json,html) dependent from requested accept header.

fxprunayre · 2020-01-09T18:50:07Z

does the landingpage (and htaccess rewrite rule) respect the request headers language and accept (encoding)?

No. As described, the config define which formatter to use and which language:

The configuration is made in config.properties:

landingPage.formatter=xsl-view
landingPage.language=${language.default}

This work funded by Ifremer has for main goal to provide a static (and up to date) HTML view of all public records of a catalogue in order to make search engine indexing all properly (with the JSON-LD representation). Then you click on get more information button and you access the catalogue with the complete UI, formats, language selection ... If we start making a full cache of all combinations of language/encoding then it is another story. Here we focus on having a static, fast and up to date version of all publich records. Ifremer was using an ETL process to take care of this that will be replace by a custom landing page formatter.

It slows down a bit the indexing process (even if done in separate thread) that we should probably improve in near future but that another story too.

Define the full portal link. Values can be: * default (will link to the main application) * group (will link to the record groupowner website URL if any, default if not) * custom url using {{uuid}} to be replaced by the record UUID. eg. http://another.portal.org/${uuid}

cmangeat

seems there is a trouble at compilation time.

also, FormatterApi.getRecordFormattedBy could be broken in two parts, a reusable one and the HTTP/REST wrapper, the reusable one to be used for generating landing page (as anyway getRecordFormattedBy not really used thru http as a distributed service).

furthermore, as, as far as I understand, landing pages are accessed as static resources, not really using the webapp but relying on the web server, first level cache (memory level) of formatter cache is of no use for landing page, considering building landing page repository apart from formatter cache could make sense (change date FormatterCache logic also does not apply).

cmangeat · 2020-02-11T09:26:56Z

domain/src/main/java/org/fao/geonet/repository/OperationAllowedRepositoryImpl.java

+        final Root<OperationAllowed> root = query.from(OperationAllowed.class);
+        final Root<Metadata> metadataRoot = query.from(Metadata.class);
+
+        Predicate mdIdEquals = cb.equal(metadataRoot.get(Metadata_.id), root.get(OperationAllowed_.id).get(OperationAllowedId_.metadataId));


One should also use a join clause => not sure, join seems to be not "straight-forward"

cmangeat · 2020-02-11T09:29:34Z

services/pom.xml

    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-csv</artifactId>
    </dependency>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>wro4j</artifactId>


Why is it please needed at service side ?

cmangeat · 2020-02-11T09:32:38Z

services/src/main/java/org/fao/geonet/api/records/formatters/SchemaLocalizations.java

        final String lang2 = appContext.getBean(IsoLanguagesMapper.class).iso639_2_to_iso639_1(lang3);
+
+        String finalLang = lang3;


one should set the final modifier ?

cmangeat · 2020-02-11T09:41:52Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FormatterCache.java

+        }
+        if (StringUtils.isNotEmpty(landingPageFormatter)) {
+            final ServletContext servletContext = _context.getServlet().getServletContext();
+            _context.setAsThreadLocal();


_context (the one from geonet startup) shared (also overriding the previous one) in Thread from which comes index MetadataIndexCompleted event, an asynchronous way... I understand that we need a context to perform service call thru http if we plan to rely on a distributed services framework, but this is not what we are doing as it seems we need both to set the context in Thread before to use it to execute request. Furthermore, we are not really executing request, just in a certain way invoking a local spring service (i.e. formatService cannot be hosted on a different node/jvm).

(with "context = createServiceContext" in getRecordFormattedBy)

cmangeat · 2020-02-11T09:48:39Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FormatterCache.java

+     * in formattersToInitialize so they get loaded eg. XSLCache init
+     * @param context
+     */
+    public static void initializeFormatters(final ServiceContext context) {


Seems that this method is never used

cmangeat · 2020-02-11T10:18:13Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/Key.java

     * @param lang         the current ui language
     * @param formatType   the content type of the output
     * @param formatterId  the formatter used to create the output
     * @param hideWithheld if true then elements in the metadata with the attribute
     *                     gco:nilreason="withheld" are being hidden
     */
-    public Key(int mdId, String lang, FormatType formatType, String formatterId, boolean hideWithheld, FormatterWidth width) {
+    public Key(int mdId, String mdUuid, String lang, FormatType formatType, String formatterId, boolean hideWithheld, FormatterWidth width) {


Why does adding uuid in the key make it better, please ?

because you need it to write/remove the landing page file.

cmangeat · 2020-02-11T10:22:25Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FormatterCache.java

+    @Autowired
+    OperationAllowedRepository operationAllowedRepo;
+    @Autowired
+    DataManager dataManager;


nerver used ?

cmangeat · 2020-02-11T10:23:04Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FormatterCache.java

+    }
+
+
+    @Autowired


I am used to look for available beans at the beginning of the file

cmangeat · 2020-02-11T10:57:49Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FilesystemStore.java

            }
        }
    }

-    private void resize() throws SQLException, IOException {
+    private void resize(String mdUuid) throws SQLException, IOException {


Resize helps making actual cache twice time smaller removing older elements from it... Older elements and uuid of the currently caching metadata are NOT related.

cmangeat · 2020-02-11T11:17:02Z

services/src/main/java/org/fao/geonet/api/records/formatters/cache/FormatterCache.java

+                            FormatType.html,
+                            true,
+                            true,
+                            true,


Why please refreshing in this case ? If not, service call should be shared with fillLandingPageCache.

pvgenuchten · 2020-05-26T22:35:46Z

hi @fxprunayre, what's the status on this, it's an impressive work, would love to see that merged soon. I have some questions.

I didn't quite understand your statement: "It is disabled by default." As far as i know landingpages are enabled by default, and accessible via the sitemap. Where is this configuration?
xsl-view is derived from editor-configuration, therefore you can add a parameter to indicate which editor to use, basic, inspire or advanced. I would suggest to add this configuration option also for the landingpage.
the header and footer of landingpages are defined in https://github.com/geonetwork/core-geonetwork/blob/master/web/src/main/webapp/xslt/skin/default/skin.xsl. The idea with 'skin' in this url is that users could easily provide an alternative skin in /xslt/skin/forest/skin.xsl and then use some configuration (for example the view=default parameter in settings) to point to the alternative skin. Is that something to build on top of this PR?
it's probably more a formatter issue, but the display of xsl-view keeps having this weird html artifacts caused by the xslt used in formatters, < i / > is not allowed in html
Would be interesting to have on the landingpage an "add to map" button for any wm(t)s links, which opens the map page in the angular application with the selected layer loaded.
For those records which have a link with a dataset/service or parent/child record, would be interesting to retrieve the links and display them on the landingpage

fxprunayre · 2020-05-27T06:00:10Z

I didn't quite understand your statement: "It is disabled by default." As far as i know landingpages are enabled by default, and accessible via the sitemap. Where is this configuration?

Landing pages are always available via the formatter call. What is disabled by default is the possibility to have a permanent HTML cache of all public records ie. if the app is down, the HTML version is always available.

xsl-view is derived from editor-configuration, therefore you can add a parameter to indicate which editor to use, basic, inspire or advanced. I would suggest to add this configuration option also for the landingpage.

It is. For Sextant which is using this PR, we have a custom view with even custom styling depending on the group of the record.

    <landingPage.formatter>xsl-view?view=sextant&amp;template=sextant-summary-view&amp;portalLink=group</landingPage.formatter>

The idea with 'skin' i

For Sextant, we also experimented on top of that PR, the possibility to add a style based on the group ... and also customize the link of the "view in the portal" button. That is still work in progress.

it's probably more a formatter issue, but the display of xsl-view keeps having this weird html artifacts caused by the xslt used in formatters, < i / > is not allowed in html

Yep I thought all those were fixed. I'll have a look.

Would be interesting to have on the landingpage an "add to map" button for any wm(t)s links, which opens the map page in the angular application with the selected layer loaded.

This is more difficult because the add to map actions are all controled by the JS logic of the angular app ... not sure how to solve that easily.

For those records which have a link with a dataset/service or parent/child record, would be interesting to retrieve the links and display them on the landingpage

Same here. All that is done in JS so that's why it is important to have a quite visible button to open the record in the JS app.

it's an impressive work, would love to see that merged soon.

We are discussing with Christophe who is improving this a bit. We are not 100% convinced on where/when/how to produce the landing page. We would like to avoid altering indexing performance. We also discussed micro services - so having a separate app taking care of that kind of jobs would make sense - having an indexing machine not part of the main webapp could help ...

pvgenuchten · 2020-05-28T13:03:43Z

if the app is down, the HTML version is always available.

this is an interesting aspect. Should be in title of the PR

Would be interesting to have on the landingpage an "add to map" button for any wm(t)s links, which opens the map page in the angular application with the selected layer loaded.

This is more difficult because the add to map actions are all controled by the JS logic of the angular app ... not sure how to solve that easily.

We should reuse as much as possible to add-to-map of the angular app, so only add:
suggestion for each online resource:

<xsl:if test="contains(gmd:protocol/*/text(),"wms")">
<a href="catalog.search#/map?type=wms&amp;url={gmd:url/*/text()}&amp;layer={gmd:name/*/text()}">Add to map</a>
</xsl:if>

For those records which have a link with a dataset/service or parent/child record, would be interesting to retrieve the links and display them on the landingpage

Same here. All that is done in JS so that's why it is important to have a quite visible button to open the record in the JS app.

Machines will not follow the link to the js app, but it is important they detect that linkage to other records is available, i guess we can use backend code from record/uuid/related as part of landing page creation

We would like to avoid altering indexing performance.

agree, to me creation at initial request (and removal at update), similar to tile-cache, makes sense. third party can set up a caching mechanism (can be a search engine crawler).

fxprunayre · 2020-09-22T15:57:03Z

Not that much interest on this PR ? It has been used live in Sextant for a couple of months now - the main drawback is that it slow down indexing a bit.

Closing for now unless someone review it. We may move to another approach having a dedicated service to take care of building landing page ...

pvgenuchten · 2020-09-22T19:39:33Z

This could be a aspect of upcoming micro services refacture / ogc api records implementation

archaeogeek · 2021-01-04T10:41:49Z

@fxprunayre I'm really keen to use this. If it was re-opened, what sort of review do you still need- is it for code or for how it works in action? Could it be implemented in 3.10.x or is it mainly for 4?

fxprunayre · 2021-01-04T12:54:05Z

@fxprunayre I'm really keen to use this. If it was re-opened, what sort of review do you still need- is it for code or for how it works in action?

The code works, it has been used for a couple of months in Ifremer for Sextant (production currently based on 3.10.x).
BTW any extra code review would be good too.

Major "complain" so far on Ifremer instance is that it slows down indexing and that why we are looking into moving this to a specific app (synergy with OGC API Records micro service - we would have a dedicated app to index and populate the landing page cache). We did some experiment, but this is not planned yet.

Could it be implemented in 3.10.x or is it mainly for 4?

For now, this PR was mainly targeting 3.10.4~.

archaeogeek · 2021-01-04T14:20:13Z

@fxprunayre I'm happy to implement and test it in 3.10.4 and see how it impacts performance. I'm not going to be much good at a code review though!

fxprunayre · 2021-01-05T07:28:18Z

Sure you can try to run and check this PR. Also an issue we have on Ifremer side is that the landing page still depends on the webapp for the CSS and attachments. They would like to completely 'detach' the landing page from the app in case it is down.

I'm trying to push more to a dedicated service taking care of this. I added an item about this in the potential January sprint we will make and will try to find support to make progress on this topic. It could be good to hook a cache mechanism/events on top of OGC API Records or indexing service. https://github.com/geonetwork/core-geonetwork/wiki/GeoNetwork-UI-and-microservices-codesprint-January-2021

fxprunayre added this to the 3.10.0 milestone Dec 16, 2019

fxprunayre marked this pull request as ready for review December 16, 2019 08:26

fxprunayre added 14 commits January 8, 2020 16:34

Formatter / Cache & landing pages / Fix test context.

a7e67b7

Formatter / Cache & landing pages / On indexing.

d25f1cb

Formatter / Cache & landing pages / Add language parameter. Add confi…

f7fa08f

…g to make the landing page creation asynch (to not extend indexing time).

Formatter / Cache & landing pages / Log in propper logger and do not …

8bc6502

…build lp for templates.

Formatter / Cache & landing pages / Add build option.

34df4d3

Formatter / Cache & landing pages / Cleaning.

7a9877a

Formatter / Cache & landing pages / Add default webserver config.

46fb13c

Formatter / Cache & landing pages / Add default webserver config.

ca24587

Formatter / Cache & landing pages / Fix config

5b293bb

Formatter / Cache & landing pages / Cleanup config

f8bf32e

Formatter / Cache & landing pages / Cleanup

6e7f7ac

Formatter / Cache & landing pages / Add possibility to use a formatte…

44b1650

…r with custom parameters.

Formatter / Cache & landing pages / Fix build.

84f018b

fxprunayre force-pushed the landingpagecache branch from 411c38f to 84f018b Compare January 8, 2020 15:35

fxprunayre requested a review from josegar74 January 8, 2020 15:36

fxprunayre modified the milestones: 3.10.0, 3.10.1 Jan 17, 2020

fxprunayre requested a review from cmangeat January 29, 2020 12:13

cmangeat reviewed Feb 11, 2020

View reviewed changes

fxprunayre modified the milestones: 3.10.1, 3.10.2 Feb 12, 2020

fxprunayre added 3 commits March 3, 2020 15:33

Merge remote-tracking branch 'upstream/master' into landingpagecache

f4f4bd7

Sitemap / Only list public records.

b36f3b2

Formatter cache / Resize does not use mduuid.

8f6e426

fxprunayre added 3 commits March 3, 2020 15:39

Fix build.

ad43924

Landing page / Create a queue of record's landing page to update.

0fd755f

Landing page / Create a queue of record's landing page to update.

19f6b79

fxprunayre modified the milestones: 3.10.2, 3.10.3 Apr 8, 2020

josegar74 modified the milestones: 3.10.3, 3.10.4 Jun 25, 2020

fxprunayre closed this Sep 22, 2020

fxprunayre mentioned this pull request Sep 22, 2020

[WIP] Landing pages with key factory #4734

Closed

fxprunayre mentioned this pull request Mar 18, 2021

Landing page / Improve multilingual support #4924

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Formatter / Cache & landing pages #4299

Formatter / Cache & landing pages #4299

fxprunayre commented Dec 10, 2019 •

edited

Loading

pvgenuchten commented Jan 9, 2020

fxprunayre commented Jan 9, 2020

cmangeat left a comment

cmangeat Feb 11, 2020 •

edited

Loading

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

fxprunayre Mar 4, 2020 •

edited

Loading

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

cmangeat Feb 11, 2020

pvgenuchten commented May 26, 2020 •

edited

Loading

fxprunayre commented May 27, 2020

pvgenuchten commented May 28, 2020 •

edited

Loading

fxprunayre commented Sep 22, 2020

pvgenuchten commented Sep 22, 2020

archaeogeek commented Jan 4, 2021

fxprunayre commented Jan 4, 2021

archaeogeek commented Jan 4, 2021

fxprunayre commented Jan 5, 2021

		final String lang2 = appContext.getBean(IsoLanguagesMapper.class).iso639_2_to_iso639_1(lang3);

		String finalLang = lang3;

Formatter / Cache & landing pages #4299

Formatter / Cache & landing pages #4299

Conversation

fxprunayre commented Dec 10, 2019 • edited Loading

Formatters

Landing pages

Other changes

Documentation

pvgenuchten commented Jan 9, 2020

fxprunayre commented Jan 9, 2020

cmangeat left a comment

Choose a reason for hiding this comment

cmangeat Feb 11, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fxprunayre Mar 4, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pvgenuchten commented May 26, 2020 • edited Loading

fxprunayre commented May 27, 2020

pvgenuchten commented May 28, 2020 • edited Loading

fxprunayre commented Sep 22, 2020

pvgenuchten commented Sep 22, 2020

archaeogeek commented Jan 4, 2021

fxprunayre commented Jan 4, 2021

archaeogeek commented Jan 4, 2021

fxprunayre commented Jan 5, 2021

fxprunayre commented Dec 10, 2019 •

edited

Loading

cmangeat Feb 11, 2020 •

edited

Loading

fxprunayre Mar 4, 2020 •

edited

Loading

pvgenuchten commented May 26, 2020 •

edited

Loading

pvgenuchten commented May 28, 2020 •

edited

Loading