-
-
Notifications
You must be signed in to change notification settings - Fork 489
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
20 changed files
with
1,546 additions
and
39 deletions.
There are no files selected for viewing
43 changes: 43 additions & 0 deletions
43
docs/manual/docs/user-guide/harvesting/harvesting-database.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # Database Harvesting {#database_harvester} | ||
|
|
||
| This harvesting type uses a database connection to harvest metadata stored in a database table. | ||
|
|
||
| ## Adding a Database harvester | ||
|
|
||
| To create a Database harvester go to `Admin console` > `Harvesting` and select `Harvest from` > `Database`: | ||
|
|
||
|  | ||
|
|
||
| Providing the following information: | ||
|
|
||
| - **Identification** | ||
| - *Node name and logo*: A unique name for the harvester and optionally a logo to assign to the harvester. | ||
| - *Group*: Group which owns the harvested records. Only the catalog administrator or users with the profile `UserAdmin` of this group can manage the harvester. | ||
| - *User*: User who owns the harvested records. | ||
|
|
||
| - **Schedule**: Scheduling options to execute the harvester. If disabled, the harvester should be executed manually from the harvesters page. If enabled a schedule expression using cron syntax should be configured ([See examples](https://www.quartz-scheduler.org/documentation/quartz-2.1.7/tutorials/crontrigger)). | ||
|
|
||
| - **Configure connection to Database** | ||
| - *Server*: The database server IP/Hostname. | ||
| - *Port*: The database port. For example, for Postgres usually 5432. | ||
| - *Database name*: The database name to connect. | ||
| - *Table name*: Table name with the metadata. The name must begin with a letter (a-z) or underscore (_). Subsequent characters in a name can be letters, digits (0-9), or underscores. | ||
| - *Metadata field name*: Table field name that contains the metadata. The name must begin with a letter (a-z) or underscore (_). Subsequent characters in a name can be letters, digits (0-9), or underscores. | ||
| - *Database type*: Database type. Currently supported Postgres and Oracle. | ||
| - *Remote authentication*: Credentials to connect to the database. | ||
|
|
||
| - **Search filter**: allows to define a simple field condition to filter the results. | ||
| - *Filter field*: Table field name used to filter the results. The name must begin with a letter (a-z) or underscore (_). Subsequent characters in a name can be letters, digits (0-9), or underscores. | ||
| - *Filter value*: Value to filter the results. It can contain wildcards (%). | ||
|
|
||
| - **Configure response processing for database** | ||
| - *Action on UUID collision*: When a harvester finds the same uuid on a record collected by another method (another harvester, importer, dashboard editor,...), should this record be skipped (default), overriden or generate a new UUID? | ||
| - *Validate records before import*: If checked, the metadata will be validated after retrieval. If the validation does not pass, the metadata will be skipped. | ||
| - *XSL filter name to apply*: (Optional) The XSL filter is applied to each metadata record. The filter is a process which depends on the metadata schema (see the `process` folder of the metadata schemas). | ||
|
|
||
| It could be composed of parameter which will be sent to XSL transformation using the following syntax: `anonymizer?protocol=MYLOCALNETWORK:FILEPATH&[email protected]&thesaurus=MYORGONLYTHEASURUS` | ||
|
|
||
| - *Batch edits*: (Optional) Allows to update harvested records, using XPATH syntax. It can be used to add, replace or delete element. | ||
| - *Translate metadata content*: (Optional) Allows to translate metadata elements. It requires a translation service provider configured in the System settings. | ||
|
|
||
| - **Privileges** - Assign privileges to harvested metadata. |
Binary file added
BIN
+8.09 KB
docs/manual/docs/user-guide/harvesting/img/add-database-harvester.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
73 changes: 73 additions & 0 deletions
73
...ers/src/main/java/org/fao/geonet/kernel/harvest/harvester/database/DatabaseHarvester.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| //============================================================================= | ||
| //=== Copyright (C) 2001-2024 Food and Agriculture Organization of the | ||
| //=== United Nations (FAO-UN), United Nations World Food Programme (WFP) | ||
| //=== and United Nations Environment Programme (UNEP) | ||
| //=== | ||
| //=== This program is free software; you can redistribute it and/or modify | ||
| //=== it under the terms of the GNU General Public License as published by | ||
| //=== the Free Software Foundation; either version 2 of the License, or (at | ||
| //=== your option) any later version. | ||
| //=== | ||
| //=== This program is distributed in the hope that it will be useful, but | ||
| //=== WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| //=== MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | ||
| //=== General Public License for more details. | ||
| //=== | ||
| //=== You should have received a copy of the GNU General Public License | ||
| //=== along with this program; if not, write to the Free Software | ||
| //=== Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA | ||
| //=== | ||
| //=== Contact: Jeroen Ticheler - FAO - Viale delle Terme di Caracalla 2, | ||
| //=== Rome - Italy. email: [email protected] | ||
| //============================================================================== | ||
|
|
||
| package org.fao.geonet.kernel.harvest.harvester.database; | ||
|
|
||
| import org.fao.geonet.Logger; | ||
| import org.fao.geonet.kernel.harvest.harvester.AbstractHarvester; | ||
| import org.fao.geonet.kernel.harvest.harvester.HarvestResult; | ||
|
|
||
| import java.sql.SQLException; | ||
|
|
||
| public class DatabaseHarvester extends AbstractHarvester<HarvestResult, DatabaseHarvesterParams> { | ||
| private static final String TABLE_NAME_PATTERN = "([_a-zA-Z]+[_a-zA-Z0-9]*)"; | ||
| private static final String FIELD_NAME_PATTERN = "([_a-zA-Z]+[_a-zA-Z0-9]*)"; | ||
|
|
||
| @Override | ||
| protected DatabaseHarvesterParams createParams() { | ||
| return new DatabaseHarvesterParams(dataMan); | ||
| } | ||
|
|
||
| @Override | ||
| protected void storeNodeExtra(DatabaseHarvesterParams params, String path, String siteId, String optionsId) throws SQLException { | ||
| // Remove non-valid characters | ||
| params.setTableName(params.getTableName().replaceAll("[^" + TABLE_NAME_PATTERN + "]", "")); | ||
| params.setMetadataField(params.getMetadataField().replaceAll("[^" + FIELD_NAME_PATTERN + "]", "")); | ||
| params.setFilterField(params.getFilterField().replaceAll("[^" + FIELD_NAME_PATTERN + "]", "")); | ||
|
|
||
| setParams(params); | ||
|
|
||
| harvesterSettingsManager.add("id:" + siteId, "icon", params.getIcon()); | ||
| harvesterSettingsManager.add("id:" + siteId, "server", params.getServer()); | ||
| harvesterSettingsManager.add("id:" + siteId, "port", params.getPort()); | ||
| harvesterSettingsManager.add("id:" + siteId, "username", params.getUsername()); | ||
| harvesterSettingsManager.add("id:" + siteId, "password", params.getPassword()); | ||
| harvesterSettingsManager.add("id:" + siteId, "database", params.getDatabase()); | ||
| harvesterSettingsManager.add("id:" + siteId, "databaseType", params.getDatabaseType()); | ||
| harvesterSettingsManager.add("id:" + siteId, "tableName", params.getTableName()); | ||
| harvesterSettingsManager.add("id:" + siteId, "metadataField", params.getMetadataField()); | ||
| harvesterSettingsManager.add("id:" + siteId, "xslfilter", params.getXslfilter()); | ||
|
|
||
| String filtersID = harvesterSettingsManager.add(path, "filter", ""); | ||
| harvesterSettingsManager.add("id:" + filtersID, "field", params.getFilterField()); | ||
| harvesterSettingsManager.add("id:" + filtersID, "value", params.getFilterValue()); | ||
| } | ||
|
|
||
| @Override | ||
| protected void doHarvest(Logger l) throws Exception { | ||
| log.info("Database harvester start"); | ||
| DatabaseHarvesterAligner h = new DatabaseHarvesterAligner(cancelMonitor, log, context, params); | ||
| result = h.harvest(log); | ||
| log.info("Database harvester end"); | ||
| } | ||
| } |
Oops, something went wrong.