Skip to content

Commit

Permalink
Merge branch 'main' into review_and_promote_asset_lineage_to_tech_pre…
Browse files Browse the repository at this point in the history
…view
  • Loading branch information
lpalashevski authored Jun 23, 2023
2 parents 1c6e0a8 + af492f7 commit dde9994
Show file tree
Hide file tree
Showing 911 changed files with 34,893 additions and 13,906 deletions.
4 changes: 2 additions & 2 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ open-metadata-implementation/governance-servers/ @mandy-chessell
open-metadata-implementation/governance-servers/data-engine-proxy-services/ @cmgrote @popa-raluca
open-metadata-implementation/governance-servers/open-lineage-services/ @marius-patrascu @popa-raluca
open-metadata-implementation/platform-services/ @mandy-chessell
open-metadata-implementation/server-chassis/ @mandy-chessell
open-metadata-implementation/server-chassis/server-chassis-spring/ @bogdan-sava @lpalashevski
open-metadata-implementation/platform-chassis/ @mandy-chessell
open-metadata-implementation/platform-chassis/platform-chassis-spring/ @bogdan-sava @lpalashevski
open-metadata-implementation/user-interface/ui-chassis/ @bogdan-sava @marius-patrascu
open-metadata-implementation/view-services/ @davidradl

Expand Down
2 changes: 1 addition & 1 deletion Content-Organization.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ The Egeria core repository contains the core Egeria functionality, and is organi

* **[repository-services](open-metadata-implementation/repository-services)** - metadata exchange and federation - aka the Open Metadata Repository Services (OMRS).

* **[server-chassis](open-metadata-implementation/server-chassis)** - the server chassis provides the server framework for the OMAG Server Platform.
* **[platform-chassis](open-metadata-implementation/platform-chassis)** - the platform chassis provides the runtime framework for the OMAG Server Platform.

* **[user-interfaces](open-metadata-implementation/user-interfaces)** - browser based user interfaces.

Expand Down
3 changes: 1 addition & 2 deletions application.properties
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ server.ssl.trust-store-password=egeria

# WARNING! setting 'strict.ssl=false' allows java clients to open https connections without checking the validity of
# certificates from the servers it is calling.
# Alternate you can import self signed certificates into java truststore or setup an truststore only for this app
# Alternate you can import self-signed certificates into java truststore or set up a truststore only for this app
# by adding the store into server.ssl.trust-store parameter
strict.ssl=true
# Comma separated values of http headers to be added to ThreadLocal
Expand Down Expand Up @@ -73,4 +73,3 @@ springdoc.swagger-ui.docExpansion=none
# Endpoints web configuration
#management.endpoints.web.exposure.include=*
management.health.cassandra.enabled=false

4 changes: 2 additions & 2 deletions developer-resources/Dependency-Management.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ This page covers the way dependencies are managed in Egeria.

### Logging dependencies for slf4j

* Any utility, sample, tool - or our server chassis, are 'applications' in that they have an entry point - typically main(). These should include a binding for slf4j. See http://www.slf4j.org/faq.html
* Any utility, sample, tool - or our OMAG Server Platform, are 'Java applications' in that they have an entry point - typically main(). These should include a binding for slf4j. See http://www.slf4j.org/faq.html
* We generally use logback (ch.qos.logback:logback-classic for example)
* Generally A configuration file should not be provided - default formatting will be used & can be overriden by logback configuration at deployment time.
* Generally A configuration file should not be provided - default formatting will be used & can be overridden by logback configuration at deployment time.
* Test code automatically includes slf4j-simple - a simple logging implementations
* Other code that forms libraries (most of our code) MUST NOT include a slf4j logging implementation. Otherwise the application loses control of the logging implementation, hidden config files can change behaviour, and a multiple_bindings issue will be raised by slf4j

Expand Down
84 changes: 84 additions & 0 deletions logback.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
<?xml version="1.0" encoding="UTF-8"?>

<!-- SPDX-License-Identifier: Apache-2.0 -->
<!-- Copyright Contributors to the ODPi Egeria project. -->

<!-- This logback file will be read every 60 seconds in case the configuration has changed. -->
<!-- Add scanPeriod option to change the refresh time. -->
<configuration scan="true">

<!-- This logback file sets up two logging destinations for the server identified by ${EGERIA_SERVER_1}. -->
<!-- It should be installed for each platform where this server is running. -->
<!-- If more servers are added to the platform then duplicate the appenders and loggers for each server. -->
<property name="LOG_DIR_PLATFORM" value="./data/platform/logs" />
<property name="EGERIA_SERVER_1" value="cocoMDS1" />
<property name="LOG_DIR_SERVER_1" value="./data/servers/${EGERIA_SERVER_1}/logs" />

<!-- This appender defines the destination of developer (debug) logging. -->
<appender name="PLATFORM_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_DIR_PLATFORM}/debug.log</file>
<!-- This filter only allows ERROR level log messages to be stored. -->
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>${LOG_DIR_PLATFORM}/debug.log.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
<maxFileSize>1000KB</maxFileSize>
<maxHistory>1000</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>

<!-- This appender defines the destination of Egeria's audit log records. -->
<!-- It requires the SLF4J audit log destination to be set up in the server's configuration. -->
<appender name="AUDIT_LOG_${EGERIA_SERVER_1}" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_DIR_SERVER_1}/audit.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<!-- daily rollover -->
<fileNamePattern>${LOG_DIR_SERVER_1}/audit.log.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
<maxFileSize>1000KB</maxFileSize>
<maxHistory>1000</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>

<!-- This appender defines the console (standard out) destination. -->
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder>
<pattern>%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}.%M:%L - %m%n</pattern>
</encoder>
</appender>

<!-- This logger defines the level of entries from Egeria's audit log records. Can be INFO or ERROR -->
<logger name="org.odpi.openmetadata.frameworks.auditlog.${EGERIA_SERVER_1}" level="INFO">
<appender-ref ref="AUDIT_LOG_${EGERIA_SERVER_1}" />
</logger>

<!-- This logger defines the level of entries for logging from the springboot chassis for the OMAG Server Platform. Can be TRACE, DEBUG, INFO, WARN and ERROR -->
<logger name="org.odpi.openmetadata.serverchassis.springboot" level="INFO">
<appender-ref ref="STDOUT" />
</logger>

<!-- This logger defines the level of entries for logging from Tomcat for the OMAG Server Platform. Can be TRACE, DEBUG, INFO, WARN and ERROR -->
<logger name="org.springframework.boot.web.embedded.tomcat" level="INFO">
<appender-ref ref="STDOUT" />
</logger>

<!-- This logger defines the level of entries for debug logging. Can be TRACE, DEBUG, INFO, WARN and ERROR -->
<root level="ERROR">
<appender-ref ref="PLATFORM_LOG" />
</root>

</configuration>


51 changes: 48 additions & 3 deletions open-metadata-distribution/open-metadata-assemblies/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ dependencies {

// Dependencies for the main server assembly - this is modeled on the pom.xml
// In future we may be able to select by type or label
implementation project(':open-metadata-implementation:server-chassis:server-chassis-spring')
implementation project(':open-metadata-implementation:platform-chassis:platform-chassis-spring')
implementation project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-console-connector')
implementation project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-event-topic-connector')
implementation project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-file-connector')
Expand Down Expand Up @@ -110,8 +110,11 @@ distributions {
//distributionBaseName = "${rootProject.name}-distribution"
contents {
into('server') {
// Just the chassis
from { project(':open-metadata-implementation:server-chassis:server-chassis-spring').bootJar }
// Just the chassis - for backward compatibility
from { project(':open-metadata-implementation:platform-chassis:platform-chassis-spring').bootJar }
rename { String fileName ->
fileName.replace("platform-chassis-spring", "server-chassis-spring")
}
fileMode = 0755
}
into('server/lib') {
Expand Down Expand Up @@ -148,6 +151,48 @@ distributions {
from { project(':open-metadata-resources:open-metadata-samples:governance-services-sample').jar }
fileMode = 0755
}
into('platform') {
// OMAG Server Platform - new name consistent with documentation
from { project(':open-metadata-implementation:platform-chassis:platform-chassis-spring').bootJar }
rename { String fileName ->
fileName.replace("platform-chassis-spring", "omag-server-platform")
}
fileMode = 0755
}
into('platform/lib') {
// All our connectors, plus discovery sample & the sample Coco security connectors
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-console-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-event-topic-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-file-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:audit-log-connectors:audit-log-slf4j-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:cohort-registry-store-connectors:cohort-registry-file-store-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:open-metadata-archive-connectors:open-metadata-archive-file-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:open-metadata-archive-connectors:open-metadata-archive-directory-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:open-metadata-collection-store-connectors:inmemory-repository-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:open-metadata-collection-store-connectors:graph-repository-connector').fatJar }
from { project(':open-metadata-implementation:adapters:open-connectors:repository-services-connectors:open-metadata-collection-store-connectors:omrs-rest-repository-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:data-store-connectors:file-connectors:avro-file-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:data-store-connectors:file-connectors:basic-file-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:data-store-connectors:file-connectors:csv-file-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:data-store-connectors:file-connectors:data-folder-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:configuration-store-connectors:configuration-encrypted-file-store-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:configuration-store-connectors:configuration-file-store-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:discovery-service-connectors').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:dynamic-archiver-connectors').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:governance-action-connectors').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:integration-connectors:files-integration-connectors').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:integration-connectors:kafka-integration-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:integration-connectors:openapi-integration-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:integration-connectors:openlineage-integration-connectors').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:integration-connectors:elasticsearch-integration-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:event-bus-connectors:open-metadata-topic-connectors:inmemory-open-metadata-topic-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:event-bus-connectors:open-metadata-topic-connectors:kafka-open-metadata-topic-connector').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:governance-daemon-connectors:open-lineage-connectors:open-lineage-janus-connector').jar }
from { project(':open-metadata-resources:open-metadata-samples:open-metadata-security-samples').jar }
from { project(':open-metadata-implementation:adapters:open-connectors:rest-client-connectors:spring-rest-client-connector').jar }
from { project(':open-metadata-resources:open-metadata-samples:governance-services-sample').jar }
fileMode = 0755
}
into('utilities') {
from { project(':open-metadata-resources:open-metadata-archives:design-model-archives:glossary-canonical-model').shadowJar }
from { project(':open-metadata-resources:open-metadata-archives:open-metadata-types-utility').shadowJar }
Expand Down
2 changes: 1 addition & 1 deletion open-metadata-implementation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ OMAG Servers that it is hosting.
implementation of the metadata exchange and federation capabilities for a metadata
repository that supports the open metadata standards.

* **[server-chassis](server-chassis)** - the server chassis is the base component for the OMAG Server Platform.
* **[platform-chassis](platform-chassis)** - the platform chassis is the base component for the OMAG Server Platform.
It includes the web server that receives the REST API requests for both the OMAG Server Platform
and the servers that run on it.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,31 +1,20 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the ODPi Egeria project. -->

![InDev](../../../images/egeria-content-status-in-development.png#pagewidth)
![TechPreview](../../../images/egeria-content-status-tech-preview.png#pagewidth)

# Asset Catalog Open Metadata Access Service (OMAS)

The Asset Catalog OMAS provides services to search for data assets including:
The Asset Catalog OMAS provides services to search for data assets.

* data stores
* event feeds
* APIs
* data sets

The search locates assets
based on the content of the Asset metadata itself and the metadata that links
to it. This includes:

* glossary terms
* schema elements
* assets
The search locates assets based on the content of the Asset metadata itself and the metadata that links
to it.

The Asset Catalog REST API supports:

* the retrieval of assets based on unique identifiers
* the retrieval of asset's relationships and classifications
* the retrieval of assets based on known classification or relationship
* to query for related assets and to retrieve an asset neighborhood
* to query for related assets
* the retrieval of assets based on their type

* [Documentation](https://egeria-project.org/services/omas/asset-catalog/overview)

Expand All @@ -43,12 +32,13 @@ The module structure for the Asset Catalog OMAS is as follows:
[cohort](https://egeria-project.org/concepts/cohort-member).
* support for the access service's API and its related event management.
* [asset-catalog-spring](asset-catalog-spring) supports the REST API using the [Spring](../../../developer-resources/Spring.md) libraries.

* [asset-catalog-topic-connector](asset-catalog-topic-connector) supports asynchronous messaging through a connector
and connector provider class

### Search solution
The search will return Assets, Glossary Terms and Schema Elements that match the search criteria.
The search will return entities that match the search criteria.
As the asset search is to be performed against on one or more repositories a search engine will be used.
The search will be performed using the existing properties of the asset, glossary terms and/or schema elements.
The search will be performed using the existing properties of the entity.
Indexing will be performed by the Asset Catalog OMAS according to supported zones.
The search result will contain: guid, name (name or displayName), description, qualifiedName, classifications, zoneMembership (the basic properties of the element).

Expand All @@ -58,10 +48,10 @@ This call is using the asset global identifier and the asset type.


![Figure 1: Search](docs/egeria-asset-search.png)
> Figure 1:Integration of search engine
> Figure 1: The integration of the search engine
###Other Services
Asset Catalog OMAS provides services to fetch the asset
### Other Services
Asset Catalog OMAS provides services to fetch the asset's:
* classifications
* relationships
* specific entities that connect two assets
Expand Down
Original file line number Diff line number Diff line change
@@ -1,19 +1,17 @@
<!-- SPDX-License-Identifier: CC-BY-4.0 -->
<!-- Copyright Contributors to the ODPi Egeria project. -->

todo:
describe the rest model
body for search and basic model -pic

# The Asset Catalog OMAS Model

The Asset Catalog OMAS API provides the shared Java classes between the
server and client. It contains the following packages:
* api - interface and listener for interacting with asynchronous events
* exception - specific OMAS exceptions
* model - that describes the Java POJO's shared between modules.
Also, here can be found the beans for the REST endpoints.



Details about exceptions used in Asset Catalog can be found [here](../exception/README.md)

----
License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@
import org.odpi.openmetadata.frameworks.connectors.ffdc.PropertyServerException;
import org.odpi.openmetadata.frameworks.connectors.ffdc.UserNotAuthorizedException;

/**
* AssetCatalogEventInterface is the interface that a client implements to
* register a listener to receive the events from the Asset Catalog OMAS's out topic.
*/
public interface AssetCatalogEventInterface {

/**
Expand Down
Loading

0 comments on commit dde9994

Please sign in to comment.